Deletion of contexts in 9.2

[jrh]

I have an application that creates instances of a context from a template and then deletes them over a period of time.  I notice that the deletion of contexts in the future release of newLISP will keep them around as "ghost" symbols and will require me to double-delete contexts to prevent them from filling up the symbol table with a bunch of garbage.



This is a pain.  What is the rationale behind changing the behavior of delete?

[Lutz]

This was necessary to avoid conflicts with another feature implemented: to be able to dynamically create contexts on local variables in functions.



Generally I don't recommend the frequent dynamic creation/deletion of namespaces as it is not very efficient speed-wise. But it all depends on the specifics of your program and may be the right approach in your case.



I would write a wrapper-function 'delete-context', which does the double duty of deleting the symbols inside the context and the deletion of the context itself.



Lutz

[jrh]

This was necessary to avoid conflicts with another feature implemented: to be able to dynamically create contexts on local variables in functions.

But can't one already do that with the sym statement?

Code Select Expand

> (setq myVar (sym (string "myCtx" 0 0 1)))
myCtx001
> (context myVar)
myCtx001
myCtx001>



[Lutz]

The original code was more elaborate, but this is the essential part of it:

Code Select Expand

; the template
(define ctx:var)

(define (foo x)
        (local (y)
                (new x 'y)
                (set 'y:var 123)
                (delete 'y) ; clean up namespace contents
        )
)

; pass the template to a function
; which creates a temporary namespace from that template
(foo 'ctx)


This creates a temporary namespace in 'y. On v.9.1.1 you can execute this only once and get a protection error the second time. If you allow cleanup you would self-modify the code and crash.



Again, I think it is a bad idea to use contexts this way, but the above code seemed reasonable in the application it was used. newLISP is not an object oriented language but more a functional Lisp language. Use namespaces for moderate volatile data-objects and for structuring code. Lists are much more agile when defining transient data structures.



Lutz

[Jeff]

The documentation says that lists are not efficient beyond several thousand cells and to use contexts as hashes in these situations.



What should we do if we needed a significantly large, deeply nested list?

[Lutz]

Contexts are very efficient for random and/or associative read/write access when you have a large number of symbols. Creation, reading and writing of symbols is fast but deletion is not very efficient and slow in memory intensive apps.



Nested or not nested doesn't make much of a difference for newLISP, but lists are always walked through in a sequential manner (1), while symbol tables are binary-balanced trees and scaling log(n).



Here is a table to summmarize it:

Code Select Expand


        sequential access   random access   creation  deletion

lists    very fast           medium-slow(3)  fast      fast

arrays   fast                fast            n/a (2)   n/a (2)

contexts fast (3)            fast            fast      slow


(1) accessing/pushing/popping the last element of a list is optimized, pushing to the end is as fast as pushing onto

 the beginning of a list.



(2) the whole array is created or deleted, cannot delete individual cells



(3) sorted acces with dotree



In the end the best method is to time the algorithm and try different ways. When I work with name spaces I often try to put the creation and usage of the context into a script which gets loaded, does it's job and then just exits, throwing everything away. This way I don't have to go through lengthy symbol deletion.  



The area where I have used contexts most is in natural language analysis, building dictionaries with hundreds-thousands to several millions of symbols. For run-off-the-mill associative or random acces of a few hundred elemens I always use lists and association lists.



Lutz

[Dmi]

Reading this I suspect that the main problem resists in that

- There is a language-provided possibility of OO-programming but it is not cpu-efficient.

- There is a Lutz's declaration that newLISP is not for OO, because contexts are for hashes, modules, module instances.

- Many people coming in newLISP from the world of C++ and so where list/symbol processing is not so strong, but OO is the major feature.

- BUT! There is no good (or well known ;) example (possible in "Code patterns") which will show how the lispish functional programming technics may successfully replace well known OO technics using large object collections.



Unfortunately, I'm not a lisp expert and, from the other hand, my OO experience is in the far past. So I haven't a competition to write such issue :-

[jrh]

The area where I have used contexts most is in natural language analysis, building dictionaries with hundreds-thousands to several millions of symbols. For run-off-the-mill associative or random acces of a few hundred elemens I always use lists and association lists.

I find the use of contexts to encapsulate data structures such as large (1000+) lists or 100000 element arrays to be very efficient as I can access them by reference and avoid the overhead of copying them when they are manipulated.  Deletion seems speedy enough as the program never has more than a couple of dozen of these container contexts around at any one time.



Having to double delete them now when using the basic language syntax in order to support obscure and esoteric template strategies doesn't seem to me to be very "clean".

---

[Lutz]

Actually it is cleaner today than it was before. 'delete' was never allowed to delete protected symbols and context symbols are protected and where allowed to be deleted in one step. There was no safeguard as it exists for other protected symbols.



Making the deletion of contexts a double step is also more logical if you consider that two namespaces are involved. The namespace to which the context symbols is the handle, and the namespace to which context symbols itself belongs, which is 'MAIN'. The first delete deletes the symbols in the namespace of the context, after which it doesn't need to be protected anymore. The second delete deletes the context symbols itself.



So even without supporting that feature of transient contexts in lambda expressions this was good change.



Lutz

[jrh]

OK, I see your point.  After considering it it also makes sense to me because I often build the sym name I use for a context "object" dynamically so  breaking it down in two steps has a certain symmetry.