[bug?] deleting referenced context symbol

[hartrock]

Hello All,

hello Lutz,



further stress testing newLisp: is this a bug or a feature (possibly for efficiency reasons)?



This leads to an endless loop:

Code Select Expand


newLISP v.10.5.3 64-bit on Linux IPv4/6 UTF-8, options: newlisp -h

>
(context 'Ctx)
(context MAIN)
(context? Ctx)
(context 'Ctx2)
(context MAIN)
;;
(set 'Ctx:sub Ctx2)
;;
(push '("foo" "bar") Ctx:sub)
;;
;; not OK:
(delete 'Ctx2)
(assoc "foo" Ctx:sub) ; -> endless loop
;; copy/paste until here ;;

Ctx
MAIN
true
Ctx2
MAIN
Ctx2
(("foo" "bar"))
true
^CERR: received SIGINT - in function assoc
(c)ontinue, (d)ebug, e(x)it, (r)eset:x


But doubling the delete - after the first delete Ctx2 remains visible in the output of '(symbols)' - repairs the problem (gives a correct error message):

Code Select Expand


newLISP v.10.5.3 64-bit on Linux IPv4/6 UTF-8, options: newlisp -h

>
(context 'Ctx)
(context MAIN)
(context? Ctx)
(context 'Ctx2)
(context MAIN)
;;
(set 'Ctx:sub Ctx2)
;;
(push '("foo" "bar") Ctx:sub)
;;
;; OK:
(delete 'Ctx2)
(delete 'Ctx2)
(assoc "foo" Ctx:sub) ; -> expected error message
;; copy/paste until here ;;

Ctx
MAIN
true
Ctx2
MAIN
Ctx2
(("foo" "bar"))
true
true

ERR: list expected in function assoc : nil
>


Due to being related to symbol lookup there could be some relation to 'pop-assoc problem: feels like a bug' (http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4377#p21584">http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4377#p21584).





Best regards,

Stephan

[Lutz]

When deleting a context symbol, the first delete removes the context namespace contents and demotes the symbol to a normal mono-variable symbol. The second delete then removes the symbol completely from the symbol table - this will be added to the manual.



This behavior is necessary when using local variable symbols in functions as contexts.



Ps: pop-assoc problem is fixed here: http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/

[hartrock]

Hello Lutz,



thank you for your quick response.

When deleting a context symbol, the first delete removes the context namespace contents and demotes the symbol to a normal mono-variable symbol. The second delete then removes the symbol completely from the symbol table - this will be added to the manual.



This behavior is necessary when using local variable symbols in functions as contexts.

Second delete is able to nil Ctx:sub like first delete did for Ctx:aSymbol before (see session below).

Couldn't this been used somehow for avoiding the endless loop just after first delete?



Best regards,

Stephan

Code Select Expand


newLISP v.10.5.3 64-bit on Linux IPv4/6 UTF-8, options: newlisp -h

>
(context 'Ctx)
(context MAIN)
;;
(context 'Ctx2)
(context MAIN)
;;
;;
"set:"
(set 'Ctx:aSymbol 'someSymbol)
(set 'Ctx:sub Ctx2)
;;
;;
"before delete:"
(list "Ctx:aSymbol" Ctx:aSymbol "Ctx:sub" Ctx:sub)
(list (context? Ctx2) (context? Ctx:sub))
;;
"1. delete:"
(list (delete 'Ctx2) (delete 'someSymbol))
(list "Ctx:aSymbol" Ctx:aSymbol "Ctx:sub" Ctx:sub)
(list (context? Ctx2) (context? Ctx:sub))
;;
"2. delete:"
(list (delete 'Ctx2) (delete 'someSymbol))
(list "Ctx:aSymbol" Ctx:aSymbol "Ctx:sub" Ctx:sub)
(list (context? Ctx2) (context? Ctx:sub))

Ctx
MAIN
Ctx2
MAIN
"set:"
someSymbol
Ctx2
"before delete:"
("Ctx:aSymbol" someSymbol "Ctx:sub" Ctx2)
(true true)
"1. delete:"
(true true)
("Ctx:aSymbol" nil "Ctx:sub" Ctx2)
(nil true)
"2. delete:"
(true true)
("Ctx:aSymbol" nil "Ctx:sub" nil)
(nil nil)
>


[Lutz]

On a general note: I wouldn't do any deleting of namespaces in newLISP program of non-trivial size. Except when using the nil flag in the delete command, symbols are checked for reference in the whole newLISP cell memory space, which can get very slow on bigger programs with bigger data in it.



Contexts are not meant to be much created and deleted. Allthough you can do it, there are more thought of as something stable, staying around for the whole execution of the program.



For that reason, in FOOP contexts are used mainly as classes holding methods, while data objects are normal LISP lists, which are memory managed automatically by newLISP. If you use contexts as objects their memory management is manual and not very efficient.

[hartrock]

On a general note: I wouldn't do any deleting of namespaces in newLISP program of non-trivial size. Except when using the nil flag in the delete command, symbols are checked for reference in the whole newLISP cell memory space, which can get very slow on bigger programs with bigger data in it.

OK: this fits to my assumption of efficiency reasons.

But what about this (to my use case see below):

Code Select Expand


(context 'Ctx)
(context MAIN)
(context 'Ctx2)
(context MAIN)
;;
(define (safe-delete symbolOrContext)
  (if (context? (eval symbolOrContext))
      (delete symbolOrContext))
  (delete symbolOrContext))
;;
(set 'Ctx:sub Ctx2)
;;
(push '("foo" "bar") Ctx:sub)
;;
;; works:
(safe-delete 'Ctx2)
(assoc "foo" Ctx:sub) ; -> expected error message

My point is to get delete safer without much cost. Checking for refs of one symbol in second delete could double the effort in the worst case, so the unchecked variant has its use cases, too.

Contexts are not meant to be much created and deleted. Allthough you can do it, there are more thought of as something stable, staying around for the whole execution of the program.



For that reason, in FOOP contexts are used mainly as classes holding methods, while data objects are normal LISP lists, which are memory managed automatically by newLISP. If you use contexts as objects their memory management is manual and not very efficient.

My current use case - very good for learning newLisp during implementing it, too - is an in-memory database with undo/redo transaction behavior. Contexts are used for holding data (current state of db) and transactions (how it has been changed in the past).

There may be multiple databases with different names at the same time with (create-db name) and (delete-db name) commands: but this kind of deletion should occur very rarely (if at all), so this perfectly fits to your advice not to delete context symbols too often. But if it is wished to delete such a named db, this should be safe.

[hartrock]

Note: second reply to split whole reply into smaller parts.



From the docu of delete (http://www.newlisp.org/downloads/newlisp_manual.html#delete">http://www.newlisp.org/downloads/newlis ... tml#delete">http://www.newlisp.org/downloads/newlisp_manual.html#delete)

delete

syntax: (delete symbol [bool])

syntax: (delete sym-context [bool])



Deletes a symbol symbol, or a context in sym-context with all contained symbols from newLISP's symbol table. References to the symbol will be changed to nil.



When the expression in bool evaluates to true, symbols are only deleted when they are not referenced.



When the expression in bool evaluates to nil, symbols will be deleted without any reference checking. Note that this mode should only be used, if no references to the symbol exist outside it's namespace. If external references exist, this mode can lead to system crashes, as the external reference is not set to nil when using this mode. This mode can be used to delete namespace hashes and to delete namespaces in object systems, where variables are strictly treated as private.



Protected symbols of built-in functions and special symbols like nil and true cannot be deleted.



delete returns true if the symbol was deleted successfully or nil if the symbol was not deleted.



When deleting a context symbol, the first delete removes the context namespace contents and demotes the context symbol to a normal mono-variable symbol. A second delete will remove the symbol from the symbol table.

I don't understand this part (first sentence emphasized by me):

When the expression in bool evaluates to nil, symbols will be deleted without any reference checking. Note that this mode should only be used, if no references to the symbol exist outside it's namespace. If external references exist, this mode can lead to system crashes, as the external reference is not set to nil when using this mode. This mode can be used to delete namespace hashes and to delete namespaces in object systems, where variables are strictly treated as private.

Why does nil'ing of Ctx:sub work with second (delete 'Ctx2) in my original post, if there is no reference checking (bool arg not given, so it should be nil)?

Because it works and there is a use case for not checking (if you know, what you are doing): what is the meaning of this param? Only be honored for contexts?

[Lutz]

Why does nil'ing of Ctx:sub work with second (delete 'Ctx2) in my original post, if there is no reference checking (bool arg not given, so it should be nil)?

 The absence of bool arg doesn't mean it's nil. The default value for the bool arg is is true. So (delete 'S) and (delete 'S true) are the same thing and do reference checking.

[hartrock]

 The absence of bool arg doesn't mean it's nil. The default value for the bool arg is is true.

OK.

My assumption has been, if there is no default value given in the docu, then an optional arg leads to a nil param, if not given.

 So (delete 'S) and (delete 'S true) are the same thing and do reference checking.

OK.

After first delete Ctx:sub refers to a symbol whose evaluation leads to a deleted (invalid) context (and not nil); and second delete repairs this by reference checking with nil'ing.

Code Select Expand


(define (safe-delete symbolOrContext (bool true))
  (if (context? (eval symbolOrContext))
      (delete symbolOrContext bool))
  (delete symbolOrContext bool))

What about making the functionality of safe-delete (like proposed above, but extended with optional bool param (code not tested)) the default behavior? Is there a use case for deleting only the context, but not its symbol? Are there other problems/tradeoffs I don't see so far?

[Lutz]

my statement:

So (delete 'S) and (delete 'S true) are the same thing and do reference checking.

was not quite correct. Both do reference checking, but (delete 'S true) will return nil when a reference is found.  (delete 'S) will replace all references found with  nil.



So we really have 3 modes:



(delete 'S)  check for references and replace with nil.



(delete 'S true) check for references and return nil when references are found.



(delete 'S nil) ignore references, just delete (unsafe if references exist).



With contexts there is a 2-step process the first delete for the context contents the second for the context symbol.



The hanging you observed after the first (delete 'CTX) when CTX had niled references will not occur anymore but give an appropriate error message:



http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/



So the first (delete 'CTX) on a context is a "safe" operation and errors will be signaled without hanging.



The two stage deleting of contexts is necessary when the same symbol get used as a context, then the context contents gets deleted, then the same symbol gets a context again. I believe this occurs in the objective newLISP, module mentioned in an other thread, and also in the DragonFly web module.

[hartrock]

The hanging you observed after the first (delete 'CTX) when CTX had niled references will not occur anymore but give an appropriate error message:



http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/



So the first (delete 'CTX) on a context is a "safe" operation and errors will be signaled without hanging.

Thanks Lutz: this change renders safe-delete from above obsolete.



One could define context-n-symbol-delete for deleting a context with its symbol:

Code Select Expand


(define (context-n-symbol-delete ctx)
  (if (not (context? ctx))
      (throw-error "context argument needed"))
  (let (symbol (sym (string ctx)))
    (delete symbol)   ; ctx with its syms
    (delete symbol))) ; ctx sym in MAIN