[patch] system variables $main-args-load-ix, $load-list

[hartrock]

Here is a patch providing system variables $main-args-load-ix and $load-list.

These system variables provide system information, which is difficult to compute outside the interpreter code, but easy from inside, where all needed information is available.



Patch against newlisp-10.6.4.tgz  2015-09-21 16:07  1.6M:

Code Select Expand


sr@free:~/newLISP_Git/mirror$ git diff -p inprogress HEAD
diff --git a/mirror/newlisp.c b/mirror/newlisp.c
index 262339d..0190bc5 100644
--- a/mirror/newlisp.c
+++ b/mirror/newlisp.c
@@ -192,6 +192,8 @@ SYMBOL * atSymbol;
 SYMBOL * currentFunc;
 SYMBOL * argsSymbol;
 SYMBOL * mainArgsSymbol;
+SYMBOL * mainArgsLoadIxSymbol;
+SYMBOL * loadListSymbol;
 SYMBOL * listIdxSymbol;
 SYMBOL * itSymbol;
 SYMBOL * sysxSymbol;
@@ -894,6 +896,7 @@ for(idx = 1; idx < argc; idx++)
         exit(0);
         }
     
+    mainArgsLoadIxSymbol->contents = (UINT)stuffInteger(idx);
     loadFile(argv[idx], 0, 0, mainContext);
     }
 
@@ -1428,6 +1431,8 @@ questionSymbol = translateCreateSymbol("?", CELL_NIL, mainContext, TRUE);
 atSymbol = translateCreateSymbol("@", CELL_NIL, mainContext, TRUE);
 argsSymbol = translateCreateSymbol("$args", CELL_NIL, mainContext, TRUE);
 mainArgsSymbol = translateCreateSymbol("$main-args", CELL_NIL, mainContext, TRUE);
+mainArgsLoadIxSymbol = translateCreateSymbol("$main-args-load-ix", CELL_NIL, mainContext, TRUE);
+loadListSymbol = translateCreateSymbol("$load-list", CELL_NIL, mainContext, TRUE);
 listIdxSymbol = translateCreateSymbol("$idx", CELL_NIL, mainContext, TRUE);
 itSymbol = translateCreateSymbol("$it", CELL_NIL, mainContext, TRUE);
 countSymbol = translateCreateSymbol("$count", CELL_NIL, mainContext, TRUE);
@@ -1454,6 +1459,8 @@ questionSymbol->flags |= SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
 atSymbol->flags |=  SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
 argsSymbol->flags |= SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
 mainArgsSymbol->flags |= SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
+mainArgsLoadIxSymbol->flags |= SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
+loadListSymbol->flags |= SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
 listIdxSymbol->flags |= SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
 itSymbol->flags |= SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
 countSymbol->flags |= SYMBOL_GLOBAL | SYMBOL_BUILTIN | SYMBOL_PROTECTED;
@@ -1465,6 +1472,8 @@ argsSymbol->contents = (UINT)getCell(CELL_EXPRESSION);
 objSymbol.contents = (UINT)nilCell;
 objSymbol.context = mainContext;
 objCell = nilCell;
+mainArgsLoadIxSymbol->contents = (UINT)nilCell;
+loadListSymbol->contents = (UINT)getCell(CELL_EXPRESSION);
 
 /* init signal handlers */
 for(i = 0; i < 32; i++)
@@ -3291,6 +3300,7 @@ if(my_strnicmp(fileName, "http://", 7) == 0)
     pushResult(result);
     if(memcmp((char *)result->contents, "ERR:", 4) == 0)
         return(errorProcExt2(ERR_ACCESSING_FILE, stuffString((char *)result->contents)));
+    addList((CELL*)loadListSymbol->contents, stuffString(fileName));
     result = copyCell(sysEvalString((char *)result->contents, context, nilCell, EVAL_STRING));
     currentContext = contextSave;
     return(result);
@@ -3314,6 +3324,7 @@ for(i = 0; i<recursionCount; i++) printf("  ");
 printf("load: %sn", fileName);
 #endif
 
+addList((CELL*)loadListSymbol->contents, stuffString(fileName));
 result = evaluateStream(&stream, 0, TRUE);
 currentContext = contextSave;



[*] $main-args-load-ix gives index into $main-args of last file/URL tried to be load by main() loop of the interpreter.

During evaluation of a script this need not be an index of its path, because it may itself been loaded indirectly by another script given as command line argument.

But exactly the index of currently evaluated $main-args script is suited for reliable and easy determination of:

[*] scriptname at command line arg: good for output of user info (e.g. errors and logging);

[*] position of script arguments: after that parsing of script arguments can start;

[*] fixpoint for relatively located dirs/files: this is good for having an alternative to NEWLISPDIR in file locations related to a user.

[/list]

[*] $load-list is a push-back list of all files/URLs already loaded by interpreter main() loop or load or module from newLISP; last element is last loaded script. This may be another script/source as given at command line, because it could have performed a load itself, which would have led to one or more additional elements in this list.

This is good for inspecting,

[*] what have been loaded at all, and

[*] if there have been any duplicate loads of script/code.

[/list] Note: code currently being evaluated may stay in any of these elements (after a return from calling load by some script, evaluation continues there).

[*] Good properties of both patches:

[*] they should work at all supported platforms;

[*] only minimal code changes;

[*] no pollution of 'normal' namespace by using symbols with '$' prefix;

[*] where are the disadvantages?

[/list][/list]

Code Select Expand


sr@free:~/newLISP/Examples$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> $main-args $main-args-load-ix $load-list
("newlisp")
nil
()
>
sr@free:~/newLISP/Examples$ newlisp notExisting
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> $main-args $main-args-load-ix $load-list
("newlisp" "notExisting")
1
()
>
sr@free:~/newLISP/Examples$ touch existing.lsp # empty
sr@free:~/newLISP/Examples$ newlisp -s 1000 existing.lsp --foo
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> $main-args $main-args-load-ix $load-list
("newlisp" "-s" "1000" "existing.lsp" "--foo")
4
("existing.lsp")
>
sr@free:~/newLISP/Examples$ cat printThese.lsp
#!/usr/bin/env newlisp
(println "$main-args: " $main-args)
(println "$main-args-load-ix: " $main-args-load-ix)
(println "$load-list: " $load-list)
sr@free:~/newLISP/Examples$ newlisp -s 1000 printThese.lsp --foo
$main-args: ("newlisp" "-s" "1000" "printThese.lsp" "--foo")
$main-args-load-ix: 3
$load-list: ("printThese.lsp")
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> $main-args $main-args-load-ix $load-list
("newlisp" "-s" "1000" "printThese.lsp" "--foo")
4
("printThese.lsp")
>

Calling newlisp loading some .init.lsp at startup, which is loading some other source itself:

Code Select Expand


sr@free:~/newLISP/Examples$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> $main-args $main-args-load-ix $load-list
("newlisp")
nil
("/home/sr/.init.lsp" "/home/sr/newLISP/modules/Init.lsp"
 "/home/sr/newLISP/modules/FOOPReference.lsp" "/home/sr/newLISP/modules/Util.lsp"
 "/home/sr/newLISP/modules/Logger.lsp" "/home/sr/newLISP/modules/LoggerTweakable.lsp"
 "/home/sr/newLISP/modules/Libs.lsp")
>
sr@free:~/newLISP/Examples$ newlisp -s 1000 printThese.lsp --foo
$main-args: ("newlisp" "-s" "1000" "printThese.lsp" "--foo")
$main-args-load-ix: 3
$load-list: ("/home/sr/.init.lsp" "/home/sr/newLISP/modules/Init.lsp"
 "/home/sr/newLISP/modules/FOOPReference.lsp" "/home/sr/newLISP/modules/Util.lsp"
 "/home/sr/newLISP/modules/Logger.lsp" "/home/sr/newLISP/modules/LoggerTweakable.lsp"
 "/home/sr/newLISP/modules/Libs.lsp" "printThese.lsp")
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> $main-args $main-args-load-ix $load-list
("newlisp" "-s" "1000" "printThese.lsp" "--foo")
4
("/home/sr/.init.lsp" "/home/sr/newLISP/modules/Init.lsp"
 "/home/sr/newLISP/modules/FOOPReference.lsp" "/home/sr/newLISP/modules/Util.lsp"
 "/home/sr/newLISP/modules/Logger.lsp" "/home/sr/newLISP/modules/LoggerTweakable.lsp"
 "/home/sr/newLISP/modules/Libs.lsp" "printThese.lsp")
>


[*] First there has been a change of (load) according the suggestion from Ralph (see http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4756#p23433">//http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4756#p23433), so that (load) (without arguments) returns a copy of $load-list. But this has problems, due to making the semantics of load more complicated (in addition the naming of the load func is questionable for this semantics), so it has been discarded.

One example: someone could have the idea to try (load int-ix) in analogue to (main-args int-ix), which would not work.

[*] After creating these patches I think, that $main-args-load-ix is the system information, which is more important for easing the life of developers than $load-list; but the latter is helpful, too.

[/list]

[xytroxon]

Good work!



In keeping with Lutz's succinct newLISP key word naming scheme, (as opposed to Scheme's the-longer-the-run-on-key-word-the-better approach ;o), I would rename $main-args-load-ix to either of the more simply remembered mnemonics:



$libraries  or  $libs



Also does/should your code include import .dll/.so library filepaths?



--xytroxon

[Lutz]

I understand the occasional necessity to remember the names of files loaded, but this should not be built-in to the language. What happens with applications which reload files over and over again to refresh or change a set of data or function definitions? This would make the the stack of filenames bigger and bigger.



One could built some functionality with little code in .init.lsp, e.g. by changing the definition of module or introduce a new loadx. This would also be easier to adapt to specific requirements.





Ps: How about adding something to the getopts.lsp module?

[hartrock]

I understand the occasional necessity to remember the names of files loaded, but this should not be built-in to the language. What happens with applications which reload files over and over again to refresh or change a set of data or function definitions? This would make the the stack of filenames bigger and bigger.

Good point! I haven't thought about this problem. Seems to be a killer argument for the simple variant of $load-list in my patch: I'm agreeing that $load-list this way is a bad idea.

But this argument does not apply to $main-args-load-ix (see below).

One could built some functionality with little code in .init.lsp, e.g. by changing the definition of module or introduce a new loadx. This would also be easier to adapt to specific requirements.

This is true.

BTW: for loading own sources I have built some infrastructure with similar commands and a loaded list to have automated loading of libs just once.

[*] a good name for logging (and this is not the name of a source, which has been loaded last from whatever other source);

[*] a dir as a good place for relative positioning of files/dirs as a lightweighted alternative to the official NEWLISPDIR;

[*] a position being a good start for reliably parsing script arguments.

[/list]
You probably have seen Ralph's sophisticated solution by looking into the procfs to get a reliable solution for specific platform(s), whose working depends on platform, and is far away from simple. The other variant he has mentioned is to

... install a newlisptar binary (embedding lsptar.lsp) at the production places ...

. If I've understood correctly - don't have experiences here - this would lead to duplicate the newlisp bin for each app/script packaged this way. OK, this is a solution having its usecases and merits, but isn't it the sense of a script interpreter to share it between all scripts? (If possible, of course.)

Ps: How about adding something to the getopts.lsp module?

This is one usecase for $main-args-load-ix: to know, where we are in the $main-args line, and to have a scriptname.



I really don't know, how to get the info, which $main-args-load-ix would give, reliably from outside the interpreter (and if possible at all, it seems to be quite complicated and/or having other drawbacks); but from inside the interpreter it's easy and just 5 (!) lines (without any resource risk).

If I would know a good way of getting this info from outside the interpreter, I'd just use it.



I think, I'm surely not the only one having this problem.

@All: what do you think?



@Lutz: I would be happy, if you could seriously reconsider this part of the patch.

[xytroxon]

Just do a unique list operation after each file path is added.

Code Select Expand


(set '$libs nil)

(define (add-lib fp)
  (set '$libs (unique (push fp $libs -1)))
  (println fp "-> " $libs)
)

(define (x-load fp)
  (add-lib fp)
)

(define (x-import fp)
  (add-lib fp)
)

(println "Load a newLISP module")
(println "Load File -> $libs")
(x-load   "p1/mod.lsp")
(x-import "p2/bin.dll" "func-1")
(x-import "p2/bin.dll" "func-2")

(println "nLoad my library")
(println  "Load File -> $libs")
(x-load   "p1/lib.lsp")
(x-import "p1/bin.dll" "func-1")
(x-import "p1/bin.dll" "func-2")

(println "nRepeated load example")
(println  "Load File -> $libs")
(x-load   "p3/usr.lsp")
(x-load   "p3/usr.lsp")
(x-load   "p3/usr.lsp")
(x-load   "p3/usr.lsp")
(x-load   "p3/usr.lsp")
(x-load   "p3/usr.lsp")

(println "n$libs-> " $libs)
(exit)


Output:



Load a newLISP module

Load File -> $libs

p1/mod.lsp-> ("p1/mod.lsp")

p2/bin.dll-> ("p1/mod.lsp" "p2/bin.dll")

p2/bin.dll-> ("p1/mod.lsp" "p2/bin.dll")



Load my library

Load File -> $libs

p1/lib.lsp-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp")

p1/bin.dll-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll")

p1/bin.dll-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll")



Repeated load example

Load File -> $libs

p3/usr.lsp-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll" "p3/usr.lsp")

p3/usr.lsp-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll" "p3/usr.lsp")

p3/usr.lsp-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll" "p3/usr.lsp")

p3/usr.lsp-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll" "p3/usr.lsp")

p3/usr.lsp-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll" "p3/usr.lsp")

p3/usr.lsp-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll" "p3/usr.lsp")



$libs-> ("p1/mod.lsp" "p2/bin.dll" "p1/lib.lsp" "p1/bin.dll" "p3/usr.lsp")

>Exit code: 0



By inspecting $libs, I can see I have a  bin.dll file conflict originating in mod.lsp



-- xytroxon

[hartrock]

Just do a unique list operation after each file path is added.

This is a good example for what can well be done from outside the interpreter (you could check for loading the same lib with same path twice, too).

But in Lutz' example such a solution (with keeping list small by unique added) just wouldn't have any benefits, but could fail: e.g. by loading a versioned source file - including version info in its name - again and again... -> Boom!



But for the other $main-args-load-ix part of the patch I don't see such risks, and I don't see how to get respective info reliably from outside the interpreter...

[xytroxon]

Hi Hartrock!



I should of said that the use of unique in my x-load x-import functions would be done at the C code level, inside of the newLISP load and import functions.



This pseudo code was written to show how the $libs list would look after preventing multiple loads from blowing up the $libs list size. This version maintains the order in which modules are loaded, (until a module is loaded more than once). But all unique filepaths are recorded for user inspection.



-- xytroxon

[hartrock]

Hello Xytroxon,
 

I should of said that the use of unique in my x-load x-import functions would be done at the C code level, inside of the newLISP load and import functions.

This would not help, if the name of some repeatedly loaded source continuously changes: e.g. by having some version info (could be a timestamp) as part of the name (so list cannot be condensed by unique)...



Some feature built-in interpreter source should be (as) robust (as possible) against all imaginable usecases, which it wouldn't be in this example.

[rrq]

As I see it, the argument for addressing this part of the interpreter is to make it possible to build a generic modularization support framework that avoids everyone having to hard code the modularization into every application.



I think the support needs from the interpreter includes the two discussed: i.e., knowing the current main-args index, and tracking which files are loading at least while they are loading, but it also will require some way to "stop" the loading/processing part-way into a file. Without those, I don't think I would be able to invent a generic modularization framework, and will have to settle for hard-coded modularization special to each application.



On the other hand, many applications are small enough to not make this an issue. And you can come a long way by compositing applications by copying snippet files to be siblings of the main.

[hartrock]

As I see it, the argument for addressing this part of the interpreter is to make it possible to build a generic modularization support framework that avoids everyone having to hard code the modularization into every application.

Introducing $main-args-load-ix is a no-brainer to me, as long as there are no good arguments against introducing it (haven't seen them so far).

It

[*] gives much for building a modularization infrastructure by module(s),

[*] is platform agnostic,

[*] does not need significant resources,

[*] needs only a minimal interpreter code change,

[*] does not pollute non-system (non-$) namespace.

[/list]
In summary: there is minimal effort for a big improvement.



What are the arguments against introducing it?

Isn't it suited to give exactly this (small) info from inside interpreter, which is needed for a modularization infrastructure, so that all other stuff could be done by module(s) outside interpreter core?

I think the support needs from the interpreter includes the two discussed: i.e., knowing the current main-args index, ...

This would be given by $main-args-load-ix.

... and tracking which files are loading at least while they are loading, ...

This could well be done by module(s) code (load functions with extended functionality) outside interpreter core.

... but it also will require some way to "stop" the loading/processing part-way into a file.

Couldn't this be done by module code? Possibly not by stopping of loading something, but by stopping of evaluating it, if some condition has been met.

 Without those, I don't think I would be able to invent a generic modularization framework, and will have to settle for hard-coded modularization special to each application.

It would be interesting to know, for which usecase $main-args-load-ix together with module(s) code would be insufficient.

On the other hand, many applications are small enough to not make this an issue. And you can come a long way by compositing applications by copying snippet files to be siblings of the main.

Putting code in siblings of a script, to be loaded by this script, is not robust, if you don't have a reliable script path for computing its directory.