[HowTo] stopping newlisp parsing of '#!'-script args ..

[hartrock]

.. with a simple patch for a first solution and further ideas for possibly better ones...

[4. update] Removed some stupid patch ideas.

[3. update]

Solution for 10.5.4: http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4416#p21792">http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4416#p21792

Solution for later versions: http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4416#p21794">http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4416#p21794

'[HowTo]' added to title.

[2. update] Changed title away from feature request: some discussion revealed a very good solution (see posts thereafter); and the patches are not a good idea compared with it.

[update] Better patch - IMHO - in self-reply to this post.



Doing more and more scripting with newLISP, it's important to me to have interactive debugging facilities.

Furthermore it's good to have some separation between CLI args targeted to the interpreter and others targeted to a script interpreted by it.



Assumed there is the following code:

Code Select Expand


#!/usr/bin/newlisp

(module "getopts.lsp")

(shortopt "p" (println "players: " getopts:arg) "int" "num of players")
(shortopt "h" (println (getopts:usage)) nil "usage")

(getopts (2 (main-args)))

;; some code here: if something goes wrong, or just for interactive development
;; no (exit) could be a good idea here...
(exit)

This gives:

Code Select Expand


sr@free:~/NewLisp$ ./game.lsp -p 10
players: 10
sr@free:~/NewLisp$

So far so good.



But if something goes wrong, or we just want to inspect some symbols before (exit), commenting out (exit) does not work:

Code Select Expand


#!/usr/bin/newlisp

(module "getopts.lsp")

(shortopt "p" (println "players: " getopts:arg) "int" "num of players")
(shortopt "h" (println (getopts:usage)) nil "usage")

(getopts (2 (main-args)))

;; some code here: if something goes wrong, or just for interactive development
;; no (exit) could be a good idea here...
;(exit)

We get:

Code Select Expand


sr@free:~/NewLisp$ ./game.lsp -p 10
players: 10
newLISP server setup on 10 failed.
sr@free:~/NewLisp$

instead of entering newLISP interpreter.



[4. update]: Removed some stupid patch ideas.



What do you think?

[hartrock]

...

[Lutz]

If you define and error handler, you can skip command line processing and are left in interactive mode:

Code Select Expand

#!/usr/bin/newlisp

(error-event (fn () (println (last-error))))

(module "getopts.lsp")

;(shortopt "-" (println "'-' opt") nil "stops parsing CLI opts")
(shortopt "p" (println "players: " getopts:arg) "int" "num of players")
(shortopt "h" (println (getopts:usage)) nil "usage")

(getopts (2 (main-args)))

(throw-error "debugging mode")

(exit) ; <-- is never reached


The throw-error statement will put you in interactive mode:

Code Select Expand


~> ./game -p 10
players: 10
(58 "ERR: user error : debugging mode")
newLISP v.10.5.4 64-bit on OSX IPv4/6 UTF-8 libffi, options: newlisp -h

>


any other error in you code would also send you to interactive mode, but you can force it using throw-error

[hartrock]

Thanks for the hint: this is a solution. I even haven't thought about exceptions for this use case, though some assert macros of mine use them, too.

Though there is more to do for the developer than just to comment out (exit); for debugging this should be fine.



What about having some kind of non-error exception for this use case?

Because there needn't be an error, it's just to keep the interpreter off from parsing of command line args and entering it instead for an interactive session.



[4. update]: Removed some stuff about a bad idea of checking the exec bit of interpreter loaded scripts for stopping interpreter CL parsing.

[Lutz]

To summarize: the only reason for newLISP doing unwanted command line processing, is programmer error.  



There is no other normal use case for a built-in mechanism to stop processing the command line. If processing occurs and is unwanted, it always means either a code error or a missing exit statement during code development and for debugging. Somehow the program lost control and finishes at the top level without exiting, causing newLISP to continue processing command line parameters.



To help debugging in future versions, a simple (reset) will also interrupt command line processing and enter interactive mode. No error handler will be required. So this will do it too:

Code Select Expand


#!/usr/bin/newlisp

(module "getopts.lsp")

(shortopt "p" (println "players: " getopts:arg) "int" "num of players")
(shortopt "h" (println (getopts:usage)) nil "usage")

(getopts (2 (main-args)))

(reset) ; stop processing - now also including command line parsing

(exit) ; <--- never gets here because reset interrupts processing


Ps: A scripting language checking the execution bit would be very unusual, no scripting language does that. The exe bit should only be required by the OS/shell for direct execution of scripts via the shebang #! mechanism.

[hartrock]

To summarize: the only reason for newLISP doing unwanted command line processing, is programmer error.  

This is not the only reason: there is also the interest in interactive development: no error, but wishing to inspect things from inside of the interpreter after the script code has run.

There is no other normal use case for a built-in mechanism to stop processing the command line. If processing occurs and is unwanted, it always means either a code error or a missing exit statement during code development and for debugging. Somehow the program lost control and finishes at the top level without exiting, causing newLISP to continue processing command line parameters.

See above.

To help debugging in future versions, a simple (reset) will also interrupt command line processing and enter interactive mode. No error handler will be required. So this will do it too:

Code Select Expand


#!/usr/bin/newlisp

(module "getopts.lsp")

(shortopt "p" (println "players: " getopts:arg) "int" "num of players")
(shortopt "h" (println (getopts:usage)) nil "usage")

(getopts (2 (main-args)))

(reset) ; stop processing - now also including command line parsing

(exit) ; <--- never gets here because reset interrupts processing


I like this solution!

(reset) with error handler already does exactly what I want: the script runs and afterwards it's possible to inspect symbols and their contents in the interpreter.



Without error handler there are problems:

[*] without (reset) it hangs at an unknown opt - e.g. '-c' - this is my use case -,

[*] with (reset) the interpreter exits.

[/list]

Thanks for your will - and work - to improve this!

After this change it's just a change of (exit) to (reset) and vice-versa (just to change one semicolon for commenting out/in (reset)).

Ps: A scripting language checking the execution bit would be very unusual, no scripting language does that.

OK: thanks for the info.



Moreover I think now, that the (reset) mechanism:

[*] has lower overhead - no need to check file status after loading it -, and

[*] it gives less suprises - with checking exec bit of the file, it depends from the position of it, if and when CL arg parsing stops (the (reset) mechanism is cristal clear in this respect: stop CL arg parsing, if developer explicitely says so).

[/list]

Note: I have changed the title of my original post away from [FR} (feature request).

[TedWalther]

I wrote the getopts modules.  I have difficulty understanding the issue.



"--" is a GNU standard.  getopts is designed to do GNU standard command line parsing.



If an unknown option makes the program hang in the getopts module, that is a bug.  Can you please post a short working sample of code so I can duplicate the problem.  Unknown options should raise an error, not hang.



Scripts take arguments as part of the POSIX standard, and it is best not to mess with that.  For instance, in some of my newlisp scripts I have to use the -m and -s options, to control the memory and stack usage of the interpeter, so it looks like this: #!/usr/bin/newlisp -m 1024 -s 1024



This happens on a script by script basis.  When you write your script and set up the command line parsing, just take that into account.



Another issue with command line parsing is that different operating systems parse the command line differently, so you may need to start your argument processing at a different place if you are on Unix versus Windows.



Can you explain your issue a bit better hartrock?

[hartrock]

Ted you are right!

I haven't known this '--' behavior.



From 'man getopts':

Each  parameter not starting with a `-', and not a required argument of

       a previous option, is a non-option parameter. Each  parameter  after  a

       `--' parameter is always interpreted as a non-option parameter.  If the

       environment variable POSIXLY_CORRECT is set, or  if  the  short  option

       string  started with a `+', all remaining parameters are interpreted as

       non-option parameters as soon as  the  first  non-option  parameter  is

       found.

Exactly this is what getopts does:

Code Select Expand

#!/usr/bin/newlisp

(module "getopts.lsp")

(shortopt "a" (println "'a' opt") nil "works as expected")
(shortopt "h" (println (getopts:usage)) nil "usage")

;; Do *not* try this (action won't be triggered):
;; (shortopt "-" (println "'-' opt") nil "stops parsing CLI opts") ; Do *not* try this!

(getopts (2 (main-args)))

(println "After calling getopts.")
(exit)


Code Select Expand


sr@free:~/NewLisp$ newlisp getopts_bug.lsp -a --
'a' opt
After calling getopts.
sr@free:~/NewLisp$ newlisp getopts_bug.lsp -- -a
After calling getopts.
sr@free:~/NewLisp$

In the first case '-a' triggers its action; in the second case it will be ignored.

[TedWalther]

Looks like it does what is supposed to.  Here, from the GNU getopt documentation:

http://www.gnu.org/software/libc/manual/html_node/Using-Getopt.html#Using-Getopt">http://www.gnu.org/software/libc/manual ... ing-Getopt">http://www.gnu.org/software/libc/manual/html_node/Using-Getopt.html#Using-Getopt



getopt has three ways to deal with options that follow non-options argv elements. The special argument '--' forces in all cases the end of option scanning.

[hartrock]

Can you explain your issue a bit better hartrock?

My original motivation for starting this has been twofold:

[*] Being able to enter the interpreter instead of exit:

    → possible by reset or throw (don't know the differences so far) together with error handler (and starting with v10.5.5 reset even without error handler).

[*] Having a separation between CL args for interpreter and script:

     → possible by correctly using exit or the former mechanism for entering the interpreter during development.

[/list]
So this thread has changed to [HowTo].



Next is to cleanup it a bit to avoid confusion: most important, removing the stupid idea with using special opt '--' for separating CL args between interpreter and script (coming into mind without knowing first mechanism above and not knowing special semantics of '--')).

[4. update]: Cleanup done.



BTW: I have extended/patched getopts.lsp (injected from the outside after loading it) for having shortlongopt: this leads to a grouping of corresponding short and long opt in usage:

Code Select Expand

sr@free:~/Ideas/Spiel$ ./gol -h
Usage: gol [options]
  -c, --cycles INT                  num of cycles
  -r, --rounds INT                  num of rounds per cycle
  -a, --players-at-all INT          players at all
  -g, --players-per-game INT        players per game
  -s, --start-balance INT           start balance for each player
  -l, --lower-risk-limit FLOAT      [opt] 0.0 <= FLOAT <= 1.0 (default: 0.0)
  -u, --upper-risk-limit FLOAT      [opt] 0.0 <= FLOAT <= 1.0 (default: 1.0)
  -d, --distribution-per-cycle FLOAT  [opt] 0.0 <= FLOAT <= 1.0 (default: 0.0) fraction of balance to be distributed
  -h, --help                        Print this help message

(printing could be improved further). May be you are interested.

[TedWalther]

That looks like a good idea.  Can you send the patch for review?

[hartrock]

Hello Ted,



I've sent you an email with the code via the User Panel. Because it's my first mail here, and it stays in the Outbox - perhaps diminishing there, if you have seen it? - I think it cannot hurt to give you a hint.



Best regards,

Stephan