newLISP Fan Club

Thanks @itistoday!! Great to be at a new forum home.

Here's a pointer to a few of my newlisp projects

Most of them are functional, but generally they are for fun.

This morning I got a blurb here about the pains of sending emails to addresses @gmail.com, detailing a bit about how finicky they, Google, are.

Whilst that is true, I've held this email address some 20 years mostly because their spam filtering has been quite outstanding. During that time I have set up some number of MTA (typically small and smallish ones), and like apparently this forum, come to realize both that in order to deliver to @gmail.com addresses all the "ducks" need to be in perfect order, and that getting attention and assistance by Google staff to figure out the "ducks" is near impossible.

This forum is close to perfect in its setup apart from lacking its reverse DNS setup, which I believe is one of the "ducks", i.e. one thing that @gmail.com is finicky about.

(And if it'd be of any help. I'm more than happy to be a trial target for MTA configuration trials.)

It seems there's something funny with mixing search and seek (or read-line). The following is an illustration with an appropriate some-data-file.txt which has "some text" here and there.

Code Select Expand

/root/newlisp-10.7.1/newlisp
newLISP v.10.7.1 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (open "some-data-file.txt" "r")
3
> (seek 3)
0
> (search 3 "some text" true)
548
> (seek 3)
0
> (search 3 "some text" nil)
64812
> (seek 3)
0
> (search 3 "some text" true)
64821
It looks like search keeps its own file position, separate from what seek uses (and what read-line uses).

Yet another embedding approach; now using tar for the application.

Code Select Expand

** pack-newlisp version 1.1. Utility for managing self executable newlisp tar.

Usage 1: pack-newlisp p <filename> <architecture> <tar data arguments>
  Pack an application for the given architecture (`uname -m`) into the
  given filename. The <tar data arguments> nominates the files and
  directories to include as per the `tar` program.

Usage 2: pack-newlisp u <filename>
  Unpack a pack application by separating out the included files as a
  compressed tar named <filename>.tgz


The attached bundle is for building your own pack-newlisp binary, which indeed is a pack-newlisp embedding in itself. It's a mixture of newlisp code, a Makefile, and documentation.



EDIT: new version that packs for defined architectures. (You'll need to make a "head" for the target architecture, and include in your pack-newlisp)



EDIT 2: new version after bug fixing. This is hopefully the final of version 1.x series.

(not too overwhelming interest though)



EDIT3: I removed the attachment here, and rather made the software available at github:

Code Select Expand

git clone https://github.com/ralph-ronnquist/pack-newlisp.git
If you happen to use it and in particular prepare a new head file, then please add it to the bundle. Right now it only includes "head files" for Linux i686, Linux x86_64 and Windows XP.

Just a general note that I've updated my various newlisp modules at http://www.realthing.com.au/files/newlisp">http://www.realthing.com.au/files/newlisp, and in particular, added "markdown"  (see http://daringfireball.net/projects/markdown">//http://daringfireball.net/projects/markdown) processing to expand-string.



Maybe markdown could be included into newlispdoc (as well)?

This problem got lost in the other thread. Namely that regex does not accept option flags 128, 256 or 1024.

Code Select Expand

> (regex "x" "xxx" 128)

ERR: regular expression in function regex : "offset 0 unknown option bit(s) set"
> (regex "x" "xxx" 256)

ERR: regular expression in function regex : "offset 0 unknown option bit(s) set"
> (regex "x" "xxx" 1024)

ERR: regular expression in function regex : "offset 0 unknown option bit(s) set"

Somewhere something changed for date-value, which now doesn't evaluate its parameters, but requires explicit ints. E.g.

Code Select Expand

(date-value 2015 1 1) ; is fine, but
(date-value (int "2015") 1 1) ; is not, and
(date-value '2015 1 1) ; is also not accepted

This changed for 10.6.3 sometime after May this year, but I didn't realize this until I quite recently upgraded a machine to the "final" 10.6.3 ...



Hopefully it is just a bug, or otherwise it certainly is something to make a clear note about in the documentation.

Newlisp is as fun as always, and in the spirit of it I like to point you at

http://www.realthing.com.au/files/newlisp/lsptar.lsp">//http://www.realthing.com.au/files/newlisp/lsptar.lsp

which is a couple of stanzas to run newlisp programs in/from/as tar files (compressed or uncompressed). This is to let you package up a program of many lsp files into a tgz file, and then run this elsewhere without unpacking them; kind of an alternative to embedding.



EDIT: that last polishing I did before presenting it here splashed egg in my face :-(

Obviously it held the bug of failing on anything but for "run the first file" option...

I've uploaded the corrected version now.

Apparently json-parse can only read 32-bit numbers, and it results in MAXINT without error for larger numbers. Luckily it seems to be a minor patch to expand this to 64 bit numbers. As attached for today's 10.6.3.


[attachment=0]json-64bit-number.patch.gz[/attachment]

I'm wondering whether there's any direct support for writing a "userland" packet filter in newlisp for iptables (with QUEUE or NFQUEUE rules). The net-listen documentation seems to hint on something in this direction, but it's not clear to me how things are done.

Anyone have any pointers?

Test scenario: running newlisp in -http mode and another newlisp querying it:

Code Select Expand

1% newlisp -http -d 19001

Code Select Expand

2% newlisp -e '(net-eval localhost 19001 {(exec "ls")})'

More specifically, in the source, I can see that the batchMode logic of executeCommandLine seems buggy; it enters batchMode=2 whenever the input starts with "[cmd]", and this simply bypasses both the command event call, and the HTTP dispatch chain, also including the end case of avoiding evaluation.



I would have liked suggesting a fix, and perhaps it's sufficient to avoid entering batchMode=2 in the case of httpMode&&noPromptMode, but I'm not totally a par with all ramifications of that.



In any case, this really is a show stopper for using it as front-line http request handler.



EDIT: Possibly one can patch this by having an exiting reader-event handler:

Code Select Expand

(reader-event (fn (s) (exit)))
This simply closes the connection for any net-eval input, i.e., when the input starts with "[cmd]", and other input (esp. http requests) is processed like before.

Note that this has to be the very last thing on a server definition file.

The other day I took a moment to device an expand-string functionality so that I could author HTML pages more easily by using HTML page templates with keys that get replaced by content somewhat similar to the expand function, but for keys in strings (and wihtout tokenization).



Eventually I realised I needed both the plain expansion of keys replaced by values, and something more complex, that maps an expansion fragment from the template over a range of key values. This would be for creating a section of a repeated HTML fragment, where its keys take on successive values from a programatically defined range.



And, just for the fun of it, I also included the ability of embedding newlisp code into the template, as an alternative to the key replacement. This gives more control to the template authoring, to decide on which data is fitted into it.



I've made it available http://www.realthing.com.au/files/newlisp/expand-string.lsp">here as it may possibly be a quite useful utility.



I'm also quite interested in learning other good approaches to the HTML authoring problem with newlisp CGI handlers. My approach typically results in that files come in pairs, with a CGI script that, following its computations, ends by expanding a template as its response. The template may of course be an inline string, but I find the mixing of HTML and newlisp code difficult to read, and therefore cumbersome to maintain. So I end up having separate HTML template files, with almost only HTML code apart from the special <MAP>..</MAP> construct for repeated expansion.



Edit: I suppose with a special interpreter, which skips the first line and then treats the rest as a template, one can do reponse authoring in the style of PHP (etc). That's the reason I've included the "<newlisp ... ?>" eval construct in addition to the "<EVAL> .. </EVAL>" construct. Though, this still suffers from the mixing of newlisp code and HTML code, which I think makes a less maintanable solution.

When tracking down why my ^C handler

Code Select Expand

(signal 2 exit)
made newlisp in -http mode crash rather then exit, I found that p_exit doesn't actually exit in daemonMode, but rather it IOchannel=0 and throws (aka longjmps) an ERR_USER_RESET error. Thus, execution eventually continues at the AFTER_ERROR_ENTRY label, where fileno stubbornly objects to the IOchannel==0 by crashing. On Linux.



I'm not sure about the intention with p_exit not actually exiting in daemonMode.



Assuming there's a good reason for it, one should avoid IOchannel=0 to let the service progress as per normal. Though, then it's seriously hard to make it exit; either import my own exit or, not so gracefully, rely on (/ 0 0).

I'd like to suggest that the HTTP attribute "Authorization" is added to the ones captured for -http mode and forwarded as HTTP_AUTHORIZATION. Doing so is in line with CGI 1.1 to support basic authorization in the cgi scripting. This otherwise requires me to rewrite the full HTTP service.



For convenience I've attached a patch doing this on 10.6.3 (2015-03-10).



Alternatively, I'd be happy with a callback solution to allow the front-end server handle the authorization. This maybe is preferrable, but probably involves more coding(?), and significant documentation.

It's almost xmas, so I thought I should point you at the Java wrapping for newlisp, using JNI that I just cleaned up. You'll need to compile it, probably after adapting the Makefile to your set up.



http://www.realthing.com.au/files/newlisp/newlisp-jni.tgz">//http://www.realthing.com.au/files/newlisp/newlisp-jni.tgz



It's all nicely self-documenting code (hah!), and it provides what I thought of as the bare essentials for running newlisp under/in Java. I'm afraid the callback solution is different from the (emerging) "standard", mostly because I didn't feel like wading through all the type variants for boxing.



It is made on and for 32 bit linux (xubuntu 12.04) using newlisp 10.6.2 and openJDK 1.6.

It might port to other platforms.



enjoy.

I ran into the following, which seems like a bug in the regex pattern handling, illustrated in the following example:

Code Select Expand

> (map char (explode (replace "['']" "'" "x" 0)))
(120 120 120)
> (map char (explode (replace "'" "'" "x" 0)))
(120)
> (map char (explode (replace "['']" "'" "x" 2048)))
(120)

Thus, when the pattern is within brackets, the replacement of char u8216 gets replicated into each of the source bytes, whereas without the brackets, the "proper" replacement occurs. The replace is also proper with the flags code 2048 raher than 0.



newLISP v.10.6.0 32-bit on Linux IPv4/6 UTF-8 libffi.

I started investigating using kmeans for vector classification, and has the problem of getting more 'nan' than I'm happy with, and I hope there's a simple answer to what I'm doing wrong:



E.g. I run kmean-train to form 5 clusters for a collection of ~50 vectors of 100 elements. Then some centroids are of all -nan values. What does that mean?



Next, I run a sequence of kmeans-query using these centroids (including the -nan ones), and then often but not always some measures are 'nan'. What does that mean?



Alternatively, is it sensible to revise the centroids, eg map -nan to something small or large, to maybe avoid the nan classifications?

I'm fairly sure I'm not alone in having some one or two or even more than a few "teething problems" whenever I'm writing a new newlisp script. Or maybe I am, since I've been unable to find much to assist in this, and the built-in error messages are quite terse.



In any case, I've augmented my public collection of newlisp utilites with a "debug.lsp" module, at http://www.realthing.com.au/files/newlisp/debug.lsp">//http://www.realthing.com.au/files/newlisp/debug.lsp.



It's is a batch oriented debugging facility to help in finding where the bug (the next bug) is by selective tracing. It thus helps in particular for the script loading and start up, and then a bit into the program logic; in particular helping to pin-point where the data type and nil versus () mistakes are when they turn up.

Now I ran into a wall trying to use FOOP, or possibly it's really a bug. Namely that it appears arguments get copied in some funny way.... I've isolated my problem to this small example code:

Code Select Expand

(context 'One)
(define (One:One v) (list (context) v))
(define (put v) (setf (self 1) v))

(context 'Two)
(define (Two:Two one x)
  (:put one x) ;---------------------- mark 1
  (list (context) one x))

(context MAIN)
(constant 'uno (One 1))
(constant 'duo (Two uno 2)) ;----- mark 2

(println "uno=" uno)
(println "duo=" duo)

Thus, there are two contexts: One and Two, where in particular the Two "objects" are supposedly tied to One "objects" on "construction". Both of them also hold an auxillary value, and when a Two object is created, its auxillary value is propagated  to its One object.



But in the example, the One object passed in to the Two construction (at mark 2 in the code) is not the same as the One object it operates on in its constructor (at mark 1 in the code), and yet only a single One object has been created!

I get the following output:

Code Select Expand

uno=(One 1)
duo=(Two (One 2) 2)
This is with "newLISP v.10.6.0 32-bit on Linux IPv4/6 UTF-8 libffi"



I really hope someone can enlighten me about what is going on here, because it kind of is a show stopper for me.

I just ran into this pebble on the beach, that the "flat" function doesn't flatten quoted sub lists.

Code Select Expand

> (flat '('(1 2) (3 4)))
('(1 2) 3 4)
Perhaps it's worth mentioning it in the doc, or fix it, if it's a mistake. By my inuition, the flat function should traverse into quoted lists the way it traverses into lambda expressions.