Building and using a shared newlisp.so library

[twotaters]

I'm trying to build the newlisp.so shared library on Red Hat Linux. Section 24 of the Code Patterns says "On all platforms, newLISP can be compiled as a shared library. On Win32, the library is called newlisp.dll, on Mac OS X newlisp.dylib and on Linux and BSDs, the library is called newlisp.so. Makefiles are included in the source distribution for most platforms." I haven't found anything that says which of the makefiles it should be.



I used "configure-alt" to build the static newlisp binary. The "--disable-static" flag doesn't seem to be the right option. It builds the the executable as shared, but doesn't build the newlisp.so library.



After running configure-alt, I've update makefile_build to add -fPIC to the .o rule and -shared -o newlisp.so. That creates a shared library. I don't see a simple way to check it though - it looks like it's impossible to build a newlisp that dynamically loads newlisp.so. It seems that driving newlisp from a C program isn't a common use case, so I'd like to step back and make sure that I'm approaching this sanely.



I have an application written in C. Currently, it has Lua embedded in it. It's a shebang executable, so script files with "#!/path/to/ox" are the most common entry point. When it executes, it initializes the environment, loads the Lua interpreter from shared libraries, does the same for Oracle and SQLite3. Once all the shared libraries are in place, it loads the text of the script, strips off the shebang, and executes the script. As it reads in the script, it does some basic substitutions so that the users can run the same script on different servers (dev/test/qa/production). The processed text is feed into Lua and Lua is the glue to run the various bits and pieces against the database.



I want to replace the Lua bits with newlisp. I have to keep the external interface the same - I can't change the name of the job script or the invocation. Going from "/path/to/there/aJobName" to "/path/to/bin/newlisp aJobName" isn't acceptable. Heck, I can't even have them set up their environment to add the library paths newlisp needs. There's no technological reason for that constraint, it's budge. There are about a thousand users working in many different departments and my group doesn't have the budget to pay them to change their jobs.



Other than that, I have a good degree of freedom. As long as the system is always operational, that is, so I plan on reusing the current system's C code that does the initialization and script substitution logic. After testing that, I can roll out changes as quickly as I can update the Lua code in a job script to newlisp.



I looked at the option of changing "/path/to/there/aJobName" to a bash script that does something like set up the new lisp environment then run "newlisp /path/to/there/aJobName.lsp." I'd prefer not to if it's reasonable to have that preference.



Sorry for the big build-up to a simple question: What's the "best practice" - embedding newlisp or extending newlisp?

[Lutz]

What's the "best practice" - embedding newlisp or extending newlisp?

Normally extending newLISP with imported functions is a better choice. The embedding path, I would only choose when embedding into existing bigger applications like Excel or applications build with Visual Basic.



In the source distribution, makefiles for building a newlisp.so or newlisp.dylib are identified via the string "lib" as part of the name of the makefile. Not for all platforms and flavors, make files are available.



Inside the distribution directory, you could use the following newLISP statement o find all lib makefiles:

Code Select Expand

(directory "." "makefile.*lib.*")
or in a UNIX shell

Code Select Expand

ls makefile*lib*

[TedWalther]

Code Select Expand

(directory "." "makefile.*lib.*")

Wow Lutz, I need to read the Manual more often!  I didn't even know that was possible.  Now I can clean up a bunch of old code that wrapped (directory) in the (filter) function.



newLisp is so inspirational.  I've been meditating about the bourne shell and POSIX shell languages, and asking myself: what do they really DO.  What is the magical thing they do that makes them so good on the command line and for shell scripts.  As languages they are simple and clunky.  But they do something very important, in a way that is fast and easy.  Once I get it properly laid out, there may be some easy way to extend newLisp to do this as well:



tab completion:

found in bash, the javascript console, and various IDEs.



connecting binary programs together, no matter what language they were implemented in.



environment variables; like global variables for your whole interactive session, a way of sharing common information between the binary blobs.



command line arguments, are like the arguments passed to a function.



pipes, are data streams, streams of bytes passed in on file descriptors.  Normally these streams are stdin, stdout, and stderr.  But as Daniel Bernstein demonstrated with qmail, much more is possible in this department.  The "pipe" concept allows a very 3 dimensional implementation, but POSIX/bourne keeps pipelines fairly linear.  It doesn't allow more complex 3 dimensional plumbing.  The mechanics of handling file descriptors other than the standard 0 1 2 are messy and not intuitive.



A pipe, or file descriptor, is a communication point that lets data go into and out of a program.



name spaces are implemented through the file system, using programs like ls and cd, so you can "discover" which information is available to the program, including which programs are available.  Code is data, data is code.



So, what is shell?  It is a common meeting ground for programs written in ANY language to communicate with each other, and for the operator to communicate with those programs and connect them together on the fly.  CORBA/COM seems to have taken some of the shell concepts and tried to turn them into an API, as has D-Bus.



I think any command shell based on newLisp will have to provide the features mentioned above (perhaps excepting tab-completion)



People have wondered what a GUI/graphical shell would look like.  Taking into account what a shell actually does, maybe some concept is now possible.



A file selector dialog, for instance, should allow you to keep selecting files until you are done, but I've never seen a file selection dialog that lets you select files in multiple directories at once.  Yet I implemented this several years ago.



Pipelines could be created graphically, using a graph drawing type of tool.



Shell, at heart, is about connecting "black boxes" to each other.  Anything that makes that easier or more intuitive is capturing the essence of shell.  If a system could change something about black boxes, what?  I would want the system to compile the black boxes with the following information: what entry points (file descriptors, unix sockets, fifos) does it offer service on, and what is the name and description of that service.  Like a registry.  We already know what libraries a black box links in, unless it is statically linked, or unless it dynamically links in the library itself at a later point.  This also is a point that causes some ambiguity from a discovery point.  Perhaps the proc filesystem already has this information.



Given the nature of what shell does, I'm not sure how much more graphical it could be made.

[rrq]

Inspirational it is, indeed, even if I'm not yet at a stage of wanting to slice all my bread with the newlisp knife :-) Though, considering the recent bash scare, it's even so more refreshing to recall the well-balanced lack of bloating in newlisp. "Practical makes perfect"?



Veering back towards the initial topic, I thought to add my opinion in chorus with Lutz, that extending is the better option.



And, though I suspect I missed a point here, you'd typically do this by writing "#!" scripts with newlisp code, instead of and without intermediate shell scripting. For me it usually ends up in a collection of .lsp snippets that get used and reused in various constallations to make up the individual "commands", so I have my final "distribution packaging" to make each "command" a self-contained "#!" newlisp script.