Release 10.1 delayed to June 22nd

[Lutz]

Release 10.1 has been delayed to June 22nd to implement improved messaging support for concurrency and multi core CPUs.



http://www.newlisp.org/downloads/development/newLISP-10.1-Release.html">http://www.newlisp.org/downloads/develo ... lease.html">http://www.newlisp.org/downloads/development/newLISP-10.1-Release.html

[itistoday]

This is exciting news indeed! :-D



I'm curious though as to how this will work.  This sounds a lot like actors for newLISP, but how will the receiving end process these messages?  And as actors need to be able to coordinate with one another, will there be a way to know which actor sent you the message so that you can store a reference to it so that you can send a message back to it?  Will this allow for a user-defined main-loop in each spawned process?



One really great actor library that I'm using right now is called http://plactorkit.googlecode.com/svn/tags/plactorkit-1.0/docs/index.html">PLActorKit.  It's for Cocoa/Objective-C, but the design is neat and the ideas universal.



It has support for two paradigms of Actors:



Paradigm #1: Send/Receive

=================

This is illustrated by the "A Simple Echo Actor" example in the above link. The basic idea is that each thread can call send/receive on another actor. 'receive' blocks and returns messages with attached objects, and each message has a reference two who sent the message so you can send a message back to that actor.



What's nifty about this paradigm is that it allows each actor to maintain its own runloop, allowing it to perform other tasks besides simply sending and receiving messages. The receive method can take an optional timeout, and therefore you can have code that checks to see if any messages have arrived, and if none have after a timeout, the actor can go ahead and perform other tasks.



Paradigm #2: Runloop handled by actor system

=============================

This is illustrated by the "Transparently Proxying Objective-C Messages with Actors" section in the link above. In this paradigm, each actor has a main run loop that is implemented by the actor system itself, not the developer.



It allows for even cleaner code at the possible expense of flexible functionality. Instead of writing your own runloop that reads in messages and calls the appropriate functions, this method is more of an RPC mechanism. You essentially call a function directly on an actor (instead of composing a message), and that function will get called for you, and a possible response returned.



In this paradigm, unless some sort of "input source hook" is provided by the actor system, the main runloop can do nothing else other than receive RPCs and execute them, it cannot for example, check with some database of its own accord, it must be told to by another actor.



==========



Another question, in both paradigms, is the question of asynchronous message passing.  Will the system allow for this, or will each message/function-invocation block? And if it does allow for it, how will responses be read? Will it incorporate the concept of "promises" aka "futures"? Or will the actor be required to simply perform another receive call to check and see if a message has arrived with the response from the other actor?



Sorry for all the questions, this is just a topic of great interest to me.

[Excalibor]

Lutz,



Sounds cool, indeed!



Anyway, lots of changes for the new release, it'll surely be worth the upgrading!



Once question, though. I know that Windows, not having fork(), it's a hindrance for multiprocess... Any possibility of having pseudo-fork support on Windows by using threads of some kind? (posix or green threads, for example?)

[Lutz]

The new message interface can act on two levels of increasing functionality:



* On the lowest level parent processes communicate via shared memory with each of their child processes. On this level the new message facility is just an incomplete actor implementation. Peer processes are not directly reachable on that level. But parent processes easily can act as a proxy to send  messages to any other node in the process tree.



The real power of the new message system comes from the fact that messages can contain lisp code, which leads us to the next higher level of functionality:



* On this higher level complex actor and agent models can be implemented. This is due to the fact that any lisp expression can be sent inside a message to be evaluated in the recipient's environment. The recipient does not need that functionality pre-implemented as in OO programming methods. newLISP's dynamic scoping mechanism allows pieces of code sent around to accomplish any tasks evaluating that code in the target environment.



Tasks could be: setting variables, calling functions, injecting entire new code pieces and in turn sending new messages or creating new processes.



* Message recipients (either parents or children) listen in a run-loop for messages using pids as addresses. Each process has the parent's pid in (sys-info -4) (new in v.10.1) and its own list of children pids can be retrieved with (sync). At this moment (non-blocking) run-loops must be implemented by the developer, which is trivial to do, including:w

 implementation of timeouts.



The new message facility can accomplish a lot with very little overhead-code and used by less experienced newLISP programmers. On that lower level the message system would be used to transfer pieces of data without the need to care about semaphores or other synchronization mechanisms to avoid collisions.



Using evaluated messages, elaborate actor and agent schemes can be implemented with no limits in complexity. As a building block for higher level functionality, the basic messageing mechanism is extremely fast, comparable to the speed of a simple variable assignment. A complete message exchange takes longer depending on the time-slicing of the scheduler in the platform OS.



This week I am testing the new features and am developing a example programs for the manual and Code Patterns documentation.



Ps: everything Mac OS X and other Linux UNIX only



Comment on Excalibor's post:

Green threads are difficult to use in a fully dynamic language like newLISP they also don't suport multiple core CPUs. Some projects are moving away from green threads towards native threads or to classic Unix forked processes (Chrome and other web browsers). Native threads and forked processes do support multicore CPUs today, offer more stability via better isolation, can be preempted and their scheduling is done by the OS. newLISP processes are lightweight because of their: short startup time and low memory footprint.

[Excalibor]

Lutz,



Sounds really enticing, can't waitto get the new version a documentation!



As for threading, I guess I was really thinking about fork() emulation under Windows... If native threads are the way to go, maybe a windows guru could provide a test implementation so we can use Cilk and concurrent code going really parallel under windows? I have several unix machines for real work, but my developing workstation is a windows xp box... I wonder if cygwin already provides this? maybe I should try to compile newlisp with cygwin and see...



anyway, I can't wait for Monday to arrive :-)



thx

[Lutz]

Cygwin does supply an implementation of fork, and a few years back the Windows version of newLISP was compiled under Cygwin and could run fork.



Unfortunately the fork was not reliable and slow. For this and other reasons (e.g. an extra Cygwin DLL had to be included), newLISP moved to MinGW.



User Pjot from http://www.turtle.dds.nl">http://www.turtle.dds.nl and http://www.gtk-server.org">http://www.gtk-server.org wrote a 'winfork.lsp' in newLISP a few years back which could run various sample programs. Basically he started a process using 'process' and  preloaded it with all of newLISP's environment. The whole thing was only about 10 statements.



But that wouldn't help you with the Cilk API and the 'message' function in upcoming 10.1 which need a 'C' fork() inside newLISP on which they can build.



The best would be to install Sun VirtualBox on your Windows machine and install UBUNTU Linux. It runs quite fast and if you load down the original Sun Java JRE (1.5 or later) you have the newLISP-GS IDE too.