development release newLISP v.9.3.8

[Lutz]

• refinements of the Cilk API. On Win32 simulated API for writing platform independent code. On Win32 spawned functions will be evaluated sequentially



• read here for Cilk API documentation (spawn function): http://www.newlisp.org/newlisp_manual_9308.html#spawn">http://www.newlisp.org/newlisp_manual_9308.html#spawn



• miscellaneous fixes in Guiserver gs:text-area



Ps: this release has been compiled on Max OS X 10.4 (Tiger) again to make parse-date work correctly on Mac OX X 10.5 (Leopard) and let Intel users run also on Tiger.

[newdep]

Hi lutz,  thanks for the addons! ;-)



Question..why is this behaviour =>

First it isn't allowed, then it is..? (works with any symbol)





> (spawn *)



symbol expected in function spawn : *



> (spawn '*)

26778

> (sync 1000)

true



> (spawn *)

26779

> (sync 1000)

true

>

[Jeff]

Thanks Lutz.  This is starting to really shape up nicely.



Is there any way to set a limit on the number of processes spawned?  Since there is already the functionality to ignore spawn and execute sequentially (as for windows), it would just be a matter of implementing a guarded share that could be incremented by each process launched.  When a process completes, it decrements the value.  Then, if the max is exceeded, spawn would just evaluate sequentially.

[newdep]

Another undocumented feature is =>



(destroy 0)



which will 'kill' the itself (the current newlisp process).

[cormullion]

Great stuff Lutz - I'm looking forward to using this. But about this new spawn and sync stuff - can I ask some dumb questions? :)



Is it useful only  when there is more than one cpu available? Does it run differently or at all with only one cpu - and how does newLISP - or even me - know how many cpus there are?  Can it work by running threads on other machines connected over the network (like net-eval)?



Say I have a 2 GHz Intel Core 2 Duo - can I use these functions to make my programs run - say - 60% faster, if I can split the task up into two reasonable chunks, one for each cpu...? Or does the OS do that anyway (eg running my newLISP programs using one processor while picking up email with the other...)?



Do these new features replace or supersede any of the existing newLISP features?



OK - you can see, I have many questions! :)

[Jeff]

Corm,



You can find out the number of cpus on a machine (on unix, anyway) using:

Code Select Expand

(define (num-cpus)
  "Returns the number of cpus as known by sysctl, or 1 if the exec call fails."
  (or (int (first (exec "sysctl -n hw.ncpu"))) 1))

Spawn is just syntactic sugar for fork and share.  In the future, it may have a simple manager that limits the number of processes spawned in relationship to the cpu.



Scheduling is done by the CPU.  fork() is a function in libc, which is a (mainly) standard library on unix systems.  Fork copies the entire process into a child process that cannot talk to the parent process outside of shared pages of memory.  These processes cannot be launched on other machines (as in Erlang).



Spawn is useful for breaking up complicated tasks into smaller tasks when each task need not know about the others.  For example, if you were to map an expensive function over a small list (with fewer elements than the max number of forks a process can spawn on your os: it is around 250 on my osx box), you might profit by using spawn, especially if the expensive function spends a lot of time doing I/O or blocking.



None of these replace the current primitives.  They just provide a handy wrapper around them.  Rather than wrapping a shared page in semaphores, forking, and setting the result of the forked work in the shared page (again wrapping in semaphores), you can simply use spawn.  It just automates a common task.



If you want to manage processes by cpu number, you need to have some method of counting child processes, which libc does not provide.  Since forking is expensive and locking/unlocking shared memory is as well, my experience is that it is not worth it.  The way to split large tasks with scheduling is to fork one proc for each cpu (a worker).  Each worker knows what it is supposed to do, and can write the result to its own shared page.  You use a semaphore to tell the worker when to evaluate function x.  The worker then semaphores the manager (the initial process that forked the workers) that the work is finished and accessible.



To schedule, you would need to write a manager function that would decide which worker should be called next and send jobs to each worker.



It's a tedious problem to solve and requires a real grasp of semaphores and shared variables.

[newdep]

..Lets assume you have a 2 Gig Dual core ;-)



Your programs wont be 'faster' if you have a dual core.

They will though run more 'efficient'. But your OS And software

needs to know how to handing Dual-Core's. If the software is not

build for Dual core your programs can even be slower than on

a simple core.



As i can see it In newlisp spawn is a fork, so actualy you could

build a fork that has the the same behavior as spawn..which is

"splitting of from the root process and returning when done"



Now im not sure  if it was 'spawn' that took the whole newlisp

environment or if it was 'fork'..



..Jeff posted at the same time it seems ;-)

[lithper]

... how does newLISP - or even me - know how many cpus there are? ...

The place with LOTS of system information (all of it, or almost) is the "/proc" filesystem.

I'm not sure of its implementation on BSD/Mac, and I remember that on Solaris it's not ASCII, so some utilities must be used from command line to read it (or APIs from a C library), but on Linux it was a specific decision to make it ASCII.



So, to find out virtually all about your machine, it's sufficient to read (read-only) the appropriate file there, and minimally match/parse it.



So something like "cat /proc/cpuinfo" could do it in a Linux box  from shell - or an obvious equivalent from newlisp.

Same technique will do with network interfaces and parameters, memory etc.



If you have enough privileges, it's possible on Linux to set some of the parameters through /proc, too

[Lutz]

.. how does newLISP - or even me - know how many cpus there are? ...

Even with only one CPU core multiple processes can save time when processes are doing I/O and are waiting, e.g. when retrieving web pages or doing other network related I/O. When your computer has multiple CPU cores, the operating system will automatically distribute processes on what you have.

the "/proc" filesystem

Only on Linux and Solaris, but to use this for a more refined 'spawn', which would stop parallelizing processes when a maximum number is reached, reading the proc file system is too slow compared to other 'spawn' overhead. On the Mac and other BSD like UNIX, we have sysctl() (I mean the system C-function sysctl(), not the command line utility), which can retrieve info about processes running. We would need something similar speedy on Linux and Solaris to do more advanced scheduling.



For most applications the API as it is, and leaving the scheduling to the platform OS seems sufficient to me. After researching more about the possibilities on Linux and Solaris something might be added.



The next development release will implement error recovery for spawned processes which currently are lost uncontrollable when finishing with error, and fix some other issues pointed out by Norman.



ps: turns out there is a sysctl() function on Linux too, to retrieve system parameters and it seems to be compatible in many parts with the BSD sysctl() call.

[newdep]

Hi Lutz,



Sometimes (spawn <time>) boosts the CPU load to 99% for <time> period

, I cant realy simmulate it as being a constant problem but it happens now

and then... If I found a good test ill drop a posting..





Norman.

[cormullion]

Thanks for all the extra information folks - I may recycle some of it... :)



Just out of interest, I found that the primes example from the manual takes about 9 seconds, compared with about 12 for running the three (primes) expressions serially with no spawning. That's on an 2 GHz Intel Core 2 Duo. That's a reasonable speed improvement!

[Lutz]

On Mac OS X you can see the usage of the 2 cores using the Activity Monitor/Window/CPU Usage menu option.

[cormullion]

I just noticed that some of my code now credits newLISP version 5239 or similar, instead of 9300... :))



Has (sys-info) changed such that asking for the version now returns the PID... I know it's changed in the development releases, but didn't realise that all existing uses might return different numbers...



Is this going to stay - will I have to change all existing code that relies on this?

[Lutz]

Is this going to stay - will I have to change all existing code that relies on this?

There is an unstated (until today) contract that new stuff in 'sys-info' will always be inserted somewhere before the last 2 values, which are also the only static once in all the list for the same newlisp executable or library.



If you use (sys-info -2) for your version number and (sys-info -1) for the compile flavor (libray, UTF-8, IPv6), you are safe for all past, present and future versions.