threadring in newlisp

[TedWalther]

If it is any consolation, at least we are beating out Ruby, Perl, and C# Mono.  Although this isn't a problem with Mono; F# Mono is as good as Erlang.

[cormullion]

Are these times comparing interpreted and compiled languages? Seems odd to allow compiled languages to do their homework before being asked to run their code, whereas the interpreted languages have to respond to the problem without prior notice... :)

[Lutz]

Seems odd to allow compiled languages to do their homework before being asked to run their code, whereas the interpreted languages have to respond to the problem without prior notice... :)

...best explanation of compilation versus interpretation, I have ever seen :)



To Ted: what version of newLISP did you use? I am surprised by the good results, I thought that spawn'ed (fork'ed internally) processes are more heavy-weight than pthreads. The advantage of using different processes instead of threads (running in the same process space) is, that multiple processes can run on multiple cores. But it seems that pthreads are really forked processes on Linux?

[TedWalther]

Lutz, I used your latest release, 10.3.4, under Ubuntu.  I installed from dpkg.reactor-core.org.



Actually, I don't have a way to compare spawn with pthreads.  The C implementation on the shootout site, uses pthreads.  And it is more than twice as fast as the newlisp implementation using fork.



Have you found the same segfault behavior in the spawn version of my code?  Any idea why the parent process is burning so much cpu inside the (sync) function?



Yes, the compiled and interpreted languages are all tested together.  I like seeing how they stack up against each other.  Now that we have byte code, virtual machines, and on the fly compilation into semantic trees, the line between compiled and interpreted languages is blurry anyway.  It doesn't make much sense to quibble much about it.

[Lutz]

You can control the time spent in the parent process by changing the proportion of time in the sync statement and a sleep time in the until loop:

Code Select Expand

(until (integer? throwaway)
    (sync 20)    ; 20ms in internal sync loop
    (sleep 400)  ; 400ms giving up time for child processes
)


As you are only waiting for one of the child processes to finish and setting the 'throwaway' variable, the parent process can spend most of the sleeping and giving up time to child processes.



The crash in development version 10.3.4 happens when 'spawn' hits a resource limit. This will be addressed in development version 10.3.5.

[TedWalther]

Using (sync 100)(sleep 400), I found that the fork version is 7% faster than the spawn version.  I did another run of the fork version, this time with 339 threads instead of 503, and found that it was only 3% faster.  There must have been some system load slowing it down.

Code Select Expand


ted@ted-desktop:~/local/src$ time ./threadring4.lsp
Done by id 180
Seconds: 107.50

real 1m47.509s
user 0m47.800s
sys 0m55.270s
ted@ted-desktop:~/local/src$ time ./threadring3.lsp
Done by id 89
Seconds: 115.45

real 1m55.778s
user 0m55.370s
sys 1m22.130s


[Lutz]

That crash which happend to you in 'spawn' in development version 10.3.4, is now fixed in pre 10.3.5:



http://www.newlisp.org/downloads/development/inprogress">http://www.newlisp.org/downloads/development/inprogress

[TedWalther]

I'll test it now.  Another error, is in the time-of-day code.  I save the time-of-day at the start, then subtract at the end and divide by 1000 to get the number of seconds elapsed.  This works mostly, but sometimes, in both the spawn and the fork versions, I get a large negative number.  Any ideas?

[TedWalther]

ted@ted-desktop:~/local/src/newlisp-10.3.5-inprogress$ ./threadring3.lsp



ERR: cannot open socket pair in function spawn

[TedWalther]

I bumped the number of threads back down to 339, and the error went away.  I therefore think that the error message is wrong, possibly because of an off-by-one error somewhere.  It isn't segfaulting, but how come I can't spawn 503 threads, but I can fork 503 times?

[Lutz]

Additionally to the pipes created by the threadring program, 'spawn' creates a socket pair (based on Unix local domain sockets) for each process spawned and to be used by the message API with 'send' and 'receive'.



I will make the the creation of these sockets optional in the 'spawn' call. This will allow applications to not use the message API or only use it for slected childs spawned:



(spawn <sym-variable> <child-process> [true])



The optional 'true' parameter would cause the socket pair to be created, while without it, no socket pair would be created for that child.

[Lutz]

See http://www.newlisp.org/downloads/development/inprogress">http://www.newlisp.org/downloads/development/inprogress for these changes now implemented.



The threadring program will now run unchanged with a higher number of child processes in the ring.



The negative timing numbers will be addressed at a later time.

[TedWalther]

The socketpair is how the child process returns its final value to the parent, right?  What happens if you spawn a process with the socketpair absent?  Isn't that the same as a fork?

[Lutz]

The final return value of the child process is transferred via shared memory, not via sockets. Perhaps in the future the low level 'fork' and 'wait-pid' should be taken out in favor of the more high-level and practical 'spawn' plus the message API.

[TedWalther]

Or at least, don't take fork and wait-pid out entirely; put them in the posix module?



I just tested, and the spawn version is 2% slower than the fork version.  So, they are almost equivalent.