threadring in newlisp

[TedWalther]

After meditating a bit, about being able to "send" and "receive" between spawned processes, I had this idea.  Could you expose the send/receive mechanism just a little bit?  That way I could "send" a message to the parent, requestion the communications port/socket/handle for some other child.  Then I could communicate directly with that other child.  And I could instruct the parent how to handle such a request from the child.  That would save having to make a socket pair for every single child/child pair.



Have you found that mmap is faster or slower than SysV shm api?  Any thoughts on switching to the pthreads api to double the speed?

[Lutz]

I found the shm API to be the slowest followed by memory sharing with semaphores, followed by pipes and socket pairs beeing the fastest.  I implemented all of these completely and compared speed on different platforms.



The pthreads API happens on only one core on most platforms and in the same process space, and because of that, doesn't scale well on multicore CPUs. Look at cores-used stats in the Debian benchmarks. Also, when running in the same process space, newLISP internal memory structures get messed up, if not duplicated for each thread.



Regarding, child to child communications, for now I suggest using either the proxy approach as outlined in the docs, or creating pipes, which works almost as fast as socket pairs. I understand that the best would be, if the same 'send' and 'receive' could be used, but haven't found yet a viable solution. Ideas, how to do this abound, but the devil lies in the details. Any solution needs some central registry for resources, either maintained by the OS kernel or by the newLISP parent process.



At the moment, the most important thing is, to use the existing API and find typical use patterns for solving real world programming problems. From there we will know, how important it is to extend send/receive by child to child comm., or how much can be done with the existing API.



On the Tips & Tricks page, there is a new category "Parallel and Distributed Processing" with an example, how to do parallel web page retrieval. We need more real world examples to show using the 'spawn' API.

[Kirill]

On the Tips & Tricks page, there is a new category "Parallel and Distributed Processing" with an example, how to do parallel web page retrieval. We need more real world examples to show using the 'spawn' API.

My http://km.krot.org/code/newlisp/yubi.lsp.src.html">YubiKey verification library uses (spawn) to query verification servers in parallel. It's quite basic, but real world anyway.

[Lutz]

Thanks for the hint. Parallel network access is definitely a good candidate for 'spawn', as is anything else waiting for I/O.

[cormullion]

My only use of spawn is in my irc-bot:

Code Select Expand

(spawn 'p1 (eval-string tmp))
(if (sync 2000)
    ; looking good
    (set 'result (string p1))
    ; no luck
    (begin
        (set 'result (string "error:" p1))
        (abort)))

[Lutz]

What is it, you have in tmp ?

[cormullion]

Usually just any newLISP string: e.g. on #newlisp at the moment



cormullion: (+ 2 3)

newlithp: 5



still a work in progress, though :)