• 'nil' and 'true' are allowed in memory shares
• new flavor osx_lib for newlisp.so on Mac OSX
• new flavor tru64 for Compaq/HP Alpha 64bit CPU
• 'timer' now has millisecond resolution
• 'read-buffer' can take a wait string similar to 'net-receive'
• Mac OSX installer package for 10.3
• flushing buffers for non terminal STD I/O lets newLISP be UNIX inetd server
For files and change notes see http://newlisp.org/downloads/development
Lutz
ps: this is the last release before v.8.6 in mid June
timer in newLISP for windows:
Quote
There is a 'SetTimer' function in Windows, but I have not come around yet to try it out for newLISP.
Any plans to get this in 8.6?
;-)
The code is already done, see win32-util.c. Volunteers please step forward and make it work ;-)
Lutz
ps: needs the right link flags
After adding the macro -DWIN32_TIMER to the makefile, commenting the WINCC macro in primes.h, and adding this line
#define TRACE_SIGALRM 0x8000
...to 'newlisp.h' the make mingw target compiles and links with no problems - no extra linkflag required, it seems?
However, it is the p_timerEvent in 'newlisp.c' which does not do anything, right...? The story at MSDN about timers is very big and requires some fundamental decisions... :-(
Peter
Instead of TRACE_SIGALRM, it should now be TRACE_TIMER. I wrote the Win timer stuff before making the change from using alarm() to setitimer() for the UNIX timer function.
It compiles and links but the callback function timerFunc() is never called, the reason is that the application has to be linked as a Windows application with one or more of the Win libraries like ws2_32 and perhaps a .def file is needed exporting timerFunc(). May be it also needs a WinMain(), the main callback function in a Windows application receiving events.
Lutz
In the MSDN docs the following remark is shown:
Quote
When you specify a TimerProc callback function, the default window procedure calls the callback function when it processes WM_TIMER. Therefore, you need to dispatch messages in the calling thread, even when you use TimerProc instead of processing WM_TIMER.
This text was really a riddle to me. But the demo program below shows a working SetTimer function:
#include <windows.h>
void timer_proc(HWND a,unsigned int b, unsigned int c, unsigned long d) {
MessageBox(0, "Hello", "MessageBox", 0);
}
int main(int argc, char *argv[])
{
int iTimerID = SetTimer(0, 0, 300, (TIMERPROC)timer_proc);
MSG m;
while (GetMessage(&m,0,0,0)) {
TranslateMessage(&m);
DispatchMessage(&m);
}
return 1;
}
Compile with 'gcc -o demo demo.c'. Pretty useless program but to get the idea. The timerevent is processed to the program itself, which must fetch it from the event queue.
So your "CELL * p_timerEvent(CELL * params)" must get the message of the event queue and dispatch it by itself. Probably you want to do this in a separate thread to let the newLisp programmer use the prompt.
Peter
Thanks Peter, for digging this out, looks like I don't need WinMain() as long as I use GetMessage().
Lutz
Quote
Volunteers please step forward and make it work ;-)
I know you were only joking but I couldn't resist the challenge!
Hope it helps.
Peter
Turns out that GetMessage() blocks when no message is received, which is none if you filter for WM_TIMER.
In the end I just created a thread using _beginthread() from the Win32 runtime library and watch the time in it, which turned out to be a very short solution with very little code added in the Windows compile.
But GetMessage() could be an interesting function for 'import', and then a Win GUI could be built fetching messages for the keyboard, mouse etc.. Not a single code change would be required in newLISP, in fact it could be done with any of the previous versions. But I leave that to the volunteers ;-)
Lutz
Sounds like a lot of work to me... Besides the GUI API of Win32 is a bit foggy. My focus is Unix anyway! ;-)
Peter
Actually it came out pretty short (look for it in the next release). The Win32 _beginthread() doesn't need much stuff around it. The resulting 'timer' funtion works similar to the 'timer' in the Linux/UNIX versions.
But I agree, having the 'timer'
Lutz
Hey Lutz,
Sorry to see you dropped cygwin support. I was one of the (few?) cygwin users.
Say, has anyone approached you about supporting/porting to SGI IRIX 6.5 (mips, n32)? We have several here at work on which I'd love to run newlisp. I really don't know where to start; however, if (1) you don't have anyone else, (2) you can suggest how to start (i.e. which makefile I could start from), I could give it a shot in my spare time.
BTW, any IRIX users out there who have newlisp running on an IRIX box, or who are interested in a port to IRIX?
--Rick
Hi rickyboy,
I just finished a port of newLisp to Tru64Unix starting with the Makefile for Solaris.
The biggest trouble came from the fact that Tru64 Unix is a 64-bit operating system. NewLisp however expects pointers to be of 32-bit size. How about Irix?
I have a list of changes which I needed to put into the newLisp sources, would you be interested to see them, just to get an idea where you may run into?
Peter
Good to hear from you rickyboy,
About CYGWIN, sorry that had to hit you rickybody ;( with the drop of CYGWIN. I don't maintain a full CYGWIN environment on my Wintel machine anymore, using the MinGW compiler exclusively when in Cygwin, the next flavor to drop will be Borland BCC, I only keep it because of the nice turbo debugger helping me out.
If you have any problems compiling the Cygwin flavor let me know and I try to help. There could be some problems, because since introduction of signals and timers I have not compiled that flavor.
About an Irix port, I love those machines, having used them more then a decade ago. I would just start with the Solaris or BSD make file and go from there. Most likely one of the two will get you there quickly. These are the areas where you might run into tweaking:
signals 'signal'
timers 'timer'
shared memory 'share'
time date stuff (several newLISP functions)
sleep 'sleep' (no big deal -DNANOSLEEP or not)
vasprintf(), vsnprintf(), my_vasprintf() 'C' functions: read man pages for vasprintf() and vsnprint(). If you don't have vasprintf() take my_vasprintf() in nl-filesys.c. In that case check return value of vsnprintf() on your platform, see my_vasprintf() source for details.
Most of that stuff is in nl-filesys.c. Probably one of the other flavors has already what you want. If you have GCC for 32bits on your machine things will go smoothest. If you can give me an account on one of your IRIX machines I can help you. I can sign NDAs or whatever you might need for your employer.
The tru64 port Peter/pjot did was probably one of the toughest, and even that post was done in a few days in Peter's spare time.
Lutz
Oh, and I almost forgot the most important thing:
Compile and run types.c from the newLISP source distribution and compare with CYGWIN/MingW/Linux output and it will tell you quickly if problems are to be expected!
Lutz
Thanks Peter! Thanks Lutz!
I will try to find some time tomorrow to start on the port.
--Rick
Quote
You forgot that the BCC-DLL is the only one which runs under .NET as unmanaged code! Should we loose this ability? Could you think of other options for .NET?
Oh yes, I forgot, then it stays, BTW timer doesn't work in with BCC.
Lutz
Quote
Unfortunately I dont's see a UNIX style fork (which creates an exact copy of the parent process) any time soon. The only version I ever saw is the one in cygwin, but I didn't find it to be reliable enough. Its just something Windows isn't build for. The Win32 thread facility turned out nice for the timer, but else doesn't have much use in newLISP.
1) Indeed the Win32 platform just does not have something like 'fork', it's not part of the OS. Still I wonder: is it not possible to simulate a 'fork' using
2) The Cygwin implementation is also a simulation, but is it really that bad? At least there is the possibility to use the 'fork' command in Windows. Besides Cygwin, also Perl uses a 'fork' emulation in a Win32 environment:
http://www.xav.com/perl/lib/Pod/perlfork.html
Though it has certain limitations, it will make newLisp more complete.
Peter
UNIX fork gives you a copy of all the whole parent process, which is what I need. For that reason simple threads won't work.
Lutz
So what do you think about an implementation similar to Perl?
I believe Perl and newLISP are doing the same thing, using a UNIX fork().
Lutz
Yes, in Unix probably. But how about the Windows emulation for a 'fork'...? Could you use the Windows emulation of 'fork' for Perl also for newLisp? It will make newLisp more complete... :-)
They seem to do a similar thing CYGWIN is doing, Perl has a 'perl-clone' function copying all memory from the parent process when simulating fork() on Win32.
Lutz
Well, so that is good news! :-)
...but serious, I am a little bit confused here... because a few messages back, you say:
Quote
UNIX fork gives you a copy of all the whole parent process, which is what I need.
And here you say:
Quote
Perl has a 'perl-clone' function copying all memory from the parent process when simulating fork() on Win32.
So isn't this exactly what you want for Windows then? Or do I misunderstand you somewhere?
Sorry to be so nagging, but it seems that a Cygwin-like 'fork' seems to work for many other Unix tools ported to Windows, including Perl. So why not use it for newLisp as well?
Peter
Yes the 'perl-clone' or the cygwin fork() are both Win32 implementations of the UNIX fork() by using Win32 threads and code to replicate/clone the parents memory.
I went to the source code of both Perl and Cygwin, its not a small task to understand it and rip it out, usable for newLISP. Anybody who wants to help out here is welcome to work on this, it requires both, good Win32 internals knowledge and UNIX knowledge and it has to be done in a way that thread-process communications by means of shared memory and syncronized by semaphores still work well, open file handles have to be duped etc.
Everything has to be small enough not to change newLISP's small footprint on disk and in memory, which is one of the reasons people like using newLISP.
Lutz
I am just wondering if there was a particular reason why there is no 'fork' emulated in newLisp for Windows. Of course I am willing to help you out but my Win32 knowledge unfortunately is limited... :-(
If it appears too difficult, too much work, or if the resulting newLisp binary becomes too large, well, then probably it might not be a wise thing to do.
Peter
It is a bigger piece of work, thats why I have not come around doing it. I also believe that most people wanting to do a multithreaded application in newLISP will be found not on Windows but in the UNIX world. So for the Win32 flavor 'fork' doesn't have the same priority as it has on UNIX.
Lutz
Is there maybe a possibility to write a 'fork' emulation with newLisp? The commands 'cpymem', 'process' and so on could be used for it, I suppose?
The whole thing can be put in a separate context, so your newLisp binary will not grow larger.
Still it is an emulation, but it's better than nothing.
Peter
(PS your suggestion about those Unix people I agree with. The whole 'fork' idea for Windows is not that urgent.)
Yes, you could do that theoretically, interesting idea, although it would be slow.
Lutz
Well, I got something working.
http://www.turtle.dds.nl/newlisp/winfork.lsp
The original "prodcons.lsp" works in Windows now with some small modifications:
1) Removed the (wait-pid) function, does not exist in newLisp for Windows
2) Load the "winfork.lsp" context first
3) Syntax is: (winfork '(function)) , so calling the symbol of the function instead of the function itself.
The "prodcons.lsp" demo program for Windows can be found here:
http://www.turtle.dds.nl/newlisp/prodcons.lsp
Anybody who can improve the WINFORK please do so (and let me know...;-) )
Peter
Excellent! but here 2 small suggestions to make the code even shorter and faster: (1) use 'source' to convert symbols to their source code:
(set 'a 123)
(set 'b 456)
(define (foo x y) (+ x y))
(set 'environment "")
(dolist (s '(a b foo)) (write-buffer environment (source s)))
; environment now contains all the source for 'a 'b and 'foo
'source' takes care of all the conversion
(2) when using (write-buffer ...) with a string instead of a file handle, you have a very fast in-memory string append.
Lutz
Nobody can beat the master! ;-)
The WINFORK context is much smaller now and works pretty cool. I uploaded new a version at
http://www.turtle.dds.nl/newlisp/winfork.lsp
Now the 'fork' command can be used in Windows as well!
Peter
Further testing seem to reveal a memory leak in the Windows-version of 'share'.
If you change the 'prodcons.lsp' example to let it start with '(run 100000)' and ALSO comment all semaphore stuff; watch your processlist, the spawned newLisp processes will grow until they stop (at 134mB on my WinXP system).
However, if I change the 'prodcons.lsp' example like this:
(load "winfork.lsp")
(constant 'wait -1 'sig 1 'release 0)
(define (consumer n)
(set 'i 0)
(while (< i n)
#(semaphore cons-sem wait)
#(println (set 'i (share data)) " <-")
(println "Consumer")
#(semaphore prod-sem sig)
)
(exit))
(define (producer n)
(for (i 1 n)
#(semaphore prod-sem wait)
#(println "-> " (share data i))
(println "Producer")
#(semaphore cons-sem sig)
)
(exit))
(define (run n)
(set 'data (share))
(share data 0)
(set 'prod-sem (semaphore)) ; get semaphores
(set 'cons-sem (semaphore))
(set 'prod-pid (winfork '(producer n))) ; start threads
(set 'cons-pid (winfork '(consumer n)))
(semaphore prod-sem sig) ; get producer started
(sleep 10000) ; make sure semaphores are not released too early
(semaphore cons-sem release) ; release semaphores
(semaphore prod-sem release)
)
(run 100000)
(exit)
...then the memory usage is not growing. So it is definitively in the (share).
If I run the original 'prodcons.lsp' in Linux, then the memory usage is OK.
Peter
If I run this on a Win2000 machine the memory usage also increases to 134 Mb (but newlisp does not crash).
Probably the issue is caused by these lines in win32-util.c (lines 224-227):
UINT * winMapView(UINT hMemory, int size)
{
return((UINT*)MapViewOfFile((HANDLE)hMemory, FILE_MAP_WRITE, 0, 0, size));
}
Every time the function MapViewOfFile is called, it will cost you some additional memory. Therefore each process should only call it once, and then refer to the same returned address every subsequent time.
I also found this remark:
Quote
Note also that MapViewOfFile() increments a usage counter that is decremented by UnmapViewOfFile() or on process termination. The physical named object is not removed from memory until the last process has unmapped the object, and the usage counter is set to zero.
...at http://www.jps.at/dev/kurs/3-19.html.
Peter
Thanks Peter this is fixed for 8.6.0.
Lutz