FFI fcall in principle working :-)

[sunmountain]

This works now:

Code Select Expand


(import "msvcrt" "printf" "cdecl")
(set 'cprintf (fcall printf c_int c_void_p c_int))
(cprintf "The answer is %i" 42)
(exit 0)


@lutz: please drop me a note with your email, I'll send you the tar file.

[Lutz]

Great! I have sent you a pm with my email address.

[sunmountain]

Now it is possible to return float values directly (which does not work with imported functions):

Code Select Expand


(import "msvcrt.dll" "log10")
log10<77C4D060>
; define a fcall object return type is double, takes one double as parameter
(set 'clog10 (fcall log10 c_double c_double))
log10<77C4D060>
(clog10 1000)
3
(number? (clog10 1000))
true
(format "%.2f" (clog10 1000))
"3.00"
(format "%.8f" (clog10 3.1415926))
"0.49714987"


Right now, not nearly all types are supported, and type checking and error detection are rather weak.

But I'm progressing.

[Lutz]

Great job! Thanks!



I have your code working on Mac OS X, UBUNTU Linux and Win32 XP SP2.



I made a few changes to some files. But ther are not important for you at the moment. I have not touched nl-import.c at all.



You can find the changed files in:



http://newlisp.nfshost.com/sunmountain/">http://newlisp.nfshost.com/sunmountain/



There is a new qa-ffi including your current tests and platform independent. We need more complex tests later with multiple argument lists with mixed types.



The win32-ffi.h is made of the Redhat ffi.h and ffitarget.h. We can put the Windows version of libffi.a on the newlisp.org website for developers.



At the moment I am including the same win32-ffi.h for UBUNTU Linux, but that should change because not everybody is on Intel platforms.  Developers have to get the development version of libffi. libffi.so.5 itself comes already installed on UBUNTO, but there was a libffi.so -> libffi.so.5 softlink necessary.



Om MAC OSX libffi and header files are part of a standard OS install.



There are some changes I want to make regarding the symbol creation stuff in newlisp.c but that has time for later.



As you say, don't worry about error detection stuff, or just throw some generic error.



ps: updates are also in http://newlisp.nfshost.com/downloads/development/inprogress/">http://newlisp.nfshost.com/downloads/de ... nprogress/">http://newlisp.nfshost.com/downloads/development/inprogress/ , but careful, content changes frequently and unannounced.

[sunmountain]

I just incoporated the files, thanks.

I've cleaned the code a bit, and patched another file (nl-filesys.c):

Code Select Expand


CELL * p_makeDir(CELL * params)
{
char * dirString;
UINT mode;

mode = 0777; /* drwxrwxrwx  gets user masked to drwxr-xr-x on most UNIX */

/* consume param regardless of OS */
params = getString(params, &dirString);

if(params != nilCell)
    {
    getInteger(params, &mode);
    }

#ifdef WIN_32
return(mkdir(dirString) == 0 ? trueCell : nilCell);
#else
return(mkdir(dirString, (mode_t)mode) == 0 ? trueCell : nilCell);
#endif
}


Here's the corresponding patch, which remedies compiler warnings:

Code Select Expand


--- ../newlisp-10.3.6/nl-filesys.c      2011-11-18 20:05:00 +0100
+++ nl-filesys.c        2011-11-22 21:15:50 +0100
@@ -771,17 +771,17 @@
 CELL * p_makeDir(CELL * params)
 {
 char * dirString;
-mode_t mode;
-UINT inMode;
+UINT mode;

+mode = 0777; /* drwxrwxrwx  gets user masked to drwxr-xr-x on most UNIX */
+
+/* consume param regardless of OS */
 params = getString(params, &dirString);
+
 if(params != nilCell)
     {
-    getInteger(params, &inMode);
-    mode = inMode;
+    getInteger(params, &mode);
     }
-else
-    mode = 0777; /* drwxrwxrwx  gets user masked to drwxr-xr-x on most UNIX */

 #ifdef WIN_32
 return(mkdir(dirString) == 0 ? trueCell : nilCell);


The only weakness here is the cast of UINT to mode_t, the maximum value for mode_t (for which mode_t makes any sense) is 7777

in octal notation, which is 4095 in decimal notation - which can be expressed in 11 bits (0xfff in hex).

On some platforms mode_t is unsigned short, on some unsigned int - so either 16 or 32 bits.



One should be aware of such things, perhaps by putting a note in the docs ?

Or by fencing the maximum value (and having a note):

Code Select Expand


...
getInteger(params, &mode);
mode = mode > 0xfff ? 0xfff : mode;
...


Then it should be safe.



EDIT:

I just compared the size of the striped newlisp.exe binaries with and without FFI.

FFI makes ist exactly 4096 Bytes larger - which is hex 0x1000 :-)

[Lutz]

Overhead for the extended is very small on OSX and Linux too. On OSX libffi.dylib comes installed by default and on Windows it seems, it's only required for compiling? After making newlisp.exe I could remove libffi.a and it would still run qa-ffi.



Maybe it will be possible to have the extended ffi then at least in the standard newLISP versions of Mac OSX and Windows?



Also could verify working on the 64-bit version of newLISP on Mac OSX.

[sunmountain]

Perhaps just run ldd on the newlisp binary on OSX to see which dylibs it requires.



Under Windows Dependcy Walker just lists libffi-5.dll (the internal) name, which is not present.

I'll take a look into this, now I don't know why it works.



BTW, which version of libffi is present in OSX ?

[Lutz]

On Mac OSX it's not ldd but otool, which lists a dependency on libffi.dylib version 1.0 ?!. The ffi.h header file on the system is a RedHat file from 2003.

[sunmountain]

That seems reasonable:

http://www.opensource.apple.com/source/libffi/libffi-18/include/ffi.h">//http://www.opensource.apple.com/source/libffi/libffi-18/include/ffi.h



The problem to me seems to be, that this file may be present at several locations in an OSX install,

for example the ruby cocoa bridge brings its own (newer) version.



DLL hell reincarnated on Macintosh ...



As Apple ships this version, it might be good enough, though.

Alternatively we could document the process of building a newer one.

[Lutz]

I simplified the API. The extended ffi API is no an extension syntax of the existing 'import':



For details see here:



http://www.newlisp.org/downloads/development/inprogress/CHANGES-10.3.7.txt">http://www.newlisp.org/downloads/develo ... 10.3.7.txt">http://www.newlisp.org/downloads/development/inprogress/CHANGES-10.3.7.txt

[sunmountain]

I see ...

Cool :-)



I did only make small progress today, but this now works as expected:

Code Select Expand


(import "msvcrt" "sprintf")
(set 'res (dup " " 50))
(set 'csprintf (fcall sprintf c_int c_string c_string c_int c_double))
(set 'PI 3.1415926)
(println (csprintf res "the answer is always %i, or %.8f" 42 PI))
(println res)
(exit 0)


which gives:

Code Select Expand


newlisp.exe fsprintf.lsp
38
the answer is always 42, or 3.14159260


This honours the fact that a pointer is just that: a pointer.

I will make this more general (for where newLisp has primitives like integer), so when you

declare a parameter as pointer (c_string, c_pointer), and later feed it with an integer symbol, the content changes respectivly - so there is no more need for unpack here.



And I did more clean up, as I think that the (old now) fcall thing is almost done.



Next on my list are these:



* finish ffi_call execution

* make closures work (aka callbacks)

* create and manage structures at runtime

* wash the dishes :-)



But first I need to merge again ...

...



I hope to have this all done before holiday season :-)

(then I'd like to create a Android port)



Thank you Lutz for the support !

[Lutz]

The "cdecl" declaration is not necessary any more when using the extended FFI, libffi handles calling conventions automatically.



All data types supported by newLISP are now implemented, for a list see:



http://www.newlisp.org/downloads/development/inprogress/CHANGES-10.3.7.txt">http://www.newlisp.org/downloads/develo ... 10.3.7.txt">http://www.newlisp.org/downloads/development/inprogress/CHANGES-10.3.7.txt



So the first part of your list: "finish ffi_call execution" is done, except for some minor details, like allowing numbers for string pointers. At the moment that flew out the window and all parameters are type checked by default as of declarations in the extended 'import' syntax, but I will disable that again for string buffers to be able to pass numbers too.



@sunmountain: looking forward to the extended ffi callbacks ;-)

[sunmountain]

I'm asking myself what was the best verb in newLisp for the ffi closures.

Many people have some anticipation when they read closure, and

would probably confuse or disappoint them.



It could even be made  a special case of the standard callback symbol,

I think that would be natural, after fcall has been merged into import.



Anyway Lutz - I'm into #2 of my list :-)