Compile DLL

[sunmountain]

Hi there,

I got here in test.cpp this

Code Select Expand


extern "C"
{
__declspec(dllexport) double adder(double a,double b) {
return (a+b);
}
}


I then compile this with

Code Select Expand


C:Usersstefanffi>cl /LD /DLL test.cpp
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

test.cpp
Microsoft (R) Incremental Linker Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:test.dll
/dll
/implib:test.lib
test.obj
   Creating library test.lib and object test.exp


test.dll now exports a function adder.

But, when I start newlisp, import the func and start it, i get this:

Code Select Expand


newLISP v.10.3.5 on Win32 IPv4/6 UTF-8, execute 'newlisp -h' for more info.

> (import "test.dll" "adder")
adder<10001000>
> (adder 2.2 3.0)
-1717986918


It doesn't matter if I append "cdecl" or not.



I wrote a test prog to see if the dll is ok:

Code Select Expand


#include <stdio.h>

double adder(double a,double b);

int main(void) {
printf("%fn",adder(3.2,4.5));
return 0;
}


compiled it

Code Select Expand


C:Usersstefanffi>cl testtest.c test.lib
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

testtest.c
Microsoft (R) Incremental Linker Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:testtest.exe
testtest.obj
test.lib


and run it:



C:Usersstefanffi>testtest.exe

7.700000



Where is my error ?

[Lutz]

The newLISP import function will pass double floats correctly into cdecl compiled function, but will not return values as doubles. You can either return arrays of double floats or pointers to double floats and then use either 'get-float' or 'unpack' on those addresses. See also here:



http://www.newlisp.org/downloads/CodePatterns.html#toc-23">http://www.newlisp.org/downloads/CodePa ... tml#toc-23">http://www.newlisp.org/downloads/CodePatterns.html#toc-23



and here:



http://www.newlisp.org/downloads/newlisp_manual.html#get-float">http://www.newlisp.org/downloads/newlis ... #get-float">http://www.newlisp.org/downloads/newlisp_manual.html#get-float

[sunmountain]

I should have read the fine manual before, though.

I guess it has internally to do with the fact, that floats can't be represented as normal numbers

(duality pointer <-> long).



Time for a FFI library :-)



I took a look at several ffi libs, and I think it would be the easiest to extend newLisp's vocabulary for this.

As far as I have seen, it would be neccessary to create something like nl-ffi.c and correspondig

entries in primes.h.



Correct ?



Do you have a process to get (if it becomes usable) this thing into newLisp source ?

[Lutz]

Yes, that is the way to implement new functions in newLISP, and put a prototype of the p_xxxx function into protos.h.



But I think libffi is too big for  what newLISP tries to accomplish. You will lose newLISP's small size and short startup time, necessary for parallel and distributed tasks with multiple newLISP instances. That is why I recommend writing stub libraries for the case where the normal 'import' is not enough. Most users will never import a single library when using newLISP. Those who do, will most of the time do fine , with what the 'import' function can deliver.

[sunmountain]

But I think libffi is too big for what newLISP tries to accomplish.

• Not really. When stripped, it ads about 25 kB.
  
  I try to accomplish to do the same things with newLisp as with Python: penetration testing, automation, regression tests, gui development etc etc

You will lose newLISP's small size and short startup time, necessary for parallel and distributed tasks with multiple newLISP instances.

• loosing small size: not really.
  
  loosing small startup times: I don't think so, for the same reason + the fact, that ffi does not need big initialization procedures
  
  if I'm doing that sort of stuff, I should be aware of that - and perhaps compile an extended version for newlisp perhaps (if it really needs some sort of alien library), which is really easy IMHO

That is why I recommend writing stub libraries for the case where the normal 'import' is not enough.

• That is another option. But that would end in "import" again - which means I cannot return or get real newlisp cells etc, which would be benefical for defining structs etc.
  
  Approx. newlisps vocabulary would grow by about 10 pieces
  
  it would perfectly possible to have #defines for compiling support for ffi on/off

Most users will never import a single library when using newLISP. Those who do, will most of the time do fine , with what the 'import' function can deliver.

• I would not kick it out - leave it for simple things, but have ffi for complex(er) things
  
  Its low hanging fruit :-)

Perhaps a runtime plugin system could be a long term goal.





But, let me see how far I can get this weekend.

[Lutz]

Perhaps you can create a nl-ffi.c and, compile with: gcc -DFFI and have #ifdef FFI for primes.h and protos.h. Then it is easy to turn it on and off. newLISP only supports LP64 and ILP32 memory models, perhaps that helps to cut down on size for FFI stuff. At the moment there is no support for LP64 on Windows, mainly because MinGW doesn't offer it (yet).



I am not completely convinced yet, but I agree that it's worth to give it a shot.

[sunmountain]

Hi Lutz,

Perhaps you can create a nl-ffi.c and, compile with: gcc -DFFI and have #ifdef FFI for primes.h and protos.h. Then it is easy to turn it on and off.

That was (is) exactly my plan.

I'll take 10.3.6 as the base for this.



Due to the fact that I have a Win 7 dev box (and 32 bit only), I'm not able to test LP64 here.

I think it is not only the compiler (there is a http://mingw-w64.sourceforge.net/">MinW-W64), but even so the fact

that the MS runtime ecosystem is http://technet.microsoft.com/en-us/library/bb496995.aspx">LLP64.



And, I as far as I'v seen, newLisp is relatively hard bound to the standard unix os interface,

so porting it to the Microsoft C++ compiler (and the Windows programming model) would be a longer journey.

If at all, it is worth the effort - maintaining a piece of C/C++ for several platforms is one thing (lots of #ifdefs) - for several platforms + several compilers squares the number of #ifdefs.

And it would not solve the LP64/ILP32 issues - it would be even more neccessary to

make the code aware of several memory models (I remember sources, where very ugly things got done

to accomplish such).



And then the beauty of the newLisp source code will be gone forever.



Now it's small, well designed, structured and easy to adopt (it took me 10 min to get my own verbs working).

[Lutz]

Good to know that MinGW-w64 v1.0 is now available, thanks for the news.



Taking care of LLP64 in Windows 7, will take a little bit more time and testing above all. Most of the changes will be kept local to newlisp.h. Because newLISP is often compiled on small platforms - now even more because of the rise of mobile platforms - both 32-bit and 64-bit versions will need to be supported for a long time.

[sunmountain]

Taking care of LLP64 in Windows 7, will take a little bit more time and testing above all.

That's funny.

I was just wondering today, if there are automated tests as a regression after (automatic) newLisp builds.



Anyway, for the nl-ffi thing I'll write some.

[Lutz]

There is a shorter and a longer test suite. And both can run with more or less reporting. After making the newlisp executable, run from the main distribution directory:



For a short suite and less reporting:

make test



For a short suite with full reporting:

make check



For a long suite with less reporting:

make testall



For a long suite with full reporting:

make checkall



On a modern machine, just do a 'make testall'. I normally do:



make clean; make; make testall



all in a row, separated by semicolons. On Windows, I work in a MSYS Bash shell.



When you write a qa-ffi, write your diagnostics, whatever else, to the console. At the end do a:



(println ">>>>> <your description here> SUCCESSFUL")



or a:



(println ">>>>> PROBLEM <your description here>")



'make test' and 'make testall' basically run check and checkall but only report lines with at least 3 angle brackets ">>>".



You can run a single qa- file, as an example:



./newlisp qa-specific-tests/qa-net6



Most qa-xxx files are in the qa-specific-tests/ directory, but there are two 'qa-xxxx' files in the main directory: 'qa-dot' and 'qa-comma'. Both do a quick test of all functions in the system. qa-dot does it for locales with a decimal "." dot and qa-comma does it for locales with a decimal "," comma.



I also use qa-dot and qa-comma to check for cell or memory leakage. Comment out the last 'exit' statement in the script and report (sys-info). From the newlisp command line, do several (load "qa-dot") to check for cell leakage. After no more than two invocations the cell count should remain constant. Once in a while I use 'valgrind' to check for general memory leaks.



For main releases, I also check some files from the example/ directory and run the old modified Debian benchmark suite and applications offered at newlisp.org (the wiki and the ide).

[sunmountain]

I sat down an made a layout of my code.

To be honest: I think it will be neccessary to create a new cell type (most likely types),

for ffi calls, callbacks and some types.



I'd like to be able to do something like this (later):

Code Select Expand


(import "msvrcrt.dll" "printf" "cdecl")
(set 'cprintf (fcall printf c_int (*c_char c_int)))
(cprintf "The answer to all is: %dn" 42)




or so

Code Select Expand


(define (cmp a b) (set 'a (get-int a))(set 'b (get-int b))(if (> a b) 1 (> b a) -1 (= a b) 0))
(set 'qsort (fcall qsort c_int c_int (*c_int c_func)))
(set 'sortcb (fclosure  cmp c_int *c_int *c_int))
(set 'clist (farray c_int (34 2 3 5 6)))
(qsort clist (fsizeof clist) sortcb))


When newlisp tries to execute the line (cprintf ...) it must be able

to figure out how, like with executeLibfunction, which is bound to CELL_IMPORT_[DLL|CDECL].



Some more intrinsic than I thought first.



So, many #ifdefs :-)

[Lutz]

One additional type CELL_IMPORT_FFI should be enough.



Then 'fcall' registers the call pattern internally. I wonder if it would be possible to use existing symbols for call patterns:



*, string - generic pointer 32-bit or 64-bit depending on the newLISP version

char - one-byte unsigned byte

int - 32-bit

long - 64-bit (new symbol)

float - IEEE 769 double float

flt - 32-bit float



* and string would be used mostly for the same thing, but having both improves readability.



Everything else gets resolved with existing functions like pack, unpack, get-char, get-float, get-int and get-long for breaking up data structures and reformatting return values.



Don't try to cover everything under the sky. We can handle 90% of functions already, if we get to 99% that is enough.



If the thing isn't small enough and easy enough to use, then it's not worth it. Remember a module writer, who has to be knowledgeable in C anyway, would always be able  to write a stub to be used with the existing 'import'.



What I mean is, not having FFI doesn't prevent people from writing modules. Having FFI only makes sense if it really is a lot easier to use than writing your import stubs library.



Not trying to discourage you, but I think it's a really hard problem to solve ;-)



Perhaps, starting out the whole thing, writing a stub-wrapper for FFI to be 'inport'ed by newLISP via a module written in newLISP, is a better start:



newlisp-program -> newlisp-module -> C-ffi-wrapper -> libffi.dll/so/dylib -> 3rd-party-lib



Then, after that works, trying to get rid of C-ffi-wrapper. Another possibility would be to modify libffi.dll/so/dylib to our own:



newlisp-program -> newlisp-module -> modified-for-newlisp-libffi.dll/so/dylib -> 3rd-party-lib



finally putting a stripped down modified-for-newlisp-libffi.dll/so/dylib into newLISP:



newlisp-program -> newlisp-module -> 3rd-party-lib



I guess the last one is, what you are shooting for.

[sunmountain]

newlisp-program -> newlisp-module -> 3rd-party-lib



I guess the last one is, what you are shooting for.

Exactly.



About my motivation:

My goal ist to make it easier working with more complex structures.

Mostly these cause more work, due to pack/unpack.



At first I thought it could be done solely with import/pack/unpack and some

define-macro magic in combination with FOOP (create context objects with default functors behaving the right way).

But then I discoverd the restricted number of callbacks, and as I do GUI development quite often,

that would be a real constrain (which could even be solved otherwise, sure.).

I thought it would be cool to have a generic way of creating something the ffi library calls

closures, so having one trampoline function which does name based dispatching and transforming

the parameters and return values transparently to the newLisp programmer.

And: I want to learn :-)

And yes, it's hard work at a first glance - but the newLisp code makes it easy - at least in theory.



Anyway, could you please explain if there is a standard pattern on using the CELL members

aux and contents ?



If I'm creating CELL_FFI_IMPORT type, I could for example do a dispatching over aux

and store pointers to concrete objects in contents, so I would only need one CELL type.



The question is, if that would break anything.

[Lutz]

In here:



http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/



everything is prepared for an extended ffi. Turns out, a new CELL type is not necessary. A 'fcall' is already in primes.h and prototyp for p_fcall in protos.h.



The ->aux field of the import cell which contains the name of the imported function, from now on contains a structure typedef FFIMPORT defined in newlisp.h.



Expand this structure to your liking. If the typefield is 0, newLISP will use the simple interface, else it will call the new interface. All other values are free for you to use. The structure gets allocated during 'import'.



At the end of nl-import.c, I added stubs for:



CELL * executeLibFFI(CELL * pcell, CELL * params)



and



CELL * p_fcall(CELL * params)



Basically everything you do, can be done with those two functions. pcell->aux contains the new structure and pcell->contents contains the imported function address as before. At the moment these functions just return nil. executeLibFFI() is called from executeLibfunction().



For the return values of fcall() and executeLibFFI() look into the stuff......() functions in newlisp.c. They generate new newLISP cells for all the specific types.



The way things are setup, no bad side effects to other portions of newLISP can happen. If you want, you can pull out executeLibFFI() and p_fcall() in extra file.



development newlisp-10.3.6 will be released later today. But you can start with the "inprogess" version, if you want.

[sunmountain]

The way things are setup, no bad side effects to other portions of newLISP can happen. If you want, you can pull out executeLibFFI() and p_fcall() in extra file.



development newlisp-10.3.6 will be released later today. But you can start with the "inprogess" version, if you want.

Somebodys done my homework :-)



Thanks alot.