extended import and callback API

[Lutz]

The import documentation in the reference manual has been rewritten with a second chapter for the extended import FFI API. Any help in improving language and punctuation is  appreciated. Simply copy paste corrected paragraphs in this thread and I will integrate them.



http://www.newlisp.org/downloads/development/inprogress/import.html">http://www.newlisp.org/downloads/develo ... mport.html">http://www.newlisp.org/downloads/development/inprogress/import.html



Special thanks again to sunmountain for implementing the extended import syntax.



The new extended syntax will work automatically for both the 32-bit and 64-bit versions of newLISP automatically and allows feeding and returning most of C's basic data types. The new API is more practical when porting code from other scripting languages. The libffi is part of standard Mac OS X and Windows 7 and XP and UBUNTU linux installs. On other UNIX libffi is easy to make and add.



The old API, which is used in existing standard modules, is preferred in embedded applications when trying to install a minimum amount of libraries and when speed performance is critical.



sunmountain is currently working on an extended FFI callback API. As soon as this is integrated a new development version 10.3.8 will be released.



ps: CodePatterns.html will be updated later

[xytroxon]

This is a great addition to newLISP!!!



Some quick thoughts:

Code Select Expand


; An idea for dlls with multiple functions. To allow for a list
; in the second argument to define a dll's functions as a group.
; So to improve readability and reduce  typing errors when
; adding new functions from header source code and
; for easier testing of multiple dll versions...

; (import MY_DLL
(import MY_DLL_NEW
'(
;; void func1(int var1);
("func1" "void" "int")
;; int func2(char var2);
("func2" "int" "char")
; etc...
)
)

; That caused a second idea, it would look better to
;  retain the original C header code format i.e.
;        return variable / function name / variable(s).
; The purpose: Tto reduce errors transposing C function
; headers from copy and paste editing

(import MY_DLL
'(
;; void func1(int var1);
("void" "func1" "int")
;; int func2(char var2, int *var3);
("int" "func2" "char" "int*")
; etc...
)
)


-- xytroxon

[xytroxon]

Here is a better way, that doesn't need to mess with import...

Code Select Expand


(define (import-ffi libname funclist)
   (dolist (func funclist)
      (eval (push import (push libname func)))
   )
)

(import-ffi "msvcrt.dll"
   '(
      ("atof" "double" "char*")
      ("atoi" "int" "char*")
      ; etc...
   )
)


-- xytroxon

[Lutz]

I can put that into CodePatterns.html.

[sunmountain]

Hi,

Yes I'm done with implementing the closure callback extention.

Now you can write code like this:

Code Select Expand


(import "msvcrt" "qsort" "void" "void*" "int" "int" "void*")
(import "msvcrt" "atexit" "void" "void*")
(define (bye) (println "See you ..."))
(atexit (callback 'bye "void" "void"))
(set 'l '())
(dotimes (i 30)
    (set 'l (append l (list (int (mul 30 (random)))))))
(set 'base (pack (dup "ld " 30) l))
(println "Unsorted:")
(println (unpack (dup "ld " 30) base))
(define (cmp a b)
    (- (get-int a) (get-int b)))
(qsort (address base) 30 4 (callback 'cmp "int" "void*" "void*"))
(println "Sorted by qsort:")
(println (unpack (dup "ld " 30) base))
(exit)


which gives (your rng may create alternate numbers, though):

Code Select Expand


Unsorted:
(0 16 5 24 17 14 10 26 24 22 5 25 21 15 9 0 2 10 4 4 29 13 3 0 0 11 15 17 18 18)
Sorted by qsort:
(0 0 0 0 2 3 4 4 5 5 9 10 10 11 13 14 15 15 16 17 17 18 18 21 22 24 24 25 26 29)
See you ...


And you can (for instance) interact in more "natural" way with ui libraries.

For example I use http://www.tecgraf.puc-rio.br/iup/">IUP very often.

To use it, I place a iup.dll in the same directory as newlisp.exe and then do this:

Code Select Expand


(import "iup.dll" "IupOpen" "void" "void")
(import "iup.dll" "IupClose" "void" "void")
(import "iup.dll" "IupButton" "int" "char*" "int")
(import "iup.dll" "IupSetAttribute" "int" "int" "char*" "char*")
(import "iup.dll" "IupSetAttributeHandle" "int" "int" "char*" "void*")
(import "iup.dll" "IupSetCallback" "int" "int" "char*" "void*")
(import "iup.dll" "IupShowXY" "int" "int" "int" "int")
(import "iup.dll" "IupDialog" "int" "int")
(import "iup.dll" "IupMainLoop" "int" "void")
(import "iup.dll" "IupLabel" "int" "char*")
(import "iup.dll" "IupVbox" "int" "int" "int" "int" "int")
(import "msvcrt.dll" "atexit" "void" "void*")

(IupOpen)
(set 'quit_bt (IupButton "Quit" 0))
(set 'quit_bt2 (IupButton "More Quit" 0))

(define (quit_cb handle button pressed x y status)
(begin
    (println "handle id " handle)
    (if (= handle quit_bt) (println "quit_bt pressed "))
    (if (= handle quit_bt2) (println "quit_bt2 pressed "))
    (println "pressed? " (if (= 1 pressed) "Yes" "No"))
    (println "x " x)
    (println "y " y)
    (println "status " (get-string status))
    -2) ; IUP_DEFAULT = -2
)

(define (end) (println "Quit !"))

(IupSetCallback quit_bt "BUTTON_CB" (callback 'quit_cb "int" "int" "int" "int" "int" "int" "char*"))
(IupSetCallback quit_bt2 "BUTTON_CB" (callback 'quit_cb "int" "int" "int" "int" "int" "int" "char*"))
(set 'label (IupLabel "Very long label"))
(IupSetAttribute label "EXPAND" "YES")
(IupSetAttribute label "ALIGNMENT" "ACENTER")
(set 'vbox (IupVbox label quit_bt quit_bt2 0))
(IupSetAttribute vbox "MARGIN" "10x10")
(IupSetAttribute vbox "GAP" "5")
(IupSetAttribute vbox "ALIGNMENT" "ACENTER")
(set 'dialog (IupDialog vbox) )
(IupSetAttribute dialog "EXPAND" "YES")
(IupSetAttribute dialog "SIZE" "QUARTER")
(IupSetAttributeHandle dialog "DEFAULTESC" quit_bt)
(IupShowXY dialog 100 100)
(IupSetAttribute dialog "TITLE" "IUP from newLisp 10.3.8_DEVEL")
(atexit (callback 'end "void" "void"))
(IupMainLoop)
(IupClose)
(exit)


If run and clicking on the first or second button gives something like this:

Code Select Expand


$ newlisp.exe iup_button.lsp
handle id 23449624
quit_bt pressed
pressed? Yes
x 22
y 9
status   1
handle id 23449624
quit_bt pressed
pressed? No
x 22
y 9
status   1
handle id 23449624
quit_bt pressed
pressed? Yes
x 22
y 9
status     3
handle id 23449624
quit_bt pressed
pressed? No
x 22
y 9
status     3
handle id 23451248
quit_bt2 pressed
pressed? Yes
x 24
y 17
status   1
handle id 23451248
quit_bt2 pressed
pressed? No
x 24
y 17
status   1
handle id 23451248
quit_bt2 pressed
pressed? Yes
x 24
y 17
status     3
handle id 23451248
quit_bt2 pressed
pressed? No
x 24
y 17
status     3
Quit !


As a sideeffect of this extension the number of callbacks is virtually unlimited.



What is missing is support for varargs (which is really hard to implement in ffi context,

and support for structs/unions, which is not so hard to implement.



By now the extension adds ca. 40 kB (Windows XP SP2, gcc 4.6.1 (mingw)).



I'm looking forward to see this in 10.3.8 :-)



And again: a big thank you to lutz for this opportunity.

[TedWalther]

Lutz, have you considered joining the Google Summer of Code?

[Lutz]

To be a Google Summer Code mentor is a lot of work and also requires a bigger project or a project in earlier stages with more people involved at the beginning.

[Lutz]

After some struggeling on Mac OS X, the extended ffi with unlimited callbacks is now working on Mac OS X, UBUNTU Linux and Win32.



There is still some minor cleanup for memory management when functions get imported repeatedly in the same newLISP session and some refactoring is in process too. All that should be done soon for an official development release 10.3.8 during this week.



On Mac OSX 10.3.8 with libffi only adds about 10k compared to 10.3.3. On UBUNTU Linux the difference is only about 5k.

[Lutz]

After some struggling on Mac OS X, the extended ffi with unlimited callbacks is now working on Mac OS X, UBUNTU Linux and Win32.



There is still some minor cleanup for memory management when functions get imported repeatedly in the same newLISP session and some refactoring is in process too. All that should be done soon for an official development release 10.3.8 during this week.



On Mac OSX 10.3.8 with libffi only adds about 10k compared to 10.3.3. On UBUNTU Linux the difference is only about 5k.

[sunmountain]

Hi lutz,

I made a small fix, sent you the code.



Under Windows (mingw, gcc 4.6.1, Win XP Pro SP 2 32 But) it adds

280590-274958 = 5632 Bytes, so the addition is neglectable I think.



But to be fair: it is compute intensive, as it creates call information on the fly using assembler

ninja technics.



Now I can put python to rest :-)

At least, as soon I implemented proper struct handling.

[Lutz]

The difference of 10.3.8 with extended FFI versus only simple FFI on Mac OSX Lion is:



   285248 - 279976 = 5272 bytes



On Linux Ubuntu 11.04:



   267072 - 262848 = 4224 bytes



That is less than 2% on the three platforms Windows, OS X and Linux.



For that < 2% we get the ability to import virtually any library.



Hopefully this will inspire more newLISP users to write many extensions ;-)