load lisp-files from inside wrapped EXE

[HPW]

Hello Lutz,



Is it possible to access lisp-files packed into the virtual file system

inside the packed EXE?



I know your tip with link.lsp to attach a Lisp-file, but I want to pack a large

number of files (when possible) with a directory-structure inside it and read from there?

[Lutz]

Yes, you would use 'load', image the following directory structure



c:bwidget*.tcl

c:myextension*.tcl

c:freewrap

c:freewrapnewlisp-tk.tcl

c:freewrapmystuff.lsp

c:freewrapimages*.gif

c:freewrapbwidget.txt

c:freewrapmyextension.txt

c:freewrapmylisptstuff.txt



content of file 'mylispstuff.txt':

>>>>

/freewrap/mystuff.lsp

>>>>



then wrap:



freewrap newlisp-tk.tcl -f images.txt -f bwidget.txt -f myextension.txt -f mylispstuff.txt



in the config file 'newlisp-tk.config' put the following option:



set Ide(initCommand)  "wm withdraw . ; NewlispEvaluate {(load "/freewrap/mystuff.lsp")}"



the first statement before the semicolon 'wm withdraw .' will supress the tk console windows coming up, so you will only see window stuff you are bringing up yourself. The command after the semicolon: NewlispEvaluate {(silent (load "/freewrap/mystuff.lsp"))}  is a tk procedure telling newLISP to load your file.



See also the manual section: Delivering Apllications with newLISP-tk



This mixed newLISP/TclTk stuff is a bit tricky with all those 100,000 different string delimeters {,", which also have different meaning in newLISP and TclTk :=), but that's what you gotta do ;)



Instead of doing the "set set Ide(initCommand) " stuff you also could put at the very end of the newlisp-tk.tcl file:



 NewlispEvaluate {(silent (load "/freewrap/mystuff.lsp"))}



That would be a little bit less cryptic, but still leave this in you newlisp-tk.config file:



set Ide(initCommand) "wm withdraw ."



(don't forget the dot)



let us know when it works



Lutz

[HPW]

Hello Lutz,



Thanks for the informations!



I think it does not what I want. So look at my try:



Content of /freewrap/lisp/mylisp.lsp: (Also wrapped inside a new TK-EXE)



(setq hpwtest "9999")





newLISP v7.2.2 Copyright (c) 2003 Lutz Mueller. All rights reserved.

(load "/freewrap/lisp/mylisp.lsp")

true

> hpwtest

"9999"

> (setq hpwtest nil)

nil

> hpwtest

nil

>(tk "NewlispEvaluate {(load "/freewrap/lisp/mylisp.lsp")}")

""

> true

> hpwtest

"9999"

> (setq hpwtest nil)

nil

>



Now renamimng mylisp.lsp to something other



> (tk "NewlispEvaluate {(load "/freewrap/lisp/mylisp.lsp")}")

""

> nil

> hpwtest

nil





So when I see it correct, the call on the TCL-function does it right, but did not look inside the wrapped EXE. So how to tell the LOAD-command where to search? Or must there be a new TCL-command for extracting the wanted file to temp and newlisp load it from there?

[HPW]

Also when I try the first hint:



set Ide(initCommand) "wm withdraw . ; NewlispEvaluate {(load "/freewrap/lisp/mylisp.lsp")}"



I get this:



Error sourcing /Freewrap/newlisp-tk.tcl: can not find channel named "0"

[Lutz]

I should have tried it first myself ;-), sorry about it, but I found the solution now and tried it out.



What went wrong:



It didn't work because newLISP, which lives outside the wrapped application, of course doesn't know about the freewrap virtual filesystem.



Also the set 'Ide(initCommand)' may not contain any newLISP code because when 'Ide(initCommand)' is executed from inside newlisp-tk.tcl, the communications channels to newLISP are not established yet, this is why you get a : can not find channel named "0".



This works:



I tried it out! after having wrapped tcltk-app.lsp, I tried the following in the newLISP-tk console window.



(tk "set fle [open /freewrap/tcltk-app.lsp r]; NewlispEvaluateBuffer [read $fle] 0")



TclTk opens the file than reads and evaluates it with NewlispEvaluateBuffer', which is a function internally used by newlisp-tk.tcl to send the edit-window to newLISP for evaluation when you click the evaluation button in the edit browser.



The last parameter set to 0 in the example tells newLISP to do a 'silent' evaluation, without seeing the evaluation result in the console.



After the first experiment went well, I pasted the following line as the last line into newlisp-tk.tcl:



set fle [open /freewrap/tcltk-app.lsp r]; NewlispEvaluateBuffer [read $fle] 0



Then wrapped again. As expected the tcltk-app.lsp (example from the windows distribution) went up without a problem. So this seems the way to go.



Lutz

[HPW]

Thanks Lutz,



now it works as expected and real smart with the TK-Buffer.

Powerfull function with a lot possibilitys!

[HPW]

Start thinking with the new possibilitys:



For example file checking:



(tk "file exists /freewrap/lisp/mylisp.lsp")



returns "1" when file is inside wrapped EXE "0" if not.



2 powerfull programming languages in one application combined

with a simple, but effektiv interface is really smart.

[HPW]

There comes a question up:



Is the complete wrapped EXE loaded into memory?

When I wrapp a lot of files inside the EXE will the startup slow down?

I think this is worth some tests.



Starting an empty newlisp set up 2 process with 5.75 MB and 1.85 MB.

So lets wrap!

[HPW]

I did a heavy wrapping test with a complete database with 16600 Lisp-Files from a 5.7 MB ZIP.

Unzipped 11.8 MB on Disk (69.4 MB on NTFS because cluster size)



Now my newlisp-tk.exe is 9.85 MB.



>Is the complete wrapped EXE loaded into memory?



Yes it seems so, memory load is about the difference higher.



>When I wrapp a lot of files inside the EXE will the startup slow down?



Only a little bit in the range about 1-2 sec.



Starting the empty newlisp set up 2 process with 9.3 MB and 2.0 MB.

Compression-ratio seems a little better then standard WINZIP.



Now remains the question if lisp-loading from the preloaded EXE (RAM) is faster than from disk-files.



Now it can be optimised to put absolutly needed file in the EXE and the other content dynamicly loaded from an ZVFS-Zip.

[Lutz]

sounds very impressive, I wonder what kind of application you are developing?



Lutz

[HPW]

> .. I wonder what kind of application you are developing?



It's not a new application, it will be a port of an autolisp-app running under autocad. Ths lisp-files comes from a oracle-database with a complete ERP-dataset for calculating sales orders. It is used in an autocad-oem app and is running for some years now. It is read only and running interpreted with dynamic caching. So even under interpreted autolisp it's performance is OK. It use dynamicly lisp-generated interfaces like TK. So now I want a self-contained standalone-app without the pricy oem-license with the identical logic (without 3D-graphics of cource). Multiplatform is also wellcomed (Therefor lisp-tables, no platform dependent database). A second step would be a server-based solution with the same logic.



It seems so far that it could be done, when I solve my problem with formula's from the ERP-system. Therefor I asked for the infix parser, to be able to process them under newlisp. Everything else should be possible under newlisp so far.

[Lutz]

Its good to hear how newLISP can solve real world problems. Many people think LISP is just for academics.



I just finished an infix/postfix/prefix parser in newLISP and have published it on http://newlisp.org/news">http://newlisp.org/news . The algorithm is sound and I have used variations on it it many times, coded in 'C' or in Java. This is the first time I have done it in newLISP, so give it a little bit more testing. Read the news post and the header of the file for more info.



The file is also in the development directory and will be included in the next newLISP release.



Lutz

[HPW]

Hello Lutz,



Thank you very much for your fast and great work.

And to see how incredible small it is, is still unbelievebel.



>Its good to hear how newLISP can solve real world problems.



After the great developments in last month, I now believe that it able to do solve such problems. It only has to show the final overall performance and the robustness and simplicity of TK-user-interfaces. This is the remaining matter to test.



Now to the infix-parser:



What I test so far does it right. The only thing remains for me, to replace all operators with surounding spaces. Also I had to replace all "," decimal delimiter to decimal-points, because possible formula-string are like this "(foo+10,3)/2)". But this should be no problem.



Again, thanks a lot.



Back to testing. :-)

[HPW]

With the new 7.3.4 I have no problem to load the huge Lisp-file from a file directly.

It is really fast. But when I try it from the wrapped EXE or file-system through the TK-frontend it seems to hang.

When I use a far more smaller file, it comes back at once.

Code Select Expand


(tk "set fle [open /DBA/artbez.lsp r]; NewlispEvaluateBuffer [read $fle] 0")


When I only use the:

Code Select Expand


(tk "set fle [open /DBA/artbez.lsp r]")

I get back a filehandle at once.



Can there be a problem with a huge file-buffers and NewlispEvaluateBuffer?

[HPW]

So far a workaround:

Code Select Expand


(set 'tmpfile(tk "::freewrap::unpack  /DBA/artbez.lsp C:/temp/"))
(load tmpfile)
(delete-file tmpfile)


This is fast enough.