Compression

[newdep]

Hi Lutz,



Its on my wishlist for some time now but how do you think about a small and

simple compression/decompression implementation in newlisp? (like a small lha implementation) someehing like: (compress str) and (decompress str).



Regards, Norman.

[HPW]

There was a thread about importing gzip:



http://www.alh.net/newlisp/phpbb/viewtopic.php?t=664&highlight=zlib">http://www.alh.net/newlisp/phpbb/viewto ... light=zlib">http://www.alh.net/newlisp/phpbb/viewtopic.php?t=664&highlight=zlib



And for windows (hpwNLUtility):



http://www.alh.net/newlisp/phpbb/viewtopic.php?t=149&highlight=zlib">http://www.alh.net/newlisp/phpbb/viewto ... light=zlib">http://www.alh.net/newlisp/phpbb/viewtopic.php?t=149&highlight=zlib

[newdep]

Your right about the thread ;-) Though I never came to the Zlib because

it simple too big for a simple compression. I also had a look at miniLha lib

but that one still is 16 Kb..

(I dont care what compression as long as it compresses :-)



How about a simple LZW imlant?



Though Im hoping of a C implemented fast simple compression routine

inside newlisp. the CRC was also done by Lutz and its small and fast.



Perhpas Lutz can do some X-mass features ;-)



Regards, Norman

[pjot]

Do you want compression for text files? Or also binary files?



If only textfiles must be compressed, you could write a compression function yourself in newLisp. This should be fast enough, even for large ASCII files.



Peter

[newdep]

compression is compression so that includes it all ;-)

No bitstuffing or mapping on ascii layers, but a real compression algorithm..



But I wont spoil the fun, if you have a newlisp version for ascii drop it in ;-)



Regards, Norman.

[newdep]

So its finaly Done, we have compression, done by Pjot!



You can download it from his site at -> http://www.turtle.dds.nl/newlisp/lzw.lsp">http://www.turtle.dds.nl/newlisp/lzw.lsp



PS: its not the worldrecord on speed, thought it is compession! ;-)

[pjot]

Speed is a problem, that's true. But there may be some improvements on the code which also may improve speed.



During this week I'll finalize the program. And let you know (of course).



Peter



(PS you can also buy a faster computer.... ;-) )

[pjot]

The LZW compression context has been optimized and is ready for consumption. Compared to the first implementation, the compression speed has improved with 25%, while the decompression speed has improved dramatically with about 80%.



LZW is a technique which normally is used to compress GIF and TIFF files. It is very suitable to compress text. Files up to 500k compress reasonably fast with this context.



http://www.turtle.dds.nl/newlisp/lzw.lsp">http://www.turtle.dds.nl/newlisp/lzw.lsp



If somebody has suggestions to increase the compression speed even more, please let me know. :-)



Enjoy,



Peter

[pjot]

The LZW compression has been optimized even more. The average compression speed compared to the former situation is 5x faster.



On my P4 2Ghz an e-text of 500kB can be compressed in 4 seconds now. If your machine is faster then it will take even less time.



The latest version is here:



http://www.turtle.dds.nl/newlisp/lzw.lsp">http://www.turtle.dds.nl/newlisp/lzw.lsp





To test, just copy some file into the same directory as 'LZW.lsp', with the name "dump.txt", and run 'LZW.lsp'. The compression also works with binary files.



Regards

Peter

[newdep]

On an AMD 64 3200+



filesize: 4614809  (pure text, dictionary file)

Starting compression, please wait... Time spent (msecs): 17020

Starting decompression, please wait... Time spent (msecs): 30283



1/4 of 4614809 (1153703 bytes or the same file) ->

Starting compression, please wait... Time spent (msecs): 4282

Starting decompression, please wait... Time spent (msecs): 7589



They both come pretty close to a 4:1 ...



Norman.

[pjot]

Self-extracting possibilities were added.



http://www.turtle.dds.nl/newlisp/lzw.lsp">http://www.turtle.dds.nl/newlisp/lzw.lsp



Just call

(LZW:lzx "myfile.ext")

 

and it will create a self-extracting ".lzx" file. If this ".lzx" file is executed by newlisp, it will be decompressed. In Unix/Linux it is also possible to run it directly, like "./myfile.ext.lzx". In Windows you need to execute "newlisp myfile.ext.lzx" to decompress.



Peter

[tom]

this needs to go in a backup/restore script.  I've been using back2cd.tcsh, a script that a guy in my lug wrote, but I think that newlisp could do at least as well, especially with lzw.lsp around...

[tom]

hey peter, stupid question:



how do you give it the file to compress?  



$./lzw.lsp



gives your error,



ERROR: File not found! Exiting...



I did say up front this was a stupid question :-)



thanks.

[pjot]

Hi Tom,



In the MAIN context of "LZW.lsp" the demonstration is looking for a file called "dump.txt". Just create a file with this name in the same directory and run. Currently the LZW creates a self extracting . lzx file, for which you need newLisp to decompress.



Peter