Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Topics - lithper

Pages1
#1
newLISP newS / nL 9.3.5-development can crash your machine
March 30, 2008, 11:25:41 PM
[1] accidentally came across a sequence that immediately, explosively eats up all memory and CPU and creates effectively a DoS condition, killing the OS.


Code Select

> (symbols Aa)
(Aa:Aa)
> (Aa p (d))
()
> (Aa "p" (d))

invalid function : (d)

//CRASH!!//
Terminated
 ->

In fact, this incorrect attempt at putting something as value in hash will murder your machine in 3-5 seconds:
Code Select

0  0 164184 225932   6932  49968    0    0     0     0 1014   162  0  1 99  0
 0  0 164184 225932   6932  49968    0    0     0     0 1011   159  0  0 100  0
 1  0 164184 205068   6932  49968    0    0     0     0 1010   162  6 12 82  0
 1  0 164184  80972   6932  49968    0    0     0     0 1002   125 35 65  0  0
 1  0 122820   3568   6932  49968    0    0     0     0 1003   203 34 66  0  0
 2  1 187264   2392    968   6936    0 70800     8 70800 1294   454 24 76  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 4  4 236640   3372    980   6804   20 49376   108 49376 1298   528  5 50  0 45
 2  0 250240   2472    980   6804  220 13608   220 13608 1112   629  9 34  0 57
 0  1 267212   2384    980   6244    0 17160     0 17160 1029   283  7 43  0 50
 0  3 267784   1560    996   6304    0  896    72   896 1035   242  5 10  0 85
 0  3 267780   1292   1068   7700   96    0  1560     0 1128   293  0  3  0 97


CPU usage will jump to almost 100%, all memory will be eaten up, the machine will thrash the disk (swapping memory out), it will freeze



[2] A similar crash OR segmentation fault happens if after an incorrect operation a bracket is omitted (usually  a benign error condition)
Code Select

-> newlisp
newLISP v.9.3.5 on Linux IPv4 UTF-8, execute 'newlisp -h' for more info.

> (define Aa:Aa)
nil
> (Aa "f" (list p))
(nil)
> (symbols Aa

missing parenthesis : "...(symbols Aa                         "
Terminated
//CRASH!!//


Sometimes in these cases newlisp generated simple segmentation faults.



I do realize the shown usage is incorrect - only if your script generates it in some unexpected case, or if your program can be manipulated into doing something similar, a very serious condition can happen.



[3] It seems that new usage -- (define Aa:Aa), then (Aa "f" "asdf") or (Aa "f" 345) -- is narrower than the one before?

It seemed that with older usage I could avoid restriction on hash key as STRING ONLY, and on data as only STRING or NUMBER ?? Or am  I mistaken and this limitation existed all the time ? (seems, not, as one of my older scripts refused to work with new notation without a typecast).
#2
newLISP newS / a newbie question
March 22, 2008, 03:09:18 PM
Reference for functions like those from the "ref" family, "find", "match" etc. - operators for work with lists - supposedly can take (list) as search arguments inside a bigger list (L):
Code Select
(ref (list) L)
It works OK as examples from the docs show if (list) is a known set of symbols:
Code Select
(set 'L '(a b c d 1 2 3))
(ref 'd L)

However, if and when you calculate your search list , this invocation becomes impossible:
Code Select
(list? (3 1 L))
         :- true

(ref (3 1 L) L)
        :- read error message from newLisp

The only variant I stumbled upon that works is converting the calculated list into a string, and then clumsily stripping its (" -- and "), if the term contains one element. Searching for a string that matches a symbol in a given list then works.



Where is the answer? It may be a very simple, newbie question.



I.e. the problem as far as i can guess is in the treatment of brackets and quotes by nL.

If our list includes (d 1) as a _sublist_ - (a b c (d 1) 2 3 , the match for '(d 1) will happen, and so examples from documentation work.

But i see no way to match for a subsequence in a flat list, if the sequence is produced by a previous calculation as a list and so is bracketed.



For example - I calculate something that produces "list_q" which is ("1" "0")

I want to find where in list L that "1" is included:

(ref  ( 0 1 list_q)  L)

     :- will fail

because NL wants to treat brackets and quotes as literal part of the symbol in searching the list L.
#3
Anything else we might add? / Fast web serving and mod_lisp mini-HOWTO
March 12, 2008, 10:31:36 AM
1. Introduction

In serving web pages. nL can (a) work as a standalone server for smaller applications (b) can be spawned in scores of copies because it's so light and/or (c) be dropped into a cgi-bin directory.

So as it is it's already more versatile than many other languages.



newLisp, however, is unlike the "big" ones in the sense that there is no immediate way to embed it into a web server, in the manner of mod_perl or the php apache module.

This is desirable because it can considerably decrease load on the server and response times - the application will remain persistent rather than restart each time.



There are several ways to solve the problem of serving newLisp scripts from Apache, i.e. a serious server capable of many functions, faster than a regualr CGI can do.



One known alternative to embedding a language into Apache (the way mod_perl and php do) is that used by FastCGI:

(a) create an extension=module for Apache that opens a socket (network or unix socket) to an external process

(b) which will listen for Apache connections, and serve replies

(c) The application looks like a regular CGI script,, but after initialization etc. it starts a loop that reads from the socket, analyzes the CGI environment resent to it by Apache, and replies with headers and content.





With FastCGI there are two ways newLisp could talk to Apache:

(a) using imports from a fastcgi library in C supplied in the fastCGI distribution

(b) or reimplementine in script (a) function(s) that unpacks fastCGI packets and speaks the fastCGI protocol.



FastCGI packets are binary, and this is would be a more complex way to use it, although the distribution includes a perl module that can be compiled (a special options) to produce a script in pure perl - it could be used as a crib.



A simpler alternative to FastCGI, which uses the same principle is mod_lisp, whith a primitive protocol in ASCII. Scripting to this module is very easy.

Contradicting its name, mod_lisp is totally generic, and can be used from any program in any language that is capable of networking.



2. Getting and compiling mod_lisp



2.1 The URLs

The tiny mod_lisp.c can be downloaded directly from

http://www.fractalconcept.com:8000/public/">http://www.fractalconcept.com:8000/public/

open-source/mod_lisp/

Pages describing the module and its protocol are linked from here (bottom of the page):

http://www.fractalconcept.com">http://www.fractalconcept.com

(download links look broken).

The modules in the mod_lisp-current.tgz was last updated some time in either 2004 or 2006, it seems.



Secondly, the pages link to sample scripting in clisp, cmucl and LispWorks that show how it can serve primitive dynamic web pages.



One more useful example and howto/tutorial is in a link to a web site that went into an Internet black hole, but can be retrieved from the Internet history archive:

search for

http://lisp.t2100cdt.kippona.net/lispy/home">http://lisp.t2100cdt.kippona.net/lispy/home

on the Wayback machine site:

http://www.archive.org/web/web.php">http://www.archive.org/web/web.php



This tutorial describes a setup with MySQL backend (which in 2003 the author's 350 MHz machime pushed at 6 hits/sec) or for a simple dynamic page (36 hits/second)





2.2 Compilation

The downloaded distribution includes file mod_lisp2.c - this version is for Apache 2.0.x

Another file mod_lisp.c is for Apache 1.3.x



Apache Foundation currently maintains (with latest patches including security patches) 3 series of its server:

1.3.x, 2.0.x and the latest 2.2.x



Apache software relies on its APR library, which between versions 2.0 and 2.2 jumped from 0.9.x to 1.x, with changes in its API. Therefore modules written for Apache 2.0.x will most probably require some porting.



2.2.1 compilation for Apache 2.0.x

mod_lisp2.c was created for 2.0.x and compiles cleanly using the usual (an apache with enabled module support is presumed)
Code Select
cd /dir/where/mod_lisp_source/is
apxs -c mod_lisp.c
cd ./.libs
cp mod_lisp.so /to/apachedir/modules

2.2.2 compilation for Apache 2.2.x

Compilation for Apache 2.2.x requires some porting. I did it comparing APR library header files between pre-1.0 and post-1.0 versions, and by using the Changes doc from the distribution.

After renaming 3-4 functions and excluding the APR_STATUS IS_SUCCESS(s) macro the module becomes usable on Apache 2.2.x

Briefly:
Code Select

1. in mod_lisp.c (line 327):
(apr_socket_create ((&socket), AF_INET, SOCK_STREAM, socket_pool));

Now a new argument "APR_PROTO_TCP" needs to be added:
(apr_socket_create ((&socket), AF_INET, SOCK_STREAM, APR_PROTO_TCP, socket_pool));

2, Depreciated function was used in the module which is no longer valid: apr_send; needs to be found in the text and changed to apr_socket_send;
APR authors mentioned in their Changes doc compatibility of such change.
See header in APR sources; as defined in $apacheindcludedir/apr_network_io.h :
APR_DECLARE(apr_status_t) apr_socket_send(apr_socket_t *sock, const char *buf, apr_size_t *len);
/** @deprecated @see apr_socket_send */
APR_DECLARE(apr_status_t) apr_send(apr_socket_t *sock, const char *buf, apr_size_t *len);

3. same substitution with apr_connect --> apr_socket_connect
4. Same with apr_recv --> apr_socket_recv

5. (the macro APR_SATUS_IS_SUCCESS was obsoleted): Changed:
APR_STATUS_IS_SUCCESS(s)
to
((s) == APR_SUCCESS))


After these several changes the module seems to compile (same procedure as for 2.0.x) and run on 2.2.x (I checked the latest 2.2.8)







3. Configuring and using mod_lisp



3.1 The minimal Apache 2.x.x configuration

consists of:



(a) adding to httpd.conf line (adjust if your modules are in a different dir):
Code Select
LoadModule lisp_module modules/mod_lisp.so

(b) and adding (minimally)


Code Select
LispServer 127.0.0.1 3000 "somenameforit"
# stupid forum sfw eats up slash-dirname; it must be
# Location slash-lisp  here:
<Location>
SetHandler lisp-handler
</Location>


This means that your apache will knock at port 3000 of your local machine to speak with your newLisp server application every time a user browses into http://www.your.server.name.com/lisp/xxx.html">www.your.server.name.com/lisp/xxx.html directory



The newLisp hadndling server can be installed on a different machine in the backend network, I believe, so creating a flexible architecture. In contrast to FastCGI this module does not seem to be able to communicate through local Unix sockets.



3.2 Scripting on newLisp side



The module sends the regular CGI header data one item on a line, splitting it into "key-value" pairs, i.e. in exchanges it should look like
Code Select
"Content-Typen"
"text/html; charset=utf-8n"

Note that "text/html; charset=..." are on the same line, however

Header information must end with "endn"



You can see what Apache sends to you by starting newLisp to listen on a configured port as a server with "-L filename" option, which logs this information.

It's a dangerous exercise, but because request headers and vars are not lisp commands, you'll see the information simply logged but interspersed with newLisp "nnil" responses or some echoes:
Code Select
server-protocol
           ......this new line
nil        ......and this nil come from newLisp response  
HTTP/1.1

nil
method

nil
GET

nil
url

nil
/lisp/index.html

nil

and so on. This is for a first look only. When your script is working, you'll inspect it from there.





So the minimal newLisp script to talk to mod_lisp will look like this:


Code Select
#!/usr/bin/newlisp

;.....initialize, do things that need to be done before serving; then

#; ------server simple---------
(define (server_simple str_content_to_send)

                ; maximum bytes to receive
        (constant 'max-bytes 1024)
        (set 'cnt 0)

        (if (not (set 'listen (net-listen 3000)))
                (print "ERR opening socket: " (net-error)))
        (set 'c_connection (net-accept listen))

        (while true  ; -- forever;  handle better in a real script
                (if (net-error)
                        (set 'c_connection (net-accept listen)) ) ;; blocking here
                       
                ; get request info and do sth with it
                (net-receive c_connection 'message-from-client max-bytes)
               
                ; send the generated response back to apache
                (net-send c_connection str_content_to_send)
                     
                ;(silent
                ;       (inc 'cnt 1)
                ;       (print (string cnt) "-" (net-error) "n")
                ;); /silent/

        );/end of inner read-write while/
); /end of server_simple/

Uncomment the "silent" block to see counter and resets on the console from which you test the driver (more precise reasons for resets can be glimpsed with "net-select"  if needed.

This is not a realistic script - in reality you would pick up lines (arguments) from "message_from_client" (print it to see) and form a response in the main part of you application, which you will send in place of my constant "str_content_to_send".  Note how it is formed in "main".

But this is all it takes to work with mod_lisp, not much to talk about.



Note that the connection gets reset in case of errors (e.g. inability to send to an already closed socket).



And "main" will set up some test page to send:
Code Select
;-------MAIN-------------
;       sets args

; content first to calculate its length
(set 'str_second (read-file "/path/name/to/some/file/to/send.html") )
; header written according to mod_lisp protocol
(set 'str_head (append
        "Content-Typen"
        "text/html; charset=utf-8n"
        "Content-Lengthn"
        (string (length str_second))
        "n"
        "Keep-Socketn"
        "0n"
        "endn"
        ) )

; run the app
(server_simple (append str_head str_second))

The simple mod_lisp exchange protocol requires

(a) splitting header info into key-value pairs and sending them on separate line each. Output "endn" to finish sending key-value pairs.

(b) announcing Content_Length and sending after the end of the header a chunk of that size. This parameter is important for the correct work of the protocol.

(c) supplying "Keep-Socketn" "1n" if you wish to keep the connection open for a subsequent transfer, or setting it to "0" as in the example to release the connection for next requests from apache.





4. Conclusion, speed, and comparison with CGI



This setup allowed to drop load on the web server - actually, it became quite negligible with NewLisp alone sending pages from the filesystem or generating it programmatically.

Secondly, it sped up one page delivery from a persistent backend server in newLisp from roughly 50/sec as CGI for printing out one 9.5kB+(html styling pages) -- to roughly between 200 and 250/second with mod_lisp (depends on the general load level, concurrency etc) with subsecond delays.



It's interesting to note, that this mod_lisp speed is 5-7 times higher than reported with a "big lisp" setup in 2003 on a machine that is probably roughly 1.3 times slower than mine.







FastCGI (which I have not tested with newLlisp) may be preferrable because it's been implemented on majority of web servers, big and small, and because much more engineering went into it, which might result in better performance or stability or God knows what else



Results of testing FastCGI on my machine were not that great (roughly twice slower than the major mode of mod_perl for a particular perl script) and depended on the web server and used FastCGI implementation. I have not tested it enough, however.



mod_lisp is in fact not for Lisp only, but is a generic solution implementing the same idea as FastCGI with a very simple ASCII protocol, and so available for any language with networking capabilities. ASCII protocol is very  easy for scripting.



mod_lisp  currently runs on Apache 2.0.x and 2.2.x (and supposedly also on 1.3.x, which I have not tested); the speedups are about the same for the simplest one-page tests.

Correction I wrote "does not run on web servers besides Apache" -- There is a "mod_lisp.c" rewritten for the popular lighttpd web server (i.e. it implements the same protocol, the author says), and the only message about it plus source code I found dates from 2007, i.e. it's recent. I have not tried to compile or use it, however.



One interesting question remains, however

Is it possible to put into apache httpd.conf (in different subsections, possibly) not one, but a number of mod_lisp references to several servers running on different ports?

Yes, they can be tucked into different virtual hosts, at least:


Code Select
NameVirtualHost 127.0.0.1:8000

<VirtualHost>
ServerName "localhost"

LispServer  127.0.0.1 3000 "fractal"
#forum sfw eat up Location slash, lisp - the dir name
<Location>
SetHandler lisp-handler
</Location>
</VirtualHost>

<VirtualHost>
ServerName "localtwo"
DocumentRoot "/usr/local/apache2.0.63/htdocs2"

LispServer  127.0.0.1 5000 "fraction"
#forum script eat up Location slash, lisptwo - the dir name
<Location>
SetHandler lisp-handler
</Location>
</VirtualHost>

How soon mod_lisp will become a bottleneck with this setup is unclear, however.

On my machine the module simply "multiplexes" between the two streams, feeding a hundred into one newLisp script instance, then switching and feeding a hundred or so into the other one for a second or two -- while the sum of two speeds remains roughly equal to the speed of a single stream from one instance.

.
#4
Anything else we might add? / Can newLisp serve 250 typical webpages per second ?
March 10, 2008, 04:52:38 AM
1. nLisp is fast.

Once I figured out perl "cheats" during "line processing" operations (perl -ne 's/a/b/g && print') by gobbling big chunks of input into memory and started to emulate this in nL, string processing speeds became equal.



This made me think of serving web pages. nL can (a) work as a standalone server for smaller applications (b) can be spawned in scores of copies because it's so light and/or (c) be dropped into a cgi-bin directory.

So as it is it's already more versatile than other languages.



newLisp, however, is unlike the "big" ones in the sense that there is no immediate way to embed it into a web server, in the manner of mod_perl or the php apache module.





2. an interesting find - and how it works

Thinking of that, I looked around and made an interesting discovery. Some lisp people several years ago (development seems to have stopped in the first half of 2005) created "mod_lisp" for Apache 2.0.x



In fact, it's much better than a lisp-specific module.  It's totally generic, and any language with networking scripting capabilities could make use of it.


QuoteWhat the module does is make Apache, when a request comes for a URL configured to be taken care of by the module (e.g. to http://www.server.com/lisp/xxx.html">www.server.com/lisp/xxx.html), try to connect to a network server at its back (written in lisp, or in any other language) and running either locally, or across a network on a backend machine.



Knocking on a preconfigured port (say, 3000), the mod_lisp from Apache will send all typical web environment and data using a very simple ASCII protocol. The kind of data we know well from playing with CGI scripts.



Then the our Listener, the Lisp server, will decipher the request and send back the header and content, again, very much the way they do over regular CGI. The difference is that there is no need any longer to start a new process with all its overhead. One binary serving the connection will just loop and send the pages.


In fact, there is one well-known protocol that works exactly as described - FastCGI. The difference is that FCGI's protocol is not ASCII, so scripting for it would be a much bigger headache. FCGI is described in an RFC full of unbelievably bureaucratic lingo, etc. - but it's the same idea.





3. what i did as an experiment

As a result of this sudden fit of hacking enthusiasm I spend last Sunday trying to make mod_lisp work in the hope that nL won't let me down and compare well against 6 responses per second from CMUCL (or was it CLISP?) when serving from a MySQL database via mod_lisp, or the whopping 36/sec when the database was detached -- /figures corrected - lithper/.  That's what authors of the module achieved with their big Lisps according to their measurements back in 2003.

(my test box is a 500MHz from 2000, so I would not have unjust advantage, I thought).



[.a.]. mod_lisp did not compile with current apache 2.2  In series 2.2 APR (apache API's to its libraries) changed to a new version.

Previous 2.0 is still maintained (the latest release 2.0.63 fixes a number of cross-scripting vulnerabilities), as is the old 1.3 series (latest with security fixes etc is 1.3.41)



I caught several obvious things that needed updating when attempting to port mod_lisp to 2.2, but finally, although the module compiled and passed apache checks,  ver2.2.8 segfaulted with the module present. Well..



[.b.]. Then I downloaded apache 2.0.53 from 2005, figuring that the module was still alive at that time to bypass possible incompatibilities.

Now it compiled and installed itself OK



[.c.] Next I  had to spend a few hours trying to understand from a very brief description the correct syntax for the protocol, reading CL lisp source of the provided samples, and experimenting with a basic setup of nL network server.

In the end, the protocol looks very simple indeed, one just needs to know what it chokes on.



So finally for a first estimation I made nL serve a 10kB web page (pure text)  as a server sitting behind apache and talking to it through a socket -- in comparison to the same text being served via a light cgi script. (it basically checks the arguments, and then copies the file picking it from the filesystem; in both cases I just "cat" the file in response to a request)



4. The results as measured by "ab"

(apache bench, a small utility in apache distrbution). are as follows:



(CGI app) - nL is was light enough to serve at the rate of 50/second. It is OK - many commercial sites with LAMP architectures can achieve 6-10 in their best days, while under the load a user will have to wait for 2-3, even 5 seconds before a page appears.



(mod_lisp), a "fastCGI" for the nerd who assembles his helicopter from parts every time before he goes to fly (and calls it "the unix way"), however, somewhat surprised me. The page was delivered at consistent 250 times/second.  (Of course, in real life the script will nave to do more than just spew a small file out, but..)



This is better than mod_perl, i.e. perl embedded into apache does on my machine (or at least it's the same speed; my fully enabled web app is a bit heavier, and I remember under mod_perl it ran at 80-100 responses/second on the same type of output files).



5. The problem is that the setup at this point is buggy. nLisp can fall into coma, frequently, and refuse to serve. Possibly the barebone script was not correct, and I would need to set up some request queue, or maybe it's the problem of consistency across several copies of the web server, or maybe the "fastCGI" scripts must be written in a special way, as my past experience with mod_perl and such suggests.

Update: it seems apache closed connections, so that newLisp server would try to write data nowhere, when apache limits on the number of processes and clients were not sufficient.

It seems the setup works.

NewLISP script should simply reset connection when this happens (rarely if at all) and keep listening from its standard server loop-function forever.



NewLisp serves all of the stuff amazingly fast. The totals confirm the volume is correct (i.e. no 0-length pages or error pages in the output, it seems to add up).


QuoteFirst Impression

My first impression is that mod_lisp, a generic simpler clone of fastCGI, should be supported and ported to the later apache releases.

And that proper server-side script should be written for nL to work in this mode.  (upd: actually, the simplest standard newLisp server handles the communication OK)

It adds to the already wider set of ways to serve web pages than with other languages, and mod_lisp web interfaces in nL might prove to be a unique performer fit for higher capacity sites, both commercial or not.  



Alternatively, it might be worth spending some effort on writing a FastCGI protocol server to drop into nL scripts in view of the encouraging results given by the mod_lisp ascii clone. FastCGI is supported by a large number of web servers, mega- or nano-sized, in contrast to native embedding like apache embeds perl and php

(one othe approach would be to import symbols from the reference implementation of Fast CGI C library; it would break the "standaloneness" of nL script, however, if that's important)

If this message interests readers, I'll add a brief how-to and links.



Below are two sample reports from "ab".

CGI run was 15kB - against 9.5-10kB from mod_lisp, but the rate per second was pretty much constant and consistent for the two setups.

I used the same web server in both cases, so its configuration supposedly affected both results in a similar way. It had no specific caching enabled. The test is, of course, purely informal.



1. 300 requests with concurrency of 100 - i.e. 3 seconds if the server can bear such load ideally, or longer, if it backlogs under the load. This test bears the load, but delays responses about 2 seconds already:


Quote
user@host-> ./ab -n 300 -c 100 'http://localhost/cgi-bin/scriptname.lsp?target=edit_file&arch_path=local-blogs&rss_user=nick&f">http://localhost/cgi-bin/scriptname.lsp ... ser=nick&f">http://localhost/cgi-bin/scriptname.lsp?target=edit_file&arch_path=local-blogs&rss_user=nick&f

ilename=asdf'

This is ApacheBench, Version 2.0.40-dev .............



Benchmarking localhost (be patient)

Completed 100 requests

Completed 200 requests

Finished 300 requests



Server Software:        Apache/2.0.49

Server Hostname:        localhost

Server Port:            80



Document Path:          /cgi-bin/script.lsp?target=edit_file&arch_path=local-blogs&rss_user=nick&filename=asdf

Document Length:        15130 bytes



Concurrency Level:      100

Time taken for tests:   5.974378 seconds

Complete requests:      300

Failed requests:        0

Write errors:           0

Total transferred:      4585956 bytes

HTML transferred:       4539000 bytes

Requests per second:    50.21 [#/sec] (mean)

Time per request:       1991.459 [ms] (mean)

Time per request:       19.915 [ms] (mean, across all concurrent requests)

Transfer rate:          749.53 [Kbytes/sec] received



Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        0    6  10.8      0      37

Processing:    51 1687 569.4   1915    2928

Waiting:       47 1669 559.9   1912    2608

Total:         88 1694 558.9   1915    2928



Percentage of the requests served within a certain time (ms)

  50%   1915

  66%   1961

  75%   2003

  80%   2028

  90%   2079

  95%   2157

  98%   2441

  99%   2610

 100%   2928 (longest request)



user@host ->


2. same 300/100 for mod_lisp

This test bore the load easily, with 0.3-0.4 seconds per response, and that at 250 requests per second. It means the server could sustain larger loads and still provide usable delays


Quote.user@host -> ./ab -n 300 -c 100 'http://localhost:80/lisp/index.html">http://localhost:80/lisp/index.html'

This is ApacheBench, Version 2.0.40-dev ........



Benchmarking localhost (be patient)

Completed 100 requests

Completed 200 requests

Finished 300 requests



Server Software:        Apache/2.0.49

Server Hostname:        localhost

Server Port:            80



Document Path:          /lisp/index.html

Document Length:        9444 bytes



Concurrency Level:      100

Time taken for tests:   1.160991 seconds

Complete requests:      300

Failed requests:        0

Write errors:           0

Total transferred:      2893500 bytes

HTML transferred:       2833200 bytes

Requests per second:    258.40 [#/sec] (mean)

Time per request:       386.997 [ms] (mean)

Time per request:       3.870 [ms] (mean, across all concurrent requests)

Transfer rate:          2433.27 [Kbytes/sec] received



Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        0    6  12.3      0      42

Processing:     3  316 107.7    371     399

Waiting:        0  316 107.6    369     398

Total:         45  323  95.5    371     399



Percentage of the requests served within a certain time (ms)

  50%    371

  66%    384

  75%    385

  80%    385

  90%    389

  95%    389

  98%    390

  99%    392

 100%    399 (longest request)

user@host ->
#5
Anything else we might add? / uses of NL - tar indexing
March 02, 2008, 10:51:35 PM
QuoteShouldn't we talk not only about abstractions and principles, but also see if and how NewLisp could be used for scripting in the spirit of perl?


..some time ago I accidentally read a message of some sysadmins on the main BBC website. They were talking about developing another "perl framework" to take care of multiple problems, one of which seemed too many files in their web directories, they became unmanageable.



There is a very simple solution, but for some incomprehensible reason few people seem to be aware of it: use tar files, and index them. You'll be able to access contents in a tiny fraction of a second, so in effect your tar becomes an equivalent of a compressed read-only filesystem with random file access.



When "tar" itself reads the archive, it linearly scans the full length, and therefore it is way too slow.



I myself use it as a backend for storing archives of, for example, last year's postings on a blogging site, or to keep other collections of documents.

One obvious advantage is that you stop relying on unbundled software (many assume a relational db for storage), and with the ability of NewLisp to create tiny standalone executables one can create a really portable cgi application with a web interface.



One might add indexed tar storage to the NewLisp wikis to manage large file and postings collections.





1. The idea

Tar files are a concatenation of their files, with added headers and written in 512-byte blocks, so the end of the content section can be padded with 00s

Usually you'll want to keep gzip-compressed files inside (or alternatively make a tar of uncompressed files and compress the whole bundle, that is discussed later)



To index a tar therefore one needs to read the archive in 512 pieces, check for the "magic string" - the word "ustar" at position 517 will signal the header of a next member file. Then the indexer should read bytes 0-100 of the header, which will contain the name of the included file, padded with 00s.

There is more information in the header - Wikipedia is a quickest reference, if one wishes to avoid dense descriptions.



Next the indexer counts the 512 blocks until the next header with a filename, and the result, written to a file in a form convenient for the reader program will  look like
Quote...

archive/dir/some/file/name 1235 14

archive/dir2/some/other/file/name 3214 3

...

Meaning for the reader program - wind/seek 1235*512 bytes from the beginning, then get 14 blocks (until the next member file starts), and process to rid them of the header and padding garbage.





A skeleton indexer

I offer a simple "skeleton script" below. My primary uses of the scripts are not as command-line utilities, so the argument reading is primitive etc.


Code Select
#!/usr/bin/newlisp
;;@module index-tar

(if ( != (length (main-args)) 3)
( and (println "ntUSAGE: " (main-args 1) " Tar_file_namen") (exit))  )
 
(set 'l_a (2 (main-args)))
 
(if (file? (set 'tarball_file (pop l_a)))
(set 'fh_tarfile (open tarball_file  "read"))
(and (println "ntcould not open tar filen") (exit))  )

(set 'cnt 0)
(set 'prev_offset 0)
(set 'prev_name "dummy")
(set 'str_accumulator "" )

Now the main part of the processing begins.

After experimentation, i found that the fastest NL can provide - is writing into a memory buffer (a string).


Quote from: redundant string cast
One "trick" I use is - after reading the filename from the first 100 bytes of the header  - which is a string - I cast it as strng again:
Code Select
(set 'prev_name (string (0 100 str_processed_block)) )
The extracted filename is padded with 00s:



"archive/my/file/name000000000000....00

 0---------------------------------------------------------100



Casting this string as string again seemed an inexpensive way to truncate the padding.



P.S. Correction and speedup the fastest way to create the string is through the use of "format" command and without explicit casts. The script below is updated


Code Select
(while (read-buffer fh_tarfile 'tar_chunk 4096)
(dotimes (block_cnt 8)

(set 'str_processed_block ((* block_cnt 512) 512 tar_chunk))
(and
(= (257 5 str_processed_block) "ustar") ; --> this is the header block
(if  ( = cnt 0 )
(set 'prev_name (string (0 100 str_processed_block)) )
(write-buffer str_accumulator
(format "%s %d %dn" prev_name prev_offset (- cnt prev_offset) ) )
 
)
(set 'prev_name (string (0 100 str_processed_block) ))
(set 'prev_offset cnt)  
)
(inc 'cnt)

); /end of dotimes/ - 512k inspection of chunks
);end of while - 4k chunks gobbling

(write-buffer str_accumulator
(format "%s %d %dn" prev_name prev_offset (- (- cnt 1) prev_offset) ) )

(write-buffer 1 str_accumulator) ; /*result to STDOUT*/

(close fh_tarfile)
(exit)
..and write the result to STDOUT.



the length of the last  file in the archive  will be wrong (too large), but it won't interfere with correct file extraction.



......USAGE: scriptname tar_file_name > index_file_name



The indexer is sort of slow. For my tests I use a 90MB tar of tiny (5-50kb) text/html files, gzipped before they were put into the archive. The whole archive uncompressed was 250MB and consists of roughly 31k files.



The indexer goes through 90MB and 37000 items (dirnames, some trash add to the 31000 useful files) in roughly 3.9 seconds on an old machine (pentium 500MHZ).

Indexers written in C do that in roughly 1.0-1.3 seconds.



Because indexing is a one-time and infrequent job, I decided I could really live with it if the main script for the project, file listing/search and extraction from this as if "read-only tar filesystem" were fast enough.







2. List and get files quickly from large tar archives



I found two ways to scan files for their contents fast in NL:

(a) use the "search" operator. To speed it up roughly 3 times one might want to reset a buffer size in the newlisp code (see my previous posting), and then it becomes comparable to perl up to thousands of lines. It will lag after that

(b)   slurp the contents of the whole file into memory, and use string operators to search for information in it.



Below are the two short functions,  "f_search_in_file" and "f_slurp_and_search"  that do that.



The remaining code snippets can be concatenated into one working skeleton script, which is fed the index file in the format shown above and the tar file.

Depending on the options, the script will either find filenames (by its substrings etc - using regexps), the number of lines of output is given as options too - or cut out the files. So first adjust your regexp to "list" what you need, then change the option to "get" the file.


Quote
USAGE:

<prg> list|get search|slurp from_match_num to_match_num index_file tar_file



for example:

 (a)  ./script.lsp list slurp 100 120 '2006-.*'  index.txt archive.toz.tar

(find btw 100th and 120th matches for string 2006- )



(b) ./script.lsp get slurp 10000 10001 '/2006-.*' index.txt archive.toz.tar |tar xOf - |zcat

cut out the 1000th match by reading the index file into memory (a faster find in this case), getting the record, then using the offset and length to get it




In this skeleton script I simply cut the file out. With the header and trailing padding it's just a small one-file tar archive itself (or - of as many files as you decided to extract at once).

I could untar in NL, but because tar itself is a tiny 140k utility, I simply redirect into "tar xOf - " to get the stream on STDOUT, and because files inside the archive are gzipped, one might decompress them with zcat (also tiny), or serve as "gz" file.

Most web browsers support gzip compression


Code Select
#!/usr/bin/newlisp
;;@module .......

#------FUNCS---------

(define (f_search_in_file)

(set 'v_cnt 0)
(while (search fh_index (string str_pattern) 0)
(inc 'v_cnt)
(if ( > v_cnt (int from_rec_num))
(apply f_output (list (read-line fh_index)) )
(read-char fh_index)
)
(and
( >= v_cnt (int to_rec_num))
(close fh_index)
(exit)  )   );--end of while = grep for given file--

(close fh_index)
); /* end of f_search */

The "search" grepping inside files is very fast for files at the beginning of the archive - faster than slurping the index file into memory.


Code Select
(define (f_slurp_and_search)

(set 'cnt 0)
(replace (string str_pattern) ; -- replace what
(read-file index_file) ; -- replace where
(and ; -- pseudo-replace with what actions
(inc 'cnt)
(if (> (int cnt) (int from_rec_num))
(apply f_output (list $0)))
;(println $0))
;(f_get $0))

(and (>= (int cnt) (int to_rec_num))
(exit)  ); --end of and
); end of "and" = replace-actions
0); end of replace  
) ; /*end of f_slurp_and_search*/

This one is faster if the number of matches is high (e.g. you are grepping for 4000, or 10000 files in the index), or when (even a few) files lie at the end of the tar archive.



Next comes the primitive function to cut the contents out of the tar:


Code Select
(define (f_get str_tar_entry)

(set 'str_parsed (parse str_tar_entry))
(seek fh_tarball ( * (int (nth -2 1 str_parsed)) 512 ))
(read-buffer fh_tarball 'tar_item (* (int (nth -1 1 str_parsed)) 512))

; writing the cutout (which is tar itself) on STDOUT
(write-buffer 1 tar_item )
; writing into a tar file
;....
; writing and untarring into fsystem  
;....

); /*end of f_get*/



Main sets command-line options.


Code Select

#------MAIN----------

(if ( != 9 (length (main-args)))
( and
(println "ntUSAGE: " (main-args 1) " 'list|get' 'slurp|search' 'From_rec_num' 'To_rec_num' 'RegexpPattern' 'Indexname' 'Tarballname'n")
(exit))  
)


; -- these are globals: no args passing to the funcs
(set 'l_a (2 (main-args)))
(set 'grep_untar (pop l_a))
(set 'slurp_search (pop l_a))
(set 'from_rec_num (pop l_a))
(set 'to_rec_num (pop l_a))
(set 'str_pattern (pop l_a) )

(if
(= grep_untar "list") (constant 'f_output 'println)
(= grep_untar "get") (constant 'f_output 'f_get)
(println "nt action not defined: list|get
grep-like find files -- or get a cutout tar of file(s)
)


(if (file? (set 'index_file (pop l_a)))
(set 'fh_index (open index_file "read"))
(println "nt could not find the index file")
)

(if (file? (set 'tarball (pop l_a)))
(set 'fh_tarball (open tarball "read"))
(println "nt could not find the tar file")
)

(if
(= slurp_search "slurp") (f_slurp_and_search)
(= slurp_search "search") (f_search_in_file)
(println "nt type of processing not defined: slurp|search")
)


(exit)




3. The results

The results of this experiment are good, the script is quite usable.



I know of 2 other utilities for indexing tars.



One is called "tarix" and its home is sourceforge.net. It's written in C, but the author made an unfortunate choice: claiming a need for compatibility, he did read index file line-by-line, but rather character-by-character.  As a result extraction is slow: on my machine with 90MB tar (and 2MB index file of 37k entries, 31k useful entries) it gets the needed line from the index in appr. 0.300 seconds (and then seeking and cutting out is sufficiently fast)



The "tarix" author, however, also implemented an idea put forward by the author of the (well-known) "zlib" library. It's also possible to index in the similar manner compressed files!

Basically, if instead of tar of many compressed file1.gz ... fileX.gz you use  tar.gz, it's possible to create a double index. So when extracting a file, (a) find it in the index, (b) stream uncompression from the point before the file begins in the tar, to a point after it ends, and cut out the file tar blocks the roughly way we have done here.

Zlib author has prototype utility to show how it can be done, and "tarix" is the only program I know that actually picked up the idea.

Another note: tarix' invocation is quirky, it's not easy to use manually, however it may be Ok in the scripts.



My tests of the scripts above are much better. Extraction is

0.035 - 0.070 - 0.090 - 0.110  of a second for the majority of the files.

Getting files from the far end of the tar archive is 0.200 second for "slurp" and 0.600 second for "search"



The third set of utilities for such applications I found in some scientific software for working with biology data. I had to tear out the sources from the project tree, and they use some interesting tricks with memory management. Therefore, they are not usable without their accompanying library, around 200k (the utilities themselves are tiny 50-60k).

Those can extract files in tens of microseconds (e.g. 0.025 - 0.070 up to 0.100), even for the farthest files.



So, this humble and rather quick NewLisp hack comes as a strong second, and is especially good for wiki and blog uses, where new files are accessed frequently, and farthest and old are read much less.



I believe using indexed tars is a really good solution to management of file collections, for websites, or, if anyone adds necessary scripts to a file manager (such as "Midnight Commander" many use on Linux), as a means to browse and read from tars without the need to linearly scan (slooow) and/ot extract their contents into temporary directories (overfilling your file systems).



some results on an old 500MHZ machine


Quote
1. Grepping for files:

list/grep one file (end of index) -- "search":    0m0.675s

list/grep  one file (end of index)  -- "slurp":             0m0.243s



This is what the utility is not supposed to do. In real web usage extraction is done by pages (say, 20 at a time):

grep/find/list 12300 filenames - "search"    0m3.072s

grep/find/list 12300 filenames - "slurp"       0m1.246s



2. Extracting files

get/extract one file from the end  (13000th match) -- "search"  0m0.675s

get/extract one file from the end (13000th match) -- "slurp"      0m0.243s



Extraction from beginning or the middle of the file:

get/extract one 122th matching file (search)   0m0.043s

get/extract one 122th matching file (slurp)      0m0.065s



get/extract one 633th matching file (search)     0m0.088s

get/extract one 633th matching file (slurp)        0m0.089s



get/extract one 1220th matching file (search)    0m0.100s

get/extract one 1220th matching file (slurp)        0m0.108s


P.S. Comparison with sqlite indexer (an excellent indexer program) shows that unless you know precisely the filename you're looking for, you (or the web site user) are likely to search for a substring.

If indexed sqlite access is very quick (0.015s for my test file), any searches with "LIKE username%" will simply scan the index linearly, and in my tests sqlite provided worse results than the described utility.



P.P.S. One can index files not only by their filenames, but by other keys (categories, date, some key words etc.) The speed of reading the index, obviously, depends on the size of the index, so all unnecessary information should be cut out.
#6
Anything else we might add? / (a) tuning "search" I/O (b) bug?
February 25, 2008, 08:44:05 PM
Quote This post is:

(a) (hopefully) a hint on how to optimize one of NL i/o file operations to at least begin to compete with perl's

(b) a bug report (an unintended behaviour pattern that becomes visible in a certain case)


1. Scenario 1 Supposing we want to write an app with a web user interface, that is to be used by lots of people. The first task is to find a language that would pack the app as one piece, and eliminate the need to install and configure programs, libraries and  systems.



Scenario 2 Then if our application is also to be used in a multiuser environment, best  if it's written in such a way that  the only installation is  copying one file into a cgi-bin directory. It must be light, however, to serve many people - say,  allow the web server give 1 second delays when it is pounded by peak loads of 20-70 requests per second.



New Lisp is somewhat unique in that it could probably do it.



2. For these scenarios - and especially the first one - it's vital not to get tied with a separate DB engine. So the data our app is going to serve and keep should be available, fast, from a filesystem and some simple text file indeces, possibly.

In my experience flat text files are OK for up to tens of thousands of records, and when numbers grow,  an indexer should be used or flat text file indeces split into multiple levels.



3. So, NL could do that. However, its I/O speed when searches in files are performed, is not on par with perl.



4. The (read-line) - (process) - (print your_line) sequence is very, very slow in Newlisp.



On my test file (text) of 31k lines-"records" it does some search of, say, 20 records from somewhere in the end in 1.5 seconds, or even several seconds. Obviously, it is beyond any usable range.

Perl scans the same file with
Code Select
perl -ne 'm/term_to_match/ && print' filename
in 0.150 seconds, i.e. 10 times faster







5. Probably because of this NL includes another operator, specifically created to search in files, "search"



It loads a chunk of the file, checks for the pattern, and if it's found, seeks back to the first symbol of it. Next you can use (read-line), and it'll show you what remained of the line.

To display the full line it's possible to use a regexp, e.g.

".*my_search_word" -  but at a great cost to speed again.



One solution would be to write the indices in such a way that the truncation does not spoil getting at the result, but that's another question.



One other approach is to slurp the whole file into memory buffer and try to march lines from memory, a method that would exclude many i/o operationss.





6. Tuning "search" operation "Search" operator is better than (read-line - ...) cycle, but still it's slower than perl.



Perl scans my 31k-line file in flat time, 0.120 seconds, picking any number of records from any place in it.

NL slows down towards the end, and competes with perl only in the first, say, 100 records.



Is it possible to find optimizations?



I checked the source code, and it looks like adjusting the size of the buffer "search" uses to read the file in, can speed the thingy roughly 3 times, at least that's what it did on my machine.
QuoteFind in source code file "nl-string.c"

It sets SEARCH_SIZE constant to 0x4000 by default.



Reset it to 0x1000, then recompile newlisp.  It will speed the "search" operator roughly three times.



Possibly because 4k is the page size on my system.

Further shrinking of the buffer did not improve the search operation in any considerable way, so 0x1000 is the size I kept (actually, because memory is allocated as SEARCH_SIZE +1, I decided to set it, rightly or wrongly, to 0xFFF)


Now my scans of lines-records in an "index" file could compete with perl scans to roughly 1000 records, and at 10k records perl was only three times  faster, not 10 times.



As I am unfamiliar with the code, the question is - is this adjustment harmless? At the first sight, from reading the source for p_search in nl-string.c it seems so.  Could someone assure me, just in case?







7. Comparisons.



 Perl on a text file of 31k records, looking for 20 matches at 1400th "depth":
Code Select
time perl -ne 'BEGIN{$i=0}; (m/allan/) && ($i++) && ( ($i > 1380) && (print))  && ($i > 1400 ) && (
exit)' data.pro|wc -l
........real    0m0.137s


 Newlisp on the same file (before tuning)
Code Select
time ./lgrep.lsp  1380 1400 'allan' data.pro |wc -l
........real    0m0.407s

 Newlisp after tuning the SEARCH_SIZE parameter:
Code Select
time ./lgrep.lsp 1380 1400 allan data.pro |wc -l
........real    0m0.137s

and, finally, search for 20 records at 10000 matches tucked in the farther end of the 31k-record file:


QuotePerl:  same invocation,  getting 20 records from 9980th to 10000th

........real    0m0.157s

Perl scans files in flat time



Newlisp: same parameters -    

........real    0m1.480s -- before optimization

........real    0m0.516s -- after tuning



Newlisp adds up time as we dive deeper into a file or the number of extracted records grow. This lag might be further reduced if the need for a dummy i/o operation is eliminated












8. Bug or Feature?

Realistic queries into this text file will not, however, accumulate all records in the manner of "grep", which prints everything it found. Realistically in such an application I would need to get, for example, 20-item chunks of records (e.g. to display TOC of available documents on the site, page by page).



So, I would need to do a query like

"get me info on records between 980 and 1000 from the index file"



This will involve doing a "search", skipping its results, then next, until 980th match is found, and only then I'll start doing (read-lines) and print - until match 1000, when I will abort the procedure.


QuoteThe implementation of "search" does not allow another search to move forward, unless some output was performed: I have not found it in the code, but it seems, the  point in the file is not advanced when "search" is repeated - unless some operation happens that would rewind it forward a bit, and then the next search will be OK.


It might as well be a feature,  I do not know, but to me it looks like unintended behaviour.









9. Can one code around it?  - Yes, but..



 -- One can do a fake (read-line)  -- tested, it slows down the whole thing

 -- One could rewind forward one byte or so - but the NL implementation of "seek" does not allow one to use relative step; some fix like importing a "seek" from libc seems too cumbersome a solution

 -- One could do a fake "search" on a different pattern, which would advance, say, one character or two - so next the real search would be good again. It's way too slow

 -- And, finally, the solution I use for now - do a fake (read-char), which seems most lightweight.



Let's look at a simplistic "test-grep" script to see what this is about:


Code Select
       (set 'fh (open processed_file "read"))
        (set 'v_cnt 0)

        (while (search fh (string str_pattern) 0)
            (inc 'v_cnt)
            (if ( > v_cnt (int from_rec_num))

                ; yes, match in the range
                (println (read-line fh))

                ; no, skip this match -- here's the ugly
                (read-char fh) ; --> other workarounds R slower
            )

            (and
                ( >= v_cnt (int to_rec_num))
                (close fh)
                (println "nt first " from_rec_num " to " v_cnt  " matches in " processed_file)
                (exit))

        );--end of while = grep for given file--
        (close fh)






As I said , this code (after the SEARCH_SIZE tuning) competes with perl's scan (also with cutoff after the needed number of matches is extracted) until, say, 1000 records in roughly any place in a 31000-line file.



NewLisp begins to lag when either the number of needed matches grows (e.g. you need 1000 of them), and/or when you try to pick up some record starting in the thousands (4000th ot 10000th). At 10k matches (scattered  in various places inside that 31k file) NL seems to be at 1/3 of perl's speed, after the suggested tuning.



I believe that if the need to "jerk" the search operator with an i/o operation is eliminated,  NL "search" operator  can be further sped up.
#7
Anything else we might add? / A newlisp twin that died 3 years ago?
February 23, 2008, 06:02:18 PM
I recently dug out a reference to New Lisp and was very, very much impressed by the quality of the language:



1. Bits of the processing flow of my perl webscript/blogging engine, which  I prided myself for optimizing for speed, when recoded for test in NL ran roughly 2 times faster. Same algorithm, just recoded straightforwardly, with the provided equivalent or close in meaning NL operators



We'll see if the difference persists after I complete the port ;))



2. "Grep" in NewLisp seems to run with the speed of the C-coded GNU grep on my linux box, or almost (say, 0.458 seconds in NL against 0.450s with grep) when I tested it on a text copy of the Origin of Species (although comparison is not straightforward; at least they are "comparable")



3. However one even bigger advantage seems to be in the precision packing of the functionality into one executable. This is probably what a newer generation scripting language should look like - no more need to add modules/libraries and/or separate tiny utilities to make a typical web or other mainstream app run from the bare core language binary.





First I explained it by the intelligence of  NL creator. However, it seems that he was standing "on the shoulders of the giants", and things in the Lisp community are generally more mm.. advanced than in other scripting language circles.





That is what I began to believe when I discovered REFLISP - another, virtually identical packaging of a dialect of LISP with a web server and all needed libraries in a 126k binary:

"Reflisp", http://reflisp.sourceforge.net">http://reflisp.sourceforge.net



The project is written in C++, uses a more "standard" CL set of commands and syntax, has poor documentation (in the form of wiki pages in a wiki coded in Reflisp, which is supposed to be available on startup after a compilation), "installs" itself in a sort of "crooked" way (one may run it from the compilation tree, and/or would have to set and keep an environment var to point to the top of the installation, $REFLISPDIR, plus run it from a wrapper script.

So it's clumsier than NL and does not have that critical entry point: excellent tutorials and documentation.



The project, however, has been dead since 2005, and googling on the author's name one can come across a message (living under 1 April 2005 in his blog) that he is abandoning the project after all those years.  No joke this time?



Has anyone looked at this forgotten twin brother of NL? Run any tests (it seems to be able to accept CL scripts, probably with the help of some  macros) ?



And could anyone enlighten me, a longtime citizen of the unix world, and a thinker in C/shell/perl terms, but a newcomer to Lisp, if it's considered standard to package a language in the way NL and Reflisp are, or are those two just lucky exceptions?
Pages1