CGI input data disappearing

[Kirill]

Hi, all,



I've had an issue that bothered me for a while now. Several times I've noticed that data, which newLISP receives from CGI, is incomplete. I don't know if it's a newLISP issue or not, but I've experienced it with CGI under both Apache (behind Squid on NearlyFreeSpeech) and mathopd (behind nginx).



Please see http://km.krot.org/cgitest.cgi">http://km.krot.org/cgitest.cgi for demonstration (source here: http://km.krot.org/cgitest.txt">http://km.krot.org/cgitest.txt)



The scripts create a string with 20000 characters. Then the script generates a page, where the string is sent as a hidden parameter and textarea. Then the script compares the lengths. What worries me a lot is that the results are varying between form submissions.



Does anyone have any clues?



Kirill

[Kirill]

I suspect the issue is with newLISP, as perl works just fine:



http://km.krot.org/cgitest2.cgi">http://km.krot.org/cgitest2.cgi (source: http://km.krot.org/cgitest2.txt">http://km.krot.org/cgitest2.txt).



All clues welcome!



Kirill

[Lutz]

This updated version 3.1 of cgi.lsp should fix this: http://www.newlisp.org/code/modules/cgi.lsp.html">http://www.newlisp.org/code/modules/cgi.lsp.html





ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67">http://newlisponrockets.com/rockets-item.lsp?p=67



ps: updated to 3.11 cgi.lsp a few minutes later

[rickyboy]

Who is Rocket Man?



And why does he/she use tabs to indent newLISP code? ;)

[Kirill]

This updated version 3.1 of cgi.lsp should fix this: http://www.newlisp.org/code/modules/cgi.lsp.html">http://www.newlisp.org/code/modules/cgi.lsp.html





ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67">http://newlisponrockets.com/rockets-item.lsp?p=67



ps: updated to 3.11 cgi.lsp a few minutes later

Thanks, but now CGI:get returns nothing at all: http://km.krot.org/cgitest3.cgi">http://km.krot.org/cgitest3.cgi

[Kirill]

I have a fix! http://km.krot.org/cgitest4.cgi">http://km.krot.org/cgitest4.cgi



Patch will follow shortly...

[Kirill]

Patch for cgi.lsp: http://km.krot.org/code/cgi.lsp.patch">http://km.krot.org/code/cgi.lsp.patch



Now http://km.krot.org/cgitest3.cgi">http://km.krot.org/cgitest3.cgi works properly as well.

[Lutz]

When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.



With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.

[Kirill]

And here's a patch for web.lsp: http://km.krot.org/code/web.lsp.patch">http://km.krot.org/code/web.lsp.patch

[Kirill]

When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.

True, but I'm using their beta "realm", so newLISP and other tools are not so very old. newLISP version there is

Code Select Expand

newLISP v.10.3.3 on BSD IPv4/6 UTF-8.

With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.

Yes, that's why the patched code reads until it has read enough.

[kanen]

Thanks to Kirill, there's now a patch that also fixes this problem for web.lsp (part of artful-code).



You can always grab the latest version at:



https://github.com/kanendosei/artful-newlisp">//https://github.com/kanendosei/artful-newlisp

[Kirill]

The issue is solved, so I remove the test CGI scripts. Thanks.

[Lutz]

cgi.lsp has been updated again, to version 3.2 : http://www.newlisp.org/code/modules/cgi.lsp.html">http://www.newlisp.org/code/modules/cgi.lsp.html

[Kirill]

cgi.lsp has been updated again, to version 3.2 : http://www.newlisp.org/code/modules/cgi.lsp.html">http://www.newlisp.org/code/modules/cgi.lsp.html

Thanks, Lutz! The file online does not seem to include the change below, though

Code Select Expand


--- cgi.lsp.orig 2012-12-05 00:48:29.632208151 +0000
+++ cgi.lsp 2012-12-05 00:51:34.799112862 +0000
@@ -132,10 +132,13 @@
 ; get POST data if present, use CONTENT_LENGTH variable
 ; if available
 (if (env "CONTENT_LENGTH")
-    (set 'post-data "")
     (when (= (env "REQUEST_METHOD") "POST")
-        (while (read (device) buffer (int (env "CONTENT_LENGTH")))
-            (write post-data buffer))
+         (set 'recvd 0)
+         (set 'conln (int (env "CONTENT_LENGTH")))
+         (set 'post-data "")
+         (do-while (< recvd conln)
+                   (inc recvd (read (device) buffer conln))
+                   (write post-data buffer))
 
         (let (boundary (when (regex "multipart/form-data; boundary=(.*)"
                 (env "CONTENT_TYPE")) (append "--" $1)))


[Lutz]

Two things:



(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.



currently in cgi.lsp 3.2:

Code Select Expand


(if (env "CONTENT_LENGTH")
    (when (= (env "REQUEST_METHOD") "POST")
        (set 'post-data "")
        (while (read (device) buffer (int (env "CONTENT_LENGTH")))
            (write post-data buffer))

(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.