Text only | Text with Images

newLISP Fan Club

Forum => newLISP and the O.S. => Topic started by: Kirill on December 04, 2012, 03:28:04 AM

Title: CGI input data disappearing
Post by: Kirill on December 04, 2012, 03:28:04 AM
Hi, all,



I've had an issue that bothered me for a while now. Several times I've noticed that data, which newLISP receives from CGI, is incomplete. I don't know if it's a newLISP issue or not, but I've experienced it with CGI under both Apache (behind Squid on NearlyFreeSpeech) and mathopd (behind nginx).



Please see http://km.krot.org/cgitest.cgi for demonstration (source here: http://km.krot.org/cgitest.txt)



The scripts create a string with 20000 characters. Then the script generates a page, where the string is sent as a hidden parameter and textarea. Then the script compares the lengths. What worries me a lot is that the results are varying between form submissions.



Does anyone have any clues?



Kirill
Title: Re: CGI input data disappearing
Post by: Kirill on December 04, 2012, 04:46:08 AM
I suspect the issue is with newLISP, as perl works just fine:



http://km.krot.org/cgitest2.cgi (source: http://km.krot.org/cgitest2.txt).



All clues welcome!



Kirill
Title: Re: CGI input data disappearing
Post by: Lutz on December 04, 2012, 12:25:31 PM
This updated version 3.1 of cgi.lsp should fix this: http://www.newlisp.org/code/modules/cgi.lsp.html





ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67



ps: updated to 3.11 cgi.lsp a few minutes later
Title: Re: CGI input data disappearing
Post by: rickyboy on December 04, 2012, 03:29:12 PM
Who is Rocket Man?



And why does he/she use tabs to indent newLISP code? ;)
Title: Re: CGI input data disappearing
Post by: Kirill on December 04, 2012, 04:27:07 PM
Quote from: "Lutz"This updated version 3.1 of cgi.lsp should fix this: http://www.newlisp.org/code/modules/cgi.lsp.html





ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67



ps: updated to 3.11 cgi.lsp a few minutes later


Thanks, but now CGI:get returns nothing at all: http://km.krot.org/cgitest3.cgi
Title: Re: CGI input data disappearing
Post by: Kirill on December 04, 2012, 04:47:58 PM
I have a fix! http://km.krot.org/cgitest4.cgi



Patch will follow shortly...
Title: Re: CGI input data disappearing
Post by: Kirill on December 04, 2012, 04:52:43 PM
Patch for cgi.lsp: http://km.krot.org/code/cgi.lsp.patch



Now http://km.krot.org/cgitest3.cgi works properly as well.
Title: Re: CGI input data disappearing
Post by: Lutz on December 04, 2012, 05:09:16 PM
When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.



With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.
Title: Re: CGI input data disappearing
Post by: Kirill on December 04, 2012, 05:14:45 PM
And here's a patch for web.lsp: http://km.krot.org/code/web.lsp.patch
Title: Re: CGI input data disappearing
Post by: Kirill on December 04, 2012, 05:17:11 PM
Quote from: "Lutz"When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.


True, but I'm using their beta "realm", so newLISP and other tools are not so very old. newLISP version there is


Code Select
newLISP v.10.3.3 on BSD IPv4/6 UTF-8.

Quote from: "Lutz"With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.


Yes, that's why the patched code reads until it has read enough.
Title: Re: CGI input data disappearing
Post by: kanen on December 04, 2012, 05:25:53 PM
Thanks to Kirill, there's now a patch that also fixes this problem for web.lsp (part of artful-code).



You can always grab the latest version at:



//https://github.com/kanendosei/artful-newlisp
Title: Re: CGI input data disappearing
Post by: Kirill on December 05, 2012, 12:56:39 AM
The issue is solved, so I remove the test CGI scripts. Thanks.
Title: Re: CGI input data disappearing
Post by: Lutz on December 05, 2012, 09:32:35 PM
cgi.lsp has been updated again, to version 3.2 : http://www.newlisp.org/code/modules/cgi.lsp.html
Title: Re: CGI input data disappearing
Post by: Kirill on December 06, 2012, 04:44:15 AM
Quote from: "Lutz"cgi.lsp has been updated again, to version 3.2 : http://www.newlisp.org/code/modules/cgi.lsp.html


Thanks, Lutz! The file online does not seem to include the change below, though


Code Select

--- cgi.lsp.orig 2012-12-05 00:48:29.632208151 +0000
+++ cgi.lsp 2012-12-05 00:51:34.799112862 +0000
@@ -132,10 +132,13 @@
 ; get POST data if present, use CONTENT_LENGTH variable
 ; if available
 (if (env "CONTENT_LENGTH")
-    (set 'post-data "")
     (when (= (env "REQUEST_METHOD") "POST")
-        (while (read (device) buffer (int (env "CONTENT_LENGTH")))
-            (write post-data buffer))
+         (set 'recvd 0)
+         (set 'conln (int (env "CONTENT_LENGTH")))
+         (set 'post-data "")
+         (do-while (< recvd conln)
+                   (inc recvd (read (device) buffer conln))
+                   (write post-data buffer))
 
         (let (boundary (when (regex "multipart/form-data; boundary=(.*)"
                 (env "CONTENT_TYPE")) (append "--" $1)))
Title: Re: CGI input data disappearing
Post by: Lutz on December 06, 2012, 06:45:04 AM
Two things:



(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.



currently in cgi.lsp 3.2:
Code Select

(if (env "CONTENT_LENGTH")
    (when (= (env "REQUEST_METHOD") "POST")
        (set 'post-data "")
        (while (read (device) buffer (int (env "CONTENT_LENGTH")))
            (write post-data buffer))

(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.
Title: Re: CGI input data disappearing
Post by: Kirill on December 06, 2012, 07:51:02 AM
Thank you, Lutz.


Quote from: "Lutz"Two things:



(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.


Yes, the diff was against 3.11, which did not work, and I did not notice that the said statement was moved from before 'when' to after 'when'.


Quote from: "Lutz"(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.


Maybe it is good to throw error when CONTENT_LENGTH does not correspond to available data?
Title: Re: CGI input data disappearing
Post by: Lutz on December 06, 2012, 08:24:43 AM
QuoteMaybe it is good to throw error when CONTENT_LENGTH does not correspond to available data?


Yes, perhaps. But then also a response or error page should be formatted and sent back to the client. I guess in the real world, CONTENT_LENGTH will be ok most of the time and its not an issue to worry about. Almost always, CONTENT_LENGTH will be generated automatically by software.



I wonder how other web frameworks deal with these cases.
Text only | Text with Images