Uploading files problem

[TPJ]

I have a web app written in newLISP. The app is served by the newLISP interpreter run in the HTTP server mode. The server runs behind a proxy, which is nginx. So the flow is simple: nginx gets the requests, and proxies it to the newLISP HTTP server.



I have problems with files uploading. The script mentioned in http://newlispfanclub.alh.net/forum/viewtopic.php?f=2&t=2585">this topic doesn't work, since no CONTENT_LENGTH (or CONTENT_TYPE) environment variable is set (I checked it). BTW, I used the upload.html code to perform all tests.



I tried to just read the stdin, but it doesn't work. My CGI scripts (pure newLISP) get nothing to read (stdin is empty).



When I tried to reach the newLISP HTTP server directly (without nginx serving as a proxy), I was able to upload a text file. I just used a simple script, that reads stdin, and gets all the content of the sent file:

Code Select Expand


(set 'infile (open "upload-file" "write"))
(while (read (device) buffer 1024)
  (write infile buffer))
(close infile)


Unfortunatelly, sending any binary file (e.g. images) doesn't work at all. The stdin is broken after a few bytes read. For example, in case of a GIF image, all I got was:

Code Select Expand

-----------------------------1124875958105779246987104586
Content-Disposition: form-data; name="uploaded_data"; filename="polsl.gif"
Content-Type: image/gif

GIF89a_



Now, the questions.



1) Why on earth my script doesn't get anything on stdin when newLISP HTTP server is run behind a proxy (nginx in my case)?

2) Why on earth binary files aren't properly sent, while text files are?



I'll appreciate any help.

[Lutz]

I am not familiar with the nginx proxy server, but when using the Apache HTTP server, use this:



http://www.newlisp.org/index.cgi?page=File_Upload_Script">http://www.newlisp.org/index.cgi?page=F ... oad_Script">http://www.newlisp.org/index.cgi?page=File_Upload_Script



also referenced from here:



http://www.newlisp.org/index.cgi?Tips_and_Tricks">http://www.newlisp.org/index.cgi?Tips_and_Tricks



The server side upload script handles a POST request from a client HTML form.



When uploading a file to a newLISP HTTP server directly without a proxy in-between, you can use '(write-file <target URL> <content>)' on the client directly, no script is necessary server side as newLISP HTTP server has GET/PUT/POST/DELETE handling built-in (only POST requests would need a script handler):

Code Select Expand

(write-file "http://thesite.com/binaryfile.bin" (read-file "binarycontent"))


'write-file' when seeing a 'http://' prefixed target will issue a HTTP PUT request to newLISP HTTP server and it works with binary content on Windows and all  Unix platforms.

[TPJ]

Thanks for your remarks. Using write-file is not a solution, since I'm working on a web application. My only option is to use the POST method.





I'll start with a little update. I've been trying to solve that problem for two days now (spending on it my full time), and I've made some interesting observations:



1) Reading the stdin from a newLISP CGI script works perfectly, as long as a text file is uploaded. It works perfectly in both cases: when using nginx as a proxy, and without any proxy server.



2) Reading the stdin from a newLISP CGI script fails when a binary file is uploaded. After reading a few bytes, the stdin suddenly breakes. Once again: it happens in both cases: with ngxinx proxy, and without it.



I have described this problem http://stackoverflow.com/questions/4133870/uploading-binary-files-problem">here, on stackoverflow (you can check there what exactly I get from stdin in my scripts). The proposed solution didn't help, though. But let's forget about this problem, for a while.



I decided to investigate it more. I've installed the http://www.acme.com/software/thttpd/">thttpd server, and used it to perform the same task: serve two pages, one upload.html, and one CGI script, that should just read the stdin and save it to some file. I used a simple shell script to do the job:

Code Select Expand


#!/bin/sh

cat > to-file

echo -e "Content-type: text/htmlrnrn"
echo "<p>Hello!</p>"


The result: everything ran fine, just as it should. I was able to upload any file I wanted.



So, I tried to do the same thing using the newLISP HTTP server, and exactly the same two pages (including the simple shell script). The result: all binary files uploaded were broken, just as in the first case I described in this post.



I'm not an HTTP server expert... Therefore I hope that my conclusion is wrong. But the conclusion is that something is wrong with the newLISP HTTP server - since it's the only thing that had been changed before everything stopped to work correctly. And I don't like this conclusion, since I intend to use the newLISP HTTP server a lot.



I tested it all on my Arch Linux box, if it is of any importance.

[Lutz]

Currently newLISP HTTP server only handles textual data in POST requests and doesn't handle multipart/form-data content-type at all because the CONTENT_TYPE environment variable is not set. This has now been added to 10.2.17:

 

http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/



and it works well with the upload.cgi script and binary content:



http://www.newlisp.org/index.cgi?page=File_Upload_Script">http://www.newlisp.org/index.cgi?page=F ... oad_Script">http://www.newlisp.org/index.cgi?page=File_Upload_Script



tested on Mac OS X 10.6.4 and UBUNTU Linux 10.4