Problems with upload.cgi on newLISP 10.2.8

[Jeremy Reimer]

Hi folks,



I'm trying to get the upload.cgi script to work from this thread:



http://newlispfanclub.alh.net/forum/viewtopic.php?f=2&t=2585">http://newlispfanclub.alh.net/forum/vie ... f=2&t=2585">http://newlispfanclub.alh.net/forum/viewtopic.php?f=2&t=2585



I'm running Apache 2 on Ubuntu Linux.  I've used the version of the script for newLISP 10 but no matter what I do I get the error "wrong upload format" when the code reaches this point:

Code Select Expand


(if (not (find ".*filename="(.*)".*" disposition 1))
(throw-error "Error: wrong upload format"))


I've set permissions for the upload directory to 755 and upload.cgi to 777.  When I check my server, it seems like the script is creating the temporary file "upload-file", but never populating it with anything, as it always ends up as 0 bytes.  Since there is never anything in the temporary file, the symbol "disposition" which is set as (read-line "upload-file"), ends up being nil, and thus the script throws the error at the above point.



Is there something wrong with the script?  Does it have to be updated for newlisp 10.2.8?  It seems to me that it is never executing this while loop:

Code Select Expand


(while (read-buffer (device) buffer 1024)
   (write-buffer infile buffer))


And thus the file "upload-file" never gets written with anything.  



Thanks for your help!

[Lutz]

I just retested on Apache using newLISP 10.2.8 and it works fine:



http://www.newlisp.org/upload/upload-html.txt">http://www.newlisp.org/upload/upload-html.txt

http://www.newlisp.org/upload/upload-10-cgi.txt">http://www.newlisp.org/upload/upload-10-cgi.txt



on my installation at nearlyfreespeech.net, I use the the following permissions:

Code Select Expand

-rwxr-xr-x  1 me   web  1789 Dec 28 16:53 upload.cgi
-rw-r--r--  1 me   web   325 Dec 28 16:53 upload.html


owner and group names may be different on your installation. In above case 'web' is the group, the Apache process is in.



May be it is an Apache configuration issue to handle the "multipart/form-data" encoding type.

[Jeremy Reimer]

Hmm.. I tried using your exact scripts, with the same permissions, and got the same problem.



I'm running under Dragonfly, and both files live under the /views/ sub-directory, could this be causing the problem?



EDIT: I checked by changing the code to:

Code Select Expand


(set 'infile (open "./upload-file" "write"))
  (println "<p> Writing file...")
(while (read-buffer (device) buffer 1024)
   (println "<p>DOOD WE ARE IN YUR FILEZ WRITINGSZ" infile buffer))
(close infile)


And it never prints out the "DOOD" message, so there is something about the (while (read-buffer (device) buffer 1024) statement where it is never true.  The uploaded file is there when I print $POST, however (I've checked with both text and image files) so the data is being sent somewhere, but maybe just isn't being stored in the buffer?



EDIT: I'm still trying to figure out what's going on, but I'm not sure what's happening in this code:

Code Select Expand


; read data into intermediate file
(set 'infile (open "upload-file" "write"))
(while (read-buffer (device) buffer 1024)
   (write-buffer infile buffer))


According to the manual, (device) is a shorthand for whatever file handle was recently opened (which I assume is (open "upload-file" "write)).  But this is a temporary file, so there is nothing in it.  The manual says that (read-buffer) works like this:

Code Select Expand


Reads a maximum of int-size bytes from a file specified in int-file into a buffer in sym-buffer.


But if there is nothing in the file, why would it loop here at all?  Isn't it reading from "upload-file" and copying everything in that file to the symbol "buffer"?  But there is nothing in "upload-file" in the first place...



I mean, obviously the code works on your site, so the issue is somewhere else, like maybe my server configuration.  But I don't understand what the code is trying to do here.

[Lutz]

In upload.cgi (device) relates to the current STD I/O.



(read-buffer (device) buffer 1024) will read from the current stdin pipe which has been set up by Apache for the current CGI process to receive the POST data. (device) only refers to the recently opened file if explicitly set to it and as shown in the manual example.



I am not familiar with Dragonfly, but perhaps it already drained the stdin pipe? Perhaps running under Dragonfly, the POST data have to be retrieved in a different way? Somebody more familiar with Dragonfly, may be able to help you.

[Jeremy Reimer]

Something must be draining the stdin pipe, because (read-buffer (device) buffer 1024) returns nil. :(   I suppose I could parse the data in $POST, but it looks like this:

Code Select Expand


(("submit" "Upload") ("uploaded_data" [text] ----binary data here--- [/text]))


Which might work for saving a file if I could parse it correctly, but it doesn't contain the file name.

[TedWalther]

Yes, it looks like Dragonfly drains stdin.



I recommend you learn to use the ref* family of functions to use the POST data.



But I also agree with you on the usability of the format of $POST.



I recommended in the past that inclusion of a MIME module in the base system.  The POST data is MIME data; it should be properly parsed as MIME data.  It isn't hard.  I regret I haven't had time to do such a module myself.  And the Dragonfly author has been very vigorous in developing his vision; I don't think he really understood what I was trying to get at.   MIME is the standard; it would be very useful to have a MIME module, because MIME is an excellent interchange format for self-describing data, much better than XML in many cases, and more traditional and easy to understand with a text editor than JSON.



Read about it in the following RFC's:



RFC 2045, RFC 2046, RFC 2047, RFC 4288, RFC 4289 and RFC 2049



MIME is a very very old standard, and is one of the backbone internet standards.  It is a good standard.  Having the capacity to parse and emit it (emitting is so trivial, a module isn't really needed) will allow us to encourage new developers to use this excellent standard, to the benefit of data interchange and the prevention of bit-rot.

[Jeremy Reimer]

Okay, I am having some problems parsing the $POST data.  Basically, I can extract the raw text inside the [text] brackets using this code:

Code Select Expand


 (setq posted (rest (rest (rest (rest (rest (rest (rest (chop (string (rest (last ($POST)))) 8)))))))))


But the main problem is that as soon as it gets converted to a string, all sorts of blank characters are gone.  So my original file (in binary) looks something like this:

Code Select Expand


ÿØÿà JFIF  ` `  ÿá hExif  II*         >      


But once it gets translated into a string, it looks more like this:

Code Select Expand


ÿØÿàJFIF``ÿáhExifII*>


How can I keep these characters from dropping?

[itistoday]

Sorry, I just skimmed over this thread and saw Dragonfly mentioned. Just want to mention that Dragonfly should take care of all the uploading stuff for you, you don't need to write your own. See the documentation for the $FILES:



http://www.rundragonfly.com/dragonfly_api">http://www.rundragonfly.com/dragonfly_api



$FILES is for multipart-form data.



There's also a different method of uploading data, which is basically a POST to the URL with binary data, in which case use $BINARY.

[Jeremy Reimer]

Hey, that's great!  I'm a bit unsure on what to do with ($FILES), however.  When I try to see what's in it with (println ($FILES)), it just comes back "()".  (length ($FILES)) returns 0.  There's lots of data in ($POST) but nothing in ($FILES).  Am I doing something wrong?  I'm using the same upload HTML as before:

Code Select Expand


<form name="FileUpload" action="upload-10.cgi"
                        method="POST" enctype="multipart/form-data">
<input type="file" name="uploaded_data" size="40">
<input type="submit" value="Upload" name="submit">
</form>


Thanks for your help!

[itistoday]

That's odd, I don't have time unfortunately to test this out immediately (in the middle of a move), but try removing the size attribute perhaps?



Also, try running it through Apache and not newlisp's server. If that doesn't work let me know (or better yet, file an issue on the http://github.com/taoeffect/dragonfly-newlisp">github project page).

[Jeremy Reimer]

Thanks for your quick reply.  I'm running under Apache on Linux.  I took the size attribute out but the same issue still exists.  I'll fiddle around with it a bit and try it on another Linux server I have and see if I can get it to work.

[Jeremy Reimer]

Sorry, I had left this issue for a long time while I was working on other projects, but now I'm revisiting it and I'm still having the problem.  (In the meantime I did learn how to use ref to extract $POST data, heh)



I've filed an issue on github about it: https://github.com/taoeffect/dragonfly-newlisp/issues/issue/1">https://github.com/taoeffect/dragonfly- ... es/issue/1">https://github.com/taoeffect/dragonfly-newlisp/issues/issue/1



EDIT: In the mean time, I was able to use the following code to solve my problem with the missing binary data I talked about above.  Using the base64-enc and base64-dec functions seemed to work perfectly, and allowed me to upload a .JPG file successfully to the server:

Code Select Expand


(set 'encoded-data (base64-enc (last (($POST) (chop (ref '"uploaded_data" ($POST)))))))
(write-file "uploadedfile.jpg" (base64-dec encoded-data))


I still can't seem to grab the file name itself from ($POST) but this is at least some progress.



EDIT: My solution was to add the file name into an additional text box, and change it using a Javascript onChange event when the user clicks on the file dialog:

Code Select Expand


<form name='FileUpload' action='fileupload.cgi' method='POST' enctype='multipart/form-data'>
<input type='file' id='uploadName' name='uploaded_data' onChange='this.form.textname.value = this.value'>
<input type='text' name='textname'>
<input type='submit' value='Upload' name='submit'>
</form>


This will add the file name to the ($POST) data when you submit the form.  I could make the 'textname' field hidden if I wanted.  On some browsers/platforms it adds a "C:\fakepath\" to the name, but that's easy enough to strip out:

Code Select Expand


; take out stupid fake path if it's there
(set 'file-name (replace "C:\fakepath\" file-name ""))

(println "<p>File name..." file-name)

(write-file file-name (base64-dec encoded-data))

(println "<P>Uploading file... complete.  File is at <a href=" file-name ">" file-name "</a>")


Anyway this seems to work.  I can upload files now from Dragonfly!  Woo!

[itistoday]

Hey Jeremy, unfortunately my plate is ridiculously full at the moment, so I can't look into this—off the top of my head I'd suggest trying with Apache and see if it works there (if you're using newlisp's built-in server for this). If that doesn't help and you're able to figure out why, I'd be happy to apply any patches.

[Lutz]

Since development version 10.2.17, newLISP HTTP server works with binary contents and upload.cgi as well. But of course this assumes, that there is no preprocessing taking place with Dragonfly, so upload.cgi gets the data directly from the HTTP server.

[shoukei]

The latest reply was somewhat nearly 6 years ago but anyway I found a fix that seems to work everyone!!



Change the two lines in

dragonfly-newlisp/example-site/dragonfly-framework/lib/request.lsp

lines 142-143



142 - (if (set 'val (regex-captcha (string var {="(.+?)"}) disp))

142 + (if (set 'val (regex-captcha (string "filename" {="(.+?)"}) disp))

143 - ($FILES var (list (list 'name val) (list 'data data) (list 'length (length data))))

143 + ($FILES val (list (list 'name var) (list 'data data) (list 'length (length data)))



Then, the following works as commented in the source.(line 49 of request.lsp)

;; use like this: (lookup 'data ($FILES "filename"))

;; Valid keys: 'data, 'name, 'length



The number of files uploaded can be obtained by (length ($FILES)).

The name of each file can be obtained by (first (nth i ($FILES))) where i is the number starting from 0 for file1 and 1 for file2 ...



I'm running

OSX10.10.5, newLISP v.10.7.1, —()o Dragonfly WEB FRAMEWORK VERSION 0.73