newLISP Fan Club

Forum => newLISP in the real world => Topic started by: grable on June 17, 2005, 09:42:56 AM

Title: Modified HTTPD
Post by: grable on June 17, 2005, 09:42:56 AM
I tinkered with the included HTTPD some, and added a modified CGI:put-page into it, which executes an *.lsp file inside the current process and sends the result back.



should be a little faster then starting new processes, and scripts have full acces to the HTTPD context.

(reads <% %> and <%= %> tags.)



heres an example usage:
<%
(set 'page-title "Your IP")
%>
<html>
<head>
<title><%=page-title%></title>
</head>
<body>
<h1><%=page-title%>: <%=(first (net-peer HTTPD:connection))%></h1>
</body>
</html>


heres the modified HTTPD file (//ftp) for those interested.



this is just so much fun =)
Title: small update...
Post by: grable on June 17, 2005, 04:21:09 PM
added support for query data with GET, but am having problems with POST.

which is wierd since its basicly the same happening on both GET and POST.

(puting data in QUERY_STRING env, and runing cgi-put-page)



maybe i need a fresh pair of eyes on this ;) hehehe



dev version with funky POST HTTPD_dev (//ftp)
Title:
Post by: Lutz on June 17, 2005, 04:35:14 PM
Look into modules/cgi.lsp, perhaps it can help you. GET and POST isn't really the same. In a GET request variable/value pairs are encoded in the QUERY_STRING but in in a POST request you have to read more to get the post data. cgi.lsp handles this in a transparent fashion. It also contains a routine to do <% %> tag preprocessing, but your approach to have it integrated in the httpd itself is probably the better way to do it.



Lutz
Title: doh!
Post by: grable on June 17, 2005, 04:37:16 PM
my bad ;) was just a pathing issue.. hehehe



ive updated the original HTTPD (//ftp) link, for those interested.
Title:
Post by: grable on June 17, 2005, 04:40:17 PM
Thats where i "borrowed" this idea from in the first place.. ;) hehehe



bit im just sending the POST data in QUERY_STRING.. thats probably not the right thing to do ? . maybe add a POST_STRING env or something?



am not that familiar with http and cgi.
Title:
Post by: grable on June 17, 2005, 05:00:07 PM
Well i think i solved it..  theres realy no need to use env variables when i have acess to the HTTPD context.. to i just put things in HTTPD:QUERY_STRING and HTTPD:POST_STRING when the data is present..



heres the recent examle (and previous url has again been updated)

il probaly revork this code, since it pretty much a hack right now to get it to work properly..


<%
(set 'page-title "Your IP")
%>
<html>
<head>
<title><%=page-title%></title>
</head>
<body>
<h1><%=page-title%>: <%=(first (net-peer HTTPD:connection))%></h1>
<hr>
QUERY_STRING: <%=HTTPD:QUERY_STRING%><br>
POST_STRING: <%=HTTPD:POST_STRING%><br>
<form method="POST" action="yourip.lsp?testing">
<input type="text" name="postdata" value="some random data">
<input type="submit">
</form>
</body>
</html>




btw lutz (the newlisp expert ;) , in the loop of the cgi-put-page that evals embeded parts i do this:
(while ...
   ...
   (context (sym file-name))
   (eval-string ...)
   (context HTTPD)
   ...
)
 ...
(delete (sym file-name))
 is this the correct way to use a context? or should i put it outside of the loop?
Title:
Post by: Lutz on June 17, 2005, 06:09:47 PM
Thats the right way to do it. Changing the context around the evaluation will protect httpd from stuff happening inside the evaluation. I think because of that same reason I left <% %> tage processing in cgi.lsp used by the spawned cgi process.



Your way of server-side processing is probably more efficient, but also brings the danger of doing something 'evil' to the httpd server. Bracketing in its own context will not protect against everything. Imagine code in the preprocessed page hangs in a loop, then the httpd server is dead ;)



Lutz
Title:
Post by: grable on June 17, 2005, 06:19:48 PM
Ah.. k.. i was just wondering.. glad i understood it.. hehe


QuoteImagine code in the preprocessed page hangs in a loop, then the httpd server is dead ;)


hehehehe, yeah.. i thought about that, il just have to be very careful ;)



Although an infinite loop could potentialy happen with another process to, since the httpd will wait for its termination anyway.
Title: Cookies!!!
Post by: grable on June 17, 2005, 07:34:34 PM
my hacked version of HTTPD now eats cookies =)



i merged the functions from CGI.lsp with HTTPD, so scripts have roughly the same interface.



script interface:
 # variables
HTTPD:QUERY_STRING  = raw query data
HTTPD:POST_STRING    = raw post data
HTTPD:COOKIES_STRING = raw cookies data
HTTPD:CONTENT_TYPE  = content type of script result (text/html is default)
HTTPD:PUT_COOKIE    = raw cookie data to send to client
HTTPD:PARAMS         = query & post data as (key value) pairs
HTTPD:COOKIES        = cookies data as (key value) pairs

# functions
(HTTPD:get-cookie "name")
(HTTPD:set-cookie "name" "value")
(HTTPD:set-cookie-ex "name" "value" "domain" "path")
(HTTPD:get-param "name")


HTTPD now does the same as CGI.lsp, eg it fill the PARAMS and COOKIES for you before execution of the script, allso you can have multiple cookies in one request.





example usage:
<%
(set 'visits (int (HTTPD:get-cookie "VISITS")))
(if (= visits nil) (set 'visits 1))
(HTTPD:set-cookie "VISITS" (+ visits 1))
%>
<html>
<head>
<title>Test Page</title>
</head>
<body>
<h1>Your IP: <%=(first (net-peer HTTPD:connection))%></h1>
<hr>
Number of visits using cookies: <%=visits%><br>
QUERY_STRING: <%=HTTPD:QUERY_STRING%><br>
POST_STRING: <%=HTTPD:POST_STRING%><br>
COOKIES_STRING: <%=HTTPD:COOKIES_STRING%><br>
<form method="POST" action="yourip.lsp?testing">
<input type="text" name="postdata" value="some random data">
<input type="submit">
</form>
</body>
</html>


hope someone else allso finds this useful =) i kinda like having full access to my server, and being able to do whatever i want with it .. hehe



newest version of HTTPD (//ftp)
Title:
Post by: HPW on June 17, 2005, 11:37:29 PM
Hi grable,



I have problems to access your FTP.

Can you offer another option or send it?
Title:
Post by: grable on June 18, 2005, 06:21:51 AM
QuoteI have problems to access your FTP.

Can you offer another option or send it?


Its in the mail =)



the ftp is on 24/7 so it "should" work.. dunno why you cant get in..



theres no password for anonymous, so the first one should work,

but some browsers may complain so the second one is more "correct".



//ftp://anonymous@grable.cjb.net/HTTPD

//ftp://anonymous:@grable.cjb.net/HTTPD



if anyone else cant get it and wants me to send it, just speak up =)
Title:
Post by: statik on June 18, 2005, 11:55:47 AM
Very cool.
Title: complete turnaround ;)
Post by: grable on June 19, 2005, 07:59:29 AM
I soon found out that having everything in a single process isnt to good either :/



its was a little bit faster then spawning a new process..

bit it could stil only serve 1 client at a time.. so it now spawns

process again ;) LOL



although this time with proper "forking" using (process)

so its can serve many clients.



it sends the "control" code over the commandline, and loads the

html+newlisp template file and runs it... so the control code must send

the result to the client and close the socket. freeing the server up to do other things =)



allmost same interface as before, just removed the HTTPD context from the script

and i removed the old CGI way completly.



heres the updated files:

grb_httpd.lsp (//ftp)

index.lsp (//ftp)



go here to test //http://grable.cjb.net/



UPDATE: forks regular file downloads as well, like images/etc.
Title:
Post by: Lutz on June 19, 2005, 08:56:53 AM
Very interesting, congratulations, I would love to see the code, but cannot connect to your ftp, can you email it to me? lutz@nuevatec.com, and put the word 'newLISP' in the subject line, so my spamfilter doesn't eat it :)



Lutz
Title:
Post by: grable on June 19, 2005, 09:13:36 AM
Thanks lutz =)



their in the mail...



Since some people are having trouble geting in to my ftp, ive

put the files on the http server as well..



//http://grable.cjb.net/grb_httpd.rar
Title:
Post by: HPW on June 19, 2005, 11:43:50 AM
Hi grable,



Just tried your HTTP download link.



It open the Save_as dialog and show the file-download window.

But there it hangs and get no progress.



When I enter only:



http://grable.cjb.net/



then I get the:



Index - test page for grb_httpd.lsp HTTP server



But there the 'Download Source' link also does not work.



Is this a newLISP-server? Seems to have problems.
Title:
Post by: grable on June 19, 2005, 11:51:00 AM
Yes, its on the the modified newlisp server.



Wierd, because i have no trouble downloading myself. I even tested a 50 mb file while constantly refreshing the index page, without any problems.



Seems a few ppl have trouble acessing my ftp too :(



What browser are you using? what os??

Myself am on Windows XP and using the latest FireFox..
Title:
Post by: HPW on June 19, 2005, 12:04:03 PM
Windows XP pro german IE 6.0.2600



It does get the response normal when using:



http://grable.cjb.net/



But clicking on the download link it takes very long to get any more.



?
Title:
Post by: grable on June 19, 2005, 12:10:35 PM
hmmm.. it must be IE ;)  the ppl having trouble with the ftp use IE to .. hehe



have you tried with newLISP's (get-url) ? that should bypass any problems in the browser.
(write-file "c:grb_httpd.rar" (get-url "http://grable.cjb.net/grb_httpd.rar"))
or better yet, switch to firefox perhaps? ;)



or i could just email them to you..



it must be IE's handling of the "application/octet-stream" mime-type or something.. for its the same process in sending a webpage as in sending a file.. its only the Content-Type (mimetype) thats different in the header.



im sorry, but theres not much i can do :(



UPDATE:

  i tested in IE myself now, and the index page was slow as hell, and the download didnt work either :(  so theres defnetly something IE does or doesnt do that firefox and (get-url) does .. il have look into it later.
Title:
Post by: HPW on June 19, 2005, 12:21:06 PM
I get it with:

(write-file "c:grb_httpd.rar" (get-url "http://grable.cjb.net/grb_httpd.rar"))

But it was stored in "c:programmenewlispgrb_httpd.rar"



My IE has no problems with all download from newLISP.org or other sites.

So a newLISP server should be able to work with IE because a majority still use it (Mozilla might be better, but we can not change everyone's browser installation)
Title:
Post by: grable on June 19, 2005, 12:29:16 PM
Yeah. i forgot the extra  for paths ;)



although it ended up in C: on my pc.. lol



yes yes.. i know it has to work in IE as well ;)..

 but for now itl have to be NON-IE only.. cuz i have no clue as to why IE is slow and refuses to work.



i havent changed that much ;) hehehe
Title:
Post by: grable on June 19, 2005, 12:44:35 PM
LOL



I forgot to close the socket when done sending data ;) stupid me!



its wierd that this worked on firefox at all! it should atleast have waited for the timeout..



even though i think newLISP/exe's closes open sockets on exit.



but now it works atleast ;) so all you happy IE users can use this shit to.
Title:
Post by: grable on June 19, 2005, 01:58:24 PM
I just changed the newLISP WIKI to work with the format i use in the modified httpd, wasnt that much to change. so far so good ;)



I just have to test all the functionality in newLISP wiki before i post it.



And i fixed some stuff in the server aswell.



newest version is here:

//http://grable.cjb.net/grb_httpd.rar
Title:
Post by: HPW on June 19, 2005, 02:11:47 PM
A (happy) IE user say: Thanks



;-)



Of cource: No shit here!
Title:
Post by: grable on June 19, 2005, 02:36:29 PM
QuoteA (happy) IE user say: Thanks


glad to help.. it was stupid of me not to see it earlier ;)



UPDATE:

  yes i know theres alot of them.. atleast you get the latest no?



the modified wiki for grb_httpd works, and its much faster this way than running on the old httpd.



just replace the files in this rar with the ones you got. TAKE BACKUP!!



//http://grable.cjb.net/moded_wiki.rar



or you can test it here:



//http://grable.cjb.net:8080
Title:
Post by: grable on June 21, 2005, 02:52:56 PM
Made the cgi processor take lisp code spaned over multiple statements..



eg:
<% (for (x 0 5 1) %>
<b><%= x %></b><br>
<% ) %>


more like real server pages does it ;)

now, its a little bit slower this way i imagine, since it converts that to several (print) statements and then evals it.



I havent tested it fully, but it works on my small test samples..



anyhow, im updating the downloads with the new code so you can try it for yourself =)



//ftp://anonymous@grable.cjb.net/grb_httpd.rar

//http://grable.cjb.net/grb_httpd.rar <-- may not be up 24/7



UPDATE:

  allso the wiki needed a minor adjustment.. you can grab it here (//http)

or fix it yourself, its just to add 1 line of code



just go to the bottom of the file "index.cgi" and change the code to this
;; take parameter as page name and display it
(set 'link QUERY_STRING)
(if (or (= link "") (not link))
  (display-page "Home")
  (display-page link)
)
(throw) # <--- this line
Title:
Post by: grable on June 21, 2005, 08:02:29 PM
I setup a small page for future projects and the HTTPD, to stop bothering this forum with my minor updates ;)



//http://grable.cjb.net



and downloads have been updated to

//http://grable.cjb.net/downloads/grb_httpd.rar

//http://grable.cjb.net/downloads/moded_wiki.rar
Title:
Post by: HPW on June 21, 2005, 10:32:15 PM
Quote from: "grable"
//http://grable.cjb.net


From my IE I get no response on this link.

Loading endless.


Quote from: "grable"
//http://grable.cjb.net/downloads/grb_httpd.rar

//http://grable.cjb.net/downloads/moded_wiki.rar


The download links works fine!
Title:
Post by: grable on June 22, 2005, 03:19:02 AM
Damit, i allways forget to check with IE ;)



Thanks again for letting me know =)



il look into it. heheheh



UPDATE:

   Seems there is some issues with the wiki and the asp syntax,

   but I think ive fixed it now though.
Title:
Post by: HPW on June 22, 2005, 03:57:09 AM
Everything runs fine now!

;-)