newLISP wiki

[cormullion]

A couple of things. First of all, I saw this in index.cgi:



{ form name="commentform" action="index.cgi" menthod="post" }



Is that 'menthod' really correct? Seems to work OK, but it looks like it should be 'method'.



Second, how would you go about adding an extra question/answer to the comments function, to prevent spam comments. Presumably the fields could be added to (comment-dialog page-name), but where should I add the checking? And how to do this without modifying the index.cgi and preventing new versions from overwriting the changes...

[cormullion]

OK, I've got a bit further...



Am I right in thinking that the index.cgi script runs and quits after every user action? So if I want some data to survive between, say, displaying a comment form, and the user posting that comment form, I've got to save that data somewhere? How would I do that?



thanks!

[Jeff]

That is the nature of cgi, yes.  Typically, saving data between sessions is done in one of three ways:



1.  Database, such as mysql

2.  Cookies (or sessions, which are local files identified by a unique id set in a cookie)

3.  Temp file, which is not an elegant solution, since it is not atomic (two users could change it at once) and it is not efficient.

[Jeff]

One other thing- I am currently working on a technique using newLISP's ability to pickle an entire scope that preserves execution between web sessions.  The idea is that loading a page involves building a context that contains a render function, which actually sends the page to the browser.  All of the data associated with that page is marshalled and then saved as a session, using a unique id stored in a cookie.



It works just like regular web sessions, but you can pickle the entire stack, like continuations in scheme, except that the current location of execution is not preserved.  However, that can be easily remedied since the marshalled context can be loaded as a string and analyzed first.



I don't have anything to really distribute yet, but it is a promising line of code :)

[cormullion]

Interesting!



My more basic explorations are in the area of comments and captchas. So when index.cgi displays a comments page, it also displays some kind of challenge. Then, when the user submits a comment, their response to this challenge is compared with the expected response. But i can't see how to avoid saving the 'right' answer between 'sessions'. Perhaps a temp file is the easiest way. Putting the right answer in a cookie doesn't feel right for some reason, unless it's encrypted in some way.

[Jeff]

You could use a temp file, but you would still need to have a cookie to uniquely id the temp file.  There is no reason not to use a cookie in this instance because the cookie would not be storing any sensitive data; the only real risk is that someone steals the person's captcha value and then posts with it.  They would still have to post under their own login.

[cormullion]

Thanks, Jeff. I'm now looking at how to do cookies, so perhaps Lutz or another wiki-guru can help me...



I don't know where to place the CGI:set-cookie call when building a comments page. It says it should go immediately before the (print  "Content-type:) call, but the only one I can find is right at the start of index.cgi file, before cgi.lsp is even loaded...

Code Select Expand


(print  "Content-type: text/htmlrnrn")

(load "cgi.lsp")
(load "pages/setup.lsp")


[Jeff]

You put the cookie before the content-type header is printed.  Here is some working code for you.  This is the CGI module that I created for myself.  Each function is documented (not using newlisp docs, just using docstrings in each function):

Code Select Expand

(define-macro (assert)
  "(assert expr msg) asserts that expr does not evaluate to nil. If so,
  it throws an error using message msg."
  (unless (eval (args 0)) (throw-error (args 1))))

(constant (global 'assert))

(context 'cgi)

;;; Parameters (GET/POST)

(set 'params '((post '()) (get '())))

(define (add-param method var val)
  "Adds (var val) to params for method ('cgi:get or 'cgi:post)."
  (push (cons var val) params
        (cons (first (ref method params)) -1)))

(define (get-param method var)
  "Finds param var in params of method ('cgi:get or 'cgi:post)."
  (lookup var (assoc (if (string? method) (sym method) method) params)))

(define (parse-param-string str , rs)
  "Parses query string and returns sets of (key value) pairs."
  (if (catch
       (letn ((sets (parse str "&")) (pairs (map (fn (s) (parse s "=")) sets)))
          (filter
            (fn (pair)
                   (and (string? (first pair)) (not (empty? (first pair)))))
            pairs))
       'rs) rs '()))

(define (find-params)
  "Gets get and post data returns parsed (key value) pairs for both
  'cgi:get and 'cgi:post."
  (let ((get-params (parse-param-string (env "QUERY_STRING")))
        (post-params (parse-param-string (read-line))))
       (list (cons 'get get-params) (cons 'post post-params))))

(set 'params (find-params))

;;; Cookies

(set 'cookies '())

(define (read-cookies , rs)
  "Reads all cookies into cgi:cookies from HTTP_COOKIE."
  (if (catch
       (letn ((cookie-str (env "HTTP_COOKIE"))
           (cookies (parse cookie-str "s*;s*" 0))
           (pairs (map (fn (c) (parse c "=")) cookies)))
          pairs) 'rs) rs '()))

(define (set-cookie var val expires domain path)
  "Prints the set-cookie string for var=val, with expiration date expires
  (which should be a (date) style string), domain, and path."
  (assert (for-all true? (list var val))
 "string expected in function set-cookie")
  (let ((cookie ""))
       (begin
         (write-buffer cookie (format "%s=%s;" var (string val)))
         (if expires (write-buffer cookie (format " expires=%s;" expires)))
         (if domain (write-buffer cookie (format " domain=%s;" domain)))
         (if path (write-buffer cookie (format " path=%s;" path)))
         (print (format "Set-Cookie: %sn" cookie)))))

(define (get-cookie var)
  "Looks up var in cgi:cookies."
  (lookup var cookies))

(set 'cookies (read-cookies))

;;; Header manipulation

(set 'output-started nil)

(define (header content-type)
  (assert (nil? output-started) "Output already started in function header.")
  (unless (string? content-type) (set 'content-type "text/html"))
  (print (format "Content-type: %srnrn" content-type))
  (set 'output-started "Output started in function header."))

(define (redirect url)
  (assert (nil? output-started) "Output already started in function redirect.")
  (assert (string? url) "String expected in function redirect.")
  (print (format "Location: %sn" url))
  (header))

(define (refresh)
  (redirect (current-url)))

;;; Miscellaneous

(define (current-url)
  (let ((host (env "HTTP_HOST")) (uri (env "REQUEST_URI")))
       (string (if (env "HTTPS") "https://" "http://") host uri)))

(context MAIN)

Here is a sample of setting a cookie:

Code Select Expand

(if (set 'cookie (cgi:get-cookie "some-key"))
  (begin
    (assert (nil? cgi:output-started) "output already started!")
    (cgi:header)
    (println "I have a cookie: " cookie))
  (begin
    (assert (nil? cgi:output-started) "output already started!")
    (cgi:set-cookie "some-key" "some-value)
    (cgi:refresh)))

The reason for the refresh at the end is that after setting the cookie, it is not available through the cgi environment until after the page has been loaded again.

[cormullion]

Cool. Is this 'compatible' with Lutz' newlisp wiki, or a more parallel/alternative version?

[Jeff]

I haven't the faintest idea.  I've never touched the newLISP wiki.  But many of the functions are named the same.  You can go through and check through.

[rickyboy]

Second, how would you go about adding an extra question/answer to the comments function, to prevent spam comments. Presumably the fields could be added to (comment-dialog page-name), but where should I add the checking? And how to do this without modifying the index.cgi and preventing new versions from overwriting the changes...

Cormullion,  I was wondering about a good way to do this myself a few moons ago; then I ran out of time.  I'd ask Bob Bae about how he did it -- I think he uses some incarnation of the newLISP wiki.  Look at the bottom of http://terpri.com/index.cgi?P-2007-07-06-18-00-22-415101">this page to see his challenge interface (which uses the figlet app BTW).



I hope that helps.  And please get back to us if Bob divulges his challenger. :-)

[cormullion]

Yes - I've had a word with Bob! He's very busy, but had time to say that he integrated the figlet distribution with his newlisp wiki.



But I had been thinking of just a simple 'question-response' thing, to deter completely-automated spam comments... I had nearly everything working, checking of the user's answer against the expected answer - when I discovered that I had to remember the expected answer across calls to index.cgi! (Duh!) Hence the move towards cookies, and working out how to send cookies at the start of a comments page. The newlisp-wiki as it's distributed doesn't use cookies, although they are supported by cgi.lsp.



I'm going to study Jeff's cgi stuff now.

[Jeff]

By the way, on that program challenge on the tepri site, I couldn't post my solution.  I don't think his form processing is properly escaping inputs.



Here is how I modified his program.  I get a *lot* better performance on a huge word list (several hundred thousand words, with duplicates):

Code Select Expand

(define (sort-by-length lst)
  (map last (sort (map (fn (w) (list (length w) w)) lst) >)))

(define (check-word sub-word-list word)
  (dolist (sub-word sub-word-list)
    (replace sub-word word ""))
  (if (= 0 (length word)) true false))

(setq file-words (parse (read-file "junk.txt") "n"))
(setq sorted-words
  (sort-by-length (unique file-words)))

(dolist (word-x sorted-words)
  (setq word-save word-x)
  (setq sub-word-list '())
  (dolist (sub-word sorted-words)
    (if (and (member sub-word word-x) (!= sub-word word-x))
        (push sub-word sub-word-list -1)))
  (setq sub-word-list (sort-by-length sub-word-list))
  (while (!= sub-word-list '())
         (setq word-x word-save)
               (if (check-word sub-word-list word-x)
                   (begin
                     (println "found: " word-save)
                     (exit)))
               (setq sub-word-list (rest sub-word-list))))

Sorting by first mapping lengths to each word means that the sort function doesn't have to call length twice on each comparison, and then in the inner dolist loop, I changed it from the file-words list to the sorted-words list, which means shorter list of words (the uniquely filtered words).

[cormullion]

After investigating a bit more, I think that I can do it without cookies. You put this in the comment-dialog function (in the newLISP wiki index.cgi), after generating the question and answer:

Code Select Expand


{Please answer the following question}
question
{input type="hidden" name="the-correct-reply" value="} (base64-enc (encrypt the-right-answer key)) ...
{input name="user-reply" value="" ...}


and put this in the (if (CGI:get "postcomment")) handler:

Code Select Expand


(set 'the-right-answer (encrypt (base64-dec (CGI:get "the-correct-reply")) key))
(set 'user-reply (CGI:get "user-reply"))


only renaming them to make it less obvious... This way, you can extract the expected and received values from the form, and the user won't be able to decrypt the answer from the HTML source without the key. (Which is another story.)



It might work...