Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - hartrock

Pages 1 ... 3 4 5 6 7 ... 10
#61
newLISP in the real world / Re: [fr] more symmetry between push and pop
August 04, 2015, 05:29:38 PM
Quote from: "Lutz"
There is a frequent code pattern used where you build lists or queues by always pushing to the end using the -1 index to build queues. You are creating a 0 or -1 element position. Then using pop without index until empty?. If you do not allow pushing on on an empty list at -1, code gets complicated as the first element to push, now is a special case to be treated by code.

This is the FIFO use case with choosing it at push side: it's good to have this choice.

There is the other FIFO use case with choosing it at pop side: it would be nice to have this choice, too.


Quote from: "Lutz"
Basically we are having a https://en.wikipedia.org/wiki/Worse_is_better">https://en.wikipedia.org/wiki/Worse_is_better discussion here ;-)

Thanks for the pointer: this may be ;-)
#62
newLISP in the real world / Re: [fr] more symmetry between push and pop
August 04, 2015, 04:10:24 PM
Let's give some additional perspective.
Quote from: "update some hours later"The pusher is feeding

the popper is eating,

both are using a queue

with LIFO;



but pusher may choose

poor popper has to loose,

to use this as a queue

with FIFO.

Prenote: here standard push/pop means, doing this without ix for back/front info; which has the well known and expected LIFO semantics, if both sides use this.



But the pusher has a choice: currently how to push decides, if standard poping has LIFO - pushing without or 0 ix - or FIFO - pushing with -1 ix - semantics.

On the other poper side there is no such choice after a standard push (without wrapper code around pop). If poping with -1 ix would be allowed, this choice could also be made afterwards by poper side, after a standard push without ix.



There has been a usecase, which has led to this thread: providing a standard push without any side argument (back or front), followed by a decision of the poper how to fetch elements from this queue.



Currently I don't see a problem with providing this additional symmetry regarding the choice of LIFO/FIFO semantics:

[*] choice how to push with standard pop (already provided), versus
  • [*] standard push with choice how to pop (missing).
    • [/list]

    PS:

    A variation of this is to introduce push-back and pop-back for the to/from back cases - both together used standard again leads to LIFO like for standard push/pop; a mixture between old and new to FIFO (problematic here could be possible confusion with indices counting backwards in the non-standard case (which would be consequent for the -back variants)).

    This would have the advantage of the same code structure for FIFO as well as for LIFO cases, at the cost of two more symbols.
    #63
    newLISP in the real world / Re: [fr] more symmetry between push and pop
    August 03, 2015, 05:21:12 PM
    Quote from: "Lutz"Up to version 9.2.11 when indexing lists, indices too big would pick the last element and indices to small or to negative would pick the first element in a list. For empty lists the result would be nil in a consistent fashion, e.g for pop, nth, first, last etc..



    In programming practice this often caused undetected programming errors and it was decided to flag all index overruns as an error starting with version 9.2.12 in January 2008.

    Agreed: to be restrict is good for detecting programming errors.



    But what do you think about my proposal of allowing (pop '() 0) and (pop '() -1) (for empty lists) returning nil?

    This would give some symmetry regarding LIFO/FIFO functionality (and the same indices for empty lists would be allowed for both pop and push).



    nil pop'ing from an empty list from front or back seems to be reasonable to me (analogue to push in the other direction).



    PS:

    One difficulty is nil as element in lists, whose difference from an empty list cannot be detected by pop alone (as it is now for pop without ix (for implicit LIFO), so no change with my proposal).

    PPS:

    push currently is less picky as pop regarding ix'es with empty lists. But as well it is reasonablle to treat 0 and -1 as push'ing back/front, this could also be the case for pop.

    PPPS: Currently push/pop-ing without indices means LIFO, this is implicitely using ix 0.; but explicitely using this ix works only for push.

    PPPS:

    The 'more more symmetry' idea could be dangerous regarding hiding of programming errors: trying of pop'ing from an unexistent list may be wrong. But the same applies for push now, which allows using a nil valued sym. Here I think, there are good reasons for both variants (comfort and denseness of code versus detecting programming errors); but it could be good to choose one of them for both push'n'pop (less to memorize amongst more 'symmetry' in code).



    Hopefully my points are more clear now.
    #64
    newLISP in the real world / [fr] more symmetry between push and pop -> [fr] cancelled
    August 02, 2015, 09:34:36 AM
    [Update 2] Feature request ([fr]) cancelled.

    There is a reason for not making (pop aList -1) as simple as (push elem aList -1): it is performance, which degrades for longer lists, because there is a pointer forwards from each cell to its next cell in the list, but not backwards. This has implications for pop back (because its predecessor has to be reached from the other list end), but not for push back: thanks to 'last element optimization' of keeping a pointer to last element in list.

    So there is an asymmetry in the sense, that regarding performance it's better to leave the FIFO/LIFO choice at pushers side, which is visible in making (or leaving) the less performant variants more difficult to use.

    Nevertheless there may be usecases for the cpop/em-cpop macros (see link in previous update).



    [Update] See http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4732#p23319">//http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4732#p23319 for better and more general macro code.




  • [*] On one side push is robust, if pushing into s symbol set to nil: it creates an empty list then, and ignores some given index just in this special case.

  • [*] On the other side pop always expects a list, no one will be created, if there is none (together with ignoring a given ix then).
    • [/list]
    But this is not the problem for FIFO functionality: this would be solved by some more symmetrie between push and pop at another place.




     
  • [*] (pop '() -1)) and (pop '() 0)

        as valid expression returning nil; this corresponds to

    •  
  • [*] (push "foo" '() 0) and (push "foo" '() -1)

        , which is working now.
    • [/list]
  • [*] More more symmetry would be reached by creating an empty list for pop'ing from a nil symbol value, just as for push (ignoring any given ix and returning nil in this case).
    • [/list]
    Interesting to note, that
    Code Select
    (push '() 1)
    does not work; pushing to an empty list only works for indices 0 and -1. It could be better for detecting coding errors, if the indices would be limited to 0 and -1 as in the nil symbol value case (should be the same for pop after 'more more symmetry' then).





    It is clear, that changing the semantics of push and/or pop could break some legacy code; so such a change would likely deserve a greater version number increase.




    ;; define some things
    (macro (push-top    E L) (push E L  ))
    (macro (push-bottom E L) (push E L -1))
    (macro (pop-top     L) (pop L  ))
    (macro (pop-bottom  L) (pop L -1))
    ;; repair by:
    (macro (pop-bottom-repaired L) (if (> (length L) 0) (pop L -1)))

    (define (doIt num_push push_op_sym num_pop pop_op_sym)
      (print (format "%45s" (string "push_op: " push_op_sym ", pop_op: " pop_op_sym)))
      (set 'l '()
           'push_op (eval push_op_sym)
           'pop_op (eval pop_op_sym))
      (dolist (e (sequence 1 num_push))
              (eval (push_op e l)))
      (print "; after push_op: " l)
      (print "; pop:")
      (dotimes (i num_pop)
               (print " " (eval (pop_op l))))
      (println))
    [/code]

    There has been the following session:
    Code Select

    newLISP v.10.6.4 64-bit on OSX IPv4/6 UTF-8 libffi, options: newlisp -h

    > ;;
    > ;; this works:
    > ;;
    > (doIt 3 'push-top 3 'pop-top) ; LIFO
               push_op: push-top, pop_op: pop-top; after push_op: (3 2 1); pop: 3 2 1
    nil
    > (doIt 3 'push-bottom 3 'pop-bottom) ; LIFO
         push_op: push-bottom, pop_op: pop-bottom; after push_op: (1 2 3); pop: 3 2 1
    nil
    > (doIt 3 'push-bottom 3 'pop-top) ; FIFO
            push_op: push-bottom, pop_op: pop-top; after push_op: (1 2 3); pop: 1 2 3
    nil
    > (doIt 3 'push-top 3 'pop-bottom) ; FIFO
            push_op: push-top, pop_op: pop-bottom; after push_op: (3 2 1); pop: 1 2 3
    nil
    > ;;
    > ;; but this fails, if 'pop-bottom is trying to pop from an empty list:
    > ;;
    > (doIt 3 'push-top 4 'pop-top) ; LIFO
               push_op: push-top, pop_op: pop-top; after push_op: (3 2 1); pop: 3 2 1 nil
    nil
    > (doIt 3 'push-bottom 4 'pop-bottom) ; LIFO
         push_op: push-bottom, pop_op: pop-bottom; after push_op: (1 2 3); pop: 3 2 1
    ERR: invalid list index in function pop
    called from user function (doIt 3 'push-bottom 4 'pop-bottom)
    > (doIt 3 'push-bottom 4 'pop-top) ; FIFO
            push_op: push-bottom, pop_op: pop-top; after push_op: (1 2 3); pop: 1 2 3 nil
    nil
    > (doIt 3 'push-top 4 'pop-bottom) ; FIFO
            push_op: push-top, pop_op: pop-bottom; after push_op: (3 2 1); pop: 1 2 3
    ERR: invalid list index in function pop
    called from user function (doIt 3 'push-top 4 'pop-bottom)
    > ;;
    > ;; using repaired version: voila!
    > ;;
    > (doIt 3 'push-top 4 'pop-top) ; LIFO
               push_op: push-top, pop_op: pop-top; after push_op: (3 2 1); pop: 3 2 1 nil
    nil
    > (doIt 3 'push-bottom 4 'pop-bottom-repaired) ; LIFO
    push_op: push-bottom, pop_op: pop-bottom-repaired; after push_op: (1 2 3); pop: 3 2 1 nil
    nil
    > (doIt 3 'push-bottom 4 'pop-top) ; FIFO
            push_op: push-bottom, pop_op: pop-top; after push_op: (1 2 3); pop: 1 2 3 nil
    nil
    > (doIt 3 'push-top 4 'pop-bottom-repaired) ; FIFO
    push_op: push-top, pop_op: pop-bottom-repaired; after push_op: (3 2 1); pop: 1 2 3 nil
    nil
    >


    Triggered by this code (for copy/paste at once):
    Code Select

    ;;
    ;; this works:
    ;;
    (doIt 3 'push-top 3 'pop-top) ; LIFO
    (doIt 3 'push-bottom 3 'pop-bottom) ; LIFO
    (doIt 3 'push-bottom 3 'pop-top) ; FIFO
    (doIt 3 'push-top 3 'pop-bottom) ; FIFO
    ;;
    ;; but this fails, if 'pop-bottom is trying to pop from an empty list:
    ;;
    (doIt 3 'push-top 4 'pop-top) ; LIFO
    (doIt 3 'push-bottom 4 'pop-bottom) ; LIFO
    (doIt 3 'push-bottom 4 'pop-top) ; FIFO
    (doIt 3 'push-top 4 'pop-bottom) ; FIFO
    ;;
    ;; using repaired version: voila!
    ;;
    (doIt 3 'push-top 4 'pop-top) ; LIFO
    (doIt 3 'push-bottom 4 'pop-bottom-repaired) ; LIFO
    (doIt 3 'push-bottom 4 'pop-top) ; FIFO
    (doIt 3 'push-top 4 'pop-bottom-repaired) ; FIFO
    #65
    newLISP in the real world / Re: get-url of empty content fails
    July 24, 2015, 01:22:13 PM
    Quote from: "Lutz"check out: http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/ 2015-07-24 02:12



    now always four members in list mode: header, content, http response line, status code

    Thank you!

    Result now seems to be a good compromise between:

    [*] effort and improvement, and
  • [*] different error handling purposes.
    • [/list]

    PS: one test has to be updated:
    Code Select

    sr@free:/tmp/newlisp-10.6.4/qa-specific-tests$ diff qa-net_orig qa-net
    88c88
    < (if (find "server code 404:" (get-url (string "http://" host ":" port "/xyz.xyz")))
    ---
    > (if (find "ERR: HTTP/1.0 404 File or Directory not found" (get-url (string "http://" host ":" port "/xyz.xyz")))
    #66
    newLISP in the real world / 2. Update: suggestion (removed errorneous "fix")
    July 23, 2015, 01:01:02 PM
    2. Update: suggestion (removed errorneous  "fix")



    What about putting the int status code (or nil, if there is an invalid HTTP status line) as fourth member after the HTTP server response string in "list" mode?

    Or is there any problem with this idea, I don't see?



    There already is HTTP status as an int in nl-web.c, so parsing it from outside seems to be unneeded. If someone wants to get exact status line - e.g. for checking HTTP version - it is there after your proposal: then parsing it is the way to go.
    #67
    newLISP in the real world / different error levels [Re: get-url of empty content fails]
    July 22, 2015, 08:25:53 AM
    Quote from: "Lutz"
    Ps: May be newLISP shouldn't prepend and error message to an exsisting server error page at all. Perhaps the error message should only be generated when no page comes back from server after the header.




    [*]Sounds like an improvement, because typical server behavior is to give some info about the error (which can be more detailed than just having the HTTP status code/message).
  • [*]A further improvement for clients handling the HTTP status computationally could be:

    [*] return nil for no content, or "" for empty one (content-length 0);

  • [*] introduce (http-status) for getting status code or nil, if there is none (because of some other error); this should work in all modes ("", "list", "list raw").
    • [/list]    [/list]


  • [*] low-level: (net-error).

  • [*] HTTP request level: check for correct HTTP message format.

      Second: similar to "3. Second:".

  • [*] HTTP response level: check for correct HTTP message format first.

       Second: there may be a correct HTTP message format with some HTTP status code, but nevertheless there may be incorrect HTTP behavior; e.g.

       
    [*] wrong HTTP status code;

    •  
  • [*] a message body where none is allowed, or no message body where it is needed.
    • [/list]
  • [*] HTTP status code: if all is OK until here, HTTP status code may indicate success or errors higher level.

  • [*] HTTP protocol level: correct server behavior comprising multiple requests? E.g. a DELETE followed by a GET  of the same resource should give a 410 or 404 status.

  • [*] server high level: functional behavior, internal server errors (returning some 5xx'er status code), etc..
    • [/list]

    Some of net-errors are lower level, some are HTTP related; from the manual:

    1 - 18: lower level errors

    19   HTTP bad formed URL

    20   HTTP file operation failed

    21   HTTP transfer failed

    22   HTTP invalid response from server

    23   HTTP no response from server

    24   HTTP document empty

    25   HTTP error in header

    26   HTTP error in chunked format



    Some questions and comments to these error codes:

    21   HTTP transfer failed:

    Which error cases are covered here?

    22   HTTP invalid response from server:

    Which error cases are covered here? All at 'HTTP response level' above? Or more at higher levels above, too?

    23   HTTP no response from server:

    What's the difference to '18   Operation timed out'?

    24   HTTP document empty:

    An empty document in sense of empty content is not an error, but missing empty content (without 'Content-Length or Transfer-Encoding header field' with empty message body).

    25   HTTP error in header:

    Probably not in client request, but server response. If so: how to get errors in client request headers?

    26   HTTP error in chunked format:

    Same question as in 25.



    Ideas (to be made more mature) for improvement:

    [*]
    [*] Reduce HTTP errors returned by (net-error) to one - e.g. "19 HTTP error";
  • [*] introduce (http-error) for more fine-granular HTTP errror handling]
  • [*] error request header URL (related to specific header entry)

  • [*] ... (other specific header entries)

  • [*] error request headers (related to all headers)

  • [*] error request format (related to whole message (headers and body))

  • [*] error response header xxx (specific header)

  • [*] error response headers

  • [*] error response format chunked (specific)

  • [*] error response format (general)

  • [*] error protocol (request and response seen together one or more times).

  • [*] ...
    • [/list][/list]
  • [*] More  fine-granular HTTP errror handling from a. could be put flat into (net-error) errors as well: but this does not reflect the different levels the errors are arising from.
    • [/list]

    Problems seen yet:

    [*] A more clear separation of different protocol levels for finer error control in some protocol stack stands against the goal of reducing error handling complexity for typical use cases.
  • [*] Going this or a similar route makes work (and time is limited for all of us). Just writing this post...
    • [/list]
    #68
    newLISP in the real world / bug hunting: success! [Re: get-url of empty content fails]
    July 22, 2015, 04:57:35 AM
    Quote from: "Lutz"Many thanks! I think it is solved now. The cpymem test clearly showed the missing zero termination of the response string in the cell. This happened for content appended to an error message.



    newLISP also stores the size of a memory buffer independent of string zero-termination. Because of this, the error doesn't always show up.



    http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/ from 2015-07-21 20:45



    Ps: May be newLISP shouldn't prepend and error message to an exsisting server error page at all. Perhaps the error message should only be generated when no page comes back from server after the header.


    Good news: bug has gone!
    Code Select

    sr@mad:~/newLISP/BoF$ newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Wed, 22 Jul 2015 11:50:22 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'r (t 1))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'rc (0 r))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (list (length r) (length rc)) (list (utf8len r) (utf8len rc))
    (63 63)
    (63 63)
    > (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
    (true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true)
    > (dump (t 1)) (dump r) (dump rc)
    (140043466192144 260 140043466192208 64 13359008)
    (140043466191856 260 140043466178576 64 13320048)
    (140043466191792 260 140043466178576 64 13358928)
    > (address (t 1)) (address r) (address rc)
    13359008
    13320048
    13358928
    > (set 's "____________________")
    "____________________"
    > (cpymem (+ (address r) 50) (address s) 20) s
    20
    " Not Found.nn00an)rnC"
    > (cpymem (+ (address rc) 50) (address s) 20) s
    20
    " Not Found.nn00000000000000"
    >
    #69
    newLISP in the real world / Re: get-url of empty content fails
    July 21, 2015, 08:39:26 AM
    Quote from: "Lutz"
    May be it has to do with UTF8 handling, although all my tests are fine without it. Can you compile without UTF8 and try again?

    Same result:
    Code Select
    sr@mad:/tmp/newlisp-10.6.4$ ./newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Tue, 21 Jul 2015 15:36:29 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'r (t 1))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'rc (0 r))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (list (length r) (length rc)) (list (utf8len r) (utf8len rc))
    (63 63)

    ERR: invalid function in function list : (utf8len r)
    > (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
    (true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true)
    > (dump (t 1)) (dump r) (dump rc)
    (140371457590416 260 140371457590480 64 29344512)
    (140371457590128 260 140371457576976 64 29343904)
    (140371457590064 260 140371457576976 64 29344432)
    > (address (t 1)) (address r) (address rc)
    29344512
    29343904
    29344432
    > (set 's "____________________")
    "____________________"
    > (cpymem (+ (address r) 50) (address s) 20) s
    20
    " Not Found.nnian)rn00"
    > (cpymem (+ (address rc) 50) (address s) 20) s
    20
    " Not Found.nn00000000000000"
    >
    #70
    newLISP in the real world / bug hunting: '\0' to late in errorneous string cell content
    July 21, 2015, 08:23:58 AM
    Found interesting difference in strings (see at bottom):
    Code Select
    sr@mad:~/newLISP/BoF$ newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Tue, 21 Jul 2015 14:49:31 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'r (t 1))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'rc (0 r))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (list (length r) (length rc)) (list (utf8len r) (utf8len rc))
    (63 63)
    (63 63)
    > (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
    (true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true)
    > (dump (t 1)) (dump r) (dump rc)
    (140457260381456 260 140457260381520 64 20146112)
    (140457260381168 260 140457260367888 64 20107152)
    (140457260381104 260 140457260367888 64 20146032)
    > (address (t 1)) (address r) (address rc)
    20146112
    20107152
    20146032
    > (set 's "____________________")
    "____________________"
    > (cpymem (+ (address r) 50) (address s) 20) s
    20
    " Not Found.nnian)rn00"
    > (cpymem (+ (address rc) 50) (address s) 20) s
    20
    " Not Found.nn00000000000000"
    >

    So '' comes to late in (t 1) and r, but not in rc.

    Interpretation:

    [*] String cell->contents have been copied in both copy cases (set 'r (t 1)) and ((set 'rc (0 r)) (see dump).
  • [*] Copying string cell->contents by (0 r) honors string length in cell and ensures ending its content with a ''; but (t 1) does only a copy until a '' byte in string content of source, without looking for string length.
    • [/list]     Hope that helps.
    #71
    newLISP in the real world / bug hunting [Re: get-url of empty content fails]
    July 21, 2015, 02:09:28 AM
    Bug stays with (info taken from http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/)
    Code Select
    newlisp-10.6.4.tgz      2015-07-20 23:50  1.6M     (dump output of differing strings added):
    Code Select
    sr@mad:~/newLISP/BoF$ newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Tue, 21 Jul 2015 09:22:58 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'r (t 1))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'rc (0 r))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (list (length r) (length rc)) (list (utf8len r) (utf8len rc))
    (63 63)
    (63 63)
    > (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
    (true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true)
    > (dump r)
    (139910737302512 260 139910737289232 64 19955600)
    > (dump rc)
    (139910737302448 260 139910737289232 64 19994480)
    >

    Is it normal, that both have same 'cell->next: linked list ptr'?
    #72
    newLISP in the real world / Re: get-url of empty content fails
    July 20, 2015, 09:49:34 AM
    Bug stays with
    Code Select
    newlisp-10.6.4.tgz      2015-07-20 15:25  1.6M    :
    Code Select
    sr@mad:~/newLISP/BoF$ newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 16:13:26 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 16:13:48 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    >

    Additional observation: with triggering bug and printing in one line, at very first time only there is no bug to see, but then it appears and stays:
    Code Select

    sr@mad:~/newLISP/BoF$ newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug")) (print (t 1))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 16:09:54 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug")) (print (t 1))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 16:10:06 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > ; repeating the same from here bug stays.


    It's difficult for me to trigger this bug under other circumstances; one example: if there is a 404er directly from Apache (without newLISP server behind), there is no bug to see:
    Code Select
    sr@mad:~/newLISP/BoF$ newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/foo" "dummy content" "text/plain" "list debug"))
    POST /foo HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 16:36:38 GMTrnServer: Apache/2.2.22 (Debian)rnVary: Accept-EncodingrnContent-Length: 276rnConnection: closernContent-Type: text/html; charset=iso-8859-1rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">n<html><head>n<title>404 Not Found</title>n</head><body>n<h1>Not Found</h1>n<p>The requested URL /foo was not found on this server.</p>n<hr>n<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>n</body></html>n"
     404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    <title>404 Not Found</title>
    </head><body>
    <h1>Not Found</h1>
    <p>The requested URL /foo was not found on this server.</p>
    <hr>
    <address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>
    </body></html>
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">n<html><head>n<title>404 Not Found</title>n</head><body>n<h1>Not Found</h1>n<p>The requested URL /foo was not found on this server.</p>n<hr>n<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>n</body></html>n"
    >


    Don't know, how to come nearer to the problem this way.



    Is there anything, which I could check in direction of 'digging deeper'?

    Like here:
    Code Select
    sr@mad:~/newLISP/BoF$ newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 17:48:02 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (set 'r (t 1))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'rc (0 r))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (list (length r) (length rc))
    (63 63)
    > (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
    (true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true)
    > (= r rc)
    nil
    > (println r)
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    i
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (println rc)
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.


    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > ;
    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 17:52:31 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (println (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)

    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    >

    That both length and chars of strings are '=', but not their comparison with '=' is very strange.

    [update] utf8len is equal, too:
    Code Select
    > (list (utf8len r) (utf8len rc))
    (63 63)


    Compiler:
    Code Select
    sr@mad:~$ gcc -v
    Using built-in specs.
    COLLECT_GCC=gcc
    COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
    Target: x86_64-linux-gnu
    Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
    Thread model: posix
    gcc version 4.7.2 (Debian 4.7.2-5)
    #73
    newLISP in the real world / Re: get-url of empty content fails
    July 20, 2015, 07:14:04 AM
    Testing with
    Code Select
    newlisp-10.6.4.tgz      2015-07-19 23:31  1.6M     bug stays (first call OK, thereafter bug):
    Code Select

    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list"))
    ("Date: Mon, 20 Jul 2015 13:51:33 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > ; -> OK
    > ;
    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 13:52:17 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > ; -> bug triggered
    > ;

    Another observation: if at first provoking the bug with "list debug", then it stays even with arg "list" not provoking it in former test, but - update - only once:
    Code Select

    sr@mad:~/newLISP/BoF$ newlisp
    newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Mon, 20 Jul 2015 14:06:13 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list"))
    ("Date: Mon, 20 Jul 2015 14:06:34 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list"))
    ("Date: Mon, 20 Jul 2015 14:18:50 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    >

    Anything else I could test?



    BTW: it would be helpful, if
    Code Select
    newlisp -v     (or -v -v) would contain a built timestamp (to be sure, that you are really using a just built binary (though just building/installing does not ensure, that you have built from the correct source)).
    #74
    newLISP in the real world / Re: get-url of empty content fails
    July 19, 2015, 01:56:04 AM
    Quote from: "Lutz"If this happened on Windows or OS2, TRU64, Solaris or AIX, it may have been fixed:



    http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/



    There was an uninitialized buffer in a custom version of vasprintf() -> my_vasprintf() used by above OSs.

    No, error has been on Debian Linux.


    Quote from: "Lutz"
    Ps: doing some more changes affecting all OSs, the way varargs is used - will post those Sunday morning PST.

    This sounds like a bigger change with many chances for errors...



    When there is a newer devel version I'll test again on Monday or other working days and give feedback again (I don't have access to corresponding system here).
    #75
    newLISP in the real world / Re: get-url of empty content fails
    July 17, 2015, 12:12:02 PM
    Thanks for the upgrade.



    Unfortunately there is some problem:
    Code Select

    > ; "OK with "list":
    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list"))
    ("Date: Fri, 17 Jul 2015 18:37:10 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > ; -> OK
    > ;
    > ; but with "list debug":
    > (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
    POST /ViPLab HTTP/1.1
    Host: localhost
    User-Agent: newLISP v10604
    Connection: close
    Content-type: text/plain
    Content-length: 13

    dummy contentHTTP/1.1 404 Not Found

    ("Date: Fri, 17 Jul 2015 18:38:13 GMTrnServer: Apache/2.2.22 (Debian)rnContent-Length: 17rnConnection: closernContent-Type: text/plain; charset=utf-8rnrn"
     "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn" 404)
    > (print (t 1))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > ; -> failure: see the string artefact 'ian)'
    > ;
    > ; it can be repaired by:
    > (print (0 (t 1)))
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"

    This is very strange. Possibly the 'repair' by copying the string gives you an idea, where the problem may be...



    Digging deeper:
    Code Select

    > (set 'r1 (t 1))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (print r1)
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    ian)
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (set 'r2 (0 r))
    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (print r2)
    ERR: server code 404: HTTP/1.1 404 Not Found
    404: Not Found.

    "ERR: server code 404: HTTP/1.1 404 Not Foundrn404: Not Found.nn"
    > (list (length r1) (length r2))
    (63 63)
    > (map (lambda (ix) (= (r1 ix) (r2 ix))) (sequence 0 (-- (length r1))))
    (true true true true true true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true true true true true true
     true true true true true true true true true true true true true true true true true true true true true)

    At interpreter level strings seem to be OK...
    Pages 1 ... 3 4 5 6 7 ... 10