Print Page - "save" makes improper [text] strings; impossible to reload

Title: "save" makes improper [text] strings; impossible to reload
Post by: TedWalther on August 19, 2015, 05:29:26 PM

I stored some text in a context. That text contained a lot of cut and pasted newlisp code.

I did (save "myfile.lsp" MySnippets)

It saved it all.

Then I do (load "myfile.lsp") Uh-oh. Error.

Code Select Expand

ERR: symbol expected : [text] blahblahblah

There is no line number, and it is a big file. grep doesn't work. Pop open vi, and see the problem. There were embedded text tags in the data.

Simple way to reproduce problem:

Make a file with more than 2048 bytes in it. I just did this:

Code Select Expand

$ vi foo.txt
type this: i[/text]<ESC>2050Im<ESC>:wq
$ newlisp
(set 'foo (read-file "foo.txt"))
 => works correctly
(save "foo.lsp" 'foo)
=> it doesn't complain...
(load "foo.lsp")
ERR: symbol expected in function set : [/text]

Lutz, would it make sense to have "" be the default representation for strings, and let the string know internally if it is 2048 bytes or not?

Update

What I mean by that, does the programmer really need to know which internal string implementation is being used? String representation is for programmers to use. Doesn't the interpreter have enough information to switch between a 2048 buffer, and an arbitrary sized string?

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: TedWalther on August 19, 2015, 06:05:57 PM

As a temporary workaround, before I ever "save" a string, I do (replace {]} my_string {]}). And then after saving and loading it, I reverse it with (replace {]} my_string {]})

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: Lutz on August 20, 2015, 06:40:03 AM

The distinction of shorter or bigger-equal than 2048 is done for speed. The internal string representation is the same. What differs is the method used for reading strings from source. For strings shorter than 2048 limited by "" or {}, allocated buffers or the stack and one read() is used. On output newLISP looks at the length than chooses the "" versus [text][/tags]. For longer strings stream reading is used. The short kind, the most used in program source, can also be handled fast on the C stack, while the long cannot.

But the three sets of delimiters also offer the programmer different ways of encoding strings. E.g. for web work [text][/text] are very convenient because they take text as is including line feeds. {} are useful for regular expression for avoiding double backslashing.

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: TedWalther on August 21, 2015, 12:35:55 PM

Lutz, can you change things so that "save" function works? Is save much good if you can't "load" the output?

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: Lutz on August 23, 2015, 04:18:07 PM

see here: http://www.newlisp.org/downloads/development/inprogress/

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 04, 2015, 03:15:47 PM

But this does not work for unprintable chars.

This is OK:

Code Select Expand

> (set 's "this is 00 a test")
"this is 00 a test"

; but here 00 has silently gone away:

Code Select Expand

> (set 'sb (dup s 150))
[text]this is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a testthis is  a test[/text]
>

Any ideas?

It would be nice to have a robust way to store/load strings whatever they contain.

Or alternatively: if strings are the wrong data type for storing unprintable chars, this could be an error case (but for shorter ones this is allowed, so the switch between "" and [text][/text] leads to different behavior here).

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: TedWalther on September 04, 2015, 04:04:26 PM

I was thinking, from point of view of correctness, easiest just to have "" for strings. Go language has `` for unescaped strings. {} strings are convenient, but can sometimes bite that they can only contain balanced parenthesis. I think overall, if "" worked for any size, I'd rarely use the other kinds, especially if "" strings allow literal characters inside, so I can use them the same was as [text] and {} for multiline strings.

It is a huge "gotcha" to have a string type (two string types) that work only on some content, but don't have any escape mechanism to include all content. And then the one type of string that does allow all content, is limited to 2048 bytes. Aaarrghghh!

Lutz, how about this; when doing "save", if the string is longer than 2048 bytes, instead of converting it to [text], convert it to (list num num num) where each num is a byte, value in the range 0..255 Or else (string str1 str2...) where str1 is a 2048 byte string in "" representation, as is str2, up until all the content is represented.

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: Lutz on September 04, 2015, 08:00:54 PM

In Hartrock's example the 00 characters are still in sb but text in [text],[/text] tags does not escape characters, so 00 is not shown, although part of sb.

Right now in 10.6.4 (in progress) base64 encoding is used for strings > 2047 characters and containing the [/text] tag when using save. I could drop the condition for [/text] and always use base64 for strings longer 2047 in the save function. That would make Hartrock's sb string usable for the save function.

But I don't want to give up the [text],[/tags] to use completely unescaped text (not binary content). This is frequently used in web programming. I also don't want to eliminate the 2047 limit for " quoted string for speed in processing.

The need to display code > 2047 and containing non-displayable binary info is very rare. The save now will work with binary contents too if it always uses base64 transformation on strings > 2047 characters.

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: TedWalther on September 06, 2015, 06:00:59 PM

The base64 solution bothers me, because it breaks the human readability of the (save) output. Can you share a bit more info about the speedups you achieved with the three different string types? How about a 4th type of string, L" for "" strings longer than 2048 bytes?

update

Or instead of L"my string", do it Python style: """my string"""

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 08, 2015, 04:35:05 AM

After doing some other newLISP related stuff back to this interesting discussion.

~~Quote from: "Lutz"~~In Hartrock's example the 00 characters are still in sb but text in [text],[/text] tags does not escape characters, so 00 is not shown, although part of sb.

This is OK for using these strings inside newLISP; but it makes problems, if transferring their [text]...[/text] representation, because this omits part of the string (information loss).

There is such a use case: I'm in the process of writing a newLISP 'Inspector' server app for inspecting all newLISP symbols by a browser (planned to publish it in a while). Therefore all strings will be transferred from server to browser (via JSON) by using their :source representation.

Currently there is (using newlisp-10.6.4.tgz 2015-08-31):

Code Select Expand

[unprintable chars] "bar00buz"

- OK, no information loss -, and

Code Select Expand

[unprintable chars] [text]this s a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is a testthis is ...

- not optimal, because some parts of the string are missing -

as part of info in browser window.

~~Quote from: "Lutz"~~Right now in 10.6.4 (in progress) base64 encoding is used for strings > 2047 characters and containing the [/text] tag when using save. I could drop the condition for [/text] and always use base64 for strings longer 2047 in the save function. That would make Hartrock's sb string usable for the save function.

~~Quote from: "TedWalther"~~The base64 solution bothers me, because it breaks the human readability of the (save) output.

Inspecting a newLISP system for development/debugging purposes needs two things:

accuracy, and

readability.

For serving 1. the internal representation has to be shown, which even for short strings is not optimal regarding 2.; e.g.

Code Select Expand

> 
[text]
One line...
second line...
third line...
[/text]

"nOne line...nsecond line...nthird line...n"

][/text] representation is more readable than its internal "..." representation (for the 'Inspector' app I'm preferring accuracy, if in doubt).

~~Quote from: "Lutz"~~
But I don't want to give up the [text],[/tags] to use completely unescaped text (not binary content). This is frequently used in web programming.

They are also very nice for showing loaded text files as symbol evaluations (*.html, *.txt, *.js, etc.).

~~Quote from: "Lutz"~~
I also don't want to eliminate the 2047 limit for " quoted string for speed in processing.

~~Quote from: "Lutz"~~
The need to display code > 2047 and containing non-displayable binary info is very rare.

For debugging purposes there is such a need.

~~Quote from: "Lutz"~~ The save now will work with binary contents too if it always uses base64 transformation on strings > 2047 characters.

This is bad for readability of - longer - text files mentioned above; just tried to encode //https://www.base64encode.org/:

Code Select Expand

Hello World!

; resulting in Base64 format (UTF-8):

Code Select Expand

SGVsbG8gV29ybGQh

Note: result longer as source.

Idea:

What about switching to base64 transformation (or another accurate variant) only then, if there are unprintable chars?

This would give readability in most cases, but would also provide accuracy in the rarer ones, too.

~~Quote from: "TedWalther"~~
Lutz, how about this; when doing "save", if the string is longer than 2048 bytes, instead of converting it to [text], convert it to (list num num num) where each num is a byte, value in the range 0..255 Or else (string str1 str2...) where str1 is a 2048 byte string in "" representation, as is str2, up until all the content is represented.

I like the simplicity and readability (compared with base64 encoding) of this approach.

Idea:

To combine the best of all variants:
~~[list=1]~~

"..." as now for short strings.

[text]...[/text] for longer strings not containing unprintable chars.

(string str1 str2 ...) encoding for longer strings containing a small amount of unprintable chars.

base64 encoding for longer strings containing a significant amount of unprintable chars (binary data in strings would not be very readable in other representations, too).

But 4. only then - due to its unreadability (checking values of single bytes not possible) -, if it gives real improvements regarding mem or speed or ? (possibly I'm missing an important point here) compared with 3. (when is this the case?).

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: Lutz on September 08, 2015, 07:58:32 AM

save strings longer than 2047 will now be broken up in portions of up to 72 characters delimited by normal quotes "..." and escaped for unprintable characters.

For example{

Code Select Expand

(set 'str (dup "Part of 00 a long string with more than 2047 chars. " 45))

produces using save:

Code Select Expand

(set 'str (append   
"Part of 00 a long string with more than 2047 chars. Part of 00 a long stri"    
"ng with more than 2047 chars. Part of 00 a long string with more than 204"  
...
"tring with more than 2047 chars. Part of 00 a long string with more than "    
"2047 chars. Part of 00 a long string with more than 2047 chars. ")

The internal representation of a string is always the same: address and length. The internal I/O routines decide the format by looking at the length field of the string and the device used.

http://www.newlisp.org/downloads/development/inprogress/

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: TedWalther on September 08, 2015, 11:30:49 AM

Thank you Lutz. I woke up from a dream this morning with another solution, but then saw you had implemented this one. I'm very glad and appreciate the one you implemented. Here was the one I saw in the dream:

Code Select Expand

(length str_one) => 5000
str_one => [text 5000].....[/text]

Also, funny enough, I came across this problem in the exact same way as hartrock; transmitting newlisp code across the internet embedded in JSON.

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 08, 2015, 03:01:42 PM

~~Quote from: "Lutz"~~save strings longer than 2047 will now be broken up in portions of up to 72 characters delimited by normal quotes "..." and escaped for unprintable characters.

I like this solution: accurate and even more readable as longer "..." chunks.

~~Quote from: "Lutz"~~
The internal representation of a string is always the same: address and length. The internal I/O routines decide the format by looking at the length field of the string and the device used.

So there will be output of longer strings in [text]...[/text] format printed by the newLISP interpreter (please correct me, if I'm wrong).

For having nice output of longer strings (e.g. from reading text files) in the browser, I like the [text]...[/text] variant for strings only containing 'normal' text chars; e.g. currently there is something like this in the browser window of 'Inspector' app:

Code Select Expand

WS_T_Content: _jqtree.css  ->
[text]ul.jqtree-tree {
  list-style: none outside;
  margin-left: 0;
  margin-bottom: 0;
  padding: 0; }
  ul.jqtree-tree ul.jqtree_common {
    list-style: none outside;
    margin-left: 12px;
    margin-right: 0;
    margin-bottom: 0;
    padding: 0;
    display: block; }
  ul.jqtree-tree li.jqtree-closed > ul.jqtree_common {
    display: none; }
  ul.jqtree-tree li.jqtree_common {
    clear: both;
    list-style-type: none; }
  ul.jqtree-tree .jqtree-toggler {
    border-bottom: none;
    color: #333;
    text-decoration: none;
    margin-right: 0.5em;
    vertical-align: middle;
    /* [sr] Solves the problem with toggler char occupying less than 1em, by
     * enforcing to use exactly this space. */
    float: left; width: 1em; }
    ul.jqtree-tree .jqtree-toggler:hover {
      color: #000;
      text-decoration: none; }
    ul.jqtree-tree .jqtree-toggler.jqtree-closed {
      background-position: 0 0; }
  ul.jqtree-tree .jqtree-element {
    cursor: pointer;
    position: relative; }
  ul.jqtree-tree...

Is it safe to assume, that all chars not represented as nnn sequences in "..." formats are printable chars suited for [text]...[/text] input/output?

Then I would have a criterium for writing a function for converting from chunked "..." into [text]...[/text] format for displaying a 'friendly' string nicely in browser: of course [text],[/text] tags inside string and probably b should be treated special (and possibly enforcing chunked "..." output to have no information loss).

Important is, to keep a correct (no information loss) "..." chunk input/output representation for 'nasty' strings.

I don't know how robust this approach would be - e.g. regarding invalid UTF-8 sequences (//https://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences) -, what it really should be to get no surprises for 'nasty' strings; so any hints could be helpful here (string representations are a huge field...).

Perfect would be to have a function like e.g. valid-UTF-8? or text? (the latter checking for valid UTF-8 and not having embedded [text],[/text] tags and possibly not having b inside (you get the idea): the latter would be ideal for my usecase, but the former would be more flexible for being used in other ones, too...

PS: Possibly best for an 'Inspector' app targeted to developers is to show internal "" rep as default (accuracy), with the possibility to switch to a user friendly [text] view (which may have information loss then).

Title: two more ideas [Re: "save" makes improper [text] strings; im
Post by: hartrock on September 09, 2015, 07:02:47 AM

Two more ideas (after sleeping).

extend seems to be faster]
> (time (append "foo" "bar" "buz") 1000)
0.733
> (time (extend "foo" "bar" "buz") 1000)
0.427
[/code]

~~Quote from: "hartrock"~~
~~Quote from: "Lutz"~~save strings longer than 2047 will now be broken up in portions of up to 72 characters delimited by normal quotes "..." and escaped for unprintable characters.

I like this solution: accurate and even more readable as longer "..." chunks.

This is not necessarily true.

But there is another variant:

Code Select Expand

(extend "First line...ntsecond line with tab...n  third line with indent...n")

could also be:

Code Select Expand

(extend
"First line...n"
"tsecond line with tab...n"
"  third line with indent...n"
)

This would preserve WS formatting to some extent!

Strings with very many n's inside (each converted to 5 instead of 2 chars in output) should be very rare, so I think this is no problem here.

As upper line length limit - if there is no n in sight - 2047 could be used...

For inspecting a newLISP system this would be a nice representation avoiding all [text]...[/text] issues: smaller strings up to 2047 chars could be shown optionally in this format, too.

BTW:

Recently I have checked exporting all syms and their values in JSON format with some JSON WS for user friendly formatting - indents, NLs - or without it for saving space: the saving was only about 7% for switching from e.g.

Code Select Expand

{
"Class:Class": {
  "type": "sym",
  "prefix": "Class", "term": "Class",
  "global?": false, "protected?": false,
  "val": {
    "type": "lambda",
    "val": "(lambda () (cons (context) (args)))"
  }
},

to (one line):

Code Select Expand

{"Class:Class":{"type":"sym","prefix":"Class","term":"Class","global?":false,"protected?":false,"val":{"type":"lambda","val":"(lambda () (cons (context) (args)))"}},

[/list]

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: Lutz on September 09, 2015, 01:22:42 PM

After doing a few more benchmarks, I stay with append, which is faster when appending more than a few strings. In the save case you have at least about 30 strings of about 72 characters each. extend does a realloc() on each string, while append allocates memory in bigger chunks.

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 09, 2015, 07:19:28 PM

~~Quote from: "Lutz"~~After doing a few more benchmarks, I stay with append, which is faster when appending more than a few strings. In the save case you have at least about 30 strings of about 72 characters each. extend does a realloc() on each string, while append allocates memory in bigger chunks.

One thought (without looking into interpreter source): extend could reallocate similar as append allocates...

And some questions:

Is append the preferred variant, if special properties of extent like being able to change contents of a literal string inside a lambda are not needed?

Until now I've thought, that extent could be faster for appending one chunk after each other at the end of an empty start string (e.g. for JSON generation); now I think using append wherever possible could be faster.

Is it generally - if special features of extent are not needed - in streaming cases (after initing by (set 'res "")) better to do:

Code Select Expand

(set 'res (append res arg)) ; many times

instead of:

Code Select Expand

(extend res arg) ; many times

Title: [backtrace] newlisp-10.6.4.tgz 2015-09-09 20:27
Post by: hartrock on September 10, 2015, 06:04:43 AM

~~Quote from: "Lutz"~~save strings longer than 2047 will now be broken up in portions of up to 72 characters delimited by normal quotes "..." and escaped for unprintable characters.

~~Quote from: "Lutz"~~http://www.newlisp.org/downloads/development/inprogress/

I've gotten a backtrace for

newlisp-10.6.4.tgz 2015-09-09 20:27

:

Code Select Expand

sr@free:~/newLISP/Inspector$ ~/newLISP_Git/mirror/newlisp inspector.lsp
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/modules/getopts_patched_Ted.lsp" to be loaded.
[log][inspector.lsp Libs:load-lib][Info] load lib for context WS, depth: 1
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/lib/WS.lsp" to be loaded.
[log][inspector.lsp Libs:load-lib][Info] load lib for context assert, depth: 2
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/lib/assert.lsp" to be loaded.
[dbg][inspector.lsp] (assert.lsp...
[log][inspector.lsp Libs:load-lib][Info] load lib for context navi, depth: 3
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/lib/navi.lsp" to be loaded.
[dbg][inspector.lsp]   (navi.lsp...
[log][inspector.lsp Libs:load-lib][Info] load lib for context assert, depth: 4
[log][inspector.lsp Libs:load-lib][Info] lib 'assert' already seen: skipped.
[log][inspector.lsp Libs:load-lib][Info] load lib for context util, depth: 4
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/lib/util.lsp" to be loaded.
[dbg][inspector.lsp]     (util.lsp...
[dbg][inspector.lsp]     ...util.lsp)
[dbg][inspector.lsp]   ...navi.lsp)
[log][inspector.lsp Libs:load-lib][Info] load lib for context algo, depth: 3
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/lib/algo.lsp" to be loaded.
[dbg][inspector.lsp]   (algo.lsp...
[dbg][inspector.lsp]   ...algo.lsp)
[log][inspector.lsp Libs:load-lib][Info] load lib for context util, depth: 3
[log][inspector.lsp Libs:load-lib][Info] lib 'util' already seen: skipped.
[log][inspector.lsp Libs:load-lib][Info] load lib for context tweak, depth: 3
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/lib/tweak.lsp" to be loaded.
[dbg][inspector.lsp]   (tweak.lsp...
[log][inspector.lsp Libs:load-lib][Info] load lib for context assert, depth: 4
[log][inspector.lsp Libs:load-lib][Info] lib 'assert' already seen: skipped.
[dbg][inspector.lsp]   ...tweak.lsp)
[dbg][inspector.lsp] ...assert.lsp)
[log][inspector.lsp Libs:load-lib][Info] load lib for context FM, depth: 2
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/lib/FM.lsp" to be loaded.
[dbg][inspector.lsp] (FM.lsp...
[log][inspector.lsp Libs:load-lib][Info] load lib for context assert, depth: 3
[log][inspector.lsp Libs:load-lib][Info] lib 'assert' already seen: skipped.
[log][inspector.lsp Libs:module-opt-overwrite][Info] "/home/sr/newLISP/modules/FOOPReference.lsp" to be loaded.
[dbg][inspector.lsp] ...FM.lsp)
[log][inspector.lsp Libs:load-libs][Info] Trying to initialize lib FM...
[dbg][inspector.lsp] (FM:initialize...
[dbg][inspector.lsp]   ((context FM, symsFromCtx assert)...
[dbg][inspector.lsp tweak:context]  4 (assert) calls modified in: FM
[dbg][inspector.lsp]   ...(context FM, symsFromCtx assert))
[dbg][inspector.lsp]   ((context FM, symsFromCtx dbg)...
[dbg][inspector.lsp]   ...(context FM, symsFromCtx dbg))
[dbg][inspector.lsp] ...FM:initialize)
[log][inspector.lsp Libs:load-libs][Info] ...FM:initialize done.
[log][inspector.lsp Libs:load-libs][Info] Trying to initialize lib tweak...
[dbg][inspector.lsp] (tweak:initialize...
[dbg][inspector.lsp tweak:initialize] nothing to tweak
[dbg][inspector.lsp] ...tweak:initialize)
[log][inspector.lsp Libs:load-libs][Info] ...tweak:initialize done.
[log][inspector.lsp Libs:load-libs][Info] Trying to initialize lib algo...
[log][inspector.lsp Libs:load-libs][Info] ...no algo:initialize found (OK).
[log][inspector.lsp Libs:load-libs][Info] Trying to initialize lib util...
[log][inspector.lsp Libs:load-libs][Info] ...no util:initialize found (OK).
[log][inspector.lsp Libs:load-libs][Info] Trying to initialize lib navi...
[dbg][inspector.lsp] (navi:initialize...
[dbg][inspector.lsp]   ((context navi, symsFromCtx assert)...
[dbg][inspector.lsp tweak:context] 14 (assert:pre) calls modified in: navi
[dbg][inspector.lsp]   ...(context navi, symsFromCtx assert))
[dbg][inspector.lsp]   ((context navi, symsFromCtx dbg)...
[dbg][inspector.lsp tweak:context]  2 (dbg:expr) calls modified in: navi
[dbg][inspector.lsp]   ...(context navi, symsFromCtx dbg))
[dbg][inspector.lsp] ...navi:initialize)
[log][inspector.lsp Libs:load-libs][Info] ...navi:initialize done.
[log][inspector.lsp Libs:load-libs][Info] Trying to initialize lib assert...
[dbg][inspector.lsp] (assert:initialize...
[dbg][inspector.lsp]   ((context assert, symsFromCtx assert)...
[dbg][inspector.lsp tweak:context]  1 (assert:pre) call  modified in: assert
[dbg][inspector.lsp]   ...(context assert, symsFromCtx assert))
[dbg][inspector.lsp]   ((context assert, symsFromCtx dbg)...
[dbg][inspector.lsp]   ...(context assert, symsFromCtx dbg))
[dbg][inspector.lsp] ...assert:initialize)
[log][inspector.lsp Libs:load-libs][Info] ...assert:initialize done.
[log][inspector.lsp Libs:load-libs][Info] Trying to initialize lib WS...
[log][inspector.lsp Libs:load-libs][Info] ...no WS:initialize found (OK).
[dbg][inspector.lsp] (append Inspector:dir "/" Inspector:file) -> "/home/sr/newLISP/Inspector/symbols.html"
[dbg][inspector.lsp] (append Inspector:dir "/" Inspector:file) -> "/home/sr/newLISP/Inspector/jquery-2.1.4.js"
[dbg][inspector.lsp] (append Inspector:dir "/" Inspector:file) -> "/home/sr/newLISP/Inspector/tree.jquery.js"
[dbg][inspector.lsp] (append Inspector:dir "/" Inspector:file) -> "/home/sr/newLISP/Inspector/egBase.js"
[dbg][inspector.lsp] (append Inspector:dir "/" Inspector:file) -> "/home/sr/newLISP/Inspector/symbols.js"
[dbg][inspector.lsp] (append Inspector:dir "/" Inspector:file) -> "/home/sr/newLISP/Inspector/inspector.css"
[dbg][inspector.lsp] (append Inspector:dir "/" Inspector:file) -> "/home/sr/newLISP/Inspector/jqtree.css"
[log][inspector.lsp][Info] {
  "type": "string",
  "length": 3,
  "rep": ""foo""
}
[log][inspector.lsp][Info] {
  "type": "string",
  "length": 3,
  "rep": ""foo""
}
[dbg][inspector.lsp] (Inspector:json-str logg) -> "{
  "type": "context",
  "term": "Logger_1",
  "default": true
}"
[dbg][inspector.lsp] (Inspector:json-expr logg) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "logg",
  "global?": true, "protected?": false,
  "val": {
    "type": "context",
    "term": "Logger_1",
    "default": true
  }
}"
[dbg][inspector.lsp] (Inspector:json-expr +) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "+",
  "global?": true, "protected?": true,
  "val": { "type": "primitive", "val": "+@41CEE0" }
}"
[dbg][inspector.lsp] (Inspector:json-expr 4711) -> "{ "type": "integer", "val": "4711" }"
[dbg][inspector.lsp] (Inspector:json-expr 'MAIN:foo) -> "{ "type": "quote", "val": "'MAIN:foo" }"
[dbg][inspector.lsp] (Inspector:json-str 'MAIN:foo) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "foo",
  "global?": false, "protected?": false,
  "val": {
    "type": "string",
    "length": 3,
    "rep": ""bar""
  }
}"
[dbg][inspector.lsp] (Inspector:json-str 'MAIN:bar) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "bar",
  "global?": false, "protected?": false,
  "val": { "type": "nil", "val": "nil" }
}"
[dbg][inspector.lsp] (Inspector:json-str (array 6 '(1 2 3))) -> "{
  "type": "array",
  "length": 6,
  "rep": "(1 2 3 1 2 3)"
}"
[dbg][inspector.lsp] (Inspector:json-str nil) -> "{ "type": "nil", "val": "nil" }"
[dbg][inspector.lsp] (Inspector:json-str 'nil) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "nil",
  "global?": true, "protected?": true,
  "val": { "type": "nil", "val": "nil" }
}"
[dbg][inspector.lsp] (Inspector:json-str default) -> "{ "type": "primitive", "val": "default@40D140" }"
[dbg][inspector.lsp] (Inspector:json-str 'default) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "default",
  "global?": true, "protected?": true,
  "val": { "type": "primitive", "val": "default@40D140" }
}"
[dbg][inspector.lsp] (Inspector:json-str logg) -> "{
  "type": "context",
  "term": "Logger_1",
  "default": true
}"
[dbg][inspector.lsp] (Inspector:json-str 'logg) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "logg",
  "global?": true, "protected?": false,
  "val": {
    "type": "context",
    "term": "Logger_1",
    "default": true
  }
}"
[dbg][inspector.lsp] (Inspector:json-str MAIN) -> "{
  "type": "context",
  "term": "MAIN",
  "default": true
}"
[dbg][inspector.lsp] (Inspector:json-str 'MAIN) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "MAIN",
  "global?": true, "protected?": true,
  "val": {
    "type": "context",
    "term": "MAIN",
    "default": true
  }
}"
[dbg][inspector.lsp] (Inspector:json-str true) -> "{ "type": "true", "val": "true" }"
[dbg][inspector.lsp] (Inspector:json-str 'true) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "true",
  "global?": true, "protected?": true,
  "val": { "type": "true", "val": "true" }
}"
[dbg][inspector.lsp] (Inspector:json-str "foo "bar" buz") -> "{
  "type": "string",
  "length": 13,
  "rep": ""foo \"bar\" buz""
}"
*** glibc detected *** /home/sr/newLISP_Git/mirror/newlisp: double free or corruption (fasttop): 0x0000000000a2ea20 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x75be6)[0x7fc1f50b9be6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fc1f50be98c]
/home/sr/newLISP_Git/mirror/newlisp[0x4082e1]
/home/sr/newLISP_Git/mirror/newlisp[0x412ecb]
/home/sr/newLISP_Git/mirror/newlisp[0x41315b]
/home/sr/newLISP_Git/mirror/newlisp[0x413226]
/home/sr/newLISP_Git/mirror/newlisp[0x40b765]
/home/sr/newLISP_Git/mirror/newlisp[0x410435]
/home/sr/newLISP_Git/mirror/newlisp[0x41062a]
/home/sr/newLISP_Git/mirror/newlisp[0x4136f2]
/home/sr/newLISP_Git/mirror/newlisp[0x40b765]
/home/sr/newLISP_Git/mirror/newlisp[0x410435]
/home/sr/newLISP_Git/mirror/newlisp[0x41062a]
/home/sr/newLISP_Git/mirror/newlisp[0x407482]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fc1f5062ead]
/home/sr/newLISP_Git/mirror/newlisp[0x407575]
======= Memory map: ========
00400000-00454000 r-xp 00000000 08:01 4671427                            /home/sr/newLISP_Git/mirror/newlisp
00654000-00658000 rw-p 00054000 08:01 4671427                            /home/sr/newLISP_Git/mirror/newlisp
00658000-0065b000 rw-p 00000000 00:00 0 
009f8000-00aa9000 rw-p 00000000 00:00 0                                  [heap]
7fc1f0000000-7fc1f0021000 rw-p 00000000 00:00 0 
7fc1f0021000-7fc1f4000000 ---p 00000000 00:00 0 
7fc1f47c3000-7fc1f47d8000 r-xp 00000000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc1f47d8000-7fc1f49d8000 ---p 00015000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc1f49d8000-7fc1f49d9000 rw-p 00015000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc1f49d9000-7fc1f4a5c000 rw-p 00000000 00:00 0 
7fc1f4a5c000-7fc1f4e1b000 r--p 00000000 08:01 3489798                    /usr/lib/locale/locale-archive
7fc1f4e1b000-7fc1f4e40000 r-xp 00000000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7fc1f4e40000-7fc1f503f000 ---p 00025000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7fc1f503f000-7fc1f5043000 r--p 00024000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7fc1f5043000-7fc1f5044000 rw-p 00028000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7fc1f5044000-7fc1f51c5000 r-xp 00000000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7fc1f51c5000-7fc1f53c5000 ---p 00181000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7fc1f53c5000-7fc1f53c9000 r--p 00181000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7fc1f53c9000-7fc1f53ca000 rw-p 00185000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7fc1f53ca000-7fc1f53cf000 rw-p 00000000 00:00 0 
7fc1f53cf000-7fc1f53db000 r-xp 00000000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7fc1f53db000-7fc1f55db000 ---p 0000c000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7fc1f55db000-7fc1f55dc000 rw-p 0000c000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7fc1f55dc000-7fc1f5619000 r-xp 00000000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7fc1f5619000-7fc1f5819000 ---p 0003d000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7fc1f5819000-7fc1f581b000 r--p 0003d000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7fc1f581b000-7fc1f5821000 rw-p 0003f000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7fc1f5821000-7fc1f5823000 rw-p 00000000 00:00 0 
7fc1f5823000-7fc1f5825000 r-xp 00000000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fc1f5825000-7fc1f5a25000 ---p 00002000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fc1f5a25000-7fc1f5a26000 r--p 00002000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fc1f5a26000-7fc1f5a27000 rw-p 00003000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fc1f5a27000-7fc1f5aa8000 r-xp 00000000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7fc1f5aa8000-7fc1f5ca7000 ---p 00081000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7fc1f5ca7000-7fc1f5ca8000 r--p 00080000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7fc1f5ca8000-7fc1f5ca9000 rw-p 00081000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7fc1f5ca9000-7fc1f5cc9000 r-xp 00000000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7fc1f5da8000-7fc1f5de5000 rw-p 00000000 00:00 0 
7fc1f5e5f000-7fc1f5ea6000 rw-p 00000000 00:00 0 
7fc1f5ec6000-7fc1f5ec8000 rw-p 00000000 00:00 0 
7fc1f5ec8000-7fc1f5ec9000 r--p 0001f000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7fc1f5ec9000-7fc1f5eca000 rw-p 00020000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7fc1f5eca000-7fc1f5ecb000 rw-p 00000000 00:00 0 
7ffeaab64000-7ffeaab85000 rw-p 00000000 00:00 0                          [stack]
7ffeaab94000-7ffeaab95000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted
sr@free:~/newLISP/Inspector$ uname -a
Linux free 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u3 x86_64 GNU/Linux

@Lutz:

What about an extra forum for errors in the inprogress version? Would be easy to track for you, and after a fix nobody cares.

Or?

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: Lutz on September 10, 2015, 06:53:34 AM

extend is destructive append is not. Many things in newLISP can be done in either a destructive or non-destructive manner and destructive functions are marked with an ! in the reference.

http://www.newlisp.org/downloads/newlisp_manual.html#destructive

Most destructive functions - like extend too - can also do in-place modification. See the last sub-chapter about this on this page: http://www.newlisp.org/index.cgi?page=Closures

Bench-marking is a tricky business you can find some benchmarks comparing to Perl and Python here: http://newlisp.nfshost.com/benchmarks/

Small changes how a function is used may cause big differences in benchmarks. Sometimes just moving between OSs or versions of compilers changes everything. That is, why I like the benchmarks in the link, because they always measure function performance in an application context.

ps: people report errors under all topics of the forum, and that has worked well in the passed times.

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 10, 2015, 08:55:23 AM

~~Quote from: "Lutz"~~
ps: people report errors under all topics of the forum, and that has worked well in the passed times.

Does this kind of backtrace help you, or do you need more or less info?

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 10, 2015, 09:13:50 AM

~~Quote from: "Lutz"~~extend is destructive append is not. Many things in newLISP can be done in either a destructive or non-destructive manner and destructive functions are marked with an ! in the reference.

http://www.newlisp.org/downloads/newlisp_manual.html#destructive

Most destructive functions - like extend too - can also do in-place modification. See the last sub-chapter about this on this page: http://www.newlisp.org/index.cgi?page=Closures

Bench-marking is a tricky business

...

Normally I don't like to think about performance: if there are only small differences in performance, I'd always prefer the more elegant - mostly the shorter - solution.

In this sense

Code Select Expand

(extend res arg) ; many times

surely is better than

Code Select Expand

(set 'res (append res arg)) ; many times

.

But sometimes some inner workings of a language implementation are bubbling up in form of surprises to the programmer not being aware of them: I'm remembering my trial to get away some asymmetries regarding push/pop, while ignoring the inner structure of lists (which is the reason for them)...

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: xytroxon on September 10, 2015, 04:51:18 PM

While being destructive, the extend function also has variable history to worry about! To use the extend function in the save output to avoid side effects when executing the load function, res MUST be explicitly RESET to nil or "the empty string" to prevent seemingly random "string errors" from occuring if the res variable had been set before. (Like in a third party module or from executing load multiple times from the REPL terminal.)

Code Select Expand

;... earlier executed code
(++ res) ;-> error [1]
; or
(set 'res '("John Doe" "1234 Main St" "Any Town, USA")) ;-> error [2]
; or
(set 'res "{string that remains from the first load}") ;-> error [3]

; ...when load is executed

(extend res arg1 arg2 arg3 ...)

; Code fails if res previously contained [1] a number, or "works" and extends either [2] a list, or [3] a string!

So the code generated by the save function would need to be:

Code Select Expand

(set 'res "") ; clear string
(extend res arg1 arg2 arg3 ...)

As per the norm, the Lutz version is optimal, (if not always beautiful) ;o)

-- xytroxon

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 10, 2015, 05:43:56 PM

~~Quote from: "xytroxon"~~
So the code generated by the save function would need to be:

Code Select Expand (set 'res "") ; clear string (extend res arg1 arg2 arg3 ...)

Sure.

Usecase in mind has been streaming: see //http://www.newlispfanclub.alh.net/forum/viewtopic.php?f=16&t=4743#p23412, where the init to an empty string at the beginning has been mentioned, too.

Possibly it had been better to repeat more accurate, without expecting knowledge about all of this very long thread.

Title: [backtrace] newlisp-10.6.4.tgz 2015-09-10 14:11
Post by: hartrock on September 10, 2015, 06:00:19 PM

Tested with changed inprogress version backtrace again:

Code Select Expand

[dbg][inspector.lsp] (Inspector:json-str 'true) -> "{
  "type": "sym",
  "prefix": "MAIN", "term": "true",
  "global?": true, "protected?": true,
  "val": { "type": "true", "val": "true" }
}"
[dbg][inspector.lsp] (Inspector:json-str "foo "bar" buz") -> "{
  "type": "string",
  "length": 13,
  "rep": ""foo \"bar\" buz""
}"
*** glibc detected *** /home/sr/newLISP_Git/mirror/newlisp: double free or corruption (fasttop): 0x0000000001f824e0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x75be6)[0x7f21d33c9be6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f21d33ce98c]
/home/sr/newLISP_Git/mirror/newlisp[0x4082e1]
/home/sr/newLISP_Git/mirror/newlisp[0x412ecb]
/home/sr/newLISP_Git/mirror/newlisp[0x41315b]
/home/sr/newLISP_Git/mirror/newlisp[0x413226]
/home/sr/newLISP_Git/mirror/newlisp[0x40b765]
/home/sr/newLISP_Git/mirror/newlisp[0x410435]
/home/sr/newLISP_Git/mirror/newlisp[0x41062a]
/home/sr/newLISP_Git/mirror/newlisp[0x4136f2]
/home/sr/newLISP_Git/mirror/newlisp[0x40b765]
/home/sr/newLISP_Git/mirror/newlisp[0x410435]
/home/sr/newLISP_Git/mirror/newlisp[0x41062a]
/home/sr/newLISP_Git/mirror/newlisp[0x407482]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f21d3372ead]
/home/sr/newLISP_Git/mirror/newlisp[0x407575]
======= Memory map: ========
00400000-00454000 r-xp 00000000 08:01 4671433                            /home/sr/newLISP_Git/mirror/newlisp
00654000-00658000 rw-p 00054000 08:01 4671433                            /home/sr/newLISP_Git/mirror/newlisp
00658000-0065b000 rw-p 00000000 00:00 0 
01f4c000-01ffd000 rw-p 00000000 00:00 0                                  [heap]
7f21cc000000-7f21cc021000 rw-p 00000000 00:00 0 
7f21cc021000-7f21d0000000 ---p 00000000 00:00 0 
7f21d2ad3000-7f21d2ae8000 r-xp 00000000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f21d2ae8000-7f21d2ce8000 ---p 00015000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f21d2ce8000-7f21d2ce9000 rw-p 00015000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f21d2ce9000-7f21d2d6c000 rw-p 00000000 00:00 0 
7f21d2d6c000-7f21d312b000 r--p 00000000 08:01 3489798                    /usr/lib/locale/locale-archive
7f21d312b000-7f21d3150000 r-xp 00000000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f21d3150000-7f21d334f000 ---p 00025000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f21d334f000-7f21d3353000 r--p 00024000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f21d3353000-7f21d3354000 rw-p 00028000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f21d3354000-7f21d34d5000 r-xp 00000000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7f21d34d5000-7f21d36d5000 ---p 00181000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7f21d36d5000-7f21d36d9000 r--p 00181000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7f21d36d9000-7f21d36da000 rw-p 00185000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7f21d36da000-7f21d36df000 rw-p 00000000 00:00 0 
7f21d36df000-7f21d36eb000 r-xp 00000000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7f21d36eb000-7f21d38eb000 ---p 0000c000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7f21d38eb000-7f21d38ec000 rw-p 0000c000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7f21d38ec000-7f21d3929000 r-xp 00000000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7f21d3929000-7f21d3b29000 ---p 0003d000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7f21d3b29000-7f21d3b2b000 r--p 0003d000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7f21d3b2b000-7f21d3b31000 rw-p 0003f000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7f21d3b31000-7f21d3b33000 rw-p 00000000 00:00 0 
7f21d3b33000-7f21d3b35000 r-xp 00000000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f21d3b35000-7f21d3d35000 ---p 00002000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f21d3d35000-7f21d3d36000 r--p 00002000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f21d3d36000-7f21d3d37000 rw-p 00003000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f21d3d37000-7f21d3db8000 r-xp 00000000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7f21d3db8000-7f21d3fb7000 ---p 00081000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7f21d3fb7000-7f21d3fb8000 r--p 00080000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7f21d3fb8000-7f21d3fb9000 rw-p 00081000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7f21d3fb9000-7f21d3fd9000 r-xp 00000000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7f21d40b8000-7f21d40f5000 rw-p 00000000 00:00 0 
7f21d416f000-7f21d41b6000 rw-p 00000000 00:00 0 
7f21d41d6000-7f21d41d8000 rw-p 00000000 00:00 0 
7f21d41d8000-7f21d41d9000 r--p 0001f000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7f21d41d9000-7f21d41da000 rw-p 00020000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7f21d41da000-7f21d41db000 rw-p 00000000 00:00 0 
7ffd68a28000-7ffd68a49000 rw-p 00000000 00:00 0                          [stack]
7ffd68ab1000-7ffd68ab2000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted
sr@free:~/newLISP/Inspector$

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: Lutz on September 11, 2015, 06:56:40 AM

Try to boil down the code to the least amount needed to produce the error. Back-tracing does not help here.

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 11, 2015, 07:38:32 AM

~~Quote from: "Lutz"~~Try to boil down the code to the least amount needed to produce the error. Back-tracing does not help here.

OK.

Problem seems to be related to strings:

Code Select Expand

sr@free:~/newLISP_Git/mirror$ ./newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 's "foo bar buz" 'r "")
""
> (time (extend r s) 200)
0.07000000000000001
> (source 'r) ; -> boom!
*** glibc detected *** ./newlisp: double free or corruption (fasttop): 0x000000000114bb60 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x75be6)[0x7fc73176bbe6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fc73177098c]
./newlisp[0x4082e1]
./newlisp[0x412ecb]
./newlisp[0x41315b]
./newlisp[0x4253dd]
./newlisp[0x40b765]
./newlisp[0x410435]
./newlisp[0x410aca]
./newlisp[0x406ede]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fc731714ead]
./newlisp[0x407575]
======= Memory map: ========
00400000-00454000 r-xp 00000000 08:01 4671433                            /home/sr/newLISP_Git/mirror/newlisp
00654000-00658000 rw-p 00054000 08:01 4671433                            /home/sr/newLISP_Git/mirror/newlisp
00658000-0065b000 rw-p 00000000 00:00 0 
010c8000-0115c000 rw-p 00000000 00:00 0                                  [heap]
7fc72c000000-7fc72c021000 rw-p 00000000 00:00 0 
7fc72c021000-7fc730000000 ---p 00000000 00:00 0 
7fc730e75000-7fc730e8a000 r-xp 00000000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc730e8a000-7fc73108a000 ---p 00015000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc73108a000-7fc73108b000 rw-p 00015000 08:01 1941506                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc73108b000-7fc7310cd000 rw-p 00000000 00:00 0 
7fc73110e000-7fc7314cd000 r--p 00000000 08:01 3489798                    /usr/lib/locale/locale-archive
7fc7314cd000-7fc7314f2000 r-xp 00000000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7fc7314f2000-7fc7316f1000 ---p 00025000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7fc7316f1000-7fc7316f5000 r--p 00024000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7fc7316f5000-7fc7316f6000 rw-p 00028000 08:01 4424152                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7fc7316f6000-7fc731877000 r-xp 00000000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7fc731877000-7fc731a77000 ---p 00181000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7fc731a77000-7fc731a7b000 r--p 00181000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7fc731a7b000-7fc731a7c000 rw-p 00185000 08:01 4423693                    /lib/x86_64-linux-gnu/libc-2.13.so
7fc731a7c000-7fc731a81000 rw-p 00000000 00:00 0 
7fc731a81000-7fc731a8d000 r-xp 00000000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7fc731a8d000-7fc731c8d000 ---p 0000c000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7fc731c8d000-7fc731c8e000 rw-p 0000c000 08:01 1974305                    /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
7fc731c8e000-7fc731ccb000 r-xp 00000000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7fc731ccb000-7fc731ecb000 ---p 0003d000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7fc731ecb000-7fc731ecd000 r--p 0003d000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7fc731ecd000-7fc731ed3000 rw-p 0003f000 08:01 1941596                    /lib/x86_64-linux-gnu/libreadline.so.6.2
7fc731ed3000-7fc731ed5000 rw-p 00000000 00:00 0 
7fc731ed5000-7fc731ed7000 r-xp 00000000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fc731ed7000-7fc7320d7000 ---p 00002000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fc7320d7000-7fc7320d8000 r--p 00002000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fc7320d8000-7fc7320d9000 rw-p 00003000 08:01 4423699                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fc7320d9000-7fc73215a000 r-xp 00000000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7fc73215a000-7fc732359000 ---p 00081000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7fc732359000-7fc73235a000 r--p 00080000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7fc73235a000-7fc73235b000 rw-p 00081000 08:01 4423686                    /lib/x86_64-linux-gnu/libm-2.13.so
7fc73235b000-7fc73237b000 r-xp 00000000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7fc732553000-7fc732558000 rw-p 00000000 00:00 0 
7fc732570000-7fc732577000 r--s 00000000 08:01 3834320                    /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7fc732577000-7fc73257a000 rw-p 00000000 00:00 0 
7fc73257a000-7fc73257b000 r--p 0001f000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7fc73257b000-7fc73257c000 rw-p 00020000 08:01 4423706                    /lib/x86_64-linux-gnu/ld-2.13.so
7fc73257c000-7fc73257d000 rw-p 00000000 00:00 0 
7ffd27859000-7ffd2787a000 rw-p 00000000 00:00 0                          [stack]
7ffd2793e000-7ffd2793f000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: Lutz on September 11, 2015, 09:21:25 AM

Thanks Hartrock, fixed here:

http://www.newlisp.org/downloads/development/inprogress/ 2015-09-11 16:19

Title: Re: "save" makes improper [text] strings; impossible to relo
Post by: hartrock on September 11, 2015, 09:46:00 AM

Thanks Lutz,

working for me.

newLISP Fan Club

Forum => newLISP in the real world => Topic started by: TedWalther on August 19, 2015, 05:29:26 PM