Text only | Text with Images

newLISP Fan Club

Forum => newLISP in the real world => Topic started by: psilwen on August 15, 2017, 05:26:07 AM

Title: Why is the behavior of "trim" function so strange?
Post by: psilwen on August 15, 2017, 05:26:07 AM
Code Select
newLISP v.10.7.1 32-bit on Windows IPv4/6 UTF-8 libffi, options: newlisp -h

> (setq str (dup "00" 10))
"00000000000000000000"
> (trim str)
""
> (setq str (dup "00" 31))
"000000000000000000000000000000000000000
00000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 32))
"000000000000000000000000000000000000000
0000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
Title: Re: Why is the behavior of "trim" function so strange?
Post by: varbanov on August 18, 2017, 06:33:52 AM
Hi,



I'm still on a previous version and there's no problem ...
Code Select

newLISP v.10.6.2 32-bit on Win32 IPv4/6 UTF-8 libffi, options: newlisp -h

> (setq str (dup "00" 32))
"0000000000000000000000000000000000000000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 52))
"00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
> (trim str)
""
>


Yours,

s.v.
Title: Re: Why is the behavior of "trim" function so strange?
Post by: CaveGuy on August 18, 2017, 07:01:54 AM
I am back from the future :) no problem there ?


Code Select
newLISP v.10.7.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "00" 52))
"00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
> (trim str)
""
>
Title: Re: Why is the behavior of "trim" function so strange?
Post by: psilwen on August 18, 2017, 11:03:24 AM
Code Select
newLISP v.10.7.3 32-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "00" 8))
"0000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 1))
"00"
> (trim str)
""
> (setq str (dup "00" 2))
"0000"
> (trim str)
""
> (setq str (dup "00" 3))
"000000"
> (trim str)
""
> (setq str (dup "00" 4))
"00000000"
> (trim str)
""
> (setq str (dup "00" 5))
"0000000000"
> (trim str)
""
> (setq str (dup "00" 6))
"000000000000"
> (trim str)
""
> (setq str (dup "00" 7))
"00000000000000"
> (trim str)
""
> (setq str (dup "00" 8))
"0000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 9))
"000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 10))
"00000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 11))
"0000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 12))
"000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 13))
"00000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 14))
"0000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 15))
"000000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 16))
"00000000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 17))
"0000000000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 18))
"000000000000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 19))
"00000000000000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 20))
"000000000000000000000000000000000000000
0"
> (trim str)
""
> (setq str (dup "00" 21))
"000000000000000000000000000000000000000
000"
> (trim str)
""
> (setq str (dup "00" 22))
"000000000000000000000000000000000000000
00000"
> (trim str)
""
> (setq str (dup "00" 23))
"000000000000000000000000000000000000000
0000000"
> (trim str)
""
> (setq str (dup "00" 24))
"000000000000000000000000000000000000000
000000000"
> (trim str)
""
> (setq str (dup "00" 25))
"000000000000000000000000000000000000000
00000000000"
> (trim str)
""
> (setq str (dup "00" 26))
"000000000000000000000000000000000000000
0000000000000"
> (trim str)
""
> (setq str (dup "00" 27))
"000000000000000000000000000000000000000
000000000000000"
> (trim str)
""
> (setq str (dup "00" 28))
"000000000000000000000000000000000000000
00000000000000000"
> (trim str)
""
> (setq str (dup "00" 29))
"000000000000000000000000000000000000000
0000000000000000000"
> (trim str)
""
> (setq str (dup "00" 30))
"000000000000000000000000000000000000000
000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 31))
"000000000000000000000000000000000000000
00000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 32))
"000000000000000000000000000000000000000
0000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 33))
"000000000000000000000000000000000000000
000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim





Code Select
newLISP v.10.7.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "00" 8))
"0000000000000000"
> (trim str)
""
> (setq str (dup "00" 1))
"00"
> (trim str)
""
> (setq str (dup "00" 2))
"0000"
> (trim str)
""
> (setq str (dup "00" 3))
"000000"
> (trim str)
""
> (setq str (dup "00" 4))
"00000000"
> (trim str)
""
> (setq str (dup "00" 5))
"0000000000"
> (trim str)
""
> (setq str (dup "00" 6))
"000000000000"
> (trim str)
""
> (setq str (dup "00" 7))
"00000000000000"
> (trim str)
""
> (setq str (dup "00" 8))
"0000000000000000"
> (trim str)
""
> (setq str (dup "00" 9))
"000000000000000000"
> (trim str)
""
> (setq str (dup "00" 10))
"00000000000000000000"
> (trim str)
""
> (setq str (dup "00" 11))
"0000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 12))
"000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 13))
"00000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 14))
"0000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 15))
"000000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 16))
"00000000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 17))
"0000000000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 18))
"000000000000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 19))
"00000000000000000000000000000000000000"
> (trim str)
""
> (setq str (dup "00" 20))
"000000000000000000000000000000000000000
0"
> (trim str)
""
> (setq str (dup "00" 21))
"000000000000000000000000000000000000000
000"
> (trim str)
""
> (setq str (dup "00" 22))
"000000000000000000000000000000000000000
00000"
> (trim str)
""
> (setq str (dup "00" 23))
"000000000000000000000000000000000000000
0000000"
> (trim str)
""
> (setq str (dup "00" 24))
"000000000000000000000000000000000000000
000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 25))
"000000000000000000000000000000000000000
00000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 26))
"000000000000000000000000000000000000000
0000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 27))
"000000000000000000000000000000000000000
000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 28))
"000000000000000000000000000000000000000
00000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 29))
"000000000000000000000000000000000000000
0000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 30))
"000000000000000000000000000000000000000
000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 31))
"000000000000000000000000000000000000000
00000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 32))
"000000000000000000000000000000000000000
0000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 33))
"000000000000000000000000000000000000000
000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 34))
"000000000000000000000000000000000000000
00000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 35))
"000000000000000000000000000000000000000
0000000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 36))
"000000000000000000000000000000000000000
000000000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 37))
"000000000000000000000000000000000000000
00000000000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 38))
"000000000000000000000000000000000000000
0000000000000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 39))
"000000000000000000000000000000000000000
000000000000000000000000000000000000000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "00" 40))
"000000000000000000000000000000000000000
0000000000000000000000000000000000000000
0"
> (trim str)
""
> (setq str (dup "00" 41))
"000000000000000000000000000000000000000
0000000000000000000000000000000000000000
000"
> (trim str)
""
> (setq str (dup "00" 42))
"000000000000000000000000000000000000000
0000000000000000000000000000000000000000
00000"
> (trim str)
""
Title: Re: Why is the behavior of "trim" function so strange?
Post by: psilwen on August 18, 2017, 11:09:01 AM
newLISP v.10.7.3

newLISP v.10.7.1

newLISP v.10.7.0

All have this issues.



Usually starting from 8, 16, 24, 32...., depending on the version.



But newLISP v.10.6.2，I tested from 1 to 65536, everything was fine.
Title: Re: Why is the behavior of "trim" function so strange?
Post by: HPW on August 18, 2017, 11:40:14 PM
Hello,



trim seems not to like the null-character.

So is it valid whitespace character?


Code Select

newLISP v.10.7.2 32-bit on Windows IPv4/6 libffi, options: newlisp -h
> (setq str "t00   t")
"t00   t"
> (trim str)

ERR: not enough memory in function trim


Code Select

> (setq str "tabc00   abct")
"tabc00   abct"
> (trim str)
"abc00   abc"
>

Regards
Title: Re: Why is the behavior of "trim" function so strange?
Post by: Lutz on August 19, 2017, 09:42:25 AM
'trim' was overrunning memory when all characters in string < 32 (space).



Fixed here:

http://newlisp.nfshost.com/downloads/development/inprogress/
Title: Re: Why is the behavior of "trim" function so strange?
Post by: HPW on August 19, 2017, 10:41:27 PM
Hello Lutz,



Thanks for the fix.

Will characters <32 now handled as whitespace and get trimmed?

Or do they stay?



Regards
Title: Re: Why is the behavior of "trim" function so strange?
Post by: Lutz on August 21, 2017, 01:01:19 PM
See last paragraph here:

http://www.newlisp.org/downloads/development/inprogress/CHANGES-10.7.3.txt



So the 10.7.3 now is more forgiving when a string contains zero characters, which it should'nt in perfect ASCII or UTF-8 where the zero character marks the end of the string.
Title: Re: Why is the behavior of "trim" function so strange?
Post by: psilwen on September 12, 2017, 10:55:10 PM
Title: Re: Why is the behavior of "trim" function so strange?
Post by: TedWalther on September 13, 2017, 01:20:36 AM
It occurs to me, if I was trimming a string, and it had 00 characters on either the left or the right side, I'd want them GONE.  If it is a binary string where the 00 character belongs, then I wouldn't be running trim on it, since trim is a textual function.
Title: Re: Why is the behavior of "trim" function so strange?
Post by: rrq on September 13, 2017, 02:24:32 AM
Technically, since neither "xbbxe1" nor "xcexaaxcaxb2xc3xb4xbbxe1" are valid UTF-8 strings, the trim behaviour is conveniently undefined. It appears the trim function expands "xbb" into a 2-byte code, and "xe1" into a three byte code, making its output valid UTF-8. ("xcexaa", "xcaxb2" and "xc3xb4" are valid UTF-8 characters).
Text only | Text with Images