Print Page - development release version 9.0.14

Title: development release version 9.0.14
Post by: Lutz on January 04, 2007, 04:58:36 AM

* This is the first version which can be compiled for large pointer LP64 memory model. Many changes where necessary to make this happen. Please handle this version with care, many of lines code have been touched to make the code compilable in either 32-bit or 64-bit memory models.

* The manual has been updated to include the additions and changes made in the 9.0.x development series. The update of the summary index and frame index for the online version will come later today.

* This version is feature complete for 9.1, due on February 15th. A preview summary page of this version can be seen here: http://newlisp.org/index.cgi?page=Preview_91

Files and changes notes: http://newlisp.org/downloads/development/

Lutz

Title:
Post by: newdep on January 04, 2007, 07:03:38 AM

newlisp has the BEST compression ratio in the whole world..

This is no joke this is a fact!..Although its a fact on my XP Pro machine.

I downloaded newlisp-9014.exe 1.7 MB big..

Here is a screendump of "Installed software" and blow your mind ;-)

~~(//%3C/s%3E%3CURL%20url=%22http://www.nodep.nl/downloads/newlisp/wow.JPG%22%3Ehttp://www.nodep.nl/downloads/newlisp/wow.JPG%3C/URL%3E%3Ce%3E)~~

Title:
Post by: newdep on January 04, 2007, 10:24:24 AM

the linux make is not without any warings anymore...

...Its a minor cosmetic issue below...

$newlisp-9.0.14$ make linux_readline

make -f makefile_linux_readline

make[1]: Entering directory `/newlisp-9.0.14'

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX newlisp.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-symbol.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-math.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-list.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-liststr.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-string.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-filesys.c

nl-filesys.c: In function `p_parseDate':

nl-filesys.c:1580: warning: implicit declaration of function `strptime'

nl-filesys.c:1580: warning: comparison between pointer and integer

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-sock.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-import.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-xml.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-web.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-matrix.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX nl-debug.c

gcc -Wall -pedantic -Wno-uninitialized -Wno-strict-aliasing -Wno-long-long -c -O2 -g -DREADLINE -DLINUX pcre.c

gcc newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o nl-import.o nl-xml.o nl-web.o nl-matrix.o nl-debug.o pcre.o -g -lm -ldl -lreadline -lncurses -o newlisp

strip newlisp

make[1]: Leaving directory `/newlisp-9.0.14'

Title:
Post by: newdep on January 04, 2007, 10:39:41 AM

qa-utf8

Nice.. i never saw those befor... and because im Dutch

I need to correct the entry inside qa-utf8 ofcourse ;-)

Just some nagging ;-)

Its say's ->

(set 'Dutch "Ik kan glas eten, het doet m196179 geen kwaad.")

>Ik kan glas eten, het doet mÄ³ geen kwaad.

There are two issues:

(1) Dutch does not need utf8, but if you use it then

"mÄ³ " is incorrect, should be just an "i" and a "j" "ij"

(2) the spelling is wrong and its very odd Dutch ;-)

This is the correct translation from

German -> "Ich kann Glas essen, ohne mir weh zu tun."

German to Dutch -> "Ik kan glas eten zonder mij pijn te doen."

Title:
Post by: newdep on January 04, 2007, 11:09:17 AM

Hi Lutz,

Just currious but what is your vision on newLisp regarding future releases and

bit structures? Will newLisp always stay on 32bit no matter what or will it eventualy move towards full 64 bits? (I would stick with 32Bits for sure with

releases and do side releases for 64 bits, as it is now).

Aaaaa i almost forgot "BEST WISHES FOR THIS NEW YEAR !"

Regards, Norman.

Title:
Post by: Lutz on January 04, 2007, 12:52:26 PM

Binary releases will be 32-bit for a longer time until 64-bit OSs are completely mainstream. The source code package will offer both compile flavors 32-bit and 64-bit as it does with the current 9.0.14 for Linux. When 64-bit becomes available for other UNIX flavors it will be added, i.e. the BSDs.

For whatever machine I can get my hands on, I will offer 64-bit binaries of the executable for major releases.

The current Tru64 UNIX version is a 32-bit version running on a 64-bit OS. I am not sure if Pjot will try to offer a full 64-bit build too. If the Tru64 OS supports the LP64 memory model, it should be not too hard.

Currently most 64-bit Linux come with support for both 32-bit and 64-bit binaries. Mac OSX is a 64-bit OS but only comes with 32-bit libraries, and I guess will stay that way for some time.

At least for several years to come 32-bit will be the software application standard even if most hardware will move to 64-bit.

The codebase is now much more portable so it will be easier to adapt to new necessities as they arise.

Lutz

Ps: regarding the strptime() compiler warning you got: on all OSs I compiled, it was declared in /usr/include/time.h. Perhaps you need a newer installation of GCC?

Best wishes for 2007 for you too and everybody else!

Title:
Post by: newdep on January 04, 2007, 11:58:12 PM

Hi Lutz,

Im using gcc version 3.4.6

I have these in my time.h ->

# ifdef __USE_XOPEN

/* Parse S according to FORMAT and store binary time information in TP.

The return value is a pointer to the first unparsed character in S. */

extern char *strptime (__const char *__restrict __s,

__const char *__restrict __fmt, struct tm *__tp)

__THROW;

# endif

# ifdef __USE_GNU

/* Similar to the two functions above but take the information from

the provided locale and not the global locale. */

# include <xlocale>

extern size_t strftime_l (char *__restrict __s, size_t __maxsize,

__const char *__restrict __format,

__const struct tm *__restrict __tp,

__locale_t __loc) __THROW;

extern char *strptime_l (__const char *__restrict __s,

__const char *__restrict __fmt, struct tm *__tp,

__locale_t __loc) __THROW;

# endif

Norman.

Title:
Post by: pjot on January 05, 2007, 03:01:21 PM

This version also compiles fine on Tru64Unix, no errors at all.

However, the tests in 'qa-dot' cause a core dump:

~~Quote~~
[peter@olga]# ./qa-dot

Testing built-in functions ...

! != $ % & * + - / < << <= = > >= >> NaN? ^ abs acos add address amb and append Memory fault(coredump)

[peter@olga]#

I will look into it this weekend. Maybe I can remove all TRU64 macros and some compilerflags to have a full 64bit binary. That would be awesome! :-)

Peter

Title:
Post by: Lutz on January 05, 2007, 04:37:24 PM

Yes, try to compile this 9.0.14 version on Tru64 as a full 64-bit app. In your makefile add -DNEWLISP64, also remove all "#pragma pointer_size short" all other macros can probably stay or they are redundant with the NEWLISP64 define.

This is what the 64-bit compile as of LP64 memory model expects:

char -> 1 byte

int -> 4 byte

short int -> 2 byte

unsigned short int -> 4 byte

long -> 8 byte

unsigned long -> 8 byte (defined as UINT)

long int -> 8 byte

long long int -> 8 byte

double -> 8 byte

float -> 4 byte

double float -> 8 bytes

size_t -> 8 bytes -> unsigned long

ssize_t -> 8 bytes -> long

all pointer types -> 8 byte

all other types like "long double" are of no interest in newLISP currently. You can compile newlisp-x.x.x/util/type.c to find out. If I remember well from one of our conversations earlier, things are fine with your type.c output. So just add -DNEWLISP64 in newlisp.h or your compile flags and things should be fine.

Lutz

PS: -> Norman: I added for Linux: a strptime() declaration to be independent of the: #ifdef __USE_GNU which gives me problems in other compile flavors. That will eliminate the warning you got.

Title:
Post by: pjot on January 06, 2007, 02:36:24 AM

Well, good news BUT with minor hiccups.

1) First I removed all 'pointer_size' stuff. I still have to use -DTRU64 because of some workarounds for unknown functionnames.

2) Also I ran '/usr/lib/cmplrs/cc/protect_headers_setup.sh -d' again to make sure all 32-bit headers are gone.

3) Adjusted the makefile so it really compiles to 64 bit by adding '-DNEWLISP64' and by removing the '-xtaso' and '-taso' compilerflags.
~~Quote~~
OBJS = newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o

nl-sock.o nl-import.o nl-xml.o nl-web.o nl-matrix.o nl-debug.o pcre.o

# CFLAGS = -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -D_POSIX_PII_SOCKET

CFLAGS = -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET

CC = cc

default: $(OBJS)

$(CC) $(OBJS) -lm -lrt -ldb -lbsd -lreadline -ltermcap -o newlisp

strip newlisp

4) Now I run 'make'.
~~Quote~~
[peter@olga]# make

./build

Discovered Tru64 Unix, please read doc/TRU64BUILD

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET newlisp.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-symbol.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-math.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-list.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-liststr.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-string.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-filesys.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-sock.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-import.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-xml.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-web.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-matrix.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-debug.c

cc -I/usr/local/include -ieee -pedantic -c -O3 -DREADLINE -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET pcre.c

cc newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o nl-import.o nl-xml.o nl-web.o nl-matrix.o nl-debug.o pcre.o -lm -lrt -ldb -lbsd -lreadline -ltermcap -o newlisp

strip newlisp

5) Well, that looks good!! Compiling works on Tru64Unix 5.1B (I still have to test 4.0f). Now I run the testfile 'qa-dot', and there is no memory crash.
~~Quote~~

[peter@olga]# ./qa-dot

<snip>

TESTING: newlisp FINISHED WITH ERRORS:

>>>> flt failed nil

>>>> pipe failed string expected in function write-file : [text]

called from user defined function QA:qa

>>>> ~ failed problem in format string in function format : "%llx"

called from user defined function QA:qa

total time: 5042

newLISP v.9.0.14 64-bit on Tru64Unix, execute 'newlisp -h' for more info.

> ring...

> (exit)

So these are the minor hiccups... I thought I already let you know, but to me the overall result looks very promising. Finally, I ran 'types.c'.
~~Quote~~
[peter@olga]# cc -o types types.c

[peter@olga]# ./types

type bytes

---------------

char 1

char * 8

void * 8

short int 2

int 4

long 8

long int 8

long long 8

float 4

double 8

long double 16

wchar_t 4

format input output

------------------------------------------------

%d 0xffffffff -1

%u 0xffffffff 4294967295

%d 0x7fffffff 2147483647

%u 0x7fffffff 2147483647

%d 0x80000000 -2147483648

%u 0x80000000 2147483648

%d 0x7fffffffffffffffLL -1

%u 0x7fffffffffffffffLL 4294967295

%x 0xffffffffffffffffLL ffffffff

%X 0x7fffffffffffffffLL ffffffff

%X 0x8000000000000000LL 0

%llX 0xFFFFFFFFFFFFFFFFLL ffffffff

%llX 0x7FFFFFFFFFFFFFFFLL ffffffff

%llX 0x8000000000000000LL 0

%lld 0xffffffffffffffffLL -1

%llu 0xffffffffffffffffLL 18446744073709551615

%lld 0x7fffffffffffffffLL 9223372036854775807

%llu 0x7fffffffffffffffLL 9223372036854775807

%lld 0x8000000000000000LL -9223372036854775808

%llu 0x8000000000000000LL 9223372036854775808

Probably something is wrong because of the types. I'll continue to troubleshoot these few issues in 'qa-dot' this weekend.

Again it looks very promising!! With a little bit of luck we have a GENUINE 64-bit newlisp during the weekend!!

Peter

Title:
Post by: pjot on January 06, 2007, 02:59:51 AM

For the '(flt)', I think it is OK.
~~Quote~~
[peter@olga]# newlisp

newLISP v.9.0.14 64-bit on Tru64Unix, execute 'newlisp -h' for more info.

> (flt 1.23)

4394818826404

The test in 'qa-dot' expects 1067282596.

But the function (flt) converts an 8-byte float to a 4-byte float. On a 32-bit system, it converts a 4 byte float to a 2-byte float, and then the result must be 1067282596.

So my bet is that this test also fails on an AMD64.

Peter

Title:
Post by: pjot on January 06, 2007, 03:08:56 AM

The pipe test fails because the '[text]' symbols are not recognized. This is how it works on Linux:
~~Quote~~
peter[~]$ newlisp

newLISP v.9.0.14 on Linux, execute 'newlisp -h' for more info.

> (println [text]blabla[/text])

blabla

"blabla"

But this is how it looks on Tru64Unix now:
~~Quote~~
[peter@olga]# newlisp

newLISP v.9.0.14 64-bit on Tru64Unix, execute 'newlisp -h' for more info.

> (println [text]blabla[/text])

nilnil

nil

Anything changed in parsing these '[text] - [/text]' symbols?

Peter

Title:
Post by: Lutz on January 06, 2007, 06:26:09 AM

For speed I coded this for parsing the [text] tag in the function getToken() in newlisp.c, but nothing has changed here for a long time and Tru64 ate it before:

Code Select Expand else if(*(UINT*)token == *(UINT*)"[tex" && *(INT16*)(token +4) == *(INT16*)"t]")

it probably should be replaced with:

Code Select Expand else if(strncmp(token, "[text]", 6) == 0)

The speed impact overall on a program is probably very small. Still I wonder why the Tru64 compiler did not eat this correctly, and it should be investigated, because it may lead to other problems in other areas. Casting of address pointer to int's and long's is done frequently throughout the code. Perhaps Tru64 translates string constant addresses in to small pointers? some kind of segmented architecture?

Lutz

Title:
Post by: Lutz on January 06, 2007, 07:13:52 AM

... oops, now I see what is happening.

Replace

Code Select Expand if(*(UINT*)token == *(UINT*)"[tex"

with:

Code Select Expand if(*(unsigned int *)token == *(unsigned int *)"[tex"

UINT is 64-bit is 8 bytes long but should be only 4 bytes for the copmparison. 'int' will be 32-bit in both models.

Lutz

Title:
Post by: Lutz on January 06, 2007, 07:24:19 AM

~~Quote~~But the function (flt) converts an 8-byte float to a 4-byte float. On a 32-bit system, it converts a 4 byte float to a 2-byte float, and then the result must be 1067282596.

No, in both system 64- and 32-bit (flt x) converts an 8-byte double float ionto a 4-byte float. In both programming models 'double float' is 8 byte and 'float' is 4 byte. The result must aleyas be: (flt 1.23) => 1067282596. Is fine on AMD64.

Lutz

ps: the types.c output (top portion) looks perfect

Title:
Post by: pjot on January 06, 2007, 08:37:16 AM

Hi Lutz,

@[text]: your proposal solved the issue!

@flt: you are right, of course, I changed the code in "nl-import.c" as follows:
Code Select Expand 320 /* used when passing 32bit floats to library routines */ 321 CELL * p_flt(CELL * params) 322 { 323 double dfloatV; 324 float floatV; 325 //UINT number; 326 unsigned int number; 327 328 getFloat(params, &dfloatV); 329 330 floatV = dfloatV; 331 memcpy(&number, &floatV, 4); 332 333 return(stuffInteger(number)); 334 }

So the UINT has been replaced by 'unsigned int'. Same issue as with the [text]-problem. The (flt) function work perfectly OK now!

Now only one problem is left, I am going to investigate it now.

Peter

Title:
Post by: pjot on January 06, 2007, 08:49:05 AM

OK this was really simple. The whole issue comes down to the format string "%llx".
~~Quote~~
[peter@olga]# newlisp

newLISP v.9.0.14 64-bit on Tru64Unix, execute 'newlisp -h' for more info.

> (format "%llx" (~ 0xa0a0a0a0a0a0a0a0))

problem in format string in function format : "%llx"

> (format "%lx" (~ 0xa0a0a0a0a0a0a0a0))

"5f5f5f5f5f5f5f5f"

> (exit)

So if I change "%llx" to "%lx" in 'qa-dot' then all tests pass!

What do you propose as a nice workaround for this issue?

Anyway everything from 'qa-dot' works, I will test some of my private programs now.

Peter

Title:
Post by: Lutz on January 06, 2007, 10:13:15 AM

Thats the way it is in TRU64, it is already documented this way since 9.0 her http://newlisp.org/downloads/newlisp_manual.html#format

the problem is not with TRU64 but with qa-dot ;), I will put a fix in qa-dot and qa-comma.

Thanks for discovering the 'unsigned int' fix for p_flt(), it was pure luck, that this did not cause problems on AMD64. Amazing how each new port can uncover new bugs.

The fact that 64-bit mode now runs on two different platforms (Tru64 and Linux) is a nice validation, that the new 64-bit flavor seems to be mostly Ok.

Please send me the new 'makefile_tru64' and all changes so I can work them into 9.0.15. I already did the fixes for 'unsigned int' for [TEXT] and 'flt', but I am not sure about all other #ifdef TRU64

Lutz

ps: I will try to put out a 9.0.15 Monday or Tuesday, or whenever you are done.

Title:
Post by: pjot on January 06, 2007, 11:08:20 AM

Lutz,

Get ready for more good news!

1) The programs I tested in Tru64Unix all work, without any problem.

2) The performance actually is impressive. On a DS-15 with an Alpha processor running at 1Ghz speed, the performance faster (about 5-10%) compared to a 32-bit Intel running at the double speed of 2Ghz!

For the changes, if I remember well, it were these:

1) Remove the 'pointer_size' stuff from newlisp.h but leave the additional defines

2) Remove the 'pointer_size' stuff from newlisp.c

3) Remove the 'pointer_size' stuff from nl-filesys.c

4) Remove the 'pointer_size' stuff from nl-string.c but leave the line with 'result = strtoul...'

The modified sources I have uploaded for your reference, I will send a PM with the location. I have modfied the makefile_tru64 also, and the TRU64BUILD in the doc-directory. You will find these in the package.

Cheers

Peter

Title:
Post by: newdep on January 06, 2007, 03:22:54 PM

Here some speed tests with the latest release ;-)

All did the same LZW (by pjot) compression with newlisp.

DS-15 1Ghz processor

--------------------------------

Tru64Unix 32-bit: 20341 msecs

Tru64Unix 64-bit: 19299 msecs

AMD Athlon 2400+ XP processor

----------------------------------------------

Linux 32-bit: 11203 msecs

AMD Athlon64 3200+ processor

----------------------------------------------

Linux 32-bit: 5431

Pentium-4 1500/500 processor

----------------------------------------------

OS/2 32-bit: 22324

Title:
Post by: newdep on January 10, 2007, 07:43:34 AM

Intel Pentium-3 700 Mhz (laptop)

----------------------------------------------

Linux 32-bit: 33100 msecs

Intel Pentium-4 2 GHz (laptop)

----------------------------------------------

WinXP-Pro 32-bit: 21930 msecs

newLISP Fan Club

Forum => newLISP newS => Topic started by: Lutz on January 04, 2007, 04:58:36 AM