Hi, I'm using the amd64 (64 bit multicore) version of OpenBSD. I compile newlisp with readline and UTF8 support using the stock bsd makefile and it seems to work, though I haven't done thorough testing.
I fixed the artful code and the default MySQL modules to load the correct library, and they can both open and close the database, making a connection, but doing a query and looking at the results gives it an instant segfault. I suspect it has something to do with sizes of datatypes, perhaps even something to do with sql.c
Is anyone willing to log in and do some debugging? What would you want per hour?
Is it possible I just need to compile and run sql.c to produce correct values for mysql5.lsp?
You can reach me at 778-320-0644 or user "tederific" on skype.
Ted
I did run sql.c and updated the offset values for mysql5.lsp and it didn't help. As soon as I try to look at the results of a query, even something simple like MySQL:num-rows, makes for instant segfault.
Ted
Hi Ted,
The segfault comes from newlisp?
Just to verify because I dont have my Linux at hand at the moment...
did you install mysql with the kernel tuning parameters?http://www.openbsdsupport.org/mysql.htm
Is sql.c mysql5 code or is it a newlisp code..
Norman.
The mysql51.lsp shipped with 10.0.2 and also available here:
http://www.newlisp.org/modules/
is independent of offsets and probably works not only for MySQL 5.1 but also for earlier versions, e.g. 5.0.
Quote from: "Lutz"
http://www.newlisp.org/modules/
is independent of offsets and probably works not only for MySQL 5.1 but also for earlier versions, e.g. 5.0.
Thank you Lutz. I will try that when I get internet access again on Monday.
If it is independent of offsets, could it also be independent of the whole 32bit-vs-64bit thing?
Ted
Quote from: "newdep"
The segfault comes from newlisp?
Just to verify because I dont have my Linux at hand at the moment...
did you install mysql with the kernel tuning parameters?http://www.openbsdsupport.org/mysql.htm
Is sql.c mysql5 code or is it a newlisp code..
Norman.
sql.c is in the util directory in the newlisp source. It is used to calculate the proper offsets.
I notice the artful code mysql module does not seem to use offsets. Lutz says the new mysql51 module doesn't rely on them either. The artful code module segfaults, so perhaps the problem isn't with the offsets.
I didn't touch the kernel tuning parameters, but I did alter login.conf as suggested by the openbsd installation instructions. Also, the database is running fine answering many queries per second on a live website. I'm just hoping to add newlisp to the mix so I'm not stuck developing entirely with PHP.
The segfault definitely happened inside newlisp; doing things using the mysql client connecting to the server works fine.
Ted
Quote from: "Lutz"
http://www.newlisp.org/modules/
is independent of offsets and probably works not only for MySQL 5.1 but also for earlier versions, e.g. 5.0.
Thank you Lutz, I just tested and it worked!
I am using OpenBSD 4.4 amd64 (running on Intel chips) with MySQL 5.0.51 installed.
Perhaps the mysql51.lsp module could just be named mysql5.lsp?
Ted
Quote from: "TedWalther"
Thank you Lutz, I just testd it and it worked!
Sorry, I was premature. It worked more, but the essential part still doesn't work.
:init works
:connect works
:query works
:num-rows works
:num-fields works
:fetch-row segfaults
:fetch-all segfaults
:databases segfaults
:tables segfaults
:fields segfaults
:data-seek works
:affected-rows works
:inserted-id works
:error works
:escape works
:close-db works
Also, the module documentation omits the "s" from the end of the "tables" function, making it appear to be MySQL:table instead of MySQL:tables.
What makefile did you use to compile newLISP on OpenBSD / AMD64 and did it pass "make test" executed inside the distribution directory?
Quote from: "Lutz"
I used makefile_bsd and also makefile_bsd_utf8 at different times. Both get a Signal 11 in qa-dot. Here is more info:
$ make -f makefile_bsd_utf8
gcc -Wall -Wno-uninitialized -O2 -c -g -DREADLINE -D_BSD -DSUPPORT_UTF8 newlisp.c
newlisp.c: In function `newlisp_completion':
newlisp.c:809: warning: implicit declaration of function `completion_matches'
newlisp.c:809: warning: return makes pointer from integer without a cast
.
.
.
gcc newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o nl-import.o nl-xml.o nl-web.o nl-matrix.o nl-debug.o nl-utf8.o pcre.o -g -lm -lreadline -lncurses -o newlisp
newlisp.o(.text+0xc3d): In function `command_generator':
/data/local/newlisp-10.0.2/newlisp.c:795: warning: strcpy() is almost always misused, please use strlcpy()
newlisp.o(.text+0x2a1): In function `loadStartup':
/data/local/newlisp-10.0.2/newlisp.c:445: warning: strcat() is almost always misused, please use strlcat()
strip newlisp
m6 $ make test
./newlisp qa-dot
Testing built-in functions ...
*** Signal 11
Stop in /data/local/newlisp-10.0.2 (line 239 of Makefile).
>>>>> THE CILK API TESTED SUCESSFULL
.
get-char get-float get-int Segmentation fault (core dumped)
.
>>>>> DICTIONARY API TESTED SUCCESSFUL
.
Testing built-in functions ...
Segmentation fault (core dumped)
.
mandelbrot text image came up nicely
.
4990000 kbytes written
ERR: symbol expected in function read-buffer
.
net-peek: 11
HELLO WORLD: Ok
.
>>>>> NETWORK EVAL AND NETWORK FILE FUNCTIONS SUCCESSFUL
.
process 1
.
>>>>> REFERENCE TESTING SUCCESSFUL
.
>>>>> SIGNAL TESTING SUCCESSFUL
.
Step 5
ERR: user error : Expression failed! (not (rename-file UnicodeStr InvalidUTF8))
.
Yiddish:××× ×§×¢× ×¢×¡× ×××60</PRICE>
.
>>>>> XML CALLBACK TESTED SUCCESSFUL
In makefile_bsd change the CFLAGS and CC line to:
DREADLINE -D_BSD
...
$(CC) $(OBJS) -m32 -g -lm -lreadline -lncurses -o newlisp
this will force a 32Bit app on a 64bit CPU then try "make test" again (note that "make test" does not run all qa-* files). Also, there is a TAB character before $(CC) in order for the makefile to work.
ps: mysql51.lsp as a test routine too. After loading the module do:
Quote from: "Lutz"
DREADLINE -D_BSD
...
$(CC) $(OBJS) -m32 -g -lm -lreadline -lncurses -o newlisp
this will force a 32Bit app on a 64bit CPU then try "make test" again (note that "make test" does not run all qa-* files). Also, there is a TAB character before $(CC) in order for the makefile to work.
ps: mysql51.lsp as a test routine too. After loading the module do:
Tried that, then also tried the -melf_i386_obsd option at the link stage. It doesn't link because the 32 bit toolchain and libraries are missing. Then I found out that OpenBSD doesn't support a mixed 64/32 bit system. It is all one or the other. If you'd like a login account to try this, let me know.
Try the makefile posted here (make sure TABs don't get lost, make will complain, use the download option in your browser):
http://www.newlisp.org/downloads/development/makefile_bsdLP64
it is full 64 bit (basically added the -m64 option and the -DNEWLISP64 flag)
when you used the original makefile_bsd, it probably already tried to compile for 64 bit already but had the -DNEWLISP64 missing in the CFLAGS line. This then caused the segfaults in some of the functions.
Quote from: "Lutz"
http://www.newlisp.org/downloads/development/makefile_bsdLP64
it is full 64 bit (basically added the -m64 option and the -DNEWLISP64 flag)
when you used the original makefile_bsd, it probably already tried to compile for 64 bit already but had the -DNEWLISP64 missing in the CFLAGS line. This then caused the segfaults in some of the functions.
Wow, that's beautiful. Passes all tests with flying colors.
When I add UTF8 into the mix, some of the tests fail, so I backed it out again:
>>>>> TESTING: ./newlisp FINISHED WITH ERRORS:
UTF-8 upper-case: failed
UTF-8 lower-case: failed
Back to the mysql51 module, it still segfaults same as before.
gdb tells this from the core dump. Core dump was generated by running (test-mysql) immediately after loading the module.
(gdb) bt
#0 0x00000002105edbea in strlen () from /usr/lib/libc.so.48.0
#1 0x00000000004060a6 in stuffString (string=0x3706038 <Address>) at newlisp.c:1823
#2 0x00000000004053e7 in evaluateExpression (cell=0x20567be20) at newlisp.c:1338
#3 0x000000000040b5c9 in p_if (params=0x20567be20) at newlisp.c:5109
#4 0x00000000004053e7 in evaluateExpression (cell=0x20567b8a0) at newlisp.c:1338
#5 0x0000000000405b98 in evaluateLambda (localLst=0x20567b300, arg=0x20418b2c0, newContext=0x205676000) at newlisp.c:1639
#6 0x000000000040544c in evaluateExpression (cell=0x20567bfe0) at newlisp.c:1318
#7 0x0000000000417173 in p_push (params=0x20567bfe0) at nl-liststr.c:343
#8 0x00000000004053e7 in evaluateExpression (cell=0x20567c0c0) at newlisp.c:1338
#9 0x000000000040b5c9 in p_if (params=0x20567c0c0) at newlisp.c:5109
#10 0x00000000004053e7 in evaluateExpression (cell=0x20567c1a0) at newlisp.c:1338
#11 0x000000000040b5a8 in evaluateBlock (cell=0x20567c1a0) at newlisp.c:5091
#12 0x000000000040be66 in loop (params=0x1, forFlag=0) at newlisp.c:5444
#13 0x00000000004053e7 in evaluateExpression (cell=0x20567c4e0) at newlisp.c:1338
#14 0x000000000040b5a8 in evaluateBlock (cell=0x20567c4e0) at newlisp.c:5091
#15 0x000000000040c3d9 in p_evalBlock (params=0x3706038) at newlisp.c:5601
#16 0x00000000004053e7 in evaluateExpression (cell=0x20567c620) at newlisp.c:1338
#17 0x000000000040b612 in p_if (params=0x20567c6a0) at newlisp.c:5113
#18 0x00000000004053e7 in evaluateExpression (cell=0x20567c600) at newlisp.c:1338
#19 0x0000000000405b98 in evaluateLambda (localLst=0x20567aaa0, arg=0x205676000, newContext=0x205676000) at newlisp.c:1639
#20 0x000000000040544c in evaluateExpression (cell=0x209fb8680) at newlisp.c:1318
#21 0x000000000040c5fe in println (params=0x209fb8680, lineFeed=1) at newlisp.c:5709
#22 0x00000000004053e7 in evaluateExpression (cell=0x209fb8640) at newlisp.c:1338
#23 0x000000000040b5a8 in evaluateBlock (cell=0x209fb8640) at newlisp.c:5091
#24 0x000000000040be66 in loop (params=0x1, forFlag=0) at newlisp.c:5444
#25 0x00000000004053e7 in evaluateExpression (cell=0x209fb8580) at newlisp.c:1338
#26 0x0000000000405b98 in evaluateLambda (localLst=0x20567d780, arg=0x205676000, newContext=0x205676000) at newlisp.c:1639
#27 0x000000000040544c in evaluateExpression (cell=0x205679640) at newlisp.c:1318
#28 0x00000000004049af in evaluateStream (stream=0x7f7ffffeddf0, outDevice=0, flag=1) at newlisp.c:1055
#29 0x0000000000407e6c in loadFile (fileName=0x7f7ffffeddf0 " 03", offset=0, encryptFlag=0, context=0x202846200) at newlisp.c:3004
#30 0x000000000040404a in main (argc=2, argv=0x7f7ffffee048) at newlisp.c:693
You can neglect the 'upper/lower-case' error in the UTF-8 version. It just means that some localizations are missing in your installation and for the language tested (Greek alphabet); but UTF-8 strings will be processed correctly except for 'upper/lower-case' of certain characters. Certainly for English it will be Ok.
The fact that MySQL doesn't work at all, could have many reasons. May be you have more luck using the MySQL module from ArtfulCode, accessible from here:
http://www.newlisp.org/modules/
and here:
http://www.artfulcode.net/
An issue may be not using the correct libmysqlclient.so. It is not always part of the server installation, and perhaps you have a mismatch. I have seen this frequently.
What I also do not understand is that your installation only offers a 64-bit API. The standard on 64-bit machines today is, to offer a full set of 32-bit libraries and utilities. This mixed model is used by Mac OS X, MS Vista and all mayor Linux installations. For details on this model see here:
http://www.unix.org/version2/whatsnew/lp64_wp.html
At this point, I don't know how to help. Compiling/linking for 32-bit seemed to work but it didn't pass the tests. The well testing 64-bit version doesn't fit with your MySQL installation.
ps: one last test: In the newlis-x.x.x/util subdirectory is a file types.c. Compile/link it 3 times:
gcc -m32 types.c -o types32
gcc -m64 types.c -o types64
and send me or post the outputs of all three programs.
ps2: I am currently traveling, but could log on to your machine by the weekend. email me: lutz at nuevatec.com
Quote from: "Lutz"
The fact that MySQL doesn't work at all, could have many reasons. May be you have more luck using the MySQL module from ArtfulCode, accessible from here:
http://www.newlisp.org/modules/
and here:
http://www.artfulcode.net/
An issue may be not using the correct libmysqlclient.so. It is not always part of the server installation, and perhaps you have a mismatch. I have seen this frequently.
What I also do not understand is that your installation only offers a 64-bit API. The standard on 64-bit machines today is, to offer a full set of 32-bit libraries and utilities. This mixed model is used by Mac OS X, MS Vista and all mayor Linux installations. For details on this model see here:
http://www.unix.org/version2/whatsnew/lp64_wp.html
At this point, I don't know how to help. Compiling/linking for 32-bit seemed to work but it didn't pass the tests. The well testing 64-bit version doesn't fit with your MySQL installation.
ps: one last test: In the newlis-x.x.x/util subdirectory is a file types.c. Compile/link it 3 times:
gcc -m32 types.c -o types32
gcc -m64 types.c -o types64
and send me or post the outputs of all three programs.
ps2: I am currently traveling, but could log on to your machine by the weekend. email me: lutz at nuevatec.com
Compilation:
m6 $ gcc types.c -o types
m6 $ gcc -m32 types.c -o types32
/usr/bin/ld: warning: i386 architecture of input file `/tmp//ccM25150.o' is incompatible with i386:x86-64 output
m6 $ gcc -m64 types.c -o types64
m6 $
type and type64 give identical output, so I only show output for type64 here.
m6 $ ./types32
Bus error (core dumped)
m6 $ ./types64
type bytes
---------------
char 1
char * 8
void * 8
short int 2
int 4
long 8
long int 8
long long 8
size_t 8
float 4
double 8
long double 16
wchar_t 4
format input output
------------------------------------------------
%d 0xffffffff -1
%u 0xffffffff 4294967295
%d 0x7fffffff 2147483647
%u 0x7fffffff 2147483647
%d 0x80000000 -2147483648
%u 0x80000000 2147483648
%d 0x7fffffffffffffffLL -1
%u 0x7fffffffffffffffLL 4294967295
%x 0xffffffffffffffffLL ffffffff
%X 0x7fffffffffffffffLL ffffffff
%X 0x8000000000000000LL 0
%llX 0xFFFFFFFFFFFFFFFFLL ffffffff
%llX 0x7FFFFFFFFFFFFFFFLL ffffffff
%llX 0x8000000000000000LL 0
%lld 0xffffffffffffffffLL -1
%llu 0xffffffffffffffffLL 18446744073709551615
%lld 0x7fffffffffffffffLL 9223372036854775807
%llu 0x7fffffffffffffffLL 9223372036854775807
%lld 0x8000000000000000LL -9223372036854775808
%llu 0x8000000000000000LL 9223372036854775808
CPU is little endian
m6 $
mmm... odd...
Could be there is a 32b and 64b library mixup?
(can you do a 'ldd newlisp' for the 32b and 64b version)
Do you do a 'make clean' every time you compile newlisp?
(just out of precaution and needed when using both 32/64)..
32Bits isnt the problem, what I do know though is that OpenBSD has
a very strick way of handling of the clibrary, what version of Obsd do you use?
Quote from: "Lutz"
The fact that MySQL doesn't work at all, could have many reasons. May be you have more luck using the MySQL module from ArtfulCode, accessible from here:
http://www.newlisp.org/modules/
The ArtfulCode module segfaults. Interestingly, it segfaults in the same function for a very similar reason; "address out of bounds" inside one of the stuffString functions, although in this case is is stuffStringN calling memcpy instead of stuffString calling strlen as happens with your module.
The OpenBSD developers told me they put some safeguards in their system to make certain programming errors segfault right away. Perhaps newlisp is hitting those safeguards?
Here is the backtrace for the ArtfulCode module:
#0 0x00000002031f1209 in memcpy (dst0=0x207364350, src0=0xddaf120, length=0) at /usr/src/lib/libc/string/bcopy.c:115
#1 0x0000000000406129 in stuffStringN (string=0xddaf120 <Address>, len=1) at newlisp.c:1838
#2 0x000000000041b480 in p_unpack (params=0x207364350) at nl-string.c:1891
#3 0x00000000004053e7 in evaluateExpression (cell=0x20a5ec920) at newlisp.c:1338
#4 0x00000000004094e9 in getEvalDefault (params=0x20a5ec920, result=0x7f7ffffbbb18) at newlisp.c:3824
#5 0x000000000040af66 in p_first (params=0x20a5ec920) at newlisp.c:4799
#6 0x00000000004053e7 in evaluateExpression (cell=0x20a5ec8e0) at newlisp.c:1338
#7 0x000000000040a782 in p_setf (params=0x20a5ec8c0) at newlisp.c:4495
#8 0x00000000004053e7 in evaluateExpression (cell=0x20a5ec880) at newlisp.c:1338
#9 0x0000000000405b98 in evaluateLambda (localLst=0x20a5ec600, arg=0x208de8740, newContext=0x204d32000) at newlisp.c:1639
#10 0x000000000040544c in evaluateExpression (cell=0x20a5ec5c0) at newlisp.c:1318
#11 0x0000000000413a4a in p_map (params=0x20a5ec5c0) at nl-list.c:93
#12 0x00000000004053e7 in evaluateExpression (cell=0x20a5e9180) at newlisp.c:1338
#13 0x000000000040a782 in p_setf (params=0x20a5e9160) at newlisp.c:4495
#14 0x00000000004053e7 in evaluateExpression (cell=0x20a5e9120) at newlisp.c:1338
#15 0x000000000040b5a8 in evaluateBlock (cell=0x20a5e9120) at newlisp.c:5091
#16 0x000000000040c3d9 in p_evalBlock (params=0x207364350) at newlisp.c:5601
#17 0x00000000004053e7 in evaluateExpression (cell=0x20a5e8ce0) at newlisp.c:1338
#18 0x000000000040b67e in p_ifNot (params=0x20a5e8c80) at newlisp.c:5133
#19 0x00000000004053e7 in evaluateExpression (cell=0x20a5e8c40) at newlisp.c:1338
#20 0x000000000040acd5 in let (params=0x204d35fc0, type=2) at newlisp.c:4773
#21 0x00000000004053e7 in evaluateExpression (cell=0x204d35f80) at newlisp.c:1338
#22 0x0000000000405e28 in evaluateMacro (localLst=0x204d35b20, arg=0x204d35f80, newContext=0x0) at newlisp.c:1733
#23 0x0000000000405425 in evaluateExpression (cell=0x206dedb40) at newlisp.c:1325
#24 0x0000000000405b98 in evaluateLambda (localLst=0x206dedb20, arg=0x20c2e1980, newContext=0x204d32000) at newlisp.c:1639
#25 0x000000000040544c in evaluateExpression (cell=0x206dee7a0) at newlisp.c:1318
#26 0x000000000040e09c in p_colon (params=0x204d32000) at newlisp.c:6774
#27 0x00000000004053e7 in evaluateExpression (cell=0x206deeb00) at newlisp.c:1338
#28 0x000000000040a782 in p_setf (params=0x206deeb40) at newlisp.c:4495
#29 0x00000000004053e7 in evaluateExpression (cell=0x206deec40) at newlisp.c:1338
#30 0x000000000040bb10 in repeat (params=0x206deec40, type=0) at newlisp.c:5289
#31 0x00000000004053e7 in evaluateExpression (cell=0x206dee980) at newlisp.c:1338
#32 0x0000000000405b98 in evaluateLambda (localLst=0x206dee960, arg=0x20c2e1ac0, newContext=0x204d32000) at newlisp.c:1639
#33 0x000000000040544c in evaluateExpression (cell=0x204d38a80) at newlisp.c:1318
#34 0x0000000000405b98 in evaluateLambda (localLst=0x206df2e80, arg=0x20c2e1040, newContext=0x204d32000) at newlisp.c:1639
#35 0x000000000040544c in evaluateExpression (cell=0x206df2140) at newlisp.c:1318
#36 0x00000000004099de in p_catch (params=0x207364350) at newlisp.c:3998
#37 0x00000000004053e7 in evaluateExpression (cell=0x206df24a0) at newlisp.c:1338
#38 0x000000000040a782 in p_setf (params=0x206df2120) at newlisp.c:4495
#39 0x00000000004053e7 in evaluateExpression (cell=0x206df2460) at newlisp.c:1338
#40 0x000000000040b612 in p_if (params=0x206def300) at newlisp.c:5113
#41 0x00000000004053e7 in evaluateExpression (cell=0x206dee580) at newlisp.c:1338
#42 0x0000000000405b98 in evaluateLambda (localLst=0x206def420, arg=0x20c2e1a80, newContext=0x204d32000) at newlisp.c:1639
#43 0x000000000040544c in evaluateExpression (cell=0x204d35640) at newlisp.c:1318
#44 0x00000000004049af in evaluateStream (stream=0x7f7ffffbc500, outDevice=0, flag=1) at newlisp.c:1055
#45 0x0000000000407e6c in loadFile (fileName=0x7f7ffffbc500 " 03", offset=0, encryptFlag=0, context=0x20a7f2140) at newlisp.c:3004
#46 0x000000000040404a in main (argc=2, argv=0x7f7ffffbc750) at newlisp.c:693
Quote from: "newdep"
Could be there is a 32b and 64b library mixup?
(can you do a 'ldd newlisp' for the 32b and 64b version)
Do you do a 'make clean' every time you compile newlisp?
(just out of precaution and needed when using both 32/64)..
32Bits isnt the problem, what I do know though is that OpenBSD has
a very strick way of handling of the clibrary, what version of Obsd do you use?
OpenBSD 4.4
On OpenBSD, there is no support for running mixed binaries; it is either 64 bit or 32 bit. There is no issue with picking the right library version. I am running the 64 bit OS.
Yes, I do make clean every time.
Quote from: "Lutz"
An issue may be not using the correct libmysqlclient.so. It is not always part of the server installation, and perhaps you have a mismatch. I have seen this frequently.
Although mysql doesn't come by default with OpenBSD, there is a standard package for OpenBSD. In OpenBSD 4.4 it installs the libraries like this:
/usr/local/lib/libmysqlclient.so.19.0
/usr/local/lib/libmysqlclient_r.so.19.0
/usr/local/lib/mysql/libmysqlclient.a
/usr/local/lib/mysql/libmysqlclient.la
/usr/local/lib/mysql/libmysqlclient.so.19.0
/usr/local/lib/mysql/libmysqlclient_r.a
/usr/local/lib/mysql/libmysqlclient_r.la
/usr/local/lib/mysql/libmysqlclient_r.so.19.0
So far I've been using /usr/local/lib/mysql/libmysqlclient.so.19.0 as the library to load. And many functions, such as query, num-rows, num-fields, do succeed.
The default on your machine seems to be 64-bit, and the 32-bit compile of types.c bombs right away.
I ran OpenBSD 32-bit with MySQL a few years back (2006/7) without a problem, but that was on 32-bit Intel CPUs. As you said: it is either or on OpenBSD, different from other OSs like OSX, Vista, Linux. Its probably due to their philosophy of keeping things safe, straight and simple.
If OpenBSD does not offer 32-bit libs on their 64-bit OS version, we are out of luck :( , unless you could install OpenBSD 32-bit on that machine?
ps: I will not be available until Thursday morning EST
ps2: isn't there a TCP/IP interface available with MySQL?
Quote from: "Lutz"
If OpenBSD does not offer 32-bit libs on their 64-bit OS version, we are out of luck :( , unless you could install OpenBSD 32-bit on that machine?
ps2: isn't there a TCP/IP interface available with MySQL?
Is there something in newlisp that depends on 32 bit libs? Or is it just harder to debug without having them to compare with? I have access to a 32 bit OpenBSD 4.4 box for testing. In fact, I justed tested it there, and it works nicely with mysql, no failures evident.
A quick two minute search of the web didn't pull up any information on the wire protocol MySQL uses. Telnetting to port 3306 locally showed that it isn't a straight text protocol like POP or SMTP.
Will email you more info after lunch.
Wow Lutz, you rock! Will the module updates be incorporated in the 10.0.5 release? Now I'm wondering how many other modules need similar fixes added.
The odbc and sqlite3 modules may need to be updated too, if 64-bit libraries exist for these at all. Other modules in the distribution are probably safe to use with 64-bit newLISP unchanged.
The new mysql.lsp module will be posted later, and then it goes into the 10.0.5 maintenance release in May.
Yes, there are 64bit libraries for all the standard modules that come with newLISP. The account is still active if you want to do any updates or testing.
crypto.lsp
* /usr/lib/libcrypto.so.14.0
gmp.lsp
* /usr/local/lib/libgmp.so.7.1
mysql.lsp
* /usr/local/lib/libmysqlclient.so.19.0
odbc.lsp
* /usr/local/lib/libiodbc.so.3.15
* /usr/local/lib/libiodbcinst.so.3.15
sqlite3.lsp
* /usr/local/lib/libsqlite3.so.11.0
unix.lsp
* /usr/lib/libc.so.48.0
zlisp.lsp
* /usr/lib/libz.so.4.1
I couldn't find any of the following libraries on your system:
* /usr/local/lib/libgmp.so.7.1
* /usr/local/lib/libiodbc.so.3.15
* /usr/local/lib/libiodbcinst.so.3.15
* /usr/local/lib/libsqlite3.so.11.0
perhaps you looked on a 32-bit machine, not on the 64-bit? But I could verify the correct working of libc, libcrypto and libz under 64-bit newLISP.
There is a new 10.0.5 on your system and I accidently removed 10.0.4-local. I hope your local adaptations can quickly be redone.
Changes are made for all the special OpenBSD compiler warnings, but one warning still shows up although already fixed. Perhaps some cacheing issue? Rebooting the machine will probably make it go away.
Thanks, I was looking at those libraries on another 64bit OpenBSD machine. They are now installed on the test box.
The warnign was referring to your use of strncat(); openbsd deprecates it as much as strcat() in favor of strlcat (and strlcpy). I'm now looking into how to incorporate those two functions portably.
Thanks Ted, all warnings are gone now and libgmp and libsqlite3 work too. Only sqlite3.lsp needed to adapt to 64-bit. The gmp.lsp module worked right away with the added library path for OpenBSD.
Adapting the odbc.lsp to 64-bit will take longer and may not happen for the next release. It never has been used on UNIX anyway and lacks functions for creating ODBC instances. On Win32, for which this module was created originally, ODBC instances are created using a control applet.
Thanks again for letting me use your OpenBSD system. If you could leave the account open for me another few weeks, that would be great. I could then more thoroughly test the upcoming maintenance release.
ps: note that mysql.lsp still contains the logon credentials on your machine.
I'm happy to make the box available as long as you'd like for continuing to make newlisp the best programming environment ever. As long as the login credentials don't get into the next release, that is alright ;-) I just made a patch to get rid of two warnings:
First, the strlcat() warning is now fixed, I updated the source to use those functions, with appropriate error checking.
Next, in the command generator for the readline support, I added some error checking and the (char*) cast that shut the compiler up.
Does the patch suit your style sufficiently?
http://reactor-core.org/newlisp.diff.txt
I am more selective on error-checking and would prefer to not have it in those places, where you added it ;-). Error-checking policy, as it is now, has proven to be more than sufficient in the past. I also prefer to stay with the strlcat() and strlcpy() versions built into gcc. The old code using strcpy() and strcat() was already safe, and the move to strlcpy() and strlcat() was made only to suppress compiler warnings.
It would be great to have that OpenBSD machine available to me in the future. If you have to change credentials once in a while, that is Ok with me, just let me know when you make a change. I have similar arrangements with other users (e.g. Solaris and TRU64) and it has helped to make newLISP safer and stable. OpenBSD is specially interesting, because they put this focus on security and a clean compile is harder to achieve than on other OSs.
Thanks again for making OpenBSD 64-bit available to me.
Quote from: "Lutz"
It would be great to have that OpenBSD machine available to me in the future. If you have to change credentials once in a while, that is Ok with me, just let me know when you make a change. I have similar arrangements with other users (e.g. Solaris and TRU64) and it has helped to make newLISP safer and stable. OpenBSD is specially interesting, because they put this focus on security and a clean compile is harder to achieve than on other OSs.
Thanks again for making OpenBSD 64-bit available to me.
The box is available to you as long as you need. It is also multicore running SMP, in case that is needed.
strlcat and strlcpy aren't included in GNU libc, the OpenBSD guys recommended direct inclusion of the functions as the portable way to do it.
yes, I just realize strlcat() and strlcpy() are only on BSD based OSs (Mac OSX, FreeBSD, Solaris) but not on Linux and Win32. In those cases strncpy() and strncat() can stay as the code was safe already (even using strcpy() and strcat()), and no warnings are generated on these platforms.
Quote
Thanks, and SMP is good news for launching parallel processes using:
http://www.newlisp.org/newlisp_manual.html#spawn
Mac OSX does a good job balancing multiple processes launched by 'spawn' on multi-core CPUs and I guess OpenBSD will do similar with SMP.
... turns out BSD SMP does an even better job balancing three processes on 2 AMD64 processors or cores (I assume) compared to OSX on Mac Mini 1.83 GHZ Intel Core Duo:
7478ms vs 5142ms on BSD -> 1.45 (BSD 64-bit, newLISP 64-bit)
17101ms vs 12574ms on OSX -> 1.36 (OSX 64-bit, newLISP 32-bit)
13535ms vs 9925ms on OSX -> 1.36 (OSX 64-bit, newLISP 64-bit)
measured with the 'spawn' prime number example in the manual.
Quote from: "Lutz"
7478ms vs 5142ms on BSD -> 1.45 (BSD 64-bit, newLISP 64-bit)
17101ms vs 12574ms on OSX -> 1.36 (OSX 64-bit, newLISP 32-bit)
13535ms vs 9925ms on OSX -> 1.36 (OSX 64-bit, newLISP 64-bit)
measured with the 'spawn' prime number example in the manual.
Actually it is a quad-core box; that might make a difference.
From what I understand there are lots of improvements to be made in the OS X kernel in relation to SMP, and that many of these improvements should find their way into Snow Leopard. Great discussion btw! Thanks Lutz for getting that mysql module to work on 64-bit!
As always, benchmarks are a tricky business with all sorts of surprises.
After Ted revealed that his machine is actually a quad core (not dual as I assumed), I tried a spawn with four processes, increasing the ratio from 1.45 to 1.58 (simple versus spawn) on BSD but on OS X dual core I now get even better to a 1.67 ratio!