pay someone for OpenBSD fixes?

[TedWalther]

gdb tells this from the core dump.  Core dump was generated by running (test-mysql) immediately after loading the module.



(gdb) bt

#0  0x00000002105edbea in strlen () from /usr/lib/libc.so.48.0

#1  0x00000000004060a6 in stuffString (string=0x3706038 <Address>) at newlisp.c:1823

#2  0x00000000004053e7 in evaluateExpression (cell=0x20567be20) at newlisp.c:1338

#3  0x000000000040b5c9 in p_if (params=0x20567be20) at newlisp.c:5109

#4  0x00000000004053e7 in evaluateExpression (cell=0x20567b8a0) at newlisp.c:1338

#5  0x0000000000405b98 in evaluateLambda (localLst=0x20567b300, arg=0x20418b2c0, newContext=0x205676000) at newlisp.c:1639

#6  0x000000000040544c in evaluateExpression (cell=0x20567bfe0) at newlisp.c:1318

#7  0x0000000000417173 in p_push (params=0x20567bfe0) at nl-liststr.c:343

#8  0x00000000004053e7 in evaluateExpression (cell=0x20567c0c0) at newlisp.c:1338

#9  0x000000000040b5c9 in p_if (params=0x20567c0c0) at newlisp.c:5109

#10 0x00000000004053e7 in evaluateExpression (cell=0x20567c1a0) at newlisp.c:1338

#11 0x000000000040b5a8 in evaluateBlock (cell=0x20567c1a0) at newlisp.c:5091

#12 0x000000000040be66 in loop (params=0x1, forFlag=0) at newlisp.c:5444

#13 0x00000000004053e7 in evaluateExpression (cell=0x20567c4e0) at newlisp.c:1338

#14 0x000000000040b5a8 in evaluateBlock (cell=0x20567c4e0) at newlisp.c:5091

#15 0x000000000040c3d9 in p_evalBlock (params=0x3706038) at newlisp.c:5601

#16 0x00000000004053e7 in evaluateExpression (cell=0x20567c620) at newlisp.c:1338

#17 0x000000000040b612 in p_if (params=0x20567c6a0) at newlisp.c:5113

#18 0x00000000004053e7 in evaluateExpression (cell=0x20567c600) at newlisp.c:1338

#19 0x0000000000405b98 in evaluateLambda (localLst=0x20567aaa0, arg=0x205676000, newContext=0x205676000) at newlisp.c:1639

#20 0x000000000040544c in evaluateExpression (cell=0x209fb8680) at newlisp.c:1318

#21 0x000000000040c5fe in println (params=0x209fb8680, lineFeed=1) at newlisp.c:5709

#22 0x00000000004053e7 in evaluateExpression (cell=0x209fb8640) at newlisp.c:1338

#23 0x000000000040b5a8 in evaluateBlock (cell=0x209fb8640) at newlisp.c:5091

#24 0x000000000040be66 in loop (params=0x1, forFlag=0) at newlisp.c:5444

#25 0x00000000004053e7 in evaluateExpression (cell=0x209fb8580) at newlisp.c:1338

#26 0x0000000000405b98 in evaluateLambda (localLst=0x20567d780, arg=0x205676000, newContext=0x205676000) at newlisp.c:1639

#27 0x000000000040544c in evaluateExpression (cell=0x205679640) at newlisp.c:1318

#28 0x00000000004049af in evaluateStream (stream=0x7f7ffffeddf0, outDevice=0, flag=1) at newlisp.c:1055

#29 0x0000000000407e6c in loadFile (fileName=0x7f7ffffeddf0 "03", offset=0, encryptFlag=0, context=0x202846200) at newlisp.c:3004

#30 0x000000000040404a in main (argc=2, argv=0x7f7ffffee048) at newlisp.c:693

[Lutz]

You can neglect the 'upper/lower-case' error in the UTF-8 version. It just means that some localizations are missing in your installation and for the language tested (Greek alphabet); but UTF-8 strings will be processed correctly except for 'upper/lower-case' of certain characters. Certainly for English it will be Ok.



The fact that MySQL doesn't work at all, could have many reasons. May be you have more luck using the MySQL module from ArtfulCode, accessible from here:



http://www.newlisp.org/modules/">http://www.newlisp.org/modules/



and here:



http://www.artfulcode.net/">http://www.artfulcode.net/



An issue may be not using the correct libmysqlclient.so. It is not always part of the server installation, and perhaps you have a mismatch. I have seen this frequently.



What I also do not understand is that your installation only offers a 64-bit API. The standard on 64-bit machines today is, to offer a full set of 32-bit libraries and utilities. This mixed model is used by Mac OS X, MS Vista and all mayor Linux installations. For details on this model see here:



http://www.unix.org/version2/whatsnew/lp64_wp.html">http://www.unix.org/version2/whatsnew/lp64_wp.html



At this point, I don't know how to help. Compiling/linking for 32-bit seemed to work but it didn't pass the tests. The well testing 64-bit version doesn't fit with your MySQL installation.



ps: one last test: In the newlis-x.x.x/util subdirectory is a file types.c. Compile/link it 3 times:

Code Select Expand

gcc types.c -o types
gcc -m32 types.c -o types32
gcc -m64 types.c -o types64


and send me or post the outputs of all three programs.



ps2: I am currently traveling, but could log on to your machine by the weekend. email me: lutz at nuevatec.com

[TedWalther]

You can neglect the 'upper/lower-case' error in the UTF-8 version. It just means that some localizations are missing in your installation and for the language tested (Greek alphabet); but UTF-8 strings will be processed correctly except for 'upper/lower-case' of certain characters. Certainly for English it will be Ok.



The fact that MySQL doesn't work at all, could have many reasons. May be you have more luck using the MySQL module from ArtfulCode, accessible from here:



http://www.newlisp.org/modules/">http://www.newlisp.org/modules/



and here:



http://www.artfulcode.net/">http://www.artfulcode.net/



An issue may be not using the correct libmysqlclient.so. It is not always part of the server installation, and perhaps you have a mismatch. I have seen this frequently.



What I also do not understand is that your installation only offers a 64-bit API. The standard on 64-bit machines today is, to offer a full set of 32-bit libraries and utilities. This mixed model is used by Mac OS X, MS Vista and all mayor Linux installations. For details on this model see here:



http://www.unix.org/version2/whatsnew/lp64_wp.html">http://www.unix.org/version2/whatsnew/lp64_wp.html



At this point, I don't know how to help. Compiling/linking for 32-bit seemed to work but it didn't pass the tests. The well testing 64-bit version doesn't fit with your MySQL installation.



ps: one last test: In the newlis-x.x.x/util subdirectory is a file types.c. Compile/link it 3 times:

Code Select Expand

gcc types.c -o types
gcc -m32 types.c -o types32
gcc -m64 types.c -o types64


and send me or post the outputs of all three programs.



ps2: I am currently traveling, but could log on to your machine by the weekend. email me: lutz at nuevatec.com

Compilation:

Code Select Expand


m6 $ gcc types.c -o types
m6 $ gcc -m32 types.c -o types32
/usr/bin/ld: warning: i386 architecture of input file `/tmp//ccM25150.o' is incompatible with i386:x86-64 output
m6 $ gcc -m64 types.c -o types64
m6 $


type and type64 give identical output, so I only show output for type64 here.

Code Select Expand


m6 $ ./types32
Bus error (core dumped)
m6 $ ./types64

type      bytes
---------------
char        1
char *      8
void *      8
short int   2
int         4
long        8
long int    8
long long   8
size_t      8
float       4
double      8
long double 16
wchar_t     4

format input              output
------------------------------------------------
%d     0xffffffff           -1
%u     0xffffffff           4294967295
%d     0x7fffffff           2147483647
%u     0x7fffffff           2147483647
%d     0x80000000           -2147483648
%u     0x80000000           2147483648
%d     0x7fffffffffffffffLL -1
%u     0x7fffffffffffffffLL 4294967295
%x     0xffffffffffffffffLL ffffffff
%X     0x7fffffffffffffffLL ffffffff
%X     0x8000000000000000LL 0
%llX   0xFFFFFFFFFFFFFFFFLL ffffffff
%llX   0x7FFFFFFFFFFFFFFFLL ffffffff
%llX   0x8000000000000000LL 0
%lld   0xffffffffffffffffLL -1
%llu   0xffffffffffffffffLL 18446744073709551615
%lld   0x7fffffffffffffffLL 9223372036854775807
%llu   0x7fffffffffffffffLL 9223372036854775807
%lld   0x8000000000000000LL -9223372036854775808
%llu   0x8000000000000000LL 9223372036854775808

CPU is little endian
m6 $


[newdep]

mmm... odd...



Could be there is a 32b and 64b library mixup?

(can you do a 'ldd newlisp' for the 32b and 64b version)



Do you do a 'make clean' every time you compile newlisp?

(just out of precaution and needed when using both 32/64)..



32Bits isnt the problem, what I do know though is that OpenBSD has

a very strick way of handling of the clibrary, what version of Obsd do you use?

[TedWalther]

The fact that MySQL doesn't work at all, could have many reasons. May be you have more luck using the MySQL module from ArtfulCode, accessible from here:



http://www.newlisp.org/modules/">http://www.newlisp.org/modules/

The ArtfulCode module segfaults.  Interestingly, it segfaults in the same function for a very similar reason; "address out of bounds" inside one of the stuffString functions, although in this case is is stuffStringN calling memcpy instead of stuffString calling strlen as happens with your module.



The OpenBSD developers told me they put some safeguards in their system to make certain programming errors segfault right away.  Perhaps newlisp is hitting those safeguards?



Here is the backtrace for the ArtfulCode module:

Code Select Expand


#0  0x00000002031f1209 in memcpy (dst0=0x207364350, src0=0xddaf120, length=0) at /usr/src/lib/libc/string/bcopy.c:115
#1  0x0000000000406129 in stuffStringN (string=0xddaf120 <Address>, len=1) at newlisp.c:1838
#2  0x000000000041b480 in p_unpack (params=0x207364350) at nl-string.c:1891
#3  0x00000000004053e7 in evaluateExpression (cell=0x20a5ec920) at newlisp.c:1338
#4  0x00000000004094e9 in getEvalDefault (params=0x20a5ec920, result=0x7f7ffffbbb18) at newlisp.c:3824
#5  0x000000000040af66 in p_first (params=0x20a5ec920) at newlisp.c:4799
#6  0x00000000004053e7 in evaluateExpression (cell=0x20a5ec8e0) at newlisp.c:1338
#7  0x000000000040a782 in p_setf (params=0x20a5ec8c0) at newlisp.c:4495
#8  0x00000000004053e7 in evaluateExpression (cell=0x20a5ec880) at newlisp.c:1338
#9  0x0000000000405b98 in evaluateLambda (localLst=0x20a5ec600, arg=0x208de8740, newContext=0x204d32000) at newlisp.c:1639
#10 0x000000000040544c in evaluateExpression (cell=0x20a5ec5c0) at newlisp.c:1318
#11 0x0000000000413a4a in p_map (params=0x20a5ec5c0) at nl-list.c:93
#12 0x00000000004053e7 in evaluateExpression (cell=0x20a5e9180) at newlisp.c:1338
#13 0x000000000040a782 in p_setf (params=0x20a5e9160) at newlisp.c:4495
#14 0x00000000004053e7 in evaluateExpression (cell=0x20a5e9120) at newlisp.c:1338
#15 0x000000000040b5a8 in evaluateBlock (cell=0x20a5e9120) at newlisp.c:5091
#16 0x000000000040c3d9 in p_evalBlock (params=0x207364350) at newlisp.c:5601
#17 0x00000000004053e7 in evaluateExpression (cell=0x20a5e8ce0) at newlisp.c:1338
#18 0x000000000040b67e in p_ifNot (params=0x20a5e8c80) at newlisp.c:5133
#19 0x00000000004053e7 in evaluateExpression (cell=0x20a5e8c40) at newlisp.c:1338
#20 0x000000000040acd5 in let (params=0x204d35fc0, type=2) at newlisp.c:4773
#21 0x00000000004053e7 in evaluateExpression (cell=0x204d35f80) at newlisp.c:1338
#22 0x0000000000405e28 in evaluateMacro (localLst=0x204d35b20, arg=0x204d35f80, newContext=0x0) at newlisp.c:1733
#23 0x0000000000405425 in evaluateExpression (cell=0x206dedb40) at newlisp.c:1325
#24 0x0000000000405b98 in evaluateLambda (localLst=0x206dedb20, arg=0x20c2e1980, newContext=0x204d32000) at newlisp.c:1639
#25 0x000000000040544c in evaluateExpression (cell=0x206dee7a0) at newlisp.c:1318
#26 0x000000000040e09c in p_colon (params=0x204d32000) at newlisp.c:6774
#27 0x00000000004053e7 in evaluateExpression (cell=0x206deeb00) at newlisp.c:1338
#28 0x000000000040a782 in p_setf (params=0x206deeb40) at newlisp.c:4495
#29 0x00000000004053e7 in evaluateExpression (cell=0x206deec40) at newlisp.c:1338
#30 0x000000000040bb10 in repeat (params=0x206deec40, type=0) at newlisp.c:5289
#31 0x00000000004053e7 in evaluateExpression (cell=0x206dee980) at newlisp.c:1338
#32 0x0000000000405b98 in evaluateLambda (localLst=0x206dee960, arg=0x20c2e1ac0, newContext=0x204d32000) at newlisp.c:1639
#33 0x000000000040544c in evaluateExpression (cell=0x204d38a80) at newlisp.c:1318
#34 0x0000000000405b98 in evaluateLambda (localLst=0x206df2e80, arg=0x20c2e1040, newContext=0x204d32000) at newlisp.c:1639
#35 0x000000000040544c in evaluateExpression (cell=0x206df2140) at newlisp.c:1318
#36 0x00000000004099de in p_catch (params=0x207364350) at newlisp.c:3998
#37 0x00000000004053e7 in evaluateExpression (cell=0x206df24a0) at newlisp.c:1338
#38 0x000000000040a782 in p_setf (params=0x206df2120) at newlisp.c:4495
#39 0x00000000004053e7 in evaluateExpression (cell=0x206df2460) at newlisp.c:1338
#40 0x000000000040b612 in p_if (params=0x206def300) at newlisp.c:5113
#41 0x00000000004053e7 in evaluateExpression (cell=0x206dee580) at newlisp.c:1338
#42 0x0000000000405b98 in evaluateLambda (localLst=0x206def420, arg=0x20c2e1a80, newContext=0x204d32000) at newlisp.c:1639
#43 0x000000000040544c in evaluateExpression (cell=0x204d35640) at newlisp.c:1318
#44 0x00000000004049af in evaluateStream (stream=0x7f7ffffbc500, outDevice=0, flag=1) at newlisp.c:1055
#45 0x0000000000407e6c in loadFile (fileName=0x7f7ffffbc500 "03", offset=0, encryptFlag=0, context=0x20a7f2140) at newlisp.c:3004
#46 0x000000000040404a in main (argc=2, argv=0x7f7ffffbc750) at newlisp.c:693


[TedWalther]

mmm... odd...



Could be there is a 32b and 64b library mixup?

(can you do a 'ldd newlisp' for the 32b and 64b version)



Do you do a 'make clean' every time you compile newlisp?

(just out of precaution and needed when using both 32/64)..



32Bits isnt the problem, what I do know though is that OpenBSD has

a very strick way of handling of the clibrary, what version of Obsd do you use?

OpenBSD 4.4



On OpenBSD, there is no support for running mixed binaries; it is either 64 bit or 32 bit.  There is no issue with picking the right library version.  I am running the 64 bit OS.



Yes, I do make clean every time.

[TedWalther]

An issue may be not using the correct libmysqlclient.so. It is not always part of the server installation, and perhaps you have a mismatch. I have seen this frequently.

Although mysql doesn't come by default with OpenBSD, there is a standard package for OpenBSD.  In OpenBSD 4.4 it installs the libraries like this:

Code Select Expand


/usr/local/lib/libmysqlclient.so.19.0
/usr/local/lib/libmysqlclient_r.so.19.0
/usr/local/lib/mysql/libmysqlclient.a
/usr/local/lib/mysql/libmysqlclient.la
/usr/local/lib/mysql/libmysqlclient.so.19.0
/usr/local/lib/mysql/libmysqlclient_r.a
/usr/local/lib/mysql/libmysqlclient_r.la
/usr/local/lib/mysql/libmysqlclient_r.so.19.0


So far I've been using /usr/local/lib/mysql/libmysqlclient.so.19.0 as the library to load.  And many functions, such as query, num-rows, num-fields, do succeed.

[Lutz]

The default on your machine seems to be 64-bit, and the 32-bit compile of types.c bombs right away.



I ran OpenBSD 32-bit with MySQL a few years back (2006/7) without a problem, but that was on 32-bit Intel CPUs. As you said: it is either or on OpenBSD, different from other OSs like OSX, Vista, Linux. Its probably due to their philosophy of keeping things safe, straight and simple.



If OpenBSD does not offer 32-bit libs on their 64-bit OS version, we are out of luck :( , unless you could install OpenBSD 32-bit on that machine?



ps: I will not be available until Thursday morning EST



ps2: isn't there a TCP/IP interface available with MySQL?

[TedWalther]

If OpenBSD does not offer 32-bit libs on their 64-bit OS version, we are out of luck :( , unless you could install OpenBSD 32-bit on that machine?



ps2: isn't there a TCP/IP interface available with MySQL?

Is there something in newlisp that depends on 32 bit libs?  Or is it just harder to debug without having them to compare with?  I have access to a 32 bit OpenBSD 4.4 box for testing.  In fact, I justed tested it there, and it works nicely with mysql, no failures evident.



A quick two minute search of the web didn't pull up any information on the wire protocol MySQL uses.  Telnetting to port 3306 locally showed that it isn't a straight text protocol like POP or SMTP.



Will email you more info after lunch.

[TedWalther]

Wow Lutz, you rock!  Will the module updates be incorporated in the 10.0.5 release?  Now I'm wondering how many other modules need similar fixes added.

[Lutz]

The odbc and sqlite3 modules may need to be updated too, if 64-bit libraries exist for these at all. Other modules in the distribution are probably safe to use with 64-bit newLISP unchanged.



The new mysql.lsp module will be posted later, and then it goes into the 10.0.5 maintenance release in May.

[TedWalther]

Yes, there are 64bit libraries for all the standard modules that come with newLISP.  The account is still active if you want to do any updates or testing.



crypto.lsp

* /usr/lib/libcrypto.so.14.0



gmp.lsp

* /usr/local/lib/libgmp.so.7.1



mysql.lsp

* /usr/local/lib/libmysqlclient.so.19.0



odbc.lsp

* /usr/local/lib/libiodbc.so.3.15

* /usr/local/lib/libiodbcinst.so.3.15



sqlite3.lsp

* /usr/local/lib/libsqlite3.so.11.0



unix.lsp

* /usr/lib/libc.so.48.0



zlisp.lsp

* /usr/lib/libz.so.4.1

[Lutz]

I couldn't find any of the following libraries on your system:



* /usr/local/lib/libgmp.so.7.1

* /usr/local/lib/libiodbc.so.3.15

* /usr/local/lib/libiodbcinst.so.3.15

* /usr/local/lib/libsqlite3.so.11.0



perhaps you looked on a 32-bit machine, not on the 64-bit? But I could verify the correct working of libc, libcrypto and libz under 64-bit newLISP.



There is a new 10.0.5 on your system and I accidently removed 10.0.4-local. I hope your local adaptations can quickly be redone.



Changes are made for all the special OpenBSD compiler warnings, but one warning still shows up although already fixed. Perhaps some cacheing issue? Rebooting the machine will probably make it go away.

[TedWalther]

Thanks, I was looking at those libraries on another 64bit OpenBSD machine.  They are now installed on the test box.

[TedWalther]

The warnign was referring to your use of strncat(); openbsd deprecates it as much as strcat() in favor of strlcat (and strlcpy).  I'm now looking into how to incorporate those two functions portably.