mingw64 build patch

[kosh]

Hi Lutz.



I wrote patch file for builds with MinGW-w64 gcc.

Changes are following:



- Define UINT as pointer size integer (stdint.h, uintptr_t)

- Use type conversion macros for printf (inttypes.h, PRIdPTR)

- Auto detect 64-bit memory model (LP64/LLP64) using C pre-defined macros

- nl-sock.c: bugfix non initialized var DefaultIn



Patch files can be viewed on GitHub:

https://github.com/kosh04/newlisp/compare/feature/mingw-w64">https://github.com/kosh04/newlisp/compa ... /mingw-w64">https://github.com/kosh04/newlisp/compare/feature/mingw-w64



(p.s. BBS attachment fails with message The extension diff is not allowed.)



Build and Test (`make testall`) passed under the following platform:



- Win7 Home amd64

-- TDM-GCC (mingw-w64 gcc 4.9.1)

-- MinGW gcc 4.8.1

-- cygwin32 gcc 4.9.2

-- cygwin64 gcc 4.9.2

- Mac OSX clang



However, this patch have some problems:



- stdint.h, inttypes.h headers available in ISO C99. but newLISP C source seem to C90.

- win32-path.c: an incomplete compatibility between `struct stat` and `struct _stat`.

- nl-sock.c: UINT type is name conflict in ws2tcpip.h.

  This patch provides the temporary workaround using push_macro/pop_macro.

  If you don't like this way, better to rename the UINT to another name (e.g. `typedef uintptr_t nl_int;`)



All reviews are welcome. Thanks.

[kosh]

Patch files are here:

https://gist.github.com/kosh04/6a9ff701d5d6358d3ad9">https://gist.github.com/kosh04/6a9ff701d5d6358d3ad9

https://rawgit.com/kosh04/6a9ff701d5d6358d3ad9/raw/">https://rawgit.com/kosh04/6a9ff701d5d6358d3ad9/raw/



It based newlisp v.10.6.2

[Lutz]

Thanks for this great work! Everything worked right away on all OS.



All integrated for 10.6.3 here: http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/

(several reposts after initial post)



Tested on: OS X 10.10, Ubuntu Linux 14.04, FreeBSD 9.3, OpenBSD 5.4 and Windows XP sp2 with mingw32.



I wonder what the speed comparison is between the 32-bit and 64-bit versions of newLISP on Windows.

[kosh]

Thanks Lutz for merge patch.



I found a missing code in newlisp.h (newlisp-10.6.3.tgz)

Code Select Expand

#undef NEWLISP64
#if defined(_LP64) || defined(_WIN64)
#define NEWLISP64
#endif


and qa-specific-tests/qa-local-domain (for `make testall`)

Code Select Expand

diff --git a/qa-specific-tests/qa-local-domain b/qa-specific-tests/qa-local-domain
index caa5901..49b41f7 100755
--- a/qa-specific-tests/qa-local-domain
+++ b/qa-specific-tests/qa-local-domain
@@ -6,6 +6,10 @@
 (println)
 (println "Testing UNIX local domain sockets")
 
+(when (find ostype '("Win32" "OS/2"))
+  (println "not tested on " ostype)
+  (exit))
+
 (define (listener path)
  (set 'lsock (net-listen path))
  (set 'csock (net-accept lsock))


NEWLISP64 is defined in files makefile_* in this time.

It will more better way to use predefined compiler macros.

Code Select Expand

# check predefined macros
$ echo | gcc -m32 -dM -E - | grep WIN64

$ echo | gcc -m64 -dM -E - | grep WIN64
#define _WIN64 1
#define __WIN64 1
#define WIN64 1
#define __WIN64__ 1


I wonder what the speed comparison is between the 32-bit and 64-bit versions of newLISP on Windows.

FYI, my AppVeryor's build log info is the following (mingw cross-compile on cygwin):

https://ci.appveyor.com/project/kosh04/newlisp/build/1.0.62">https://ci.appveyor.com/project/kosh04/ ... ild/1.0.62">https://ci.appveyor.com/project/kosh04/newlisp/build/1.0.62

Free plans run builds on VM instances with 1 virtual core, 1.75 GB of memory and 127 GB of disk space (Windows Azure "Small").

http://www.appveyor.com/docs/build-configuration">http://www.appveyor.com/docs/build-configuration

Code Select Expand

# 32-bit
>>>>> total time: 14779.53999999999
>>>>> Performance ratio: 5.70 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)

# 64-bit
>>>>> total time: 12015.73600000001
>>>>> Performance ratio: 4.64 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)


My build environment info will be posted at a later.



Regards.

[Lutz]

Thanks Kosh. The speed difference between Win32 and Win64 seems to be similar to other platforms.



Currently the selection between 32 and 64 bit is made by the configure script, which then copies one of the makefile_xxx to makefile_build. This gives people the possibility to decide for a different memory model by picking a different makefile_xxxx. This happens often when building for embedded or other resource constrained systems. So, at this point automatic NEWLISP64 detection will not happen yet, but may be added in the future.



The configure-alt script from Ted Walther works a bit different creating makefile_build by building it from scratch and doing more automation in discovering features on different OS platforms. Win64 capability has not yet been added, but that should be easy.



I added detection of Windows 64-bit to configure but currently don't have the possibility to test it, as my Windows development platforms currently is only 32-bit. I hope the script and new makefiles work for you.



Many qa-specific-test/qa-xxx files have been changed testing for Win64, where appropriate. A few makefiles for mingw64 have been added and should be picked out automatically by the configure script. After fresh install of the source distribution or after a 'make clean' a simple 'make' will call the configure script first then make the newlisp executable.



Let me know if all this works when using mingw64 tools.



Ps: some more fixes uploaded at 2015-2-18 15:47 UTC

[kosh]

makefile_mingwLLP64_utf8 is here:

Code Select Expand

# makefile for newLISP v. 10.x.x on MinGW-w64 with UTF-8 support

OBJS = newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o
nl-import.o nl-xml-json.o nl-web.o nl-matrix.o nl-debug.o pcre.o nl-utf8.o win-util.o win-path.o

CFLAGS = -Wall -Wno-uninitialized -Wno-long-long -O1 -g
CPPFLAGS = -DWIN_64 -DSUPPORT_UTF8
#CPPFLAGS = -DWIN_32 -NEWLISP64 -DSUPPORT_UTF8
LDLIBS = -lws2_32

CC = gcc
STRIP = strip

default: newlisp.exe

newlisp.exe: $(OBJS)
$(CC) $(LDFLAGS) $^ $(LDLIBS) -o $@
$(STRIP) $@

newlisp.dll: LDFLAGS += -static-libgcc -Wl,--enable-stdcall-fixup
newlisp.dll: $(OBJS) win-dll.o win-dll.def
$(CC) -shared $(LDFLAGS) $^ $(LDLIBS) -o $@

$(OBJS): primes.h protos.h newlisp.h makefile_mingwLLP64_utf8


This makefile can also make DLL file:

Code Select Expand

make -f makefile_mingwLLP64_utf8 newlisp.dll

---

Unfortunatly, new newlisp-10.6.3.tgz compile with errors:

Code Select Expand

gcc -Wall -Wno-uninitialized -Wno-long-long -O1 -g -DWIN_64 -DSUPPORT_UTF8  -c -o nl-sock.o nl-sock.c
In file included from nl-sock.c:29:0:

C:/MinGW64/x86_64-w64-mingw32/include/ws2tcpip.h:160:1: error: size of unnamed array is negative

 C_ASSERT(sizeof(IN_PKTINFO)==8);

 ^

C:/MinGW64/x86_64-w64-mingw32/include/ws2tcpip.h:167:1: error: size of unnamed array is negative

 C_ASSERT(sizeof(IN6_PKTINFO)==20);

 ^

make: *** [nl-sock.o] Error 1


in nl-sock.c: `#define UINTP uintptr_t` is not ineffective. I recommend to use push_macro/pop_macro, or include newlisp.h after ws2tcpip.h if you don't rename the type name UINT.



---

I hope the value ostype be always "Win32".

This value should not be distinguished by 32/64 bit as well as in other platform. To determine the 64-bit platform can use sys-info.

[Lutz]

Thanks Kosh, I will continue to be integrate Windows 64-bit development when I have a development environment similar to yours, with both mingw32 and mingw64 together and the latest stable release.



The logical would be to use a new ostype string "Windows", although it would break backwards compatibility with code using "Win32". Using both "Win32" and "Win64" would leave compatibility but be impractical when running old "Win32" scripts on "Win64" installations, although probably those scripts are compatible.



As soon, as I have a new development environment installed, a decision will be made.

[kosh]

Lutz, you have completed the setup of new Windows machine?



As an aside, http://tdm-gcc.tdragon.net/">http://tdm-gcc.tdragon.net/ provide "TDM32 Packages". so it works on 32-bit Windows XP.

Of course, a generated newlisp.exe runs on 64-bit Windows.

[Lutz]

Not until April, I will have access to a Windows 7 64-bit machine, but thanks for the tip using TDM gcc. That will make the installation a lot easier. I understand that you can cross compile to 64-bint on a 32-bit machine, but I don't have enough space too install, so it has to wait.

[Lutz]

A Windows 64-bit version made with tdm gcc 64 is now working, but only for IPv4. Perhaps IPv6 is not supported in the tdm version of mingw gcc? Hve not looked into it yet.



The following makefiles have been added:  makefile_tdmgcc64 and  makefile_tdmgcc64dll and the builds pass make testall except with qa-net6.



Other makefiles including UTF8 and extended FFI using libffi will follow during the next days.



Currently I am still using Win32 and Win64 for the newLISP variable ostype. May be this will change to Windows (64/32 info in in sys-info), as Kosh is suggesting. When the other 64-bit flavors are ready, a .exe will published too for people to test, who can not compile themselves.

[kosh]

> but only for IPv4. Perhaps IPv6 is not supported in the tdm version of mingw gcc? Hve not looked into it yet.



Maybe, It because you use variable `defaultIn` without zero-initialization.



https://github.com/kosh04/newlisp/compare/feature/mingw-w64#diff-ab3071b4db4911dec3a809c84f1a5cbfL159">https://github.com/kosh04/newlisp/compa ... 1a5cbfL159">https://github.com/kosh04/newlisp/compare/feature/mingw-w64#diff-ab3071b4db4911dec3a809c84f1a5cbfL159



initDefaultInAddr() in nl-sock.c

Code Select Expand

-defaultIn = allocMemory(defaultInLen);
+defaultIn = callocMemory(defaultInLen);

[Lutz]

Yes, exactly, thanks Kosh! Updated here: http://www.newlisp.org/downloads/development/inprogress/">http://www.newlisp.org/downloads/develo ... nprogress/">http://www.newlisp.org/downloads/development/inprogress/

[Lutz]

... and updated again.

[Lutz]

64-bit 10.6.3 newLISP executables with and without UTF8 suppport can be found here:



http://www.newlisp.org/downloads/development/inprogress">http://www.newlisp.org/downloads/development/inprogress



Both .exe are compiled on Windows 7 using mingw64 gcc.

[xytroxon]

Hi Lutz!



This Win64 version is noticeably faster than the Win32 version - like 7 times faster for some string operations!!!



Then I noticed I was having some missing output using either newlisp64.exe  or  newlisp64utf8.exe.



Assuming an 80 character x 25 line console display - the following code should display 6000 chars in three sections of 2000 chars each of "+", "-" and "*". But the Win64 newlisp versions will display the first 2000 char "+" section,  and of the second 2000 char "-" section, only about the first 50 chars are displayed, and nothing of the third 2000 char "*" section. Sometimes the end of the last line will also show 5 or 6 random chars from the 0 to 31 and 127 to 255 ranges of ascii characters - like a wild pointer running on into another memory variable.



The string length correctly shows all 6000 chars of the "test" string being present, and this code works fine with the last Win32 release of newlisp.



Code was tested on both "console2" and "conemu" Windows console emulator programs with simular results.

Code Select Expand


(setq a (string (dup "+" (* 80 25))))
(setq b (string (dup "-" (* 80 25))))
(setq c (string (dup "*" (* 80 25))))
(setq test (string a b c))

(print (length test) "n" test)
(exit)


-- xytroxon