Text only | Text with Images

newLISP Fan Club

Forum => newLISP and the O.S. => Topic started by: kosh on February 14, 2015, 11:16:01 AM

Title: mingw64 build patch
Post by: kosh on February 14, 2015, 11:16:01 AM
Hi Lutz.



I wrote patch file for builds with MinGW-w64 gcc.

Changes are following:



- Define UINT as pointer size integer (stdint.h, uintptr_t)

- Use type conversion macros for printf (inttypes.h, PRIdPTR)

- Auto detect 64-bit memory model (LP64/LLP64) using C pre-defined macros

- nl-sock.c: bugfix non initialized var DefaultIn



Patch files can be viewed on GitHub:

https://github.com/kosh04/newlisp/compare/feature/mingw-w64



(p.s. BBS attachment fails with message The extension diff is not allowed.)



Build and Test (`make testall`) passed under the following platform:



- Win7 Home amd64

-- TDM-GCC (mingw-w64 gcc 4.9.1)

-- MinGW gcc 4.8.1

-- cygwin32 gcc 4.9.2

-- cygwin64 gcc 4.9.2

- Mac OSX clang



However, this patch have some problems:



- stdint.h, inttypes.h headers available in ISO C99. but newLISP C source seem to C90.

- win32-path.c: an incomplete compatibility between `struct stat` and `struct _stat`.

- nl-sock.c: UINT type is name conflict in ws2tcpip.h.

  This patch provides the temporary workaround using push_macro/pop_macro.

  If you don't like this way, better to rename the UINT to another name (e.g. `typedef uintptr_t nl_int;`)



All reviews are welcome. Thanks.
Title: Re: mingw64 build patch
Post by: kosh on February 14, 2015, 11:32:20 AM
Patch files are here:

https://gist.github.com/kosh04/6a9ff701d5d6358d3ad9

https://rawgit.com/kosh04/6a9ff701d5d6358d3ad9/raw/



It based newlisp v.10.6.2
Title: Re: mingw64 build patch
Post by: Lutz on February 15, 2015, 08:17:09 AM
Thanks for this great work! Everything worked right away on all OS.



All integrated for 10.6.3 here: http://www.newlisp.org/downloads/development/inprogress/

(several reposts after initial post)



Tested on: OS X 10.10, Ubuntu Linux 14.04, FreeBSD 9.3, OpenBSD 5.4 and Windows XP sp2 with mingw32.



I wonder what the speed comparison is between the 32-bit and 64-bit versions of newLISP on Windows.
Title: Re: mingw64 build patch
Post by: kosh on February 16, 2015, 11:48:44 AM
Thanks Lutz for merge patch.



I found a missing code in newlisp.h (newlisp-10.6.3.tgz)


Code Select
#undef NEWLISP64
#if defined(_LP64) || defined(_WIN64)
#define NEWLISP64
#endif


and qa-specific-tests/qa-local-domain (for `make testall`)
Code Select
diff --git a/qa-specific-tests/qa-local-domain b/qa-specific-tests/qa-local-domain
index caa5901..49b41f7 100755
--- a/qa-specific-tests/qa-local-domain
+++ b/qa-specific-tests/qa-local-domain
@@ -6,6 +6,10 @@
 (println)
 (println "Testing UNIX local domain sockets")
 
+(when (find ostype '("Win32" "OS/2"))
+  (println "not tested on " ostype)
+  (exit))
+
 (define (listener path)
  (set 'lsock (net-listen path))
  (set 'csock (net-accept lsock))


NEWLISP64 is defined in files makefile_* in this time.

It will more better way to use predefined compiler macros.


Code Select
# check predefined macros
$ echo | gcc -m32 -dM -E - | grep WIN64

$ echo | gcc -m64 -dM -E - | grep WIN64
#define _WIN64 1
#define __WIN64 1
#define WIN64 1
#define __WIN64__ 1


Quote from: "Lutz"I wonder what the speed comparison is between the 32-bit and 64-bit versions of newLISP on Windows.


FYI, my AppVeryor's build log info is the following (mingw cross-compile on cygwin):

https://ci.appveyor.com/project/kosh04/newlisp/build/1.0.62


QuoteFree plans run builds on VM instances with 1 virtual core, 1.75 GB of memory and 127 GB of disk space (Windows Azure "Small").

http://www.appveyor.com/docs/build-configuration

Code Select
# 32-bit
>>>>> total time: 14779.53999999999
>>>>> Performance ratio: 5.70 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)

# 64-bit
>>>>> total time: 12015.73600000001
>>>>> Performance ratio: 4.64 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)


My build environment info will be posted at a later.



Regards.
Title: Re: mingw64 build patch
Post by: Lutz on February 17, 2015, 12:34:26 PM
Thanks Kosh. The speed difference between Win32 and Win64 seems to be similar to other platforms.



Currently the selection between 32 and 64 bit is made by the configure script, which then copies one of the makefile_xxx to makefile_build. This gives people the possibility to decide for a different memory model by picking a different makefile_xxxx. This happens often when building for embedded or other resource constrained systems. So, at this point automatic NEWLISP64 detection will not happen yet, but may be added in the future.



The configure-alt script from Ted Walther works a bit different creating makefile_build by building it from scratch and doing more automation in discovering features on different OS platforms. Win64 capability has not yet been added, but that should be easy.



I added detection of Windows 64-bit to configure but currently don't have the possibility to test it, as my Windows development platforms currently is only 32-bit. I hope the script and new makefiles work for you.



Many qa-specific-test/qa-xxx files have been changed testing for Win64, where appropriate. A few makefiles for mingw64 have been added and should be picked out automatically by the configure script. After fresh install of the source distribution or after a 'make clean' a simple 'make' will call the configure script first then make the newlisp executable.



Let me know if all this works when using mingw64 tools.



Ps: some more fixes uploaded at 2015-2-18 15:47 UTC
Title: Re: mingw64 build patch
Post by: kosh on February 18, 2015, 12:43:04 PM
makefile_mingwLLP64_utf8 is here:


Code Select
# makefile for newLISP v. 10.x.x on MinGW-w64 with UTF-8 support

OBJS = newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o
nl-import.o nl-xml-json.o nl-web.o nl-matrix.o nl-debug.o pcre.o nl-utf8.o win-util.o win-path.o

CFLAGS = -Wall -Wno-uninitialized -Wno-long-long -O1 -g
CPPFLAGS = -DWIN_64 -DSUPPORT_UTF8
#CPPFLAGS = -DWIN_32 -NEWLISP64 -DSUPPORT_UTF8
LDLIBS = -lws2_32

CC = gcc
STRIP = strip

default: newlisp.exe

newlisp.exe: $(OBJS)
$(CC) $(LDFLAGS) $^ $(LDLIBS) -o $@
$(STRIP) $@

newlisp.dll: LDFLAGS += -static-libgcc -Wl,--enable-stdcall-fixup
newlisp.dll: $(OBJS) win-dll.o win-dll.def
$(CC) -shared $(LDFLAGS) $^ $(LDLIBS) -o $@

$(OBJS): primes.h protos.h newlisp.h makefile_mingwLLP64_utf8


This makefile can also make DLL file:
Code Select
make -f makefile_mingwLLP64_utf8 newlisp.dll

---

Unfortunatly, new newlisp-10.6.3.tgz compile with errors:
Code Select
gcc -Wall -Wno-uninitialized -Wno-long-long -O1 -g -DWIN_64 -DSUPPORT_UTF8  -c -o nl-sock.o nl-sock.c
In file included from nl-sock.c:29:0:

C:/MinGW64/x86_64-w64-mingw32/include/ws2tcpip.h:160:1: error: size of unnamed array is negative

 C_ASSERT(sizeof(IN_PKTINFO)==8);

 ^

C:/MinGW64/x86_64-w64-mingw32/include/ws2tcpip.h:167:1: error: size of unnamed array is negative

 C_ASSERT(sizeof(IN6_PKTINFO)==20);

 ^

make: *** [nl-sock.o] Error 1


in nl-sock.c: `#define UINTP uintptr_t` is not ineffective. I recommend to use push_macro/pop_macro, or include newlisp.h after ws2tcpip.h if you don't rename the type name UINT.



---

I hope the value ostype be always "Win32".

This value should not be distinguished by 32/64 bit as well as in other platform. To determine the 64-bit platform can use sys-info.
Title: Re: mingw64 build patch
Post by: Lutz on February 19, 2015, 07:57:21 AM
Thanks Kosh, I will continue to be integrate Windows 64-bit development when I have a development environment similar to yours, with both mingw32 and mingw64 together and the latest stable release.



The logical would be to use a new ostype string "Windows", although it would break backwards compatibility with code using "Win32". Using both "Win32" and "Win64" would leave compatibility but be impractical when running old "Win32" scripts on "Win64" installations, although probably those scripts are compatible.



As soon, as I have a new development environment installed, a decision will be made.
Title: Re: mingw64 build patch
Post by: kosh on March 19, 2015, 01:31:43 PM
Lutz, you have completed the setup of new Windows machine?



As an aside, http://tdm-gcc.tdragon.net/ provide "TDM32 Packages". so it works on 32-bit Windows XP.

Of course, a generated newlisp.exe runs on 64-bit Windows.
Title: Re: mingw64 build patch
Post by: Lutz on March 19, 2015, 04:52:31 PM
Not until April, I will have access to a Windows 7 64-bit machine, but thanks for the tip using TDM gcc. That will make the installation a lot easier. I understand that you can cross compile to 64-bint on a 32-bit machine, but I don't have enough space too install, so it has to wait.
Title: Re: mingw64 build patch
Post by: Lutz on April 02, 2015, 06:25:06 AM
A Windows 64-bit version made with tdm gcc 64 is now working, but only for IPv4. Perhaps IPv6 is not supported in the tdm version of mingw gcc? Hve not looked into it yet.



The following makefiles have been added:  makefile_tdmgcc64 and  makefile_tdmgcc64dll and the builds pass make testall except with qa-net6.



Other makefiles including UTF8 and extended FFI using libffi will follow during the next days.



Currently I am still using Win32 and Win64 for the newLISP variable ostype. May be this will change to Windows (64/32 info in in sys-info), as Kosh is suggesting. When the other 64-bit flavors are ready, a .exe will published too for people to test, who can not compile themselves.
Title: Re: mingw64 build patch
Post by: kosh on April 02, 2015, 08:10:36 AM
> but only for IPv4. Perhaps IPv6 is not supported in the tdm version of mingw gcc? Hve not looked into it yet.



Maybe, It because you use variable `defaultIn` without zero-initialization.



https://github.com/kosh04/newlisp/compare/feature/mingw-w64#diff-ab3071b4db4911dec3a809c84f1a5cbfL159



initDefaultInAddr() in nl-sock.c
Code Select
-defaultIn = allocMemory(defaultInLen);
+defaultIn = callocMemory(defaultInLen);
Title: Re: mingw64 build patch
Post by: Lutz on April 02, 2015, 09:06:26 AM
Yes, exactly, thanks Kosh! Updated here: http://www.newlisp.org/downloads/development/inprogress/
Title: Re: mingw64 build patch
Post by: Lutz on April 02, 2015, 09:18:22 AM
... and updated again.
Title: Re: mingw64 build patch
Post by: Lutz on April 09, 2015, 01:10:21 AM
64-bit 10.6.3 newLISP executables with and without UTF8 suppport can be found here:



http://www.newlisp.org/downloads/development/inprogress



Both .exe are compiled on Windows 7 using mingw64 gcc.
Title: Re: mingw64 build patch
Post by: xytroxon on April 11, 2015, 10:11:30 AM
Hi Lutz!



This Win64 version is noticeably faster than the Win32 version - like 7 times faster for some string operations!!!



Then I noticed I was having some missing output using either newlisp64.exe  or  newlisp64utf8.exe.



Assuming an 80 character x 25 line console display - the following code should display 6000 chars in three sections of 2000 chars each of "+", "-" and "*". But the Win64 newlisp versions will display the first 2000 char "+" section,  and of the second 2000 char "-" section, only about the first 50 chars are displayed, and nothing of the third 2000 char "*" section. Sometimes the end of the last line will also show 5 or 6 random chars from the 0 to 31 and 127 to 255 ranges of ascii characters - like a wild pointer running on into another memory variable.



The string length correctly shows all 6000 chars of the "test" string being present, and this code works fine with the last Win32 release of newlisp.



Code was tested on both "console2" and "conemu" Windows console emulator programs with simular results.


Code Select

(setq a (string (dup "+" (* 80 25))))
(setq b (string (dup "-" (* 80 25))))
(setq c (string (dup "*" (* 80 25))))
(setq test (string a b c))

(print (length test) "n" test)
(exit)


-- xytroxon
Title: Re: mingw64 build patch
Post by: xytroxon on April 11, 2015, 01:31:34 PM
Doing a write-file of the "test" string shows no premature shortening of the string or any extra "odd" chars in the file on disk.
Code Select

(write-file "test.txt" test)


But reading and printing the file has the same display failure.
Code Select

(print (read-file "test.txt"))


So maybe the problem lies within the print function.



--xytroxon
Title: Re: mingw64 build patch
Post by: TedWalther on April 11, 2015, 10:15:41 PM
I remember a few years ago, Phillip Whelan was able to very quickly locate and patch a 64 bitness problem in newLisp.  Wish he was still around, this sounds like a 64 bitness type of problem, right up his alley.
Title: Re: mingw64 build patch
Post by: Lutz on April 12, 2015, 12:08:16 AM
Cannot repeat this problem. Your test code or printing any other string >= 6000 characters always works on Windows 7 Home Premium SP1 command shell  cmd.exe or using the MSYS/MinGW bash shell. I wonder if other Windows 7 users can repeat this problem. What version of Windows are you using?
Title: Re: mingw64 build patch
Post by: HPW on April 12, 2015, 01:43:07 AM
Hello,



I tried it on my win7 home premium tablet.

When I paste it line by line into the newlisp Shell it works as expected.



But when i make a file Test.lsp with:
Code Select

(setq a (string (dup "+" (* 80 25))))
(setq b (string (dup "-" (* 80 25))))
(setq c (string (dup "*" (* 80 25))))
(setq test (string a b c))

(silent(print (length test) "n" test))


and do a (load "Test.lsp") in a fresh started Shell I get:


Code Select

newLISP v.10.6.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (load "Test.lsp")
6000
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------¦.6Å-ï?


This seems to be Buggy.



Edit: The 32 bit Exe does this correctly.



Regards

Hans-Peter
Title: Re: mingw64 build patch
Post by: Lutz on April 12, 2015, 04:19:36 AM
Yes, now I can repeat it too when loading the test code from a file. I could not repeat it when testing the code interactively, but it also fails when interactively when using silent.



It also fails on the 32-bit version in 10.6.3. It seems that this has nothing to too with 32-bit versus 64-bit, but is related to other changes in 10.3.6 development. I am glad we are catching this before 10.6.3 goes out of "inprogress".
Title: Re: mingw64 build patch
Post by: kosh on April 12, 2015, 06:02:49 AM
I also confirmed this IO problem.


Code Select
$ newlisp64.exe -n
> (setf str (string (dup {X} 2049) {HELLO}))
[text]XXXX...<repeat 2049 times>HELLO[/text]
> (println str)
XXXX...<repeat 2049 times>


It seems that cause internal output function `my_vasprintf` (in nl-filesys.c, called from varPrintf)



`vasprintf` has been implemented in mingw64 in stdio.h (but mingw32 yet).

then, It could be fixed in this way:


Code Select
/* newlisp.h */
#ifdef WINDOWS
#ifndef __MINGW64__ /* macros depends on mingw implementation */
# define MY_VASPRINTF
# define vasprintf my_vasprintf
#endif
...
#endif


Regards.
Title: Re: mingw64 build patch
Post by: Lutz on April 12, 2015, 07:40:13 AM
Yes, it was vasprintf, thanks Kosh:



http://www.newlisp.org/downloads/development/inprogress/
Title: -
Post by: ValentiMr on June 29, 2022, 04:33:18 PM
Attached is a patch to fix the test build on head.  All it does it make it so the tests compile:  they still don't pass at least they don't pass on mac os x.



Also, I've noticed a bunch of build issues on Mac OS X.  Anyone mind if I fix those?




Text only | Text with Images