Opening Win32 pipe strange behaviour

[pjot]

Hi Lutz,



Long time no speak! In the meantime I have been using newLisp for a couple of programs. I discovered a very strange thing however.



With my GTK-server it is possible now to use Win32 pipes. From the client script point of view, these pipes behave like a file. So in order to communicate, the newLisp script has to open a named pipe for writing, and also another named pipe for reading; this is how I have implemented it in the GTK-server.



With newLisp as client, I have programmed this as follows:



(set 'pipeout (open "\\.\pipe\out" "write"))

(println pipeout)

(sleep 500)

(set 'pipein (open "\\.\pipe\in" "read"))

(println pipein)

(sleep 500)



Now, the strange thing is: the opening as "write" works OK, I receive an integer number as a file handle. But the opening as "read" fails! The 'pipein' variable receives a NIL.



After some testing, I found out that the problem is the "read". Since if I try to open the 'out'-pipe as "read", I get the same problem.



I would like to mention that opening pipes on Linux work OK. Also other type of programming languages work OK with "read" and "write" on the Win32 pipes.



What am I doing wrong with newLisp in Windows?



Regards



Peter

[Lutz]

Nn Linux and other UNIXs the opening/reading and writing of pipes is done with the same functions as for files and named pipes are created using a OS system level call: mkfifo, This is not the case in Win32 where a completely different API is used to create/open/read/write named pipes.



This is the reason that named pipes are only workng on the Linux/UNIX versions of newLISP and will never be implemented on the Win32 version of newLISP.



But you still could use the newLISP 'pipe' functions, but it is not for named pipes, but else a pipe mechanism working like a named pipe.



Another possibility is to import the Win32 functions necessary with the newLISP 'import' function and write a 'namepipe.lsp'  module similar to odbc.lsp etc., but I do not have the time for this at the moment.



The Win32 API's on the system level are so much different from the APIs used on the UNIX family of operating systems. And it is just too much work to deliever this in a transparent manner in newLISP, so that these difference are not visible from a user perspective.



Lutz

[pjot]

Hi Lutz,



Thanks for your reply.



Still, I find it very peculiar that with newLisp an "Open for Writing" seems to work fine with the Windows Named Pipes. Only the "Open for Reading" is malfunctioning!



You say that Named Pipes on Windows never will be implemented; I can understand that for the server-side of Named Pipes. Indeed a Named Pipe server on Windows is very different compared to Linux/Unix. The Win32 API for Named Pipes is rather clumsy, if you ask me; I experienced many problems implementing a Named Pipe server in my GTK-server.



But for the client-side access it should be really easy, since a Named Pipe behave like a file. As client, you only have to open this file. The Windows-function for opening files/directorys/disks/pipes/mailslots is in all cases the same "CreateFile" Win32 function. Hence my question: why does opening an existing Named Pipe for "write" works OK with newLisp, and why does opening of an existing Named Pipe for "read" fail?



Other languages like Scriptbasic, VB script, Yabasic etc. can open the GTK-server Named Pipe for "read", so the server-side implemented in the GTK-server appear to work well...?



Regards



Peter



(Actually I am testing on WinXP now.)

[Lutz]

Actually named pipe seems to work perfectly on Win32, at least for me running WinXP home edition. But you have to open for write and read on different programs or newLISP invocations:



In first Command Prompt:



C:newlisp>newlisp

newLISP v8.0.9 Copyright (c) 2004 Lutz Mueller. All rights reserved.



> (open "ptest" "w")

3

> (write-line "hello" 3)

"hello"

>



and in second Command Prompt:





C:newlisp>newlisp

newLISP v8.0.9 Copyright (c) 2004 Lutz Mueller. All rights reserved.



> (open "ptest" "r")

3

> (read-line 3)

"hello"

>



What you can not do In Windows is pipe in and out of a named pipe from the same application.



Lutz



also: when running the same newlisp.exe in a cygwin shell then you can open for both read and write as in Linux, I guess in my example above I am really just sharing a file. Look in the file nl-filesys.c for the function:



int openFile(char * fileName, char * accessMode, char * option)



which is opening files in newLISP

[Lutz]

Here is the trick to make it working opening both, read/write, in the same application of newLISP under Win32 using the 'non-blocking' option:



C:newlisp>newlisp

newLISP v8.0.9 Copyright (c) 2004 Lutz Mueller. All rights reserved.



> (open "ptest" "w")

3

> (open "ptest" "r" "n")    ;; << use the non-blocking option!

4

> (write-line "hi there" 3)

"hi there"

> (read-line 4)

"hi there"

>



Lutz



its also mentioned in the manual for 'open'

[pjot]

Hi Lutz,



Thanks for your replys!



Still it does not work, but I see you use newLisp 8.0.9, while I am using 8.0.6. Maybe this makes a difference? I will upgrade this evening.



Actually, I do not want R/W access to the same pipe. The GTK-server creates 2 independent, individual pipes, a Named Pipe called "in" and a Named Pipe called "out". I have intentionally implemented it this way, since many program languages cannot open a file for R/W at the same time.



So, one pipe must be opened "r" and the other with "w" within the same script. The client script first has to write something to pipe "out", after which the GTK-server puts something to pipe "in".



This last pipe must be opened in "read" mode by the client script, and this is where the trouble starts. If I do this:



(set 'inpi (open "\\.\pipe\in" "r" "n"))

(sleep 500)

(println inpi)



I still receive a NIL. Again, this is an existing pipe, created by the GTK-server. Also first opening this pipe in "w" mode, after that as "r" "n" delivers a nil. I'll come back with testresults for newLisp 8.0.9.



Regards



Peter

[Lutz]

8.0.9 should be the same as 8.0.6 if it comes to opening file/pipes etc. perhaps you want to have a look in tot the 'C' function I am using. As you did the pipe stuff for GTK-Server in 'C', perhaps you spot something regarding the flags I am using:

Code Select Expand


open( fileName, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IREAD | S_IWRITE | blocking ) );


I just realize the non-blocking option is commented out for win32 and further testing shows me that opening for read/write from the same invocation of newlisp works (most) of the time. So on win32 the variable 'blocking' is always 0.



I wonder what kind of flags you use when opening/creating pipes.



Lutz



ps: instead of:



(set 'inpi (open "\\.\pipe\in" "r" "n"))



you could do:



(set 'inpi (open {\.pipein} "r" "n"))



saving some quotes

[pjot]

Well, in my GTK-server I create a Named Pipe for Win32 as follows (API only works on NT/2000/XP):



hPipe2 = CreateNamedPipe(

   "\\.\pipe\in",                   // pipe name

   PIPE_ACCESS_OUTBOUND,       // write access

   PIPE_TYPE_MESSAGE |       // message type pipe

   PIPE_READMODE_MESSAGE |   // message-read mode

   PIPE_WAIT,                // blocking mode

   PIPE_UNLIMITED_INSTANCES, // max. instances  

   MAX_LEN,                  // output buffer size

   MAX_LEN,                  // input buffer size

   NMPWAIT_USE_DEFAULT_WAIT, // client time-out

   NULL);                    // no security attribute





(As you can see, the name of the pipe is hardcoded here, but this is intentionally, since the client script will create a file handle for the pipename anyway.)



As I can see from your source, you use the generic C 'open' function for READ_ONLY access like this (nl-filesys.c line#416):



return(open(fileName, O_RDONLY | O_BINARY | blocking, 0));



As you mention, the variable 'blocking' has the value 0 here, so the relevant flags are O_RDONLY and O_BINARY. This seems perfectly OK. Compared with your open for "write" (line #420):



return(open( fileName, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IREAD | S_IWRITE | blocking ) );



the O_CREAT and O_TRUNC flags are extra, but these concern the creation and truncation of the new file. The Linux manpages mention explicitely that if the file is a FIFO, the O_TRUNC will be ignored. Also it is mentioned that the O_NONBLOCK option only will have effect when using FIFO's!



In other words: it might help that the O_NONBLOCK actually will be used in your sourcecode for Win32. Why did you ever comment it out?



On my side, maybe it helps to create the Named Pipe with R/W access, instead of using "OUTBOUND" only. I am going to test that now.



And thanks for the {} remark! The code will be more readable indeed.

[Lutz]

The Win32 version would not compile with O_NONBLOCK. This constant/option exists only in Linux/UNIX header files, no way to use it on Win32 :-(



Lutz

[pjot]

I think I have found the problem.



Look at this testprogram:



--------------------------------------



#include <stdio.h>

#include <fcntl.h>

#include <sys/stat.h>



int main() {



int n;



printf("Connecting to PIPE 'out'...n");

if ((n = open ("\\.\pipe\out", O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IREAD | S_IWRITE)) < 0){

   printf("Open FAILED!n");

   exit(-1);

}

else printf("Filehandle is: %dn", n);



printf("Connecting to PIPE 'in'...n");

if ((n = open ("\\.\pipe\in", O_RDONLY | O_BINARY, 0)) < 0){

   printf("Open FAILED!n");

   exit(-1);

}

else printf("Filehandle is: %dn", n);



printf("All pipes can connect!n");



return 0;

}



--------------------------------------



You see I use your 'open' constructs to test. When I compile with MinGW 3.1, and test it with my GTK-server, this is the output:



C:Gtk-serverGtk1>pipe

Connecting to PIPE 'out'...

Filehandle is: 3

Connecting to PIPE 'in'...

Filehandle is: 4

All pipes can connect!



So this looks good. But now, when I compile with Borland 5.5, this is the output:



C:Gtk-serverGtk1>pipe

Connecting to PIPE 'out'...

Filehandle is: 3

Connecting to PIPE 'in'...

Open FAILED!



It seems a problem of the Borland compiler. I saw 2 patches for these Free Commandline Tools 5.5, which I will apply and test now.

[pjot]

Hmmm... I applied the 1st patch, recompiled the testprogram, same result... then the 2nd patch, recompiled the testprogram, also same result.



Actually, my GTK-server for Windows itself is compiled using MinGW since I need to do some GTK stuff, and the GTK port for Windows is optimized for MinGW.



I guess we face a compiler incompatibility here?? Though the generic 'open' should work regardless the compiler.

[newdep]

Pjot.. Seems you have your BUG day today... First this Ksh 'alias' memory overflowbug under unix and know this Borland issue... whazzzup ;-)...

[pjot]

Yeah... that was big fun!



-----

alias ls='ls'

for i in `ls`; do echo $ i; done

-----



KSH Stack Overflow!

[Lutz]

Actually I (adamss3 pointed me to it) tried to go with MinGW, expecting it to be more compatible with GCC on Linux, but never could get it compiled, too much stuff missing/different. Perhaps I will try again, but it will take some time. These ports look easy at first, but require tons of testing for subtle differences.



Until then: is it not possible to connect to GTK-Server using normal pipes?



Lutz

[pjot]

Well, in Win32 the TCP interface works fine. Regarding the pipes: I guess I have to use Named Pipes so the client script knows which "files" it has to open. Nameless pipes might work, but then the client script has to be started by the GTK-server. This will deliver some unconvenient usage complications.



Anyway, I myself used Borland many times in the past, but I have found compilation with MinGW more convenient; the source compatibility is much bigger, and also, the resulting binary is smaller when using MinGW. The test program above for example, has a size of 16106 bytes when compiled with Mingw, but a size of 54272 when compiled with Borland; that is more than 3 times bigger.