Problem with UDP

[newdep]

Hello Lutz,



Well i came to the status on where im porting some udp tools to newlisp and im running in to a problem...



The net-send-udp and net-receive-udp are actualy oke but not 100% according to the "protocol" ;-)



What is the case? ->



If an UDP packet is send to the remote it sends with it the source port, the remote server will reply to that source port to be able to communicate.

The newlisp  TCP stack is implemented that way, but the UDP stack not.





So what is missing? ->



Actualy the udp Stack needs the same behaviour as the TCP stack, to

be able to communicate towards a server on the current sessions and that

is a complete protocol.



It is a matter of Async communication the same as TCP works. Every UDP

sessions needs to have a (net-sessions-udp) too.



Also every UDP sessions must be able to grab the remote port , that is a

(net-peer 16)  for UDP that return the remote ip and port...



I hope you are able to rewrite/change the net-send-udp & net-receive-udp because the current implementation has its limits now.



I think its not a big rewrite, but you know your code better that anyone, I hope you consider the above... :)



the impact would be on:

(net-send-udp)

(net-receive-udp)

(net-sessions) -> (net-sessions-udp)

(net-peer)

(net-local)

(net-peek) "probably wont work on udp, not sure anymore"

(net-select)



If the above it changed you will have a fully UDP async communication way,

the same as TCP works, accept for the network-Timeout in UDP which is buildin into the (net-receive-udp) already OR build be used inside (net-select) !!!



Keep on lisping..

Regards, Norman

[Lutz]

By definition the UDP (User Datagram Protocol) does not have any sender/receiver handshakig protocol like Tcp does. The send goes into "empty space" and if nobody is listening on the receiving port of the receipient addrress or ip broadcast-space, the packet is lost.



For this reason there is no peek or select for pending bytes to read by UDP. If nobody is listening on the receiving end the packed goes into never-never land.



A UDP packed contains (optional no implemented by all OSs) the port number of the sender. It is available on Windows, Linux and BSD and it could be returned theoretically with the remote ip address on reception. But this port numer is deliberately chosen by the sender. On Linux it seems to be 32768 (hex 8000) most of the time, but on BSD it is jumping around and on Windows counting up starting with some deliberate value. So there is not much sense in returning it to the user, because it might not be available the next moment.



UDP is meant to be a very low level protocol and fast for all of the above reasons. You have to implement any kind of handshaking your self as part of the data you send, or use TCP right away.



Lutz

[newdep]

Hello Lutz,



Yes your completly right on UDP..



Still making client-server independent communication possible there

should be a way for the (net-receive-udp) to know what the source ip & port is.



a (net-peer) on udp could return this. And a (net-receive-udp)

could be provided with a return of the source port?

or perhaps a (net-peer-udp) ?





If the above is possible ,(even net-select would be great here, still a next step)

then the UDP implementation is more complete and able to communicate independent.



Indeed the UDP protocol behavior stays the same but the functions in newlisp

related to UDP would be far more flexible.



I.e.



Currently im unable to setup a UDP listener because im not sure what

the local port was my (net-send-udp) send the packet on, so i cant listen

localy on the port for incoming data. protocols like tftp and Some Game protocols do so..



Regards, Norman

[Lutz]

The source ip is contained in the list net-receive-udp returns as the second field. When you talk bag to the sender you use a so called "well known" port which is agreed upon prior. Normally this is a pair one listen port for the client and another listen port for the sender.



The sender specifies the address and port of the receiver is does not (cannot) specify the outgoing port, which is picked delieberately by the lower level socket routines. The receiver listens on that port targeted by the sender and knows the ip-address where a packet is coming from. The receiver then sends back a message on the "well known'' agreed upon port where previous sender now is listening on.



Again there is no select, net-receive-udp listens and immedeately returns the UDP packet when it arrives.



A net-peer is not necessary because you know already the address where a packet comes from, from the list returned by net-receive-udp.



The outgoing port used by net-send-udp is picked by the socket implementation of the OS and there is no way to specify those. You can only specify the port you are sending to.



The port targeted by the sender is agreed upon prior in the higher level protocol you are using those ports are kneown as "well known ports" and are listed in the /etc/services table. You can use net-service to query them, i.e.:



(net-service "domain" "udp") = 53



Lutz

[newdep]

Hello Lutz,



Yes we are talking about the same issue..let me go one step further...



if server always send back to the client-source-port then the following

will never work in newlisp for the client to receive the data from the server ->



client - > (net-send-udp "127.0.0.1" 2222 "im going back" )

client - > (net-receive-udp  ?????? 512 )

server -> (net-send-udp "127.0.0.1" "remote-source-port" "im going back" )



how can (net-receive-udp) know what its local port was?



Anyway.. i tested in my C code and it works nicely... If the OS does not

provide a source port in de Datagram (because its optional indeed) then

its simply "0" (and the OS needs a fixup ;-)



I hope you think about it with a nice cold beer ;-)



Regards, Norman.

[pjot]

One night away from holidays... but still hacking. Actually I am implementing the UDP protocol for the GTK-server. I've got it running with the following GNU AWK script:



-------------------

BEGIN{

system("gtk-server udp localhost:50000 &")

# Wait for the server to initialize

pause(1)

# Setup UDP socket to server

GTK = "/inet/udp/0/localhost/50000"

# Now define the GUI

print "gtk_init(NULL, NULL)" |& GTK; GTK |& getline

print "gtk_window_new(0)" |& GTK; GTK |& getline WINDOW

....etc

-------------------



However, with newLisp I have to wait for a UDP packet first, before I can send something to the server (see documentation example with "net-receive-udp").  I wonder why this is the case with newLisp. Why is it not possible to send to a UDP port right away?



If I do this anyway, the "net-receive-udp" command produces a NIL.



Is there a way to get around this?



Peter

[newdep]

Hiya Pjot,



with UDP you can send whatever you want, but the server needs to

be listening to be able to get it..With newlisp you can send! but you cannot

wait for data at the same time, that is because its not async. :-)





PS: its a coincidence that we both work on a UDP problem Lutz ;-)



Norman.

[newdep]

Pjot,



Yes if you are using th Gtk-server to listen and to write on 1 port then your client

on the SAME machine will fail, because you cannot have 2 server on 1 machien running with the same listening port.



Actualy that is my point too (indirectly), see above..



Hope Lutz likes beer ;-)



Regards, Norman.

[pjot]

Actually it is very easy. The GTK-server acts as UDP server, waiting for incoming packets. As soon as the UDP server receives something, it will reply. Then it returns to wait state again. BUT: this all happens on the same port. The port is determined by the client script, so the UDP-server will know to which port it has to send it's packets back.



It is almost like TCP, with the difference that it is UDP... ;-) Again, this is how the AWK demo works, so I know this works... Cheers!

[newdep]

Ofcaurse it works ;-)



But i think we need to buy a 6-pack for Lutz first :-)

[pjot]

Funny we work on UDP at the same time... let's buy a 6-pack Belgian beer (10%)... ;-)

[Lutz]

The Belgian would be fine, but the dark one please! ;-)



somehow still I don't get what you guys are up to? If you send a packet with



(net-send-udp receiver-ip receiver-listen-port message)



you have the receiver at receiver port doing:



(net-receive-udp receiver-listen-port max-bytes) => (sender-ip sender-message)





There is no such thing as:



(net-send-udp receiver-ip receiver-listen-port port-where-i-will-listen-for-your-answer message)



sender would have to make port-where-i-will-listen-for-your-answer part of the message itself. Because when you set up your socket for sending you can only specify the target port:



dest_sin.sin_port = htons((u_short)remotePort); /* the target/receiver port */

dest_sin.sin_family = AF_INET;



bytesSent = sendto(sock, buffer, size, NO_FLAGS_SET,

        (struct sockaddr *)&dest_sin, sizeof(dest_sin));



Perhaps if you explain me what you are doing in 'C' I can understand?



Lutz

[pjot]

In my situation, the newLisp script acts as a client, while the GTK-server is the UDP server. So there is an interaction between newLisp and C.



OK here my beta version with UDP (just the mainloop):



------------------------

while (1){

   if ((numbytes=recvfrom(sockfd, buf, MAX_LEN - 1, 0, (struct sockaddr *)&their_addr, &addr_len)) > 0) {



      /* Terminate incoming string */

      buf[numbytes] = '';



      /* Only determine values when empty */

      if (se == NULL) {

         se = gethostbyname(inet_ntoa(their_addr.sin_addr));

         their_addr.sin_family = AF_INET;      /* host byte order */

         their_addr.sin_port = htons((long)atol(port));  /* short, network byte order */

         their_addr.sin_addr = *((struct in_addr *)se->h_addr);

         memset(&(my_addr.sin_zero), '', 8);     /* zero the rest of the struct */

      }



      <do_something_with_incoming: buf[], send back pointer: *retstr>



      /* Add newline to returned string */

      strcat(retstr, "n");



      /* Now send the result back to the socket */

      if ((numbytes=sendto(sockfd, retstr, strlen(retstr), 0, (struct sockaddr *)&their_addr, sizeof(struct sockaddr))) == -1) {

         Print_Error(2, "Could not send DATAGRAM packet", errno);

      }

   }

   else Print_Error(2, "Could not receive DATAGRAM packet", errno);

}

------------------------



As you can see, I determine the destination port from the port given as argument ('port' is one of the arguments to the GTK-server binary).



I just compiled my beta version with GCC and MinGW, this works. Tested UDP with Rebol in Windows, and the GUI came up.



So the idea is:



loop

client-script: send-udp port

server: receive-udp port

server: send-udp port

client-script: receive-udp port

until callback



The 'port' is in all the lines the same port. With newLisp however, it seems I have to do the following:



client-script: receive-udp port

loop

client-script: send-udp port

server: receive-udp port

server: send-udp port

client-script: receive-udp port

until callback



Now, why do I have to wait for a UDP packet first? Since if I do not, the client script will receive a NIL.



Maybe I am not having the same problem as Norman here; again, I just want to know why my newLisp client script has to wait for a UDP-packet first?

[Lutz]

Ok, now I see what is going on:



In your 'C' example the server receives on a aocket and immedeately sends back the replay on the same socket. You



keep the same socket open like in a TCP client/server session where the listener send back on the connected socket.



In newLISP the net-send-udp closes the socket after sending and will not try to receive back on it after response.



Your example:



loop

client-script: send-udp port

server: receive-udp port

server: send-udp port

client-script: receive-udp port

until callback



Actually works with newLISP if both client and server are newLISP. If one of them is GTK-server then



GTK-server tries to send back on the socket wher it received, but that connection is already closed. That is



fundamentally a very different thing.



In newLISP/UDP sockets are immedeately closed on both sides:



net-send-ip:

(1)create socket,

(2)use it to send to receiver address and port,

(3) close socket.



net-receive-udp:

(1)create socket

(2)listen on socket

(3)read udp packet on socket

(4)socket



The reason for this implenation is that UDP was primarily meant to be a protocol with no inherent handshaking, i.e.



when you do a syslog(), you just fire away your log message without waiting for anything back.



Of course you can work it the way you do, but I think than you should do Tcp/Ip. I am not familiar with the Gaming



usage of UDP perhaps the method you are describing is the way is the way they work UDP? The newLISP implementaion



is targeted towards controlling hardware over a network (i.e. video equipment), where the sending of a UDP packet



and it's reception is the complete transaction and anything else happens on a higher level in an application



protocol.



But there may be a solution! The net-connect function already has an undocumented additional "udp" option and I



could add one to net-listen too and then you could work this way:



server:

(net-listen port local-ip "udp") => socket

(while (not (net-select ...) ...)

(net-receive socket)

(net-send socket)



client:

(net-connect port remote-ip "udp") => socket

(net-send socket message)



etc.



So the idea is to work with normal net-send and net-receive but open the sockets for UDP with listen and connect. (the listen would only make the UDP socket not really 'listen' like a stream socket)





Lutz

[pjot]

Hi Lutz,



Good news! Your tip works. Right now, the newLisp script with UDP looks like:



-----------------------

# Define communication function

(define (gtk soc str)

(net-send soc str)

(net-receive soc 'tmp 16)

tmp)



# Start the gtk-server

(process "gtk-server udp localhost:50000")

(sleep 1000)



# Connect to the GTK-server

(set 'socket (net-connect "localhost" 50000 "udp"))



# Define GUI

(set 'tmp (gtk socket "gtk_init(NULL, NULL)"))

(set 'win (gtk socket "gtk_window_new(0)"))

.....etc

-----------------------



This works exactly as expected.



It will take some time before the UDP version of the GTK-server will be released (me being on holidays) but newLisp will be a client language supporting it.



Thank you!



Peter