- Welcome to newLISP Fan Club.
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
#1
newLISP newS / libevent2
January 04, 2013, 12:01:31 PM #2
Whither newLISP? / Asynchronous I/O with libevent
October 12, 2009, 07:18:22 AMI am in the process of polishing up the interface to a wrapper module around libevent, an asynchronous I/O library for Windows, Linux, Unix, and OSX.
Threading and multiprocessing are expensive ways to take advantage of multi-core systems. They add complexity to any application that wishes to use them safely.
In most applications, the single largest bottleneck is input and output - reading/writing a file, downloading a URL, or talking back and forth with a server over a socket. In many cases, the application can do other things while waiting for a request to be sent or a response to arrive. Newlisp provides access to the common denominator of async I/O with net-select.
Most systems provide faster polling mechanisms, and libevent provides a unified API to many of them. It allows you to write your software as a series of responses to particular events - much like GUI programming with the Newlisp GUI server.
Central to this module is the Reactor context. The Reactor can be though of as a scheduler; it does the work of managing registered events and responses, and running callback functions as needed.
Take a simple downloader. In synchronous Newlisp:
Code Select (get-url "http://www.artfulcode.net")Now, the user has to wait for that command to finish before anything else can happen. Using event-based methods and the Reactor module:
Code Select ; The data buffer
(setf data nil)
; This is called when the socket is read to read and reads into the global buffer, data.
(define (reader socket event reactor)
(if (zero? (net-peek socket))
(begin
(setf data (Reactor:get-event-storage event))
(Reactor:kill-event event)
(net-close socket))
(local (buf)
(net-receive socket buf (net-peek socket))
(Reactor:write-event-buffer event buf))))
; Write the request to the socket. Then, remove the write event and add a read event.
(define (writer socket event reactor)
(net-send socket "GET / HTTP/1.1rn")
(net-send socket "Host: www.artfulcode.netrn")
(net-send socket "Connection: closernrn")
(Reactor:kill-event event)
(Reactor:make-event reactor socket reader nil nil))
; Open a connection to the server
(setf socket (net-connect "www.artfulcode.net" 80))
; Create a reactor
(setf reactor (Reactor:make-reactor))
; Create an event handler that calls 'writer' when the socket is ready to be written to
(setf event (Reactor:make-event reactor socket nil writer nil))
; Run the reactor
(Reactor:run-reactor reactor)
; The reactor runs until it is told to stop in one of the callbacks, and then this code is run.
(println "Received " (length data) " bytes")This looks a lot more complicated. However, for complex networking programs managing multiple connections, this technique is much faster, more efficient, and with some experience, just as easy to follow.
It works like this. The reactor is created. The initial event is created for the first open socket. The reactor and relevant event are passed to each callback, allowing the callbacks to kill events, create new events, and stop the reactor (effectively moving past the "run-reactor" function call).
Think about this - what if the writer callback not only scheduled a reader for the same socket, but also added a new writer event for another url in a list. Instead of killing the reactor, the reader would do something with the complete download (like write it to a file or store it in a database) and processing would continue. The urls would be downloaded concurrently, and only the work at the end would cause newlisp to do any real work.
Using this technique, I have written applications that can process hundreds of URLs every minute.
You can see the docs here:
This module is new and has not been extensively tested. Please try it out and see what bugs you can find!
#3
Whither newLISP? / Contexts and macros
October 07, 2009, 06:37:18 AMCode Select (context 'my-macro)
(define-macro (my-macro ...)
...)
(context 'previous-context) #4
newLISP in the real world / net-receive and wait-string
October 06, 2009, 12:05:37 PM1) it is not found before int-max-bytes are read
2) it is not found in the bytes immediately available from the remote socket
If it does block for situation 2, is there any way to evade that behavior while still using the built-in buffer?
#5
newLISP newS / Artful Code modules
August 14, 2009, 01:31:21 PM #6
So, what can you actually DO with newLISP? / web.lsp 0.3
August 13, 2009, 09:54:54 AMDownload:
#7
newLISP newS / import, libmysqlclient problems
June 02, 2009, 07:50:51 AMThis happens in newlisp 10.2 and the development version. I have tried both versions using the compiled binary as well as compiling myself (darwin/utf8/readline). It happens with all of them.
Here is the sample code:
Code Select (constant 'libmysqlclient "/usr/local/mysql/lib/libmysqlclient.dylib")
(import libmysqlclient "mysql_init")
(import libmysqlclient "mysql_real_connect")
(import libmysqlclient "mysql_real_query")
(import libmysqlclient "mysql_store_result")
(import libmysqlclient "mysql_free_result")
(import libmysqlclient "mysql_close")
(import libmysqlclient "mysql_num_rows")
(import libmysqlclient "mysql_field_count")
(setf sql "select * from some_table_with_lots_of_records")
(setf MYSQL (mysql_init 0))
(println "Connecting: " (mysql_real_connect MYSQL "localhost" "user" "secret" "db" 0 0 0))
(println "Querying: " (mysql_real_query MYSQL sql (+ 1 (length sql))))
(println "Storing: " (setf MYSQL_RES (mysql_store_result MYSQL)))
;; !!! This should return a number in the thousands, but always returns 0.
(println "Rows found: " (mysql_num_rows MYSQL_RES))
(println "Freeing: " (mysql_free_result MYSQL_RES))
(println "Cleaning up: " (mysql_close MYSQL))
(exit 0)mysql_affected_rows is also behaving similarly.
I tried recompiling mysql after the upgrade, and using the new library, I get the same results.
Does anyone have *any* idea what might be happening?
#8
newLISP newS / Facebook group
May 29, 2009, 09:48:44 AM #9
newLISP newS / A new web-based library for newLISP
May 29, 2009, 06:35:23 AMIt has extensive support for URL-encoding, entities, and sessions, as well as all of the functionality of the CGI module.
#10
newLISP newS / Updated Json module
May 22, 2009, 11:19:00 AMGet it here:
#11
Anything else we might add? / Feature request
March 16, 2009, 10:46:52 AMWould it be possible to make swap work with arrays? Would it also be possible to have push/pop ops for arrays that increases/decreases an array's size destructively?
#12
Anything else we might add? / Creating functions with reference returns
February 21, 2009, 07:37:44 AMCode Select (define-macro (foo) (first (args 0)))
(setf bar '(1 2 3))
(setf (foo bar) 3)
; bar now is '(3 2 3) #13
newLISP in the real world / Printing binary content
February 19, 2009, 06:38:36 AM #14
newLISP newS / Release: updated TextMate bundle
February 18, 2009, 11:18:35 AM #15
newLISP newS / Release: nl-install
February 18, 2009, 09:57:17 AMI added a repository for my own modules at artfulcode.net. If you keep your own repository of newlisp modules, it's simple to set them up as a repository for nl-install. Once a few have been added, I will put up a listings page for them.
#16
newLISP newS / Sockets module
January 30, 2009, 08:31:37 AMOne really neat feature is the ability to pass lisp expressions directly back and forth over a socket. This works very much like net-eval, but lets you control what the server does with the expression. This is extremely useful if you want to pass complex data back and forth or create your own multi-processing net-eval using the Cilk API - and, perhaps, my MP module :).
Code Select
(let ((server (SocketServer "/tmp/newlisp.sock")))
(:run-server server
(lambda (client server , request)
(if client
(begin
(setf request (:read-expr client))
(eval request) ; not safe unless you know your client
(:write-expr client '(println "Hello, client!"))
(:close client))
(println "client connection error: " (net-error))))
nil ; no operations between connections
(lambda (err server) (println "An error has occurred: " err))))Code Select ; Connect to server process as a client
(let ((socket (Sockets:make-file-socket "/tmp/newlisp.sock")))
(:write-expr socket '(println "Hello, server!"))
(setf expr (:read-expr socket))
(:close socket))
(eval expr) ; not safe unless you know your server #17
newLISP newS / Multiprocessing with newlisp
January 29, 2009, 09:51:04 AMA summary can be found here:
The module itself is here:
#18
Anything else we might add? / Feature requests
January 26, 2009, 09:36:11 AMI have a few feature requests.
When using trees, setf fails when the symbol has not yet been defined:
Code Select (setf (Foo "bar") 123)I know that this is because their is no symbol to reference, but in this case, shouldn't the symbol be implicitly created? The intention is obvious to my mind.
I would like to be able to sync individual processes by symbol. As it stands at the moment, I cannot distinguish between two sets of processes spawned in the same parent process. This is not as rare a situation as you might imagine. For example, a socket server might fork to respond to clients, but also have another process that is forked every so often to perform a background task without tying up the listener.
I imagine it would be hard to implement, but it is a change that I think really ought to be made.
Could you add a timeout parameter to semaphores when decrementing, so that I can specify in milliseconds how long I am willing to wait for a semaphore?
#19
Whither newLISP? / GeSHi plugin
January 26, 2009, 09:29:22 AM #20
newLISP newS / Mysql module
January 22, 2009, 09:37:22 AMTake a look and find any bugs you can!