Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Topics - Jeff

newLISP newS / libevent2
January 04, 2013, 12:01:31 PM
Howdy folks. Haven't been around in a while. Wanted to let you know that I put together a wrapper for libevent2 using the new FFI features. Made it very easy. You can find it on github here -">
Whither newLISP? / Asynchronous I/O with libevent
October 12, 2009, 07:18:22 AM
Hey folks,

I am in the process of polishing up the interface to a wrapper module around libevent, an asynchronous I/O library for Windows, Linux, Unix, and OSX.

What is asynchronous I/O?

Threading and multiprocessing are expensive ways to take advantage of multi-core systems. They add complexity to any application that wishes to use them safely.

In most applications, the single largest bottleneck is input and output - reading/writing a file, downloading a URL, or talking back and forth with a server over a socket. In many cases, the application can do other things while waiting for a request to be sent or a response to arrive. Newlisp provides access to the common denominator of async I/O with net-select.

Most systems provide faster polling mechanisms, and libevent provides a unified API to many of them. It allows you to write your software as a series of responses to particular events - much like GUI programming with the Newlisp GUI server.

Central to this module is the Reactor context. The Reactor can be though of as a scheduler; it does the work of managing registered events and responses, and running callback functions as needed.

A (relatively) simple example

Take a simple downloader. In synchronous Newlisp:

(get-url "")

Now, the user has to wait for that command to finish before anything else can happen. Using event-based methods and the Reactor module:

; The data buffer
(setf data nil)

; This is called when the socket is read to read and reads into the global buffer, data.
(define (reader socket event reactor)
 (if (zero? (net-peek socket))
   (setf data (Reactor:get-event-storage event))
   (Reactor:kill-event event)
   (net-close socket))
  (local (buf)
   (net-receive socket buf (net-peek socket))
   (Reactor:write-event-buffer event buf))))

; Write the request to the socket. Then, remove the write event and add a read event.
(define (writer socket event reactor)
 (net-send socket "GET / HTTP/1.1rn")
 (net-send socket "Host: www.artfulcode.netrn")
 (net-send socket "Connection: closernrn")
 (Reactor:kill-event event)
 (Reactor:make-event reactor socket reader nil nil))

; Open a connection to the server
(setf socket (net-connect "" 80))

; Create a reactor
(setf reactor (Reactor:make-reactor))

; Create an event handler that calls 'writer' when the socket is ready to be written to
(setf event (Reactor:make-event reactor socket nil writer nil))

; Run the reactor
(Reactor:run-reactor reactor)

; The reactor runs until it is told to stop in one of the callbacks, and then this code is run.
(println "Received " (length data) " bytes")

This looks a lot more complicated. However, for complex networking programs managing multiple connections, this technique is much faster, more efficient, and with some experience, just as easy to follow.

It works like this. The reactor is created. The initial event is created for the first open socket. The reactor and relevant event are passed to each callback, allowing the callbacks to kill events, create new events, and stop the reactor (effectively moving past the "run-reactor" function call).

Think about this - what if the writer callback not only scheduled a reader for the same socket, but also added a new writer event for another url in a list. Instead of killing the reactor, the reader would do something with the complete download (like write it to a file or store it in a database) and processing would continue. The urls would be downloaded concurrently, and only the work at the end would cause newlisp to do any real work.

Using this technique, I have written applications that can process hundreds of URLs every minute.

Where to get it

You can see the docs here:">// You can download the module here:">// You can get libevent (the required C library) here:">//

This module is new and has not been extensively tested. Please try it out and see what bugs you can find!
Whither newLISP? / Contexts and macros
October 07, 2009, 06:37:18 AM
If the preferred method to avoid variable capture is to use lexically scoped macros, why not make all macros lexically scoped by default? Why not have the reader implicitly read local variables in a macro as if the macro were written:

(context 'my-macro)
(define-macro (my-macro ...)
(context 'previous-context)
newLISP in the real world / net-receive and wait-string
October 06, 2009, 12:05:37 PM
Does net-receive block while waiting on wait-string if:

1) it is not found before int-max-bytes are read

2) it is not found in the bytes immediately available from the remote socket

If it does block for situation 2, is there any way to evade that behavior while still using the built-in buffer?
newLISP newS / Artful Code modules
August 14, 2009, 01:31:21 PM
I created a google code project for the artfulcode modules. If anyone is interested in contributing, shoot me an email ( I'd love to have some other skill sets involved in the code.">
Fixed (hopefully) a query parsing bug, made the query parser more tolerant, and added a parameter to cookie to limit cookies to secure (https) sessions.

newLISP newS / import, libmysqlclient problems
June 02, 2009, 07:50:51 AM
After upgrading to OSX 10.5, I am having some problems with imported functions from libmysqlclient. Many functions that return my_ulonglong are all returning 0. Other applications using the same libmysqlclient and importing the same functions work fine, and a short program I wrote in C using the library confirms that it is returning the correct values.

This happens in newlisp 10.2 and the development version. I have tried both versions using the compiled binary as well as compiling myself (darwin/utf8/readline). It happens with all of them.

Here is the sample code:

(constant 'libmysqlclient "/usr/local/mysql/lib/libmysqlclient.dylib")

(import libmysqlclient "mysql_init")
(import libmysqlclient "mysql_real_connect")
(import libmysqlclient "mysql_real_query")
(import libmysqlclient "mysql_store_result")
(import libmysqlclient "mysql_free_result")
(import libmysqlclient "mysql_close")
(import libmysqlclient "mysql_num_rows")
(import libmysqlclient "mysql_field_count")

(setf sql "select * from some_table_with_lots_of_records")

(setf MYSQL (mysql_init 0))
(println "Connecting: " (mysql_real_connect MYSQL "localhost" "user" "secret" "db" 0 0 0))
(println "Querying: " (mysql_real_query MYSQL sql (+ 1 (length sql))))
(println "Storing: " (setf MYSQL_RES (mysql_store_result MYSQL)))
;; !!! This should return a number in the thousands, but always returns 0.
(println "Rows found: " (mysql_num_rows MYSQL_RES))
(println "Freeing: " (mysql_free_result MYSQL_RES))
(println "Cleaning up: " (mysql_close MYSQL))
(exit 0)

mysql_affected_rows is also behaving similarly.

I tried recompiling mysql after the upgrade, and using the new library, I get the same results.

Does anyone have *any* idea what might be happening?
newLISP newS / Facebook group
May 29, 2009, 09:48:44 AM
I created a facebook group for newlisp -"> . Everyone is invited to join.
I've written a replacement CGI library, which can optionally work side-by-side with the official CGI module, with some caveats (see the docs for details.)

It has extensive support for URL-encoding, entities, and sessions, as well as all of the functionality of the CGI module."> ... b-library/">
newLISP newS / Updated Json module
May 22, 2009, 11:19:00 AM
I've updated my Json module. This update includes a complete rewrite of the decoder and fixes a few rather serious bugs.

Get it here:">
Anything else we might add? / Feature request
March 16, 2009, 10:46:52 AM

Would it be possible to make swap work with arrays?  Would it also be possible to have push/pop ops for arrays that increases/decreases an array's size destructively?
Is there a way to return a reference that could be used in setf?  For example:

(define-macro (foo) (first (args 0)))
(setf bar '(1 2 3))
(setf (foo bar) 3)
; bar now is '(3 2 3)
newLISP in the real world / Printing binary content
February 19, 2009, 06:38:36 AM
I am writing an application that needs to sort of proxy data between two sources over HTTP. I am using get-url to fetch the data, but cannot output the data to a web browser using write-buffer or print, because the data may contain null characters and these seem to halt output at null chars. Is there any way to force it?  I can pack and unpack it and see it if it is serialized to a string in a list, I can even write it to a file. But if it is going to an output like stdio or a browser, it stops at the first null. Any ideas?
newLISP newS / Release: updated TextMate bundle
February 18, 2009, 11:18:35 AM
The textmate bundle has been updated and now has much better syntax support, a new theme, and many snippets, including support for doc comments.
newLISP newS / Release: nl-install
February 18, 2009, 09:57:17 AM
I wrote a small package manager for newlisp. You can download it at">// You can download the first version directly here:">//

I added a repository for my own modules at If you keep your own repository of newlisp modules, it's simple to set them up as a repository for nl-install. Once a few have been added, I will put up a listings page for them.
newLISP newS / Sockets module
January 30, 2009, 08:31:37 AM
For those of you sick of manually polling sockets when trying to write a simple server app, I wrote a Sockets module that neatly wraps both client and server operations into a couple of classes and helper functions.  Complete with examples and newlisp10 compatible.

One really neat feature is the ability to pass lisp expressions directly back and forth over a socket.  This works very much like net-eval, but lets you control what the server does with the expression.  This is extremely useful if you want to pass complex data back and forth or create your own multi-processing net-eval using the Cilk API - and, perhaps, my MP module :).">//

Running a server

(let ((server (SocketServer "/tmp/newlisp.sock")))
  (:run-server server
    (lambda (client server , request)
      (if client
          (setf request (:read-expr client))
          (eval request) ; not safe unless you know your client
          (:write-expr client '(println "Hello, client!"))
          (:close client))
        (println "client connection error: " (net-error))))
    nil ; no operations between connections
    (lambda (err server) (println "An error has occurred: " err))))

Connecting a client
; Connect to server process as a client
(let ((socket (Sockets:make-file-socket "/tmp/newlisp.sock")))
  (:write-expr socket '(println "Hello, server!"))
  (setf expr (:read-expr socket))
  (:close socket))
(eval expr) ; not safe unless you know your server
newLISP newS / Multiprocessing with newlisp
January 29, 2009, 09:51:04 AM
I have added a module to my repository (">// with common utilites for multi-processing.

A summary can be found here:">//

The module itself is here:">//
Anything else we might add? / Feature requests
January 26, 2009, 09:36:11 AM

I have a few feature requests.

Trees and setf

When using trees, setf fails when the symbol has not yet been defined:

(setf (Foo "bar") 123)

I know that this is because their is no symbol to reference, but in this case, shouldn't the symbol be implicitly created?  The intention is obvious to my mind.

Cilk interface

I would like to be able to sync individual processes by symbol. As it stands at the moment, I cannot distinguish between two sets of processes spawned in the same parent process. This is not as rare a situation as you might imagine. For example, a socket server might fork to respond to clients, but also have another process that is forked every so often to perform a background task without tying up the listener.

I imagine it would be hard to implement, but it is a change that I think really ought to be made.


Could you add a timeout parameter to semaphores when decrementing, so that I can specify in milliseconds how long I am willing to wait for a semaphore?
Whither newLISP? / GeSHi plugin
January 26, 2009, 09:29:22 AM
I've written a plugin for GeSHi for newlisp for those who use it to colorize code on their websites. It colorizes code using the same colors as the newlisp colorizer cgi and adds links to documentation for functions.">
newLISP newS / Mysql module
January 22, 2009, 09:37:22 AM
I just posted a new Mysql module that allows run-time creation of new, anonymous connections, multiple connections, automatic escaping and statement formatting, as well as a host of handy higher order functions to work with MySQL data sets.

Take a look and find any bugs you can!"> ... r-newlisp/">