Few questions ...

[Kazimir Majorinc]

I'm trying to write an article on Newlisp macros (mostly stuff I already discussed) and I need few bits of information:



Lutz: did you reinvented macros in the form of fexprs through your own experiments, or you analysed the past and concluded that you liked MacLisp and Interlisp "special forms" more than CL or Scheme.  When it happened (year?)



Everyone: theoretically, dynamic scope causes funarg problems, discussed on this forum (itistoday, Jeff recently) and on my blog. (Briefly, if your program uses variable x, and it calls the function or macro that uses local variable x, both variables of the same name are accessible, and if one is not careful, called function can confuse the two. One of the main purpose of the contexts is to solve the problem if it occured.)



Did you have such problem in practice? How serious it was? Did it happened that contexts were not enough for your problem? [I can answer this on myself- I cannot recall I ever came into such problem in practice]



Please, answer on this questions, because if you do not comment, I can assume that you had no problems, but I cannot prove it.

[Lutz]

Fexprs were in  newLISP from the beginning (v.1.0 was released in 1993). I thought that fexprs could do almost anything expansion/rewrite macros could do: creating special forms with non-standard evaluation rules. This is the reason why the name 'define-macro' has 'macro' in it. Its really a function, but its purpose is to let you do, what you normally do with macros.





But rewrite macros are coming to newLISP too:



Using a new 'reader-event' function in newLISP v.10.1.6, rewrite macros can be created. A rewrite function in an ew module macros.lsp packaged with v.10.1.6 intercepts code translation allowing for code tranformations during load of a program.



For details see here: http://www.newlisp.org/macros/">http://www.newlisp.org/macros/

[cormullion]

newLISP is cool because you can program in a Lisp-like way if you want to, but otherwise it's a powerful and easy to use scripting language without them. I mostly use it in the latter way, and very rarely use macros unless they're essential - ie no other way to do it. I don't think there's a single macro in Dragonfly, and there was only one in my previous blogging app. I think I should be using them in nldb.lsp but can't work out how to do it, so I control evaluations another way...



I'm assuming that the use of contexts (my understanding is outlined http://en.wikibooks.org/wiki/Introduction_to_newLISP/Macros#Symbol_confusion">here) is sufficient to prevent the problems of confusing identically-named variables (this is what I thought I did http://unbalanced-parentheses.nfshost.com/downloads/mycroft.nl.src.html">here but I might be wrong).



Looking forward to see what the new macros can do...!

[Kazimir Majorinc]

Aha, I see. If I understood well, these macros are similar to C macros in a sense that you explicitly define source and target expression, while other Lisp macros specify the code that expands the expression.



Maybe you could consider some other name, something like expand- (or reader- or rewrite- or preprocessing- or inline-) -patterns because of possible, almost unavoidable confusions with (1) current, very expressive define- and lambda-macros in Newlisp (2) macros in other Lisps specifying the code that calculates target expressons.



Especially if there is a possibility that people might conclude that "Newlisp macros are not as good as X-Lisp macros" while - intentionally or not - stay silent about fexpr-macros. These issues are complex and even very experienced programmers can be confused.

[cormullion]

While I have mixed feelings about doing things just so that Common Lisp users will be 'happier' and less confused than usual about newLISP, I can also see an argument to naming things precisely...

[Lutz]

The new 'macro' facfility in upcoming 10.1.6 is actually very similar to macros CL. No mixed feeling regarding naming are necessary :-)



From "The Common LISP Cookbook":

http://cl-cookbook.sourceforge.net/macros.html">http://cl-cookbook.sourceforge.net/macros.html

A macro is an ordinary piece of Lisp code that operates on another piece of putative Lisp code, translating it into (a version closer to) executable Lisp.

or from: "An Introduction to Common Lisp Macros"



http://www.apl.jhu.edu/~hall/Lisp-Notes/Macros.html">http://www.apl.jhu.edu/~hall/Lisp-Notes/Macros.html

1.  Basic Idea: Macros take unevaluated Lisp code and return a Lisp form. This form should be code that calculates the proper value. Example:



(defmacro Square (X)

  '(* ,X ,X))



This means that wherever the pre-processor sees (Square XXX) to replaces it with (* XXX XXX). The resultant code is what the compiler sees.

That is similar to what the 'macro' in macro.lsp does:

Code Select Expand

(macro (square X) (mul X X))

=> (lambda-macro (X) (expand '(mul X X)))

square => (lambda-macro (X) (expand '(mul X X)))

((lambda-macro (X) (expand '(mul X X))) 3) => (mul 3 3) ; the resultant code


'square' now contains the function generating the code the newLISP translater uses to rewrite the code. After having done (macro (square X) (mul X X)) all occurrences of (square 3) will be translated to (mul 3 3) in future expressions read by newLISP.



The last statement happens in the rewrite function of the macro.lsp module:

Code Select Expand

(set-ref-all pattern expr (eval $it) match))


Pattern is (square 3), evaluating it in (eval $it) returns the expansion replacement (mul 3 3).

Code Select Expand

(define (test) (square 3)) => (lambda () (mul 3 3))

test => (lambda () (mul 3 3))


In both cases the macro facility does template expansion to generate the target LISP code.



PS: note that the following square macro is more efficient when a complex expression is past in X: (macro (square X) (pow X 2) ). It avoids the double evaluation of an expression in X as in (mul X X). In common Lisp the compiler would take care of this doing "common subexpression elimination". When we do macro expansion in a scripting language, we have to be sensitive to efficiency issues ourselves.

[cormullion]

Cool. Look forward to playing with it. I'm happy for you to call it what you like, Lutz! :)



I'd like to know more about what these are really suitable for. I read the page you linked to first, and I get the general idea, but it did go on to say

Both Square and Square-Sum are inappropriate uses of macros.

There's a performance hit too, I suppose. What are people going to do with this...?

[Lutz]

Macros with macro.lsp are good, where ever you would want to write a small function, but are afraid of wasting function-call overhead using 'define-macro'. Macros with macros.lsp are efficient but shorter to read and more descriptive in the program text:

Code Select Expand

newLISP v.10.1.6 on OSX IPv4 UTF-8, execute 'newlisp -h' for more info.

> (module "macro.lsp")
MAIN
> (set 'x 2)
2
> (time (pow x 3) 1000000)
197.213

> (define (cube-a x) (pow x 3))
(lambda (x) (pow x 3))
> (time (cube-a x) 1000000)
381.352

> (macro (cube-b X) (pow X 3))
(lambda-macro (X) (expand '(pow X 3)))
> (time (cube-b x) 1000000)
191.922
>


the macros takes the same time as the original, because internally it is doing the same. Internally its doing (time (pow x 3) 1000000) because (cube-b x) has been transformed.

[Jeff]

This nice thing about these macros is that you are actually performing a compile-time transformation on the code. You can use them to easily define destructive setters on OOP classes, for example, because they would translate into the actual code that returns a reference. Perhaps something like this, using an assoc list to store your member vars in an instance variable:

Code Select Expand

(macro (slot-value inst slot val)
  (if (nil? val) val (setf (assoc slot inst) val)))

[Lutz]

Yes, exactly, good example how readability gets improved.



But note that the macro has to be written with uppercase variable names so 'expand' can work with it internally:

Code Select Expand

(macro (slot-value Inst Slot Val)
  (if (nil? Val) Val (setf (assoc Slot Inst) Val)))


so when you code does:

Code Select Expand

(slot-value my-inst the-slot my-val)

it will be rewritten to:

Code Select Expand

(if (nil? my-val) my-val (setf (assoc the-slot my-inst) my-val)



As 'my-val' occurs 3 times in this macro and it could be that a caller passes some complex expression (blah (foo (x)), it may be better to define:

Code Select Expand

(macro (slot-value Inst Slot Val)
  (let (value Val) (if (nil? value) value (setf (assoc Slot Inst) value))))


this way Val -> (blah (foo (x)) is only evaluated once.

[TedWalther]

Fexprs were in  newLISP from the beginning (v.1.0 was released in 1993). I thought that fexprs could do almost anything expansion/rewrite macros could do: creating special forms with non-standard evaluation rules. This is the reason why the name 'define-macro' has 'macro' in it. Its really a function, but its purpose is to let you do, what you normally do with macros.





But rewrite macros are coming to newLISP too:



Using a new 'reader-event' function in newLISP v.10.1.6, rewrite macros can be created. A rewrite function in an ew module macros.lsp packaged with v.10.1.6 intercepts code translation allowing for code tranformations during load of a program.



For details see here: http://www.newlisp.org/macros/">http://www.newlisp.org/macros/

Hi Lutz, interesting module.  Any chance this will be extended to work on character streams as well as well-formed list expressions?

[Kazimir Majorinc]

Yes, but other Lisp macros can do more complex things, for example

Code Select Expand


(defmacro Square (X)
   '(* ,X ,X))

vs

(defmacro setq2 (v1 v2 e)
   (let ((e1 (comtran e)))
      (list 'progn (list 'setq v1 e1) (list 'setq v2 e1))))

There is no top apostrophe in the second example. If I understand well, your current macros cannot do that, that's why these are more C-like than CL-like. But it is not really hard to reach or improve over CL- macros. I defined one toy-example here:



http://kazimirmajorinc.blogspot.com/2008/08/two-phases-evaluation.html">http://kazimirmajorinc.blogspot.com/200 ... ation.html">http://kazimirmajorinc.blogspot.com/2008/08/two-phases-evaluation.html

Code Select Expand

(load "http://www.instprog.com/Instprog.default-library.lsp")

(set 'x 3)
(println (time (pow x 3) 1000000)) ;=> 304

(define (cube-a x)(pow x 3))
(println (time (cube-a x 3) 1000000)); => 575

(define-macro (cube-b y)
              'prepare-time    ; this means that it will be
              (list 'pow y 3)) ; evaluated in "prepare time"
             
(println (prepare '(cube-b (cube-b x)))) ;=> (pow (pow x 3) 3)
(println (eval (prepare '(cube-b x)))); => 27
(println (eval (prepare '(time (cube-b x) 1000000)))) ;=> 317

(exit)

There is no reader-hook, so this eval - prepare - ' is necessary, but as said, it is only toy example.



You can have many kind of macros. You already have (1) current fexpr-like macros, (2) C-like macros you are demonstrating here, (3) CL-like-macros as in my library, some Scheme-like macros in future (4) ...  clever naming strategy might be important now.

[Jeff]

You know, Lutz, it would be nice if there were an operator that expanded at compile-time, rather than the entire function definition. That way, it could be evaluated inside a macro and provide the full power of both f-expressions and compile-time syntax transformations.

[Lutz]

Any chance this will be extended to work on character streams as well as well-formed list expressions?

it would be nice if there were an operator that expanded at compile-time

Working the at the character stream level, rather than s-expression level would allow non-Lisp syntax, which for my taste is a bad idea. It also would have a much larger impact on newLISP's very fast source-code load time. Remember that in a scripting language everything has to happen during load/run-time. Initially I also explored the idea to expose the raw string stream to the user and let them do whatever they want to, using Unix an Awk like regular expressions API. But that would make the whole thing to impracticable, unless you are an regex artist



I think what we have now fits best into the general character of newLISP. Manipulation is limited to what well-formed lisp-expressions let us do, but the programmer who has chosen newLISP made that choice already.



'reader-event' is called with all top-level expressions. A top-level expression doesn't have to be parenthesized, it also can be single token, string or identifier. The current module macro.lsp limits to parenthesized expressions, but that does not need to be.

You can have many kind of macros. You already have (1) current fexpr-like macros, (2) C-like macros you are demonstrating here, (3) CL-like-macros as in my library, some Scheme-like macros in future (4) ... clever naming strategy might be important now.

Nobody has to take the current "macro.lsp" as religion, there are many other ways to define a macro function and use the new 'reader-event' hook. The current macro.lsp module is just the quickest, straight forward way to implement template-expansion macros and demo the basic principle of intercepting expressions and rewriting them. Other ways are only limited by one's imagination ... and coming up with enough confusing names for it :)

[Jeff]

That's not what I meant. I meant that it would be nice to have the type of compile-time expansion you built using read-expr but in the context of a non-defining expression. More like letex than define, in other words. That way we could use them as part of  a regular newlisp macro. Like the backtick in cl.