Macros and deficiencies from CL

[eliben]

Hello,



I have a couple of questions, one of which is probably controversial. I want to note first that I'm here out of sheer curiosity as newLisp looks interesting. I come from a Perl / Ruby / C++ background with some knowledge and experience in Common Lisp. My intention is not flame, but rather a sheer interest in newlisp.



So, the questions:



1) Does newlisp have macros in the full Lisp sense ? I.e. can newlisp's macros do what CL's macros do ? Can the code from Paul Graham's "On Lisp" be implemented in newlisp ?



2) newlisp looks to be not too well received in the CL community. Granted, it's a pretty demanding community, but still it's surprising to see that newlisp is perceived practically as a joke in comp.lang.lisp

I wonder about the cause of this. It looks peculiar since newlisp seems at first sight quite complete and useful.

Has a serious attempt been ever made to understand c.l.l's criticism and answer it ?



Thanks in advance

[Lutz]

Welcome to the group Eliben!



Macros and many other things are quite different from Common LISP or SCHEME. Most code except for very simple examples from books about LISP will not run under newLISP.



To find out more about the differences from newLISP to Common LISP and SCHEME see the following page: http://newlisp.org/index.cgi?page=Differences_to_Other_LISPs">http://newlisp.org/index.cgi?page=Diffe ... ther_LISPs">http://newlisp.org/index.cgi?page=Differences_to_Other_LISPs



Then there is also the 'About' page: http://newlisp.org/index.cgi?FAQ">http://newlisp.org/index.cgi?FAQ which will further clarify newLISP's capabilities and intentions. This page also contains links to pages about memory management and benchmarks.



You can also reach the same page via the words "the new LISP" on top of the page to the right of the newLISP logo.



I suggest to try out newLISP yourself and come to your own judgement ;)



Lutz

[cormullion]

In my opinion (as a newcomer to newLISP myself), the main problem with newLISP is the name. :-) It obviously gets interpreted by uninformed or excessively opinionated people as containing some claims to superiority (which was not the author's intention). I think it would have got less attention and less criticism had it been named something different...



You'll find some references to the opinions of people on c.l.l on this forum, but nobody seems to worry too much about arguing with them.  Better things to do, perhaps, than try to sell things to people who aren't going to buy.

[eliben]

Welcome to the group Eliben!



Macros and many other things are quite different from Common LISP or SCHEME. Most code except for very simple examples from books about LISP will not run under newLISP.



To find out more about the differences from newLISP to Common LISP and SCHEME see the following page: http://newlisp.org/index.cgi?page=Differences_to_Other_LISPs">http://newlisp.org/index.cgi?page=Diffe ... ther_LISPs">http://newlisp.org/index.cgi?page=Differences_to_Other_LISPs



Lutz

Hello Lutz, thanks for the response.

The page you refer to says a few words about the difference of macros, and I've already read it, but without a few examples it is not too useful. Examples are the best way to demonstrate things.



I'm taking some time to look at newlisp, but my first goal is to understand if its macros are as powerful as Common Lisp's. I don't expect to copy paste CL code into a newlisp interpreter and see it run, but I want to know that the expressive power is comparable.



I will provide more concrete questions:



* Are newlisp macros evaluated at "compile time" - like CL macros (and C "macros", for that matter) ? This is important for performance, of course.



For example, here is a CL function and macro for 1+:

Code Select Expand

(defun 1+ (x) (+ 1 x))
(defmacro 1+ (x) '(+ 1 ,x))

The macro is better, since it saves a function call.



* Can macros be recursive ? For example, here is a CL macro that mimics the 'and' behavior:

Code Select Expand

(defmacro our-and (&rest args)
  (case (length args)
    (0 t)
    (1 (car args))
    (t '(if ,(car args)
            (our-and ,@(cdr args))))))

Can this be done in newlisp ? Can you please show how ?



That's it for now. Thanks :-)

[HPW]

Are newlisp macros evaluated at "compile time" - like CL macros (and C "macros", for that matter) ? This is important for performance, of course.

No, since there is no compiling with newLISP.

So everything in newLISP is interpreted.

In CL you can save your modified image to file and gets your own modified lisp-system.



But the newLISP interpreter is substantial smaller than every current CL implementation, it is easy embedable in other enviroments and it is quit fast for an interpeter. For performance-critical tasks you can easy call external DLL's.



As always use the right tool for the job.

;-)

[cormullion]

Seems newLISP's best feature is its ability to confuse people:



http://reddit.com/info/4hao/comments">//http://reddit.com/info/4hao/comments



;-)

[eliben]

Are newlisp macros evaluated at "compile time" - like CL macros (and C "macros", for that matter) ? This is important for performance, of course.

No, since there is no compiling with newLISP.

So everything in newLISP is interpreted.

In CL you can save your modified image to file and gets your own modified lisp-system.



But the newLISP interpreter is substantial smaller than every current CL implementation, it is easy embedable in other enviroments and it is quit fast for an interpeter. For performance-critical tasks you can easy call external DLL's.



As always use the right tool for the job.

;-)

I will explain what I mean by "compile time". In Common Lisp, even when it is interpreted and not compiled - there is a preliminary pass where all macros are "expanded" and only then the interpreter begins.



I hope that the performance benefits of this are obvious.

[rickyboy]

I don't know what is your particular motivation for using/trying newLISP.  As for me, I like, if you'll pardon the expression, kickass scripting languages.  :-)  I use them from mundane scripting tasks to prototyping mainline applications.   And in any of these cases, I want this language to be able stretch beyond the bounds I might have anticipated for it in the realm of programming power; newLISP accomplishes this very nicely.  And as you may have anticipated, that prototype could very well be the production system (hey, that's OK by me -- I can move on to new coding projects :-).



So, regarding the question of macros. the short of it is that I use newLISP macros solely for the purpose of expressing the kind of abstractions not afforded to me by 'define'ing newLISP functions, and I don't concern myself with performance, as the interpretive speed of newLISP is good enough for me.  Abstraction making (i.e. *programming* power) is the more important issue for me; I let Moore's Law take care of the *processing* power issue for me.  :-)



Hope that helps out.  Happy hacking!

[Lutz]

newLISP is not a compiled language, but translates all code into an internal list representation for evaluation. No textual interpretation of source code happens after a program is loaded.



Macros in newLISP are like functions, but without evaluation of the function parameters which have to be evaluated explicitely inside the body of the macro. Variable expansion is done explicetely using the function 'expand'. A newLISP macro call is still a function call but with much less call overhead and the possibility to access symbols directly without the need to quote them. The 'args' function permits to access arguments in an hygienic fashion without the danger of variable capture.



There are macro examples here: http://newlisp.org/downloads/newlisp_manual.html#define-macro">http://newlisp.org/downloads/newlisp_ma ... fine-macro">http://newlisp.org/downloads/newlisp_manual.html#define-macro and here for 'expand' http://newlisp.org/downloads/newlisp_manual.html#expand">http://newlisp.org/downloads/newlisp_manual.html#expand



Your first example could be done like this:

Code Select Expand


(define-macro (++) (+ 1 (eval (args 0 ))))

; but as you have to evaluate x anyway, better you do:
(define (++ x) (+ 1 x))

; but in newLISP you just do
(inc 'x)


The second example you gave, is another example you would not solve with a macro in newLISP, because all arguments would have to be evaluated, and you would solve it iteratively. In newLISP using first/last (car/cdr) to iterate recursevely thru a list is almost always the wrong approach for newLISP, where you do it with an iteration most of the time (but yes, you could use recursion in macros):

Code Select Expand


(define (my-and)
    (catch (dolist (i (args))
        (if (not i) (throw nil) true))))

(my-and 1 2 (< 4 3) 5 6) => nil
(my-and 1 2 (> 4 3) 5 6) => true


Last an example where you would use define-macro in newLISP, because you want to pass arguments without evaluation.

Code Select Expand


(define-macro (defun _func-name _arguments)
    (set _func-name (append '(lambda ) (list _arguments) (args))))

; or better to avoid variable capture

(define-macro (defun)
    (set (args 0) (append '(lambda ) (list (args 1)) (2 (args)))))

(defun double(x) (+ x x)) => (lambda (x) (+ x x))


If your newLISP installation uses init.lsp, than this macro is already defined and protected against re-definition.



The examples show that newLISP is quite different from Common LISP and it does not make sense to compare many features directly.



A comment about comparing languages: There are many examples on the net where programming languages are discussed comparing specific language features between languages. This never addresses the ability of a language to solve problems, only shows that mechanisms used in different languages are different. The person using language A will say that his language is better than language B, because the A mechanism is best implemented in A, while B uses a different approach to solve the problem.



The best way to compare programming languages is to state a real world problem, then show how it is solved in one versus the other language.



The common purpose of macros in Common LISP and newLISP is to enable special form language extensions, which can break out of the usual LISP -> (func arg1 arg2 ..) pattern, where all arguments are evaluated in applicative evaluation order. The mechanics of achieving this goal are quite different in both languages.



Lutz

[Lutz]

Yes Rickyboy, the scripting aspect of newLISP is important and defines many characteristics and the feel of the language: no predefinition of variables, variable arguments in user defined functions, etc.



Lutz

[eliben]

Lutz,



Thanks for the very informative post - this is exactly what I was looking for. I want to address a few points:



1) I agree that the division between "wanting to evaluate all arguments" and "not wanting to evaluate some of the argumetns" is logical regarding usage of macros, in most cases. However, in the established Lisp flavors, macros are also often used for optimization purposes, being expanded at "compile" time. I'm placing the word "compile" in quotes intentionally, since as I mentioned, it doesn't necessarily imply a compiler, but rather some pre-processing before the code is interpreted. In this sense it is similar to the "macros" of C and C++, which are expanded by the preprocessor.



But as I understand from you, such optimization is impossible in newlisp because macros are just like functions. I understand, and it's not a big deal really. However, wouldn't it be quite simple to implement, in your opinion ? A single "expansion" pass of the interpreter on the code, detecting and expanding all the instances of macros, can make this optimization possible. What don't I see that makes it difficult ?



2) The example you provided of the 'defun' macro using (args) for hygienic identifiers looks a little cumbersome, can't you just say:

Code Select Expand

(set (args 0) (append '(lambda ) (list args)))

Or something of the sort ?



3) I completely agree with what you are saying on comparing languages. However, experiences programmers who know well several languages from different groups (imperative, functional, OO) have more ammunition to reason and compare between languages based on features. Sometimes it is biased, but it can't be dismissed as irrelevant.

[Lutz]

Eli, I agree with most you have said and want to add a few comments to the three points you made:



(1) Yes, a preprocessing pass while loading doing macro expansion would be possible. At the moment though, not a priority, it also masks in an interactive newLISP session the possibility to just enter the macro name and get the source translation back (see also 'save' and 'source'), but it certainly would make writing macros more comfortable.



(2) The 'defun' macro: Oops, yes it can be shorter, the translation from the first version with parameter names was done too quick, thanks for the hint.

Code Select Expand


(define-macro (defun)
    (set (args 0) (append '(lambda ) (rest (args)))))

; or a mixed/shortest solution

(define-macro (defun func-name)
    (set func-name (append '(lambda )  (args))))


I have to take (rest (args)) or (1 (args)) which is the same, because I don't want 'double' in the body. Note that 'args' in newLISP is a function, which either returns a list when given without arguments or returns an element when used with an index, as in (args 0). (args) will only have a list of parameters not bound by parameters in a parameter list. A mixed approach would be possible only specifying the function name 'double' and then have only the parameter list and the body left. The expression (1 (args)) is implicit indexing/slicing/resting od the list returned by (args).



(3) Yes, for programmers who know several programming languages well, your point is taken. Just wanted to emphasize the more pragmatic, application oriented thinking.



I hope you stay with us and show us some new tricks with newLISP ;)



Lutz

[rickyboy]

Oops, we have to be careful.  This version of 'my-and' is too eager:

Code Select Expand

(define (my-and)
    (catch (dolist (i (args))
        (if (not i) (throw nil) true))))

> (my-and (< 4 3) (println "hello"))
hello
nil
> (and (< 4 3) (println "hello"))
nil


Here's one that is more lazy:

Code Select Expand

(define-macro (my-and)
  (catch
    (dolist (i (args))
      (let ((res (eval (expand '(first (list i)) 'i))))
        (if (not res) (throw nil) res)))))


It also returns the result of the last "true" expression, instead of 'true'.

Code Select Expand

> (and (> 4 3) (println "hello"))
hello
"hello"
> (my-and (> 4 3) (println "hello"))
hello
"hello"


Is there a better way to write it?  Curious.

[Lutz]

Yes, here is another similar 'lazy' evaluating version (also written as a macro):

Code Select Expand


(define-macro (my-and)
    (catch (dolist (i (args))  
        (if (not (eval i)) (throw nil) true))))


now only tested elements up to the thrown are evaluated



Lutz

[rickyboy]

Yes, thank you Lutz!



I will pick one bone though.  Notice the following:

Code Select Expand

(define-macro (lutz-and)
    (catch (dolist (i (args))
        (if (not (eval i)) (throw nil) true))))

(define-macro (ricky-and)
  (catch
    (dolist (i (args))
      (let ((res (eval i)))
        (if (not res) (throw nil) res)))))

> (lutz-and (> 4 3) (println "hello"))
hello
true
> (ricky-and (> 4 3) (println "hello"))
hello
"hello"
> (and (> 4 3) (println "hello"))
hello
"hello"