variable capture?

[ant]

I don't understand what I'm doing wrong here. Is it variable capture?



Using the example from the define-macro description in the newLISP doc I wrote two identical test functions, apart from the name of the parameter:

Code Select Expand

(define-macro (dolist-while)
  (letex (var (args 0 0)
          lst (args 0 1)
          cnd (args 0 2)
          body (cons 'begin (1 (args))))
    (let (res)
      (catch (dolist (var lst)
               (if (set 'res cnd) body (throw res)))))))

(define (test1 a-list)
  (dolist-while (x a-list (!= x 'd)) (println x)))

(define (test2 lst)
  (dolist-while (x lst (!= x 'd)) (println x)))

(test1 '(a b c d e f))
(test2 '(a b c d e f))


here is the output

Code Select Expand

newLISP v.10.3.3 on OSX IPv4/6 UTF-8, execute 'newlisp -h' for more info.

>
(lambda-macro ()
 (letex (var (args 0 0) lst (args 0 1) cnd (args 0 2) body (cons 'begin (1 (args))))
  (let (res)
   (catch
    (dolist (var lst)
     (if (set 'res cnd)
      body
      (throw res)))))))
(lambda (a-list) (dolist-while (x a-list (!= x 'd)) (println x)))
(lambda (lst) (dolist-while (x lst (!= x 'd)) (println x)))
a
b
c
nil

ERR: list expected in function dolist : lst
called from user defined function dolist-while
called from user defined function test2
>


[cormullion]

Think so. It looks like the same example as here:



http://en.wikibooks.org/wiki/Introduction_to_newLISP/Macros#Symbol_confusion">//http://en.wikibooks.org/wiki/Introduction_to_newLISP/Macros#Symbol_confusion



and the fix should be the same...

[Lutz]

The example is wrong but can be fixed this way:

Code Select Expand


(define-macro (dolist-while)
  (letex (var (args 0 0)
          lst (eval (args 0 1))
          cnd (args 0 2)
          body (cons 'begin (1 (args))))

    (let (res)
      (catch (dolist (var 'lst)
               (if (set 'res cnd) body (throw res)))))))

Note the eval and the quote before lst in the let form.



The variable capture happened during expansion of the let form. The letex parameter assignments itself are safe, because they are still evaluated in the parent environment.  



But having define-macros in their own namespace as default functions is probably the best idea. There are examples for this in Cormullion's WikiBook introduction as well as in the manual.



ps: the example is also fixed in the online manual.

[ant]

Hi and thanks both for replying. I'm still trying to puzzle it out.



I don't quite get why your fix, Lutz, affects just the lst variable. Why aren't the other variables susceptible to the same problem?

[Lutz]

They are not a problem only in this function. cnd and body will always pass a parenthesized expression and var just would stay var if you pass it the symbol var.



There is a second potential trap with the res variable, because the statements in body ar executed in the scope of the let function. Cormullion is showing this case in his introduction.



Probably the example should be taken out of the manual completely. Or only the safe solution should be shown with the function in it's own namespace.

[rickyboy]

Or only the safe solution should be shown with the function in it's own namespace.

Exactly.

[Lutz]

Sometimes I think we need a "Style Guide" defining certain conventions, how to use newLISP's feature set. Then again, I think that would be a bad thing. One of the things many newLISP users are attracted to, is it's malleability and flexibility, which is also the reason that so many people from creative professions like newLISP.



newLISP gives you the freedom to express the same functionality in different ways and styles, without trying to box you into the "one right way" of doing things.



But definitely, there are conventions emerging from the community. Enclosing define-macro's in their own namespace, is one convention frequently adopted.

[ant]

I am a newLISP noob. There is almost nothing I understand of the culture. I thought I was using the style suggested in the article http://www.newlisp.org/downloads/newlisp_manual.html#define-macro">//http://www.newlisp.org/downloads/newlisp_manual.html#define-macro. I thought it meant: use letex with args to avoid variable capture. I'm still confused. I don't understand why only lst is affected. In fact, this

Code Select Expand

(define (test3 condition)
  (println "test3")
  (dolist-while (x '(a b c d e f) condition) (println x)))

(define (test4 cnd)
  (println "test4")
  (dolist-while (x '(a b c d e f) cnd) (println x)))

(test3 nil)
(test4 nil)


gives this



test3

nil

test4

a

b

c

d

e

f

f



In fact, I can't work out how to safely use args in a macro at all. E.g.

Code Select Expand

(define-macro (ant)
  (let ((x 3))
    (println "(args 0)=" (args 0) " x=" x " (eval(args 0))=" (eval(args 0)))))

(define (foo y)
  (ant y))
(define (bar x)
  (ant x))

(foo 7)
(bar 7)


gives



(args 0)=y x=3 (eval(args 0))=7

7

(args 0)=x x=3 (eval(args 0))=3

3

[rickyboy]

Hello ant,



This seems to work (the macro definition is the same as your original on this thread[*]):

Code Select Expand

(context 'dolist-while)
(define-macro (dolist-while:dolist-while)
  (letex (var (args 0 0)
          lst (args 0 1)
          cnd (args 0 2)
          body (cons 'begin (1 (args))))
    (let (res)
      (catch (dolist (var lst)
                     (if (set 'res cnd) body (throw res)))))))
(context 'MAIN)

> (test3 nil)
test3
nil
> (test4 nil)
test4
nil

Don't know why, but it seems that you have to have those context expressions there.  When I left them out, I got the same (erroneous, or at least unexpected) result for the expression (test4 nil) that you did.  Hope this helps.

______________

[*] -- Not *exactly*.  The definition is the same as your original except for the "dolist-while:" qualification.  Leave that out and newLISP will complain about the symbol being protected.

[Lutz]

The problem with the letex example was, that it passed code into the define-macro, then expanded and executed it in the define-macro's own variable environment.



Your last example behaves as expected per the rules of dynamic scoping: the variable x is bound locally in ant to 3, shadowing the definition of x as 7 in the enclosing scope of bar and will be printed as 3 in both cases. The variable y evaluated inside ant is not shadowed and therefore will evaluate to 7 from the enclosing variable scope.



But define-macros which do not bring in external code or variables for expansion or evaluation, can still be made safe without their own namespace when using the (args n) construct, e.g.:

Code Select Expand

(define-macro (my-add)
    (let (a (eval (args 0)) b (eval (args 1))) (+ a b)))

(set 'a 3 'b 4)

> (my-add a b)
7
> (my-add b a)
7


In all other cases, put the define-macro in it's own namespace for total separation of lexical environments of caller and define-macro.

[Lutz]

If you feel uncomfortable to program in a dynamically scoped environment, do the following:



(1) Never pass quoted symbols into functions. To pass references, use contexts, default functors.



(2) Never use free variables inside a function, except for variables clearly marked for global usage by some naming convention.



(3) Always put define-macros's in their own namespace. Or ignore them completely. They are not an essential newLISP feature as reader macros are in compiled LISPs.



Now, newLISP will behave like a lexically scoped language. If you are looking for a closure-like mechanism study this page for alternatives in newLISP:



http://www.newlisp.org/index.cgi?Closures">http://www.newlisp.org/index.cgi?Closures

[ant]

I think I'm starting to understand. Thank you every one who patiently explained and suggested alternatives.



I'm using macros because I'm passing large-ish objects around (bitmaps) and for performance reasons I don't want newLISP to create a copy of these objects every time I call a function passing one as an argument.



Possibly my problem was that when I read the doc on define-macro it said (still says) you can avoid variable capture by using (args n), but it didn't mention all the various caveats that I now have a little more understanding of thanks to this thread.

[Lutz]

Pass your bitmap around like this:

Code Select Expand

(set 'mybitmap:mybitmap '(1 2 3 4 ..... 5 6 7 8 9 0))

(define (process-bitmap bitmap)
...
...
)

(process mybitmap)


it's now passed by reference.



Or you could do something like this:

Code Select Expand

(set 'mybitmap:data '(1 2 3 4 5 ... 6 7 8 9 0))
(set 'mybitmap:config other-stuff)

(define (process bitmap)
; do something with bitmap:data
; and with bitmap:config
)

(process mybitmap)


last not least, look into FOOP, which also passes objects by reference.

[Kazimir Majorinc]

My experience is that dynamic scope almost never causes the problems. I think it happened to me something like two times in last four years, and then I fixed it easily. I have two functions in my library, protect1 and protect2 I'm using to check whether that is the case. I write something like

Code Select Expand

(define (my-function ...) ...) ; using variables x, y, z
(protect1 'my-function '(x y)) ; in case I want overshadowing of z, say, it is some global counter

and that protect1 will change the names of the variables in something like my-function.x, my-function.y, my-function.z, so accidental overshadowing with variables from other functions is impossible. There is no performance penalty for doing that. For example:

Code Select Expand

(set 'set-protected1
  (lambda(function/macro-name definition-code variables)
    (set function/macro-name
      (expand definition-code
              (map (lambda(x)
                        (list x (sym (string function/macro-name "." x))))
                        variables)))))
                       
(set 'protect1 (lambda(function/macro-name variables)
                 (set-protected1 function/macro-name
                                 (eval function/macro-name)
                                 variables)))

;-------------
; your code with additional 'protection'

    (define-macro (dolist-while)
      (letex (var (args 0 0)
              lst (args 0 1)
              cnd (args 0 2)
              body (cons 'begin (1 (args))))
        (let (res)
          (catch (dolist (var lst)
                   (if (set 'res cnd) body (throw res)))))))
                   
     (protect1 'dolist-while '(var lst cnd body res))              

    (define (test1 a-list)
      (dolist-while (x a-list (!= x 'd)) (println x)))

    (define (test2 lst)
      (dolist-while (x lst (!= x 'd)) (println x)))

    (test1 '(a b c d e f))
    (test2 '(a b c d e f))
;----------------

It works.



Here is how your macro looks like after 'protection':

Code Select Expand

(lambda-macro ()
 (letex (dolist-while.var (args 0 0) dolist-while.lst (args 0 1) dolist-while.cnd
   (args 0 2) dolist-while.body
   (cons 'begin (1 (args))))
  (let (dolist-while.res)
   (catch
    (dolist (dolist-while.var dolist-while.lst)
     (if (set 'dolist-while.res dolist-while.cnd)
      dolist-while.body
      (throw dolist-while.res)))))))

In most cases, I do not use that protect1, only if my program has error I cannot find, then I test whether it is caused by name overshadowing. But almost never it is. I use it "just in case" when I write functions for library. Particularly, I used protect1 to protect protect1.



This solution is equivalent to officially recommended use of contexts.



Very rarely, accidental overshadowing might happen between two instances of the same function or macro (if function calls itself recursively.) It is particularly severe form of the name overshadowing, it cannot be solved with protect1 or with contexts, and for that purpose, I have protect2. It is very sophisticated function, and its use has high price in performances, but the fact is, I never needed it, neither once. It sits in my library 'just in case'. For years now. Discussion on this forum has shown that noone actually had that problem at all.



With very little experience, one learns how to prevent accidental overshadowing, and problems are very rare in practice.

[ant]

Thanks again for the suggestions. I have a lot to learn!