getting a loop index for dolist

[cormullion]

A quickie for you. I find I write this quite a bit:

Code Select Expand

(set 'counter 0)
(dolist (x a-list )
   ...
   (inc 'counter)
   )


ie I want to iterate over a list but also keep count of how many loops have been done.



Is there a way to find out the loop index inside a (dolist) loop? Or is it possible to write a macro that could do it? Ideally this is the type of construction:

Code Select Expand


(dolist-index (e a-list i)
    (println "element " i " of " a-list " is " e " ))


Is it possible that newLISP secretly 'knows' how many iterations it's done at a certain stage in the loop?

[rickyboy]

Given a file called 'dolist-index.lsp' with the following contents,

Code Select Expand

(context 'dolist-index)

(define-macro (dolist-index:dolist-index ctrl-form body)
  (letn ((elt (ctrl-form 0))
         (lst (ctrl-form 1))
         (idx (ctrl-form 2))
         (inc-form (list 'inc (expand ''idx 'idx)))
         (dolist-form (cons 'dolist
                            (cons (list elt lst)
                                  (append (list body)
                                          (list inc-form))))))
    (eval (expand '(let ((idx 0))
                     dolist-form)
                  'idx 'dolist-form))))

(context MAIN)


the following applications seem to work.

Code Select Expand

> (load "dolist-index.lsp")
true
> (setq a-list '(1 2 3 4))
(1 2 3 4)
> (dolist-index (e a-list i) (println "element " i " of " a-list " is " e ))
element 0 of (1 2 3 4) is 1
element 1 of (1 2 3 4) is 2
element 2 of (1 2 3 4) is 3
element 3 of (1 2 3 4) is 4
4


I would be interested if someone could show me a shorter solution.  I hate to say this, given Lutz's detestation of this, but backquote-style list splicing would have made the definition shorter.

[Lutz]

... or starting with the last development version 8.8.6 which introduced 'letex' you can combine Rick's steps into a shorter form:

Code Select Expand


> !cat dolist-index
(define-macro (dolist-index)
  (letex (var ((args) 0 0)
          lst ((args) 0 1)
          idx ((args) 0 2)
          body (cons 'begin (1 (args))))
          (set 'idx 0)
          (dolist (var lst)
              body
              (inc 'idx) )
   )
)

> (dolist-index (i '(a b c d e f) id) (println id ":" i))
0:a
1:b
2:c
3:d
4:e
5:f
6


Lutz



ps: ther usage of (args) makes the macro hygienic, allthough Ricks's hygienic method using an isolated default function would make the code more readable and perhaps faster?

[rickyboy]

Thank you Lutz!  Your version of 'dolist-index' is better for two reasons: (1) it is shorter and easier to read and (2) you exposed a bug in my version: I only assigned the first form of the 'dolist-index' body to 'body', instead of all of them -- yours is correct by usage of '(1 (args))'.



Lutz, could you also clear up some confusion I have about 'letex'?  I realized before that you claimed that the use of 'letex' allowed for hygenic macros, but I don't understand how this works.  Please let me proceed by illustration.  I tried the following in 8.8.6 (where 'dolist-index' is given by your definition).

Code Select Expand

> (set 'idx 42)
42
> (dolist-index (i '(a b c d) idx) (println idx ":" i))
0:a
1:b
2:c
3:d
4
> idx
42


Wow!  I thought the macro would trounce the 'idx' at the top level, but it did not.  I thought this because the macro definition did not scope 'idx' with a 'let'.  Look at the difference between the above and the following, which is an expanded version of it.

Code Select Expand

> (set 'idx 42)
42
> (begin (set 'idx 0) (dolist (i '(a b c d)) (println idx ":" i) (inc 'idx)))
0:a
1:b
2:c
3:d
4
> idx
4


As we expect, the "expanded" version trounces the top level 'idx'.  So clearly, the use of 'letex' in the macro is doing a little more than just expansion and evaluation.  'letex' must somehow be locally scoping the symbols which get expanded into the 'letex' body (that is, the symbols of the application, e.g. 'i' and 'idx'; not the the symbols of the definition, i.e. 'var', 'lst', 'idx', and 'body').



Lutz, could you explain a little more about how this happens?  Thanks a lot!

[Lutz]

I corrected the original post, but now when passing idx the new 'letex' crashes in 8.8.7.



The reason idx behaved as it did in your example, is because of the way dynamic scoping works. All local vars in initializers of let, letn, letx and loops like for, dotimes, dolist and dotree are local with dynamic scope. This means the same symbol is used locally but dynamic scoping saves the value of local symbols and restores them after returning from the 'let', 'letex'  etc.

Code Select Expand


> (define (report str) (println str "->" idx))
(lambda (str) (println str "->" idx))
> (set 'idx 123)
123
> (let (idx 4) (report "inside let" idx ))
inside let->4    ;; 123 in a static scoped language
4
> (report "outside let" idx)
outside let->123
123  
>


When inside 'let' the toplevel value of 'idx' is shadowed and all routines called from inside 'let' are in the dynamic scope of the caller



Lutz

[Lutz]

I edited the post again, following version works without a problem passing either 'id' or the 'idx' and 'idx' will be preserved with it's toplevel value.

Code Select Expand


(define-macro (dolist-index)
  (letex (var ((args) 0 0)
          lst ((args) 0 1)
          idx ((args) 0 2)
          body (cons 'begin (1 (args))))
          (set 'idx 0)
          (dolist (var lst)
              body
              (inc 'idx) )
   )
)

> (dolist-index (i '(a b c d e f) id) (println id ":" i))
0:a
1:b
2:c
3:d
4:e
5:f
6
> (dolist-index (i '(a b c d e f) idx) (println idx ":" i))
0:a
1:b
2:c
3:d
4:e
5:f
6
>


but there are still crashing issues with the new 'letex', which I have to investigate.



Lutz

[rickyboy]

OK, thanks to your last example, it's become clear to me how this works -- but not without a pitfall in usage which I should mention.  Consider the following.

Code Select Expand

;; Your last definition of 'dolist-index':
(define-macro (dolist-index)
  (letex (var ((args) 0 0)
          lst ((args) 0 1)
          idx ((args) 0 2)
          body (cons 'begin (1 (args))))
          (set 'idx 0)
          (dolist (var lst)
              body
              (inc 'idx) )
  )
)

> (set 'idx 42)
42
> (dolist-index (i '(a b c d e f) idx) (println idx ":" i))
0:a
1:b
2:c
3:d
4:e
5:f
6
> idx
42

;; But now look at this:

> (set 'id 42)
42
> (dolist-index (i '(a b c d e f) id) (println id ":" i))
0:a
1:b
2:c
3:d
4:e
5:f
6
> id
6


Aha!  So, we can trounce a top level 'id' with the use of the 'dolist-index' macro.  Hence, we do need a local 'let' scoping for 'idx' (which expands to 'id') in the body of the definition of the 'dolist-index' macro.

Code Select Expand

(define-macro (dolist-index)
  (letex (var ((args) 0 0)
          lst ((args) 0 1)
          idx ((args) 0 2)
          body (cons 'begin (1 (args))))
          (let (idx 0)
            (dolist (var lst)
               body
               (inc 'idx)))))

> (set 'id 42)
42
> (dolist-index (i '(a b c d e f) id) (println id ":" i))
0:a
1:b
2:c
3:d
4:e
5:f
6
> id
42


There, that's better.  :-)



So, I guess my confusion before was due to the fact that I was using 'idx' for my index variable reference in the usage.  This meant that the 'letex' was binding its symbol 'idx' to the symbol 'idx' passed in by 'args' which happened to be the same symbol -- in my usage they are both 'MAIN:idx'.  Hence, 'idx' was being locally bound by the 'letex', essentially, for free.



But when 'idx' is bound to the symbol 'id' instead, we don't get the local 'letex' binding for free (since '(= 'idx 'id)' does not hold -- duh).  That's why we needed to add the 'let' in our macro definition.



So, my wrongheaded thought that there was something magical going on with 'letex' beyond expansion and evaluation was due to my poor choice of symbol in my example, combined with just the right amount of muddled thinking.  Sorry about that.  :-)

[Lutz]

localizing the symbol passed with another expanded 'let' is a nice addition and makes the whole thing squeaking clean hygienic now ;)



the upcoming 8.8.7 also adds to more modes to 'expand', here a preview of the manual examples:

Code Select Expand


(expand '(a b c) '((a 1) (b 2))) => (1 2 c)
(expand '(a b c) '((a 1) (b 2) (c (x y z)))) => (1 2 (x y z))


there is no assignment here, vars are the same before, during and after, there is also no evaluation of initializers in the assoc-list, so this is not the same as using 'let' with an expand in the body. There is also not evaluation of the expanded expression as in 'letex'. 'expand' in this form is useful when there is an variable association list from another computation.

Code Select Expand


(set 'A 1 'Bvar 2 'C nil 'd 5 'e 6)
(expand '(A (Bvar) C d e f)) => (1 (2) C d e f)


this would be the shortest form only uppercase non-nil variables are expanded with their contents.



both forms are useful together with 'unify', specially the second one working only on prolog uppercase variables. But both are also useful and usable in other situations.



we have now very detailed control of expansion, not only for usage in macros but also to be used in other situations.



Lutz

[rickyboy]

The new stuff are great additions to newlisp.  Thanks for that and also thanks for explaining evalexpansion.  Cheers, --Ricky

[cormullion]

It's been great to see you guys sort it out. I confess that I don't feel too comfortable in this area, yet, although I can see the power, and one day I hope to set aside some time to learn it.



(I also think this "dolist-index" capability would be nice to have inside dolist one day... :-))

[cormullion]

Hey, Lutz, you've added this option in 8.8.8! How cool is that...! Thanks.



I really like the way you surprise me by implementing a feature request. So many other developers promise to do it or say that they might do it one day and never do. :-)

[HPW]

I really like the way you surprise me by implementing a feature request. So many other developers promise to do it or say that they might do it one day and never do. :-)

And Lutz does this for so many years now! Outstanding!



You can have a look to the early days here:

http://www.alh.net/newlisp/cyphor19/">http://www.alh.net/newlisp/cyphor19/

And newLISP has a history even before that forum.



;-))

[cormullion]

You can have a look to the early days here:

http://www.alh.net/newlisp/cyphor19/">http://www.alh.net/newlisp/cyphor19/

Interesting stuff!



Now I admire Lutz's patience and understanding of us newbies even more. It can't be easy to patiently explain the obvious for the umpteenth time!