Request for a $this variable

[TedWalther]

Lutz, $0-$15 are useful.  I just learnt about $idx today.  And $it is also useful.  Can you please add $this to refer to the current context?



One idiom I use a lot is to make an object the way I want it, then use (new) to duplicate it.  Having a $this variable will make it easier to call methods between contexts which then respond asynchronously by invoking another method in the calling context.

[Lutz]

you can use:

Code Select Expand

(context)

to get the current context. FOOP uses this in the default constructor:

Code Select Expand

(define (Class:Class)
    (cons (context) (args)))

(new Class 'MyClass)


See also here:

http://www.newlisp.org/downloads/newlisp_manual.html#system_symbols">http://www.newlisp.org/downloads/newlis ... em_symbols">http://www.newlisp.org/downloads/newlisp_manual.html#system_symbols

[TedWalther]

Thanks

[m35]

Might there be some value in making a $this refer to the current lambda expression being executed? Would that make self-modifying functions more portable?

[Lutz]

I have experimented with this quite a bit. newLISP internally keeps a lambda stack for error reporting. That makes it easy to access the current lambda.



In the end I left this idea alone because I couldn't come up with really usable use-cases. The only interesting usages are those where the current lambda expression was invoked by a symbol functor. But in those cases the lambda function can be accessed easily using the function name and a 'this' would not be required:

Code Select Expand

> (define (foo x) (inc (last foo) x) 0)
(lambda (x) (inc (last foo) x) 0)
> (foo 1)
1
> (foo 1)
2
> (foo 10)
12
>


All kinds of variations of above example have been shown over the years. The most interesting beeing Kazimir's code of a crawler-tractor, which runs eternally without using iteration or recursion:

Code Select Expand

(define (crawler-tractor)
    (begin (println "Hi for the " (inc counter) ". time. ")
           (push (last crawler-tractor) crawler-tractor -1)
           (when (> (length crawler-tractor) 3)
                 (pop crawler-tractor 1))))


All other cases, where the lambda is anonymous and a 'this' would be required, are really not that useful, or the task they accomplish can easier be solved using conventional code.



I believe that functions with state are best realized using namespaces, as discussed today in another thread and as shown in the docs as the method of choice for functions with state in newLISP:

Code Select Expand

(define (gen:gen)
    (inc gen:sum))

(gen) => 1
(gen) => 2




All these examples can also be found here: http://www.newlisp.org/index.cgi?Closures">http://www.newlisp.org/index.cgi?Closures



Ps: note that it is only possible to modify a lambda expression this way. Reassigning the function inside its body would crash the system:

Code Select Expand

(define (foo) (define (foo) "hello"))

(foo) => crash

[Kazimir Majorinc]

Really? I thought "$this" referencing the function is impossible, because of "one reference only."



I could think about two arguments for $this.



* anonymous functions changing themselves to store the state do not need to be explicitly deleted.

* the lists of the functions able to analyse or change themselves. Because of ORO these functions should reference themselves through their address in list, i.e. (L 1), (L 17) etc. It would be very fragile if someone pushes something in the list. Advantage of $this would be significant.



From practical point of view, self mutating functions can be used for similar purposes as closures. People seems to prefer contexts for such purposes, but that is another possibility.

[m35]

Hmm. Using self-modifying functions as closures might be pretty cool. You could create as many clones of those closures, each with their own state.



A very very sloppy example that could make use of the $this variable.

Code Select Expand

(define (truck-class func)
(let ((speed 100)
(color "red"))
(case func
("set speed" (setf
(lookup 'speed (nth '(1 1) truck-class))
(args 0)
))
("get speed" speed)
("set color"
(setf (lookup 'color (nth '(1 1) truck-class))
(args 0)
))
("get color" color)
)
)
)

(setq truck-instance truck-class)
(truck-instance "get color")         ; -> "red"
(truck-instance "set color" "blue")
(truck-instance "get color")         ; -> "red" (should be "blue")
(truck-class "get color")            ; -> "blue" (modified the original)

[Lutz]

Really? I thought "$this" referencing the function is impossible, because of "one reference only."

That is correct, and the way I had implemented this experimentally, was not with a variable $this but a function (this) returning a reference, so you could do:

Code Select Expand

(define (foo x) (push x (this) -1)))

(this) would always return a lambda expression, either referenced by 'foo' or an anonymous copy, if lambda was anonymous.

self mutating functions can be used for similar purposes as closures

Yes, and all patterns known so far are discussed here (among other stuff):



http://www.newlisp.org/index.cgi?Closures">http://www.newlisp.org/index.cgi?Closures



Kazimir's crawler-tractor was added to the closure page yesterday

You could create as many clones

m35's example makes a good case for a (this): when copying lambda expressions. But again: I think the code is too complex for what it achieves. It is also not very efficient (for an interpreted language), because method-dispatch is done wit a case statement.



Nevertheless it is an interesting approach and the last I want to do, is discourage further experimentation.



My current take on all of this is:



State is better captured using namespaces when dealing with object rarely deleted (because its not very efficient and manual).



For volatile objects, which come and go in bigger numbers FOOP is a better solution. FOOP keeps data and methods separate. Methods go into a class-namespace and instance-objects point to the class with their first list member. FOOP objects can be anonymous and are memory managed automatically.



Perhaps, as long as we try to use self-modifying functions to reproduce classic OO we are on the wrong track. newLISP is just not designed to be an classic OO language. The namespace system was primarily designed to facilitate grouping functions into modules, and this is how FOOP uses namespaces. Instead of trying to press newLISP into the traditional OO paradigm, perhaps we should try to come up with new ideas / paradigms. So far Michael's FOOP is the most efficient and elegant I have seen.

[itistoday]

The problem with contexts though is that they're global.



This makes it very difficult to write modules (libraries) for newLISP.



In most other languages, for example, it is no problem to define a class called "Record" or "Person" or "Employee". And while this is possible in newLISP, you'd be a fool if you did, because contexts are global and can't be nested.



So instead, you must adopt some sort of naming convention. For example, for Dragonfly I'm creating artificial nested contexts by having a convention of prepending words to them. For example, all RESTful resources should have "Resource." prepended to their name, and all routes have "Route." prepended to their name.



If newLISP implemented real reference counting (which it can since it controls the '=' operator, setf), all sorts of things would be possible, you could then easily implement anonymous functions that have their own anonymous state, and you could pass them around without copying the entire state/context. If you didn't want to implement closures the way Scheme does, you could instead implement the function 'this' to create an anonymous context attached to the function itself:

Code Select Expand

(define (counter) (fn () (inc (this x))))
(set 'cnt (counter))
(cnt) => 1
(cnt) => 2

'(this x)' would create (if it didn't exist) an anonymous context attached to the anonymous function containing the symbol 'x', and return a reference to x's data. You could do so many things with that, like create a real object oriented library where state is modified immediately, without creating an entire copy of all of the object's members. You would also obviously have anonymous closures in newLISP, and many of the name-conflict worries would go away.

[m35]

I'm really impressed how newLISP's very simple memory management system (ORO: one-reference-only) has worked so well for so long.



Its simplicity makes its execution very predictable. You don't have to deal with a garbage collector arbitrarily pausing your scripts, and you don't have to worry about circular reference counting memory leaks. I'm sure it also keeps the executable smaller and easier to maintain and understand. It's also somewhat surprising that essentially no performance issues have come from the simple model. Lutz even managed to optimize much of it in the new v10.



For everything that newLISP can do, the ORO model has rarely been a problem, and then contexts can handle many of those problematic cases.



A new memory management system would be a huge change to newLISP. I would vouch for keeping ORO, if nothing else, just to be a real-life example that GC may not always be the ultimate solution for memory handling.





I would however like to see some changes made to context handling. As itistoday has shown, it really seems to have limitations.

[cormullion]

contexts are global and can't be nested.

I remember a similar discussion some months ago. I don't remember Lutz saying that any changes to the context system were on the cards...

[itistoday]

It's also somewhat surprising that essentially no performance issues have come from the simple model. Lutz even managed to optimize much of it in the new v10.

I agree what Lutz has done is great, but I see some fundamental issues with ORO, and one of them, although I could be wrong on this, is performance. I find it very hard to believe that copying lots and lots of lists all over the place, all the while allocating and deallocating that memory, is more efficient than simply incrementing or decrementing a register and testing it for 0.

I'm really impressed how newLISP's very simple memory management system (ORO: one-reference-only) has worked so well for so long.



Its simplicity makes its execution very predictable. You don't have to deal with a garbage collector arbitrarily pausing your scripts, and you don't have to worry about circular reference counting memory leaks. I'm sure it also keeps the executable smaller and easier to maintain and understand.

When I say reference counting, I don't mean garbage collection, and in the scheme I'm advocating I think we could avoid the circular reference issue. Please let me know if I'm speaking gobble-de-gook (as I very well may be, I haven't studied this issue in depth), but here is what I mean:



In other languages, such as C, the '=' operator simply assigns values to registers, nothing more.



1) newLISP, and other interpreted languages, have the capability of overriding what '=' does.

2) newLISP could also, similar to Clojure, separate values from their symbols.



I've taken some time to think all this through, and I've provided my full thoughts on the issue (as well as how it could actually be done) here:



"RFC: A reference counted newLISP (and how to implement it)."

http://newlispfanclub.alh.net/forum/viewtopic.php?f=8&t=3151">http://newlispfanclub.alh.net/forum/vie ... f=8&t=3151">http://newlispfanclub.alh.net/forum/viewtopic.php?f=8&t=3151

[Kazimir Majorinc]

Newlisp with ORO is similar to assembler. Just like in assembler, one object cannot be stored on two memory addresses, in Newlisp, one list or function cannot be the value of two different symbols. In assembler, if one wants the same object on two addresses, he has to copy. Since I realized that, I think about Newlisp symbol as about humanized, but also generalized memory address. (In assembler, one cannot delete memory address or insert new between two, in Newlisp one can do that with symbols.) So, how assembler programmers deal with problems that occur here, like excessive copying? They use indirection, not just addresses, but addresses to addresses. If list is stored in memory at address 100, assembler programmer can store number 100 into cells with addresses 500 and 600 and now 500 and 600 "indirectly" contain list. The problem is solved. Programming is bit more complicated, but that's it. Same indirection need to be used in Newlisp, and problem is solved in principle.



(set 'private-1889 (lambda()...))

(set 'f 'private-1889)

(set 'g 'private-1889)



Now, f and g indirectly evaluate to same function, and there was no copies. Of course, it is now "indirect" so one always must use that extra level of indirection, write (push 3 (eval f)) etc, but it is not essential problem. It is like pointers in C.



In real life, we automatically handle such indirections. I give you telephone number of pretty girl that want to meet you, and when you call that number, the voice tells that she has new phone number. If it happens, you do not simply proclaim ERROR]> (setf mysin 'sin)

sin

> (mysin 3)



ERR: invalid function : (mysin 3)

>[/color]



It could - because that change only damages previous error behaviour. And it actually does that - with contexts:



> (set' MYSIN]



Here Newlisp wasn't insulted with context, but it automatically looked into its default functor. Contexts are limited on the way they do not allow further nesting, but with symbols, one can use as much indirection as he needs.  He losses ORO and GC didn't existed, so he is turned back to manual memory management, like in C. Not that bad. I'll stop here.



Itistoday: it was not the answer on your proposal, I just think loudly about nature of the ORO.



--

http://www.instprog.com/various/Tocak-ORO.mp3">ORO by Tocak, the rock version of Macedonian folk dance music.

[Lutz]

Misconceptions about ORO memory management:



- Memory gets allocated and deallocated all the time, and much cell copying is going on.



Not al all. ORO (One Reference Only) preallocates cell heaps and keeps a free-list from which cells are recycled and reused. Most of the time, copying or deleting does not mean allocating or freeing memory.



ORO is much about deciding what happens to a new created Lisp object in the future. From investigations other researchers did for Lisp and other languages we know, that most newly created objects can go away soon.



Traditional memory management methods manage much more memory objects than ORO, which schedules many for deletion right away, even for a slightly increased cost of copying. ORO does this with its result-stack; it holds result objects marked for deletion (recycling) in the future.



- ORO management is similar to C.



Far away from it. The result-stack in ORO is not a C stack. The C stack is used do push parameters and pop results after subroutine calls. ORO passes parameters using variables symbols and uses a symbol environment stack to keep track of symbol values up the calling hierarchy.



Also, in C memory is not managed automatically, if you copy a string, you have to allocate memory for the copy then do the copy. If this was just an intermediate result in a complex expression, you have to free that memory yourself.



- ORO is new in newLISP and unproven.



ORO was developed over the years and started out on two language projects before newLISP. The first was a string manipulating language similar to Tcl or Snobol, called DO, the second was a Prolog dialect called RuleTalk. Read more about it at http://www.donlucio.net">http://www.donlucio.net on the projects page. DO was used in more then 150 companies in the early 80's to implement accounting and other IT software on early micro computers. RuleTalk was sold commercially and used mostly by European AI researchers on the DOS/MS-Windows PC.



- Automatic memory management follows just one method or the other.



Not at all. Most languages use a mixture of different techniques. There are many flavors of garbage collection. Many years, newLISP also had a mark and sweep garbage collector working under certain conditions. It was  used less and less over the years removed completely only recently in v. 10.1.3.



Over time ORO has gotten less pure. In the beginning every memory object was treated the same way and only 'set' was optimized. Today all destructive built-ins have the same optimization which avoids to copying and pushing the result stack.

[itistoday]

Thanks Lutz for clarifying some of those misconceptions about ORO!

Misconceptions about ORO memory management:



- Memory gets allocated and deallocated all the time, and much cell copying is going on.



Not al all. ORO (One Reference Only) preallocates cell heaps and keeps a free-list from which cells are recycled and reused. Most of the time, copying or deleting does not mean allocating or freeing memory.

That's good to know that it doesn't allocate/free memory each time, but isn't copying data still a possibly significant overhead that could be avoided through reference counting?



I readily believe that it's possible to make ORO efficient in certain ways, but my main gripe with it is not even efficiency, but rather how it limits certain coding possibilities, or provides difficult and unclean workarounds to problems that could be easily solved if everything was passed by reference. It just so happens that passing things by reference is a very efficient way to do stuff, and while there might be some overhead in other areas to implement such a scheme, just as you've implemented optimization strategies for ORO, I'm sure strategies could be implemented for the sort of reference counting I'm referring to. It could result in an even faster language, and it would certainly result in more code possibilities.