My time has suddenly been crunched. I'll return to start on QA work when I'm able.
-Noah
- Welcome to newLISP Fan Club.
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
#1
newLISP newS / Regarding QA work on newLISP
August 02, 2006, 03:13:44 PMMy time has suddenly been crunched. I'll return to start on QA work when I'm able.
-Noah
#2
Anything else we might add? / Suggestions for your functional regex approach.
July 31, 2006, 06:14:46 PMOnce you're happy with your code, check whether it's suitable for publishing as a part of newLISP's main module collection.
Check the Perl documentation for use cases and examples of Perl-extended regex. Provide comparable use cases and examples in the documentation for your module.
Consider if you'd like to integrate your module into a GUI PCRE authoring tool (for example, KDE offers such a tool). You could write a TK add-on to the newLISP TK editor, or write a separate application. Your application would have use outside the newLISP community as well.
-Noah
#3
newLISP newS / Suggested edits to the Automatic Memory Management page
July 25, 2006, 12:31:23 PMHere are some suggested edits to the newly-posted rev 13 version of the Automatic Memory Management document.
-Noah
Code Select
- This article explains how ORO memory mangement works.
+ This article explains how ORO memory management works.
- A ireference-counting scheme registers each allocated memory object together with a count of references to the object.
+ A reference-counting scheme registers each allocated memory object together with a count of references to the object.
- When newLISP reaches a higher evaluation level, it removes the last evaluation results reference from the result stack and deletes the evaluation result's memory object.
+ When newLISP reaches a higher evaluation level, it removes the last evaluation result's reference from the result stack and deletes the evaluation result's memory object.
- If a list is packaged int a context (a namespace) in newLISP, then newLISP can pass the list by reference.
+ If a list is packaged in a context (a namespace) in newLISP, then newLISP can pass the list by reference.
- Arrays in newLISP are LISP cells allocated in memory in a linear fashion for faster random access when using indices. Only a subset of the list functions can be used on arrays. Automatic memory management in newLISP handles arrays similar to lists.
+ newLISP allocates an array as a group of LISP cells. The LISP cells are allocated linearly. As a result, array indices have faster random access to the LISP cells. Only a subset of newLISP list functions can be used on arrays. Automatic memory management in newLISP handles arrays in a manner similar to how it handles lists.
- Refererences
+ References
- Addision Wesley Publishing Company
+ Addison-Wesley Publishing Company
#4
newLISP newS / Suggested edit to the Memory Management Page
July 25, 2006, 02:07:13 AMHere's my first suggested edit to the Memory Management page. The edit contains two alternative paragraphs that cover the discussion of the non-use of one-bit (non)sticky counts. Let me know what's what about that; I really didn't know how to interpret the source paragraph.
I attempted a grammar/syntax edit here, but it crept over to a deep edit. Let me know if any mistakes, I'll happily rework what I've posted here.
Michael, Cormullion, if you have suggestions to improve or correct this suggested edit, feel free to post them to this thread or message them to me.
-Noah
Code Select
Automatic Memory Management in newLISP
Lutz Mueller, 2005-09-26 rev 12
newLISP and any other interactive language system will constantly generate new memory objects during expression evaluation. The new memory objects are intermediate evaluation results, reassigned memory objects, or memory objects whose content was changed. If newLISP did not delete these objects, it would eventually run out of available memory.
In order to understand newLISP's automatic memory management, it is necessary to first review the traditional methods employed by other languages.
Traditional automatic memory management (Garbage Collection)
In most programming languages, a process registers allocated memory, and another process finds and recycles the unused parts of the allocated memory pool. The recycling process can be triggered by some memory allocation limit or can be scheduled to happen between evaluation steps. This form of automatic memory management is called Garbage Collection.
Traditional garbage collection schemes employ one of two algorithms:
(1) The mark and sweep algorithm registers each allocated memory object. A mark phase periodically flags each object in the allocated memory pool. A named object (a variable) directly or indirectly references each memory object in the system. The sweep phase frees the memory of the marked objects when they are no longer in use.
(2) A reference-counting scheme registers each allocated memory object together with a count of references to the object. This reference count gets incremented or decremented during expression evaluation. Whenever an object's reference count reaches zero, the object's allocated memory is freed.
Over time, many elaborate garbage collection schemes have been attempted using these algorithms. The first garbage collection algorithms appeared in LISP. The inventors of the Smalltalk language used more elaborate garbage collection schemes. The history of Smalltalk-80 is an exciting account of the challenges of implementing memory management in an interactive programming language; see [Glenn Krasner, 1983 Smalltalk-80, Bits of History, Words of Advice]. A more recent overview of garbage collection methods can be found in [Richard Jones, Rafael Lins, 1996 Garbage Collection, Algorithms for Automatic Dynamic Memory Management].
One reference only (ORO) memory management
Memory management in newLISP does not rely on a garbage collection algorithm. Memory is not marked or reference-counted. Instead, a decision whether to delete a newly created memory object is made right after the memory object is created.
{Empirical studies of LISP have shown that most LISP cells are not shared and so can be reclaimed during the evaluation process. Aside from some optimizations for primitives like set, define, and eval, newLISP deletes memory objects containing intermediate evaluation results once it reaches a higher evaluation level. newLISP does this by pushing a reference to each created memory object onto a result stack. When newLISP reaches a higher evaluation level, it removes the last evaluation result's reference from the result stack and deletes the evaluation result's memory object. This should not be confused with one-bit reference counting. ORO memory management does not set bits to mark objects as sticky. }
- OR -
{ Empirical studies of LISP have shown that most LISP cells are not shared and so can be reclaimed during the evaluation process. newLISP deletes memory objects containing intermediate evaluation results once it reaches a higher evaluation level. newLISP does this by pushing a reference to each created memory object onto a result stack. When newLISP reaches a higher evaluation level, it removes the last evaluation result's reference from the result stack and deletes the evaluation result's memory object. This should not be confused with one-bit reference counting. newLISP's memory management does not set bits to mark objects as sticky, aside from its optimizations for primitives like set, define, and eval. }
newLISP follows a one reference only (ORO) rule. Every memory object not referenced by a symbol or context reference is obsolete once newLISP reaches a higher evaluation level during expression evaluation. Objects in newLISP (excluding symbols and contexts) are passed by value to other functions. As a result, each newLISP object only requires one reference.
newLISP's ORO rule has advantages. It simplifies not only memory management but also other aspects of the newLISP language. For example, while users of traditional LISP have to distinguish between equality of copied memory objects and equality of references to memory objects, newLISP users do not.
newLISP's ORO rule also has disadvantages. It forces newLISP to constantly allocate and then free LISP cells. newLISP optimizes this process by allocating large chunks of cell memory from the host operating system. newLISP will request LISP cells from a free cell list and then recycle those cells back into that list. As a result, only a few CPU instructions (pointer assignments) are needed to unlink a free cell or to re-insert a deleted cell.
The overall effect of ORO memory management is a faster evaluation time and a smaller memory and disk footprint than traditional LISP's can offer. The lack of garbage collection in newLISP more than compensates for its high frequency of cell creation/deletion. Note that under error conditions, newLISP will employ a mark and sweep algorithm to free un-referenced cells.
Performance considerations with value passing
Passing parameters by value (memory copying) instead of by reference poses a potential disadvantage when dealing with large lists. For practical purposes, however, the overhead needed to copy a large list is negligible compared to the processing done on the list. Nevertheless, to achieve maximum performance, newLISP offers a group of destructive functions that can efficiently create and modify large lists. While cons and set-nth return a new memory object of the changed list, push, pop and nth-set change the existing list and only return a copy of the list elements that they added or removed. In order for any function to operate destructively on a large list, the large list must be passed by reference. If a list is assigned to a context (a namespace) in newLISP, then newLISP can pass the list by reference. newLISP contexts are the best choice when passing big lists or string buffers by reference.
In general, the speed of ORO memory management more than compensates for the overhead required to pass parameters by value.
Memory and data types in newLISP
The memory objects of newLISP strings are allocated from and freed to the host's OS whenever newLISP recycles the cells from its allocated chunks of cell memory. This means that newLISP handles cell memory more efficiently than string memory. As a result, it is often better to use symbols than strings for efficient text processing. For example, when handling natural language it is more efficient to handle natural language words as individual symbols in a separated name-space, rather than as a single string. The 'spam-filter' program in the newLISP source distribution uses this method. newLISP can handle millions of symbols without degrading performance.
Programmers coming from other programming languages frequently overlook that symbols in LISP can act as more than just variables or object references. The symbol is a useful data type in itself, which in many cases can replace the string data type.
Integer numbers and double floating-point numbers are stored directly in newLISP's LISP cells and do not need a separate memory allocation cycle.
For efficiency during matrix operations like matrix multiplication or inversion, newLISP allocates non-cell memory objects for matrices, converts the results to LISP cells, and then frees the matrix memory objects.
References
- Glenn Krasner, 1983. Smalltalk-80, Bits of History, Words of Advice
Addison-Wesley Publishing Company
- Richard Jones, Rafael Lins, 1996 Garbage Collection, Algorithms for Automatic Dynamic Memory Management
John Wiley & Sons
#5
newLISP newS / Regarding editing the Memory Management document
July 24, 2006, 05:07:55 PMLutz, I'm half done with a first suggested edit to the Memory Management document. I started this morning, after reading your post mentioning it.
-Noah
#6
newLISP newS / Regarding newLISPdoc
July 22, 2006, 11:22:56 PM-Noah
#7
newLISP newS / Regarding QA testing for newLISP modules
July 22, 2006, 04:53:06 PMDo you think its a good idea to include dependency information in the newLISPdoc fields? Should the newLISPdoc format include sufficient information from which to generate test suites, or is that outside the purpose of newLISPdoc?
-Noah
#8
newLISP newS / newLISP core development expectations
July 22, 2006, 02:32:51 PMWhat do you think about QA testing basic newLISP functionality? For example, a set of tests of newLISP functions called under normal, boundary, and error conditions. I do not have lots of time, but I can offer some time toward that effort.
What do you think about module pre-installation tests? newLISP's module installation process could verify that a target system meets the module requirements before newLISP installs the module. For example, a useful test of the ps module is a check that Ghostscript is available on the target system.
If you wanted, you could standardize how developers create pre-installation module tests. One way is to specify fields in the newLISPdoc format that another program uses automatically to create a pre-installation test for a documented module (for example, a test of the module's dependencies on other modules).
-Noah
#9
newLISP newS / About that newLISP executable
July 21, 2006, 10:45:33 PMNot meaning to ignore your point.
-noah
#10
newLISP newS /
July 21, 2006, 10:44:27 PMLutz brought up the idea of embedding documentation in Javadoc-style comments in newLISP modules. He might be working on it, I don't know.
-Noah
#11
Anything else we might add? / Edits to the Differences page
July 18, 2006, 12:44:19 AMHere's a proposed edit. This one includes corrections to my last proposed edit.
-Noah
Code Select
newLISP compared to Common Lisp and Scheme
Last updated July 12th, 2006
This page clarifies some differences between newLISP and the older standards Common Lisp and Scheme. Read the About page to find out more about newLISP's unique approach to LISP.
1. Unlike Common Lisp, newLISP and Scheme evaluate the operator part of an expression before applying the operator to its arguments.
2. In newLISP, lambda expressions evaluate to themselves. They are a subtype of the list data type, a first-class data object that can be manipulated like any other list. In Common Lisp and Scheme, lambda expressions, when evaluated, return a special 'function' data type with its free variables binding to the current environment and forming a lexical closure.
In newLISP, binding of variables in lambda expressions takes place during application of the lambda expression. In other LISPs, lambda expressions are used to realize static variables and content via lexical closures.
Static variables are a precondition for a programming language to allow object-oriented programming in that language. In newLISP, lexically closed contexts (namespaces) are used for prototype-based object-oriented programming. Like lambda expressions, contexts in newLISP are first-class objects. Contexts can be created and destroyed during runtime, passed as parameters, and referred to by symbols.
3. In newLISP and Scheme, variables, primitives, and user-defined functions share the same symbol space. In Common Lisp, function symbols and variable symbols each use a dedicated symbol space. This is why Common LISP function symbols sometimes must be prefixed with the sharp quote #'.
4. In newLISP, all variables are dynamically scoped by default. However, by defining a function in its own context, static/lexical scoping can be achieved and variable capture can be avoided. In newLISP, functions and data can share a namespace. Common Lisp and Scheme are lexically scoped by default and use lambda expressions as the closure mechanism. Common Lisp also offers 'special variables' for dynamic scoping.
5. In Common Lisp and Scheme, the 'cdr' part of the lisp cell can be used to contain another data object, in which case we have a dotted pair. In newLISP, there are no dotted pairs. Instead, each newLISP cell contains one object and a pointer to another object if the cell is part of a list. As a result, 'cons' behaves differently in newLISP than in other LISPs.
;; Common Lisp and Scheme
(cons 'a 'b) => (a . b) ; a dotted pair
[a | b]
;; newLISP
(cons 'a 'b) => (a b) ; a list
;; LISP cells in newLISP
(+ 2 3 (* 4 3))
[ ]
[+] -> [2] -> [3] -> [ ]
[*] -> [4] -> [3]
6. In newLISP, all arguments to a user-defined function are optional. Unassigned argument variables will assume the value 'nil' inside their function.
7. Logically, there are no unbound or non-existing symbols in newLISP. Any unbound or non-existing symbol is bound to 'nil' in the current namespace when it is first used.
8. 'nil' and 'true' are Boolean constants in newLISP. In Common Lisp, 'nil' has an additional role as a list terminator:
;; Common Lisp
(cons 'x nil) => (x)
;; newLISP
(cons 'x nil) => (x nil)
Scheme has the two Boolean constants '#t' and '#f' for true and false.
9. In newLISP, every object is only referenced once (ORO), with the exception of symbols, contexts, and context objects. newLISP's ORO rule allows automatic, stack-based, 'on the go' memory management without the traditional garbage collection algorithms used in Common Lisp and Scheme. newLISP passes parameters by value and stores intermediate results on a 'result stack' where their memory gets recycled after function return. Only under error conditions will newLISP employ a classic mark-and-sweep algorithm to reclaim unreferenced memory.
newLISP's automatic memory management is fully transparent to the programmer but faster and lighter on resource requirements when compared to classic garbage collection algorithms. Because ORO memory management is synchronous, newLISP code has constant and repeatable execution time. The combination of newLISP's value-passing style and unique memory management make it the fastest interactive (non-compiled) LISP available and one of the fastest scripting languages available in general.
10. In newLISP, macros don't evaluate their arguments. As a result, newLISP macros can work like built-in newLISP primitives:
(define-macro (my-setq x y) (set x (eval y)))
;; or the same avoiding variable capture
(define-macro (my-setq) (set (args 0) (eval (args 1))))
newLISP offers better control over parameter evaluation than Common Lisp and Scheme offer. In newLISP, a built-in function 'expand' can be used to initiate variable expansion explicitly:
(define (raise-to power)
(expand (fn (base) (pow base power)) 'power))
(define square (raise-to 2))
(define cube (raise-to 3))
(square 5) => 25
(cube 5) => 125
11. newLISP only requires the equal sign '=' to test for equality. This is a consequence of newLISP's value-passing style. Common Lisp requires 'eq', 'eql', 'equal', 'equalp', '=', 'string=', 'string-equal', 'char=', and 'char-eq', for equality tests of expressions, data types, identical objects, and referenced objects.
12. newLISP features implicit indexing. This is a logical extension of LISP evaluation rules overloading lists or numbers with the indexing functionality available from built-in list and string functions like nth, rest or slice, i.e.:
(set 'myList '(a b c (d e) f g))
; using nth
(nth 2 myList) => c
(nth 3 1 myList) => e
(nth -3 0 myList) => d
; using implicit indexing
(myList 2) => c
(myList 3 1) => e
(myList -3 0) => d
; implicit rest, slice
(1 myList) => (b c (d e) f g)
(-3 myList) => ((d e) f g)
(1 2 myList) => (b c)
Using implicit indexing is optional, but in many instances it speeds up coding and increases readability.
#12
Anything else we might add? / Edits to the Differences page
July 14, 2006, 09:01:28 PMThank you for your helpful reply. I understand that you'd like the Difference page to read well for educated LISPers.
Your example of granular argument evaluation shows CL and Scheme coders what you mean, without requiring additional explanatory text, so it would fit in well, if you want to add it.
Here's my proposal for an edit that includes your example and does not add explanatory text:
Code Select
---------------
In newLISP macros work like normal functions, but without evaluating their arguments. This way functions can be built, which look like built-in newLISP primitives. In newLISP a built-in function 'expand' can be used to initiate variable expansion in macros or anywhere else explicitely:
(define (raise-to power)
(expand (fn (base) (pow base power)) 'power))
(define square (raise-to 2))
(define cube (raise-to 3))
(square 5) => 25
(cube 5) => 125
Separating evaluation and expansion of parameters offers more detailed control.
---------------
+++++++++++++++
In newLISP, macros don't evaluate their arguments. As a result, newLISP macros can work like built-in newLISP primitives:
(define-macro (my-setq x y) (set x (eval y)))
;; or the same avoiding variable capture
(define-macro (my-setq) (set (args 0) (eval (args 1))))
newLISP offers better control over parameter evaluation than Common Lisp and Scheme do. In newLISP, a built-in function 'expand' can be used to initiate variable expansion explicitly:
(define (raise-to power)
(expand (fn (base) (pow base power)) 'power))
(define square (raise-to 2))
(define cube (raise-to 3))
(square 5) => 25
(cube 5) => 125
++++++++++++++
-Noah
#13
Anything else we might add? / Edits to the "Differences" page
July 14, 2006, 02:21:51 PMHere are some suggested edits to the "Differences_to_Other_Lisps" page.
In section 1, you wrote:
Like Scheme, newLISP evaluates the operator part of an expression before applying it to the arguments.
Is that in contrast with Common Lisp? I'm not sure how you're identifying a difference from other LISP's.
In section 4, you wrote:
In newLISP macros work like normal functions, but without evaluating their arguments. This way functions can be built, which look like built-in newLISP primitives.
...
Separating evaluation and expansion of parameters offers more detailed control.
How does not evaluating macro arguments let you create a function that looks like a built-in newLISP primitive? Put another way, what do you mean by 'look like built-in newLISP primitives'?
Also, can you give an example on this page that shows when you would not want to use the 'expand' function in a macro?
Code Select
newLISP compared to Common Lisp and Scheme
Last updated July 12th, 2006
This page clarifies some differences between newLISP and the older standards Common Lisp and Scheme. Read the About page too find out more about newLISP's unique approach to LISP.
1. Like Scheme, newLISP evaluates the operator part of an expression before applying it to the arguments.
2. In newLISP, lambda expressions evaluate to themselves. They are a subtype of the list data type, a first-class data object that can be manipulated like any other list. In Common Lisp and Scheme, lambda functions have their own data type. The lambda list, when evaluated, returns a special 'function' data type with its free variables bound to the current environment and forming a lexical closure.
In newLISP, binding of variables in lambda expressions takes place during application of the lambda expression. In other LISPs, lambda expressions are used to realize static variables and content via lexical closures.
Static variables are a precondition for a programming language to allow object-oriented programming in that language. In newLISP, lexically closed contexts (namespaces) are used for prototype-based object-oriented programming. Like lambda expressions, contexts in newLISP are first-class objects, which can be created or destroyed during runtime, passed as parameters, and referred to by symbols.
3. In newLISP and Scheme, variables, primitives, and user-defined functions share the same symbol space. In Common Lisp, function symbols and variable symbols each use a dedicated symbol space. This is why Common LISP function symbols sometimes must be prefixed with the sharp quote #'.
4. In newLISP, all variables are dynamically scoped by default, but newLISP also provides a lexically closed namespace called a context. By defining a function in its own context, static/lexical scoping can be achieved, and variable capture can be avoided. Several functions and data can share a namespace. Common Lisp and Scheme are lexically scoped by default and use lambda expressions as the closure mechanism. Common Lisp also offers 'special variables' for dynamic scoping.
5. In newLISP, there are no dotted pairs, because each newLISP cell contains one object and a pointer to the next object if the cell is part of a list. In Common Lisp and Scheme, the 'cdr' part of the lisp cell can be used to contain another data object, in which case we have a dotted pair. As a result, 'cons' behaves differently in newLISP than in other lisps.
;; Common Lisp and Scheme
(cons 'a 'b) => (a . b) ; a dotted pair
[a | b]
;; newLISP
(cons 'a 'b) => (a b) ; a list
;; LISP cells in newLISP
(+ 2 3 (* 4 3))
[ ]
[+] -> [2] -> [3] -> [ ]
[*] -> [4] -> [3]
6. In newLISP, all arguments to a user-defined function are optional. Unassigned argument variables will assume the value 'nil' inside their function.
7. Logically, there are no unbound or non-existing symbols in newLISP. Any unbound or non-existing symbol is bound to 'nil' in the current namespace when it is first used.
8. 'nil' and 'true' are Boolean constants in newLISP. In Common Lisp, 'nil' has an additional role as a list terminator:
;; Common Lisp
(cons 'x nil) => (x)
;; newLISP
(cons 'x nil) => (x nil)
Scheme has the two Boolean constants '#t' and '#f' for true and false.
9. In newLISP, every object is only referenced once (ORO), with the exception of symbols, contexts, and context objects. newLISP's ORO rule allows automatic, stack-based, 'on the go' memory management, without the traditional garbage collection algorithms used in Common Lisp and Scheme. newLISP passes parameters by value and stores intermediate results on a 'result stack' where their memory gets recycled after function return. Only under error conditions will newLISP employ a classic mark-and-sweep algorithm to reclaim unreferenced memory. newLISP's automatic memory management is fully transparent to the programmer but faster and lighter on resource requirements when compared to classic garbage collection algorithms. Because ORO memory management is synchronous, newLISP code has constant and repeatable execution time.
The combination of newLISP's value-passing style and unique memory management make it the fastest interactive (non-compiled) LISP available and one of the fastest scripting languages available in general.
10. Macro evaluation is a single step in newLISP and two steps in Common Lisp. In other LISPs, the macro first expands symbols in its body to their contents and then evaluates the resulting function. In newLISP, macros work like normal functions, but without evaluating their arguments. In this way, functions can be built that look like built-in newLISP primitives. In newLISP, the built-in function 'expand' is used to initiate variable expansion in macros:
(define (raise-to power)
(expand (fn (base) (pow base power)) 'power))
(define square (raise-to 2))
(define cube (raise-to 3))
(square 5) => 25
(cube 5) => 125
Separating evaluation and expansion of parameters offers better control than is available from Common Lisp and Scheme.
11. newLISP only requires the equal sign '=' to test for equality. This is a consequence of newLISP's value-passing style. Common Lisp requires 'eq', 'eql', 'equal', 'equalp', '=', 'string=', 'string-equal', 'char=', and 'char-eq', for equality tests of expressions, data types, identical objects, and referenced objects.
12. newLISP features implicit indexing. This is a logical extension of LISP evaluation rules overloading lists or numbers with the indexing functionality available from built-in list and string functions like nth, rest or slice, i.e.:
(set 'myList '(a b c (d e) f g))
; using nth
(nth 2 myList) => c
(nth 3 1 myList) => e
(nth -3 0 myList) => d
; using implicit indexing
(myList 2) => c
(myList 3 1) => e
(myList -3 0) => d
; implicit rest, slice
(1 myList) => (b c (d e) f g)
(-3 myList) => ((d e) f g)
(1 2 myList) => (b c)
Using implicit indexing is optional but in many instances speeds up coding and increases readability.
#14
newLISP newS / Postscript production in newLISP
July 14, 2006, 12:30:59 AMFor anyone who's interested in more postscript stuff, the
The so-called
Once you do, you'll see what I'm seeing, which is a fantastic shortcut to producing postscript graphics, courtesy of Lutz. This might be a good time to turn to the book
I'm actually interested in producing print-ready documents using newLISP, creating something like a formatting language processor in newLISP for SXML
-Noah
#15
Anything else we might add? / Suggested Edits to the Homepage
July 13, 2006, 11:11:53 AMHere's another suggested edit:
Code Select
- Tcl/Tk Graphical Frontend
+ Tcl/TK graphical front-end
-Noah