Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Topics - fdb

Pages1
#1
newLISP in the real world / Get rid of (some of) the parentheses with sublisp!
October 27, 2020, 03:24:55 PM
I really like (new)lisp and even the parentheses but they can get kind of annoying, especially at the end of function so in a flash of insight I got the idea to replace  the right parentheses with a subscripted count if there is more then one, see below original code:
Code Select

(define (initial-clauses str)
  (set 'start (array rows columns (map int (explode str))))
  (for (row 0 (sub1 rows))
    (for (col 0 (sub1 columns))
      (when (> (start row col) 0)
        (push (string (to-var row col (sub1 (start row col))) " 0")
               problem)))))

And below if run through sublisp:
Code Select

(define (initial-clauses str)
  (set 'start (array rows columns (map int (explode str)₄
  (for (row 0 (sub1 rows)₂
    (for (col 0 (sub1 columns)₂
      (when (> (start row col) 0)
        (push (string (to-var row col (sub1 (start row col)₃ " 0")
               problem)₅

And the code run through itself
Code Select

(define (sublisp txt)
  (set 'offset 0x2080)
  (for (x 9 2 -1)
    (replace (dup ")" x)
             txt
             (string ")" (char (+ offset x)₅
   (print (string txt)₂
   nil)


Now if someone hacks into newlisp to change this back before it gets translated you could even execute it!  ;-)
#2
So, what can you actually DO with newLISP? / Newlisp bindings for DuckDB
August 05, 2020, 03:30:02 PM
Hi,



I've added newLISP bindings for DuckDB at https://github.com/luxint/duckdb">//https://github.com/luxint/duckdb



So what is DuckDB? Have a look at their website: https://duckdb.org">//https://duckdb.org

They state that :DuckDB is an embeddable SQL OLAP database management system



In practice it is almost like SQLITE (only much faster!) so the  API I made is almost the same as the sqlite3.lsp standard module. Only exception is that BLOB's are not supported (yet) (They are in DuckDB but IMHO of little use for the expected things you would do with it).



What also doesn't work (yet) is prepared SQL statements/parameter bindings so no protection against SQL injection attacks however since this will mostly be used as a single person DB this is probably not a big issue but maybe I'll add it later, the DuckDB dynamic library supports it.



What is different from the  SQLITE module is that there is a  *very* fast CSV  import & export command, on my laptop it creates a table and imports 1.5M records within a second!



Furthermore DuckDB is a columnar database (as opposed to SQLITE which is a row based db), which means that is is *much* faster for select statements with lots of aggregations, windows functions etc. What I was also looking for is a PIVOT statement, (think about MS Excel pivot tables) but this isn't supported as it needs dynamic column creation which is only possible (AFAIK) with some dynamic SQL creation so that is what I've also created on top of this API. I haven't included this in this API but will make a separate module for that.
#3
So, what can you actually DO with newLISP? / Newlisp bindings for Termbox
July 13, 2020, 02:23:22 PM
Hi,



I made Newlisp bindings to the Termbox library. With these bindings you can easily create text-based user interfaces in Newlisp. You can find it at https://github.com/luxint/termbox">//https://github.com/luxint/termbox%20 Let me know what you think or if you have any questions!
#4
newLISP newS / newLISP language support for Visual Studio Code
June 01, 2019, 01:53:03 PM
I've made a GitHub repository with language support for newLISP in Visual Studio code: https://github.com/luxint/vscode-newlisp">//https://github.com/luxint/vscode-newlisp.



If you don't now VS Code have a look at https://code.visualstudio.com">//https://code.visualstudio.com,  a very nice free editor, built on open source,  from Microsoft, runs on Windows, Mac and Linux.



Current state of the extension is it does newLISP syntax highlighting, launching a newLISP REPL and evaluating the active file in the REPL. What I want to add is several color theme's and snippets (to show function syntax and explanations when 'hovering').



This is how it looks with theme Monokai: https://github.com/luxint/vscode-newlisp/blob/master/images/Screenshot%202019-06-01%20at%2022.18.57.png">//https://github.com/luxint/vscode-newlisp/blob/master/images/Screenshot%202019-06-01%20at%2022.18.57.png



Blue: the 'meta' functions (define, define-macro, macro, fn, lambda, lambda-macro)

Red: mutable primitive functions (set, push, ..)

Green: non-mutable primitive functions (list, array, append, ..)

Purple: numbers (integers, floats, bigints, ..)

Yellow: strings

Grey: quoted literals and comments

White: the rest



If you want to collaborate,  need different colors or have ideas how to make this better, let me know!
#5
So, what can you actually DO with newLISP? / Fun with primes (and newLISP)
May 14, 2019, 03:36:05 PM
I think newLISP is a great tool for tinkering with some ideas in your head, AKA 'exploratory computing'. If you like this as well maybe you'll also like the little write up below.



Recently I had to explain 'what is a prime' to my eldest daughter, while explaining 'there are an infinite number of primes' I thought, what about the distance between primes.., maybe there is a certain distribution?



I resisted the easiest way (google it) and went for the best (explore it in newLisp).



Ok , so we need to the primes, how to generate them ? Look in the documentation and I got this:
Code Select

;; primes.lsp - return all primes in a list, up to n

(define (primes n , p)
  (dotimes (e n)
    (if (= (length (factor e)) 1)
      (push e p -1))) p)

Great a fast factor function! So how long this take for all the primes up to 1,000,000 ?
Code Select
> > (time primes 1000000)
937.407
Ok within a second so how much primes are there?
Code Select
> (length (primes 1000000))
78498
So about 80000 primes /second, can we do better? Well an obvious improvement is to step through the odd numbers only and start with 2:
Code Select
(define (primes-to n , (p-list '(2)))
(for (x 3 n 2)
(unless (rest (factor x))
(push x p-list -1)))
p-list
)
I really like my '(unless (rest (factor(.. bit here, so what about the time?
Code Select
> (time (primes-to 1000000))
569.302

Aha! as expected almost twice as fast, now one last improvement before we go to the main topic, of course to generate primes has been known since antiquity so how does the  'Sieve of Eratosthenes' look in newLisp? Here is my version:
Code Select
(define (sieve n)
(set 'arr (array (+ n 1)) 'lst '(2))
(for (x 3 n 2)
(when (not (arr x))
(push x lst -1)
(for (y (* x x) n (* 2 x) (> y n))
(setf (arr y) true))))
lst
)
But is it faster then using the built in factor function?
Code Select
> (time (sieve 1000000))
259.257
Not bad for antiquity, we'll be using this function from now on.So what we want to do now is calculate the distance between primes. After some tinkering I came up with this:
Code Select
(define (funlist lst func rev)
(if rev
(map func (chop lst) (rest lst))
(map func (rest lst) (chop lst))))
So a generic function which we can use to calculate the differences between consecutive list members:
Code Select
> (funlist (sieve 1000) -)
(1 2 2 4 2 4 2 4 6 2 6 4 2 4 6 6 2 6 4 2 6 4 6 8 4 2 4 2 4 14 4 6 2 10 2 6 6 4 6
 6 2 10 2 4 2 12 12 4 2 4 6 2 10 6 6 6 2 6 4 2 10 14 4 2 4 14 6 10 2 4 6 8 6 6 4
 6 8 4 8 10 2 10 2 6 4 6 8 4 2 4 12 8 4 8 4 6 12 2 18 6 10 6 6 2 6 10 6 6 2 6 6 4
 2 12 10 2 4 6 6 2 12 4 6 8 10 8 10 8 6 6 4 8 6 4 8 4 14 10 12 2 10 2 4 2 10 14 4
 2 4 14 4 2 4 20 4 8 10 8 4 6 6 14 4 6 6 8 6)
Ok great no we want to count all the differences, we'll use this function:
Code Select
(define (freq lst)
(let (ulist (unique (sort lst)))
(map list ulist (count ulist lst))))
It is so nice to have all this functions in newLISP like count!

So applying this we get:
Code Select
> (freq (funlist (sieve 1000) -))
((1 1) (2 35) (4 40) (6 44) (8 15) (10 16) (12 7) (14 7) (18 1) (20 1))
Hmm for all the primes below 1000, 6 is the most frequent difference,i didn't expect that, how about all the primes below 1,000,000 ? We probably need to sort the list to get the highest first, so lets make a function to use in the sort:
Code Select
(define (comp x y)
    (>= (last x) (last y)))
And use it:
Code Select
(sort(freq(funlist (sieve 1000000) -)) comp)
((6 13549) (2 8169) (4 8143) (12 8005) (10 7079) (8 5569) (18 4909) (14 4233) (16
  2881)
 (24 2682)
 (20 2401)
 (22 2172)
 (30 1914)
 (28 1234)
 (26 1175)
 (36 767)
 (34 557)
 (32 550)

Definitely looks like number 6, does this take forever? Have a look at https://en.wikipedia.org/wiki/Prime_gap">//https://en.wikipedia.org/wiki/Prime_gap, hope you enjoyed this little  exercise!
#6
So, what can you actually DO with newLISP? / Neural network in Newlisp
December 21, 2017, 04:21:02 PM
Interested in playing with neural networks and don't want to use or learn python?  Now you can also use your favorite programming language!



I've converted the python program written by  Michael Nielsen from his his excellent on-line book at http://neuralnetworksanddeeplearning.com">//http://neuralnetworksanddeeplearning.com to newLisp. Presented some challenges as I'm not really well versed in Python or the library used (numpy) and I needed to brush up my knowledge on matrices as well to get it all working reasonably fast. (thanks Lutz for the fast matrix functions!).



With this version you can play with stochastic gradient descent for a feedforward neural network.I've included load and convert functions for the MNIST dataset, which is kind of a standard to test learning algorithms against. Download it from http://yann.lecun.com/exdb/mnist/">//http://yann.lecun.com/exdb/mnist/ and with below program you're good to go.



Let me know if you encounter any bugs (most probably!) and Lutz let me know if and how this can can be included as a module.



Ferry


Code Select

;; Mini-batch stochastic gradient descent learning algorithm for a feedforward neural network in newLISP. Based on a python program and book by Michael Nielsen, see http://neuralnetworksanddeeplearning.com/about.html and below for the MIT license.

;Usage (nn:sgd) with arbitrary number and sequence of named parameters .
; For instance (nn:sgd eta 1 network '(784 100 10) lmbda 1), other parameters will use default values, see below.

;; network         contains the number of neurons in the respective layers of the network.  For example, if the list was [2, 3, 1] then it would be a three-layer network, with the first layer containing 2 neurons, the second layer 3 neurons, and the third layer 1 neuron. For the included MNIST data and load functions the first (input) layer has to contain 784 neurons and the last (ouptput) layer 10. There can be 0-n  number of (hiden) layers and neurons in  between.

;; epochs          The number of times the whole dataset is traversed
;; mini-batch-size the numnber of instances trained together
;; eta             the learning rate, positive number, try 0.1 or 1 or 10 and then tweak
;; number          the number of instances to train upon (max 60000 for number+test data together for the MNIST training data
;; test-data       the number of instances to test against, used for accuracy calculation
;; w-init          weight initiliazer function, choice between default-w and large-w
;; cost            cost function, choice between quadratic and cross-entropy
;; lmbda           used for regularization to prevent overfitting, must be greater or equal to zero.
;; i-path          path to the input file, for MNIST the name of the file is train-images.idx3-ubyte,
;; l-path          path to output file For MNIST the name of the file is train-labels.idx1-ubyte, download from http://yann.lecun.com/exdb/mnist/


(context 'nn)

(define-macro (sgd)
(set
'network '(784 20 10)
'epochs 10
'mini-batch-size 10
'eta 4
'number 10000
'test-data 100
'w-init default-w
'cost cross-entropy
'lmbda 0
'i-path "Downloads/train-images.idx3-ubyte"
'l-path "Downloads/train-labels.idx1-ubyte"
)
(when (args) (bind (explode (args) 2) true))
(when (< (length input) number)
(load-mnist-train-images (+ number test-data) i-path)
(load-mnist-train-labels (+ number test-data) l-path))
(set 'biases (map (fn(x) (array x 1 (normal 0 1 x))) (rest network)))
(set 'weights (map w-init (rest network) (chop network)))
(dotimes (j epochs)
(setq start (time-of-day))
(setq mini-batches (explode (randomize (sequence 0 number)) mini-batch-size))
(dolist (mb mini-batches)
(update-mini-batch mb))
(if test-data
(silent (fork (println "Epoch:"(+ j 1)
" Duration:"(/ (sub (time-of-day) start) 1000)
" Accuracy:" (format "%2.2f%%" (mul 100 (evaluate number (+ number (- test-data 1))))))))
(println "Epoch: " j " complete"))))

(define (update-mini-batch mini-batch)
  (set 'batchsize (length mini-batch))
  (backprop mini-batch)
(set 'biases (map (fn(b nb) (mat - b (mat * nb (div eta batchsize))))
biases
(map (fn(x) (multiply x (array batchsize 1 '(1)))) nabla-b)))
(set 'weights (map (fn(w nw) (mat - (mat * w (sub 1 (mul eta (div lmbda number)))) (mat * nw (div eta batchsize))))
weights
nabla-w)))

(define (backprop lst)
(set 'activations (list (transpose (map(fn (x) (flat (input x)))lst)))
'b-biases (map (fn(x) (multiply x (array 1 batchsize '(1)))) biases)
'b-weights weights
'nabla-b '() 'nabla-w '() 'zs '())
(dolist (bw (map list b-biases b-weights))
(push (mat + (multiply (bw 1) (activations -1)) (bw 0)) zs -1)
(push (sigmoid (zs -1)) activations -1))
(set 'delta (cost (activations -1) (transpose (map (fn(x) (flat (output x)))lst)) (zs -1)))
(push delta nabla-b)
(push (multiply delta (transpose (activations -2))) nabla-w)
(setq x -2)
(while (> x (sub (length network)))
(set 'delta (mat * (multiply (transpose (b-weights (+ x 1))) delta) (sigmoid-prime (zs x))))
(push delta nabla-b)
(push (multiply delta (transpose (activations (sub x 1)))) nabla-w)
(dec x)))

;; weight initializers
(define (default-w x y)
  (array x y (map (fn(z) (div z (pow x 0.5))) (normal 0 1 (* x y)))))

(define (large-w x y)
  (array x y (normal 0 1 (* x y))))

;; cost functions

(define (quadratic a y z)
(mat * (mat - a y) (sigmoid-prime z)))


(define (cross-entropy a y z)
(mat - a y))


;; utility functions

(define (ff f)
(let(a (list (input f)))
(argmax (flat(nth '(-1 -1)(map (fn (b w) (push (sigmoid (mat + (multiply w (a -1)) b)) a -1)) biases weights))))))

(define (ev nr)
(if (= (ff nr) (argmax(flat(output nr))))
1 0))

(define (evaluate start end)
(let (teller 0)
(for (x start end)
(inc teller (ev x)))
(div teller (+(- end start)1))))

(define (load-mnist-data var file number block offset)
  (let (buff (read-file file)
parse-bytes (dup "b " block)
lst '())
(for (x offset (+ (- offset 1) (* number block)) block)
(push (explode (unpack parse-bytes (slice buff x block))) lst -1))
(set var lst))true)

(define (load-mnist-train-images number i-path)
(load-mnist-data (sym 'input) i-path number 784 16)
(set 'input (map m-normalize (eval 'input)))true)

(define (load-mnist-train-labels number l-path)
(load-mnist-data (sym 'output) l-path number 1 8)
(set 'output (convert (eval 'output)))true)

(define (convert labels)
(map (fn (x) (setq t (dup 0 10)) (setq (t (x 0)) 1)(explode t)) labels))

(define (m-map function matrix)
(map (fn(x) (map function x)) matrix))

(define (sigmoid-n z)
(div 1 (add 1 (exp (sub  z)))))

(define (sigmoid-prime-n z)
(let (sig (sigmoid-n z))
(mul sig (sub 1 sig))))

(define (sigmoid m)
(m-map sigmoid-n m))

(define (sigmoid-prime m)
(m-map sigmoid-prime-n m))

(define (argmax lst)
(find (apply max lst) lst))

(define (shape lst)
(list (length lst)  (length (lst 0))))

(define (normalize n (qty 255))
(div n qty))

(define (m-normalize m )
(m-map normalize m))

(define (m-log m)
(m-map log m))

(define (m-log-1 m)
(m-map log-1 m))

(define (log-1 n)
(log (sub 1 n)))


(context 'MAIN)

; NewLIps version based on Python program wriiten by Michael Nielsen, see below license of Python program.


;MIT License

;Copyright (c) 2012-2015 Michael Nielsen

;Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

;The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

;THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#7
newLISP and the O.S. / Calling OBJC runtime from newlisp
August 31, 2014, 03:40:49 AM
I'm trying to call the objc runtime from new lisp (from mac osx10.9) but i've got a problem:
Code Select

> (set 'get-class (import "libobjc.dylib" "objc_getClass" "void*" "char*"))
objc_getClass@7FFF9391BF70
> (get-class "NSString")
0

What am i doing wrong here? I was expecting a pointer to the NSString class. The definition of the c function is defined in Apple's Objective-C Runtime Reference:https://developer.apple.com/library/mac/documentation/Cocoa/Reference/ObjCRuntimeRef/Reference/reference.html#//apple_ref/c/func/objc_getClass">//https://developer.apple.com/library/mac/documentation/Cocoa/Reference/ObjCRuntimeRef/Reference/reference.html#//apple_ref/c/func/objc_getClass
Quote
objc_getClass

Returns the class definition of a specified class.



id objc_getClass(const char *name)

Parameters

name

The name of the class to look up.

Return Value

The Class object for the named class, or nil if the class is not registered with the Objective-C runtime.


I got the idea from this blog post http://sjhannah.com/blog/?p=219">//http://sjhannah.com/blog/?p=219 which calls the objective-C runtime from Java by using the c-funtions:

objc_msgSend(), to pass messages to objects,

objc_getClass(), to get a pointer to a class and

selector_getUid(), to get the pointer to a selector.
#8
newLISP in the real world / strange result with map and apply
April 26, 2014, 02:57:34 PM
Code Select

> (map (apply +) '((0 1 2) (3 4 5)))
((0 1 2) (3 4 5))
> (map (fn(x) (apply + x)) '((0 1 2) (3 4 5)))
(3 12)


Why doesn't the first example work?
#9
newLISP Graphics & Sound / How to delete rows,columns and table's?
December 16, 2013, 02:00:07 PM
How can i delete rows and/or columns in a gs:table? there seems to be only the option to add them..or am i missing something?  Furthermore i cannot delete a gs:table when i try to do so with gs:dispose i get the message

dispose cannot be applied to MAIN:row-table
Pages1