Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - nallen05

Pages1 2
#1
newLISP in the real world / approximate string matching using trigrams
April 30, 2012, 01:21:31 PM
My friend Ric Szopa wrote a "similarity" function in clojure to compare how similar two strings are using the trigrams approach (find all the sequential 3-letter combos in both words and measure the overlap):


Code Select

# trigrams in clojure (author: Ric Szopa)
(defn- trigrams [str]
 (let [n 3
       padding (repeat (dec n) $)
       cleaned (remove #(Character/isWhitespace %) (concat padding
(clojure.string/lower-case str)))
       freqs (frequencies (partition n 1 cleaned))
       norm (Math/sqrt (reduce + (map #(Math/pow % 2) (vals freqs))))]
   (into {} (for [[k v] freqs] [k (/ v norm)]))))

(defn similarity [left right]
 (if (= left right)
   1.0
   (let [left (trigrams left)
         right (trigrams right)]
     (apply + (for [[trigram value] left] (* value (or (right trigram) 0)))))))


I decided to try writing it in newlisp...


Code Select

(define (ngrams d str, (norm 0) (str-clean (append "  " (lower-case (replace " " str "")))))
  (dotimes (i (- (length str-clean) 2))
(bayes-train (unpack "s3" (+ i (address str-clean))) d))
  (dotree (x d true)
    (inc norm (pow ((context d x) 0))))
  (setq norm (sqrt norm))
  (dotree (x d true)
    (context d x (div ((context d x) 0) norm)))
d)

(define (similarity left right, (accum 0))
  (if (= left right)
 1.0
 (let ((left (ngrams 'Left left))
(right (ngrams 'Right right)))
  (dotree (s left true)
 (inc accum (mul (context Left s) (or (context Right s) 0))))
accum)))

#  (similarity "banana" "bananana") => 0.9722718241


...which turned out to be only a few lines longer.



In order to speed it up I tried the following strategies:

* store data set in a context and iterate over it using dotree (instead of list + dolist)

* i used low-level "unpack" for faster string iteration

* use built-in methods when possible (eg bayes-train)



One consequence of all the dotree's is that I felt like I ended up doing a lot more stateful variable manipulation (eg creating a variable and then calling "inc" on it within the block of the 'dotree') than I would have in other dialects of lisp (where I would have used eg 'reduce').



My question for the forum is: does anyone have any tips or tricks for optimizing newlisp code (using trigrams as an example) or corrections to my approach?



Take care



Nick
#2
Anything else we might add? / Re: newlisp reference in blog post
September 26, 2011, 11:00:00 AM
thanks! :-)



maybe some day
#3
Anything else we might add? / newlisp reference in blog post
September 25, 2011, 07:16:02 PM
it has been relatively quiet around here these days



so here's a link to a blog post I wrote  to make some noise (one of the 3 blog posts I have ever written in my life ;-)



I wrote it a while ago and forgot about it. I just remembered that it exists and that it references newLisp:



http://ginzametrics.com/how-search-affects-language-design.html">//http://ginzametrics.com/how-search-affects-language-design.html



Take care



Nick
#4
newLISP and the O.S. / Re: newLISP 'really' on the iPhone...?
May 30, 2011, 10:44:01 AM
Quote from: "HPW"There is also a sheme for the ipad:



http://web.mac.com/Jay_Reynolds_Freeman/My/Pixie_Scheme_III.html">http://web.mac.com/Jay_Reynolds_Freeman ... e_III.html">http://web.mac.com/Jay_Reynolds_Freeman/My/Pixie_Scheme_III.html



http://itunes.apple.com/us/app/pixie-scheme-iii/id401023057?mt=8">http://itunes.apple.com/us/app/pixie-sc ... 23057?mt=8">http://itunes.apple.com/us/app/pixie-scheme-iii/id401023057?mt=8


FWIW Jay Reynolds Freeman (the author of Pixie Scheme III for the iPad, referenced above, as well as Wraith Scheme, a Scheme implementation for OS X) is going to present on "Implementing a Scheme Interpreter from Scratch" at the Googleplex in Mountain View on Friday night:



http://www.meetup.com/balisp/events/20214831/">http://www.meetup.com/balisp/events/20214831/



If anyone is near the San Francisco Bay Area this weekend you are warmly invited to attend.



If someone would like to make a presentation about newLISP, we would be very happy to organize a presentation space for this event somewhere in the bay area.



Take care



Nick
#5
newLISP in the real world / Re: Lisp + observation... (unless)
May 22, 2011, 04:31:01 PM
hello kanen!



so your question is how to implement a varient of the UNTIL macro in Common Lisp that saves:

1. the return value of the test expression to a variable, and

2. the iteration count to a variable



note that in your example:


Code Select

(until (= (set 'x (rand 10)) 0)
  (println  $idx ":" x " not 0"))


you are NOT saving the return value of the test, you are saving the value of "X", which is used inside the test form



sticking with your example, the easiest way to impliment it with the built-in functionality of common lisp is to use the LOOP macro:


Code Select

(loop for x = (random 10)
  counting x into i
  do (format t "~a: ~a not 0~%" (1- i) x)
  until (= x 0))


but a lot of people don't like to use the extended LOOP form, since it's not pure s-expression language



believe it or not the DO macro is considered a nicer way to do it:


Code Select

(do ((x #1=(random 10) #1#)
 (i 0 (1+ i)))
((= x 0))
  (format t "~a: ~a not 0~%" i x))


Kazmir's example is an implementation of the common UNTIL macro. This specific implementation saves the iteration count in the variable $IDX:


Code Select

(defmacro until (test &body body)
  (let ((start-tag (gensym "START"))
        (end-tag   (gensym "END")))  
    `(progn (setf $idx 0)
            (tagbody ,start-tag
                    (when ,test (go ,end-tag))
                    (progn ,@body)
                    (incf $idx)
                    (go ,start-tag)
                    ,end-tag))))


in use:


Code Select

(until(= (setf x (random 10)) 0)
      (format t "~d: ~d not 0 ~c" $idx x #Newline))


but this macro will fail if there is a nested UNTIL, because it will mess up the value of the outer $IDX. It can be modified to address this issue:


Code Select

(defmacro until (test &body body)
  (let ((start-tag (gensym "START"))
        (end-tag   (gensym "END")))
    `(let (($idx 0))                        ;
       (tagbody ,start-tag
                (when ,test (go ,end-tag))
                (progn ,@body)
                (incf $idx)
                (go ,start-tag)
                ,end-tag))))


this macro could be done a little more simply by not using tagbody,

which is a considered a low-level-only-use-it-when-you-really-need-it special form,

and instead using a more simple, pre-baked iteration/control-flow macro (in this example LOOP

is considered a simple, pre-baked iteration/control-flow macro because there are no special keywords used--in this usage all the macro does is repeately execute it's arguments over and over and over again):


Code Select

(defmacro until (test &body body)
  `(let (($idx 0))
      (loop (if ,test                              ;                  
                   (return)                        ;
                   (progn ,@body (incf $idx)))))) ;


so... none of these macros actually implement what you propose... they don't save the value of the test result. we can modify our UNTIL macro to do this:


Code Select

(defmacro auntil (test &body body)
  `(let ($it                                  ;
           ($idx 0))
     (loop (if (setf $it ,test)                ;
  (return)
  (progn,@body (incf $idx))))))


Note that I named this macro AUNTIL, which is a common, but optional, convention for "anaphoric" macros:



http://www.bookshelf.jp//texi/onlisp/onlisp_15.html#SEC99">//http://www.bookshelf.jp//texi/onlisp/onlisp_15.html#SEC99



remember xetroxon's recent post about "The Definition of Simple is a Complex Thing"?



http://newlispfanclub.alh.net/forum/viewtopic.php?f=5&t=3863">//http://newlispfanclub.alh.net/forum/viewtopic.php?f=5&t=3863



In it he referenced Cal Sassenrath's definition of simple:


Quote
Simple is:

1. Clear abstraction: smart, well-drawn layers of "knowledge focus" that allow hiding of details.

2. Clean expression: meaningful, concise but not cryptic, representation and communication of concepts.


this definition is a good guide for us to use to reimplement AUNTIL in a more "lispy" way:


Code Select

(defmacro until (test &body body)
`(loop (if ,test
  (return)
  (progn ,@body))))

(defmacro auntil (test &body body)
  `(let ($it
($idx 0))
     (until (setf $it ,test)
       ,@body
  (incf $idx))))


we broke our macro up into two well-drawn layers: an "until" macro that implements the control flow, and a more abstract AUNTIL that adds the anaphoric properties. This gives us Simple #1 (clearer abstracion). and as a corollary, our DEFMACRO forms become simpler, giving us Simple #2 (clean expression).



One cool thing about lisp's uniform semi-concise syntax (Lisp in the canonical sense, so including common lisp & newlisp) is that you can build the language up to be able to represent any part of your program you want semi-concisely in this uniform syntax:


Code Select

(defmacro while (test &body body)
  `(loop (if ,test
      (progn ,@body)
(return))))

(defmacro awhile (test &body body)
  `(let ($it
         ($idx 0))
     (while (setf $it ,test)
,@body
(incf $idx))))

(defun not-zero (n)
  (unless (= n 0)
n))

(defmacro until-0 (form &body body)
  `(awhile (not-zero ,form)
,@body))

   

then your example becomes:


Code Select

(until-0 (random 10)
  (format t "~a: ~a not 0~%" $idx $it))


0: 7 not 0

1: 5 not 0

2: 4 not 0

3: 9 not 0

4: 6 not 0

5: 6 not 0

6: 8 not 0

7: 8 not 0

8: 5 not 0

9: 1 not 0

10: 2 not 0

11: 8 not 0

NIL



in newLisp you could also build up the language to include until-0:


Code Select

(define-macro (until-0)
  (letex (zero-form (args 0)
   body      (cons 'begin (1 (args))))
   (let ($num nil)
(until (= 0 (setq $num zero-form))
body))))


and then your example becomes more simple:


Code Select

(until-0 (rand 10)
  (println  $idx ":" $num " not 0"))




0:2 not 0

1:9 not 0

2:9 not 0

3:8 not 0

4:2 not 0

5:5 not 0

6:3 not 0

7:7 not 0

8:5 not 0

9:6 not 0

10:5 not 0



Anyway, hope this helps



Take care



Nick
#6
Anything else we might add? / More newLISP beer!
May 09, 2011, 08:39:15 PM
if anyone wants to grab a newlisp beer in Germany, Poland, Austria, or Croatia in June/July let me know!



otherwise it will have to be when you come to berkeley ;-)



take care



nick
#7
newLISP newS / Re: What the FOOP?
April 24, 2011, 02:57:03 AM
Great video m i c h a e l !



Just watched it



Thanks!
#8
Anything else we might add? / Re: newLISP beer: ILC2010 in Reno, NV / NYC
October 06, 2010, 10:43:15 AM
QuoteI confess I started thinking about a crowded bar of Common Lispers, and the arrival of a Stranger (played by Clint Eastwood, possibly) who orders a beer and stands at the other end of the bar, wearing a worn newLISP T-shirt under his poncho.



"Hey, Stranger," says one of the drinkers, " if yer looking fer trouble, you've come to the right place. We don't like them f-exprs round these parts"...



then it got a bit confused...


Hehehehe.......



seriously, though, I don't think it's like this... if elisp hackers are welcome at lisp meetups I don't see why newLISPERs wouldn't be... I for one have enjoyed both... more than anything, I think one of the drinkers would ask "hey what's a noo-lisp?" then they would both geek out over the ability to use lisp to do useful things (only later get into a nerd fistfight over dynamic vs lexical scope ;-)
#9
Anything else we might add? / newLISP beer: ILC2010 in Reno, NV / NYC
October 05, 2010, 02:23:58 AM
Any newLISPers going to be at ILC2010 in Reno, NV this month? If so do you want to plan a newLISP beer one night at the conference?



Also, any newLISPers in NYC? If so how about a newLISP beer somewhere around Manhattan the last weekend of October 2010?



Take care



Nick
#10
newLISP in the real world / Re: interacting with a "busy" newlisp server with another prgrm
January 15, 2010, 11:18:28 AM
Thanks for your reply Lutz



The posts above give me good ideas on how to approach the problem. It looks like: If I want to "stop the world" and critically debug the server I can telnet in (since it seems to block other programs from accessing). But if I just want to get some data from the server here and there I can go with the NET-EVAL approach



I guess my inspiration for this question was the concurrency issue. If I have a server that is expecting to respond to many frequent queries from client program #1, and I want to "inspect" it from client program #2, is it safe to make a request to the IP and port the server is listening to at any time, relying on the mechanisms in place to prevent simultaneous requests from the two programs from interfering with each other?
#11
newLISP in the real world / interacting with a "busy" newlisp server with another prgrm
January 15, 2010, 12:17:42 AM
So if I start a newlisp server


Code Select
newlisp -c -d 4711

and I connect to it with client program #1 and it does stuff (like responding to a request for information)


Code Select
echo "(begin (sleep 10000) 'return-stuff)" | nc 127.0.0.1 4711

and then I suspect something is wrong inside the newlisp server because the output or behavior of client program #1 doesn't seem right, I can send code to the place the newlisp server is listening to (assuming I know it) from client program #2 (like emacs) to debug it


Code Select
echo '(symbols)' | nc 127.0.0.1 4711

when I try (on my mac, OS X 10.5.8) this client program #2 seems to correctly block until client program #1 gets its answer (assuming #1 exits) then makes its request



My question: is there a better way i should be doing this? (motivation: I want to be able to "debug" a running newlisp server in the least hacky way posible)
#12
newLISP in the real world / Re: Binding a value to multiple values
January 11, 2010, 11:37:44 PM
an easier hack in this case might be to just to use the symbol itself as a container and evaluate it upon access


Quote
 (set 'outer1 '() 'outer2 '())

()

> (set 'inner '())

()

> (push 'inner outer1) ; quoted inner

(inner)

> (push 'inner outer2) ; quoted inner

(inner)

> (push 1 inner)

(1)

> (map eval outer1) ; evaluating elements of outer1

((1))

> (map eval outer2) ; evaluating elements of outer2

((1))
#13
Anything else we might add? / dang too late
July 15, 2009, 03:17:26 PM
polls closed :-(
#14
newLISP newS / Still going to support newLISP Manual?
June 27, 2009, 11:33:23 AM
Lutz



Are you still going to support the "Manual and Reference" document distributed with newLISP? In its current format (strict HTML with or without index frame) it is probably the nicest lisp manuals I have ever seen...



Nick
#15
newLISP newS /
June 25, 2009, 09:47:18 AM
Quote from: "Cormullion"we still don't have a good way of doing this distributed proof-reading. I think more people would contribute if the mechanics were easier. Something like WikiBooks might be something to consider one day...


Is the manual /code patterns HTML generated from another text format? My humble suggestion is to check the docs into github in their most basic form. Then if someone sees something that needs to be changed they just click "fork", make the change (which can be done through the github web app, without ever downloading the doc to your computer), then sending the patch to Lutz who can choose to accept it or not...



Nick
Pages1 2