newLISP Fan Club

Forum => newLISP in the real world => Topic started by: newdep on December 09, 2006, 05:48:23 PM

Title: Decode me! (Part #2 - Winter Session)
Post by: newdep on December 09, 2006, 05:48:23 PM
Part #2 of "Decode me"... Just to keep your brains warm during the winter...(if that winter is coming at all....)



To make it easier (because cracking this in not that easy without it)

I'll pass you the crypt code -> "norman"



...oh yes...the source code is listed below (which includes also the decoder :-)



Enjoy... Norman.









;; cerokq tur Vwxqnres cusg

(fskc aycvr (dognhv (yac pvrd (srdivzcr 97 122))))

(qckumrf (l 26) (ggsu (eckmtr nzgta -1) iywjf -1))



;; rrgiiz tur qfdrrfdfzdvau cqtgrfj rrbz l rzd l pcuunt vbuqxrf

(rvriar (fvfuea-seooqr l p)

 (hlvfh (wunq k (tzdsg izzet)) (svbu k (fvegk hlvfh))))



;; iqtheb kte pbfiqscbbuunt yskfeef tiam l nbu j cbqwes iaqsoqs

;; guwj dohgweq nrrrj m scrsugp!

(qrtzze (erhldn-qrqfpe k l)

 (gvfq i (swep x (svfjf vyvgk)))

  (poyvgk (i vyvgk)

   (uf (= l (j j)) (jqtd m ( (tzdsg izzet) (svbu i vyvgk) )))) l )



;; 'tehs zr a yrhkqr vf zfieepojq

(drsweq (lbjsi-oafr? l) (rzd (>= (puoi j) 97) (<= (cunf o) 122)))



;; qnpbrv

(pesvbv (qnpbrv)

 (pogvave (x (yrbxfh grlk))

  (uf (ybkvd-cnfs? (kqxg k))

        (pvsia

          (cijt (rrgiiz-eapcuq (trkh o) (ooqr 0)) seooqrr -1)

          (iatngs tadr -1))

          (cijt (trkh o) qnpbrvp -1)  )))



;; drpcuq

(drsweq (drpcuq)

 (dbgwdqs (k (ysestu gsof))

  (is (ycnqr-pngv? (fekg l))

        (sqgva

         (dleh (erhldn-qrqfpe (pbrv 0) (fekg l)) uqcbqsu -1)

         (dognhv ooqr -1))

         (dleh (grlk j) drpcuqd -1)  )))



;; nfy war frqiqt pbrv

(brvah "Yfgr Cbqs: ")

(jqtd pcuq (rrnr-cunr))



;; ngb roe vbggt svzv

(brvah "Oiugvaoc riyr: ")

(gvfq vatzxe (erou-xiar))



;; fvmd gus fditvbrx fvys

(jqtd gsof (ekczfpe (erou-riyr weriyr)))



;; ojw fbe oe qnpbrv ar qrqfpe

(cewef "(E)npbrv? ar (D)rpcuq?")

(is (= "r" (zfiee-pojq (cunf (iqaq-xsp))))

        (netvb (vzcbqs) (ndigr-tzxe (ncdvzd vatzxe ".raq") (aaia rbtadrq "")))

        (pvsia (qstadr) (jfzfe-svzv (mpcrbu unsvzv ".pep") (wczz drpcuqd ""))) )
Title:
Post by: Sleeper on December 10, 2006, 05:40:36 AM

;; create the Vigenere list
(setq alpha (rotate (map char (sequence 97 122))))
(dotimes (x 26) (push (rotate alpha -1) vlist -1))

;; return the corresponding letters from x and y coding indexes
(define (return-encode x y)
(vlist (find x (first vlist)) (find y (first vlist))))

;; return the corresponding letters from y and x coding indexes
;; this routine needs a speedup!
(define (return-decode x y)
(setq v (find x (first vlist)))
(dolist (w vlist)
(if (= y (w v)) (setq z ( (first vlist) (find w vlist) )))) z )

;; 'true if a letter is lowercase
(define (lower-case? x) (and (>= (char x) 97) (<= (char x) 122)))

;; encode
(define (encode)
(dotimes (x (length text))
(if (lower-case? (text x))
(begin
(push (return-encode (text x) (code 0)) encoded -1)
(rotate code -1))
(push (text x) encoded -1) )))

;; decode
(define (decode)
(dotimes (x (length text))
(if (lower-case? (text x))
(begin
(push (return-decode (code 0) (text x)) decoded -1)
(rotate code -1))
(push (text x) decoded -1) )))

;; ask for secret code
(print "Your Code: ")
(setq code (read-line))

;; ask for input file
(print "Original file: ")
(setq infile (read-line))

;; read the original file
(setq text (explode (read-file infile)))

;; ask for an encode or decode
(print "(E)ncode? or (D)ecode?")
(if (= "e" (lower-case (char (read-key))))
(begin (encode) (write-file (append infile ".enc") (join encoded "")))
(begin (decode) (write-file (append infile ".dec") (join decoded ""))) )


it seems it doesn't work on my system.. am i doing something wrong?

anyway that's fun :p
Title:
Post by: newdep on December 10, 2006, 05:50:29 AM
Wooowww...thats quick! Very very very nice, my compliments! ;-)



What brought you to the idea of that decoding type?









The listing above is already old, I have a newer version in the make..

Which is quicker and much smaller too...(ill post it when done)





How do you start the program? I tested it only under linux...

(./newlisp vinegere.lsp is what should work)





Regards, Norman.
Title:
Post by: newdep on December 10, 2006, 05:59:23 AM
does thisone run for you? (run it as script)





;; create the Vigenere list

(setq alpha (rotate (map char (sequence 97 122))))

(dotimes (x (length alpha)) (push (rotate alpha -1) vlist -1))



;; return the corresponding letters from x and y coding indexes

(define (return-encode x y) (vlist (find x (first vlist)) (find y (first vlist))))



;; return the corresponding letters from y and x coding indexes

(define (return-decode x y) (setq a (find x (first vlist))) ((first vlist) (find y (map (lambda (x) (nth a x)) vlist))))



;; true if a letter is lowercase

(define (lower-case? x) (and (>= (char x) 97) (<= (char x) 122)))



;; create coded-list with encoded letter

(define (encode) (push (return-encode (text x) (code 0)) coded -1))



;; create coded-list with decoded letter

(define (decode) (push (return-decode (code 0) (text x)) coded -1))



;; encode or decode or ignore

(define (vigenere)

 (dotimes (x (length text))

  (if (lower-case? (text x))

        (begin

         (if (= input "e") (encode) (decode))

         (rotate code -1))

        (push (text x) coded -1))))



;; ask for secret code

(print "Your Code: ")

(setq code (read-line))



;; ask for input file

(print "Original file: ")

(setq infile (read-line))



;; read the original file

(setq text (explode (read-file infile)))



;; ask for an encode or decode

(print "(E)ncode? or (D)ecode?")

(if (= "e" (setq input (lower-case (char (read-key)))))

        (begin (vigenere) (write-file (append infile ".enc") (join coded "")))

        (begin (vigenere) (write-file (append infile ".dec") (join coded ""))) )



(exit)
Title:
Post by: cormullion on December 10, 2006, 06:37:16 AM
Hi Norman. Very nice work. I like a good puzzle -  You should have told people to not post the answer immediately, but to post a coded reply - I nearly didn't bother when i saw that sleeper had done it immediately (smart man - well done sleeper!)...



But I forced myself not to look and to have a go anyway.... :-)



The clues were:

1: you were asking about 'rotate' recently

2: you gave us the key! :-)

3: (srdivzcr 97 122) was obviously the lower-case alphabet generated by sequence, but substitutions didn't work out

4: the first line contains "Vwxqnres" - which immediately reminded me of Vigenere...(upper-case not encoded...?)!



So I tried to write a quick decoder of a Vigenere text. Unfortunately the results didn't quite work out - getting out of step quite often. So I used some brute force to cycle through the text a few times with varying amounts of 'delay':



(set 'alphabet (map char (sequence 97 122)))
(set 'key (explode (dup "norman" 2000)))

(for (i 0 30)
(set 'counter i)
(dolist (c (explode ciphertext))
(if (find c "abcdefghijklmnopqrstuvwxyz")
(begin
(set 'a-n (find c alphabet))
(set 'k (key counter))
(set 'k-n (find k alphabet))
(set 'k1 (mod (- a-n k-n ) 26))
(print (alphabet k1)))
(begin
(print c)))
(inc 'counter)))


If you look at the resulting text in a newLISP-aware editor, all the keywords stand out:


;; rrcpgz
(qdbked (mfpphc)
(qnpkddf (y (length sbcf))
(if (hqnde-ybgi? (text u))
(begin
(dyrt (eeuxmb-decode (bkfv 0) (frxs z)) gzqbdfg -1)
(rotate obdd -1))
(dyrt (geyw x) qdyqudq -1) )))


So i could then piece together the original from those clues. Not smart, but good enough to foil the plans of Doctor Evil, given half an hour or so...



By the way, if you search google for 'vigenere newlisp' the first hit will probably be a certain blog-entry from ...
Title:
Post by: newdep on December 10, 2006, 09:51:50 AM
Hahahah fantasic creatifity cormullion! I did not expect people to

recognize code at all ..this is great.. also the thought about my posts,

yes your right..there was someting brewing ;-) Bu that your recognized "Vinegere"

from the code was good!! but that has probably someting to do with your Blog ..hahaha (I did not know it was inthere..although I read it then..)



Im always amazed how smart people can be when it comes to de-coding contents..it must be some "old age" instinct that comes in our genes..



I like your code cracker btw ;-) nice....



PS: but i forced now create "Decode me" version #3 because we have some

pretty creative people inhere in the forum...:-)



Regards,

Norman.
Title:
Post by: Sleeper on December 10, 2006, 09:58:36 AM
to solve this i found some new-lisp function names like "map" "char" "sequence" and calculated difference between char-codes.



then i found out that these differences repeat every six characters..





encoded version works fine, but looks like decodes itself with code "annorm" instead of "norman" :)
Title:
Post by: newdep on December 10, 2006, 10:08:39 AM
Realy creative, realy! I must say I did not right away noticed that in the

coded context but I was more focused on encoding ;-) You did it very quick!

(the bigger the key, the more difficult the coding is to crack..)



The second code I posted works here fine, the 'code "norman"

is rotated after the first encoding.. overhere its correct..mmm odd...



..there is a new version on its way, which ill put on my website when its done...





PS: I created this vigenere (which is indeed upper-case as cormullion mentioned) to encrypt base64 listings.. ;-) Just so it stays readable..



Regards, Norman.
Title:
Post by: Sleeper on December 10, 2006, 02:15:45 PM
Quote from: "newdep"
PS: I created this vigenere (which is indeed upper-case as cormullion mentioned) to encrypt base64 listings.. ;-) Just so it stays readable..

that's a good idea!
Title:
Post by: cormullion on December 11, 2006, 02:09:01 AM
hey norman - have you tried a simpler method of encoding:


(set 'alphabet (map char (sequence 97 122)))
(set 'key "norman")
(set 'ciphertext (join text)) ; if it's pre-exploded...
(set 'counter 0)
(dolist (c (explode ciphertext))
  (set 'a-n (find c alphabet))
  (if a-n
      (begin
         (set 'k (key (% counter (length key))))
         (set 'k-n (find k alphabet))
         (set 'k1 (% (- a-n k-n) 26))
         (print (alphabet k1))
         (inc 'counter))
      (print c))
   )


It looks like it might be 10X faster - using modulo arithmetic on the key rather than building the table and rotating lists. Of course the table method is nice because it explains what's happening and is retro 19th century too. ;-)
Title:
Post by: newdep on December 11, 2006, 03:09:47 AM
Nice code ! yes i was in my way of programming using the lists and then

i always stay in lists ;-) But the decode of my function was soooo slow

i moved towards a calculated rotate for decoding instead of a list find..

That was also a big performance boost... and with the tips from Lutz,

using arrays instead of lists, its quiet fast now..

(http://www.nodep.nl/downloads/newlisp/vigenere.lsp)



..the vigenere coding is from the 16th century and cracked in 19th...irrc..

Still a very nice way to use historic material on advanced technology ;-)



Ill see if i can test you code speed here...
Title:
Post by: newdep on December 11, 2006, 03:32:40 AM
Yes your code is quicker! If you remove the explode from the dolist even quicker

and if your move from lists to direct string manipulation even more quicker...



When im programming in newlisp I always have an odd habbit, i always try to

code in an "artistic way" instead of a "logical way".. Its like the language pushes

me in creating nice looking code instead of fast code..



I think i have this was of programming because i hate regex and perl and always

take to short way into "visual liguistic programming" that looks better then

first decoding the code with my mind and then let the computer decode it again..;-)



I believe if you tighten your code a little we have a very fast encode...but how quick is your decode?



Norman.
Title:
Post by: cormullion on December 11, 2006, 04:08:05 AM
Does dolist evaluate explode each time? Doh - I never knew that...



Anyway, I don't really understand the algorithm but I think to decode you just change - to + in :


(set 'k1 (% (- a-n k-n) 26))

- now that's something very lisp-y you can do.



I think there's always a three-way eternal triangle at work: speed, elegance, and ease of writing - for example. It's a shame when you have to sacrifice readability and elegance for speed...but even with newLISP you have to do that sometimes...



A true master can do all three, of course!
Title:
Post by: Lutz on December 11, 2006, 04:59:56 AM
QuoteDoes dolist evaluate explode each time?


'dolist' will evaluate the list expression only once. Nice to see this vignere program evolving into a master piece. Incorporating Corumullion's speedup suggestions it would come close.



Lutz
Title:
Post by: newdep on December 11, 2006, 06:48:57 AM
my latest version (still with rotate) does 35650 ms. where the version of

cormullian does 43957 ms. on a 4.6 MB text file with an amd 64 3200+.

(Note: im testing my version with a read - write and a nth-set on the list

where  cormullians version is doing print to screen..not a good measurement

 between those two but they come very close....)

Still i booked over 100% thusfar..performance speed by ->





* Avoid double nested lambda functions in map, dont use map at all

* Avoid double nested find functions

* Dont use set-nth but use nth-set

* Remove boolean selects from loops

* Dont use lists but use array's instead! (this is  a flexiblity issue on you code)

* Moving to direct text manipulation without lists could be quicker..



I think thought it can just go a few bits quicker in my code but not much anymore...



ps: encoding and decoding the newlisp_manual.html takes here 4 seconds.





Norman.
Title:
Post by: cormullion on December 11, 2006, 07:27:45 AM
Printing to screen - that can be quite slow, especially on my machine - all those translucent windows and Unicode fonts... :-)



To write to file just precede the code with this:


(device (open "myfile" "write"))

and then follow it with


(close (device))

You don't want your code to run too fast - your justification for going to get a cup of coffee while it's running would disappear!
Title:
Post by: newdep on December 11, 2006, 08:58:40 AM
That coffee break is crusial in programming ;-)
Title:
Post by: newdep on December 11, 2006, 01:00:18 PM
Regarindg to speed in newlisp the last few hours where quite intresting

to observe how code can be speeded up, a nice profiler would a nice

to have for newlisp ;-)



Finaly the conclusion is, when it comes to search and replace of big files

there is nothing quicker in newlist then using the straight arithmetic, in this

case modulo was the best solution!



This is changed from vigenere.lsp ->



;; push is slightly quicker then set-nth

;; (% x length code) is 50% quicker then a rotate, wrapping around 26..nice arithmatic flavor..

;; index on a list of chars is just a performance eater for nothing, removed!

;; removed array's again, normal lists are quicker! difference is nihil to 2 seconds.

;; IO read + write to file takes seconds (bus specific)

;; find is quick!



Under the line there is removed a delay of 20 seconds in total.. from my last code..

There is no time to put the water on the fire for cup of coffee  anymore...





PS; the original idea behind the code was indeed to show the working of

vigenere, performance was an issue for me until it killed my program and

i had to rewrite it as above...Le Roi est mort. Vive le Roi!



Norman.
Title:
Post by: cormullion on December 11, 2006, 11:42:49 PM
speed king norman!



There are some timing comparison routines (by Fanda) - now over at Noodles, but perhaps for profiling it would be easier to have something more automatic... but I think it would have to be done at a lower level...



By the way, Lutz's (encrypt) function is pretty cool... I don't know what the method of encryption is though.
Title:
Post by: newdep on December 12, 2006, 03:48:00 AM
Lutz uses XOR one-pad encryption, which is very save if you have

a random long key which is used only once every ;-)
Title:
Post by: newdep on December 12, 2006, 02:11:52 PM
Last post...



Finaly.. this is the whole vigenere -> 3 lines of code ;-)

and 13 seconds on 5 MB text file! encode and decode.



(setq head (map char (sequence 97 122)))



(define (enc) (setq y ((rotate code -1) 0))  (head  (% (S (find x head) (find y head)) 26)))



(define (vigenere) (dolist (x text)  (if (find x head)  (push (enc) dexd -1)   (push x dexd -1)) ))
Title:
Post by: cormullion on December 12, 2006, 03:35:55 PM
it's elegant, fast, and concise enough, now! :-)





But I think you should try harder:



//http://cybertiggyr.com/gene/vig/vigenere.lisp



:-/
Title:
Post by: newdep on December 12, 2006, 03:51:12 PM
you mean regarding speed or options?



regarding speed, I dont want to go quicker ;-)



My very first vigenere also has uppercase and numbers..



Perhpas ill just do the whole bunch from #0 - #127 one day ;-)



Norman.
Title:
Post by: Lutz on December 13, 2006, 05:02:49 AM
Congratulations, this was a long way from 30 minutes to a few seconds. But you can still more than double the speed, if you let your algorithm work on numbers instead of exploded strings.



some small changes:


;; create the lowercase Vigenere list
(setq head (sequence 97 122))

;; ask for secret code
(setq code (map char (explode (rotate (ask "Secret code: ")))))

; reading file
...
(setq text (read-file infile))
(setq text (unpack (dup "c" (length text)) text))
...

(if (= + S) (write-file (append infile ".enc") (join (map char dexd)))
            (write-file (append infile ".dec") (join (map char dexd))))


numbers take less memory then the one-character strings and less time to manipulate.



Lutz
Title:
Post by: newdep on December 13, 2006, 05:21:13 AM
The last Hint is from the Guru ;-)  



Lets see what it brings ;-)
Title:
Post by: newdep on December 13, 2006, 05:38:39 AM
Amazing ;-)



From 13 seconds to 8 seconds ;-)



Ill make it version 1.0 ;-)



Thanks, Norman