Base64 encoder

[pjot]

Hi all,



After all contributions of newdep I couldn't stay behind... ;-)   Below my base64 encoder written in newLISP. Have phun with it.



Cheers

Peter.

--------------------------------------------------------------------------



#!/usr/bin/newlisp

;;

;; Base64 converter using newLISP. Tested on Slackware 9.1 with newLISP 7.5.4.

;;

;; Proxy-servers require a base64 encoded "username:password" to pass through.

;;

;; With this encoder you can hack your way out :-)

;;

;;----------------------------------------------------------------------------

 

;; Setup base64 encode string

(set 'BASE64 "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/")



;; Get input from user

(print "Enter a string to convert: ")

(set 'dat (read-line))



;; Initialize result variable to string

(set 'enc "")



;; Mainloop

(while (> (length dat) 0) (begin

   

   ;; Find ASCII values

   (if (= (length dat) 1)

      (begin

         (set 'byte1 (char dat))

         (set 'byte2 0)

         (set 'byte3 0)))

   (if (= (length dat) 2)

      (begin

         (set 'byte1 (char dat))

         (set 'byte2 (char dat 1))

         (set 'byte3 0)))

   (if (> (length dat) 2)

      (begin

         (set 'byte1 (char dat))

         (set 'byte2 (char dat 1))

         (set 'byte3 (char dat 2))))



   ;; Now create BASE64 values

   (set 'base1 (/ byte1 4))

   (set 'base2 (+ (* (& byte1 3) 16) (/ (& byte2 240) 16)))

   (set 'base3 (+ (* (& byte2 15) 4) (/ (& byte3 192) 64)))

   (set 'base4 (& byte3 63))



   ;; Find BASE64 characters

   (if (= (length dat) 1)

      (begin

         (set 'enc (append enc (nth base1 BASE64)))

         (set 'enc (append enc (nth base2 BASE64)))

         (set 'enc (append enc "=="))

         ;; Put 'dat' to empty list

         (set 'dat "")))

   (if (= (length dat) 2)

      (begin

         (set 'enc (append enc (nth base1 BASE64)))

         (set 'enc (append enc (nth base2 BASE64)))

         (set 'enc (append enc (nth base3 BASE64)))

         (set 'enc (append enc "="))

         ;; Put 'dat' to empty list

         (set 'dat "")))

   (if (> (length dat) 2)

      (begin

         (set 'enc (append enc (nth base1 BASE64)))

         (set 'enc (append enc (nth base2 BASE64)))

         (set 'enc (append enc (nth base3 BASE64)))

         (set 'enc (append enc (nth base4 BASE64)))

         ;; Decrease 'dat' with 3 characters

         (set 'dat (slice dat 3))))

))



;; Print resulting string

(println enc)



;; Exit

(exit)

[newdep]

Hello Pjot,



Great invention ;-) nice work... ill plug it in as a module...



Norman.

[Lutz]

Thanks Pjot,



a small improvment:



>>>> instead of <<<<

(set 'enc (append enc (nth base1 BASE64)))

(set 'enc (append enc (nth base2 BASE64)))

(set 'enc (append enc "=="))





>>>> you can do this <<<<

(set 'enc (append (nth base1 BASE64) (nth base2 BASE64)  "=="))



in newLISP in all places where it makes sense you can specify more than one arg. Makes the code shorter and faster.



Lutz

[pjot]

Hi Lutz,



Thanks for the tip! I did not know that, I will change it here in my code.  (I am sorry for the layout of the program, but the tabs do not seem to appear....)



A decoder will follow.



Regards



Peter.

[Lutz]

thanks Peter, our pops3.lsp users will be delighted, I will put your code in a future base64.lsp module in the newLISP distribution, if this is Ok with you.



Lutz

[pjot]

Hi Lutz,



Of course, it's ok.



Peter.

[pjot]

As promised, the Base64 decoder below. I've applied your tip to this code but it does not improve readability...



Regards



Peter



-----------------------------------------------------------



#!/usr/bin/newlisp

;;

;; Base64 decoder

;;

;; Due to the nature of newLISP, this is the smallest

;; BASE64 decoder I've ever written.

;;

;;-------------------------------------------------------



;; Setup base64 string

(set 'BASE64 "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/")



;; Get input from user

(print "Enter a string to convert: ")

(set 'dat (read-line))



;; Initialize result variable to string

(set 'res "")



;; Mainloop

(while (> (length dat) 0) (begin



   ;; Find the indexnumber in the BASE64 definition

   (set 'byte1 (find (nth 0 dat) BASE64))

   (if (= byte1 nil)(set 'byte1 0))

   (set 'byte2 (find (nth 1 dat) BASE64))

   (if (= byte2 nil)(set 'byte2 0))

   (set 'byte3 (find (nth 2 dat) BASE64))

   (if (= byte3 nil)(set 'byte3 0))

   (set 'byte4 (find (nth 3 dat) BASE64))

   (if (= byte4 nil)(set 'byte4 0))



   ;; Recalculate to ASCII value

   (set 'res (append res (char (+ (* (& byte1 63) 4) (/ (& byte2 48) 16))) (char (+ (* (& byte2 15) 16) (/ (& byte3 60) 4))) (char (+ (* (& byte3 3) 64) byte4))))



   ;; Decrease string with 4

   (set 'dat (slice dat 4))))



;; Print resulting string

(println res)



;; Exit

(exit)

[newdep]

Hello pjot,



Your going too fast :)



Regards, Norman.

[Lutz]

thanks Peter, getting shorter and faster, I wonder if anybody can top this ;-) ?



Lutz

[HPW]

Testing on WIN with this sample code:

(PS: BASE64 is a protected symbol!)

Code Select Expand


;;
;; Base64 converter using newLISP. Tested on Slackware 9.1 with newLISP 7.5.4.
;;
;; Proxy-servers require a base64 encoded "username:password" to pass through.
;;
;; With this encoder you can hack your way out :-)
;;
;;----------------------------------------------------------------------------

;; Setup base64 encode string

(context 'BASE64)



(define (encode dat)

(set 'base64charset "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/")

;; Get input from user
;(print "Enter a string to convert: ")
;(set 'dat (read-line))

;; Initialize result variable to string
(set 'enc "")

;; Mainloop
(while (> (length dat) 0)
(begin
;; Find ASCII values
(if (= (length dat) 1)
(begin
(set 'byte1 (char dat))
(set 'byte2 0)
(set 'byte3 0)))
(if (= (length dat) 2)
(begin
(set 'byte1 (char dat))
(set 'byte2 (char dat 1))
(set 'byte3 0)))
(if (> (length dat) 2)
(begin
(set 'byte1 (char dat))
(set 'byte2 (char dat 1))
(set 'byte3 (char dat 2))))

;; Now create BASE64 values
(set 'base1 (/ byte1 4))
(set 'base2 (+ (* (& byte1 3) 16) (/ (& byte2 240) 16)))
(set 'base3 (+ (* (& byte2 15) 4) (/ (& byte3 192) 64)))
(set 'base4 (& byte3 63))

;; Find BASE64 characters
(if (= (length dat) 1)
(begin
(set 'enc (append enc(nth base1 base64charset)(nth base2 base64charset)"=="))

;; Put 'dat' to empty list
(set 'dat "")))
(if (= (length dat) 2)
(begin
(set 'enc (append enc (nth base1 base64charset)(nth base2 base64charset)(nth base3 base64charset)"="))

;; Put 'dat' to empty list
(set 'dat "")))
(if (> (length dat) 2)
(begin
(set 'enc (append enc (nth base1 base64charset)(nth base2 base64charset)(nth base3 base64charset)(nth base4 base64charset)))
;; Decrease 'dat' with 3 characters
(set 'dat (slice dat 3))))
))

;; Return resulting string
enc)








;;
;; Base64 decoder
;;
;; Due to the nature of newLISP, this is the smallest
;; BASE64 decoder I've ever written.
;;
;;-------------------------------------------------------

(define (decode dat)

;; Setup base64 string
(set 'base64charset "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/")

;; Get input from user
;(print "Enter a string to convert: ")
;(set 'dat (read-line))

;; Initialize result variable to string
(set 'res "")

;; Mainloop
(while (> (length dat) 0)
(begin
;; Find the indexnumber in the base64charset definition
(set 'byte1 (find (nth 0 dat) base64charset))
(if (= byte1 nil)(set 'byte1 0))
(set 'byte2 (find (nth 1 dat) base64charset))
(if (= byte2 nil)(set 'byte2 0))
(set 'byte3 (find (nth 2 dat) base64charset))
(if (= byte3 nil)(set 'byte3 0))
(set 'byte4 (find (nth 3 dat) base64charset))
(if (= byte4 nil)(set 'byte4 0))

;; Recalculate to ASCII value
(set 'res (append res
(char (+ (* (& byte1 63) 4) (/ (& byte2 48) 16)))
(char (+ (* (& byte2 15) 16) (/ (& byte3 60) 4)))
(char (+ (* (& byte3 3) 64) byte4))))

;; Decrease string with 4
(set 'dat (slice dat 4))))

;; Print resulting string
(trim res "00")
)

(context 'MAIN)




Bugs corrected!

[newdep]

Hello HPW,



"00" odd actualy... Looks like windows displays the <EOF> or <EOL>

which is an NULL-byte normaly but im not sure on this...



Norman.

[Lutz]

newLISP will only display the null character if it was somehow specified, internally there is one more character behind it, i.e:



(set 'var "000000") => "000000"



(length var) => 3



but internally newLISP allocates one more byte, which is always 00 this way strings are limited when using 'print', 'format' etc. i.e.



(set 'var "A0000") => "A0000"



but



(println var)

A

"A0000"



the first A is the printed one and the other thing is the return value from the print statment. Internally the var string is allocated with 4 bytes.



most of newLISP's functions can work on binary content allthough not documented spefically because strings are always stored in a buffer with a zero appended and the buffer length stored in a different field of the LISP cell.



Lutz

[Lutz]

Perhaps Peters BASE64 code is just fine, from:



http://www.securecode.net/Base64Convert+main.html">http://www.securecode.net/Base64Convert+main.html



>>>At the end of the encoding process we might run into a problem. If the size of the original data in bytes is a multiple of three, everything works fine. If it is not, we might end up with one or two 8-bit bytes. For proper encoding, we need three bytes, however.



The solution is to append enough bytes with a value of '0' to create a 3-byte group. Two such values are appended if we have one extra byte of data, one is appended for two extra bytes.

>>>



Si I think you can just strip of the trailing 00 doing a: (trim str)



Lutz

[pjot]

Hi all,



Indeed it is the nature of Base64 which is bothering us, I think. For the ENCODING part, we are going from 3 bytes to 4 'bytes' (3x8 to 4x6). The decoding part is the reverse (4x6 to 3x8).



The padding sign '=' is used to fill up the empty places when the input string originally is not a multiple of 4. This   '=' sign however is not part of the BASE64 string, therefore with the decoder a '0' is produced in order to perform a succesfull binary calculation backwards. This might result in a string with "00" at the end.



Even not shown in my prompt here, the length of the encoded "Peter" = "UGV0ZXI=" will deliver a decoded length of 6 which is "Peter00" again, since the base64 string ends with the padding symbol '='.



Indeed the best workaround for all this is to TRIM your resulting string in the BASE64 decoder function towards a regular one. (How is this with other languages? Gnu AWK and Scriptbasic happen to do this automatically.)



So the last line in the decoder must be:



;; Print resulting string

(trim res "00"))





Thanks for the tip.



Peter.

[Lutz]

GNU Awk and Script Basic only handle 'C' strings which always end with a zero. These languages cut of anything after a zero implicitly, as a drawback they don't handle binary data as you can using newLISP.



Lutz