How to take one byte from a string

[Fritz]

I'm trying to read the string byte-per-byte (for encoding from 8-bit codepage to UTF-8). But (pop the-string) returns some random number of bytes, so does (the-string 0) etc:

http://img7.imageshost.ru/imgs/091008/3b8db7732c/11005.png">http://img7.imageshost.ru/imgs/091008/3 ... /11005.png">http://img7.imageshost.ru/imgs/091008/3b8db7732c/11005.png



(set-locale "C") did not help too. Only working way I have found is to write temporary file and then use read-char function.

Code Select Expand


; Usage: (cyr-win-utf "text in windows-1251 encoding")
; Decodes text from windows-1251 to utf-8
(define (cyr-win-utf t-linea)
  ; Loading encoding table
  (set 'en-win-1251 '((255 "я") (254 "ю") (253 "э") (252 "ь") (251 "ы")
  (250 "ъ") (249 "щ") (248 "ш") (247 "ч") (246 "ц") (245 "х") (244 "ф")
  (243 "у") (242 "т") (241 "с") (240 "р") (239 "п") (238 "о") (237 "н")
  (236 "м") (235 "л") (234 "к") (233 "й") (232 "и") (231 "з") (230 "ж")
  (184 "ё") (229 "е") (228 "д") (227 "г") (226 "в") (225 "б") (224 "а")
  (223 "Я") (222 "Ю") (221 "Э") (220 "Ь") (219 "Ы") (218 "Ъ") (217 "Щ")
  (216 "Ш") (215 "Ч") (214 "Ц") (213 "Х") (212 "Ф") (211 "У") (210 "Т")
  (209 "С") (208 "Р") (207 "П") (206 "О") (205 "Н") (204 "М") (203 "Л")
  (202 "К") (201 "Й") (200 "И") (199 "З") (198 "Ж") (168 "Ё") (197 "Е")
  (196 "Д") (195 "Г") (194 "В") (193 "Б") (192 "А")))
  ; saving string to a temp file
  (set 't-file-name (append "/tmp/" (crypto:md5 (string (random)))))
  (write-file t-file-name t-linea)
  ; loading characters to the t-out
  (set 't-out "")
  (set 't-file (open t-file-name "read"))
  (while (set 't-char (read-char t-file))
    (push (or (lookup t-char en-win-1251) (char t-char)) t-out -1))
  (close t-file)
  t-out)

May be, there is a shorter way, without file-writing? I need this function in both Linux and Windows, and Windows temp directory has another name.

[cormullion]

Does unpack help at all?

[Fritz]

Does unpack help at all?

Thank you! I think, yes, "unpack" is a solution. Function is much shorter now:

Code Select Expand

(define (cyr-koi-utf-2 t-linea)
  ; putting character codes to the list
  (set 't-list (unpack (dup "b" (mul 2 (length t-linea))) t-linea))
  ; decoding characters from 't-list to the 't-out
  (set 't-out "")
  (dolist (t-char t-list)
    (push (or (lookup t-char en-koi8r) (char t-char)) t-out -1))
  t-out)

It works ok. Have found a funny thing, btw. Manual says: "Length... returns... the number of characters in a string". But (length "one-russian-letter-in-utf-8") returns 2, not 1.

[Jeff]

dostring processes a string one char at a time...

[Fritz]

dostring processes a string one char at a time...

Dostring takes several bytes from string per time, and I need one byte only:

http://img7.imageshost.ru/imgs/091008/ee0e3865a2/4489d.png">http://img7.imageshost.ru/imgs/091008/e ... /4489d.png">http://img7.imageshost.ru/imgs/091008/ee0e3865a2/4489d.png

[m35]

Manual says: "Length... returns... the number of characters in a string". But (length "one-russian-letter-in-utf-8") returns 2, not 1.

What version of the manual are you using? The http://www.newlisp.org/downloads/newlisp_manual.html#length">current manual says

Returns ... the number of bytes in a string.

There is also http://www.newlisp.org/downloads/newlisp_manual.html#utf8len">utf8len for utf8 strings.



I've run into troubles myself when treating strings as binary data. It would work fine in normal newlisp then blow up when running in utf8 newlisp. Can't remember what I did to make things universal though.



Edit

Looked at the functions in the manual and I see 3 functions that work with bytes regardless: unpack (as you know), slice, and get-char.



You could just loop over the bytes with slice or get-char

Code Select Expand

(for (i 0 (- (length s) 1))
   (setq c (slice s i 1))
   ' or
   (setq c (char (get-char (+ i (address s)))))
)


[cormullion]

You could even use implicit slicing:


 

Code Select Expand

(offset length utf8-str)

but don't confuse it with implicit indexing:


   

Code Select Expand

(utf8-str offset length)

which does work on characters not bytes.



You can sometimes write code for both UTF8 and non-UTF8. Eg:

Code Select Expand

(define (string-length s)
    (if unicode (utf8len s) (length s)))


[Fritz]

I think I have old manual. It is good: now I can be sure my "unpack" will work in future versions too.

You could just loop over the bytes with slice or get-char

Code Select Expand

(for (i 0 (- (length s) 1))
   (setq c (slice s i 1))
   ' or
   (setq c (char (get-char (+ i (address s)))))
)


Slice works, at least, with uft8 locale and ASCII encoded line. But get-char gives me only some strange negative numbers. Only this entangled construction works:

Code Select Expand


(dotimes (i (length rln))
  (print (or (lookup (+ 256 (get-char (+ i (address rln)))) en-win-1251) "?")))


[Fritz]

You could even use implicit slicing:


 

Code Select Expand

(offset length utf8-str)

But how? ((address str) 1 1) ?

[cormullion]

How about

Code Select Expand

(set 's "04030201")
(for (i 0 3)
  (println (get-char (address (i 1 s)))))
4
3
2
1


where i is the offset, 1 is the length, and s is the string you're slicing...

[Lutz]

You can do without 'address' if the argument is a string. This will do it too:

Code Select Expand

(for (i 0 3) (println (get-char (i 1 s))))

[Fritz]

You can do without 'address' if the argument is a string. This will do it too:

Code Select Expand

(for (i 0 3) (println (get-char (i 1 s))))

Code Select Expand

(get-char (address (i 1 s)))



gives always "0" as a result.

Code Select Expand

(get-char (i 1 s))

works, but only in this strange form:

Code Select Expand

(+ 256 (get-char (i 1 s)))

PS: its a pity "explode" can not work with raw bytes, so I can not use "map".