read-file doesn't work with /dev/stdin

[TedWalther]

how can I make read-file work with /dev/stdin?  when my application doesn't have a file specified by the user, I want to fall back to stdin to slurp my data in.  It is potentially utf8 data input, and I will want to iterate over each character in the stream..



Writing to /dev/stdout works just fine.



Perhaps the simplest solution is to let read-file etc use file descriptors (integers) as well as strings?



Example code:



(setq from-format (sym (or (env "FROM") "utf8")))

(setq   to-format (sym (or (env "TO") "neo-paleo-hebrew")))

(setq output-file (or (env "SAVEAS") "/dev/stdout"))

(setq  input-file (or (env "OPEN")   "/dev/stdin"))



(dostring (c (read-file input-file))

   (append-file output-file (char c)))

[Lutz]

'read-file' can only be used to read from files.



You could use 'read-char' to read from stdin using 0 as the device. Returned values will be one byte at a time, or use 'read-line' then 'explode' the line into UTF-8 multibyte characters. In both cases processing does not start until a line-feed.

Code Select Expand

#!/usr/bin/newlisp

(while (!= (setq ch (read-char 0)) (char "q"))
(println "->" ch))

(exit)


Or use 'read-key', which will process immediately after each key-stroke:

Code Select Expand

#!/usr/bin/newlisp

(while (!= (setq ch (read-key)) (char "q"))
(println "->" ch))

(exit)


[TedWalther]

Lutz, could it be time to change read-char with read-byte, and make read-char do the utf8 thing?



Would it be hard to slurp in fd 0 for read-file?  I mean, generally, I cat a file into the program, and it is no different from reading a regular file.  The only difference between read-file and read-char is that one takes an already opened file descriptor, and the other wants to do the open itself.  Once read-file accepted all possible input from fd0, it makes sense it would close it.

[Lutz]

I see 'read-char', 'read-buffer' and 'read-line' as the low-level API using file descriptors and dealing with octets and 'read/write/append -file' as the highlevel API dealing with file names.

[Lutz]

A future version of newLISP has a 'read-utf8' which works like 'read-char', but reads UTF-8 characters from a file handle or stdin. Until then I suggest using (explode (read-line ...)) instead. 'explode' splits a string on multi-byte character borders in UTF8 enabled versions of newLISP.

[TedWalther]

A future version of newLISP has a 'read-utf8' which works like 'read-char', but reads UTF-8 characters from a file handle or stdin. Until then I suggest using (explode (read-line ...)) instead. 'explode' splits a string on multi-byte character borders in UTF8 enabled versions of newLISP.

Thank you for the explanation.  I'm meditating now on the whole utf8 vs byte oriented paradigm works in newLISP.  I should have spent more time meditating on the pack and unpack functions.



Does (char ...) use (pack ...) internally?  Should it?  I recently had some problems when trying to output some Hebrew characters in codepage iso8859-8.  I think (char ...) was interpreting them as utf8 characters.

[Lutz]

'char' does not use 'pack' or 'unpack' internally, but 'char' is UTF-8-sensitive when using the UTF-8 enabled version of newLISP. If a Windows installation is working with ISO-8859-x character sets, then one should not use the UTF-8 enabled version for Hebrew and most Eastern European (cyrillic) character sets.



When the UTF-8 enabled version of newLISP sees Hebrew ISO-8859-8 characters (greater 127), it sees them not as Hebrew but something else. All characters below 128 will interpreted as ASCII in both, ISO-8859-x or UTf-8/UNICODE. All above 127 will be one-byte-wide characters in ISO-8859-x characters sets but could initiate different one- or multi-byte-wide characters in UTF-8.



Here is a list of all functions sensitive to UTF-8 characters when using the UTF-8 enabled version: http://www.newlisp.org/newlisp_manual.html#utf8_capable">http://www.newlisp.org/newlisp_manual.html#utf8_capable



Ps: by default the Windows installer shipped is non-UTF8, and the Mac OS X installer is UTF-8.

[newdep]

Lutz,



Why do you make a differences in function names between utf8 and none utf8?



Its somewhat awkward to have that inside a language...

I would expect global functionality in a function instead of separate behaviour..



Just a though....

[Lutz]

Some functions work differently in utf8 and non-utf8 versions like these: http://www.newlisp.org/newlisp_manual.html#utf8_capable">http://www.newlisp.org/newlisp_manual.html#utf8_capable They are meant to work strictly on displayable strings and can display this global behavior. But there are also functions where you need both version like 'read-char' and 'read-utf8'. If you let 'read-char' switch behavior, like the others in the link, you would not be able to read binary files or ISO-8859 files correctly.