Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Topics - John_Small

Pages1
#1
newLISP in the real world / Out of memory message
June 25, 2007, 10:33:43 AM
I'm using newLISP for filtering large number of

huge ascii text files (one line equals one record).



The filter is making simple changes to the input

text line (record) and printing it to a tmp file

and then when finished replacing the original

file (after it was closed of course).



After running for an hour or so it finally

crashes with a message to the effect it is

out of memory in function such and such.



I'm not using any recursive algorithm so

I don't think it is the stack running out of

space.  I am running a huge number of

huge files through the filter (using "for"

loops to read the archive directory files).

The filter is making changes to these

archive files and this is why it can run

for hours.



I didn't save the exact error message but

I can start the job again and get the message

verbatim if you need it.
#2
newLISP newS / implementation of file-map
June 08, 2007, 05:34:07 PM
This was done in a hurry and briefly tested.



It allows you to map the lines from the input to

the output file.  The optional 4th argument

indicates the out-file is temporary and that the output

file replaces the input file when finished.  If the function

succeeds it returns the final file name otherwise it returns

nil.  I didn't look to see if newLISP has a temp file

name generator so the function prototype could

be changed to something more appropriate.

I'm using newLISP to write file stream combinators

and filters so I'll be using this all the time.  Seems

like it would be easy to include something similar

in functionality with the standard library.  The reason

I didn't use read-file and write-file is because

the files I'm filtering are gigantic and can't fit into

memory. 







(define (file-map fn-line-map filename out-filename out-temp)

  (setq out-file (open out-filename "write"))

  (if out-file

    (begin

      (setq in-file (open filename "read"))

      (setq out-written nil)

      (if in-file

        (begin

          (while (read-line in-file)

            (write-line (fn-line-map (current-line)) out-file))

          (setq out-written true)

          (close in-file)

        )

      )

      (close out-file)

      (if out-written

        (if out-temp

          (begin

            (delete-file filename)

            (rename-file out-filename filename)

            filename

          )

          out-filename

        )

        nil

      ) 

    )

    nil

  )

)
#3
newLISP newS / read-file and write-file
June 07, 2007, 04:38:29 PM
Suppose read-file allowed an optional 2nd argument



   (read-file filename fnc)



This fnc is called once for each line in the file.

But instead of returning the entire file in a str-buffer

it returns a continuation function that can be repeatedly

called until returning EOF.  If this is too confusing

suppose it were called read-file-line or something.

The idea here is to dispense with having to test for

EOF and closing the file which is all hidden then just

like it is with read-file.  The difference it larger

files don't have to be read entirely into memory

all at once.



Likewise if  write-file could differentiate between

a str-buffer and an continuation argument then

it could repeatedly call the continuation (i.e. a function

taking no arguments).



Cutting to the chase instead of changing read-file and write-file

suppose instead (or in addition) that copy-file took

an optional 3rd argument that was a current-line mapping

function.  If copy-file returned the name of the destination

file (or nil) we could write something like



    (rename-file src (copy-file src dst map-fnc))



or better yet



   (map-file src map-fnc)
#4
newLISP newS / (for-all cond list1 [... listn])
June 07, 2007, 03:55:26 PM
It would be nice if (for-all ..) allowed for additional,

optional list arguments.



    (for-all (fn (item) ...)  list1 ... listn)



It may not always be efficient to use append to

concatenate the lists since the for-all may be in

a loop itself.



It would also be nice to have a disjunction complement to

for-all called for-any



    (for-any cond  list1 ... listn)
#5
newLISP and the O.S. / write-line outputs CRCRLF on my XP box
May 31, 2007, 05:03:03 PM
Hi,



I'm using version 9.1.1 on windows XP.



  (write-line "")



and



   (println "")



both output a three character sequence



      CRCRLF



instead of just



     CRLF



Is there something I'm doing wrong?
Pages1