format

[eddier]

Thanks so much for the crash proof format function! This is already saving me time.



I was looking back in my old files and noticed some work on an ftp library. I never finished it. However I do have a sequence of commands that logs in, changes directory, and ftp-s a file to a server. It's only a hack (a bad one at that) but might serve as a starting point for an ftp library.  Anyone interested in making an ftp library?



Eddie

[Ryon]

I have wanted to have an FTP area for this site, but my hosting plan does not include anonymous FTP. Darn!

[Lutz]

a ftp library would be a good thing to add to the existing internet protocol modules: smtp, pop3 etc.. Just put it up here on the board for everybody to use/learn from it.



good to hear that the new 'format' function is doing its job!



Lutz

[eddier]

This code stores a file to an ftp server (it works on our server).



I've never tried to retrieve a file from an ftp server, but I think you'ld set up a buffer to store the data, make a data connection, do a RETR filename command to activate the transfer, and write the buffer into a file. Be carefull to have a space between each command and its parameter and terminate each command line with an "rn"

Code Select Expand


;; sequence for ftping a file to an ftp server
;; use the well known port 21 for communicating with the ftp server
;; use passv to let the server create a data transfer connection
;; Note this is a TERRIBLE HACK, I don't even check for response numbers!

(set 'socket (net-connect "host" 21)) ;; connect to a host
(net-receive socket 'rec-buff 256 "rn")

(setq send-buff "USER idrn") ;; send an id for logging on
(net-send socket 'send-buff)

(net-receive socket 'rec-buff 256 "rn")

(setq send-buff "PASS -passwordrn") ;; suppress the opening greeting

(net-receive socket 'rec-buff 256 "rn")

(setq send-buff "CWD subdirrn") ;; change the working directory to subdir
(net-send socket 'send-buff)
(net-receive socket 'rec-buff 256 "rn")

(setq send-buff "TYPE Arn") ;; choose A for ASCII or I for binary transfer
(net-send socket 'send-buff)
(net-receive socket 'rec-buff 256 "rn")

(setq send-buff "PASVrn") ;; let server give a port number for data transfer
(net-send socket 'send-buff)
(net-receive socket 'rec-buff 256 "rn")

;; going to use default mode for transfer (i.e. stream)
(regex {(d+),(d+),(d+),(d+),(d+),(d+)} rec-buff) ;; parse out ip and port
(setq port (+ (* 256 (integer $5)) (integer $6)))
;; connect to that ip and port for data transfer (note: two diff connections)
(setq socket2 (net-connect (string $1 "." $2 "." $3 "." $4) port))

;; the next command sets up a file name and tells the server to accept a
;; stream over the designated port
(setq send-buff "STOR filenamern") ;; send store command over poert 21
(net-send socket 'send-buff)

;; Send the data string for the file over designated port.
(setq send-buff2 (read-file ("filename))
(net-send socket2 'send-buff2)

;; I STAT command to get a response from the server.
(setq send-buff "STATrn")
(net-send socket 'send-buff)
(net-receive socket 'rec-buff 256 "rn")

;; terminate the data port connection
(net-close socket2)

;; terminate the communication connection
(setq send-buff "QUITrn")
(net-send socket send-buff)
(net-close socket1)

(exit)


Eddie

[Lutz]

thankyou so much. I parameterized user-id etc. and shortened the 'net-send', remember that since 7.2.0 'net-send' can take a string or variable directly.



This one works fine for me. Note that there is a delay after "connected" is printed:

Code Select Expand


;; sequence for ftping a file to an ftp server
;; use the well known port 21 for communicating with the ftp server
;; use passv to let the server create a data transfer connection
;; Note this is a TERRIBLE HACK, I don't even check for response numbers!


(set 'host "somehost.com")
(set 'user-id "myid")
(set 'password "secret")
(set 'direc "tmp")
(set 'file-name "testfile")


(set 'socket (net-connect host 21)) ;; connect to a host
(if socket (println "connectet") (print "could not connect"))
(net-receive socket 'rec-buff 256 "rn")
(println rec-buff)

(net-send socket (append "USER " user-id "rn")) ;; send an id for logging on
(net-receive socket 'rec-buff 256 "rn")
(println rec-buff)

(net-send socket (append "PASS " password "rn")) ;; suppress the opening greeting
(net-receive socket 'rec-buff 256 "rn")
(println rec-buff)

(net-send socket (append "CWD " direc "rn")) ;; change the working directory to subdir
(net-receive socket 'rec-buff 256 "rn")
(println rec-buff)

(net-send socket "TYPE Irn") ;; choose A for ASCII or I for binary transfer
(net-receive socket 'rec-buff 256 "rn")
(println rec-buff)

(net-send socket "PASVrn") ;; let server give a port number for data transfer
(net-receive socket 'rec-buff 256 "rn")
(println rec-buff)

;; going to use default mode for transfer (i.e. stream)
(regex {(d+),(d+),(d+),(d+),(d+),(d+)} rec-buff) ;; parse out ip and port
(setq port (+ (* 256 (integer $5)) (integer $6)))
;; connect to that ip and port for data transfer (note: two diff connections)
(setq socket2 (net-connect (string $1 "." $2 "." $3 "." $4) port))

;; the next command sets up a file name and tells the server to accept a
;; stream over the designated port
(net-send socket (append "STOR " file-name "rn")) ;; send store command over poert 21

;; Send the data string for the file over designated port.
(net-send socket2 (read-file file-name))

;; I STAT command to get a response from the server.
(net-send socket "STATrn")
(net-receive socket 'rec-buff 256 "rn")
(println rec-buff)


;; terminate the data port connection
(net-close socket2)

;; terminate the communication connection
(net-send socket "QUITrn")
(net-close socket)

(exit)


Lutz

[eddier]

Your welcome! I see you've already made some improvements. To make a library, one could use the new regular expression stuff to parse out the response numbers and use the new throw and catch mechanism to implement a nice ftp library.



Anyone interested in trying to make an ftp library?



I noticed that regex and parse are very similar. Do we need both? I also noticed that match and find perform similar functions.



Eddie

[Lutz]

This is a working FTP module, read the header for help. It is also in the http://newlisp.org/download/development">http://newlisp.org/download/development directory. I only could test it with my ISP. Thanks again to Eddie for getting this started.



This uses very little memory even with large files.

Code Select Expand


;; ftp.lsp - module for newLISP for FTP transfers
;;
;; v.1.0
;;
;; example:
;;
;; (FTP:put "somebody" "secret" "host.com" "subdir" "file")  ;; upload
;;
;; (FTP:get "somebody" "secret" "host.com" "subdir" "file")  ;; download
;;
;; returns 'true' on sucess else 'nil' and check the variable FTP:result
;; for the last result message from the ftp host
;;
;; to set debug mode, which shows all dialog woth the server:
;;
;; (set 'FTP:debug-flag true)
;;
;;
;;

(context 'FTP)

;; debuggung mode
(set 'debug-mode nil)

;; mode of transfer
(define GET 1)
(define PUT 2)

(define (get user-id password host subdir file-name)
    (transfer user-id password host subdir file-name GET))


(define (put user-id password host subdir file-name)
    (transfer user-id password host subdir file-name PUT))


(define (transfer user-id password host subdir file-name mode)
  (and
    (connect-to host 21)
    (send-get-result (append "USER " user-id "rn") "3")
    (send-get-result (append "PASS " password "rn") "2")
    (send-get-result (append "CWD " subdir "rn") "2")
    (send-get-result "TYPE Irn" "2")
    (set 'buff (send-get-result "PASVrn" "2"))
    (regex {(d+),(d+),(d+),(d+),(d+),(d+)} buff)
    (set 'port (+ (* 256 (integer $5)) (integer $6)))
    (set 'ip (string $1 "." $2 "." $3 "." $4))
    (set 'socket2 (net-connect ip port))

    (if (= mode PUT)
        (and
            (check-file file-name)
            (net-send socket (append "STOR " file-name "rn"))
            (set 'fle (open file-name "r"))
            (while (> (read-buffer fle 'buffer 512) 0)
                (if debug-mode (print "."))
                (net-send socket2 buffer 512))
            (close fle)) true)

    (if (= mode GET)
        (begin
            (net-send socket (append "RETR " file-name "rn"))
            (set 'fle (open file-name "w"))
            (while (net-receive socket2 'buffer 512)
                (if debug-mode (print "."))
                (write-buffer fle buffer))
            (close fle)) true)

    (send-get-result "STATrn" "1")
    (or (net-close socket2) true)
    (net-send socket "QUITrn")
    (or (net-close socket) true)))


(define (send-get-result str code)
    (net-send socket str)
    (if debug-mode (println "sent:" str))
    (net-receive socket 'result 256 "rn")
    (if debug-mode (println result))
    (if (starts-with result code) result))


(define (connect-to host port)
    (set 'FTP:result nil)
    (set 'socket (net-connect host port))
    (if socket
        (net-receive socket 'result 256 "rn")
        (begin
            (set 'result "could not connect")
            nil)))    
   
(define (check-file file-name)
    (if (file? file-name)
        true
        (begin
            (set 'result (append file-name " does not exist"))
            nil)))

(context 'MAIN)


;; test
;
;(set 'FTP:debug-mode true)
;
;(FTP:put "userid" "password" "site.com" "tmp" "testfile")
;
;(FTP:get "userid" "password" "site.com" "tmp" "testfile")
;
;(exit)

;; eof





Lutz

[eddier]

Works with our server as well :)  I like it.



I am working on all kinds of scripts to generate html pages. For example one that collects all html files in the directory except "index.html" and creates an index page with update information, logo, etc...

I made a script this week to convert a tab delimited file (that others can edit with a spreadsheet) and convert it to a faculty/staff and alumnae directory and upload it to our server (You can see the results at http://www.bmc.edu/meetouralumnae.html">www.bmc.edu/meetouralumnae.html or http://www.bmc.edu/meetourfacultyandstaff.html">www.bmc.edu/meetourfacultyandstaff.html).



I am going to modify " get" to  download a file into a buffer and "put" to upload a buffer into a file. I can check to see if file-name is a symbol and tread file-name as a buffer.  What do you think?



Eddie

[Lutz]

I had a look at those pages, nice! and learned something new:



<div style="overflow:auto">



did not know scrolling tables are so easy.



>>>

I am going to modify " get" to download a file into a buffer and "put" to upload a buffer into a file. I can check to see if file-name is a symbol and tread file-name as a buffer. What do you think?

>>>



reading all in one swoop into a buffer is probably faster, of course it will not work for 100Mbyte files which go better to a file first.



BTW, there is a new improved ftp.lsp version 1.1 in the development directory.



Lutz