Some issuses about socket function

[winger]

first:

I have wirted a multiporcess scanner.

i use ten process.

All of them blocked sometimes later!!!

Code Select Expand


(define (check_path host , url url2 result str  socket)
            (if-not (starts-with host "http://") (setf url2  (string "http://" host)))
            (dolist (port portlst)
                (when (and
                            (setf socket (net-connect host (int port) TIMEOUT)) ;have question
                            (close socket)
                            (find "^ERR: server code 404" (get-url (append url2 ":" port FOOLCHECK) ) 1)
                            )
                            (dolist (path ALLPATH)
                            (setf url (append url2 ":" port  path));notice here
                            ;(setf result (get-url url  "list debug"TIMEOUT AGENT))
                            (setf result (get-url url  TIMEOUT AGENT))
                            (if-not (find "^ERR: Operation timed out" result 1)
                                (if-not (find "^ERR.*n*" result 1)
                                    (and
                                    (println "rnrn----------------------Bingo----------------------rn" url "rn")
                                    (set 'str  (string  "<a href="" url " ">" url "</a>        200 </br>"))
                                    (write FID_R  str)
                                    )
                                   ; (
                                    ;can add other err log ex:503 forbiden
                                    ;)
                              )
                              (and
                                (println "rnrn----------------------Time out----------------------rn" url "rn")
                                (set 'str  (string  "<a href="" url "">" url "</a>        time out </br>"))
                                (write FID_R  str)
                              )
                        )
                    )

                )
            )
)
(set 'pnum (min pnum allipnum))
 (dotimes (n pnum)
    (spawn (sym (allip pcount)) (brute_path  (allip pcount)))
    (inc pcount)
    (println "")
)

(define (report pid)
    (semaphore sid -1)
    (when (< pcount allipnum)
        (spawn (sym (allip  pcount)) (check_path (allip pcount)))
    )
    (inc pcount)
    (semaphore sid 1)
)

i strace process find :

then always thus:



!netstat -anp  -c 1| grep newlisp



tcp        1      0 192.168.51.103:46220    115.36.76.90:80        CLOSE_WAIT  9771/newlisp    

tcp        0      0 192.168.51.103:52180    113.17.16.64:8000    ESTABLISHED 23789/newlisp  

tcp        1      0 192.168.51.103:46192    115.26.76.90:80        CLOSE_WAIT  9771/newlisp    

tcp        1      0 192.168.51.103:52520    115.26.76.90:80        CLOSE_WAIT  9771/newlisp    

tcp        0      0 192.168.51.103:35091    121.1.45.14:8080      ESTABLISHED 24664/newlisp  

tcp        1      0 192.168.51.103:46187    115.36.76.90:80        CLOSE_WAIT  9771/newlisp    

tcp        0      0 192.168.51.103:48741    115.8.11.11:8001    ESTABLISHED 24305/newlisp  

tcp        0      0 192.168.51.103:36782    58.9.19.181:80       ESTABLISHED 24934/newlisp  

tcp        0      0 192.168.51.103:32917    121.4.45.10:8002      ESTABLISHED 24575/newlisp  

tcp        0      0 192.168.51.103:37811    58.24.19.13:8080     ESTABLISHED 24926/newlisp  

tcp        1      0 192.168.51.103:46200    115.23.76.90:80        CLOSE_WAIT  9771/newlisp    

tcp        0      0 192.168.51.103:44936    121.14.45.12:80        ESTABLISHED 24607/newlisp  

tcp        0      0 192.168.51.103:45617    115.28.1.206:80      ESTABLISHED 24423/newlisp  

tcp        0      0 192.168.51.103:41055    121.1.45.121:8002      ESTABLISHED 24596/newlisp  

tcp        0      0 192.168.51.103:43155    121.4.45.240:80        ESTABLISHED 24720/newlisp  



strace -v -p  24664

select(6, [5], NULL, NULL, {0, 1000})   = 0 (Timeout)

select(6, [5], NULL, NULL, {0, 1000})   = 0 (Timeout)

select(6, [5], NULL, NULL, {0, 1000})   = 0 (Timeout)

.

.

.

second:

Code Select Expand

(set 'socket (net-connect "www.baidu.com" 80))
(net-send socket "GET /rnrn")
(net-receive socket buffer 10000);have issuses
(println buffer)
(exit)

[winger]

I found where the problem.

Code Select Expand

get-url

Function get-url always not return on child process.

Even if you set the timeout does not work。

Comment out the line that contains the  "get-url", will be able to perform good.

[Lutz]

Whenever you use spawn, you also have to use sync. This function processes the return signals of child-processes. Without doing this, they will stay in memory.



See here for a programs using spawn to retrieve multiple web pages in parallel using get-url with a timeout:



http://www.nuevatec.com/query.html">http://www.nuevatec.com/query.html

and here for the complete source:

http://www.nuevatec.com/syntax.cgi?query-cgi.txt">http://www.nuevatec.com/syntax.cgi?query-cgi.txt



this critical snippet is towards the bottom of the program:

Code Select Expand

; spawn a childprocess for each link
(dolist (lnk (0 20 links))
(set 'pid (spawn 'page (get-url lnk 4000)))
(push (list pid lnk) SITES -1))

; this gets executed whenever a page has been retrieved
(define (page-ready-event pid)
    (let (link (0 80 (lookup pid SITES)))
        (set 'link (url-decode link))
        (println (inc cnt) " pid:" pid " " (- (time-of-day) START_TIME) " ms " link BRK)
        (push (lower-case (clean-html page)) CONTENT -1)
        (inc xferred (length page)))
)

; start waiting for pages
(println "waiting: ..." BRK BRK)
(unless (sync 10000  page-ready-event)
(println BRK "timeout" BRK))


see the sync statement at the end - scroll up.

[winger]

thx lutz...



冏

I forgot to paste the code about "sync" portion.

I use  "sync"from the outset. --!



Post it to github now:

https://github.com/freewinger/wsec_toolkit/blob/master/svn_scanner.lsp">https://github.com/freewinger/wsec_tool ... canner.lsp">https://github.com/freewinger/wsec_toolkit/blob/master/svn_scanner.lsp

[Lutz]

Perhaps you have some unknown error thrown in brute_path. Try the following to catch anything occurring in that function:

Code Select Expand


(spawn (sym (allip  pcount))
    (begin (catch (brute_path (allip pcount)) 'result) result)
)


I also wonder about another spawn in the event funtion report handling the returns from spawns. The event function is called from inside sync in the same process. Semaphores would not be necessary in that case.

[winger]

Code Select Expand


;i change report function to  following :
(define (report pid)
    (semaphore sid -1)
    (when (< pcount allipnum)
        (spawn (sym (allip  pcount))
            (if (begin (catch (brute_path (allip pcount)) 'result) result) (println  "----:----> "result  (read-key)))
        )
    )
    (inc pcount)
    (semaphore sid 1)
)

But  code  

Code Select Expand

(println  "----:----> "result  (read-key)) never execute (This should be a syntax error--!) .



Three days i satart script use

newlisp svnscan.lsp -n 60 -f ip2.txt -sc

Find script just output

" ."

when  i come back today .

It is work done after I press the "Enter" key  four times . (it output num 10 every time . mybe it's  number of bytes written by the function "write" ?).

........................................................................................................................................................................10

.......................................................................................

10

10

10

Begin process scan result

Spend 0 5397 minutes 27 seconds

Result file: svnscan_result2013年02月19日17时48分56秒.html

# ps -A |grep  newlisp

12626 pts/11   3-17:38:38 newlisp

21272 pts/11   00:00:12 newlisp

27581 pts/11   00:00:11 newlisp

28713 pts/11   00:00:11 newlisp





#strace -v -p 12626

....

gettimeofday({1361591102, 962949}, NULL) = 0

waitpid(-1, 0xbfeff0e8, WNOHANG)        = 0

gettimeofday({1361591102, 962993}, NULL) = 0

waitpid(-1, 0xbfeff0e8, WNOHANG)        = 0

gettimeofday({1361591102, 963037}, NULL) = 0

....



#strace -v -p 21272

Process 21272 attached - interrupt to quit

read(0,

.........

[Lutz]

sync is not able to continue when waiting on read-key in the event function, it has called. The four times you hit the enter key is for four spawned processes, which had finished, but could not be processed further, because sync was waiting for return from the report event function. Don't put anything blocking into the event function.