W3C home > Mailing lists > Public > www-talk@w3.org > March to April 2004

webbot

From: Matthew Carey <matthew@ssl.co.uk>
Date: Tue, 02 Mar 2004 17:08:39 +0000
Message-ID: <4044BF97.1000602@ssl.co.uk>
To: www-talk@w3.org

The webbot goes into an infinite tight loop on some sites. (latest checkout)

When started like this:

./webbot -depth 99 -v -prefix 'http://www.uni4me.com/uni4me' 
'http://www.uni4me.com/uni4me/index.cfm'

Most of the time it gets stuck a little way into walking this site.

Now this may be because some vital parameter is missing but it works 
with most sites just the occaisonal site causes problems.

This happens right down in the event loop. Any ideas?


Host Event.. WRITE passed to 
`http://www.uni4me.com/uni4me/what-is-uni/index.cfm#'
HTDoConnect. WOULD BLOCK `www.uni4me.com'
Memory Free. 0x80e5d10
Event Loop.. calling select: maxfds is 4
............ READ :
............ WRITE: 4
............ OOB  :
............ Timeout is 20 s, 0 microsecs
Event Loop.. select returns 1
............ READ :
............ WRITE: 4
............ OOB  :
............ Timeout is 20 s, 0 microsecs
Memory Free. 0x80ac700
Timer....... Found timer 0x80dff78 with callback 0x8050cec, context 
0x80c7e48, and relative timeout 20000
EventOrder.. execute ordered events
Memory Free. 0x80ac9b0
EventList... calling socket 4, request 0x80c4ef0 handler 0x8079608 type 
HTEvent_WRITE
Host Event.. WRITE passed to 
`http://www.uni4me.com/uni4me/what-is-uni/index.cfm#'
HTDoConnect. WOULD BLOCK `www.uni4me.com'
Memory Free. 0x80ab740
Event Loop.. calling select: maxfds is 4
............ READ :
............ WRITE: 4
............ OOB  :
............ Timeout is 20 s, 0 microsecs
Event Loop.. select returns 1
............ READ :
............ WRITE: 4
............ OOB  :
............ Timeout is 20 s, 0 microsecs
Memory Free. 0x80adae8
Timer....... Found timer 0x80dff78 with callback 0x8050cec, context 
0x80c7e48, and relative timeout 20000
EventOrder.. execute ordered events
Memory Free. 0x80c51e0
EventList... calling socket 4, request 0x80c4ef0 handler 0x8079608 type 
HTEvent_WRITE
Host Event.. WRITE passed to 
`http://www.uni4me.com/uni4me/what-is-uni/index.cfm#'
HTDoConnect. WOULD BLOCK `www.uni4me.com'
Received on Tuesday, 2 March 2004 12:12:25 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 27 October 2010 18:14:28 GMT