W3C home > Mailing lists > Public > www-lib@w3.org > January to March 2001

Webbot "Broken Pipe" problem

From: Q. Alex Zhao <azhao@cc.gatech.edu>
Date: Wed, 3 Jan 2001 08:48:38 -0500 (EST)
Message-Id: <200101031348.IAA26881@gvu2.cc.gatech.edu>
To: www-lib@w3.org
I built version 5.3.2 on Solaris 2.5.1 and tried to use webbot to index a
set of web servers. Somehow after about 10 to 30 seconds, webbot always got
a "Broken Pipe" signal and exited.

This is worse than the previous version -- previously I could get one
complete run out of three or four tries.

Could it be that reading from sockets in the webbot program didn't check
whether the socket is still valid before the read?

Am I the only person using webbot, or this problem only happens to me?

The command I used was nothing special:

	webbot -n -ss -cache -cache_size 48 -cacheroot my_dir	\
		-r webbot.conf -prefix http -depth 256	\
		-exclude '\.gz$|...' -img -check '\.gif$|...'	\
		-redir -referer my_result "$URL"

and the webbot.conf file just listed certain URLs I didn't want the program
to crawl.

Thanks in advance for any help.
= Q. Alex Zhao
  http://www.cc.gatech.edu/~qiang.a.zhao/
  mailto:aZhao@cc.gatech.edu voiceto:404-385-2447 faxto:404-385-1253
  Graphics, Visualization & Usability Center, Georgia Inst. of Tech.
Received on Wednesday, 3 January 2001 08:48:43 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 23 April 2007 18:18:38 GMT