W3C home > Mailing lists > Public > www-lib@w3.org > January to March 2000

webbot problem

From: Amir Michail <amir@cs.washington.edu>
Date: Sun, 23 Jan 2000 22:38:44 -0800
To: www-lib@w3.org
Message-Id: <00012322433406.03786@fester.cs.washington.edu>

I tried to use webbot to crawl our dept. web site but for some
reason, some urls give error code 1 in the clf log and are not followed.

These are correct urls that work fine in any browser.

Moreover, if I crawl closer to the url (e.g., not the entrance of the site),
there are no problems.

I tried increasing the crawl depth significantly and this seems to have make no
difference.  I also have redirection turned on.

I am using version 5.2.9 compiled for Linux.


P.S.  Here is the command invocation:

${ROBOT} ${FLAGS} -nopipe -redir -ss -n -depth 5 \
-exclude "\?|/hype/|/mailing-list/|/archive/" \
-check "\.wav$|\.dvi$|\.tex$|\.au$|\.eps$|\.mp3$|\.mov$|\.qt$|\.gz$|\.tar$|\.tgz$|\.Z$|\.zip$|\.ZIP$|\.exe$|\.EXE$|\.ps$|\.doc$|\.pdf$|\.xplot$|\.java$|\.c$|\.h$|\.txt$|\.ppt$|\.gif$|\.GIF$|\.tiff$|\.png$|\.PNG$|\.jpeg$|\.jpg$|\.JPE$" \
-prefix ${ROOT} \
-img -imgprefix ${IMGROOT} \
-l ${LOG}-log-clf.txt \
-referer ${LOG}-log-referer.txt \
-404 ${LOG}-log-notfound.txt \
-reject ${LOG}-log-reject.txt \
Received on Monday, 24 January 2000 01:43:36 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:33:52 UTC