W3C home > Mailing lists > Public > www-lib@w3.org > July to September 1998

Re: Feeding the webbot with a list of addresses to check

From: Bob Racko <bobr@dprc.net>
Date: Tue, 29 Sep 1998 21:57:56 -0400
Message-Id: <>
To: www-lib@w3.org
Cc: "Lola Garcia Santiago" <mdolo@platon.ugr.es>
mostly when I want to have the robot check
multiple URI's I run it multiple times
from a script or bat file.

I suppose we can change the interpretation
of the second and subsequent arguments to mean
additional URI's. Right now they mean keywords to search for.

I personally would prefer (yet another) command line
option along the lines of -addURI *
which would add the named link to the list of initial documents to load.
(The same as finding an <A href="*" > in a document except this link
would name an alternate root or starting-point).

This begs the issue of return-status. What does it mean to
try to fetch one or more documents and they fail to fetch
(or fail to parse, or fail to... ) ?

When only one document-root is given (as it is now)
then the return status is wether the root and all children
came through ok. [at least that is what _I_ find most useful
when building web spiders]

Would it be better to return "OK" if ALL of them fetch and
parse [AND] -or ok if any one of them comes out ok [OR]?

For instance, I have a growing list of web-sites and pages within sites
that I want to monitor to see when they come online.
So I want to construct a robot that tells me when
any one of them is "visible". (ping is not enough)
I also want a robot that tells me when all of them are healthy.

The inverse is also useful. Suppose I want to know
when a page I am interested in is no longer available?

At 06:09 AM 9/29/98 -0400, you wrote:
>At 08:10 9/26/98 -0400, Lola Garcia Santiago wrote:
>>Dear Sirs,	
>>	I've looked some informations about Webbot but I'm interested to know if
>>Webbot works with one URI but I don't know if it
>>works using a pre-generated list of URIs. In case Webbot couldn't use that
>>list, do you know any robot or mapper that can check websites using a
>>previous list of URIs?.
>>		Thank you for your answer,
>>Lola Garcia Santiago
>>Doctoral Student
>>Universidad de Granada
>>e-mail: mdolo@platon.ugr.es 
>This actually sounds like a good idea. Bob Racko and John Punin are doing
>some wonderful work on the webbot now but I think we should put this on the
>wish list.
>What do you think guys?
>Henrik Frystyk Nielsen,
>World Wide Web Consortium

Received on Tuesday, 29 September 1998 22:10:22 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:33:48 UTC