W3C home > Mailing lists > Public > www-lib@w3.org > October to December 1998

Major update to the libwww webbot: robots.txt parser and BFS algorithm

From: Henrik Frystyk Nielsen <frystyk@w3.org>
Date: Mon, 26 Oct 1998 17:50:33 -0500
Message-Id: <>
To: www-lib@w3.org
Cc: puninj@cs.rpi.edu, bobr@dprc.net

Thanks to John Punin we now have a full robots.txt file and a breadth first
search algorithm in the webbot! Many thanks to both John Punin and Bob
Racko for handling this very elegantly without me being the loop!

The webbot is as usual at


and the new code is committed to CVS and you can try it out at


Have fun!

Henrik Frystyk Nielsen,
World Wide Web Consortium
Received on Monday, 26 October 1998 17:50:36 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:33:48 UTC