W3C home > Mailing lists > Public > www-lib@w3.org > October to December 1998

Major update to the libwww webbot: robots.txt parser and BFS algorithm

From: Henrik Frystyk Nielsen <frystyk@w3.org>
Date: Mon, 26 Oct 1998 17:50:33 -0500
Message-Id: <3.0.5.32.19981026175033.00a84ba0@localhost>
To: www-lib@w3.org
Cc: puninj@cs.rpi.edu, bobr@dprc.net

Thanks to John Punin we now have a full robots.txt file and a breadth first
search algorithm in the webbot! Many thanks to both John Punin and Bob
Racko for handling this very elegantly without me being the loop!

The webbot is as usual at

	http://www.w3.org/Robot/

and the new code is committed to CVS and you can try it out at

	http://www.w3.org/Library/cvs.html#update

Have fun!

Henrik
--
Henrik Frystyk Nielsen,
World Wide Web Consortium
http://www.w3.org/People/Frystyk
Received on Monday, 26 October 1998 17:50:36 EST

This archive was generated by hypermail pre-2.1.9 : Wednesday, 3 September 2003 17:59:20 EDT