- From: Henrik Frystyk Nielsen <frystyk@w3.org>
- Date: Mon, 26 Oct 1998 17:50:33 -0500
- To: www-lib@w3.org
- Cc: puninj@cs.rpi.edu, bobr@dprc.net
Thanks to John Punin we now have a full robots.txt file and a breadth first search algorithm in the webbot! Many thanks to both John Punin and Bob Racko for handling this very elegantly without me being the loop! The webbot is as usual at http://www.w3.org/Robot/ and the new code is committed to CVS and you can try it out at http://www.w3.org/Library/cvs.html#update Have fun! Henrik -- Henrik Frystyk Nielsen, World Wide Web Consortium http://www.w3.org/People/Frystyk
Received on Monday, 26 October 1998 17:50:36 UTC