W3C home > Mailing lists > Public > www-talk@w3.org > November to December 1999

Informing robots that site has changed significantly?

From: Conan Callen <ccallen@windowpane.com>
Date: Wed, 15 Dec 1999 20:00:49 -0500 (EST)
Message-ID: <00ea01bf4760$c5e9a740$3a8314d1@nwlink.com>
To: <www-talk@www.cern.ch>
Cc: <m.koster@webcrawler.com>

I was wondering if there is a way to tell a robot to drop everything it knows about your site and reindex the whole thing? We have recently reworked our whole site and now robots have been requesting pages that are no longer there. It would be real convient if there was a reindex directive that could be place into the robots.txt that would let the robot know that significant changes have occured to this site and it would probably be best to start over from the top (especially if many new pages have been added, and old ones moved or deleted). 

Im not on this mailing list, if there are any replies please cc my email - ccallen@windowpane.com

Received on Thursday, 16 December 1999 09:03:44 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:33:02 UTC