W3C home > Mailing lists > Public > www-talk@w3.org > November to December 1999

Re: Informing robots that site has changed significantly?

From: Gerald Oskoboiny <gerald@w3.org>
Date: Thu, 16 Dec 1999 09:26:39 -0500
To: Conan Callen <ccallen@windowpane.com>
Cc: www-talk@w3.org
Message-ID: <19991216092639.H15123@w3.org>
On Wed, Dec 15, 1999 at 08:00:49PM -0500, Conan Callen wrote:
> I was wondering if there is a way to tell a robot to drop
> everything it knows about your site and reindex the whole
> thing? We have recently reworked our whole site and now robots
> have been requesting pages that are no longer there. It would
> be real convient if there was a reindex directive that could be
> place into the robots.txt that would let the robot know that
> significant changes have occured to this site and it would
> probably be best to start over from the top (especially if many
> new pages have been added, and old ones moved or deleted). 

Removing pages from a site is generally a bad idea, because even
if you manage to get it reindexed by various robots, there will
still be many people with direct links to your site from their
own sites or bookmarks, resulting in frustrated users. (Haven't
you ever been frustrated by broken links on the web?)

If you must move pages on your site, you should set up redirects
from the old locations to the new ones, so browsers can silently
take people to the new address, and robots can see that the page
has moved and index it at its new location.

I recommend reading:

    Jakob Nielsen's Alertbox for June 14, 1998:
    Fighting Linkrot
    http://www.useit.com/alertbox/980614.html

    Cool URIs don't change (Tim Berners-Lee)
    http://www.w3.org/Provider/Style/URI

Hope this helps,

-- 
Gerald Oskoboiny       <gerald@w3.org>  +1 617 253 2920
System Administrator   http://www.w3.org/People/Gerald/
World Wide Web Consortium (W3C)      http://www.w3.org/
Received on Thursday, 16 December 1999 09:26:41 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 27 October 2010 18:14:24 GMT