W3C home > Mailing lists > Public > www-talk@w3.org > November to December 2003

robots.txt proposals - handy summary

From: Dan Brickley <danbri@w3.org>
Date: Mon, 17 Nov 2003 23:23:37 -0500
To: www-talk@w3.org
Message-ID: <20031118042337.GA18941@w3.org>

Fwd'ed from the syndication list, where interest is growing w.r.t.
successors to robots.txt in its current form. Thought might useful to 
folk here.

danbri

----- Forwarded message from Ian Graham <ian.graham@utoronto.ca> -----

From: Ian Graham <ian.graham@utoronto.ca>
Date: Sun, 16 Nov 2003 13:37:39 -0500 (EST)
To: syndication@yahoogroups.com
Subject: Re: [syndication] Compromise time....
Message-ID: <Pine.SOL.4.21.0311161231590.2664-100000@ic-unix.ic.utoronto.ca>
Reply-To: syndication@yahoogroups.com

So, back from holidays. I spend some time digging around trying to find a
good summary of historical proposals for robots.txt, but didn't find
anything.   Here are a few links that cover off some proposals (you'll
have to follow the followups yourself):

http://www.robotstxt.org/wc/mailing-list/1460.html 
http://customultraseek.org/server-sitelist.html (sitelist proposal)
http://www.utoronto.ca/ian/docs/Indexing/server.html (server resource d/b)
http://www.robotstxt.org/wc/mailing-list/0194.html (inter-robot
communication)
http://www.robotstxt.org/wc/mailing-list/0199.html (inter-robot comm II)
http://www.robotstxt.org/wc/mailing-list/0235.html (extensions proposal)
http://www.robotstxt.org/wc/mailing-list/0322.html (preferred site
domain name)
http://www.robotstxt.org/wc/mailing-list/0752.html (instructions for
robots)
http://www.robotstxt.org/wc/mailing-list/1494.html (proposed features)
[led to later discussion
http://www.robotstxt.org/wc/mailing-list/1460.html 
which referenced teh
following:
http://customultraseek.org/server-sitelist.html (sitelist proposal)
http://www.utoronto.ca/ian/docs/Indexing/server.html (server resource
d/b) ]

Basic conclusion: robots.txt not worth changing, all other issues are
really about providing metadata about site contents, and this doesn't
belong in robots.txt (but no one figured out where it should go...)

Ian


On Fri, 17 Oct 2003, Ian Graham wrote:

> 
> On Fri, 17 Oct 2003, Bill Kearney wrote:
> 
> > From: "Ian Graham" <ian.graham@utoronto.ca>
> > > http://www.robotstxt.org/wc/mailing-list/
> > > there is some discussion on additional features for robots files -- and
> > > some pretty good arguments as to why those things weren't added ;-)
> > 
> > Ian, can you help us out here and cite which articles focus on pro/con
> > positions?
> > 
> > -Bill Kearney
> > 
> 
> I can't recall all of them (memory being what it once was). I know I
> proposed adding field to indicate when domain names have been moved, or
> expired (sort of like a mega redirect), but that was felt to solve only
> part of a larger cataloging problem. 
> 
> I'm going to be off for 2 weeks, and away from all things digital, but
> I'll put this on my to-do list for when I get back.
> 
> Ian
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 
> 
> 
> 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 


----- End forwarded message -----
Received on Monday, 17 November 2003 23:26:17 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 27 October 2010 18:14:28 GMT