- From: Dan Brickley <danbri@w3.org>
- Date: Mon, 17 Nov 2003 23:23:37 -0500
- To: www-talk@w3.org
Fwd'ed from the syndication list, where interest is growing w.r.t. successors to robots.txt in its current form. Thought might useful to folk here. danbri ----- Forwarded message from Ian Graham <ian.graham@utoronto.ca> ----- From: Ian Graham <ian.graham@utoronto.ca> Date: Sun, 16 Nov 2003 13:37:39 -0500 (EST) To: syndication@yahoogroups.com Subject: Re: [syndication] Compromise time.... Message-ID: <Pine.SOL.4.21.0311161231590.2664-100000@ic-unix.ic.utoronto.ca> Reply-To: syndication@yahoogroups.com So, back from holidays. I spend some time digging around trying to find a good summary of historical proposals for robots.txt, but didn't find anything. Here are a few links that cover off some proposals (you'll have to follow the followups yourself): http://www.robotstxt.org/wc/mailing-list/1460.html http://customultraseek.org/server-sitelist.html (sitelist proposal) http://www.utoronto.ca/ian/docs/Indexing/server.html (server resource d/b) http://www.robotstxt.org/wc/mailing-list/0194.html (inter-robot communication) http://www.robotstxt.org/wc/mailing-list/0199.html (inter-robot comm II) http://www.robotstxt.org/wc/mailing-list/0235.html (extensions proposal) http://www.robotstxt.org/wc/mailing-list/0322.html (preferred site domain name) http://www.robotstxt.org/wc/mailing-list/0752.html (instructions for robots) http://www.robotstxt.org/wc/mailing-list/1494.html (proposed features) [led to later discussion http://www.robotstxt.org/wc/mailing-list/1460.html which referenced teh following: http://customultraseek.org/server-sitelist.html (sitelist proposal) http://www.utoronto.ca/ian/docs/Indexing/server.html (server resource d/b) ] Basic conclusion: robots.txt not worth changing, all other issues are really about providing metadata about site contents, and this doesn't belong in robots.txt (but no one figured out where it should go...) Ian On Fri, 17 Oct 2003, Ian Graham wrote: > > On Fri, 17 Oct 2003, Bill Kearney wrote: > > > From: "Ian Graham" <ian.graham@utoronto.ca> > > > http://www.robotstxt.org/wc/mailing-list/ > > > there is some discussion on additional features for robots files -- and > > > some pretty good arguments as to why those things weren't added ;-) > > > > Ian, can you help us out here and cite which articles focus on pro/con > > positions? > > > > -Bill Kearney > > > > I can't recall all of them (memory being what it once was). I know I > proposed adding field to indicate when domain names have been moved, or > expired (sort of like a mega redirect), but that was felt to solve only > part of a larger cataloging problem. > > I'm going to be off for 2 weeks, and away from all things digital, but > I'll put this on my to-do list for when I get back. > > Ian > > > > > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ > > > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ ----- End forwarded message -----
Received on Monday, 17 November 2003 23:26:17 UTC