W3C home > Mailing lists > Public > www-talk@w3.org > November to December 2003

Re: robots.txt proposals - handy summary

From: Yukihiko Yoshida <yukihiko@sfc.keio.ac.jp>
Date: Fri, 21 Nov 2003 21:05:38 +0900
Message-Id: <200311211205.hALC5c626629@ccz03.sfc.keio.ac.jp>
To: Dan Brickley <danbri@w3.org>
Cc: www-talk@w3.org, yukihiko@sfc.keio.ac.jp

Hello from TOKYO,

I am looking for web robot for beginners.
Is there any good program ?

I searched WebWalker program, but I could not find their web page.

Yours Sincerely,

Yukihiko YOSHIDA



> 
> Fwd'ed from the syndication list, where interest is growing w.r.t.
> successors to robots.txt in its current form. Thought might useful to 
> folk here.
> 
> danbri
> 
> ----- Forwarded message from Ian Graham <ian.graham@utoronto.ca> -----
> 
> From: Ian Graham <ian.graham@utoronto.ca>
> Date: Sun, 16 Nov 2003 13:37:39 -0500 (EST)
> To: syndication@yahoogroups.com
> Subject: Re: [syndication] Compromise time....
> Message-ID: <Pine.SOL.4.21.0311161231590.2664-100000@ic-unix.ic.utoronto.ca>
> Reply-To: syndication@yahoogroups.com
> 
> So, back from holidays. I spend some time digging around trying to find a
> good summary of historical proposals for robots.txt, but didn't find
> anything.   Here are a few links that cover off some proposals (you'll
> have to follow the followups yourself):
> 
> http://www.robotstxt.org/wc/mailing-list/1460.html 
> http://customultraseek.org/server-sitelist.html (sitelist proposal)
> http://www.utoronto.ca/ian/docs/Indexing/server.html (server resource d/b)
> http://www.robotstxt.org/wc/mailing-list/0194.html (inter-robot
> communication)
> http://www.robotstxt.org/wc/mailing-list/0199.html (inter-robot comm II)
> http://www.robotstxt.org/wc/mailing-list/0235.html (extensions proposal)
> http://www.robotstxt.org/wc/mailing-list/0322.html (preferred site
> domain name)
> http://www.robotstxt.org/wc/mailing-list/0752.html (instructions for
> robots)
> http://www.robotstxt.org/wc/mailing-list/1494.html (proposed features)
> [led to later discussion
> http://www.robotstxt.org/wc/mailing-list/1460.html 
> which referenced teh
> following:
> http://customultraseek.org/server-sitelist.html (sitelist proposal)
> http://www.utoronto.ca/ian/docs/Indexing/server.html (server resource
> d/b) ]
> 
> Basic conclusion: robots.txt not worth changing, all other issues are
> really about providing metadata about site contents, and this doesn't
> belong in robots.txt (but no one figured out where it should go...)
> 
> Ian
> 
> 
> On Fri, 17 Oct 2003, Ian Graham wrote:
> 
> > 
> > On Fri, 17 Oct 2003, Bill Kearney wrote:
> > 
> > > From: "Ian Graham" <ian.graham@utoronto.ca>
> > > > http://www.robotstxt.org/wc/mailing-list/
> > > > there is some discussion on additional features for robots files -- and
> > > > some pretty good arguments as to why those things weren't added ;-)
> > > 
> > > Ian, can you help us out here and cite which articles focus on pro/con
> > > positions?
> > > 
> > > -Bill Kearney
> > > 
> > 
> > I can't recall all of them (memory being what it once was). I know I
> > proposed adding field to indicate when domain names have been moved, or
> > expired (sort of like a mega redirect), but that was felt to solve only
> > part of a larger cataloging problem. 
> > 
> > I'm going to be off for 2 weeks, and away from all things digital, but
> > I'll put this on my to-do list for when I get back.
> > Ian
Received on Friday, 21 November 2003 07:15:28 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 27 October 2010 18:14:28 GMT