- From: Yukihiko Yoshida <yukihiko@sfc.keio.ac.jp>
- Date: Fri, 21 Nov 2003 21:05:38 +0900
- To: Dan Brickley <danbri@w3.org>
- Cc: www-talk@w3.org, yukihiko@sfc.keio.ac.jp
Hello from TOKYO, I am looking for web robot for beginners. Is there any good program ? I searched WebWalker program, but I could not find their web page. Yours Sincerely, Yukihiko YOSHIDA > > Fwd'ed from the syndication list, where interest is growing w.r.t. > successors to robots.txt in its current form. Thought might useful to > folk here. > > danbri > > ----- Forwarded message from Ian Graham <ian.graham@utoronto.ca> ----- > > From: Ian Graham <ian.graham@utoronto.ca> > Date: Sun, 16 Nov 2003 13:37:39 -0500 (EST) > To: syndication@yahoogroups.com > Subject: Re: [syndication] Compromise time.... > Message-ID: <Pine.SOL.4.21.0311161231590.2664-100000@ic-unix.ic.utoronto.ca> > Reply-To: syndication@yahoogroups.com > > So, back from holidays. I spend some time digging around trying to find a > good summary of historical proposals for robots.txt, but didn't find > anything. Here are a few links that cover off some proposals (you'll > have to follow the followups yourself): > > http://www.robotstxt.org/wc/mailing-list/1460.html > http://customultraseek.org/server-sitelist.html (sitelist proposal) > http://www.utoronto.ca/ian/docs/Indexing/server.html (server resource d/b) > http://www.robotstxt.org/wc/mailing-list/0194.html (inter-robot > communication) > http://www.robotstxt.org/wc/mailing-list/0199.html (inter-robot comm II) > http://www.robotstxt.org/wc/mailing-list/0235.html (extensions proposal) > http://www.robotstxt.org/wc/mailing-list/0322.html (preferred site > domain name) > http://www.robotstxt.org/wc/mailing-list/0752.html (instructions for > robots) > http://www.robotstxt.org/wc/mailing-list/1494.html (proposed features) > [led to later discussion > http://www.robotstxt.org/wc/mailing-list/1460.html > which referenced teh > following: > http://customultraseek.org/server-sitelist.html (sitelist proposal) > http://www.utoronto.ca/ian/docs/Indexing/server.html (server resource > d/b) ] > > Basic conclusion: robots.txt not worth changing, all other issues are > really about providing metadata about site contents, and this doesn't > belong in robots.txt (but no one figured out where it should go...) > > Ian > > > On Fri, 17 Oct 2003, Ian Graham wrote: > > > > > On Fri, 17 Oct 2003, Bill Kearney wrote: > > > > > From: "Ian Graham" <ian.graham@utoronto.ca> > > > > http://www.robotstxt.org/wc/mailing-list/ > > > > there is some discussion on additional features for robots files -- and > > > > some pretty good arguments as to why those things weren't added ;-) > > > > > > Ian, can you help us out here and cite which articles focus on pro/con > > > positions? > > > > > > -Bill Kearney > > > > > > > I can't recall all of them (memory being what it once was). I know I > > proposed adding field to indicate when domain names have been moved, or > > expired (sort of like a mega redirect), but that was felt to solve only > > part of a larger cataloging problem. > > > > I'm going to be off for 2 weeks, and away from all things digital, but > > I'll put this on my to-do list for when I get back. > > Ian
Received on Friday, 21 November 2003 07:15:28 UTC