W3C home > Mailing lists > Public > www-tag@w3.org > May 2011

Changing the meaning of Robots.txt

From: Karl Dubost <karld@opera.com>
Date: Tue, 10 May 2011 10:13:13 -0400
Message-Id: <5DC435DC-E438-4FCB-AB11-EC632E8C5BE6@opera.com>
To: "www-tag@w3.org List" <www-tag@w3.org>
On Archive.org wiki, there was a Web page about robots.txt [1] created by JSCott [2]

    ROBOTS.TXT is a stupid, silly idea in the modern era.
    Archive Team entirely ignores it and with precisely one
    exception, everyone else should too.

    If you do not know what ROBOTS.TXT is and you run a
    site... excellent. If you do know what it is and you
    have one, delete it. Regardless, Archive Team will
    ignore it and we'll delete your complaints, just like
    you should be deleting ROBOTS.TXT.

My big issue with the rant is not necessary the will to kill robots.txt but more exactly to change the expectation that people had with regards to this protocol. I expressed some of my concerns in the Discuss page [3] of the wiki.

[1]: http://www.archiveteam.org/index.php?title=Robots.txt
[2]: http://www.archiveteam.org/index.php?title=User:Jscott
[3]: http://www.archiveteam.org/index.php?title=Talk:Robots.txt

Karl Dubost - http://dev.opera.com/
Developer Relations & Tools, Opera Software
Received on Tuesday, 10 May 2011 14:13:45 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 7 January 2015 15:33:10 UTC