Changing the meaning of Robots.txt

On Archive.org wiki, there was a Web page about robots.txt [1] created by JSCott [2]
The title is  "ROBOTS.TXT IS A SUICIDE NOTE"

    ROBOTS.TXT is a stupid, silly idea in the modern era.
    Archive Team entirely ignores it and with precisely one
    exception, everyone else should too.

    If you do not know what ROBOTS.TXT is and you run a
    site... excellent. If you do know what it is and you
    have one, delete it. Regardless, Archive Team will
    ignore it and we'll delete your complaints, just like
    you should be deleting ROBOTS.TXT.

My big issue with the rant is not necessary the will to kill robots.txt but more exactly to change the expectation that people had with regards to this protocol. I expressed some of my concerns in the Discuss page [3] of the wiki.


[1]: http://www.archiveteam.org/index.php?title=Robots.txt
[2]: http://www.archiveteam.org/index.php?title=User:Jscott
[3]: http://www.archiveteam.org/index.php?title=Talk:Robots.txt

-- 
Karl Dubost - http://dev.opera.com/
Developer Relations & Tools, Opera Software

Received on Tuesday, 10 May 2011 14:13:45 UTC