- From: Susan Lesch <lesch@w3.org>
- Date: Fri, 22 Jul 2005 13:31:39 -0700
- To: donny@prnewsnow.com
- Cc: site-comments@w3.org, Olivier Thereaux <ot@w3.org>
Received on Friday, 22 July 2005 20:31:49 UTC
donny@prnewsnow.com wrote: > It came to my attention that w3 has placed a disallow tag in their > Robots.txt to prevent robots from seeing th Check pages. I believe that the file [1] excludes the W3C Link Checker [2], not other robots. For more information, the section on disallowing a single robot in this help file [3] may explain this better. [1] http://www.w3.org/robots.txt [2] http://validator.w3.org/checklink [3] http://www.robotstxt.org/wc/exclusion-admin.html Best wishes, -- Susan Lesch http://www.w3.org/People/Lesch/ mailto:lesch@w3.org tel:+1.858.483.4819 World Wide Web Consortium (W3C) http://www.w3.org/
Received on Friday, 22 July 2005 20:31:49 UTC