perl/modules/W3C/LinkChecker/docs checklink.html,1.15,1.16 linkchecker.css,1.1,1.2

Update of /sources/public/perl/modules/W3C/LinkChecker/docs
In directory hutz:/tmp/cvs-serv6539/docs

Modified Files:
	checklink.html linkchecker.css 
Log Message:
Add instructions how to allow us in /robots.txt, and include a link to it in the results.

Index: checklink.html
===================================================================
RCS file: /sources/public/perl/modules/W3C/LinkChecker/docs/checklink.html,v
retrieving revision 1.15
retrieving revision 1.16
diff -u -d -r1.15 -r1.16
--- checklink.html	11 Apr 2004 20:27:27 -0000	1.15
+++ checklink.html	20 Apr 2004 17:13:28 -0000	1.16
@@ -24,6 +24,7 @@
       <li><a href="#what">What it does</a></li>
       <li><a href="#online">Use it online</a></li>
       <li><a href="#install">Install it locally</a></li>
+      <li><a href="#bot">Robots exclusion</a></li>
       <li><a href="#csb">Comments, suggestions and bugs</a></li>
     </ul>
 
@@ -187,6 +188,30 @@
       for more information.
     </p>
 
+    <h2><a name="bot" id="bot">Robots exclusion</a></h2>
+
+    <p>
+      As of version 3.9.3, the link checker honors
+      <a href="http://www.robotstxt.org/wc/exclusion.html#robotstxt">robots exclusion rules</a>.  To place rules specific to the W3C Link Checker in
+      <code>/robots.txt</code> files, sites can use the
+      <code>W3C-checklink</code> user agent string.  For example, to allow
+      the link checker to access all documents on a server and to disallow
+      all other robots, one could use the following:
+    </p>
+
+    <pre>
+User-Agent: *
+Disallow: /
+
+User-Agent: W3C-checklink
+Disallow:
+</pre>
+
+    <p>
+      Note that <code>/robots.txt</code> rules affect only user agents
+      that honor it; it is not a generic method for access control.
+    </p>
+
     <h2><a name="csb" id="csb">Comments, suggestions and bugs</a></h2>
 
     <p>

Index: linkchecker.css
===================================================================
RCS file: /sources/public/perl/modules/W3C/LinkChecker/docs/linkchecker.css,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -d -r1.1 -r1.2
--- linkchecker.css	9 Apr 2004 11:36:17 -0000	1.1
+++ linkchecker.css	20 Apr 2004 17:13:28 -0000	1.2
@@ -52,6 +52,9 @@
   line-height:      100%;
   white-space:      pre;
 }
+pre {
+  padding-left: 2em;
+}
 
 a:link img, a:visited img {
   border-style:     none;

Received on Tuesday, 20 April 2004 13:13:33 UTC