W3C home > Mailing lists > Public > whatwg@whatwg.org > November 2011

[whatwg] Proposal: Exclude robots on a per-link basis

From: Kornel Lesiński <kornel@geekhood.net>
Date: Sat, 26 Nov 2011 14:26:42 -0000
Message-ID: <op.v5j42slote2ec8@aimac.local>
On Sat, 26 Nov 2011 12:20:28 -0000, Markus Ernst <derernst at gmx.ch> wrote:

> Viewing the logs of applications I wrote, I noticed that a considerable  
> number of requests are from robots following links of types such as "Add  
> to shopping cart" or "Remember this item" - links that typically point  
> to the same page they are clicked on, with some GET variable that  
> triggers an action on the server.

Actions that have significant side effects, like "Remember this item",  
should be performed using POST method, which well-behaved bots do not  

Exclusion of URLs based on on query string arguments could be done with  
wildcards in robots.txt (Googlebot supports this extension already):

Disallow: *add-item=*

regards, Kornel Lesi?ski
Received on Saturday, 26 November 2011 06:26:42 UTC

This archive was generated by hypermail 2.4.0 : Wednesday, 22 January 2020 16:59:38 UTC