Proposal to advertise UA automation


I'd like to discuss benefits of advertising user agent automation. I
started the topic at webappsec (,
but this mailing list seems to be more suitable for the topic.

The idea is to attach an HTTP request header to navigation requests that
are initiated by automation tools, by which I mean headless browsers, web
driver driven browsers, etc.

The benefit for the webste operator is to have a choice in responding to
such requests differently. For example, do not serve ads, suggest using API
scraping rather that loading heavy resources, send through failed CAPTCHA
route, etc.

This approach intersects with robots.txt a little, but none of modern UA
automation tools honor robots.txt, and implementing the advertising flag
seems to be relatively easy.

Sergey Shekyan

Received on Wednesday, 18 January 2017 05:13:25 UTC