Re: Proposal to advertise UA automation


I didn't see the thread on discourse, so I'll respond here.

One thing to keep in mind is that sites which actually do send difference content will need to also send a Vary header if the content is cacheable; otherwise, a cache on the same path could serve "headless" content to a browser, or vice versa. E.g., if a reverse proxy or CDN is being used.

On its own that's not a huge deal, but it will inflate the size of responses a bit; Vary needs to be sent on *all* responses for a resource that it applies to, including the "default" ones (i.e., responses to requests that don't have this header).

Aside from that, I wonder how many headless agents will actually use this, since their typical use is to get whatever the browser does, or as close to it as possible. 


> On 18 Jan 2017, at 4:12 pm, Sergey Shekyan <> wrote:
> Hi, 
> I'd like to discuss benefits of advertising user agent automation. I started the topic at webappsec (, but this mailing list seems to be more suitable for the topic. 
> The idea is to attach an HTTP request header to navigation requests that are initiated by automation tools, by which I mean headless browsers, web driver driven browsers, etc.
> The benefit for the webste operator is to have a choice in responding to such requests differently. For example, do not serve ads, suggest using API scraping rather that loading heavy resources, send through failed CAPTCHA route, etc.
> This approach intersects with robots.txt a little, but none of modern UA automation tools honor robots.txt, and implementing the advertising flag seems to be relatively easy.
> Thanks, 
> Sergey Shekyan

Mark Nottingham

Received on Thursday, 19 January 2017 04:01:50 UTC