Re: Proposal to advertise UA automation

Hi,

I didn't see the thread on discourse, so I'll respond here.

One thing to keep in mind is that sites which actually do send difference content will need to also send a Vary header if the content is cacheable; otherwise, a cache on the same path could serve "headless" content to a browser, or vice versa. E.g., if a reverse proxy or CDN is being used.

On its own that's not a huge deal, but it will inflate the size of responses a bit; Vary needs to be sent on *all* responses for a resource that it applies to, including the "default" ones (i.e., responses to requests that don't have this header).

Aside from that, I wonder how many headless agents will actually use this, since their typical use is to get whatever the browser does, or as close to it as possible. 

Cheers,


> On 18 Jan 2017, at 4:12 pm, Sergey Shekyan <shekyan@gmail.com> wrote:
> 
> Hi, 
> 
> I'd like to discuss benefits of advertising user agent automation. I started the topic at webappsec (https://lists.w3.org/Archives/Public/public-webappsec/2017Jan/0004.html), but this mailing list seems to be more suitable for the topic. 
> 
> The idea is to attach an HTTP request header to navigation requests that are initiated by automation tools, by which I mean headless browsers, web driver driven browsers, etc.
> 
> The benefit for the webste operator is to have a choice in responding to such requests differently. For example, do not serve ads, suggest using API scraping rather that loading heavy resources, send through failed CAPTCHA route, etc.
> 
> This approach intersects with robots.txt a little, but none of modern UA automation tools honor robots.txt, and implementing the advertising flag seems to be relatively easy.
> 
> Thanks, 
> Sergey Shekyan
> 

--
Mark Nottingham    mnot@akamai.com    https://www.mnot.net/

Received on Thursday, 19 January 2017 04:01:50 UTC