Re: Proposal to advertise automation of UA

I'm what way should they respond differently? The site has absolutely no
context as to why headless is being used. Why mangle the response without
any context and just hope your users still get benefit from it?

On Mon, Jan 16, 2017, 4:47 PM Sergey Shekyan <shekyan@gmail.com> wrote:

> robots.txt is either is an on/off switch, while what I propose is more
> granular, allowing websites to chose how to respond.
>
>
> On Sat, Jan 14, 2017 at 5:52 AM, Jonathan Garbee <
> jonathan.garbee@gmail.com> wrote:
>
> I don't see where having a header or something to help detect automated
> access will be beneficial. We can already automate browser engines.
> Headless mode is just a native way to do it. So, if someone is already not
> taking your robots.txt into account, they'll just use another method or
> strip whatever we add to say headless mode is in use out. Sites don't gain
> any true benefit from having this kind of detection. If someone wants to
> automate tasks they do regularly, that's their prerogative. We have
> robots.txt as a respectful way to ask people automating things to avoid
> certain areas and actions, that easily continues into headless mode.
>
> On Sat, Jan 14, 2017, 4:28 AM Sergey Shekyan <shekyan@gmail.com> wrote:
>
> I am talking about tools that automate user agents, e.g. headless browsers
> (PhantomJS, SlimerJS, headless Chrome), Selenium, curl, etc.
> I mentioned navigation requests as don't see so far how advertising
> automation to non-navigation requests would help.
> Another option to advertise can be a property on navigator object, which
> would defer possible actions by authors to second request.
>
>
> On Sat, Jan 14, 2017 at 12:56 AM, Daniel Veditz <dveditz@mozilla.com>
> wrote:
>
> On Fri, Jan 13, 2017 at 5:11 PM, Sergey Shekyan <shekyan@gmail.com> wrote:
>
> I think that attaching a HTTP request header to synthetically initiated
> navigation requests (https://fetch.spec.whatwg.org/#navigation-request)
> will help authors to build more reliable mechanisms to detect unwanted
> automation.
>
>
> ​I don't see anything in that spec about "synthetic" navigation requests.
> Where would you define that? How would you define that? Is a scripted
> window.open() in a browser "synthetic"? what about an iframe in a page?
> Does it matter if the user expected the iframe to be there or not (such as
> ads)? What if the page had 100 iframes?
>
> Are you trying to solve the same problem robots.txt is trying to solve? If
> not what kind of automation are you talking about?​
>
> -
> ​Dan Veditz​
>
>
>
>