Re: Proposal to advertise automation of UA

robots.txt is either is an on/off switch, while what I propose is more
granular, allowing websites to chose how to respond.


On Sat, Jan 14, 2017 at 5:52 AM, Jonathan Garbee <jonathan.garbee@gmail.com>
wrote:

> I don't see where having a header or something to help detect automated
> access will be beneficial. We can already automate browser engines.
> Headless mode is just a native way to do it. So, if someone is already not
> taking your robots.txt into account, they'll just use another method or
> strip whatever we add to say headless mode is in use out. Sites don't gain
> any true benefit from having this kind of detection. If someone wants to
> automate tasks they do regularly, that's their prerogative. We have
> robots.txt as a respectful way to ask people automating things to avoid
> certain areas and actions, that easily continues into headless mode.
>
> On Sat, Jan 14, 2017, 4:28 AM Sergey Shekyan <shekyan@gmail.com> wrote:
>
>> I am talking about tools that automate user agents, e.g. headless
>> browsers (PhantomJS, SlimerJS, headless Chrome), Selenium, curl, etc.
>> I mentioned navigation requests as don't see so far how advertising
>> automation to non-navigation requests would help.
>> Another option to advertise can be a property on navigator object, which
>> would defer possible actions by authors to second request.
>>
>>
>> On Sat, Jan 14, 2017 at 12:56 AM, Daniel Veditz <dveditz@mozilla.com>
>> wrote:
>>
>> On Fri, Jan 13, 2017 at 5:11 PM, Sergey Shekyan <shekyan@gmail.com>
>> wrote:
>>
>> I think that attaching a HTTP request header to synthetically initiated
>> navigation requests (https://fetch.spec.whatwg.org/#navigation-request)
>> will help authors to build more reliable mechanisms to detect unwanted
>> automation.
>>
>>
>> ​I don't see anything in that spec about "synthetic" navigation requests.
>> Where would you define that? How would you define that? Is a scripted
>> window.open() in a browser "synthetic"? what about an iframe in a page?
>> Does it matter if the user expected the iframe to be there or not (such as
>> ads)? What if the page had 100 iframes?
>>
>> Are you trying to solve the same problem robots.txt is trying to solve?
>> If not what kind of automation are you talking about?​
>>
>> -
>> ​Dan Veditz​
>>
>>
>>