W3C home > Mailing lists > Public > public-webappsec@w3.org > January 2017

Re: Proposal to advertise automation of UA

From: Mike West <mkwst@google.com>
Date: Tue, 17 Jan 2017 13:16:53 +0100
Message-ID: <CAKXHy=cR5QSkb=GF-HifPoFow1ndE0T+jF2_3q_TQufWaR-u+w@mail.gmail.com>
To: Jonathan Garbee <jonathan.garbee@gmail.com>
Cc: Sergey Shekyan <shekyan@gmail.com>, Daniel Veditz <dveditz@mozilla.com>, "public-webappsec@w3.org" <public-webappsec@w3.org>
Hey Sergey,

If your goal is to reduce malicious traffic on a website, why would you
expect the malicious-traffic generator to opt-into sending a header
advertising their automated nature? Doesn't this in some way boil down
to setting
the evil bit <https://www.ietf.org/rfc/rfc3514.txt>?


On Tue, Jan 17, 2017 at 8:18 AM, Jonathan Garbee <jonathan.garbee@gmail.com>

> I'm what way should they respond differently? The site has absolutely no
> context as to why headless is being used. Why mangle the response without
> any context and just hope your users still get benefit from it?
> On Mon, Jan 16, 2017, 4:47 PM Sergey Shekyan <shekyan@gmail.com> wrote:
>> robots.txt is either is an on/off switch, while what I propose is more
>> granular, allowing websites to chose how to respond.
>> On Sat, Jan 14, 2017 at 5:52 AM, Jonathan Garbee <
>> jonathan.garbee@gmail.com> wrote:
>> I don't see where having a header or something to help detect automated
>> access will be beneficial. We can already automate browser engines.
>> Headless mode is just a native way to do it. So, if someone is already not
>> taking your robots.txt into account, they'll just use another method or
>> strip whatever we add to say headless mode is in use out. Sites don't gain
>> any true benefit from having this kind of detection. If someone wants to
>> automate tasks they do regularly, that's their prerogative. We have
>> robots.txt as a respectful way to ask people automating things to avoid
>> certain areas and actions, that easily continues into headless mode.
>> On Sat, Jan 14, 2017, 4:28 AM Sergey Shekyan <shekyan@gmail.com> wrote:
>> I am talking about tools that automate user agents, e.g. headless
>> browsers (PhantomJS, SlimerJS, headless Chrome), Selenium, curl, etc.
>> I mentioned navigation requests as don't see so far how advertising
>> automation to non-navigation requests would help.
>> Another option to advertise can be a property on navigator object, which
>> would defer possible actions by authors to second request.
>> On Sat, Jan 14, 2017 at 12:56 AM, Daniel Veditz <dveditz@mozilla.com>
>> wrote:
>> On Fri, Jan 13, 2017 at 5:11 PM, Sergey Shekyan <shekyan@gmail.com>
>> wrote:
>> I think that attaching a HTTP request header to synthetically initiated
>> navigation requests (https://fetch.spec.whatwg.org/#navigation-request)
>> will help authors to build more reliable mechanisms to detect unwanted
>> automation.
>> ​I don't see anything in that spec about "synthetic" navigation requests.
>> Where would you define that? How would you define that? Is a scripted
>> window.open() in a browser "synthetic"? what about an iframe in a page?
>> Does it matter if the user expected the iframe to be there or not (such as
>> ads)? What if the page had 100 iframes?
>> Are you trying to solve the same problem robots.txt is trying to solve?
>> If not what kind of automation are you talking about?​
>> -
>> ​Dan Veditz​
Received on Tuesday, 17 January 2017 12:17:47 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 18:54:59 UTC