Re: Structured request headers deployment issues

On Fri, Jun 19, 2020 at 10:38 AM Yoav Weiss <yoav@yoav.ws> wrote:

>
>
> On Fri, Jun 19, 2020 at 9:03 AM Mike West <mkwst@google.com> wrote:
>
>> On Fri, Jun 19, 2020 at 7:13 AM Mark Nottingham <mnot@mnot.net> wrote:
>>
>>> > On 16 Jun 2020, at 8:15 am, Yoav Weiss <yoav@yoav.ws> wrote:
>>> >
>>> > Hey all,
>>> >
>>> > Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent
>>> Client Hints enabled by default, which is using Structured Headers.
>>> >
>>> > As a result of that, we found multiple sites which seem to have a
>>> somewhat allergic reaction to the presence of certain characters (that are
>>> part of the SH format) in request values.
>>> > While each site in question is different (in what appears to be coming
>>> from different stacks), we've seen sites that reject requests with quotes,
>>> question marks or equals signs in them.
>>> > It's still early, so it's hard to know how widespread the issue is,
>>> but we seem to be adding sites to the list at a faster pace than the pace
>>> of removing fixed ones from it.
>>>
>>
>> Yoav, have we experimented at all with alternate spellings? Do
>> single-quotes have the same impact as double-quotes, for example? Are other
>> (uglier) non-alphanumeric delimiters less likely to cause trouble? There's
>> a limited appetite for breakage, and it might be worth exploring alternate
>> aesthetics rather than forcing through the particular spelling we like the
>> most.
>>
>
> We have not. But given the fact that we've seen issues with the use of
> `"`, `?` and `=` separately, I'm somewhat assuming that those separate
> implementations are using an allow-list approach.
> With that said, I can try to throw various values at those known-to-break
> servers and see if I can come up with an alternative spelling that would
> work with them. It won't give us any guarantees that these spellings would
> work at scale, but it might be a good starting point...
>

I played around with trying to find a safe sete of chars that would work as
delimiters on the set of sites we know are broken. I have failed
<https://bugs.chromium.org/p/chromium/issues/detail?id=1091285#c10> to
find such a subset :/


>
>>
>> > So, I wanted to give this group a heads-up on that front, and maybe get
>>> folks' opinions regarding possible things we could do on that front, other
>>> than outreach and waiting for said sites to fix themselves.
>>>
>>> AIUI these aren't new; e.g., IIRC quite a few months ago Chrome
>>> encountered several Austrian sites that had this problem, traced back to a
>>> local(?) WAF vendor there. I believe that's been corrected since, after
>>> reaching out to them.
>>>
>>
>> I think you might be referring to some conversations around the
>> (since-renamed) `Sec-Metadata` header in 2018, like those captured in
>> https://bugs.chromium.org/p/chromium/issues/detail?id=861678#c11.
>>
>> Personally, I think that outreach and waiting is the right approach; if
>>> browsers consistently send these headers, they'll adapt, and the numbers
>>> are still relatively small -- or at least small enough that it's not likely
>>> the numbers will be reduced if the syntax is changed (due to _other_ WAFs'
>>> opinions about what a "good" request is).
>>>
>>
>> The flip-side of this is that we break sites users rely upon. That's
>> tough to do at scale. In this case in particular, I think I agree with the
>> underlying assertion that we'd like this syntax to work, and it seems
>> reasonable to me to roll it out to a small percentage of stable users. That
>> said, if we end up breaking the internet for even a small percentage of
>> users, it doesn't seem like a good idea to hold them hostage in the hopes
>> of increasing pressure on WAF vendors.
>>
>> My expectation is that we'd roll this out to a small percentage of
>> stable, see a spike in bug reports (especially from enterprise folks who
>> are likely to have WAF deployments, unlikely to broadly deploy beta
>> channels, and unlikely to choose to enable usage statistics or histograms),
>> see a corresponding spike in error codes for top-level navigations, and
>> roll it back.
>>
>> I think it might be reasonable to explore alternate spellings at the same
>> time, perhaps with some A/B testing to evaluate how well each weaves its
>> way through middleware.
>>
>> Related, we're also seeing more examples WAFs limiting how we can evolve
>>> the protocol (e.g., <
>>> https://github.com/coreruleset/coreruleset/pull/1777>). There's been a
>>> bit of background chatter about writing something about this and creating
>>> better communication with that community; I'm not sure what that will look
>>> like yet, but if anyone has ideas or is interested, please say so.
>>>
>>
>> I am interested!
>>
>> -mike
>>
>