Re: Structured request headers deployment issues

On Fri, Jun 19, 2020 at 7:13 AM Mark Nottingham <mnot@mnot.net> wrote:

> > On 16 Jun 2020, at 8:15 am, Yoav Weiss <yoav@yoav.ws> wrote:
> >
> > Hey all,
> >
> > Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent
> Client Hints enabled by default, which is using Structured Headers.
> >
> > As a result of that, we found multiple sites which seem to have a
> somewhat allergic reaction to the presence of certain characters (that are
> part of the SH format) in request values.
> > While each site in question is different (in what appears to be coming
> from different stacks), we've seen sites that reject requests with quotes,
> question marks or equals signs in them.
> > It's still early, so it's hard to know how widespread the issue is, but
> we seem to be adding sites to the list at a faster pace than the pace of
> removing fixed ones from it.
>

Yoav, have we experimented at all with alternate spellings? Do
single-quotes have the same impact as double-quotes, for example? Are other
(uglier) non-alphanumeric delimiters less likely to cause trouble? There's
a limited appetite for breakage, and it might be worth exploring alternate
aesthetics rather than forcing through the particular spelling we like the
most.

> So, I wanted to give this group a heads-up on that front, and maybe get
> folks' opinions regarding possible things we could do on that front, other
> than outreach and waiting for said sites to fix themselves.
>
> AIUI these aren't new; e.g., IIRC quite a few months ago Chrome
> encountered several Austrian sites that had this problem, traced back to a
> local(?) WAF vendor there. I believe that's been corrected since, after
> reaching out to them.
>

I think you might be referring to some conversations around the
(since-renamed) `Sec-Metadata` header in 2018, like those captured in
https://bugs.chromium.org/p/chromium/issues/detail?id=861678#c11.

Personally, I think that outreach and waiting is the right approach; if
> browsers consistently send these headers, they'll adapt, and the numbers
> are still relatively small -- or at least small enough that it's not likely
> the numbers will be reduced if the syntax is changed (due to _other_ WAFs'
> opinions about what a "good" request is).
>

The flip-side of this is that we break sites users rely upon. That's tough
to do at scale. In this case in particular, I think I agree with the
underlying assertion that we'd like this syntax to work, and it seems
reasonable to me to roll it out to a small percentage of stable users. That
said, if we end up breaking the internet for even a small percentage of
users, it doesn't seem like a good idea to hold them hostage in the hopes
of increasing pressure on WAF vendors.

My expectation is that we'd roll this out to a small percentage of stable,
see a spike in bug reports (especially from enterprise folks who are likely
to have WAF deployments, unlikely to broadly deploy beta channels, and
unlikely to choose to enable usage statistics or histograms), see a
corresponding spike in error codes for top-level navigations, and roll it
back.

I think it might be reasonable to explore alternate spellings at the same
time, perhaps with some A/B testing to evaluate how well each weaves its
way through middleware.

Related, we're also seeing more examples WAFs limiting how we can evolve
> the protocol (e.g., <https://github.com/coreruleset/coreruleset/pull/1777>).
> There's been a bit of background chatter about writing something about this
> and creating better communication with that community; I'm not sure what
> that will look like yet, but if anyone has ideas or is interested, please
> say so.
>

I am interested!

-mike