Re: Prague side meeting: HTTP/2 concurrency and request cancellation (CVE-2023-44487)

On Fri, Oct 13, 2023 at 12:30:04PM +0100, Lucas Pardue wrote:
> On Fri, 13 Oct 2023, 12:13 Poul-Henning Kamp, <phk@phk.freebsd.dk> wrote:
> 
> > --------
> > Kazuho Oku writes:
> >
> > > If we take this approach, there will be a guarantee that the client will
> > > open no more than 100 streams initially,
> >
> > Does any published data exist on how "100" relates to how many streams
> > real-life legit clients /actually/ open on a new H2 connection ?
> >
> 
> As outlined in the Cloudflare blog post[1], in response to the DoS traffic
> dropping the concurrency was something thst was trialled as a remediation.
> We quickly received reports about websites hit by this change. The best
> example I saw was something like example.com index.html loading an image
> gallery of well over 100 images. These were loaded via <img> and pointed to
> other.com. Chrome would parse the HTML, discover all the things to fetch,
> open a new connection, and then send 100 requests immediately before the
> server SETTINGS arrived.
> 
> So while the RFC says the limit is undefined until server SETTINGS arrives,
> my survey of clients shows that most don't wait and simply restrict
> themselves to 100 streams. A server that attempts any value lower either
> has to soak up the burst (I.e. soak up the raciness of H2) or reset the
> streams and then hope the client can retry them.
> 
> I have no doubts that certain page constructions could lead to greater peak
> concurrency if it were so permitted without the magic client clamp.
> 
> Cheers
> Lucas
> 
> [1] -
> https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/

The HTTP/2 spec optimized for 0-RTT requests but with no limit on
the number of requests in that initial client send.  HTTP/3 made a
better choice than 2^30.  If that is working well for HTTP/3, then I
hope we can change the HTTP/2 guidance from 100 to something else,
let the popular clients catch up, and then the server side can
implement stricter defenses against attacks such as request floods.

> A server that attempts any value lower either
> has to soak up the burst (I.e. soak up the raciness of H2) or reset
> the streams and then hope the client can retry them.

Unfortunately, there are sufficient number of javascript developers
(not all of them!) whose javascript code is not robust in the face of
REFUSED_STREAM, so hoping the client retries is not a good position.

If Chrome fires off 100 stream requests and some of those a refused,
then Chrome passes that on to javascript.  If the guidance were
something like 10 until server HTTP/2 SETTINGS frame was received,
then Chrome would handle queueing and the javascript app would be
better behaved without any change in the javascript app.  (If the
server HTTP/2 SETTINGS did not set SETTINGS_MAX_CONCURRENT_STREAMS,
then Chrome could go back to the current assumption of 100 once
server SETTINGS was received.)


Since lighttpd added HTTP/2 in 2020, lighttpd sends SETTINGS with
SETTINGS_MAX_CONCURRENT_STREAMS 8 following the HTTP/2 server preface.
Then, lighttpd has to somehow handle up to 2^30 requests sent before a
(well-behaved) client returns SETTINGS ackn.  In practice, this is 100,
and why the RFC guidance is important to note as having an effect!

The implementation detail that lighttpd uses to handle this initial load
is that lighttpd will defer processing additional HTTP/2 frames from the
client until already active requests are completed, as long as at least
one active request is in state HALF_CLOSED_REMOTE (has no request body).
(Otherwise the requests might block waiting for DATA frames which are
behind HEADERS frames for new streams.)  While deferring processing of
all HTTP/2 frames is less than ideal, it seems to work well enough in
practice, and once a well-behaved client sends SETTINGS ackn, the client
no longer opens streams that exceed the concurrency limit.  A malicious
client might not send SETTINGS ackn and might do other nefarious things.

Generally speaking, if a web page fires off 100 requests, those requests
are often for static files of reasonable size and that can be served
reasonably quickly, 8 at a time.  Not always.

(A more complex approach could be taken than deferring processing
additional HTTP/2 frames, e.g. a separate queue for deferred HEADERS
and CONTINUATION frames, which need to be processed in order to
maintain consistency in the HPACK decoder state.)

Apologies that it has taken so long to get to the point:

In almost three years, I have heard of one (1) user having an issue with
lighttpd enforcing SETTINGS_MAX_CONCURRENT_STREAMS 8 as described above,
and the issue was that lighttpd had to send REFUSED_STREAM to an
unmanaged flood of PUT requests (containing request bodies) sent from
his javascript application, and his javascript did not handle
REFUSED_STREAM returned by Chrome.

If popular browsers helped app developers queue the requests sent before
server SETTINGS is received, I am confident that the guidance for the
initial value of SETTINGS_MAX_CONCURRENT_STREAMS can safely be lowered.
After some time for deployment, a lower value would allow for servers to
better distinguish legitimate client request behavior from some attacks.

Cheers, Glenn

Received on Friday, 13 October 2023 12:50:12 UTC