Re: Prague side meeting: HTTP/2 concurrency and request cancellation (CVE-2023-44487)

Hi Roy,

On Sat, Oct 14, 2023 at 11:20:59AM -0700, Roy T. Fielding wrote:
> FTR, I don't see any need to change the protocol; just change the
> implementations that are vulnerable because their expectations are wrong
> about clients on the open Internet. That's why many HTTP/2 implementations
> were already prepared for this attack.

I don't see a *need* either, rather an opportunity to improve
communications between both ends when the server needs to regulate
the traffic.

At the moment, those who care about allocated resources have essentially
two choices when facing a HEADERS frame they're not willing to immediately
process:
  - reject it with RST_STREAM, but not all clients are able to retry,
    so in some cases it causes some errors

  - pause frame processing waiting for resources to get better, but it
    also means no more ability to process WINDOW_UPDATE frames, possibly
    causing a deadlock with some blocked streams we'd like to see finish.

FWIW in haproxy we're doing the second option above, which doesn't show
any problem in field with regular clients since clients do not massively
abort transfers with all streams being in use. In the case it would
happen, the streams and connection would just time out as if the client
disappeared or stopped reading by not updating windows anyway.

But having the ability for the server to invite the client to send new
streams would really be awesome. It would even allow to offer less
depending on the local load, something that cannot realistically be
done right now except by playing dirty games with SETTINGS, keeping
in mind that you can't reject streams anyway until the new SETTINGS
gets acked.

So that's why I do support this possible extension: offering more control
to the server, nothing less, nothing more. I don't count on this to suddenly
address all of the protocol's shortcomings and weaknesses that already
require special cases to resist against attacks.

> Speaking of which, that CVE is completely irresponsible. A CVE is supposed to
> list known vulnerabilities in released software,

Sadly, the days where CVEs had anything to do with security are long gone.
Nowadays they're essentially used to sell automated misleading reports
to clueless CISOs, to justify backports of important fixes in places
where there is a strict "no backport" policy, to force customers to apply
major upgrades and pay for a new version, to try to get fame quickly for
newbie reporters (hence the "Curriculum Vitae Enhancer" designation), and
even for marketing, as a cheap way to advertise your products. I can only
recommend this video from GregKH and the following podcast from two ex-
insiders who admit that at least 80% are fake:

   https://www.youtube.com/watch?v=HeeoTE9jLjM
   https://player.fm/series/series-1502626/episode-392-curl-and-the-calamity-of-cve

> not potential
> vulnerabilities in all implementations of a single protocol. Now we have
> security poodles from all over the world asking each and every HTTP project
> whether they have a fix for a vulnerability that they never had in the first
> place, all because the CVE authors prefer to blame the protocol instead of
> their own internet-facing implementations. Don't do that, especially not for
> a low severity DDoS load-based attack. It has created a DDoS of its own,
> killing time we have for all of our open source projects, and we don't scale
> like a server.

I totally agree on this and I anticipated it when I was first contacted
about it and refused to participate based on this! I was wrong on one
point, I expected to see a new name, with a logo and domain for the
issue and there was not any.

> The RFC cannot guess what is the appropriate number of max concurrent streams
> for a given interface because h2 might be used in both trusted and untrusted
> environments, with both custom and generic clients, and with a great deal of
> variance in server capabilities (memory and CPU). It is fair to say that any
> server should be capable of receiving 100 concurrent stream openings. That
> does not mean the server has to provide an equal service to those 100 open
> streams, nor does it say that a server has to ignore the reset streams count
> just because the client said so. The server is in control of the interface
> and is fully capable of adjusting its services regardless of the RFC. The
> server can make the choice of availability versus interoperability far better
> than we can.

Agreed, but the server has to process H2 frames in order and its control is
limited on what it wants to process or not. That's the essential problem
that Martin's draft can improve.

> The working group should prepare a detailed explanation for a Security
> Considerations section that describes how such attacks work and how they can
> be mitigated by reducing service allocations to misbehaving clients. There is
> no need to change the protocol itself, other than to acknowledge (once again)
> that anything specified by the protocol is subject to change by the server if
> it perceives the client to be an attack rather than an interoperable client.

There are plenty of issues caused by H2's multiplexing inside an in-order
protocol and I think by now we all know them (they were discussed to death
during the design anyway):
  - client can flood the server with HEADERS frames and even reset them.
  - one side can start many downloads and not refill WINDOW_UPDATES, leaving
    all streams pending with data in buffers. That's the same as the good
    old zero-window TCP attacks but multiplied by the number of streams
  - one side can send as many 0-byte CONTINUATION header as it wants, these
    are permitted by the protocol (not even restricted to only those having
    END_HEADERS flag)
  - even if 0-byte CONTINUATION without EH are rejected, sending 1-byte
    CONTINUATION is quite a pain as it forces the peer to reassemble
    these bytes.
  - header compression can have significant amplification factors and take
    considerable processing time: one byte per name:value is sufficient to
    replicate a large one; even when not pre-loading it, 0x10 is "accept-
    encoding: gzip, deflate" or about a 30x inflate factor.
  - some control frames (WINDOW_UPDATE, PRIORITY, RST_STREAM) can reference
    a stream and require a lookup on the peer, yet they're small and can be
    stuffed at a very high rate, forcing the peer to use a lot of CPU
  - client-chosen stream IDs can limit the choice of efficient indexing
    mechanisms on the server.
  - server never knows when the client received a complete response
    before closing and have to be careful not to destroy data in flight
    from the incoming frames that cause TCP RSTs.
  - SETTING are not necessarily effective until acked

And these are the few that come to my mind. These have to be dealt with
by implementations but it's very likely that not everyone is aware of
these. It's likely that having documented these explicitly like this by
then could have improved implementations but it could also have discouraged
some implementers from adopting that spec :-/

Cheers,
Willy

Received on Saturday, 14 October 2023 19:27:07 UTC