Re: Prague side meeting: HTTP/2 concurrency and request cancellation (CVE-2023-44487)

Hi Kazuho,

On Fri, Oct 13, 2023 at 08:05:08AM +0900, Kazuho Oku wrote:
> > Not if you combine it with settings like I suggested earlier in the thread:
> >   1) send SETTINGS with MAX_CONCURRENT_STREAMS=100
> >   2) send MAX_STREAMS announcing 100 streams (e.g. max_stream_id=201)
> >   3) send SETTINGS with MAX_CONCURRENT_STREAMS=10 (or even less)
> >
> > At this point if you get a new stream with an ID higher than the
> > MAX_STREAMS you advertised and the total is above MAX_CONCURRENT_STREAMS,
> > you know for certain it's an abuse.
> >
> 
> Is that so?
> 
> I could very well have lost my memory, but the initial value of
> MAX_CONCURRENT_STREAM is unlimited. The client is allowed to initiate
> requests before it receives the first SETTINGS frame. It is designed as
> such to allow clients to send requests in 0-RTT. Advertised limit is
> applied only when the client receives the first SETTINGS frame

Absolutely but clients also know that if they emit more streams than
what the server supports, they'll have to deal with the situation
themselves. That's a pretty annoying design, of course, but it stems
from nobody being able to agree on the number of initial streams in
the past. 100 is far too much for some servers or their developers,
and less is not enough for clients. Thus the current approach is
"send as much as you want and deal with rejects" for the client, and
"accept only as much as you want" for the server. The gray area between
the two will only lead to series of RST_STREAM(REFUSED_STREAM).

> Therefore, if we want to define an extension that has a hard limit on the
> number of concurrent requests that a client can issue, I think we should do
> something like:
> * state in the extension that clients implementing the extension MUST NOT
> initiate more than 100 requests until it receives a SETTINGS frame, and

Not needed. The recommendation is sufficient to save them from having
to deal with refused streams most of the time.

> * negotiate the use of MAX_STREAMS frames using SETTINGS.

I'm not seeing this as necessary. However if advertised by the client
it could allow the server to decide on a different limit (low limit
for old clients, higher limit for new clients based on MAX_STREAMS).

> If we take this approach, there will be a guarantee that the client will
> open no more than 100 streams initially, and that the new credits will
> become available only by the server taking action.

It's still not a guarantee because not all existing clients will be
updated to reflect the new spec. And the server is already allowed to
kill excess streams.

> But even with something like MAX_STREAMS, an attacker can issue requests at
> a very high rate. It is not bound by RTT, because a server will be ready to
> accept additional requests as soon as it sends a MAX_STREAMS frame, rather
> than when MAX_STREAMS is acked.
> 
> Therefore, assuming that the server is configured to allow 100 concurrent
> requests on the connection, an attacker can mount like 100 requests every
> 100 microseconds, assuming that it takes 100 microseconds for a server to
> process 100 requests just to cancel and send MAX_STREAMS frame.
>
> This duration (100 microseconds) depends on the server load. So, as the
> load increases, the attacks would start to fail, reducing the efficiency of
> the attack.

Yes and that's the goal, to give the control of new streams back to the
server. And in practice the problem is not a rate issue but a concurrency
issue. I.e. the server can refill MAX_STREAMS only once the resources of
pending streams are completely released.

> However, I would argue that it would still be an effective way to put load
> on the victim server even though it would not be considered as a
> vulnerability of the protocol or the server.

I'm not sure what you mean here, given that it's up to the server to
invite the client to send new requests.

> To paraphrase, the protection provided by MAX_STREAMS might not be adequate
> for real deployments.
> 
> We know that many existing servers throttle request concurrency separately
> from connection-level concurrency. As stated previously, my preference goes
> to emphasising the importance of having such a throttling scheme.

I totally agree with this last sentence. But MAX_STREAMS does give an
easy tool for the server to throttle clients.

I really like your proposal to let the client advertise support for
MAX_STREAMS, because we could encourage support and deployment on both
sides this way:

  - clients supporting MAX_STREAMS must advertise it in SETTINGS
  - servers supporting MAX_STREAMS should advertise and enforce a very
    low MAX_CONCURRENT_STREAMS limit with clients that do not support
    it (something between 1 and 10) in order to encourage clients to
    adopt it for better performance
  - servers should then use MAX_STREAMS to invite the client to send
    new requests based on what their resources allow them to process.

> > Servers can (and should) use their *real* stream count and not just the
> > apparent one at the protocol level. Apache, Nginx and Haproxy all have
> > their own variants of this and cope well with this situation.
> >
> 
> FWIW, h2o also has this kind of request-level throttling.

Great!

> The only problem with h2o was that there is a way of cancelling requests
> through this throttling scheme. So when using h2o as a h2 to h1 proxy, when
> facing a Rapid Reset attack, h2o would issue many socket(2) and connect(2)
> syscalls immediately followed by close(2).

I see. That's definitely among the difficulties that H2 brings in such
setups, and HTTP/1 is never far away for gateways like ours, with its
limitations imposed by TCP :-/

Willy

Received on Saturday, 14 October 2023 15:36:16 UTC