- From: Willy Tarreau <w@1wt.eu>
- Date: Wed, 11 Oct 2023 11:42:54 +0200
- To: Mark Nottingham <mnot@mnot.net>
- Cc: HTTP Working Group <ietf-http-wg@w3.org>
Hi Mark, On Wed, Oct 11, 2023 at 10:45:13AM +1100, Mark Nottingham wrote: > Martin has already drafted one proposal: > https://martinthomson.github.io/h2-stream-limits/draft-thomson-httpbis-h2-stream-limits.html I find that there are interesting points here. Yesterday my coworker Amaury who works on H3 explained to me how QUIC streams are flow- controlled and I found the principle much safer. I thought we could slightly emulate it by sending lots of SETTINGS frames and keeping track of which one advertised what so that we know what the client ACKs but that would be totally ugly and complicated. I do like Martin's idea above however. The only thing is that it will not protect servers against older clients, hence attackers would just act like an older client. And if for safety we decide to start lower with a small MAX_CONCURRENT_STREAMS then it would penalize modern clients during the first round trip. Or maybe we'd advertise both a low value for MAX_CONCURRENT_STREAMS for legacy clients immediately followed by a MAX_STREAMS frame advertising a larger one ? If so we still need to make sure that it will not cause massive RSTs from clients between the two :-/ Or maybe something like this could work: MAX_CONCURRENT_STREAMS = 100 MAX_STREAMS = 100 MAX_CONCURRENT_STREAMS = 10 Older clients would learn 100 then 10, possibly dropping excess streams, while new clients would learn MAX_STREAMS=100 and from that point ignore MAX_CONCURRENT_STREAMS=10. > Other discussions might touch on whether there are other measures (protocol > mechanisms or otherwise) that clients and servers can take, how concurrency > is exposed to calling code, and whether we can give better guidance about how > concurrency violations should be handled. A few of these were discussed already in the thread below opened by Cory 4 years ago, where it was discussed how to count streams vs limit, and where I even mentioned this exact method of attack consisting in sending HEADERS followed by RST_STREAM that would not change the total stream count from the protocol perspective: https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar/0131.html We already faced that issue long ago when multiple haproxy instances were stacked on top of each other over H2 and too short timeouts on the front would cause long series of HEADERS+RST_STREAM on the back before the request had a chance to be processed due to the second layer being configured with a nice value making it slower than the first one. What we've been doing in haproxy against this is that instead of counting the streams at the protocol level, we count attached ones at the application layer: these are created at the same moment, but they're released once they're aware of the close. And we stop processing new streams once the configured limit is passed. This means that for a limit of 100 streams for example, if we receive 100 HEADERS and their 100 respective RST_STREAM, it will indeed create 100 streams that are immediately orphaned (and closed from an H2 perspective), but the 101th HEADERS frame will interrupt processing until some of these streams are effectively closed and freed (and not just at the protocol layer). And from what I've read below from Maxim Dounin, it seems like nginx applies a very similar strategy (they use x2 margin instead of +1 but the principle is the same, let streams finish first): https://mailman.nginx.org/pipermail/nginx-devel/2023-October/S36Q5HBXR7CAIMPLLPRSSSYR4PCMWILK.html As such, I suspect that these approaches might be much more common than what sensationalist mass media want us to believe and that the issue is more a matter of implementation choices (i.e. resource management) than of the protocol. We know that the protocol has some deficiencies that make naive implementations easily attackable (this one, setting stream windows of 1 byte during transfers, long series of 1-byte HEADERS+CONTINUATION, various forms of HOL blocking such as zero-windows, unlimited number of PUSH_PROMISE etc) but I think that once you care about your resource usage you have to make some reasonable compromises on all of these. With that said I'd love to plug that hole with an elegant mechanism involving just an optional new frame ;-) Cheers, Willy
Received on Wednesday, 11 October 2023 09:43:07 UTC