RE: HTTP/2 re-sync connection level flow control?

I was initially mystified by how this would happen in HTTP/2.  The frames always arrive in the same order, so if they're all processed correctly, there should never be a disagreement about what the actual allowed amount is.  Presumably the disagreement is about which sent bytes count.

TL;DR of the bug you reference: an implementation bug was sending additional DATA frames on an already-closed stream.  Per RFC7540, that's prohibited, but only fatal to the stream.  This exposed a different bug, in which connection flow control updates were not sent to replenish the flow control consumed by incoming DATA frames once a RST_STREAM had been sent.  The merged fix in Go is to always return the relevant connection flow control credit along with the RST_STREAM.

You don't see it in HTTP/3 for a couple of reasons.  First, flow control (and therefore BLOCKED) are a transport-level function.  Because there's no fixed ordering, MAX_DATA frames (the equivalent of WINDOW_UPDATE for stream 0) always carry an absolute value, so implementations can't get out of sync about what the allowed window is -- or rather, the consumer will always lag behind the issuer due to transmission delay, which is always safe.  You also don't get out of sync on the number of bytes sent -- or at least, if you do, it's immediately obvious.  A RST_STREAM tells you how many bytes were sent on the stream (the "final size"), and subsequently exceeding the final size is a connection-fatal error.

-----Original Message-----
From: Graham King <> 
Sent: Wednesday, February 13, 2019 8:48 PM
Subject: HTTP/2 re-sync connection level flow control?

Given a range of HTTP/2 implementations with their bugs, edge cases, varied interpretations and understandings of the spec, not to mentioned network issues, the connection-level flow control values could get out of sync between client and server. 

Once client and server get out of sync, there is no mechanism for them to re-sync. The WINDOW_UPDATE values are always relative (increment). If the client reaches a window of 0 it will block. With enough deployments, and sufficiently long running connections, it seems almost inevitable that some will block because of this.

This happened in Go 1.11. DATA frames sent after a stream close weren't being counted towards flow control on the server side, but were on the client side (It is fixed in Go 1.12):

In QUIC (at least at some point, I couldn't find this in HTTP/3 spec) the sender sends a BLOCKED frame when it's window reaches 0. QUIC also includes a flow control value in RST frames to permit resync.

Should HTTP/2 include a means for client and server to resync absolute connection level flow control values?

Received on Thursday, 14 February 2019 18:23:41 UTC