Re: HTTP/2 flow control <draft-ietf-httpbis-http2-17>

H2 does both connection-level and stream-level flow control. Both are
necessary to prevent infinite buffering if one assumes, as intended, that
the max number of streams should be very large.

The per-stream control is necessary
On Mar 19, 2015 9:01 AM, "Jason Greene" <jason.greene@redhat.com> wrote:

I think thats a good argument for why it’s not suitable for rate-limiting
relative to a variable bandwidth product of a single connection (which i
took as your use-case a). However, memory limits are typically constant,
and an HTTP/2 server can be configured to limit the capacity of the server
based on the combination of the “window” size, the number of streams
allowed, and the number of connections allowed. Without a hard data credit,
the server would have to either HOL block, prevent valid large requests
from processing, or run out of resources and start dropping requests.

On Mar 19, 2015, at 10:38 AM, Bob Briscoe <bob.briscoe@bt.com> wrote:

 Jason,

Yes, I agree this is what we /want/ per-stream flow control to do. My
review explained why the flow control in h2 won't be able to do this.

Specifically:

 ==Flow control problem summary==

With only a credit signal in the protocol, a receiver is going to have to
allow generous credit in the WINDOW_UPDATEs so as not to hurt performance.
But then, the receiver will not be able to quickly close down one stream
(e.g. when the user's focus changes), because it cannot claw back the
generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock
is.



Bob

At 15:08 19/03/2015, Jason Greene wrote:

Hi Bob,

I agree with you that HTTP/2 flow control is not useful for your a)
use-case, intermediate buffer control. It is, however, necessary for the b)
use case, memory management of an endpoint.

An HTTP server is often divided into a core set of components which manage
work scheduling and HTTP protocol aspects, as well as a set of
independantly developed  “application” components, that  are often, but not
necessarily,  provided by the end-user of the server. There is typically an
abstraction between application code and server code, where the application
is only ever aware of its stream. The stream may be provided by a single
HTTP/1.1 TCP socket, or it might be multiplexed over HTTP/2 or SPDY, or
some other protocol.  In all of these cases the abstraction appears the
same to the application, as it need not care. The application code
typically isn’t required (nor able) to read everything immediately;
instead, it is  allowed to read as it processes the data, and this
ultimately requires the server to buffer the data in the interim (since
other streams should not be blocked). HTTP/2 flow control allows the server
to limit the maximum memory usage of this buffering, which would otherwise
be significant since HTTP data can be of arbitrary length.

-Jason

On Mar 19, 2015, at 6:54 AM, Bob Briscoe <bob.briscoe@bt.com> wrote:

Patrick,

Thank you for taking the time to read my review carefully. I've been away
from mail a few days, which should have allowed time for anything
substantive to come from the people with SPDY experience.

We only had the Rob/Greg exchange which merely talked about what some
theoretical thing called flow control might be useful for, rather than
really addressing my concern that the credit-based protocol provided in h2
is unlikely to be able to provide control if it is needed. Nonetheless,
Greg did summarise well the limited usefulness of h2's mechanism.

I should probably have provided a summary of my (long) review, which I try
to do below.


I agree that there could be some use for the h2 credit-based protocol at
the very start of a stream (in both the C->S and S->C PUSH cases you
mentioned). And it might possibly be useful for the cases you mention about
unsolicited data, which sound like they occur at the start too. Altho
without knowing more about these DoS cases I'm not sure; RST_STREAM might
have been sufficient.

However, once a stream has been allowed to open up it's rate, your
observation that we mostly see very large windows is what I predicted. It
demonstrates that the credit-based protocol does not have sufficient
information to be useful to regulate flow. It is effectively being treated
like the human appendix - something that no longer serves any purpose but
you have to continually put in effort to keep it healthy otherwise it could
stop the rest of the system from working.

For this reason, I questioned why flow control has been made mandatory. And
I suggested instead that the credit-based flow control in h2 could be
i) mandatory for a data sender to respond to incoming WINDOW_UPDATEs (and
therefore a data receiver can gracefully protect itself from DoS by
discarding data that exceeds the credit it has previously made available)
ii) optional for a data receiver to emit WINDOW_UPDATEs (i.e. does not even
have to implement this part of the code).


Bob

At 21:12 10/03/2015, Patrick McManus wrote:

Hi Bob - I think your comments are appreciated. Its just one of those
things where people have dispersed to other things and aren't necessarily
in a place to revisit all the ground work again at this stage for a new go
round. It was in large part the operational feedback and needs of the
google.com team, who has a lot of experience operating spdy at scale, that
created the flow control provisions. hopefully those folks will chime in
more authoritatively than my musings below:

I'm sure there is quite a bit to learn here - indeed poorly configured use
of the window_update mechanism has been (predictably) a source of
unintended bottlenecks during both spdy and h2 trials. The spec does try
and highlight that there can be dragons here and implementations that don't
need the features it can bring should provide essentially infinite credits
to steer clear of them.

During the various trials I've seen h2 per stream flow control deployed
successfully for a couple of use cases - both of them essentially deal with
unsolicited data.

The primary one is essentially a more flexible version of h1's
100-continue. When a client presents a large message body (e.g. a file
upload) a multiplexing server needs a way of saying "these are how many
buffers I've got available while I figure out where I'm going to sink this
incoming data (perhaps to another server I need to connect to)". Presenting
this on a per-stream basis allows the server to limit one stream while
another (with a distinct sink) can proceed independently.  IMO this value
should represent resources available and should be independent of BDP. This
is why in practice you see clients with extremely large stream windows -
most circumstances just want the data to flow at line rate (as you
describe) and aren't trying to invoke flow control. The firefox default
window is 256MB per stream - that's not going to slow down the sender nor
require frequent window_update generation.

The other use case is when the server pushes resources at a client without
them being requested, which is a new feature of h2. This is conceptually
similar to the server receiving a POST - suddenly there is a large amount
of inbound data that the implementation might not have the resources to
store completely. We can't just let TCP flow control take care of the
situation because the TCP session is being multiplexed between multiple
streams that need to be serviced. In this case the client accepts "some" of
the stream based on policy and resource availability and can leave the
stream in limbo until an event comes along that tells it to resume the
transfer by issuing credits or reject it via cancel.

hth a least a bit.


On Tue, Mar 10, 2015 at 4:31 PM, Bob Briscoe <bob.briscoe@bt.com> wrote: HTTP/2
folks,
I know extensibility had already been discussed and put to bed, so the WG
is entitled to rule out opening healed wounds.
But have points like those I've made about flow control been raised before?
Please argue. I may be wrong. Discussion can go on in parallel to the RFC
publication process, even tho the process doesn't /require/ you to talk to
me.

If I'm right, then implementers are being mandated to write complex flow
control code, when it might have little bearing on the performance benefits
measured for http/2.
Even if I'm right, and the WG goes ahead anyway, /I/ will understand. My
review came in after your deadline.
However, bear in mind that the Webosphere might not be so forgiving. If h2
goes ahead when potential problems have been identified, it could get a bad
reputation simply due to the uncertainty, just when you want more people to
take it up and try it out. Given you've put in a few person-years of
effort, I would have thought you would not want to risk a reputation flop.
I'm trying to help - I just can't go any faster.

Bob

At 14:43 06/03/2015, Bob Briscoe wrote:

HTTP/2 folks,
As I said, consider this as a late review from a clueful but fresh pair of
eyes. My main concerns with the draft are: * extensibility (previous
posting) * flow control (this posting - apologies for the length - I've
tried to explain properly) * numerous open issues left dangling (see
subsequent postings)
===HTTP/2 FLOW CONTROL===
The term 'window' as used throughout is incorrect and highly confusing, in: *
'flow control window' (44 occurrences), * 'initial window size' (5), * or
just 'window size' (8) * and the terms WINDOW_UPDATE and
SETTINGS_INITIAL_WINDOW.
The HTTP/2 WINDOW_UPDATE mechanism constrains HTTP/2 to use only
credit-based flow control, not window-based. At one point, it actually says
it is credit-based (in flow control principle #2 <
https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-5.2.1 > ),
but otherwise it incorrectly uses the term window.
This is not just an issue of terminology. The more I re-read the flow
control sections the more I became convinced that this terminology is not
just /confusing/, rather it's evidence of /confusion/. It raises the
questions * "Is HTTP/2 capable of the flow control it says it's capable
of?" * "What type of flow-control protocol ought HTTP/2 to be capable of?" *
"Can the WINDOW_UPDATE frame support the flow-control that HTTP/2 needs?"
To address these questions, it may help if I separate the two different
cases HTTP/2 flow control attempts to cover (my own separation, not from
the draft):
a) Intermediate buffer control Here, a stream's flow enters /and/ leaves a
buffer (e.g. at the app-layer of an intermediate node).
b) Flow control by the ultimate client app. Here flow never releases memory
(at least not during the life of the connection). The flow is solely
consuming more and more memory (e.g. data being rendered into a client
app's memory).
==a) Intermediate buffer control== For this, sliding window-based flow
control would be appropriate, because the goal is to keep the e2e pipeline
full without wasting buffer.
Let me prove HTTP/2 cannot do window flow control. For window flow control,
the sender needs to be able to advance both the leading and trailing edges
of the window. In the draft: * WINDOW_UPDATE frames can only advance the
leading edge of a 'window' (and they are constrained to positive values). *
To advance the trailing edge, window flow control would need a continuous
stream of acknowledgements back to the sender (like TCP). The draft does
not provide ACKs at the app-layer, and the app-layer cannot monitor ACKs at
the transport layer, so the sending app-layer cannot advance the trailing
edge of a 'window'. So the protocol can only support credit-based flow
control. It is incapable of supporting window flow control.
Next, I don't understand how a receiver can set the credit in
'WINDOW_UPDATE' to a useful value. If the sender needed the receiver to
answer the question "How much more can I send than I have seen ACK'd?" that
would be easy. But because the protocol is restricted to credit, the sender
needs the receiver to answer the much harder open-ended question, "How much
more can I send?" So the sender needs the receiver to know how many ACKs
the sender has seen, but neither of them know that.
The receiver can try, by taking a guess at the bandwidth-delay product, and
adjusting the guess up or down, depending on whether its buffer is growing
or shrinking. But this only works if the unknown bandwidth-delay product
stays constant.
However, BDP will usually be highly variable, as other streams come and go.
So, in the time it takes to get a good estimate of the per-stream BDP, it
will probably have changed radically, or the stream will most likely have
finished anyway. This is why TCP bases flow control on a window, not
credit. By complementing window updates with ACK stream info, a TCP sender
has sufficient info to control the flow.
The draft is indeed correct when it says: "Â Â  this can lead to suboptimal
use of available    network resources if flow control is enabled without
knowledge of the    bandwidth-delay product (see [RFC7323]). " Was this
meant to be a veiled criticism of the protocol's own design? A credit-based
flow control protocol like that in the draft does not provide sufficient
information for either end to estimate the bandwidth-delay product, given
it will be varying rapidly.
==b) Control by the ultimate client app== For this case, I believe neither
window nor credit-based flow control is appropriate: * There is no memory
management issue at the client end - even if there's a separate HTTP/2
layer of memory between TCP and the app, it would be pointless to limit the
memory used by HTTP/2, because the data is still going to sit in the same
user-space memory (or at least about the same amount of memory) when HTTP/2
passes it over for rendering. * Nonetheless, the receiving client does need
to send messages to the sender to supplement stream priorities, by
notifying when the state of the receiving application has changed (e.g. if
the user's focus switches from one browser tab to another). * However,
credit-based flow control would be very sluggish for such control, because
credit cannot be taken back once it has been given (except HTTP/2 allows
SETTINGS_INITIAL_WINDOW_SIZE to be reduced, but that's a drastic measure
that hits all streams together).
==Flow control problem summary==
With only a credit signal in the protocol, a receiver is going to have to
allow generous credit in the WINDOW_UPDATEs so as not to hurt performance.
But then, the receiver will not be able to quickly close down one stream
(e.g. when the user's focus changes), because it cannot claw back the
generous credit it gave, it can only stop giving out more.

IOW: Between a rock and a hard place,... but don't tell them where the rock
is.
==Towards a solution?==
I think 'type-a' flow control (for intermediate buffer control) does not
need to be at stream-granularity. Indeed, I suspect a proxy could control
its app-layer buffering by controlling the receive window of the incoming
TCP connection. Has anyone assessed whether this would be sufficient?
I can understand the need for 'type-b' per-stream flow control (by the
ultimate client endpoint). Perhaps it would be useful for the receiver to
emit a new 'PAUSE_HINT' frame on a stream? Or perhaps updating per-stream
PRIORITY would be sufficient? Either would minimise the response time to a
half round trip. Whereas credit flow-control will be much more sluggish
(see 'Flow control problem summary').
Either approach would correctly propagate e2e. An intermediate node would
naturally tend to prioritise incoming streams that fed into prioritised
outgoing streams, so priority updates would tend to propagate from the
ultimate receiver, through intermediate nodes, up to the ultimate sender.
==Flow control coverage==
The draft exempts all TCP payload bytes from flow control except HTTP/2
data frames. No rationale is given for this decision. The draft says it's
important to manage per-stream memory, then it exempts all the frame types
except data, even tho each byte of a non-data frame consumes no less memory
than a byte of a data frame.
What message does this put out? "Flow control is not important for one type
of bytes with unlimited total size, but flow control is so important that
it has to be mandatory for the other type of bytes."
It is certainly critical that WINDOW_UPDATE messages are not covered by
flow control, otherwise there would be a real risk of deadlock. It might be
that there are dependencies on other frame types that would lead to a
dependency loop and deadlock. It would be good to know what the rationale
behind these rules was.
==Theory?==
I am concerned that HTTP/2 flow control may have entered new theoretical
territory, without suitable proof of safety. The only reassurance we have
is one implementation of a flow control algorithm (SPDY), and the anecdotal
non-evidence that no-one using SPDY has noticed a deadlock yet (however, is
anyone monitoring for deadlocks?).
Whereas SPDY has been an existence proof that an approach like http/2
'works', so far all the flow control algos have been pretty much identical
(I think that's true?). I am concerned that the draft takes the InterWeb
into uncharted waters, because it allows unconstrained diversity in flow
control algos, which is an untested degree of freedom.
The only constraints the draft sets are: * per-stream flow control is
mandatory * the only protocol message for flow control algos to use is the
WINDOW_UPDATE credit message, which cannot be negative * no constraints on
flow control algorithms. * and all this must work within the outer flow
control constraints of TCP.
Some algos might use priority messages to make flow control assumptions.
While other algos might associate PRI and WINDOW_UPDATE with different
meanings. What confidence do we have that everyone's optimisation
algorithms will interoperate? Do we know there will not be certain types of
application where deadlock is likely?

"   When using flow    control, the receiver MUST read from the TCP
receive buffer in a    timely fashion.  Failure to do so could lead to a
deadlock when    critical frames, such as WINDOW_UPDATE, are not read and
acted upon. " I've been convinced (offlist) that deadlock will not occur as
long as the app consumes data 'greedily' from TCP. That has since been
articulated in the above normative text. But how sure can we be that every
implementer's different interpretations of 'timely' will still prevent
deadlock?
Until a good autotuning algorithm for TCP receive window management was
developed, good window management code was nearly non-existent. Managing
hundreds of interdependent stream buffers is a much harder problem. But
implementers are being allowed to just 'Go forth and innovate'. This might
work if everyone copies available open source algo(s). But they might not,
and they don't have to.
This all seems like 'flying by the seat of the pants'.
==Mandatory Flow Control? ==
"      3. [...] A sender        MUST respect flow control limits
imposed by a receiver." This ought to be a 'SHOULD' because it is
contradicted later - if settings change.
"   6.  Flow control cannot be disabled." Also effectively contradicted
half a page later: "Â Â  Deployments that do not require this capability
can advertise a flow    control window of the maximum size (2^31-1), and
by maintaining this    window by sending a WINDOW_UPDATE frame when any
data is received. Â Â  This effectively disables flow control for that
receiver."
And contradicted in the definition of half closed (remote): "Â  half closed
(remote):       [...] an endpoint is no longer       obligated to
maintain a receiver flow control window. "
And contradicted in 8.3. The CONNECT Method
<https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-8.3>,
which says: "Â  Frame types other than DATA Â Â  or stream management
frames (RST_STREAM, WINDOW_UPDATE, and PRIORITY) Â Â  MUST NOT be sent on a
connected stream, and MUST be treated as a    stream error (Section
5.4.2) if received. "
Why is flow control so important that it's mandatory, but so unimportant
that you MUST NOT do it when using TLS e2e?
Going back to the earlier quote about using the max window size, it seems
perverse for the spec to require endpoints to go through the motions of
flow control, even if they arrange for it to affect nothing, but to still
require implementation complexity and bandwidth waste with a load of
redundant WINDOW_UPDATE frames.
HTTP is used on a wide range of devices, down to the very small and
challenged. HTTP/2 might be desirable in such cases, because of the
improved efficiency (e.g. header compression), but in many cases the stream
model may not be complex enough to need stream flow control.
So why not make flow control optional on the receiving side, but mandatory
to implement on the sending side? Then an implementation could have no
machinery for tuning window sizes, but it would respond correctly to those
set by the other end, which requires much simpler code.
If a receiving implemention chose not to do stream flow control, it could
still control flow at the connection (stream 0) level, or at least at the
TCP level.

==Inefficiency?==

 5.2. Flow Control
<https://tools.ietf.org/html/draft-ietf-httpbis-http2-16#section-5.2> "Flow
control is used for both individual    streams and for the connection as
a whole." Does this means that every WINDOW_UPDATE on a stream has to be
accompanied by another WINDOW_UPDATE frame on stream zero? If so, this
seems like 100% message redundancy. Surely I must  have misunderstood.
==Flow Control Requirements===
I'm not convinced that clear understanding of flow control requirements has
driven flow control design decisions.
The draft states various needs for flow-control without giving me a feel of
confidence that it has separated out the different cases, and chosen a
protocol suitable for each. I tried to go back to the early draft on flow
control requirements <
http://tools.ietf.org/html/draft-montenegro-httpbis-http2-fc-principles-01
>, and I was not impressed.
I have quoted below the various sentences in the draft that state what flow
control is believed to be for. Below that, I have attempted to crystalize
out the different concepts, each of which I have tagged within the quotes.
* 2. HTTP/2 Protocol Overview
<https://tools.ietf.org/html/draft-ietf-httpbis-http2-16#section-2> says Â
"Flow control and prioritization ensure that it is possible to efficiently
use multiplexed streams. [Y] Â Â  Flow control (Section 5.2) helps to
ensure that only data that can be used by a receiver is transmitted.
[X]" * 5.2.
Flow Control
<https://tools.ietf.org/html/draft-ietf-httpbis-http2-16#section-5.2> says: Â
"Using streams for multiplexing introduces contention over use of the TCP
connection [X], resulting in blocked streams [Z]. A flow control scheme
ensures that streams on the same connection do not destructively interfere
with each other [Z]." * 5.2.2. Appropriate Use of Flow Control
<https://tools.ietf.org/html/draft-ietf-httpbis-http2-16#section-5.2.2> "Â
Flow control is defined to protect endpoints that are operating under  Â
resource constraints.  For example, a proxy needs to share memory  Â
between many connections, and also might have a slow upstream  Â
connection and a fast downstream one [Y].  Flow control addresses cases Â
  where the receiver is unable to process data on one stream, yet wants Â
  to continue to process other streams in the same connection [X]." "Â
Deployments with constrained resources (for example, memory) can  Â
employ flow control to limit the amount of memory a peer can consume. [Y]
Each requirement has been tagged as follows: [X] Notification of the
receiver's changing utility for each stream [Y] Prioritisation of streams
due to contention over the streaming capacity available to the whole
connection. [Z] Ensuring one stream is not blocked by another.
[Z] might be a variant of [Y], but [Z] sounds more binary, whereas [Y]
sounds more like optimisation across a continuous spectrum.

Regards


Bob
________________________________________________________________ Bob
Briscoe,                                 Â
                BT

________________________________________________________________ Bob
Briscoe,                                 Â
                BT

________________________________________________________________
Bob Briscoe,                                                  BT


--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

 ________________________________________________________________
Bob Briscoe,                                                  BT


--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat