Re: On the abuse of chunking for interactive usages

Hi Jamie,

On Wed, May 11, 2011 at 01:01:03PM +0100, Jamie Lokier wrote:
> Willy Tarreau wrote:
> > Example 1:
> >    video streaming sent by chunks might start playing only once download
> >    is complete. However, once transferred, it works, so even if this causes
> >    discomfort to the end user, it does not break the original goal of the
> >    application.
> 
> I know of several *infinitely* long video streaming applications, and
> a plenty of internet radio audio streaming applications, which expect
> to be able to stream continuously.
> 
> They will break if an intermediary tries to buffer it all.

Indeed, I agree that for *infinitely* long streaming it makes a difference.
Maybe some rewording is necessary then, but the general idea is that an
application must not expect any partial or complete chunk to be delivered
ASAP.

> > Example 2:
> >    an application which sends exactly one image in each chunk, hoping
> >    for the recipient to use chunks as delimiters might break if an
> >    intermediary rechunks them.
> 
> I see rechunking as similar to TCP packet boundaries.  Nobody should
> depend on chunk boundaries to mean anything.

I agree with your analogy.

(...)
> I think there are 3 different issues being mixed up here:
> 
>   1. Streaming or buffering a message (in one direction only).
>   2. Streaming request while streaming response.
>   3. Depending on reliable chunk boundaries.
> 
> I think it's important to distinguish them, instead of declaring them
> all invalid while lumped together.
> 
> 1. Streaming or buffering a message (in one direction only)
> -----------------------------------------------------------
> 
> 1a: The ability to continuously stream a response is a fairly well
> established use for some applications, such as internet radio.  Those
> applications don't require devery byte to be forwarded immediately,
> only that they are delivered in a reasonable time.

I agree with your principle of "reasonable delay". A 200ms delay for
a streaming radio is not an issue at all, and at worst, all the streamcast
will be delayed by 200ms. But when expecting interactive uses, it becomes
a problem.

> 1b: A fair number of applications expect HTML or Javascript response
> streaming to work.  HTML streaming goes back to the early days of
> Netscape Navigator - when it was possible to transmit multiple HTML
> documents, each replacing the previous one, and it was *expected* that
> they'd be delivered in a reasonable time.  Javacript streaming is what
> people do with iframes these days (still) in lieu of XmlHttpRequest
> for a variety of reasons.  However there are still question marks
> around how reliable streaming is in practice.

OK, but still, you can't prevent an intermediary to require more bytes
to be sent in order to complete a processing. For instance, you advertise
that there are more data to come, so the next hop waits for those because
an intermediate encoding needs full blocks, and it will block until you
send the next chunk. For instance, if data were to be encoded in base64
in the middle, it would be reasonable to expect that any block that is
not multiple of 3 will wait for either 1 or 2 next bytes, or for an end.

> 1c: It's clear that if an intermediary buffers whole messages, it will
> limit the message size.  For example if you PUT or GET an 80GB file
> (haven't we all), not many full-buffering proxies are going to be
> happy with that.  Yet it is clearly a valid use of RFC2616 HTTP.  That
> would be an example of something which works until you insert a proxy
> (of a certain kind) where you can't really complain about the
> application, and perhaps should complain about the proxy.

I agree, and that's what happens with content analysis at many places.

> 1d: For unidirectional streaming, it should be noted that CONNECT's
> buffering/forwarding behaviour is not fully specified either
> (e.g. forward every byte eagerly, or delay for a *limited* time to
> coalesce TCP segments).  If a proxy has a sane CONNECT implementation
> it's not unreasonable for it to use the same strategy for streaming
> *unidirectional* requests or response, if it does not have a reason to
> buffer them more.

That's a good point. That said, I've never seen a proxy which delays
data in CONNECTs, probably because if they did so, they'd quickly
notice that HTTPS does not work well at all.

> 1e: In practice, if unidirectional streaming is not reliable enough,
> the common strategy is to transmit a stream of bulky HTTP entire
> messages instead to accomplish the same task.  Or to try using CONNECT.

I perfectly agree. That was really my point about the "abuse" of chunking.

> 2. Streaming request while streaming response
> ---------------------------------------------
> 
> 2a: This is certainly debatable and/or dubious.  I know from
> experiments it does not work with all browser clients.  Sometimes the
> client waits until it has sent the whole request before reading the
> response (so deadlock is possible if the server doesn't handle this).
> 
> 2b: I wouldn't expect all proxies to support this, even those which
> handle CONNECT fine, which is a shame as it would be quite useful if
> it worked.

We could even imagine that some proxies will stop streaming the request
as soon as they see the server start responding, because it's pointless
to make the client send more bytes if the server appears to be happy
with what it has seen (think about a login redirect upon a webmail POST
involving a large attached file for instance)..

> 2c: In practice applications can simply open two HTTP connections, and
> stream one direction over each connection (at some loss of
> efficiency).  This removes the question of bidirectional streaming, while
> still depending on unidirectional streaming behaviours.

Yes.

> 3. Depending on reliable chunk boundaries
> -----------------------------------------
> 
> 3a: Chunk boundaries are analogous to TCP segment boundaries:
> Absolutely nothing should *ever* depend on their position.  If a
> protocol wants to do something fancy with incremental messages, that
> should be encoded in the data bytes of the messages *only*.  Even
> transparent proxies may rechunk arbitrarily according to instantaneous
> low-level TCP states.

I 100% agree with your point, eventhough I'm fairly sure I've observed
horrors in the past (streamed images with 1 chunk == 1 image).

> 3b: It isn't specified, but any behaviour that depends on a proxy
> forwarding bytes in a reasonable time also requires it to forward
> *parts* of chunks in a reasonable time.  That follows from the chunk
> boundaries having no semantic significance, and practically the fact
> an earlier sender may merge chunks arbitrarily.

Yes, but as I explained, when you're saying someone "Hey, I'm sending
you 10 bytes" and only emit 9, it's hard to expect it to forward those
9 bytes alone without waiting for the last one. Some internal block
processing might even not easily permit to do so.

> 3c: The sole entire purpose of chunks is to allow the sender to
> terminate the data stream when it is ready without advance knowledge
> of the message size.  This is why chunk boundaries don't (or
> shouldn't) mean anything. It's just a mechanism for encoding an out
> of band "end of message".

Perfectly agree too.

> 3d: The example which started this thread, of an application expecting
> individual chunks to be forwarded through haproxy immediately, is
> broken by design if it requires chunk boundaries to be preserved.

I don't think it requires boundaries, however since it uses only one
chunk at a time per direction, chunk merging cannot happen. Still, we
could imagine an intermediary which rechunks once in a while when a
chunk crosses an internal buffer boundary, resulting in two outgoing
chunks for an incoming one.

> This is *separate* from whether it requires simultaneous request and
> response streaming, and whether it requires its data bytes to be
> forwarded in a reasonable time (and in that case it should use a
> sliding window so that small delays don't stall the protocol's
> progress disproportionately).

That's the problem with this protocol : it alternatively sends 10 bytes
at a time in each direction. Even a delay of one millisecond makes the
protocol stall to at most 1000 packets per second ~= 10 kB/s.

> Finally: Buffering / buffer timing behaviour of CONNECT is not
> specified either.  It is commonly understood that all bytes must be
> forwarded in a reasonable time (otherwise HTTPS wouldn't work), but
> not whether they are forwarded *eagerly*, or delayed for a short time
> for TCP efficiency, as haproxy does to streamed messages in the
> example which started this thread.

At least if CONNECT or GET+Upgrade had been used there, no delaying would
have been performed (speaking for haproxy but other products seem OK on
this point).

> An application like the one which
> started this thread, but running over CONNECT, may well be bitten by
> the same delay RTT issue over CONNECT with a proxy in the middle, and
> this is likely to affect WebSocket users who design protocols
> sensitive to RTT as well.

The RTT issues will always impact poorly designed protocols which work
with short request/response patterns. TCP makes that doubly worse,
because short reqs means a PUSH which generally implies an immediate
ACK on the other side.

Thanks for your insights, Jamie, you are right that I probably mixed
several issues in a single report, probably because I was concerned by
all the dirty misuses of a message body.

Best regards,
Willy

Received on Wednesday, 11 May 2011 13:15:12 UTC