Re: Out-of-order delivery of chunked transfers

My advice:

* You are creating an HTTP resource, not an HTTP extension.  Use HTTP via its semantic interface (RFC 9110), not a version-specific syntax.
* Stream chunks out of order in the response body using a simple "Offset-length-value" syntax.
* Feel free to reuse QUIC varints if that's convenient, even if the response is over HTTP/1.1.

--Ben
________________________________
From: Richard Bradbury <richard.bradbury@rd.bbc.co.uk>
Sent: Monday, October 20, 2025 11:40 AM
To: HTTP Working Group <ietf-http-wg@w3.org>
Subject: Out-of-order delivery of chunked transfers

Hello. I am seeking expert opinion on a candidate solution to a Real World Use Case with which I was recently presented. TL;DR: Would it be legitimate to extend HTTP/1. 1 chunked transfer coding to support delivery of chunks out of order? Is


Hello.

I am seeking expert opinion on a candidate solution to a Real World Use Case with which I was recently presented.

TL;DR:

  *   Would it be legitimate to extend HTTP/1.1 chunked transfer coding to support delivery of chunks out of order?
  *   Is such an extension a useful thing to standardise?

Use Case description

The Use Case involves transmitting Binary Large Objects (BLOBs) in a multicast FLUTE v1<https://urldefense.com/v3/__https://www.rfc-editor.org/rfc/rfc3926.html__;!!Bt8RZUm9aw!-2dr-D75Lm1QkJHuB3sRt_9nsouTsa-aAJXtmnhbbATWrj-Na89Tap9ld_fzK4YAAg4_4OihQnnJbRn03tfB4n8NvQ$> session operating in carousel mode (i.e., FLUTE transmission objects in the session are sent repeatedly, possibly interleaved). The receiver is a gateway device offering an HTTP endpoint to downstream clients that want to acquire one or more transmission objects of interest to them.

Steps:

  1.  A client requests an HTTP resource from the gateway, citing it's URL.
  2.  The gateway joins the appropriate multicast group (based on some look-up mechanism not of concern here) and starts receiving FLUTE transmission objects, including the File Delivery Table<https://urldefense.com/v3/__https://www.rfc-editor.org/rfc/rfc3926.html*section-3.2__;Iw!!Bt8RZUm9aw!-2dr-D75Lm1QkJHuB3sRt_9nsouTsa-aAJXtmnhbbATWrj-Na89Tap9ld_fzK4YAAg4_4OihQnnJbRn03tcm1A-QOw$> transmission object.
  3.  The gateway looks up the URL requested by the client in the File Delivery Table of the FLUTE session and this yields the Transmission Object Identifier that it needs to acquire.
  4.  The gateway starts receiving FLUTE packets (well, actually ALC/LCT packets) for the identified transmission object and sends their payloads to the requesting client in the HTTP response body.

So far, so vanilla.

The twist is that the gateway has a low memory footprint, much smaller than the size of the BLOBs being transmitted in the FLUTE session. Hence, it cannot operate in a store-and-forward mode, and needs instead to deliver incoming data from the FLUTE session to its client in an efficient, pipelined fashion, minimising the need for buffering and anything beyond simple reassembly of received data blocks. Moreover, the client request to the gateway (step 1 above) arrives at a random time during the FLUTE session, in the worst case towards the end of a rotation of the carousel. In addition, the gateway may occasionally drop received FLUTE packets.

Candidate solutions

My first instinct is always to try and reuse existing building blocks wherever possible, rather than inventing something new.

HTTP/1.1

The chunked transfer coding feature that all HTTP/1.1 clients and servers are required to implement seems a good fit, except for one small detail: it assumes in-order delivery of chunks. Each chunk follows on from the last and is appended by the HTTP recipient to the previously received chunk (if any).

My idea is to include with every chunk a name–value pair (e.g. "offset=...") that additionally signals the byte position of this chunk in the overall resource representation. I reckon this could be achieved in the chunk delimiter using the chunk-ext production specified in section 7.1 of RFC 9112<https://urldefense.com/v3/__https://www.rfc-editor.org/rfc/rfc9112*name-chunked-transfer-coding__;Iw!!Bt8RZUm9aw!-2dr-D75Lm1QkJHuB3sRt_9nsouTsa-aAJXtmnhbbATWrj-Na89Tap9ld_fzK4YAAg4_4OihQnnJbRn03tcogJKX4g$>. For example:

1fb;offset="40d4"

With this additional signalling, the chunks could be sent on by the gateway to the client in whichever order they are received from the multicast FLUTE session, even if the gateway joins the multicast group partway through a carousel rotation. And the client could, for example, write received chunks to a sparse file using lseek() as they arrive in the HTTP response body. Happy days.

Once the entire transmission object has been received by the gateway and forwarded to the requesting client, the HTTP response body is terminated with the standard zero-length chunk. (More than one full rotation of the FLUTE carousel may be required to satisfy this termination condition in the presence of packet loss.)

In the absence of the proposed "offset=..." extension, a received chunk is assumed to belong immediately after the one previously received, as now, so the signalling is backwards-compatible with RFC 9112.

HTTP/2

(Not an immediate requirement, but I include an analysis for completeness.)

This protocol version effectively uses chunked transfer for all message bodies, thanks to its binary framing. The HTTP/2 DATA frame includes length but not position. Being based on TCP, HTTP/2 client implementations are likely not expecting to receive DATA frames out of order. Since chunked transfer coding is explicitly prohibited by section 8.1 of RFC 9113<https://urldefense.com/v3/__https://www.rfc-editor.org/rfc/rfc9113.html*name-http-message-framing__;Iw!!Bt8RZUm9aw!-2dr-D75Lm1QkJHuB3sRt_9nsouTsa-aAJXtmnhbbATWrj-Na89Tap9ld_fzK4YAAg4_4OihQnnJbRn03teCZSRugw$> it seems very difficult to achieve our of order delivery with this version of HTTP.
HTTP/3

(Not an immediate requirement, but I include an analysis for completeness.)

Again, this protocol version natively uses chunked transfers. Similar to HTTP/2, the HTTP/3 DATA frame header signals the frame length. The position of this in the underlying QUIC stream is explicitly encoded in the Offset field of the QUIC STREAM frame header, so there is potential to get cute with HTTP/3 DATA frame delivery order, provided the size of the HTTP/3 DATA frame headers (which can contain variable-length integers) is properly accounted for and no other frames are sent on that QUIC stream. I can't decide if that's elegant or evil. Maybe both at the same time.

Although QUIC streams are intended to offer an in-order bytestream abstraction to applications, being a UDP-based protocol, QUIC endpoints need to be robust to receiving datagrams out of order, to some extent. I note in particular the following in section 2.2 of RFC 9000<https://urldefense.com/v3/__https://www.rfc-editor.org/rfc/rfc9000.html*name-sending-and-receiving-data__;Iw!!Bt8RZUm9aw!-2dr-D75Lm1QkJHuB3sRt_9nsouTsa-aAJXtmnhbbATWrj-Na89Tap9ld_fzK4YAAg4_4OihQnnJbRn03tdKkIHISg$>:

"Endpoints MUST be able to deliver stream data to an application as an ordered byte stream. Delivering an ordered byte stream requires that an endpoint buffer any data that is received out of order, up to the advertised flow control limit.

QUIC makes no specific allowances for delivery of stream data out of order. However, implementations MAY choose to offer the ability to deliver data out of order to a receiving application."

Hence, sending QUIC STREAM frames with unordered Offset values seems to be a potentially viable solution to satisfy the Use Case that is sort of in keeping with the spirit of RFC 9000.

Whether HTTP/3 client implementations are robust to receiving DATA frames out of order is another matter, of course, but that would be a requirement for supporting this Use Case. (Unlike the resource-constrained gateway, the end client in this Use Case is less constrained.)