Re: Feedback for Resumable Uploads

Hello Marius,

Replies inline:

> On Jul 19, 2025, at 07:50, Marius Kleidl <marius@transloadit.com> wrote:
> 
> Hello Austin,
> 
> thank you very much for the feedback! It's greatly appreciated. Please find my comments inline.
> 
> On Wed, Jul 16, 2025 at 10:16 PM Austin Wright <aaa@bzfx.net <mailto:aaa@bzfx.net>> wrote:
>> Hello HTTP WG and Resumable Uploads authors,
>> 
>> This document moved in a very good direction since I last reviewed it! The fundamental design is on point, and my comments mostly relate to expanding the interoperability with other applications of HTTP:
>> 
>> 
>> 
>> 4.2.1. Client Behavior
>> 
>> The section says "All request methods that allow content are possible.” Do you anticipate compatibility with QUERY?
> 
> Yes, QUERY would also be covered. If the content size is rather small, it might be easier and more efficient for the client and server to just retry the entire request without using resumable uploads. It's up to the client to ask for resumable uploads if it deems them useful, but the server can also choose to not offer resumability if the overhead is not worth it. However, that's the case for any request and not just QUERY.

A follow up: I wonder, reading this section, if it should be more obvious that recovering the response is not standardized or guaranteed? I’m thinking of applications such as “Generate a cryptographic signature for this 30 GB file." 

>> 5. Status Code 104 (Upload Resumption Supported)
>> 
>> Is the “Upload-Offset” field found in an Upload Resumption Supported response usable by the client in place of making the Offset Retrieval request?
> 
> The Upload-Offset header field in 104 responses is not usable to resume an upload after interruptions as the resource might have received additional representation data since it sent the last 104 response. Since upload resources don't support overwriting representation data, append requests cannot start before the current offset. In addition, if the client chunked the upload into multiple requests and received a 2XX response, it can just use the Upload-Offset header field from the final response.
> 
> In general, the offsets indicated in the interim responses are intended to show progress to the client and let it know that the offset will never fall below the announced value. Thus allowing the client to free associated buffers. 
>  
>> Considering that the server does not acknowledge every single byte it receives, how does this work if Upload-Offset in an Upload Append operation is expected to exactly match the current upload offset?
> 
> The client should obtain the current offset either from a HEAD request (if the previous upload request was interrupted) or from the final response to the previous upload requests (if it received a 2xx response, but didn't finish the upload). In the latter case, the client should know anyways how by how much the offset advanced since the entire request content was appended.
>  
>> 
>> Now even if the client can’t reliably use Upload Resumption Supported alone to determine where to start an upload append operation, section 4.3.1. Client Behavior says “The client is expected to handle backtracking of a reasonable length”. I think this would be a good place to explain the Upload-Offset field in a 104 (Resumption Supported) response indicates data that has been committed to the upload resource, and need not be stored by the client for future use by this upload resource, but can be freed.
> 
> This is already covered in https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-09.html#section-4.1.1-4. Is this paragraph sufficient or do you think it needs expansion?

One sentence says "The client is expected to handle backtracking of a reasonable length” and another “Clients can use this guarantee to free resources associated to transferred representation data”. While these are consistent with each other, it may be helpful to connect them for the reader. Perhaps:

 “Clients should be prepared to backtrack by any amount not confirmed by the server in an Upload-Offset response field."

and

 “Since the server will not rewind the Upload-Offset, clients can free any resources associated with the portion of the upload up to the given offset.”

>> In general, the ability for a server to signal that a certain amount of the input has been committed and processed would be very useful to have more generally. The only similar functionality I know of is the 102 (Processing) status, and it doesn’t actually have a way of indicating progress, just activity. Could this use of Upload-Offset be defined more generally?
> 
> I see how this could be useful for other applications as well, but I wonder what the exact semantics of such progress updates would. For some applications, these progress reports mean that the data has been fully committed, while in other cases it could mean that the resource just received a certain amount of data but might lose it if an error appears at a later stage in the request. It might be worth talking about this in the httpapi WG, though.

A fair point. It’s a more complicated problem than I could describe in a couple sentences. There’s some good future work in being able to specify when and how often an origin server can select checkpoints that the client has signaled it is capable of recovering from.
 
>> 6. Media Type application/partial-upload
>> 
>> After our early discussions I floated a substantially similar media type for draft-httpapi-patch-byterange, essentially being “apply the enclosed bytes to the range of bytes in the Content-Range header”, but I recall there was some push-back that a media type would rely on out-of-band information in HTTP header fields.
> 
> Yes, I'm concerned about the fact that message/byterange places the offset inside the request content instead of transmitting it inside the header. I don't see a reason why the offset shouldn't be put in the request header for such requests as it avoids the need to parse the request content.

With respect to “message/byterange” I agree, I now think "application/partial-upload” is the correct design here.

>> It appears this document satisfies that concern by instead reading the `Upload-Offset` header first, and evaluating it as a condition, before proceeding with evaluating the contents of the body (at which point, the specific value of the header does not come into play).
>> 
>> Such a header, a new conditional request header that tests the length of the existing resource, could be useful in other applications, especially log writing and journal synchronization applications. What if “Upload-Offset” for making requests was called “If-Length”?
> 
> Yes, the Upload-Offset request header acts as a precondition for conditional requests. It would also be possible to name the request header If-Upload-Offset, I presume.

I do think the functionality in each direction is different enough it warrants distinct names, and an If- prefix would clarify that it makes the request conditional.

However, consider this as bikeshedding.

>> And likewise, the media type itself could be a generic media type that represents “content to append to the target”, so it could be re-used more generally outside the resumable uploads protocol?
>> 
> 
> Such a more generic media type was briefly discussed in https://github.com/httpwg/http-extensions/issues/2962#issuecomment-2500455810, but it didn't seem to attract much interest so far.

In my experience, “append this data" is an often repeated pattern in the API space, so I've proposed it as an addition for the byte range patch document <https://github.com/ietf-wg-httpapi/patch-byterange/issues/3>.

On one hand, I don’t think it’s wrong to have a spec-specific media type, this strategy can be beneficial for future evolution.

On the other hand, I struggle to imagine how evolution could benefit this media type in particular, since the entire possible value space is utilized (the ABNF definition would simply be *OCTET), therefore any change would be a breaking change. If a revision to Resumable Uploads ever needed functionality different than “append bytes”, a successor RFC would have to register a new media type.

I don’t believe this is a technical problem, I just find it odd if there ended up being two media types that have identical “append” semantics; and equally as odd if “application/partial-upload” became the single media type to do this.

>> 11. Responses to Uploads
>> 
>> Content-Location specifies the URI of the attached response body. Normally, it’s for Content-Type negotiation, but as a consequence of its definition, if it were found in a 4xx response, I would expect it to identify a permanent URI that I can use to reference that error later.
>> 
>> Likewise, if included in 1xx, this would suggest that the client has the URI where it can re-request the result of the operation, whether it’s an error or a success. This doesn’t let you retrieve the status code, but for API applications that can infer success or error from the response body, so this may still be an improvement.
>> 
>> So, to assist in recovering the response in the case the response is lost, would there be a benefit to mentioning Content-Location field for 104 (Upload Resumption Supported)?
> 
> The latest iteration of the draft (-09) doesn't mention Content-Location anymore as there were concerns raised that the previously recommended use of Content-Location didn't align with the header field's semantics. Hence, we reworded the section to be more generic in that it recommends servers to put information that should be recoverable either in header fields or point in header fields to resources where such information can be fetched from.

This is also a fair argument. In the future I would like to take up work on tracking progress and outcomes of long-running operations, the general solution to this problem. Treating this as a separate problem for a separate document may be for the best.

>> Are resumable upload resources resumable?
>> 
>> The answer seems to be “no,” as there is a separate, well-defined process for recovering from a failed upload append. However, it may still be worth noting something like:
>> 
>>  An upload resource MUST NOT itself be resumable. An interrupted request to an upload resource is simply retried from the “Offset Retrieval” step.
> 
> Good point, it would be helpful to point out that it doesn't make much sense to make PATCH requests to upload resources resumable as well. In theory, it should work, but in practice there is no use for this and might just enable new abuse patterns.
>  
>> 
>> Preserving Incomplete Uploads with `Incremental: ?1` and/or `Prefer: transaction=persist`
>> 
>> Resumable uploads are necessarily incremental messages, in that every byte uploaded necessarily changes the state of the server. So, would it be worth suggesting the use of the `Incremental: ?1` field from draft-ietf-httpbis-incremental-latest <https://httpwg.org/http-extensions/draft-ietf-httpbis-incremental.html>, and/or the `Prefer: transaction=persist` preference in draft-ietf-httpapi-patch-byterange <https://www.ietf.org/archive/id/draft-ietf-httpapi-patch-byterange-03.html#name-preserving-incomplete-uploa>, in the request headers?
>> 
>  
> Recommending the use of such headers would be a good idea, but I'm hesitant about adding a dependency on another draft. We hope to wrap up work on resumable uploads rather sooner than later and such a dependency might cause more delays.

Agreed, and I don’t see any technical reason it should be required.

>> Thanks,
>> 
>> Austin Wright.
>> 
>> 
> 
> Thanks, Austin! Overall, your feedback reads as if you would prefer the draft to use less upload-specific approaches (e.g. to communicate offset, progress information and data to append). I can understand that desire but I worry that we would end up in a rabbit hole trying to find the correct semantics. I wonder how the working group thinks in general about such situations. When should more generic solutions be preferred over specific ones that are catered to concrete use cases.

You describe my view well. I do see a balance; protocol-specific registrations may even make it easier to evolve this feature in the future. (Compare to how the HTTP Field Name registry was split apart from the Message Headers registry.)

I think ideally, we would spec out primitives (e.g. caching directives); and then informatively publish best practices utilizing them. Though as you suggest, some features are more prone to rabbit holes than others. An “append” media type is very straightforward, but synchronization of “sparse” resources, or negotiating transaction strategies, are not so straightforward, and this document makes effective compromises by not trying to solve every problem (for example, parallel uploads to the same resource). I speculate that in the long run, synchronization will become more generic and displace features that are more narrowly defined; API interactions will look less like “POST some content to this URL, fall back on resumable uploads as necessary” and function more like “Take this crash log and synchronize it with the crash analysis server using whatever features it has available."

> Best regards,
> Marius 

Cheers,

Austin.