Re: Draft v1 Update for Resumable Uploads

Hi Austin,

Thanks for the review, replying inline:

> On Jun 16, 2022, at 17:59, Austin William Wright <aaa@bzfx.net> wrote:
> 
>> 
>> On Jun 16, 2022, at 14:30, Guoye Zhang <guoye_zhang@apple.com <mailto:guoye_zhang@apple.com>> wrote:
>> 
>> 1. Content-Range
>> 
>> We attempted adopting Content-Range header, however, we realized that it doesn’t support unknown lengths which is an important use case that our clients require. Therefore we kept Upload-Offset and Upload-Incomplete headers.
>> 
>> We are open to discuss other options, such as modifying the semantics of the Content-Range header if that’s preferred, although it might cause more breakages than defining new headers.
> 
> The Content-Range header does support unknown lengths <https://httpwg.org/specs/rfc9110.html#field.content-range>:
> 
> > An asterisk character ("*") in place of the complete-length indicates that the representation length was unknown when the header field was generated.
> 
> For example:
> Content-Range: bytes 42-1233/*
> And again, while you can’t use this in requests… you could use it in a new media type (see below).

For dynamically generated content, we don’t know the full length and we also don’t want to chunk them into small upload requests. Essentially we need “bytes 42-/*” which isn’t currently allowed by Content-Range.
> 
>> 2. Media types
>> 
>> PATCH currently doesn’t define a media type. We went through the list of media types but couldn’t find the appropriate category for the Upload Appending Procedure. It is a generic byte-appending operation that can modify any types of media, so we don’t think it fits into an application media type.
>> 
>> We are open to suggestions if a media type is desired.
> 
> I believe a media type is required, you’re supposed to define one to use with PATCH.
> 
> By doing this, you also guarantee that the server will support resumption, since servers that do not will error "415 Unsupported Media Type”.
> 
> I’m working on a draft for “message/byte-range” that would suit this purpose perfectly. Its entire function would be to overwrite the specified byte range with some content. I’ve previously shared an I-D that includes this media type among other things, but I am going to split it out because it’s reusable for so many other purposes. Watch the httpapi list.

Happy to adopt this media type if it becomes available. One thing I would ask to support is the ability for the server to reject all operations other than appending. Many existing resumable upload implementations do not support overriding previously uploaded content, as the bytes might have already been processed and discarded.
> 
>> 3. 1xx intermediate response
>> 
>> We surveyed the most popular HTTP libraries in many languages, and nearly all of them consider 1xx responses an internal signaling mechanism so they don’t expose the ability for applications to handle them. (We are also guilty of this as maintainers of URLSession API on Apple platforms.) If we use 1xx response for any critical information, it would prevent nearly all tus-v1 adopters to switch to this new protocol until it’s natively supported in HTTP libraries.
>> 
>> We think having just the feature detection part using 1xx response is a good balance, both eliminating any extra round trips for HTTP libraries implementing this protocol and allowing application adopters to ignore it.
>> 
>> 4. Can we PATCH a PATCH?
>> 
>> Yes, Upload Creation Procedure supports any method, including PATCH. We included a section “Request Identification” about the nuances in this area. Unfortunately, this added complexity is the result of splitting the procedures, but we don’t think it will complicate the implementations in most cases. Servers can still decide what methods make sense for their use case and whether to support PATCH.
> 
> Some other remarks:
> 
> 1. Upload-Token should be unnecessary in requests, the server should use a URI to identify operations in progress.

Let’s suppose there is a service that creates a thumbnail from a full-sized image: https://example.com/generate_thumbnail?size=512x512. Anybody can send an image in the request body and receive a thumbnail in the response.

Our goal is that any interrupted uploads should be resumable. However, there isn’t a straightforward to mechanically change the URI to distinguish between attempts. Many other resumable upload protocols require you to first request a token then use it to upload which adds a roundtrip, so having a client-generated token is the best approach we could come up.
> 
> 2. Upload-Offset as a new header should be unnecessary, this data can be stored in the PATCH body.
> 
> 3. Feature detection: Feature detection is not a reliable way to ensure that your request will be understood correctly. In general, the request must have the expected effect even if sent to a server that doesn’t understand it. This easy to ensure this by using a new method, or a new media type, in the request.

The Upload Creation Procedure is designed so that a server that ignores unknown headers would treat it as a regular non-resumable upload. This allows generic HTTP client implementations to convert all uploads to resumable upload under the hood. This is the reason we need a feature detection mechanism.

Guoye
> 
> 
> Cheers,
> 
> Austin.
> 
>> 
>> 
>> Looking forward to continuing the discussions and refinements of the draft.
>> 
>> Best regards,
>> Guoye Zhang