Re: Partial review of Resumable Uploads draft-ietf-httpbis-resumable-upload-05

Thanks for the very detailed review and actionable feedback Roy!

It'll take some time for us to go through it. The largest _design_ issue you raise is around PATCH vs POST. This is something we should track in an issue, the authors will create one once at a computer.

Cheers
Lucas

On Sat, Nov 16, 2024, at 21:06, Roy T. Fielding wrote:
> The quoted text is from https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-05.txt
> 
> > Resumable Uploads for HTTP
> > draft-ietf-httpbis-resumable-upload-05
> > 
> 
> General comments:
> 
> The protocol described is generally fit for purpose. However, it is overly constrained as an HTTP extension and editorially contorted. I assume this is because some folks want it specified as HTTP and others want it specified as file upload. IOW, “too many reviewers” with contrary opinions.
> 
> I can appreciate that problem, but the context and purpose of this draft is to extend HTTP. It’s perfectly reasonable to document how to do a file upload using the protocol mechanism described here, but such documentation can be supplied as an example, appendix, or on some other website. *This* document must define the proposed extension in HTTP terms.
> 
> I know there has been some discussion of a future editorial pass, along with an issue for that on GitHub, but it looks like that pass is intending to be more file-specific. I disagree.
> There are no files in HTTP. Specifying an extension in that way is a crutch that will lead to interop failure as HTTP changes over time. Specifically, it interferes with a more natural description of the protocol as a continuation of the initial request.
> 
> 1) Obey the principle of orthogonality:
> 
> a) Use HTTP, don’t constrain it.
> 
> The protocol extensions are mostly normal. Their description is overly specific and treads on a number of HTTP landmines as a result.
> 
> Resources are identified, messages are received, and semantics are revealed accordingly. Requirements need to be stated in terms that are consistent with the rest of HTTP, rather than taking a very specific use case and forcing it into a specific set of message requirements.
> 
> b) Do not redefine things that are already defined by HTTP.
> 
> Status codes have self-descriptive semantics. The protocol does not need to require the server to send a specific code. It can suggest certain codes for certain responses, but a server is always free to respond with any status code, and a client must be prepared to receive any code. In some cases it makes sense to define “what to do next” based on a specific status code, but that should always be done within the context of any code being a valid response.
> 
> The spec should use “initial request” (not target request) and “resumption request” (not “the request”), where needed to disambiguate.
> 
> The spec should also require HTTP/1.1 (or later) as a minimum. This means a server receiving such a request via HTTP/1.0 must ignore the Upload-* fields because 104 isn’t supported in HTTP/1.0.
> 
> c) Assume that the extension applies to all resources receiving content
>    (i.e., do not limit requests/responses to what this draft anticipates).
> 
> For example, if a given semantic is not specific to the method, don’t mention the method. Any request that conveys content might be interrupted, so define the initial request as any request that conveys content. Let the server decide which methods can be resumed. The server needs to remember the initial request target and method along with the upload resource state.
> 
> 2) Apply requirements to roles, not to messages.
>    See "https://httpwg.org/specs/rfc9110.html#requirements.notation”.
> 
>    a) The {sender|client|server} MUST/SHOULD {generate|send} X to indicate Y …
>    b) A message received with X indicates …
>    c) After receiving X, the recipient can/MUST/SHOULD do …
> 
> For example, this paragraph is inscrutable:
> 
> > Upload-Complete MUST be set to false if the end of the request
> > content is not the end of the upload. Otherwise, it MUST be set to
> > true. This header field can be used for request identification by a
> > server. The request MUST NOT include the Upload-Offset header field.
> 
> 
> 3) Do not assume a single use case
> 
>    a) define what the protocol means, not when to use it.
>    b) provide real examples separate from definitions.
> 
> 
> Specific comments:
> 
> The introduction is struggling to say the simple stuff. REST terminology is not helping here because the text is being overly specific about things that don’t matter in this context. [Talking about representations is useful later in the draft when discussing digests, content-encoding, and method semantics. It is okay to just reference RFC9110 as needed.]
> 
> An introduction just needs to motivate the spec. The details come later.
> 
> My suggested translation follows after:
> 
> > 1. Introduction
> > 
> > HTTP clients often encounter interrupted data transfers as a result
> > of canceled requests or dropped connections. Prior to interruption,
> > part of a representation (see Section 3.2 of [HTTP]) might have been
> > exchanged. To complete the data transfer of the entire
> > representation, it is often desirable to issue subsequent requests
> > that transfer only the remainder of the representation. HTTP range
> > requests (see Section 14 of [HTTP]) support this concept of resumable
> > downloads from server to client.
> > 
> 
> > HTTP methods such as POST or PUT can be used by clients to request
> > processing of representation data enclosed in the request message.
> > The transfer of representation data from client to server is often
> > referred to as an upload. Uploads are just as likely as downloads to
> > suffer from the effects of data transfer interruption. Humans can
> > play a role in upload interruptions through manual actions such as
> > pausing an upload. Regardless of the cause of an interruption,
> > servers may have received part of the representation before its
> > occurrence and it is desirable if clients can complete the data
> > transfer by sending only the remainder of the representation. The
> > process of sending additional parts of a representation using
> > subsequent HTTP requests from client to server is herein referred to
> > as a resumable upload.
> > 
> > Connection interruptions are common and the absence of a standard
> > mechanism for resumable uploads has lead to a proliferation of custom
> > solutions. Some of those use HTTP, while others rely on other
> > transfer mechanisms entirely. An HTTP-based standard solution is
> > desirable for such a common class of problem.
> > 
> > This document defines an optional mechanism for HTTP that enables
> > resumable uploads in a way that is backwards-compatible with
> > conventional HTTP uploads. When an upload is interrupted, clients
> > can send subsequent requests to query the server state and use this
> > information to send the remaining data. Alternatively, they can
> > cancel the upload entirely. Different from ranged downloads, this
> > protocol does not support transferring different parts of the same
> > representation in parallel.
> 
> Data transfer using the Hypertext Transfer Protocol [HTTP] is often
> interrupted by canceled requests or dropped connections. Interruptions
> are more likely as the size of data transfer increases. If the intended
> recipient can indicate how much of the data was received prior to
> interruption, a sender can resume data transfer at that point instead
> of attempting to transfer all of the data again.
> 
> Range requests (Section 14 of [HTTP]) provide support for resumable
> data transfers from server to client (a.k.a., downloads).
> 
> This specification defines a mechanism for resumable data transfers
> from client to server (a.k.a., uploads). New header fields are
> defined to indicate support for resumable upload and describe the
> extent of what is being uploaded. A new informational status
> code is defined to identify an upload resource where the current
> data transfer can be resumed or canceled.
> 
> This specification does not define support for transferring
> an upload as multiple requests in parallel.
> 
> > 2. Conventions and Definitions
> 
> > 
> > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
> > "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
> > "OPTIONAL" in this document are to be interpreted as described in
> > BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
> > capitals, as shown here.
> > 
> > The terms Byte Sequence, Item, String, Token, Integer, and Boolean
> > are imported from [STRUCTURED-FIELDS].
> > 
> > The terms "representation", "representation data", "representation
> > metadata", "content", "client" and "server" are from [HTTP].
> 
> URL is also being used without a reference. For this draft, it
> should be URI and reference Section 4 of [HTTP].
> 
> > 3. Overview
> > 
> > Resumable uploads are supported in HTTP through use of a temporary
> > resource, an _upload resource_, that is separate from the resource
> > being uploaded to (hereafter, the _target resource_) and specific to
> > that upload. By interacting with the upload resource, a client can
> > retrieve the current offset of the upload (Section 5), append to the
> > upload (Section 6), and cancel the upload (Section 7).
> 
> 
> IOW,
> 
>    Resumable uploads are supported through server creation of a separately
>    identified (usually temporary) resource specific to the initial request.
>    This separate “upload resource" can be accessed by the client to discover
>    how much data was received, resume the upload, or cancel the upload.
> 
> Personally, I would just place the above at the end of Introduction and
> relocate section 3 to the end of the spec, with real examples instead
> of these arrow diagrams with prose. The diagrams are more complicated
> than the protocol exchanged as HTTP/1.1.
> 
> > The remainder of this section uses an example of a file upload to
> > illustrate different interactions with the upload resource. Note,
> > however, that HTTP message exchanges use representation data (see
> > Section 8.1 of [HTTP]), which means that resumable uploads can be
> > used with many forms of content -- not just static files.
> 
> Argh. These are not different things. A file is a storage mechanism, not a type of data transfer. Even when the client thinks it is uploading a file, the server remains in complete control over how that data will be interpreted by the initial resource (and thus how the resumed data will be interpreted as well). Hence, it is not a file upload even when a client thinks it might be.
> 
> This assumption makes the spec more complicated than it needs to be and doesn’t define the protocol in the process. The overview can be deleted or provided at the end as actual examples using HTTP/1.1.
> 
> ====
> 
> The remaining parts of the specification are defined in an
> unnatural order. I expected the protocol to be defined in the
> same order that the mechanisms would be used in HTTP:
> 
>   3. Initial request
>      a) define what client sends (if anything) to indicate support
>         for resumption?
>      b) describe case where server only provides final response
>      c) maybe describe case where client already expects that the
>         server will support a resumable upload
> 
>   4. 104 status code
>      a) define semantic and how it identifies upload resource
>      b) describe use cases that would want 104 prior to interrupt
>      c) suggest that a client needs to look for 104 while sending
>         the initial data transfer
> 
>   5. Upload resource (or resumption resource)
>      a) what state the server needs to retain (initial request target, method, etc.)
>      b) how a client resumes the data transfer
>      c) how a client completes the data transfer
> 
>   6. Upload cancel
>      a) how a client cancels the data transfer
>      b) how a server cancels the data transfer
>      c) suggestions (non-requirements) on cleaning up after
> 
>   7. Examples
>      a) resuming a file upload
>      b) resuming a large form
>      c) resuming a long poll
>      … 
> 
> 
> I've run out of time/energy to go further, but once the protocol is described in order, instead of an iteration over several use cases, then I think it will be clear what other requirements can be removed because they are already defined by HTTP semantics. All of the use cases fit within those semantics.
> 
> For example, remove any requirement that says a server must respond with 204 or 404 or 403 (as opposed to just describing what that means), and anything that redefines method semantics.
> 
> For example, use of the method PATCH for appending opaque data is incorrect. PATCH already has a defined semantic that could be used in parallel (e.g., another extension might use PATCH correctly). The resumable upload extension does not use a patch format, so both use of PATCH and creation of a new media type "application/partial-upload” are inappropriate.
> 
> POST should be used for appending data to the upload resource. This matches the semantics of POST and is easier to deploy in practice. Each POST can use “application/octet-stream” (or not send any Content-Type, since it has no role to play in processing).
> 
> I know this message is kind of annoying, particularly this late in the process. I hope it helps in the long run.
> 
> Cheers,
> 
> Roy T. Fielding, Senior Principal Scientist, Adobe
> 
> 
>