Re: Resumable Upload draft updates

Hi Rob,

thanks for the feedback!

I don't really have any strong opinions, but it would be nice to compare it
> to popular storage solutions and see if you really need all of the correct
> but maybe too clever HTTP mechanisms.


AWS S3 offers multipart uploads (not to be confused with the multipart
media type):
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html. The
client first creates a multipart upload resource and can then upload
individual parts to the resource. Each part (except for the last ones) has
a minimum size of 5 MB. Once all parts are uploaded, an additional request
is used to commit the upload. Parts from interrupted requests is not saved
and must be retransmitted.

Google Cloud Storage offers resumable uploads as described in
https://cloud.google.com/storage/docs/resumable-uploads. Their approach is
very similar to the draft in that a resumable upload resource is created,
to which the client can then append data to via individual requests. Data
from interrupted requests seems to be saved. A client resumes the upload by
querying the uploaded data range. Overall, our methods are very similar,
although the used methods and header fields are slightly different.

Azure Blob Storage offers Append Blobs, which seem to act similar to AWS
S3' multipart uploads:
https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs
But I haven't used them much on my own.

I wonder if it's helpful to include such a comparison in the draft.

The one thing that stuck out to me was the OPTIONS request, I thought that
> might be better in the TLS handshake.


I would like to emphasize here again that the OPTIONS request is optional
and a client can start an upload without. Putting upload limits into the
TLS handshake is interesting but not ideal as upload limits belong to a
resource and are not a property of the connection. In addition, getting
this information from the TLS handshake on the client-side is not always
possible, depending on the HTTP API (e.g. the Fetch API does not expose
this). Configuring this on the server-side could also be non-trivial if a
TLS-terminating proxy is involved.

The draft contains a similar discussion of different methods to indicate
server support for resumable uploads:
https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-resumable-upload-04#name-feature-detection-2.
It does not mention TLS handshake, but touches on similar topics.

Best regards,
Marius

On Wed, Jul 24, 2024 at 9:03 PM Rob Sayre <sayrer@gmail.com> wrote:

> Hi,
>
> This is in reference to the IETF 120 session and
>
> <
> https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-resumable-upload-04
> >
>
> Firstly, people continually reimplement this feature, so it would be nice
> to standardize it. I don't really have any strong opinions, but it would be
> nice to compare it to popular storage solutions and see if you really need
> all of the correct but maybe too clever HTTP mechanisms. The one thing that
> stuck out to me was the OPTIONS request, I thought that might be better in
> the TLS handshake.
>
> All of those storage servers have this, but the chunks are often very
> large, like 100MB. That works great if you're uploading giant files
> on servers. But the other thing is that your favorite social networks
> definitely do this in much smaller slices when you upload a photo. This is
> so you can take a picture, get on the subway, and it all works in a few
> minutes. I guess "small objects in the presence of packet loss" would be
> the general term. That is the same design as the storage servers, it just
> slices files up a lot more.
>
> thanks,
> Rob
>
>