Re: HTTP Resumable Uploads on Apple Platforms

Hi Marius,

Thanks for spending your time testing the code!

> On Jul 9, 2023, at 02:59, Marius Kleidl <marius@transloadit.com> wrote:
> 
> Hi Jonathan,
> 
> I played around with the example code using SwiftNIO and wanted to provide some feedback. First of all, it follows a very interesting and different approach than other resumable upload servers. In the past, I have mostly seen resumable upload servers implemented as additional services, which are separate from the main backend services. Once a resumable upload is complete, the main service will be notified about the complete upload using some custom communication channel. However, your example takes a different approach by wrapping the main backend logic allowing every request to be turned into a resumable upload, while being completely transparent to the main backend logic. It is going to be interesting to see what the benefits and drawbacks of these two approaches will be over time.
> 
> You can find my example code in https://github.com/tus/draft-example/tree/main/servers/swift-nio, where I try to collect different implementations of the resumable upload draft. It's my first exposure to SwiftNIO, so it might be quite rough around the edges. Nevertheless, there are a few points worth mentioning:
> If the offsets in the Upload Appending Produce do not match the server's expectation, a 409 Conflict is returned and the entire resumable upload is cancelled. Subsequent requests to the upload URL result in a 404 Not Found. While this is not prohibited by the draft, I am wondering if it's better to keep the upload around and give the client a second chance to resume the upload.
I made it strict here because the client must use the offset specified in the previous offset retrieving procedure, so appending with an incorrect offset is only possible if the client is not compliant. That being said, there is not much harm in keeping the upload around, so we could change the code to not terminate the upload.
> The decision whether an upload is complete seems to depend on the value of Content-Length and Upload-Incomplete. For example, an Upload Creation Procedure with Upload-Incomplete: ?1, Content-Length: X, and a request body of X bytes creates an upload that is marked as incomplete in responses for the Offset Retrieving Procedure, but which cannot be extended using the Upload Appending Procedure. All subsequent PATCH requests are rejected with 409 Conflict responses because the server considers the upload to have a maximum length of X bytes (taken from the first request's Content-Length), which is already satisfied through the first request. The server seems to ignore the Upload-Incomplete header here, which is the main indicator for an upload's completion state (as far as I understood the specification). I could also be wrong here, but we should improve the draft's wording to make it clearer when an upload is considered complete. I also included a brief curl exchange below to demonstrate what I meant here.
Oh this is a bug. I will push a new version with the fix.
> The sample code from https://developer.apple.com/documentation/foundation/urlsession/building_a_resumable_upload_server_with_swiftnio requires the so far unreleased Swift 5.9 version to run. Is 5.9 necessary or could 5.8 also suffice, which is the latest released version?
The version requirement will be reduced to 5.7. The only reason for targeting 5.9 was the lack of time testing older compiler versions.
> Are there any plans to continue maintaining NIOResumableUpload as a library or was it intended as a one-off example? If you to don't have such plans, maybe we can host it in the tus GitHub organization and make it more accessible for other developers.
The plan is to eventually migrate it to apple/swift-nio-extras repo. However, it might take a while since they require all dependent packages to be stable. In the meantime, feel free to mirror it in tus organization.

Guoye

> Thank you again for releasing the code and features!
> 
> Best regards
> Marius
> 
>> ~/workspace/tus/draft-example (main) $ curl -i -X POST 127.0.0.1:8080 <http://127.0.0.1:8080/> -H 'upload-draft-interop-version: 3' -H 'upload-incomplete: ?1' -d "hello "
>> HTTP/1.1 104 Upload Resumption Supported
>> Upload-Draft-Interop-Version: 3
>> Location: http://127.0.0.1:8080/resumable_upload/14803916209729849474-1337128731644763746
>> 
>> HTTP/1.1 201 Created
>> Upload-Draft-Interop-Version: 3
>> Location: http://127.0.0.1:8080/resumable_upload/14803916209729849474-1337128731644763746
>> Upload-Incomplete: ?1
>> Upload-Offset: 6
>> transfer-encoding: chunked
>> 
>> ~/workspace/tus/draft-example (main) $ curl -i --head http://127.0.0.1:8080/resumable_upload/14803916209729849474-1337128731644763746 -H 'upload-d
>> raft-interop-version: 3'
>> HTTP/1.1 204 No Content
>> Upload-Draft-Interop-Version: 3
>> Upload-Incomplete: ?1
>> Upload-Offset: 6
>> Cache-Control: no-store
>> 
>> ~/workspace/tus/draft-example (main) $ curl -i -X PATCH http://127.0.0.1:8080/resumable_upload/14803916209729849474-1337128731644763746 -H 'upload
>> -draft-interop-version: 3' -H 'upload-offset: 6' -H 'upload-incomplete: ?
>> 0' -d "world"
>> HTTP/1.1 409 Conflict
>> Upload-Draft-Interop-Version: 3
>> Upload-Incomplete: ?1
>> Upload-Offset: 6
>> Content-Length: 0
> 
> On Fri, Jun 9, 2023 at 1:13 AM Jonathan Flat <jflat@apple.com <mailto:jflat@apple.com>> wrote:
>> Hi all,
>> 
>> New on all Apple platforms, URLSession supports HTTP resumable uploads using the protocol in draft-ietf-httpbis-resumable-upload-01 <https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-01.txt>. The WWDC23 session “Build robust and resumable file transfers <https://developer.apple.com/videos/play/wwdc2023/10006/>" describes the new API and is accompanied by a sample code project “Building a resumable upload server with SwiftNIO <https://developer.apple.com/documentation/foundation/urlsession/building_a_resumable_upload_server_with_swiftnio>". Excited to hear your thoughts and feedback, and looking forward to more implementations!
>> 
>> Best,
>> Jonathan
>> jflat@apple.com <mailto:jflat@apple.com>