Re: Sending very large chunks over the data channel

On 28 May 2014, at 11:44, Harald Alvestrand <harald@alvestrand.no> wrote:

> On 05/28/2014 12:35 PM, Tim Panton new wrote:
>> 
>> On 28 May 2014, at 11:02, Harald Alvestrand <harald@alvestrand.no> wrote:
>> 
>>> On 05/28/2014 11:46 AM, tim panton wrote:
>>>> On 28 May 2014, at 10:18, Harald Alvestrand <harald@alvestrand.no> wrote:
>>>> 
>>>>> On 05/28/2014 10:52 AM, Wolfgang Beck wrote:
>>>>>> Adding another data transfer protocol on top of SCTP wll not solve your problem.
>>>>>> 
>>>>>> The Websocket-style API is the problem.
>>>>>> It does not allow the JS to delay reception and does not tell you when it is apporpriate to send more data.
>>>>>> 
>>>>>> Sending a chunk and wait for an ACK? That means you will spend most of the time waiting for Acks instead of
>>>>>> transmitting data. Of course you can somehow negotiate how many chunks you can send without having to
>>>>>> wait for an ACK. Now you have re-implemented a substantial part of SCTP, probably with more errors and less sophistication.
>>>>>> 
>>>>>> What's wrong with the Streams API?
>>>>> The first thing wrong about the Streams API as described in the link below is that it does not preserve message boundaries; a Stream is a sequence of bytes.
>>>>> 
>>>>> Our chosen abstraction is a sequence of messages.
>>>>> 
>>>>> Something like the Streams API may be a Good Thing (and applicable to websockets too), but the current proposal just has the wrong model for our purposes.
>>>>> 
>>>>> If you have a suggestion to bridge the gap, please bring it forward.
>>>> Thinking some more about the back pressure issue, how about an optional callback on send()
>>>> onSendSpaceAvailable(int amount)
>>>> which gets called whenever it is next possible to send and the block size that is available.
>>> 
>>> I think a lot could be done if we just defined the semantics of send():
>>> 
>>> - send either succeeds fully or fails fully. There is no partial success. (Needed to ensure integrity of messages.)
>>> - the channel should stay up after a failed send (nothing got sent, this is not fatal)
>> 
>> I have a philosophical  problem with  that. Say the requested semantics of the channel are : sequenced, reliable message delivery.
>> Now you have the possibility that one of the sent() messages will never be delivered, but a subsequent one will. 
> 
> That's why I want the semantics of send() to be totally binary outcome: Either it's sent, or it's not.
> If it's not sent, it doesn't enter the sequence.

You'd need to make that abundantly clear in the spec. That is an ordered reliable channel for 'accepted' messages, not 
submitted messages. You'll also have to make send() blocking at least to the point where it can no longer fail, otherwise
the javascript could already have called the subsequent send() - possibly getting us to the position where the subsequent
(smaller?) send succeeds without the previous larger one. (Which in no way resembles ordered reliability in my book).

> 
>> If you are sending incremental changes (think database transactions) and one in the middle is dropped but the channel
>> remains operational, you have broken that semantic. 
> 
> Only if the sending code assumes that a failure is ignorable. Caveat emptor.
> (Many files have been lost to code written under the assumption that write() will always succeed... "what do you mean - disk full?")

But you are the one making it ignorable. The current spec is very clear, if a channel can't meet it's obligations it is closed, there can be 
no subsequent messages and no possibility to ignore the error and just keep sending the rest of the file.

> 
>> 
>>> - the code for "the buffer is full, try later" and "this message is too big to ever send" should be different (so that we can know the difference between "back off" and "you must be kidding")
>> 
>> perhaps that works for the unreliable case, but I don't like it for reliable. 
>> 
>>> 
>>> There's a difficulty in the callback definition you suggest - I think it needs to say what size it is waiting for. Otherwise, we get a sequence like:
>>> 
>>> -> send(20000) -> fail, temporary too big
>>> <- spaceAvailable(100)
>>> -> send(20000) -> fail
>>> <- spaceAvailable(1000)
>>> -> send(20000) -> fail
>>> 
>>> and so on. That's not right.
>> 
>> Agreed. Given that perhaps we need a bulkSend( ) which takes an array of messages or even a function that generates messages.
>> bulkSend( function(avail) { return /nextmessageOfAvailBytes} );
> 
> Now we're getting complicated :-)

But very much in the spirit of modern javascript. 

> 
>> 
>> T.
>> 
>>> 
>>> 
>>>> 
>>>> T.
>>>>  
>>>>>> Wolfgang
>>>>>> 
>>>>>> On 05/27/14 09:37, Stefan Håkansson LK wrote:
>>>>>>> This was discussed at the f2f, and the Streams API was mentioned, but as
>>>>>>> Harald pointed out yesterday the applicability of Streams with the data
>>>>>>> channel is not clear yet.
>>>>>>> 
>>>>>>> But there is another option that is supported right now. The blob
>>>>>>> (defined in http://dev.w3.org/2006/webapi/FileAPI/) supports slicing up
>>>>>>> data chunks in smaller parts (and of course re-assembling them back).
>>>>>>> So, it is quite simple to split up a large chunk in smaller ones, and
>>>>>>> then add some simple acking on the app layer (hold back sending next
>>>>>>> slice until the previous one is acked).
>>>>>>> 
>>>>>>> This is not elegant, but should work.
>>>>>>> 
>>>>>>> The quota API (https://dvcs.w3.org/hg/quota/raw-file/tip/Overview.html)
>>>>>>> allows for a bit more sophistication, but it seems to be supported by
>>>>>>> Chrome only (and then only an older version of the API).
>>>>>>> 
>>>>>>> Stefan
>> 
>