Re: A proposal for putting QUIC streams into ORTC

I think a low-level SCTP primitive would make sense, but SCTP and QUIC are
different enough that low-level primitive for SCTP would be very different
than a QuicStream.

I think it would look something like this:

interface SctpAssociation {
  void send(Uint8Array data, SctpSendParameters parameters);
  attribute EventHandler onrecv;  // SctpReceivedMessage;
}

dictionary SctpSendParameters {
  unsigned short sid;
  unsigned long ppid;
  boolean unordered;
  int max_rtx_count;
  int max_rtx_ms;
}

dictionary SctpReceivedMessage {
  unsigned short sid;
  unsigned long ppid;
  unsigned short seqnum;
  unsigned long timestamp;
  Uint8Array data;
}


Everything else (data channels, streams, etc) could be built from there.  I
don't think it's a problem if QUIC and SCTP have different low-level
primitives.


On Thu, Jul 13, 2017 at 6:02 AM Lennart Grahl <lennart.grahl@gmail.com>
wrote:

> I really like the concept as I believe that we need a streaming API for
> arbitrary data transfer in the WebRTC environment. And I agree that an
> optional low-level approach is better than confronting application
> developers with low-level buffering techniques when they just don't need
> it (this is where the existing data channel API makes sense).
>
> However, the concept itself is applicable to the SCTP transport as well.
> So, personally, I believe it would make sense if we would have a unified
> DataStream (renamed from your QuicStream proposal) that can work on top
> of either the SctpTransport or the QuicTransport, so we do not have
> separate APIs for SCTP and QUIC. The DataChannel would then work on top
> of the DataStream.
>
> Yes, for such an API, we will have to look at technical differences
> between SCTP and QUIC. If these are too large for a direct DataStream
> interface, the QuicStream and the SctpStream interfaces could be just
> another layer in between. But I believe a well-tailored API should be
> possible as there are APIs such as the NEAT project's API
> (https://neat.readthedocs.io/en/latest/tutorial.html) which unify
> transport protocols with even larger differences (granted, they don't
> need the concept of streams but both QUIC and SCTP seem to have it).
>
> Maybe we should even consider creating WHATWG streams in an extended
> form (that support the concept of unreliable and unordered data
> transfer) directly, so the concept and logic of data streams, buffering,
> back-pressure, etc., can be applied to other transport protocols in the
> future and they don't need to reinvent the wheel.
>
> I'd be happy to participate in creating such an API.
>
> Cheers,
> Lennart
>
>
> On 13.07.2017 13:42, Peter Thatcher wrote:
> > There has been a lot of interest in adding QUIC-based data channels to
> > ORTC, and it seems like a natural fit to layer some new QuicTransport on
> > top of an IceTransport to produce data channels the same way we do form
> an
> > SctpTransport layered on top of a DtlsTransport on top of an
> IceTransport.
> >
> > But as I have spent time thinking about it and designing it, I realized
> > that there is a more fundamental primitive that we can expose, slightly
> > lower-level than data channels.  And if we give that primitive an API, it
> > would not only be possible to implement data channels on top, it would be
> > possible to build many other things on top in a way that would be more
> > simple, powerful, and flexible at the same time.
> >
> > That powerful fundamental primitive is the QUIC stream.
> >
> > Here's as simplified version of the API I propose (I slightly more full
> one
> > a little later), centered around QUIC streams:
> >
> > [Constructor(IceTransport)]
> > partial interface QuicTransport {
> >   QuicStream createStream();
> >   attribute EventHandler onstream; // QuicStream from remote side
> > };
> >
> > partial interface QuicStream {
> >   // If there is space in the local buffer, write to the buffer.
> >   // Otherwise, fail.
> >   void write(Uint8Array data);
> >
> >   // Will maybe write one last value before finishing
> >   // and then a "fin" bit.
> >   void finish(optional Uint8Array data);
> >
> >   // The remote side has acked everything, including the "fin" bit
> >   // Which means it is safe to close the stream.
> >   readonly attribute EventHandler onacked;
> >
> >   // Reads all that is in the buffer.
> >   Uint8Array read();
> >
> >   // Fills the array exactly from the read buffer.
> >   // If that amount is not available, it fails.
> >   void readInto(Uint8Array data);
> >
> >   // Closing on either side trigger on close on both sides.
> >   // Once closed, all future reads and writes will fail.
> >   // Nothing will be sent or received any longer.
> >   // Everything buffered will be thrown away.
> >   // Closing an already closed stream does nothing.
> >   void close();
> >   attribute EventHandler onclosed;
> > }
> >
> >
> > Basically, you can write, read, and close.   You can also "finish", know
> > when the remote side has "finished" and know when the remote side has
> acked
> > everything.
> >
> >>From here, as I'll show with some examples, you can send and receive
> small
> > message out of order or with time-bounded reliability.  You can also send
> > and receive large messages with back pressure.  You can build data
> > channels.  Or you can even build WHATWG streams.
> >
> > But there's just one last piece to the puzzle to explain: buffering.  In
> > order for back pressure to work properly and easily we need a way of
> > causing the app to wait until there is room in a limited-size buffer to
> > send and receive.    In the examples, you'll see how this is used, and
> then
> > I'll show the methods to accomplish this.
> >
> > Example of sending small unreliable/unordered messages with a short
> timeout:
> > let ice = ...;
> > let quic = new QuicTransport(ice);
> > let maxMessageSize = 4096;
> > let timeout = 5000;
> >
> > // Write outgoing message
> > let data = ...;
> > let qstream = quic.createStream();
> > await qstream.waitForWritable(data.byteLength);
> > qstream.finish(data);
> > setTimeout(() => qstream.close(), timeout);
> >
> > // Read incoming messages.
> > quic.onstream = qstream => {
> >   await qstream.waitForReadableOrFinished(maxMessageSize);
> >   let data = qstream.read();
> >   // Do something with data.
> > };
> >
> >
> > Example of sending a large message in chunks, with back pressure
> > let ice = ...;
> > let quic = new QuicTransport(ice);
> >
> > // Write out in chunks
> > let qstream = quic.createStream();
> > let chunks = ...;
> > for chunk in chunks {
> >   await qstream.waitForWritable(chunk.byteLength);
> >   qstream.write(chunk);
> > }
> >
> > // Read in the chunks
> > quic.onstream = qstream => {
> >   await qstream.waitForReadableOrFinished();
> >   while (!qstream.finished) {
> >     let data = qstream.read();
> >     // .... Use the data somewhere
> >     await qstream.waitForReadableOrFinished();
> >   }
> > }
> >
> >
> >
> > Building reliable, ordered DataChannels on top of streams:
> >
> > A reliable, ordered DataChannel can be implemented on top of a single
> > QuicStream with a simple framing mechanism to insert messages into the
> > stream.  For example, a framing mechanism like the following could be
> used:
> >
> > DataChannel.send writes the following to the stream:
> > - 2 bits to indicate the type (0x01 == string; 0x10 == binary)
> > - 2 bits for the number of additional bytes (N) for the length of the
> > message payload (0x00 = 0, 0x01 = 1, 0x10 = 2, 0x11 = 4)
> > - 4 + (8 * N) bits for the length of the message (M)
> > - M bytes for the message
> >
> >
> > Building unreliable or unordered DataChannels:
> >
> > To  send unordered or unreliable messages over QUIC, multiple QUIC
> streams
> > must be used, perhaps one for each message.  A DataChannel could be built
> > which creates a new QUIC stream for each message which writes a data
> > channel ID as metadata first and then a message write after.  However, an
> > application will typically simply skip this level of abstraction and just
> > use QuicStreams directly for sending and receiving unreliable or
> unordered
> > messages, since it is more simple that way.
> >
> >
> > WHATWG streams:
> >
> > WHATWG streams wrap an "underlying source" to create ReadableStreams and
> > WritableStreams which provide some convenient functionality.  We can
> create
> > "underlying sources" from QuicStreams with something like this:
> >
> > var qstream = quic.createStream();
> > new ReadableStream(
> >   {
> >     type: "bytes",
> >     start: function(controller) {
> >       qstream.onclose = () => controller.close();
> >     },
> >     pull: async function(controller) {
> >       if (controller.byobRequest) {
> >         var data = controller.byobRequest.view;
> >         await qstream.waitForReadableOrFinished(data.byteLength);
> >         if (qstream.finished) {
> >           let finishedData = qstream.read()
> >           copyArray(finishedData, data);
> >           controller.byobRequest.respond(finishedData.byteLength);
> >         } else {
> >           qstream.readInto(data);
> >           controller.byobRequest.respond(data.byteLength);
> >         }
> >       } else {
> >         await qstream.waitForReadableOrFinished();
> >         controller.enqueue(qstream.read());
> >       }
> >     },
> >     cancel: function() {
> >       qstream.close();
> >     }
> >   },
> >   new ByteLengthQueuingStrategy({ highWaterMark: 4096 })
> > );
> >
> > new WritableStream(
> >   {
> >     type: "bytes",
> >     start: function (controller) {
> >       qstream.onclose = () => controller.close();
> >     },
> >     write: async function(chunk, controller) {
> >       await qstream.waitForWritableAmount(chunk.byteLength);
> >       qstream.write(chunk);
> >     }
> >     close: async function() {
> >       qstream.finish();
> >       await waitForEvent(qstream.onacked);
> >       qstream.close();
> >     }
> >     abort: function(reason) {
> >       qstream.close();
> >     }
> >   },
> >   new ByteLengthQueuingStrategy({ highWaterMark: 4096 })
> > );
> >
> >
> > Finally, here are the two buffering methods I mentioned that are need to
> > make the back pressure and buffer work correctly:
> >
> > partial interface QuicStream {
> >   // Resolves when the amount can be written or buffered,
> >   // Up to the maxBufferedAmount (== amount if not given)
> >   Promise waitForWritable(unsigned long amount,
> >                                           optional unsigned long
> > maxBufferedAmount);
> >
> >   // Resolves when the amount can be written or buffered,
> >   // Up to the maxBufferedAmount (== amount if not given)
> >   // Will also resolved if the stream finished.
> >   Promise waitForReadableOrFinished(
> >     unsigned long amount, optional unsigned long maxBufferedAmount);
> > }
> >
> >
> > I think this approach of QUIC streams with data channels and other
> > abstractions built on top will be flexible, powerful, and about as simple
> > as it can get.
> >
>