- From: Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com>
- Date: Mon, 2 Oct 2017 23:34:43 +0200
- To: Lennart Grahl <lennart.grahl@gmail.com>
- Cc: "public-ortc@w3.org" <public-ortc@w3.org>
- Message-ID: <CA+ag07Z8AK4YfRL35NBwXDGDOAMdgQ8e3PK=MSMgpQxMMZfygQ@mail.gmail.com>
Hi Lennart, welcome! ;)
I agree with your diagnosis.
I don't have an use case in mind, but I think we have to clearly specify
what is the expected behavior in this case. After giving it a second
thought, maybe it is just enough to recommend not sending big messages
because they can produce the effect described before and produce a waste of
bandwidth. After all if you send a 1400bytes message and it is split in two
quic packets and one of them is lost, it is not much of a difference as if
it was just sent in one quic packet and it was lost.
Best regards
Sergio
2017-10-02 22:55 GMT+02:00 Lennart Grahl <lennart.grahl@gmail.com>:
> Hi Sergio,
>
> sorry for jumping in here. :)
>
> What is the "maximum-packet-size" by your definition? Let's take
> SCTP-based data channels for an example: If you have a completely
> unreliable channel (by that, I mean nothing is being retransmitted) and
> you don't want to waste any bandwidth, then your message is limited by
>
> mtu(interface) - headroom_for([udp, dtls, sctp])
>
> which is the available payload size per chunk at a specific point in
> time. So, the message's size would have to be less or equal to that. Is
> that the information you want to retrieve? Be aware that this value
> isn't constant. For some implementations, it may even change in between
> the "send" call and the actual transmission of the data because the
> underlying UDP socket has changed. Things may even get more complicated
> when diving into the topic of chunk bundling (which both SCTP and QUIC
> do as far as I can tell).
>
> A question about your use case: In my mind, unreliable channels will
> usually be used for time-related information such as the current
> temperature of some device. Taking into consideration that large
> messages will lead to the problem you've stated, I'd compress that
> information but that's about it. It's not really dynamically sizeable
> data unless I have bundled update information that *could* be sent
> independently (for example, data from a bunch of sensors). Is this the
> use case you have? If yes, why bundle this information in the first
> place? If not, could you elaborate? (Note that I'm not questioning the
> usefulness of your proposal to be able to retrieve that value via the
> API. Just trying to understand the use case.)
>
> Cheers,
> Lennart
>
> On 02.10.2017 20:51, Sergio Garcia Murillo wrote:
> > Hi Bernard,
> >
> > One question regarding unreliable QUIC streams (which could be applied to
> > current SCTP DC).
> >
> > If I read correctly the draft, my understanding is that we will have to
> use
> > one QUIC stream per message sent over an unreliable&unordered DC. The
> > problem that I see is that that message will be then chunked into one or
> > several quic packets and those quic packets are the ones that are
> > unreliable.
> >
> > The problem is what happens if we send a 65kb message over the DC, which
> > (for example) is chunked in 65 1kb packets, and only one is lost? I don't
> > think we should provide an API to allow retrieving the individual byte
> > ranges, so we most probably end up discarding the whole message (wasting
> > bandwidth).
> >
> > You would say that that is an unlikely to happen, as devs will fine tune
> > their messages sizes to be sent over a single packet, which I agree.
> > Currently, this the approach on SCTP DC, but this value has been
> calculated
> > by trial&error and reverse engineering, so I think that to do things
> > properly, we should provide in the API what is the "max-packet-size" that
> > ensures that the message will fit into a single packet.
> >
> > Please correct me if anything is wrong.
> > Sergio
> >
> > 2017-10-02 20:12 GMT+02:00 Bernard Aboba <Bernard.Aboba@microsoft.com>:
> >
> >> Specification work on the QUIC protocol is continuing within the IETF
> QUIC
> >> WG, and a number of implementations are now in progress.
> >>
> >> With the QUIC WG hosting an interim meeting in Seattle this week, I
> >> thought I'd summarize where we are with QUIC support in the ORTC API,
> and
> >> mention a few recent developments that may be relevant to that work.
> >>
> >> At this point, we have incorporated an RTCQuicTransport interface into
> the
> >> ORTC API:
> >> http://draft.ortc.org/#quic-transport*
> >>
> >> However, while the current "Big Picture" mentions QuicStreams, we do not
> >> yet have a section defining the QuicStream API.
> >>
> >> In addition to Peter's proposal for a QuicStream API, EKR has recently
> >> posted a (C) QUIC API proposal on the QUIC WG mailing list, which bears
> >> some similarity to Peter's proposal:
> >> https://www.ietf.org/mail-archive/web/quic/current/msg02295.html
> >>
> >> So far, work in the IETF QUIC WG has focused on defining QUIC support
> for
> >> reliable transport:
> >> https://tools.ietf.org/html/draft-ietf-quic-transport
> >>
> >> This has lead to some questions about how QUIC can be used for
> unreliable
> >> transport (e.g. hosting an unreliable RTCDataChannel interface over
> QUIC).
> >> Recently a draft has been submitted on the subject of unreliable
> transport:
> >> https://tools.ietf.org/html/draft-tiesel-quic-unreliable-streams
> >>
> >> Aside from these notes of progress, there is at least one major sticking
> >> point. Multiplexing of QUIC with other WebRTC protocols (e.g. RTP,
> RTCP,
> >> STUN, TURN, DTLS, ZRTP, etc.) has not yet been defined. Here is a
> link to
> >> the issue raised in the QUIC WG. An initial proposal (involving
> changes to
> >> the header bits) was rejected:
> >> https://github.com/quicwg/base-drafts/issues/426
> >>
> >> -----Original Message-----
> >> From: Lennart Grahl [mailto:lennart.grahl@gmail.com]
> >> Sent: Thursday, July 13, 2017 6:02 AM
> >> To: Peter Thatcher <pthatcher@google.com>
> >> Cc: public-ortc@w3.org
> >> Subject: Re: A proposal for putting QUIC streams into ORTC
> >>
> >> I really like the concept as I believe that we need a streaming API for
> >> arbitrary data transfer in the WebRTC environment. And I agree that an
> >> optional low-level approach is better than confronting application
> >> developers with low-level buffering techniques when they just don't
> need it
> >> (this is where the existing data channel API makes sense).
> >>
> >> However, the concept itself is applicable to the SCTP transport as well.
> >> So, personally, I believe it would make sense if we would have a unified
> >> DataStream (renamed from your QuicStream proposal) that can work on top
> of
> >> either the SctpTransport or the QuicTransport, so we do not have
> separate
> >> APIs for SCTP and QUIC. The DataChannel would then work on top of the
> >> DataStream.
> >>
> >> Yes, for such an API, we will have to look at technical differences
> >> between SCTP and QUIC. If these are too large for a direct DataStream
> >> interface, the QuicStream and the SctpStream interfaces could be just
> >> another layer in between. But I believe a well-tailored API should be
> >> possible as there are APIs such as the NEAT project's API
> >> (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fneat.
> >> readthedocs.io%2Fen%2Flatest%2Ftutorial.html&data=02%7C01%
> >> 7CBernard.Aboba%40microsoft.com%7C12990010786e425a9e0108d4c9efe71f%
> >> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636355479601344448&sdata=
> >> SXf5IbQdfrtnB547ymZCnHrCjX5XRc77BPkJAzHVzH8%3D&reserved=0) which unify
> >> transport protocols with even larger differences (granted, they don't
> need
> >> the concept of streams but both QUIC and SCTP seem to have it).
> >>
> >> Maybe we should even consider creating WHATWG streams in an extended
> form
> >> (that support the concept of unreliable and unordered data
> >> transfer) directly, so the concept and logic of data streams, buffering,
> >> back-pressure, etc., can be applied to other transport protocols in the
> >> future and they don't need to reinvent the wheel.
> >>
> >> I'd be happy to participate in creating such an API.
> >>
> >> Cheers,
> >> Lennart
> >>
> >>
> >> On 13.07.2017 13:42, Peter Thatcher wrote:
> >>> There has been a lot of interest in adding QUIC-based data channels to
> >>> ORTC, and it seems like a natural fit to layer some new QuicTransport
> >>> on top of an IceTransport to produce data channels the same way we do
> >>> form an SctpTransport layered on top of a DtlsTransport on top of an
> >> IceTransport.
> >>>
> >>> But as I have spent time thinking about it and designing it, I
> >>> realized that there is a more fundamental primitive that we can
> >>> expose, slightly lower-level than data channels. And if we give that
> >>> primitive an API, it would not only be possible to implement data
> >>> channels on top, it would be possible to build many other things on
> >>> top in a way that would be more simple, powerful, and flexible at the
> >> same time.
> >>>
> >>> That powerful fundamental primitive is the QUIC stream.
> >>>
> >>> Here's as simplified version of the API I propose (I slightly more
> >>> full one a little later), centered around QUIC streams:
> >>>
> >>> [Constructor(IceTransport)]
> >>> partial interface QuicTransport {
> >>> QuicStream createStream();
> >>> attribute EventHandler onstream; // QuicStream from remote side };
> >>>
> >>> partial interface QuicStream {
> >>> // If there is space in the local buffer, write to the buffer.
> >>> // Otherwise, fail.
> >>> void write(Uint8Array data);
> >>>
> >>> // Will maybe write one last value before finishing
> >>> // and then a "fin" bit.
> >>> void finish(optional Uint8Array data);
> >>>
> >>> // The remote side has acked everything, including the "fin" bit
> >>> // Which means it is safe to close the stream.
> >>> readonly attribute EventHandler onacked;
> >>>
> >>> // Reads all that is in the buffer.
> >>> Uint8Array read();
> >>>
> >>> // Fills the array exactly from the read buffer.
> >>> // If that amount is not available, it fails.
> >>> void readInto(Uint8Array data);
> >>>
> >>> // Closing on either side trigger on close on both sides.
> >>> // Once closed, all future reads and writes will fail.
> >>> // Nothing will be sent or received any longer.
> >>> // Everything buffered will be thrown away.
> >>> // Closing an already closed stream does nothing.
> >>> void close();
> >>> attribute EventHandler onclosed;
> >>> }
> >>>
> >>>
> >>> Basically, you can write, read, and close. You can also "finish",
> know
> >>> when the remote side has "finished" and know when the remote side has
> >>> acked everything.
> >>>
> >>> >From here, as I'll show with some examples, you can send and receive
> >>>> small
> >>> message out of order or with time-bounded reliability. You can also
> >>> send and receive large messages with back pressure. You can build
> >>> data channels. Or you can even build WHATWG streams.
> >>>
> >>> But there's just one last piece to the puzzle to explain: buffering.
> >>> In order for back pressure to work properly and easily we need a way
> >>> of causing the app to wait until there is room in a limited-size buffer
> >> to
> >>> send and receive. In the examples, you'll see how this is used, and
> >> then
> >>> I'll show the methods to accomplish this.
> >>>
> >>> Example of sending small unreliable/unordered messages with a short
> >> timeout:
> >>> let ice = ...;
> >>> let quic = new QuicTransport(ice);
> >>> let maxMessageSize = 4096;
> >>> let timeout = 5000;
> >>>
> >>> // Write outgoing message
> >>> let data = ...;
> >>> let qstream = quic.createStream();
> >>> await qstream.waitForWritable(data.byteLength);
> >>> qstream.finish(data);
> >>> setTimeout(() => qstream.close(), timeout);
> >>>
> >>> // Read incoming messages.
> >>> quic.onstream = qstream => {
> >>> await qstream.waitForReadableOrFinished(maxMessageSize);
> >>> let data = qstream.read();
> >>> // Do something with data.
> >>> };
> >>>
> >>>
> >>> Example of sending a large message in chunks, with back pressure let
> >>> ice = ...; let quic = new QuicTransport(ice);
> >>>
> >>> // Write out in chunks
> >>> let qstream = quic.createStream();
> >>> let chunks = ...;
> >>> for chunk in chunks {
> >>> await qstream.waitForWritable(chunk.byteLength);
> >>> qstream.write(chunk);
> >>> }
> >>>
> >>> // Read in the chunks
> >>> quic.onstream = qstream => {
> >>> await qstream.waitForReadableOrFinished();
> >>> while (!qstream.finished) {
> >>> let data = qstream.read();
> >>> // .... Use the data somewhere
> >>> await qstream.waitForReadableOrFinished();
> >>> }
> >>> }
> >>>
> >>>
> >>>
> >>> Building reliable, ordered DataChannels on top of streams:
> >>>
> >>> A reliable, ordered DataChannel can be implemented on top of a single
> >>> QuicStream with a simple framing mechanism to insert messages into the
> >>> stream. For example, a framing mechanism like the following could be
> >> used:
> >>>
> >>> DataChannel.send writes the following to the stream:
> >>> - 2 bits to indicate the type (0x01 == string; 0x10 == binary)
> >>> - 2 bits for the number of additional bytes (N) for the length of the
> >>> message payload (0x00 = 0, 0x01 = 1, 0x10 = 2, 0x11 = 4)
> >>> - 4 + (8 * N) bits for the length of the message (M)
> >>> - M bytes for the message
> >>>
> >>>
> >>> Building unreliable or unordered DataChannels:
> >>>
> >>> To send unordered or unreliable messages over QUIC, multiple QUIC
> >>> streams must be used, perhaps one for each message. A DataChannel
> >>> could be built which creates a new QUIC stream for each message which
> >>> writes a data channel ID as metadata first and then a message write
> >>> after. However, an application will typically simply skip this level
> >>> of abstraction and just use QuicStreams directly for sending and
> >>> receiving unreliable or unordered messages, since it is more simple
> that
> >> way.
> >>>
> >>>
> >>> WHATWG streams:
> >>>
> >>> WHATWG streams wrap an "underlying source" to create ReadableStreams
> >>> and WritableStreams which provide some convenient functionality. We
> >>> can create "underlying sources" from QuicStreams with something like
> >> this:
> >>>
> >>> var qstream = quic.createStream();
> >>> new ReadableStream(
> >>> {
> >>> type: "bytes",
> >>> start: function(controller) {
> >>> qstream.onclose = () => controller.close();
> >>> },
> >>> pull: async function(controller) {
> >>> if (controller.byobRequest) {
> >>> var data = controller.byobRequest.view;
> >>> await qstream.waitForReadableOrFinished(data.byteLength);
> >>> if (qstream.finished) {
> >>> let finishedData = qstream.read()
> >>> copyArray(finishedData, data);
> >>> controller.byobRequest.respond(finishedData.byteLength);
> >>> } else {
> >>> qstream.readInto(data);
> >>> controller.byobRequest.respond(data.byteLength);
> >>> }
> >>> } else {
> >>> await qstream.waitForReadableOrFinished();
> >>> controller.enqueue(qstream.read());
> >>> }
> >>> },
> >>> cancel: function() {
> >>> qstream.close();
> >>> }
> >>> },
> >>> new ByteLengthQueuingStrategy({ highWaterMark: 4096 }) );
> >>>
> >>> new WritableStream(
> >>> {
> >>> type: "bytes",
> >>> start: function (controller) {
> >>> qstream.onclose = () => controller.close();
> >>> },
> >>> write: async function(chunk, controller) {
> >>> await qstream.waitForWritableAmount(chunk.byteLength);
> >>> qstream.write(chunk);
> >>> }
> >>> close: async function() {
> >>> qstream.finish();
> >>> await waitForEvent(qstream.onacked);
> >>> qstream.close();
> >>> }
> >>> abort: function(reason) {
> >>> qstream.close();
> >>> }
> >>> },
> >>> new ByteLengthQueuingStrategy({ highWaterMark: 4096 }) );
> >>>
> >>>
> >>> Finally, here are the two buffering methods I mentioned that are need
> >>> to make the back pressure and buffer work correctly:
> >>>
> >>> partial interface QuicStream {
> >>> // Resolves when the amount can be written or buffered,
> >>> // Up to the maxBufferedAmount (== amount if not given)
> >>> Promise waitForWritable(unsigned long amount,
> >>> optional unsigned long
> >>> maxBufferedAmount);
> >>>
> >>> // Resolves when the amount can be written or buffered,
> >>> // Up to the maxBufferedAmount (== amount if not given)
> >>> // Will also resolved if the stream finished.
> >>> Promise waitForReadableOrFinished(
> >>> unsigned long amount, optional unsigned long maxBufferedAmount); }
> >>>
> >>>
> >>> I think this approach of QUIC streams with data channels and other
> >>> abstractions built on top will be flexible, powerful, and about as
> >>> simple as it can get.
> >>>
> >>
> >>
> >>
> >
>
Received on Monday, 2 October 2017 21:35:08 UTC