Re: MediaStream Recording : record() method

But the upshot would be that for real-time speech recognition you should
use a particular codec/protocol configuration of PeerConnection, not
MediaRecorder. The use case for MediaRecorder is more like "I'm recording a
voice memo, and don't want to deal with unreliable connections and
uncompressed bits, so I record it and send it asynchronously."

Of course it'd be nice if that encoding and upload all happened as quickly
as possible, but the sweet spot for apps is that they shouldn't have to
deal with stringency in a highly-constrained environment. (As the sweet
spot for server-side speech recognition is to do the best possible job
given the results of the PeerConnection configuration and capability.)

Or to put it another way, in use cases where MediaRecorder is the right
tool you want a consistently applied encoding producing a highly
interoperable file format. When PeerConnection is the right tool you might
switch up codecs and protocols mid-stream to give the best real-time
performance.




On Wed, Aug 28, 2013 at 11:08 AM, Mandyam, Giridhar <mandyam@quicinc.com>wrote:

>  There is an outstanding bug for WebRTC to have a TCP fallback:  see
> https://www.w3.org/Bugs/Public/show_bug.cgi?id=20818.  It has not been
> closed yet, so I assume that a TCP-based PC is still on the table for
> WebRTC.  This should be sufficient for lossless real-time data streaming in
> my opinion, and satisfy the speech recognition use case.****
>
> ** **
>
> The timeframe for when we’ll see TCP-based PC implementations is an
> unknown (at least to me).****
>
> ** **
>
> *From:* Greg Billock [mailto:gbillock@google.com]
> *Sent:* Wednesday, August 28, 2013 10:40 AM
> *To:* Robert O'Callahan
> *Cc:* public-media-capture@w3.org
> *Subject:* Re: MediaStream Recording : record() method****
>
> ** **
>
> ** **
>
> ** **
>
> On Mon, Aug 26, 2013 at 6:46 PM, Robert O'Callahan <robert@ocallahan.org>
> wrote:****
>
> On Tue, Aug 27, 2013 at 12:36 PM, Greg Billock <gbillock@google.com>
> wrote:****
>
>  First, I think the recording API should be tuned for latent behavior, so
> that the recording product can be stored to long-term storage. That is, an
> app shouldn't expect to be able to get real-time recording from the API.
> For use cases where that's important, we have PeerConnection.****
>
>  ** **
>
> Depends on what you mean by "real-time recording". Using MediaRecorder to
> losslessly stream data as close to real time as possible is a valid
> use-case --- our WebRTC people have told me that lossless PeerConnection is
> not going to happen anytime soon, if ever.****
>
> ** **
>
> Real-time to me means "adapting lossy-ness to ensure minimum latency" --
> that is, what PeerConnection does, and why it may will never support
> uniform compression methods.****
>
> ** **
>
> On an unloaded high-end machine, I agree that the MediaRecorder ought to
> produce very consistent and relatively low-latency callbacks so the stream
> will end up being pretty real-time. Our plan in Chrome is to buffer through
> long-term storage to make handling the pressures of the recording process
> consistently and make constraints on the app using the API less stringent.
> That approach is optimized for non-real-time recording, which I think is
> fine, as that's the sweet spot for the API. The advice would be "if you're
> using MediaRecorder to essentially write your own PeerConnection, you're
> doing it wrong." On the other hand, if you have a good system and a good
> connection, you should be able to, for example, upload a recorded stream to
> cloud storage in quasi-real-time with good responsivity.****
>
> ** **
>
> ** **
>
>  ****
>
>     ****
>
>  This means that not all timeslice values will be honored. We could look
> at language like the setTimeout() spec uses -- where timeout values set
> inside a nested setTimeout won't be honored if they are greater than 4ms.
> ****
>
> ** **
>
> Here we could use something more UA-specific, such that either the
> timeslice value is to be regarded as a hint, or just say up-front that
> sufficiently low values of |timeslice| won't be honored, and some
> UA-specific minimum will be used instead. It'd be good to mention in the
> spec that apps should not attempt to rely on the API for real-time behavior.
> ****
>
> ** **
>
> Concrete edit:
>
> 3. If the timeSlice argument has been provided, then once timeSlice
> milliseconds of data have been collected, raise a dataavailable event
> containing the Blob of collected data, and start gathering a new Blob of
> data. Otherwise (if timeSlice has not been provided), continue gathering
> data into the original Blob.
>
> could become:
>
> 3. If the timeSlice argument has been provided, then once at least
> timeSlice milliseconds of data have been collected, or some minimum time
> slice imposed by the user agent, whichever is greater, raise a
> dataavailable event containing the Blob of collected data, and start
> gathering a new Blob of data. Otherwise (if timeSlice has not been
> provided), continue gathering data into the original Blob. Callers should
> not rely on exactness of the timeSlice value, especially if the timeSlice
> value is small. Callers should consider timeSlice as a minimum value.****
>
>  ** **
>
> That sounds reasonable to me.****
>
>  Shouldn't this be a "should" requirement? The UA should maintain
> fidelity to the inputs, but suppose an app configures the recorder to
> accept multiple superimposed audio tracks into a format that doesn't
> support such superposition. That seems like a pretty nice use case
> (web-based audio editing) that apps would want to deliberately support. **
> **
>
> ** **
>
> The original tracks make it sound like playback would produce either the
> same JS objects that went in, or ones with the same bits. I don't think
> this is the intended interpretation. Perhaps a better interpretation is
> that the recorder should represent all recorded tracks in the output. (Not,
> for example, just the first track in a set.)****
>
> ** **
>
> Concrete edit:****
>
> ** **
>
> "The UA should record the MediaStream in such a way that all compatible
> Tracks in the original are represented at playback time. The UA should do
> so in the highest fidelity to the input track composition which it can,
> given recording options and output format."****
>
>   ****
>
> That sounds reasonable to me too.****
>
>
> Rob
> -- ****
>
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
> le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w  ****
>
>  ** **
>