RE: MediaRecorder and using Streams

If the Stream API is unstable, as Anne indicates, then it's too soon to move to it.  This doesn't mean that we couldn't do so later.  We're in a hurry to get out Media Capture and WebRTC 1.0, but MediaRecorder will lag behind, so there'll be a longer time in which to consider changes.

Most of your arguments for the Stream API seem to be based on implementation considerations inside the UA,  so I'm not qualified to evaluate them.  Genesys will be a consumer, not an implementer of MediaRecorder.  Our main use case is grabbing buffers of audio data and sending it off to a remote speech recognizer.  For that purpose, I find the Blob API/dataavailable event somewhat easier to use.  You call record specifying the buffer size (200ms is a fairly common sample size for speech rec engines), and then handle the dataavailable events as they arrive, stuffing the data into a socket.  With the Streams API, it looks like you have to keep calling readAsX and then blocking till the appropriate amount of data becomes available.  That's not a fatal objection, of course, but it may give you some idea why we structured the API they way we did.

I'd like to hear what other implementers think of this.


-          Jim

From: groby@google.com [mailto:groby@google.com] On Behalf Of Rachel Blum
Sent: Wednesday, July 31, 2013 3:06 PM
To: public-media-capture@w3.org
Subject: MediaRecorder and using Streams


Greg Billock and I have been taking a look at the MediaRecorder API for Chromium, and we were wondering if there's a reason MediaRecorder doesn't use the Streams API. Intuitively, it seems to make sense, since the end product is an encoded video stream.


*         There seem to be two different use-cases, sliced recording and one-shot recording

*         These two cases face different resource constraints. Sliced recording is presumably used mostly for streaming purposes, and thus a bit more sensitive to resource constraints.

*         The current API can easily create an unbounded number of blobs.

*         The use of blobs in ondataavailable raises the interesting question of the proper mime type for each blob.

*         The possibility automatically switching to disk-backed blobs raises another interesting question, that of quota interaction.


Given these issues, we've looked at an alternative approach using Streams instead. Specifically, the API would be restructured to allow the following calls:


Stream recordToStream(optional StreamBuilder b);

void recordToBlob();


recordToBlob covers the use case of recording in one go - it will simply have a finalized Blob available when onstop() is fired.


recordToStream() offers the ability for timesliced recording. It will build a Stream populated with the encoded data, either using an internally constructed StreamBuilder or using the passed-in StreamBuilder.


Here are the advantages as we see them:


1) It separates out the two use cases of sliced vs. non-sliced recording. The UA is in full control of use of memory vs. disk storage for the one-shot recording..

2) It is more resilient against accidental blob leakage. Instead of having to decide for each blob if it should have disk backing or not, this is up to the Stream implementation. Users are guided more naturally into not creating code that queues up blobs that it fails to consume at sufficient speed - this buffering is all more naturally left to the Stream object.

2a) That means the UA can implement "smart" streams that react accordingly to resource constraints - discard data up to the next I-frame, replace frames with small placeholder frames to indicate they were skipped, etc.

3) Users can still get more control over the creation of the stream if they pass a StreamBuilder.

3a) A lot of the error handling and event processing move into the stock File API object.

3b) A user-built StreamBuilder writing to long-term storage has more predictable interaction with the quota system for disk-stored recordings without needing to move the Blob resulting from one-shot recording.

4) There is no type mystery any more - there is exactly one definitive type, and it is stored on the Stream. No need to deal with data fragments of unknowable type.


If recordToStream is not given the optional StreamBuilder parameter, the UA can use an internal implementation.

Would love to hear your thoughts,
- rachel