Re: Assigning MediaStream objects to HTML media elements

Rich,

thanks for good input! There is some more comment towards the end of 
this mail.

On 12/13/2011 12:58 AM, Rich Tibbett wrote:
> Nine months ago we (Opera) were in the middle of some early
> experimentation work implementing an early proposal of the getUserMedia
> proposal. We found that we had to patch the then-current WHATWG<device>
> proposal with a number of additional API behaviors for when HTML media
> elements are hosting Media Stream objects.
>
> In our early specification we defined the term 'stream mode' for when
> such DOM elements are assigned Stream objects for the purposes of local
> media playback. We documented the behavior of the DOM APIs for HTML
> media elements operating in stream mode in [1].
>
> One of the key concepts we established is that MediaStream objects, when
> applied to audio/video elements, should act within a 'linear media
> timeline' model (also a defined term in [1]). Such a timeline is, for
> example, not seekable and has well-defined behavior in the case that the
> video element is paused and resumed. When a developer interfaces with
> the DOM APIs representing HTML Media elements, we define specific setter
> and getter behaviors to switch off things that don't make sense in a
> 'linear media timeline' model.
>
> For example, a developer cannot adjust the playbackRate of a live
> stream, captured from a user's webcam. I think we need to review and add
> these behaviors to the current specification to ensure we end up with
> consistent behavior across UAs implementing our future recommendation.
>
> In the course of our early prototyping and also as part of the provided
> spec [1] we also introduced the ability to assign Stream objects
> directly to the .src of a video or audio HTML element (e.g. video.src =
> stream). This was not included in the official WHATWG draft at the time
> (since we didn't have window.URL.createObjectURL at that time and no-one
> really knew how else we could do it).
>
> Right now in the W3C proposal developers are expected to indirectly mint
> a new temporary URL (via window.URL.createObjectURL) to assign a
> MediaStream to a video/audio element. To coincide with direct assignment
> of a Stream object to a video element, our work also defined how Stream
> objects that have been assigned to HTML media elements should be labeled
> if or when a developer attempts to resolve the src URL/URI. We settled
> on 'about:streamurl' to be a reserved, though unresolvable, about: URI,
> to indicate that the media element is displaying/playing an object that
> implements the MediaStream interface (and hence the media element is in
> stream mode and has all the behaviors of that mode as defined in [1]).
>
> This direct object assignment requires less code to assign Stream
> objects to video/audio elements. It works really well to date in all
> Opera implementations and we'd like to apply this behavior to the W3C
> spec pending further discussion and feedback.
>
> There are notably a number of things we might want to also discuss, such
> as what we should do if particular attributes (i.e. autoplay, loop) are
> included on any HTML media element declaration (but I'd assume they mean
> nothing in stream mode). The documentation in [1] should be a reasonable
> starting point for that discussion also.
>
> I notice that roc's MediaStream API Processing proposal [2] touches on a
> few of these aspects to some degree also, albeit in less formal prose
> right now. I guess we're all thinking along the same thing and we just
> need to define the exact terms and algorithms in our specification.
>
> Any feedback would be appreciated. The main question is: should we add
> such behaviors as these to our specification or not?
Personally, I have been thinking along the lines that we must add 
something like the table in [1] to explain what attributes and data of 
the media elements [3] that have a meaning for streams, and how they 
should be interpreted.

And [1] seems like a good starting point indeed.

Then, I must admit, I saw this task as something for the webrtc wg 
(cc'd) rather than for this TF.

Br,
Stefan

>
> br/ Rich
>
> [1]
> http://people.opera.com/richt/release/specs/device/o-device.html#stream-mode-behavior
>
> [2]
> https://dvcs.w3.org/hg/audio/raw-file/tip/streams/StreamProcessing.html#mediastreams
>
[3] http://dev.w3.org/html5/spec/Overview.html#media-elements

Received on Tuesday, 13 December 2011 15:04:27 UTC