Re: TV Control API and getUserMedia

Thanks for your reply, and apologies for the delay in following up.

A few comments inline below:

> Thanks for the note!
> 
> Speaking as individual:
> 
> On 12/16/2016 03:02 PM, Francois Daoust wrote:
> > Dear Media Capture task force,
> >
> > The TV Control Working Group is re-modeling the TV Control API specification around sources. That new model is well aligned with the model used in Media Capture and Streams. In fact, it would seem possible to re-use Media Capture and Streams interfaces as-is. However, some details do not seem entirely right when we do that, and we would like to get your feedback on the following questions.
> >
> > Context
> > -----
> > The TV Control API specification defines an API for sourcing audio and video media, such as TV and radio from broadcast, IPTV, or other sources, and allows presentation of the media using the <video> and <audio> HTML elements:
> >
> >   http://w3c.github.io/tvcontrol-api/
> >
> > The API typically produces MediaStreams, with methods to switch from one TV/radio channel to the other, and various classes to retrieve associated channel and program metadata.
> >
> > The current API is designed around the notion of tuners but the group agreed to a re-design centered on the notion of sources. To keep things simple, a TV source is something attached to a broadcast signal (which could come through a cable, terrestrial antennas, satellites, etc.) that can be tuned to a specific channel to produce a MediaStream, composed of a set of MediaStreamTracks for that channel. Users may switch the source to another channel at any time.
> >
> > Looking at it from a getUserMedia perspective, it seems possible to consider TV sources as input devices, and channels as a constraint that could be applied to the main video track. This would lead to the following code:
> >
> > var source = null;
> >
> > navigator.mediaDevices.enumerateDevices()
> >     .then(function (devicesInfo) {
> >       source = devicesInfo.find(function (deviceInfo) {
> >         // Or use "getCapabilities" to detect support for the "tvchannel"
> >         // constraint if kind cannot be extended?
> >         return deviceInfo.kind === 'tvsource';
> >       });
> >       return source.getChannels();
> >     })
> >     .then(function (channels) {})
> >       var channel = channels[0];
> >       secondChannel = channels[1];
> >       navigator.mediaDevices.getUserMedia({
> >         video: {
> >           deviceId: source.deviceId,
> >           tvchannel: channel
> >         }
> >       })
> >     })
> >     .then(function (stream) {
> >       tvStream = stream;
> >
> >       // Render the stream
> >       document.getElementById('video').srcObject = tvStream;
> >
> >       // Switch to another channel
> >       return tvStream.getVideoTracks()[0].applyConstraints({
> >         tvchannel: secondChannel
> >       });
> >     })
> >     .then(function () {
> >       // Stop stream
> >       tvStream.getVideoTracks()[0].stop();
> >     });
> >
> > This triggers a few questions though.
> >
> > Questions
> > -----
> >
> > 1. Attaching to "enumerateDevices"?
> > --
> > We need some way to enumerate the TV/radio sources but it feels a bit strange to mix TV/radio sources with camera/microphone sources. Use cases that want to get media from a camera are roughly disjoint from use cases that want to tune to a particular TV channel. Shouldn't APIs rather be separated?
> >
> > If not, what would be the proper way to distinguish TV sources? Should we introduce a new "tvsource" kind of source for instance?
> 
> It might be more convenient to model your API on top of the
> getScreenContent API, which is also an API for getting MediaStreams from
> stuff that is not cameras or microphones.
> 
> https://github.com/w3c/mediacapture-screen-share
> 
> This API is not directly applicable, since it is intended to invoke a
> browser-chrome chooser rather than just opening a screen capture, but it
> is another example of getting streams from non-device objects.

Thanks for this. So our options become:

1) Extend getUserMedia to allow TV source and channel constraints 
2) Extend MediaDevices to add a TV-specific stream-returning method
3) Continue to use our existing TVManager interface (exposed as nagivator.tv)

It seems that either (2) or (3) seem preferable.

> > 2. Channel constraint at the track level?
> > --
> > Similarly, it does not seem natural to apply the constraint to change the channel at the track level. The channel "constraint" rather seems to apply at the MediaStream level, as it is going to affect all tracks. Are we trying to push the constraint model too far?
>
> It seems somehow strange to use a stream control to change the channel.
> I'd rather want to model this as getting a new stream from the new
> channel. Streams are not very heavy-weight objects; creating a new one
> is probably more sensible than re-sourcing an old one.

This is something we've discussed in the TV Control WG, where we considered that creating a MediaStream involves allocating the underlying hardware resources for tuning to and decoding the media. It seemed to us that reusing the stream allowed for a simpler implementation, and provided a reasonable API for users.

> > The same comment applies to the "stop" method, which again is going to affect all tracks of the MediaStream in our case.
> 
> I don't think it's necessary to reintroduce stream.stop(); it is a very
> short loop to stop all tracks in a stream.

We don't really have a use case for stopping streams individually, so we may want a stop method in our MediaStream-derived interface, or an interface that acts as a parent to the MediaStream.

> > 3. TV specific semantics?
> > --
> > While it works, is also seems counter-intuitive to "apply a constraint" to switch from one channel to another. Developers would rather expect something like a "tuneToChannel" method instead.
>
> If we don't retarget streams, this problem goes away.

Yes, true.

> > 4. Extending or wrapping MediaStream?
> > --
> > The TV Control API extends MediaStream to add buffering. This seems to preserve the spirit of MediaStreams and there has already been exchanges about that in the past.
> >
> > We may need to introduce other attributes at that level, such as an "isRecordable" property to tell whether the channel may be recorded. This is also where the "stop" method mentioned above would fit. Such extensions seem of different nature. We're wondering whether it could be preferable to introduce an extra layer that wraps the MediaStream and exposes TV specific properties.
> 
> What are the buffering properties you want? Do you want to make the
> result seekable (infininte-size buffering), or do you want merely to
> ensure that buffers are deep enough that glitches don't happen?

We want the stream to be seekable, to enable pause/resume for live TV and timeshifted playback, so essentially we're removing a lot of the constraints that MediaStream applies to the HTMLMediaElement. Here's a link to our previous discussion:

https://lists.w3.org/Archives/Public/public-media-capture/2015Jun/0011.html

and there are some specifics in my original email:

https://lists.w3.org/Archives/Public/public-tvapi/2015Jun/0000.html

> The Identity framework should offer some capabilities for restricting
> recording; basically you could just assert a random identity on channels
> that are not allowed to be inspected by the JS, and the semantics would
> do the Right Thing.

We'll take a closer look at this, thank you.

> > Alternative proposal
> > -----
> > All in all, an alternative proposal that separates the APIs, moves operations back to the source level, and creates a wrapping tuner class around MediaStream could lead to the following code:
> >
> > var source = null;
> > var secondChannel = null;
> > var tvTuner = null;
> >
> > navigator.tv.getSources()
> >     .then(function (sources) {
> >       source = sources[0];
> >       return source.getChannels();
> >     })
> >     .then(function (channels) {
> >       var channel = channels[0];
> >       secondChannel = channels[1];
> >       return source.tuneToChannel(channel);
> >     })
> >     .then(function (tuner)) {
> >       tvTuner = tuner;
> >
> >       // Render the stream
> >       document.getElementById('video').srcObject = tvTuner.stream;
> >
> >       // Switch the source to another channel, re-using the same tuner
> >       return source.tuneToChannel(secondChannel, tuner);
> >     })
> >     .then(function () {
> >       // Stop stream and release resource
> >       tvTuner.stop();
> >     });
> >
> > That alternative proposal re-uses MediaStreams, but is less aligned with getUserMedia. What do you think?
> 
> Putting the ideas above into (attempted) code:
> 
> // Get the channel
> navigator.tv.getChannelStream(channel)
>     .then(function(stream) {
>         document.getElementById('video').srcObject = stream;
>     })
> }
> 
> // Change the channel
> navigator.tv.getChannelStream(otherChannel)
>     .then(function(stream) {
>         document.getElementById('video').srcObject = stream;
>     })
> }
> 
> // Stop the channel
> document.etElementById('video').srcObject = null;
> // Since there are no more references to "stream", it will be garbage
> collected.

Our preference is to have a more explicit mechanism for releasing resources, rather than only relying on GC, although we do realise we can't rely on users doing the right thing.

> > Feedback through email is fine. Feel free to send feedback on GitHub if preferred. This is being tracked by the following issue on our side:
> >  https://github.com/w3c/tvcontrol-api/issues/4
> >
> > Note that we would be happy to schedule a call as needed to clarify questions and thoughts.
> >
> > Thanks,
> > Francois.
> 
> -- 
> Surveillance is pervasive. Go Dark.

Many thanks,

Chris

Received on Thursday, 12 January 2017 10:08:38 UTC