RE: TV Control API and getUserMedia

Hi Stefan,

Thanks, that's precisely the sort of feedback we are looking for!
See inline for comments. Also speaking as an individual.

> De : Stefan Håkansson LK [mailto:stefan.lk.hakansson@ericsson.com]
> Envoyé : lundi 19 décembre 2016 11:12
> 
> (speaking as an individual)
> 
> To me it seems that using getUserMedia, as specced now, with constraints
> to select a tvtuner as the source is a bit clumsy, since the constraints
> you can supply are of type "MediaStreamConstraints" which in turn has
> (up to) one entry for a video track and one for an audio track.
> 
> If you were to extend getUserMedia to also be able to accept a (e.g.)
> "TVTuner" constraint it could perhaps look a little better.

OK, that would create another "getUserMedia" for a different usage. It's probably cleaner to stay away from "getUserMedia" altogether in that case, as in the "alternative proposal".


> Also, changing channel by applying a track constraint seems (as you
> point out) clumsy and non intuitive. Here, you could perhaps define
> constraints that apply on TVMediaStreams instead?

Right. Which triggers our question 4: this seems to attach new semantics to TVMediaStream that do not map well to the "spirit" of MediaStreams. It may be worth "wrapping" instead of "extending".


> To me, your 'Alternative proposal' actually seems cleaner.

Good to know :)


> Now I am far away from my area of knowledge, but anyway: is there any
> specific reason for not going along the path of Media Source Extensions
> and define a TV Tuner in a similar way? To me it seems that the only
> uses of TVMediaStream is a) recording and b) text tracks. But a) will
> (?) be solvable by combining mediacapture-fromelement with
> mediacapture-recodring, and for b) I had the feeling that there was
> already support for handing text tracks in the video element itself.

Let me try to reformulate to make sure I understand. What you're suggesting is that we investigate a design such as:

var video = document.getElementById('v');
var tvSource = new TVSource();
video.src = window.URL.createObjectURL(tvSource);

... and attach our logic to TVSource without ever using MediaStreams. Does that roughly capture what you have in mind?

I don't think the TV Control API can directly reuse Media Source Extensions, as SourceBuffer objects take bytes as input. The stream of bytes in our case does not need to be exposed to the application, and should not be exposed in most cases. The main expected flow is from the tuner to the graphics card through the decoding pipeline, without bytes actually going through the user agent. Now, the TV Control WG could indeed define another type of media source. The part that worries me is that the spec would need to define how media playback algorithms in HTML5 are to be adjusted when such a media source gets attached to a media element, which is something MediaStream seems to provide out of the box. But maybe that's not a big deal.

Thanks,
Francois.



> 
> Br,
> Stefan
> 
> On 16/12/16 15:03, Francois Daoust wrote:
> > Dear Media Capture task force,
> >
> > The TV Control Working Group is re-modeling the TV Control API
> > specification around sources. That new model is well aligned with the
> > model used in Media Capture and Streams. In fact, it would seem
> > possible to re-use Media Capture and Streams interfaces as-is.
> > However, some details do not seem entirely right when we do that, and
> > we would like to get your feedback on the following questions.
> >
> >
> > Context ----- The TV Control API specification defines an API for
> > sourcing audio and video media, such as TV and radio from broadcast,
> > IPTV, or other sources, and allows presentation of the media using
> > the <video> and <audio> HTML elements:
> >
> > http://w3c.github.io/tvcontrol-api/
> >
> > The API typically produces MediaStreams, with methods to switch from
> > one TV/radio channel to the other, and various classes to retrieve
> > associated channel and program metadata.
> >
> > The current API is designed around the notion of tuners but the group
> > agreed to a re-design centered on the notion of sources. To keep
> > things simple, a TV source is something attached to a broadcast
> > signal (which could come through a cable, terrestrial antennas,
> > satellites, etc.) that can be tuned to a specific channel to produce
> > a MediaStream, composed of a set of MediaStreamTracks for that
> > channel. Users may switch the source to another channel at any time.
> >
> > Looking at it from a getUserMedia perspective, it seems possible to
> > consider TV sources as input devices, and channels as a constraint
> > that could be applied to the main video track. This would lead to the
> > following code:
> >
> > var source = null;
> >
> > navigator.mediaDevices.enumerateDevices() .then(function
> > (devicesInfo) { source = devicesInfo.find(function (deviceInfo) { //
> > Or use "getCapabilities" to detect support for the "tvchannel" //
> > constraint if kind cannot be extended? return deviceInfo.kind ===
> > 'tvsource'; }); return source.getChannels(); }) .then(function
> > (channels) {}) var channel = channels[0]; secondChannel =
> > channels[1]; navigator.mediaDevices.getUserMedia({ video: { deviceId:
> > source.deviceId, tvchannel: channel } }) }) .then(function (stream)
> > { tvStream = stream;
> >
> > // Render the stream document.getElementById('video').srcObject =
> > tvStream;
> >
> > // Switch to another channel return
> > tvStream.getVideoTracks()[0].applyConstraints({ tvchannel:
> > secondChannel }); }) .then(function () { // Stop stream
> > tvStream.getVideoTracks()[0].stop(); });
> >
> >
> > This triggers a few questions though.
> >
> > Questions -----
> >
> > 1. Attaching to "enumerateDevices"? -- We need some way to enumerate
> > the TV/radio sources but it feels a bit strange to mix TV/radio
> > sources with camera/microphone sources. Use cases that want to get
> > media from a camera are roughly disjoint from use cases that want to
> > tune to a particular TV channel. Shouldn't APIs rather be separated?
> >
> > If not, what would be the proper way to distinguish TV sources?
> > Should we introduce a new "tvsource" kind of source for instance?
> >
> >
> > 2. Channel constraint at the track level? -- Similarly, it does not
> > seem natural to apply the constraint to change the channel at the
> > track level. The channel "constraint" rather seems to apply at the
> > MediaStream level, as it is going to affect all tracks. Are we trying
> > to push the constraint model too far?
> >
> > The same comment applies to the "stop" method, which again is going
> > to affect all tracks of the MediaStream in our case.
> >
> >
> > 3. TV specific semantics? -- While it works, is also seems
> > counter-intuitive to "apply a constraint" to switch from one channel
> > to another. Developers would rather expect something like a
> > "tuneToChannel" method instead.
> >
> >
> > 4. Extending or wrapping MediaStream? -- The TV Control API extends
> > MediaStream to add buffering. This seems to preserve the spirit of
> > MediaStreams and there has already been exchanges about that in the
> > past.
> >
> > We may need to introduce other attributes at that level, such as an
> > "isRecordable" property to tell whether the channel may be recorded.
> > This is also where the "stop" method mentioned above would fit. Such
> > extensions seem of different nature. We're wondering whether it could
> > be preferable to introduce an extra layer that wraps the MediaStream
> > and exposes TV specific properties.
> >
> >
> > Alternative proposal ----- All in all, an alternative proposal that
> > separates the APIs, moves operations back to the source level, and
> > creates a wrapping tuner class around MediaStream could lead to the
> > following code:
> >
> > var source = null; var secondChannel = null; var tvTuner = null;
> >
> > navigator.tv.getSources() .then(function (sources) { source =
> > sources[0]; return source.getChannels(); }) .then(function (channels)
> > { var channel = channels[0]; secondChannel = channels[1]; return
> > source.tuneToChannel(channel); }) .then(function (tuner)) { tvTuner =
> > tuner;
> >
> > // Render the stream document.getElementById('video').srcObject =
> > tvTuner.stream;
> >
> > // Switch the source to another channel, re-using the same tuner
> > return source.tuneToChannel(secondChannel, tuner); }) .then(function
> > () { // Stop stream and release resource tvTuner.stop(); });
> >
> > That alternative proposal re-uses MediaStreams, but is less aligned
> > with getUserMedia. What do you think?
> >
> > Feedback through email is fine. Feel free to send feedback on GitHub
> > if preferred. This is being tracked by the following issue on our
> > side: https://github.com/w3c/tvcontrol-api/issues/4
> >
> > Note that we would be happy to schedule a call as needed to clarify
> > questions and thoughts.
> >
> > Thanks, Francois.
> >
> >
> >

Received on Tuesday, 20 December 2016 15:28:13 UTC