Re: Constructors for MediaStreamTrack subtypes - remove?

On 2013-08-12 09:29, Harald Alvestrand wrote:
> On 08/12/2013 08:34 AM, Stefan Håkansson LK wrote:
>> On 2013-08-11 17:13, Harald Alvestrand wrote:
>>> On 08/08/2013 03:35 PM, Stefan Håkansson LK wrote:
>>>> On 2013-08-05 10:46, Harald Alvestrand wrote:
>>>>> Hi,
>>>>> at the moment, the spec contains constructors for VideoMediaStreamTrack
>>>>> and AudioMediaStreamTrack - each taking a TrackConstraints set as parameter.
>>>>>
>>>>> These were part of an idea to revisit the way one acquires media
>>>>> (different from getUserMedia, which returns tracks) - but the chairs are
>>>>> unable to tell whether any of the brief proposals floated in Lyon and
>>>>> Boston have gathered consensus as "THE proposal", or indeed whether we
>>>>> have consensus to go ahead with this API addition.
>>>>>
>>>>> In the absence of such consensus, I suggest that we remove the
>>>>> constructors from the spec at this time, so that it's clear that
>>>>> implementations don't have to support these constructors that don't
>>>>> produce useful objects at the moment.
>>>>>
>>>>> If we converge on a specific proposal that makes these constructors
>>>>> useful, I think it is easy to add them back at that time.
>>>>>
>>>>> People who have a different opinion should reply to this message....
>>>> Personally I think the possibility to create AudioMediaStreamTrack's and
>>>> VideoMediaStreamTrack's (and out of them MediaStream's) is quite
>>>> valuable in situations where the destination is a remote peer.
>>>>
>>>> It allows the application to set up the connection and media codecs etc.
>>>> before (or in parallel to) asking for consent to use microphone and
>>>> camera. This can enhance the user experience since it reduces the risk
>>>> of audio clipping (as everything is already set up when the user accepts
>>>> the use of camera and microphone). It also allows the app to not bother
>>>> the user by asking for consent if for some reason the connection can't
>>>> be established (or if the codecs supported do not match - we still don't
>>>> have an MTI video codec).
>>> That's where I get confused .... what if the properties of the stream
>>> depend on the device chosen at GetUserMedia time? Such as Cullen's
>>> standard example of a VP8-hardware camera and an H.264-hardware camera,
>>> where you can choose one or the other? (Of course, if you can tell the
>>> capabilities of the camera without opening it, we're back into
>>> fingerprinting land, for which user prompts are supposed to be our
>>> shield.... but "new" happens before the user prompt.)
>>>
>>> So if one does this:
>>>
>>> videoTrack = new VideoMediaStreamTrack(..)
>>> videoStream = new MediaStream(videoTrack)
>>>
>>> peerConnection.addStream(videoStream)
>>> offer = peerConnection.createOffer(...)
>>>
>>> will the offer contain all the codec configurations available to the
>>> browser, only those codec configurations that the browser knows it can
>>> support across all devices, some "browser default", or something else?
>> To me, it would be only those codec configurations the browser knows it
>> can support across all devices. That is all that can be done at this stage.
>
> Two corner cases that come to mind:
>
> 1) The browser has two non-intersecting possible sets (the H.264 vs VP8
> camera case).
> Should the offer contain no video?

This was what I talked about in the section you could not parse. See 
further below.

>
> 2) The browser's "support across all devices" is outside the constraints
> given for the streams , but all reasonable results of an attachUserMedia
> result in a media stream that satisfies the constraints (for example,
> min size may be 640x480, transcoding may be supported only up to
> 240x160, while both the VP8 camera and the H.264 camera are HD capable).
>
> Should CreateOffer fail? Or create a "no video" offer?

I think this problem is more generic. What if getUSerMedia is used with 
mandatory constraints to generate a MediaStream with one video track of 
certain resolution and frame-rate.

If that MediaStream is attached to the PeerConnection, and the browser 
can't support that high resolution or frame-rate for the encoded media 
when sending over the network, should CreateOffer fail, or create a "no 
video" offer? And what if the constraints of the video track is changed 
to something that can't be supported after attaching the MediaStream to 
the PeerConnection? Should a subsequent "CreateOffer" fail? (But this 
discussion belongs to the WebRTC WG).

>>
>> But for this to be possible, all devices must be able to deliver media
>> in a more raw format than h.264/VP8 otherwise the browser would have to
>> transcode. If this is not possible, then I agree this path is questionable.
> I'm not sure I can parse those words into something I can translate into
> code...
>
> http://alvesnotes.blogspot.se/2012/03/byte-stream-fallacy.html
>
> The format actually exchanged on the network once data flows will be in
> a codec format.
> That codec format may be VP8, or something else - it will definitely not
> be in a "raw" format, so negotiating a "raw" format won't just be
> useless, it'll be harmful.
> So that can't be what you intended to say - could you explain more?

Yep. I meant the format (borrowing the example in you blog post) on the 
USB wire. If the devices (cameras) can't deliver e.g. YUV2 (more raw 
than H.264/VP8) but only VP8 or H.264, and you have one VP8 only capable 
camera and one h.264 only - then the browser has no video format it can 
support across all devices (since it can't be expected to transcode from 
VP8 to H.264 or vice versa before transmission). I think this is your 
corner case 1 above. Of course you'd never transmit e.g. YUV2 over the 
network.

My point was that if we foresee it being common that cameras would 
support only VP8/VP9/H.264/H.265 and not YUV2 - then the idea of 
constructed tracks and "attachUserMedia" is perhaps not that attractive.

>
>
>>> And if one later does (my preferred naming - I see no reason to reuse
>>> the name "getUserMedia):
>>>
>>> navigator.attachUserMedia(videoStream, ....)
>>>
>>> will a negotationNeeded event be fired if the capabilities of the source
>>> are different from the capabilities previously negotiated? If it's more
>>> capable than announced, less capable than announced, or only with
>>> different preferences?
>> I think it should fire if the attached device adds capabilities.
>>
>>> It's this kind of complexity that leads me to think that we don't have a
>>> complete proposal for what I'm calling "attachUserMedia" yet.
>>>
>>>
>>>
>>>
>>>> I also think it would be quite a minor addition to spec up how a
>>>> MediaStream that contains only constructed audio and video tracks can be
>>>> an argument to getUserMedia (but I don't know how big the implementation
>>>> effort would be though).
>>>>
>>>>
>>>>>            Harald
>>>>>
>>>>>
>>>>>
>>>
>>>
>

Received on Monday, 12 August 2013 10:01:17 UTC