Re: Tech Discussions on the Multitrack Media (issue-152)

On Wed, 02 Mar 2011 17:40:07 +0100, Mark Watson <watsonm@netflix.com>  
wrote:

>
> On Mar 1, 2011, at 11:58 PM, Philip Jägenstedt wrote:
>
>> On Tue, 01 Mar 2011 23:51:30 +0100, David Singer <singer@apple.com>  
>> wrote:
>>
>>> Hi Silvia, friends
>>>
>>> Eric and I have been discussing this, and we've added an 8th option to
>>> the Wiki, for your consideration.
>>>
>>> at <http://www.w3.org/WAI/PF/HTML/wiki/Media_Multitrack_Media_API>
>>
>> This is like option 2 except that <track> is not a void element and can
>> have <source> child elements, right?
>>
>> For all of the options that don't use <audio> and <video> to represent
>> audio and video tracks, do you consider it a problem that properties of
>> HTMLMediaElement are not available? Some of these allow for interesting
>> conflicts such as looping a slave video but not the master video, but I
>> imagine at least videoWidth, videoHeight and volume would be useful to
>> have.
>
> It seems that the requirements for the object representing a track are  
> more than the proposed MediaTrack but less than HTMLMediaElement.

I agree.

> If something other than strict time alignment in the playout of the  
> different tracks is required, then shouldn't you have completely  
> separate media elements, rather than allowing currentTime, playbackRate,  
> loop attributes & load(), play(), pause() methods for each track ? It  
> could surely get quite complex if each track is allowed to have  
> independent timing, but somehow they are still supposed to be linked  
> together.

I've been arguing that we should have separate media elements because it  
makes styling so much easier. It is a problem that one then has to define  
what happens when one e.g. sets currentTime on a slave, but it could be  
solved, e.g. by throwing WRONG_STATE_ERR. It's far from obvious what is  
the best solution here.

> What attributes from HTMLMediaElement would really make sense to add to  
> MediaTrack ?

I'm not sure we should duplicate parts of HTMLMediaElement, but in any  
case these parts of HTMLMediaElement I think we want to use for "slave"  
audio and video tracks:

* error
* src, currentSrc and the resource selection algorithm (picking between  
src="" and multiple <source> elements)
* networkState
* buffered
* readyState
* seeking
* seekable
* ended
* volume
* muted (if you want just the video track of a second resource)

> I have another question related to an earlier discussion on this thread:  
> do I understand correctly that in order to apply CSS styling there needs  
> to be an HTML markup element to apply the styling to ? i.e. that you  
> couldn't apply styling to control the rendering of an in-band track that  
> appears only as a MediaTrack object and not as a <track> element (with  
> associated HTMLTrackElement object) ?
>
> In that case it seems we need to be able to refer to an in-band track in  
> the main resource from a <track> element in order to apply styling. For  
> example the MediaTrack API could expose an "id" attribute (at least for  
> in-band tracks) which could be used to refer to this track from a  
> <track> element.

The discussion about rendering has mostly been about whether or not video  
tracks should be in a <video> element and styles using plain CSS or if it  
should be something more magical like pseudo-elements of <track> that are  
still rendered outside of the <video> content area.

About in-band video tracks, one solution I've been thinking about is  
exposing additional video tracks as Stream objects, which can then be set  
as the src on another video element. This is similar to how the <device>  
element directs output to a <video>.

>> About multiple text track formats, my working assumption is that we will
>> be able to agree on a single format that everyone supports as there is  
>> not
>> really any of the legal of business issues that we have for audio and
>> video.
>
> Even if all browsers, including those embedded in TVs etc., support  
> WebVTT, it seems likely some will support other formats as well (e.g.  
> TTML/SMPTE-TT) and that content providers might have a preference about  
> which is used on devices which support both.

I'm still hoping that everyone will support exactly 1 format like for  
scripts and stylesheets, but sure, it's possible that won't happen.

>> If, however, we should both fail at finding a baseline text format and
>> decide that we should use <track> for audio and video as well, then
>> wouldn't multiple <track> elements with a type parameter be enough? We
>> could define it such that multiple <track> elements that differ only in
>> type should be assumed to representing the same content and have some  
>> kind
>> of track selection algorithm for picking one. Am I too optimistic?
>>
>
> That seems like a secondary markup language design issue. <source>  
> elements does allow you to express the semantics that some things are  
> mutually exclusive. The version with <source> elements inside <track> is  
> a superset of the version with only <track> elements as you can always  
> list multiple <track> elements instead. I would be cautious about being  
> too prescriptive with a "pick one" algorithm: if the tracks are  
> unambiguously described and the browser has a good way to make use of  
> two of the same type and that is what the user wants then it should be  
> allowed to do that.

If we have <source> child elements to <track>, I think it should work  
exactly like the resource selection algorithm for <video>, which is a  
"pick one" algorithm. If each <source> represents the same resource,  
there's no reason to have it otherwise.

Still, it seems to me that just having many <track> elements would work.

-- 
Philip Jägenstedt
Core Developer
Opera Software

Received on Thursday, 3 March 2011 11:47:32 UTC