Re: Feedback on MediaController

On Thu, 21 Apr 2011, Silvia Pfeiffer wrote:
> On Thu, Apr 21, 2011 at 9:01 AM, Ian Hickson <ian@hixie.ch> wrote:
> > On Mon, 18 Apr 2011, Silvia Pfeiffer wrote:
> >>
> >> The kind is necessary to classify the track correctly in menus, e.g. 
> >> as sign language, audio description, or even a transparent caption 
> >> track.
> >
> > I'm fine with exposing kind; is there any documentation on what video 
> > formats expose for this?
> >
> > We should derive these from the kinds that are exposed in media 
> > formats, it doesn't make sense for us to come up with them.
> 
> http://wiki.xiph.org/Ogg_Skeleton_4 has a specification for what it is
> in Ogg now.
> This includes the roles as specified here:
> http://wiki.xiph.org/SkeletonHeaders#Role
> 
> I don't know if MPEG has anything like it.
> 
> WebM has a bunch of metadata on tracks e.g. TrackType, but not much
> semantics IIUC.
> http://www.webmproject.org/code/specs/container/#track

TrackType doesn't seem appropriate here. There doesn't seem to be an 
appropriate field.

If Ogg is the only format with this kind of thing, then practically 
speaking I don't see how we can expose it. User agents simply wouldn't 
have the information available.

I've written the spec for getKind(), but have left it commented out for 
now since it doesn't seem to be usefully implementable.


> To me looping on individual slaves of a multitrack resource individually 
> doesn't make sense. That would mean that this element is independent of 
> the others. Either they all loop or none does.

I think this is overly limiting, but it's academic at this point since 
"loop" is defined as doing nothing currently anyway.


> When attributes on slaves are changed, the state of the group changes. 
> Autoplay changed during playback has no effect anyway. Autoplay is only 
> relevant right after the first load.

It's relevant after any load.


> I also think that having autoplay on individual elements while not on 
> others decouples their timeline and is wrong. That should not be 
> possible. They either slave to the same timeline or they don't.

(The decoupling problem was a bug. Fixed.)

Imagine a scenario where you have five audio tracks and a video track, 
and you just want one audio track at a time to play. You can pause the 
other four, and play the fifth. If you want this to start automatically, 
you'd only want the video and one audio track to autoplay.


> >> > But what's the use case?
> >>
> >> If I reach the end, I want to present something different, such as a 
> >> post-roll add or an overlay with links to other videos that are 
> >> related. It is much easier to wait on a onended event on the combined 
> >> resource than having to register an event handler with each slave and 
> >> then try and combine the result.
> >
> > If you want to display an overlay at the end of a video, it seems like 
> > you'd want to do that as soon as the video ended, you wouldn't want to 
> > wait until the end of the entire resource, no? So you'd want onended 
> > on the <video> element, not the controller.
> 
> No, I'd actually want to wait until all elements are ended before 
> showing an overlay, in particular if an audio description or a sign 
> language video or a commentary continues. The grouped resource is not 
> ended before all of the slaves are ended and I don't want random 
> advertising in my face before it's all over. That would be really 
> disturbing.

Fair enough. I've added an 'ended' event on MediaController that fires 
whenever all the slaves are ended.


On Thu, 21 Apr 2011, Philip Jägenstedt wrote:
> 
> One kind of quite likely error is that people try to draw the videos to a
> canvas in the loadedmetadata event handler. Most of the time this will work
> because of the very narrow window between HAVE_METADATA and HAVE_CURRENT_DATA.
> However, for those elements that are still actually in HAVE_METADATA, nothing
> will be painted per the drawImage spec: "if the image argument is an
> HTMLVideoElement object whose readyState attribute is either HAVE_NOTHING or
> HAVE_METADATA, then the implementation must return without drawing anything."
> If readyState is racy, then of course such scripts will fail randomly, but if
> such failure is rare enough, some sites will inevitably come to depend on
> readyState being HAVE_CURRENT_DATA in the loadedmetadata event handler, which
> would make HAVE_METADATA and HAVE_CURRENT_DATA completely redundant.

To fix this we'd need to do two things: first, make the changing of 
readyState happen in the task the event fires in (which we'll probably do 
as a result of the bugs you filed), and second, make drawImage() fail if 
readyState isn't HAVE_CURRENT_DATA, even if the data actually is 
available. Also, we'll have to define what happens if the handler for 
loadeddata seeks to something that doesn't have data yet and then calls 
drawImage(). Or seeks to something that doesn't have data yet, spins 
synchronously for just long enough to _get_ data, and then calls 
drawImage(). I guess we could make seeking synchronously reset readyState.

Can you clarify if this is what you mean?


> > > I'd very much like to see at least the overall issue of race 
> > > conditions resolved. For these aggregate events on MediaController, 
> > > we would have to make sure that they are fired in the same task as 
> > > the last media element changes its readyState and fires the 
> > > corresponding event.
> >
> > Firing it in the same task that the last media element fires the 
> > corresponding event seems reasonable, but I still don't understand why 
> > you think it will be helpful to freeze readyState until the event 
> > fires. Can you explain (maybe again) with a concrete example?
> 
> These are race conditions that I or people I work with have actually 
> tripped on:
> 
> * When removing the autoplay attribute in any of the loadstart, 
> loadedmetadata, loadeddata or canplay event handlers, it's racy whether 
> or not the media element will actually play.

I'll handle that as part of bug 11981.


> * When registering a loadeddata event handler in the loadedmetadata 
> event handler, it's racy whether or not that event handler will run. The 
> same is true of any other pair of events that depend on readyState or 
> networkState transitions. Having this not be racy would be useful when 
> writing tests, e.g. to catch the first canplaythrough event after the 
> seeked event.

I'll handle this as part of these same bugs.


> * For preload=metadata, when checking networkState in a progress event 
> handler, it's racy between NETWORK_LOADING and NETWORK_IDLE. This makes 
> it impossible to test if we follow the spec on fast networks.

Is there a bug on this one?

(I don't really understand the problem here.)


> * When currentTime is set in the loadstart event handler, it's racy 
> whether or not it will actually seek (fails if readyState is 
> HAVE_NOTHING).

Making currentTime's behaviour depend on the exposed readyState rather 
than the actual underlying state sounds like a world of pain, but I guess 
we can do that to a limited extent. Is there a bug on this one?


> * When the loop attribute is present and you pause() in the ended event 
> handler, it's racy if you end up paused at the beginning or the end.

I don't see how to fix that. We can't make the event handler synchronous 
since that would make the playback jerky.

In fact, pause() in general is unavoidably "racy" as far as I can tell.


On Thu, 21 Apr 2011, Philip Jägenstedt wrote:
> 
> Supposedly, the spec should no longer allow slave media elements to be 
> out of sync. Yet, HTMLMediaElement.play() still does what it always has, 
> including seeking when playback has ended.

That was an oversight, hopefully now fixed. Let me know if there's 
anything still broken here.


> Perhaps play() and pause() should throw INVALID_STATE_ERR when there's a 
> controller, just like setting currentTime?

You can still turn certain tracks on and off, it's just that while they're 
playing they'll be forced to be in sync.


> > I'm happy to follow implementations on this.
> 
> At what point would you remove it from the spec because of lack of 
> implementor interest?

I go through the spec every year or two getting rid of dead wood. See 
also:

   http://wiki.whatwg.org/wiki/FAQ#Is_there_a_process_for_removing_bad_ideas_from_a_specification.3F


On Thu, 21 Apr 2011, Eric Carlson wrote:
>
>   For a QuickTime movie, videoWidth and videoHeight are calculated from 
> the union of the bounding boxes of all enabled visual tracks.  
> Therefore, videoWidth and videoHeight will change if disabling a track 
> changes the union rect. This is true in Safari today.

It seems really weird that enabling a video track might change where 
another one is rendered.


> > The use case for looping a single track is things like this:
> > 
> >   http://www.google.com/green/
> > 
> > ...but I don't see why you would use a MediaController to do that kind 
> > of thing. It's not like you'd want the multiple videos there in sync, 
> > they're just background.
>
> We have had several requests from developers who want to create 
> something very much like http://www.google.com/green/, but with the 
> background videos playing in sync (and looping). The developers 
> essentially want a non-rectangular video, which in theory could be done 
> with a single file with alpha, but alpha is currently not widely 
> supported in video formats.

If that's the use case, it seems like we'd be better off adding alpha than 
adding synchronised looping.

I don't mind adding synchronised looping, it's just that it should be the 
best solution to its use case if we add it.


> > I'm also skeptical of introducing loop at the MediaController level 
> > even in the simple case of finite resources, because it's not clear 
> > how to make it work with looping subresources. Say you had two 
> > resources, both set to loop, one of which was 5s and one 3s, and that 
> > you then further say that the whole thing should loop. What should 
> > happen? We don't want to define MediaController looping in a way that 
> > precludes that from being possible, IMHO, at least not unless we have 
> > a strong use case.
>
>   MediaController.currentTime would be set to 0 when it reaches the 
> duration of the longest subresource.

This would mean that if we had sublooping as well as overall looping we'd 
preclude the ability for the longer track to be a non-integral multiple of 
the subloops without the subloop getting cut short each time the overall 
loop went around. Is that ok?


> >> * onended: is raised when all slave media elements are in ended state
> > 
> > This isn't supported; a media controller can't be "ended" currently.
>
>   It could be "ended" when currentTime reaches the duration of the 
> longest subresource.

Done.


> > Looking at the metadata list cited above, I don't see anything in 
> > either ogg, mp4, or webm that maps to "kind", so I don't see much 
> > point exposing that on the audio/video track lists, though I agree 
> > that in principle it would be a good idea.
>
> For MPEG-4 and QuickTime files, the mapping can sometimes be made from a 
> track's media format. For example, a closed caption track has 'clcp' 
> sample data, 'tx3g' is MPEG-4/3GPP Timed Text, etc. It is also possible 
> for a track's metadata to list its "kind".

Can you point me to the documentation for this?


> > As currently specified, grouped tracks always autoplay any tracks that 
> > are labeled autoplay, and those that are not simply don't play (while 
> > the controller's position advances regardless).
>
>   I think it will be confusing for developers that some tracks in a 
> group can play while others do not. It seems more logical to me that the 
> playback state of all elements in a group should be kept in sync.

I've fixed the playing-when-nothing-is-playing problem, but I strongly 
disagree that we should always have all the tracks enabled at once. It 
would be a pain to unhook a media element to stop a track from playing 
instead of just pausing it.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'