Re: State transitions for media elements

On Wed, 2008-10-15 at 00:26 +0000, Ian Hickson wrote:
> On Thu, 18 Sep 2008, Dave Singer wrote:
> >
> > So, here is a brief revision, obviously derived from the existing 
> > document but, we hope, avoiding these issues and supporting more (we 
> > hope, all) protocols:
> > 
> > Network state:  documents whether the network is being used (e.g. for a 
> > network activity indicator)
> > 
> > Empty:  initial state, or state when there is a failure that'll need 
> > some action to escape from
> > 
> > Idle:  the URI is known, but the UA has no need to use the network right 
> > now (e.g. download resource for which 'enough' is cached, streaming 
> > resource which is not active...)
> > 
> > Loading: the network is being used right now (you can show an activity 
> > indicator)
> > 
> > Loaded:  for a loadable resource, we've both loaded it all and don't 
> > intend to unload it (you could disconnect and walk away)
> > 
> > There is an event, Stalled, which is fired once during Loading if data 
> > doesn't seem to be arriving after a reasonable timeout (as now).
> >
> > 
> > Media state:  documents what you can do with the media.  (Each state is 
> > a superset of the one preceding).
> > 
> > Empty: initial state
> > 
> > Metadata_loaded:  enough data has been loaded that a well-defined set of 
> > questions can now be answered as well as they ever could be (e.g. 
> > duration, width/height, codecs used, and so on).
> > 
> > Can_display (or Can_display_at_current_time; currently called 
> > can_display_current_frame):  the UA has done all it can or intends to do 
> > for the media resource to be displayable at the current_time.  For a 
> > downloadable resource, this means that the current video frame (if 
> > applicable) can be painted, at least one sample of audio (if applicable) 
> > played, and so on.  For a streaming resource, it may mean very little 
> > more than that if you are waiting for something before you displayed the 
> > media element, stop waiting: it won't get any more displayable.
> > 
> > Can_play:  if playback were requested, the UA expects it would be able 
> > to actually start within a reasonable period and play a reasonable 
> > amount (before a stall, for example).  For a downloadable protocol, that 
> > means that at least some data ahead of current_time is available; for a 
> > streaming protocol, that if playback was requested, playback would start 
> > 'soon'.
> > 
> > Can_play_through:  if playback was requested, the UA is reasonably 
> > confident that it could play to the end without a playback stall.  
> > (This state might never get entered if the network bandwidth is 
> > insufficient and the resource cannot be cached, either because of cache 
> > limitations or because it's a streaming service)
> 
> I also added a state for "all the metadata is known and video has been 
> shown at least once so there is no need to show placeholder images like 
> the poster frame".
> 
> The states thus are:
> 
>    NETWORK_EMPTY
>    NETWORK_IDLE
>    NETWORK_LOADING
>    NETWORK_DONE
> 
> ...and:
> 
>    HAVE_NOTHING
>    HAVE_METADATA
>    HAVE_SOME_DATA
>    HAVE_CURRENT_DATA
>    HAVE_FUTURE_DATA
>    HAVE_ENOUGH_DATA
> 
> I'm not good at naming constants so if anyone has a truly better idea, 
> please let me know. Ideas should be self-consistent, intuitive, have a 
> common leading prefix, be relatively short, have correct grammar, and not 
> be cute. If you're not sure, don't suggest it. I don't want to wade 
> through three dozen sets of names tomorrow. :-P
> 
> 
> > Play_request state:  documents what has been asked of the media.  We 
> > need state+events for this because UAs can display a play/pause 
> > controller that the scripts cannot 'see'.
> > 
> > Empty:  initial state
> > 
> > Pause_requested:  the UA has been asked to pause playback
> > 
> > Play_requested:  the UA has been asked to play
> > 
> > (This could probably be a single boolean if we don't need the empty 
> > initial state).
> 
> How is this different from the "paused" boolean attribute?
> 
> 
> > Actual playing is reflected by the is_playing property and the 
> > Rate_changed event.
> > 
> > Rate_changed gets dispatched if either of
> > a) is_playing changes value (between true and false)
> > or
> > b) is_playing is true and the current playback rate changes
> > 
> > Specifically:
> > a) for a streaming protocol, after a play_request, the network connection is
> > opened, data is requested, some amount of de-jitter buffer accumulated, and
> > then is_playing changes to true and a Rate_changed event happens
> > b) for a download or streaming protocol, if the buffer runs dry while playing,
> > is_playing changes to false and the Rate_changed event is dispatched.
> 
> You could implement these on top of the existing events, but1 I guess we 
> could add these events too. Do people really want them?
> 
> 
> On Mon, 22 Sep 2008, Philip Jgenstedt wrote:
> > 
> > In particular I've noticed that if the file is available locally (in 
> > cache or file://) one will know that it is fully loaded before actually 
> > reading any data. But since network states must go through 
> > LOADED_METADATA and LOADED_FIRST_FRAME (which require actually reading 
> > data) reporting that state has to wait until later. It wouldn't be a big 
> > issue implementation-wise, but is a symptom of the mixing of separate 
> > issues into a single state.
> 
> As defined, the networkState will never be set to NETWORK_LOADED before 
> the metadata and first frame have been obtained explicitly. Is that ok?

That's the order of things in the load() algorithm currently. It's not a
huge problem, but would it hurt to simply merge step 14 and 15, so that
NETWORK_LOADED can occur independently of HAVE_METADATA and
HAVE_CURRENT_DATA?

> > I'm thinking that the EMPTY state isn't needed at all, the lowest state 
> > could be IDLE. Currently the EMPTY state is used in several algorithms 
> > to detect a "fresh" media element, but as far as I can see an empty flag 
> > would suffice, and it would not need to be exposed via the API at all.
> 
> It seems like it is useful to expose what the browser thinks the state is. 
> If people think we should drop NETWORK_EMPTY and just have NETWORK_IDLE, 
> though, I'm certainly open to changing the spec to that. What do people want?

Exposing NETWORK_EMPTY in networkState seems neither useful nor harmful,
so if nobody else cares either way then perhaps change for the sake of
change would be a waste of time.

> -- 
> Ian Hickson               U+1047E                )\._.,--....,'``.    fL
> http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
-- 
Philip Jägenstedt
Opera Software

Received on Monday, 20 October 2008 13:47:53 UTC