Re: [whatwg] Memory management problem of video elements

于 2014年08月20日 15:52, Philip Jägenstedt 写道:
> On Tue, Aug 19, 2014 at 3:54 PM, duanyao <duanyao@ustc.edu> wrote:
>> 于 2014年08月19日 20:23, Philip Jägenstedt 写道:
>>
>>> On Tue, Aug 19, 2014 at 11:56 AM, duanyao <duanyao@ustc.edu> wrote:
>>>> If the media element object keeps track of its current playing url and
>>>> current position (this requires little memory), and the media file is
>>>> seekable, then
>>>> the media is always resumable. UA can drop any other associated memory of
>>>> the media element, and users will not notice any difference except a
>>>> small
>>>> delay
>>>> when they resume playing.
>>> That small delay is a problem, at least when it comes to audio
>>> elements used for sound effects. For video elements, there's the
>>> additional problem that getting back to the same state will require
>>> decoding video from the previous keyframe, which could take several
>>> seconds of CPU time.
>>>
>>> Of course, anything is better than crashing, but tearing down a media
>>> pipeline and recreating it in the exact same state is quite difficult,
>>> which is probably why nobody has tried it, AFAIK.
>> UA can pre-create the media pipeline according to some hints, e.g. the video
>> element is becoming visible,
>> so that the delay may be minimized.
>>
>> There is a load() method on media element, can it be extended to instruct
>> the UA to recreate
>> the media pipeline? Thus script can reduce the delay if it knows the media
>> is about to be played.
> load() resets all state and starts resource selection anew, so without
> a way of detecting when a media element has destroyed its media
> pipeline to save memory, calling load() can in the worst case increase
> the time until play.
I meant we could add an optional parameter to load() to support "soft 
reload", e.g. load(boolean soft),
which doesn't reset states and re-select resource.

Maybe it is better to reuse pause() method to request UA to recreate the 
media pipeline. If a media element is in
memory-saving state, it must be in paused state as well, so invoke 
pause() should not have undesired side effects.

Anyway, it seems the spec needs to introduce a new state of media 
element: "memory-saving state".
In low memory condition, UA can select some low-priority media elements 
and turn them into memory-saving state.

Suggested priorities for videos are:
(1) recently (re)started, playing, and visible videos
(2) previously (re)started, playing, and visible videos
(3) paused and visible videos; playing and invisible videos
(4) paused and invisible videos

Priorities for audios are to be considered.

Memory-saving state implies paused state.

If memory becomes sufficient, or a media elements priorities are about 
to change, UA can restore some of them to
normal paused state (previously playing media doesn't automatically 
resume playback).

If pause() method is invoked on a media element in memory-saving state, 
UA must restore it to normal paused state.

>> Audios usually eat much less memory, so UAs may have a different strategy
>> for them.
>>
>> Many native media players can save playing position on exit, and resume the
>> playing from that position on the next run.
>> Most users are satisfied with such feature. Is recovering to "exact same
>> state" important to some web applications?
> I don't know what is required for site compat, but ideally destroying
> and recreating a pipeline should get you back to the exact same
> currentTime and continue playback at the correct video frame and audio
> sample. It could be done.
>
>> I'm not familiar with game programing. Are sound effects small audio files
>> that are usually
>> played as a whole? Then it should be safe to recreate the pipeline.
> There's also a trick called audio sprites where you put all sound
> effects into a single file with some silence in between and then seek
> to the appropriate offset.
I think if UA can get and set currentTime property accurately, it should 
be able to recreate the pipeline
with the same accuracy. What are the main factors limiting the accuracy?
However, a UA using priorities to manage media memory is unlikely to 
reclaim a in-use audio sprites element's memory.
>
> Philip

Received on Wednesday, 20 August 2014 10:05:49 UTC