Re: [whatwg] Memory management problem of video elements

于 2014年08月19日 16:00, Philip Jägenstedt 写道:
> On Tue, Aug 19, 2014 at 9:12 AM, duanyao <duanyao@ustc.edu> wrote:
>> Hi,
>>
>> Recently I have investigated memory usage of HTML video element in
>> several desktop browsers (firefox and chrome on windows and linux, and
>> IE 11), and have found some disappointing results:
>>
>> 1. A video element in a playable state consumes significant amount of
>> memory. For each playing or paused or preload=auto video element, the
>> memory usage
>> is up to 30~80MB; for those with preload=metadata, memory usage is
>> 6~13MB; for those with preload=none, memory usage is not notable. Above
>> numbers are measured with 720p to 1080p H.264 videos, and videos in
>> lower resolutions use less memory.
>>
>> 2. For a page having multiple video elements, memory usage is scaled up
>> linearly. So a page with tens of videos can exhaust the memory space of
>> a 32bit browser. In my tests, such a page may crash the browser or
>> freeze a low memory system.
>>
>> 3. Even if a video element is or becomes invisible, either by being out
>> of viewport, having display:none style, or being removed from the active
>> DOM tree (but not released),
>> almost same amount of memory is still occupied.
>>
>> 4. The methods to reduce memory occupied by video elements requires
>> script, and the element must be modified. For example, remove and
>> release the element.
>>
>> Although this looks like a implementors' problem, not a spec's problem,
>> but I think the current spec is encouraging implementors to push the
>> responsibility of memory management of media elements to authors, which
>> is very bad. See the section 4.8.14.18
>> (http://www.whatwg.org/specs/web-apps/current-work/multipage/embedded-content.html#best-practices-for-authors-using-media-elements):
>>
>>> 4.8.14.18 Best practices for authors using media elements
>>> it is a good practice to release resources held by media elements when
>> they are done playing, either by being very careful about removing all
>> references to the element and allowing it to be garbage collected, or,
>> even better, by removing the element's src attribute and any source
>> element descendants, and invoking the element's load() method.
>>
>> Why this is BAD in my opinion?
>>
>> 1. It requires script. What if the UA doesn't support or disables script
>> (email reader, epub reader, etc), or the script is simply failed to
>> download? What if users insert many video elements to a page hosted by a
>> site that is not aware of this problem (so no video management script
>> available)? Users' browsers may be crashed, or systems may be freezed,
>> with no obvious reason.
>>
>> 2. It is hard to make the script correct. Authors can't simply depend on
>> "done playing", because users may simply pause a video in the middle and
>> start playing another one, and then resume the first one. So authors
>> have to determine which video is out of viewport, and remove its src,
>> and record its currentTime; when it comes back to viewport, set src and
>> seek to previous currentTime. This is quite complicated. For WYSIWYG
>> html editors based on browsers, this is even more complicated because of
>> the interaction with undo manager.
>>
>> 3. Browsers are at a much better position to make memory management
>> correct. Browsers should be able to save most of the memory of an
>> invisible video by only keep its state (or with a current frame), and
>> limit the total amount of memory used by media elements.
>>
>> So I think the spec should remove section 4.8.14.1, and instead stresses
>> the the responsibility of UA to memory management of media elements.
> What concrete advice should the spec give to UAs on memory management?
> If a script creates a thousand media elements and seeks those to a
> thousand different offsets, what is a browser to do? It looks like a
> game preparing a lot of sound effects with the expectation that they
> will be ready to go, so which ones should be thrown out?

UA can limit the number of simultaneously playing medias according to 
available memory or user preference,
and fire error events on media elements if the limit is hit. We may need 
another error code, currently some UAs fire MEDIA_ERR_DECODE,
which is misleading.

If the thousand media elements are just sought, not playing, UA can seek 
them one by one, and drop cached frames afterwards, only keep current 
frames;
if memory is even more limited, the current frames can also be dropped.

For a html based slideshows or textbooks, it is quite possible to have 
tens of videos in one html file.

For audio elements, I think it is less problematic because they usually 
use far less memory than videos.
> A media element in an active document never gets into a state where it
> could never start playing again, so I don't know what to do other than
> trying to use less memory per media element.
What do you mean by "a state where it could never start playing again"?
If the media element object keeps track of its current playing url and 
current position (this requires little memory), and the media file is 
seekable, then
the media is always resumable. UA can drop any other associated memory 
of the media element, and users will not notice any difference except a 
small delay
when they resume playing.
> Have you filed bugs at
> the browsers that crash or freeze the system?
Yes, for firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1054170
> Regardless of what the UA does, section 4.8.14.1 is still good advice
> when the script knows that a resource won't be needed but the browser
> cannot. Example: a sound effect is played for last time in a game as
> the last secret in the level is found.
I think UAs can manage this situation quite well if a LRU mechanism is 
used to reclaim media elements' associated memory.
In a some games, which sound effect is to be played is determined by 
user actions, so game developers have to implement their own LRU
if UAs don't provide it.

If a page use many images, it may also consume a lot of memory. Does the 
spec want to add a similar advice for img element? I don't think so.
>
> Philip

Received on Tuesday, 19 August 2014 09:57:28 UTC