Re: [media] discussion on how to render captions in HTML5

On Aug 18, 2010, at 4:40 PM, Silvia Pfeiffer wrote:

> Hi media a11y folks, hi Eric,
> 
> In today's call we had a brief discussion on the WHATWG specification of rendering of time-synchronized text with audio and video resources, see http://www.whatwg.org/specs/web-apps/current-work/complete/rendering.html#timed-tracks-0 .
> 
> The point I was making was that I am disappointed that for <audio> elements there is no rendering. I was suggesting that for both, <audio> and <video> elements, rendering of time-synchronized text should depend on the @controls attribute of the <audio> and <video> elements. The reason behind this is that I expect a menu to be made available to the user through the @controls that allows the user to activate/deactivate text tracks from the list of available text tracks. Because that list is made available through the @controls, I would also expect that the rendering of the text cues depends on this @controls attribute being available.
> 
> However, the specification says that <audio> elements don't render any time-synchronized text, but only <video> elements do.
> 
> We didn't get very far in the discussion - in particular Eric had some important points to make. Thus, I'd like to take up this discussion here again.
> 
  My point is that semantically, the *only* difference between <video> and <audio> elements is the former renders visual media while the later does not. There is absolutely no requirement that a file in a <video> element must have visual media, eg. it is perfectly legal to use an mp3 file in a <video> element. For example, the following :

    <video src="song.mp3" id="video" controls> </video>

creates a 300x150 element that has only audio data so it doesn't draw anything (300x150 is the default size of a <video> element).

  There is also no requirement that an <audio> element must not support files with visual media, it just doesn't render visual data. In the following example, an HD movie trailer in an <audio> element. The 'controls' attribute tells the UA to show the default controls, so it does take up space on the page but it does not render the visual track:

    <audio src="HD_trailer_1024x768.mp4" id="audio" controls> </audio>

 I believe that time-synchronized text, whether it come from a track in the media file or is loaded from an external file, is *visual media* - it has a visual representation - so I don't believe it make sense for an <audio> element to render them. 

  Silvia's proposal is to render text cues when an <audio> element has the 'controls' attribute. This might work for text-only cues, but if we allow an <audio> element to render time-synchronized text we of course have to allow it to render burned-in captions, sign language tracks, etc. Those are both video tracks, so her proposal is actually to make an <audio> element behave like a <video> element when it has a 'controls' attribute. This change would break the previous example because the video track would be rendered, requiring the page author to edit the movie to remove the video track.

  I believe that the person creating the web page should use a <video> element if a file has visual media, whether it comes from a video track or from time-synchronized text.

eric