[media] how to support extended text descriptions

Hi all,

I'm aware that we are currently focused on trying to sort out
hierarchical navigation, but I also want to bring the extended text
descriptions into the mix to make sure we understand how that can
work.

Ian and I have discussed recently how we would address this need and
whether there would be a requirement for extra markup.

I believe the below discussion is a good summary of how we envisage it
to work. Please provide feedback and ask questions if it anything is
unclear.

With the below described approach (which we discussed at the WHATWG),
there is no need for extra markup, but there is a requirement on the
accessibility API between a screen reader and the browser. On that API
we would need the possibility for the screen reader to influence how
the player works.

I suggest we need to talk to developers of browser accessibility APIs
and also to screen reader developers to see what they say about it and
whether this is technically realistic. I would encourage you to try
and get such information.

Looking forward to your comments.

Cheers,
Silvia.


On Tue, 24 May 2011, Silvia Pfeiffer wrote:
>
> Ian and I had a brief conversation recently where I mentioned a problem
> with extended text descriptions with screen readers (and worse still
> with braille devices) and the suggestion was that the "paused for user
> interaction" state of a media element may be the solution. I would like
> to pick this up and discuss in detail how that would work to confirm my
> sketchy understanding.
>
> *The use case:*
>
> In the specification for media elements we have a <track> kind of
> "descriptions", which are:
> "Textual descriptions of the video component of the media resource,
> intended for audio synthesis when the visual component is unavailable
> (e.g. because the user is interacting with the application without a
> screen while driving, or because the user is blind). Synthesized as a
> separate audio track."
>
> I'm for now assuming that the synthesis will be done through a screen
> reader and not through the browser itself, thus making the
> descriptions available to users as synthesized audio or as braille if
> the screen reader is set up for a braille device.
>
> The textual descriptions are provided as chunks of text with a start
> and a end time (so-called "cues"). The cues are processed during video
> playback as the video's playback time starts to fall within the time
> frame of the cue. Thus, it is expected the that cues are consumed
> during the cue's time frame and are not present any more when the end
> time of the cue is reached, so they don't conflict with the video's
> normal audio.
>
> However, on many occasions, it is not possible to consume the cue text
> in the given time frame. In particular not in the following
> situations:
>
> 1. The screen reader takes longer to read out the cue text than the
> cue's time frame provides for. This is particularly the case with long
> cue text, but also when the screen reader's reading rate is slower
> than what the author of the cue text expected.
>
> 2. The braille device is used for reading. Since reading braille is
> much slower than listening to read-out text, the cue time frame will
> invariably be too short.
>
> 3. The user seeked right into the middle of a cue and thus the time
> frame that is available for reading out the cue text is shorter than
> the cue author calculated with.
>
> Correct me if I'm wrong, but it seems that what we need is a way for
> the screen reader to pause the video element from continuing to play
> while the screen reader is still busy delivering the cue text. (In
> a11y talk: what is required is a means to deal with "extended
> descriptions", which extend the timeline of the video.) Once it's
> finished presenting, it can resume the video element's playback.
>
> IIUC, a video is "paused for user interaction" basically when the UA has
> decided to pause the video without the user asking to pause it (i.e. the
> paused attribute is false) and the pausing happened not for network
> buffering reasons, but for other reasons. IIUC one concrete situation
> where this state is used is when the UA has reached the end of the
> resource and is waiting for more data to come (e.g. on a live stream).

Ian's comment:
That latter state is not "paused for user interaction", it's just stalled
due to lack of data. The rest is accurate though.


> To use "paused for user interaction" for extending descriptions, we need
> to introduce a means for the screen reader to tell the UA to pause the
> video when it reaches the end of the cue and it's still busy delivering
> a cue's text. Then, as it finishes, it will un-pause the video to let it
> continue playing.
>
> To me it sounds like a feasible solution.
>
> The screen reader could even provide a user setting and a short-cut so a
> user can decide that they don't want this pausing to happen or that they
> want to move on from the current cue.
>
> Another advantage of this approach is that e.g. a deaf-blind user could
> hook up their braille device such that it will deliver the extended
> descriptions and also deliver captions through braille with such
> extension pausing happening. (Not sure that such a user would even want
> to play the video, but it would be possible.)
>
> Now, I think there is one problem though (at least as far as I can
> tell). Right now, IIUC, screen readers are only passive listeners on the
> UA. They don't influence the behaviour of the UA. The accessibility API
> is basically only a one-way street from the UA to the AT. I wonder if
> that is a major inhibitor of using this approach or whether it's easy
> for UAs to overcome this limitation? (Or if such a limitation even
> exists - I don't know enough about how AT work...).
>
> Is that an issue? Are there other issues that I have overlooked?

Ian's comment:
That seems to be entirely an implementation issue.

Received on Saturday, 4 June 2011 05:42:06 UTC