Re: [media] handling multitrack audio / video

On Dec 3, 2010, at 12:31 PM, Silvia Pfeiffer wrote:

> 
> Now there's only one other major use case that we haven't considered
> yet: the case where we have completely alternative resources to the
> main audio/video that contain signing/descriptions/captions/subtitles.
> So we have, for example:
> 
> * v_main.{ogv,mp4,webm} which has main audio & main video
> * v_main_sign.{ogv,mp4,webm} which has main audio & main video & sign
> language video & captions - being particularly targeted at the HoH
> * v_main_desc.{ogv,mp4,webm} which has main audio & main video & audio
> description - being particularly targeted at the VI
> * plus all of this in different languages
> 
> I think this is also a major use case that we will find, in particular
> where the auxiliary accessibility content has been burnt-in.
> 
> I almost think that in this case we have to provide completely
> alternative <video> elements where only one of them is allowed to be
> active. This is where a manifest may come in handy.

This case is challenging, because you need to find a supported format, but want best-fit, not first-fit, on accessibility and language aspects.

Just thinking through how that work work with <source> elements and a kind-like attribute, I could imagine this:

- Source selection algorithm becomes two-pass.
- First pass stable sorts the <source> elements according to a best-fit metric of accessibility affordances, relative to user preferences (this metric would have to be defined). It's a stable sort so that elements are not reoderdered if they are equally good fits.
- The second pass is the usual source-selection algorithm.

This will always result in getting the best accessibility fit out of sources with a supported format.

We would also have to be able ti distinguish optional accommodations vs. burned-in ones. Burned-in accommodations (e.g. burned-in captions or a sign language translation combined with the main video track) should score negatively for users who don't want that accommodation, but optional ones (e.g. built-in but optional closed caption track, or built-in but separate descriptive audio track) do not score positively or negatively.

Whether this is viable depends on whether we can come up with a good scoring rule for matching available accommodations to user needs.

Regards,
Maciej

Received on Friday, 3 December 2010 21:36:36 UTC