Re: Change Proposals toward Issue-9: "how accessibility works for <video> is unclear"

On Tue, Apr 13, 2010 at 7:11 PM, Henri Sivonen <hsivonen@iki.fi> wrote:

> "Sean Hayes" <Sean.Hayes@microsoft.com> wrote:
>
> The point of TTML is that by defining a declarative model of time, the
> > caption engine can pre-prepare overlay frames at video rates and
> > inject them into the video pipeline, something your made up semantics
> > doesn’t seem to deal with.
>
> Indeed, what I suggested, by design, doesn't try to achieve per-frame
> alignment of timed text and video frames. Instead, my suggestion optimizes
> for maximal reuse of the existing browser engine capabilities. I'm assuming
> that for video accessibility it is not critically important to achieve
> per-frame time alignment of text and video frames and it's acceptable to
> have slight jitter in the alignment of the video and timed text display
> times.
>
> I was assuming that browser engines will have a way to composite a main
> thread-owned RGBA overlay on top of the video frame "for free" on the GPU,
> so that preparing the overlays and compositing them with the video frames
> ahead of video frame display time is a non-requirement for retaining proper
> video frame rate.
>

That's a safe assumption, for Gecko at least.

If we had to have per-frame alignment of timed text and video frames, we
could do it in Gecko even if the timed text is rendered by our
main-thread-only CSS layout engine and we are compositing the entire Web
page, including video, on a dedicated composition thread. You just have to
maintain a queue of pre-rendered timed text frames alongside the queue of
decoded video frames. That adds complexity but it would still be a lot
better than including another layout engine.

However, I'm not aware of any use-cases that require per-frame alignment of
timed text and video frames.

Rob
-- 
"He was pierced for our transgressions, he was crushed for our iniquities;
the punishment that brought us peace was upon him, and by his wounds we are
healed. We all, like sheep, have gone astray, each of us has turned to his
own way; and the LORD has laid on him the iniquity of us all." [Isaiah
53:5-6]