Presentation at Video Accessibility Workshop, 1st November 2009
Dr Silvia Pfeiffer
Mozilla/Xiph/Annodex/Vquence


---

User Requirements: Video Accessibility

* accessibility of the <video> and <audio> element (solved)
* needs for hard of hearing (HoH):
  - captions
  - signing video
  - signing captions (!)
* needs for foreigners:
  - subtitles
* needs for vision impaired (VI):
  - audio descriptions
  - textual audio descriptions (timed transcript!) with screen reader
  - textual audio descriptions (timed transcript!) with braille



---

Related Video/Audio User Requirements

* usability:
  - ticker text
  - active regions (timed)
  - chapter breakdown (cue ranges)

* searchability:
  - annotations
  - timed metadata
  - timed transcript

* entertainment/education:
  - karaoke
  - lyrics
  - linguistic markup


---

Technical Requirements

Videa a11y creates more content tracks for video

* support of multi-track audio/video files
* support of multiple text tracks in video files
* standard JavaScript and DOM interface to a/v/t tracks
* content negotiation with server on required tracks










---

Technical Requirements

Text tracks are special: text should be text and not binary

* edit-ability: crowd source text tracks
* integration: CMS requires text in DB
* extensibility: text track from other server/organisation
* sharability: embedding of video with associated text tracks

Requirement for an extension of HTML5 <video> to relate to external text tracks







---

Proposal: <itext> element (V1)

  <video src="example.ogv" poster="example.png"  controls>

    <itext lang="en" type="text/srt" charset="ISO-8859-1" display="auto"
           src="example.english.srt" category="SUB"></itext>
    <itext lang="es-mx" type="text/srt" charset="ISO-8859-1" display="auto"
           src="example.spanish-us.srt" category="SUB"></itext>
    <itext lang="zh" type="text/srt" charset="GB18030" display="auto"
           src="example.chinese.srt" category="SUB"></itext>   
    <itext lang="ja" type="text/srt" charset="EUC-JP" display="auto"
           src="example.japanese-euc.srt" category="SUB"></itext>

    <itext lang="en" type="text/srt" charset="ISO-8859" display="yes"
           src="audiodescription.srt" category="TAD"></itext>

    <itext lang="en" type="text/srt" charset="ISO-8859" display="yes"
           src="chapters.srt" category="CUE"></itext>

  </video>

https://wiki.mozilla.org/Accessibility/HTML5_captions

---

Categories

* CC: closed captions
* SUB: subtitles
* TAD: textual audio descriptions
* KTV: karaoke
* TIK: ticker text
* AR: active regions
* NB: semantic annotations, including speech bubbles and director comments
* META: metadata, mostly machine-readable
* TRX: transcripts / scripts
* LRC: lyrics
* LIN: linguistic markup
* CUE: cue points, DVD style chapter markers and similar navigational landmarks


---

Demo: http://www.annodex.net/~silvia/itext/elephant_no_skin.html

* Note: extra menu
* Note: textual audio description
* Note: default display styles













---

Analysis of the time-aligned text formats:

- sequence of text segments with start/end time markers
- potentially spatial positioning
- potentially styling
- default position, if visible
- default language










---

Multitrack text support in Ogg

Demo: Firefox and Ogg Kate

file:///Users/silvia/Sites/mozilla/itext/elephants_dream/multitrack/elephant.vat_en_t_fr_t_de_t_sp.ogv

* Note: integrated menu










---

Common JavaScript Interface

    void fetch();
    DOMString currentText(currentTime);
    void enable();
    void disable();
    void delay(seconds);










---

Proposal: <itext> element (V2)

  <video class="v" src="elephants_dream/elephant.ogv" poster="elephants_dream/elephant.png" >
    <itextlist category="SUB">
      <itext lang="en" type="text/srt" charset="ISO-8859-1" src="example.english.srt">
      </itext>
      <itext lang="es-mx" type="text/srt" charset="ISO-8859-1" src="example.spanish-us.srt">
      </itext>
      <itext lang="zh" type="text/srt" charset="GB18030" src="example.chinese.srt">
      </itext>   
      <itext lang="ja" type="text/srt" charset="EUC-JP" src="example.japanese-euc.srt">
      </itext>
    </itextlist>

    <itextlist category="TAD">
      <itext lang="en" type="text/srt" charset="ISO-8859" src="audiodescription.srt">
      </itext>
   </itextlist>

    <itextlist category="SUB">
      <itext lang="en" type="text/srt" charset="ISO-8859" src="chapters.srt"></itext>
    </itextlist>


https://wiki.mozilla.org/Accessibility/HTML5_captions_v2

---

Demo: http://www.annodex.net/~silvia/itext/elephant_no_skin_v2.html















---

Common JavaScript Interface:

iTextList:
              attribute Function onenter;
              attribute Function onleave;

iText:
 	interface HTMLItextElement : HTMLElement {
              attribute DOMString src;
              attribute DOMString lang;
              attribute DOMString type;
              attribute DOMString charset;
              attribute float     delay;
              attribute unsigned long stretch;

     readonly attribute boolean fetched;
     readonly attribute ItextError error;
     readonly attribute HTMLCollection allText;
     readonly attribute langName;

     DOMString currentText(currentTime);
   };    

---

Demo: http://oggify.com/


---

Proposal: Discuss text / itextlist proposal and move forward with specification

---


My demos are at: http://www.annodex.net/~silvia/itext/


Thanks.