RE: HTML and Multimedia Synchronization

Silvia Pfeiffer,

The email was broad, introducing concepts for discussion, suggestions for features for HTML.  Concretely, the suggested features are multimedia synchronization with speech synthesis and with 3D graphics, WebGL or X3DOM.  The functionality can be achieved with XHTML and JavaScript; HTML could come to include multimedia synchronization features; additionally, a new HTML+SMIL document could be a deliverable.  The interoperabilities between multimedia synchronization, JavaScript, speech synthesis, and 3D graphics are suggested features.

A specific example includes a digital mathematics book or textbook with a page of hypertext and a 3D graphic or widget which renders 3D graphics with multitouch ergonomics.  In the example, the 3D graphic is a mathematical function and some hypertext on the page discusses the mathematical function.  The hypertext content contains hyperlinks which activate rotation, scaling, translation, and animations of the 3D graphic or objects in the 3D graphic.

In the example, there is audio overlay interoperability.  The page has one or more audio overlays available for users and, when a voice actor reads the content, the 3D graphic of the mathematical function rotates, scales, translates, and animates as if the hyperlinks were being clicked as the voice actor read each.

In the example, there is also speech synthesis interoperability.  When a speech synthesizer synthesizes the hypertext content, the 3D graphic of the mathematical function rotates, scales, translates, and animates as if the hyperlinks were being clicked as the synthesizer synthesized each.

An example hypertext region:

<p>{1}... as you can see in {2}, the <a href="{3}" {4}>...</a> of the function...{/1}</p>

Region {1} is a sentence element, and could be a <span>, <ssml:s> in XHTML, or new element.

Region {2} is hypertext to refer to or cross-reference a figure or 3D graphics element.

Region {3} calls a JavaScript function.

Region {4} is markup pertaining to multimedia synchronization including for audio overlays and speech synthesis.



Kind regards,

Adam