Re: HTML and Multimedia Synchronization

Hi Adam,

On Sun, Jan 6, 2013 at 2:57 AM, Adam Sobieski <adamsobieski@hotmail.com>wrote:

> Silvia,
>
> I have implemented mathematics documents, document pages and portions of
> digital mathematics books and textbooks, including based upon pages and
> portions of existing documents, AMS documents, math books, and textbooks,
> including while looking at specific features and features in combination,
> for example CSS columns and MathML3 line wrapping (
> https://groups.google.com/forum/#!msg/mozilla.dev.platform/KCIMqFwVaIY/bkIfgjPERo0J
> ).
>
> On the topic of scholarly and scientific communication, an interesting use
> case is mathematics documents with device-specific formatting (
> http://www.w3.org/TR/css3-mediaqueries/), for example columnization (
> http://www.w3.org/TR/css3-multicol/) and widescreen displays, that print
> styled as per AMS document style (
> ftp://ftp.ams.org/pub/author-info/documentation/handbk.pdf).
>
> I participate in some discussion groups specifically interested in digital
> STEM books and textbooks and, more specifically, digital mathematics books
> and textbooks; I could collect together their observations and invite them
> and other participants here to provide feedback upcoming;
>

That would be valuable.


a digital textbook team indicated to me, for example, an interest in the
> speech recognition of mathematics. Some more topics will be available
> upcoming with some available on the 15th when papers from the upcoming W3C
> digital publishing workshop will be available online.
>

Are you aware that the Speech API CG has developed a proposed specification
for speech recognition and synthesis:
http://www.w3.org/community/speech-api/ and
http://dvcs.w3.org/hg/speech-api/raw-file/9a0075d25326/speechapi.html ? I
am not sure about its status and whether it is being adopted into the work
of a W3C WG. Is your suggestion to adopt it as a piece of work for this
Working Group?


Some HTML and multimedia synchronization techniques are possible with with
> HTML, XHTML, and JavaScript. We can observe the timesheets.js project (
> http://wam.inrialpes.fr/timesheets/), for example, with internal
> timesheets (http://wam.inrialpes.fr/timesheets/markup/intTimesheet.xhtml),
> external timesheets (
> http://wam.inrialpes.fr/timesheets/markup/extTimesheet.html), smil-*
> attributes (http://wam.inrialpes.fr/timesheets/markup/smilPrefix.html),
> data-* attributes (
> http://wam.inrialpes.fr/timesheets/markup/dataPrefix.html), smil:*
> attributes (http://wam.inrialpes.fr/timesheets/markup/smilNS.xhtml), and
> inline SMIL (http://wam.inrialpes.fr/timesheets/markup/noPrefix.html).
>
> Enhancing the Web-based and desktop-based indexing, searching, and
> retrieving of RDFa-enhanced hypertext-based time-based multimedia documents
> or configurations of such documents, or bookmarks, can become topical.
>

Indexing, search and retrieval are applications and don't need improvements
to the HTML specification. The W3C Media Fragments WG also has developed
specifications that allow the creation of hyperlinks into time-based
multimedia documents, see http://www.w3.org/TR/media-frags/ . As URLs can
be stored as bookmarks, these use cases are already possible.



> Standardizing HTML and multimedia synchronization, for example with HTML
> 5.1, a new HTML+SMIL document (
> http://www.w3.org/TR/2000/WD-smil-boston-20000622/html-smil-profile.html),
> and/or a SMIL 4 specification, can enhance the indexing, searching and
> retrieving of, the searchability of, content with multimedia
> synchronization features. Some topics in multimedia synchronization include
> speech synthesis and 3D graphics.
>

3D graphics are indeed interesting. There is a whole consortium looking at
3D graphics on the Web: http://www.web3d.org/realtime-3d/ and X3D files can
be used in browser applets. Are you suggesting we should make it more of a
native Web format? Have browser vendors expressed interest in implementing
3D graphics support natively?


*EPUB Media Overlays 3.0* (
> http://idpf.org/epub/30/spec/epub30-mediaoverlays.html) describes
> multimedia synchronization, specifically media overlays, with regard to the
> EPUB3 format.
>

I am not convinced that Web pages need media overlays. We have many years
of experience in making Web pages accessible by proving text alternatives
for certain pieces of content and by having AT communicate either existing
text on a page or provided text alternatives to a11y users. Media overlays
make sense for books where there is a linear reading experience. Web pages,
and more so modern Web applications don't generally follow a linear reading
experience and thus text and text alternatives that are associated with
user's focus seem more adequate. I would, however, suggest discussing such
needs with WAI.

Best Regards,
Silvia.