Fwd: [Filter Effects][css3-transforms] Using MathML for formulas

Cameron, Dirk and Vincent:

Peter forwarded your message to me, as he felt I might be able to  
contribute to this discussion.  Let me preface my remarks with a  
disclaimer:  I am not an expert in assistive technology, and have only  
worked with a few screen readers, so I don't have a great deal of  
experience with the needs to those who are using assistive  
technology.  What little I do know, however, suggests that screen  
readers vary widely in their adherence to the various standards, and  
that it will be difficult to produce a solution that will work for all  
readers that people have available.

Cameron's code samples are interesting ones, and I appreciated getting  
to see them.  My experience with screen readers, however, suggests  
that text that is visually understandable (like the equation  
"miterLimit / stroke-width = 1 / sin ( theta / 2 )") is only going to  
be appropriate for screen readers only in the simplest of cases.  What  
needs to be spoken for mathematical text is quite different from what  
is needed for ordinary English text, and different again from a visual  
representation of mathematics as a text string (as you have found  
out).  The reason that the parens are not spoken, for example, is that  
parens in a sentence are not spoken -- they affect the pitch and tone,  
but produce no sounds themselves.  Similarly, the comma in the  
mathematics "(x,y)" probably should be spoken, but commas in sentences  
aren't.  Special care must be taken to indicate where quantities are  
begin and end, to distinguish between 1/(x+1) and 1/x + 1, for  
example, or to indicate the extent of a square root's operand, or  
where a matrix cell starts and stops.  There are a whole range of  
issues of this sort that makes it difficult to use a visual  
representation for spoken purposes, unless the screen reader has  
special knowledge of mathematical notation.  For a screen reader with  
knowledge of mathematics, that would most likely be a knowledge of  
MathML, not of some plain-text notation for mathematics, and so trying  
to use such plain-text strings for screen readers at the same time  
they are used as visual alternatives for browser that don't support  
MathML is going to be very difficult.

Since screen readers don't have direct knowledge of how to read  
mathematics (at least I am not aware of one), the knowledge of how to  
speak mathematics has to come from another source.  This is why the  
current accessibility options for mathematics all involve MathPlayer,  
as it is that component that tells the screen readers what to say.   
The problem, of course, is that MathPlayer is only available for  
Internet Explorer, so that limits its applicability.  What you need is  
a more widely available component that can perform a similar function.

My feeling is that MathJax can do much more in this area than it  
currently does.  Since MathJax does understand the structure of the  
mathematics (since it understand MathML), it could produce text  
strings that appropriately represent that mathematics and could  
present those to the screen readers to read, while still providing a  
good visual representation for the sighted.  The idea that has not yet  
been discussed here is that these strings can be generated  
automatically, and MathJax can do that.  It doesn't currently do this,  
but it COULD.

As a proof of concept, I cobbled together a little bit of code that  
extends MathJax to do just that (using the CDN copy of MathJax,  
unaltered).  It is not complete, but includes enough to see that it  
could work.  I have only tested it with Apple's VoiceOver, so I know  
it works with that, but don't know about other screen readers.  My  
past experience was that not all screen readers handle the ARIA  
attributes, and thee are crucial to the functioning of this test  
page.  (Without them, for example, the screen reader will try to read  
the HTML output directly, as well as the special strings that MathJax  
produced for the screen reader, which will produce gibberish (in  
addition to the correct spoken string).  There seem to be effective  
ways of adding text for screen readers that are not visible to the  
sighted, but the reverse is not so universal; we should be able to use  
aria-hidden="true", but this doesn't work for all screen readers.   
(Even VoiceOver doesn't always respect this setting; for example, I  
could not get VoiceOver not to read the pop-up menu in the same page I  
link to below).

That being said, you can try out the sample at:

 http://www.math.union.edu/locate/Cervone/transfer/mathjax/speech-lab.html

You can type TeX code into the area at the top, and press the  
"Typeset" button (or CTRL-RETURN) to get the math to be typeset.  If  
you are using a current enough version of VoiceOver, it should speak  
the new math automatically, otherwise you may need to do CRTL-OPTION-A  
to get it to speak it (and it may speak the pop-up menu as well).   
There are a number of pre-defined examples in the pop-up menu, several  
taken from the SVG documentation.  There is a checkbox that lets you  
see the actual string that is generated.  Note that this is pretty  
worthless for a sighted person, as everything is written out in words,  
and there are a lot of commas inserted to introduce pauses that are  
crucial for the understanding of the spoken versions.  For example,  
your miterLimit to stroke-width ratio expression produces:

 fraction, miterLimit, over stroke-width, end fraction, = fraction, 1,  
over sine of quantity fraction, theta, over 2, end fraction, end  
quantity, end fraction,

Not everything is spoken in an optimal way (since I only spent about 8  
hours putting it all together), and the page linked above only knows  
the words for a small number of symbols.  It seems that VoiceOver says  
nothing for symbols it doesn't understand, so if you use something  
like \subset it will probably skip that.  In any case, there is still  
work to be done to make something like this production-ready, but this  
should give you a sense of what might be possible.

I would be interested in what you think of this approach, and of the  
results of this prototype version.

Davide


> ---------- Forwarded message ----------
> From: Cameron McCormack <cam@mcc.id.au>
> Date: Fri, May 18, 2012 at 9:35 PM
> Subject: Re: [Filter Effects][css3-transforms] Using MathML for  
> formulas
> To: Dirk Schulze <dschulze@adobe.com>
> Cc: Peter Krautzberger <pkrautzb@umich.edu>, Vincent Hardy <vhardy@adobe.com 
> >, "public-fx@w3.org" <public-fx@w3.org>
>
>
> I think some of the existing <pre>-formatted formulae in the spec  
> aren't really accessible yet anyway, especially those that render  
> say matrices with repeated "[" and "]" characters.  The example that  
> I replaced,
>
>  <pre>miterLimit / stroke-width = 1 / sin ( theta / 2 )</pre>
>
> is (at a guess) probably OK.
>
> I think we can use ARIA attributes to mark up the math with  
> equivalent descriptions[1].  Here's a first go at representing the  
> above formula:
>
>  <div role="math" aria-describedby="a">
>    <math display="block">
>      <semantics>
>        <mrow>
>          <mfrac>
>            <mi>miterLength</mi>
>            <mi>stroke-width</mi>
>          </mfrac>
>          <mo>=</mo>
>          <mfrac>
>            <mn>1</mn>
>            <mrow>
>              <mi>sin</mi>
>              <mo>&#x2061;</mo>
>              <mfrac>
>                <mi>θ</mi>
>                <mn>2</mn>
>              </mfrac>
>            </mrow>
>          </mfrac>
>        </mrow>
>        <annotation-xml encoding="application/xhtml+xml">
>          <pre id="a">miterLimit / stroke-width = 1 / sin(theta / 2)</ 
> pre>
>        </annotation-xml>
>      </semantics>
>    </math>
>  </div>
>
>
> This is not going to work with MathJax -- either with the SVG or HTML 
> +CSS renderings -- since as people have pointed out the <math>  
> element gets replaced with content like this (in the SVG case):
>
>  <div style="text-align: center;" aria-readonly="true" role="textbox"
>       class="MathJax_SVG_Display">
>    <span class="MathJax_SVG" style="display: inline-block;">
>      <svg>
>        ...
>      </svg>
>    </span>
>  </div>
>  <script type="math/mml">
>    (CDATA representing the serialised <math> tree we started with)
>  </script>
>
> so the element with id="a" is gone.
>
> (I'm not sure why a read only text box is the best way to present  
> the <svg> representation of the math to ATs.)
>
>
> Regardless, I'm after a practical solution here -- so perhaps we can  
> just fall back on the old "position some text for the screen reader  
> off the page where the sighted reader can't see it".
>
>  <style>
>    div[role="math"] > :first-child + * {
>      position: absolute;
>      left: -400px;
>      width: 400px;
>      overflow: hidden;
>    }
>  </style>
>  <div role="math" aria-describedby="a">
>    <math display="block">
>      <mrow>
>        <mfrac>
>          <mi>miterLength</mi>
>          <mi>stroke-width</mi>
>        </mfrac>
>        <mo>=</mo>
>        <mfrac>
>          <mn>1</mn>
>          <mrow>
>            <mi>sin</mi>
>            <mo>&#x2061;</mo>
>            <mfrac>
>              <mi>θ</mi>
>              <mn>2</mn>
>            </mfrac>
>          </mrow>
>        </mfrac>
>      </mrow>
>    </math>
>    <pre id="a">miterLimit / stroke-width = 1 / sin(theta / 2)</pre>
>  </div>
>
> This worked in Safari with VoiceOver -- although surprisingly to me,  
> it didn't read the parentheses; I'm not sure if there is a way to  
> investigate the exact characters that are in there with VO or other  
> ATs.
>
> Can anyone suggest any better way to represent the math content in  
> HTML besides a <pre> with content that looks like code?  Especially  
> for more complex formulae like the ones in http://www.w3.org/TR/SVG11/implnote.html#ArcConversionEndpointToCenter 
> .  Are screen reader users who read math on the web comfortable with  
> TeX?  (I notice that Wikipedia uses <img>s of rendered math with TeX  
> in the alt="".)
>
>
> Aside: it'd be neat if I could use CSS to provide aria-describedby  
> relationships for all the math in the page at once, for example  
> something like:
>
>  div[role="math"] {
>    aria-describedby: selector(:scope > :first-child + *);
>  };
>
>
> [1] http://www.w3.org/TR/wai-aria/roles#math
>
>
>