RE: DPub-ARIA Testing

Thanks, Shane

In addition to formal vocabulary testing, we are also collecting testimonials from User Agents, such as reading systems. It is helpful to provide documentation of systems that do things like alter their actions or UI based on the presence of DPUB-ARIA or epub:type terms in our tests.  Thanks to those who have already submitted.

Tzviya

Tzviya Siegman
Information Standards Lead
Wiley
201-748-6884
tsiegman@wiley.com<mailto:tsiegman@wiley.com>

From: Shane McCarron [mailto:shane@spec-ops.io]
Sent: Monday, December 26, 2016 11:43 AM
To: public-digipub.; W3C PF - DPUB Joint Task Force
Subject: DPub-ARIA Testing

As many of you know, we are entering a phase where we need to be testing both vocabulary use and support for the dpub-aria roles via the various AT APIs.  I have not really communicated with most of you before, so I thought I would send a long a quick overview of how W3C testing for these sorts of things works (will work).

tl;dr: There is a w3c test framework and we can do the testing within that framework, gathering information about implementation support at a central site W3C uses for these purposes.

TEST FRAMEWORK - or W3C TESTING 101

The W3C has a test framework called "Web Platform Tests" or WPT.  This framework is a rich environment for exercising components of the web platform.  Dpub-ARIA is, of course, part of that platform.  If you want to learn about WPT, check out [1]

Each W3C spec has a top level folder in this framework.  Within that folder each spec group is responsible for populating and validating the tests.  Each test is called a "test case".  Each "test case" can be automated, manual, or some other modes.

Once tests are run, they produce a JSON file that is sent as input to the test results repository [2].  Results in that repo are processed by various reporting tools, including wptreport [3] to generate the reports that can help inform transitions from W3C CR to PR to Rec.

The rest of this mail talks about how I plan to use these tools to exercise implementations of the dpub-aria and dpub-aam specifications.  This will also be reflected and updated in a wiki at [4].

VOCABULARY TESTING

The basic requirement here is that we examine a bunch of real-world content to demonstrate use of the terms we have introduced.  My simplistic test for this includes a single "test case" with a sub-test for each term.  This "test case" has a textarea into which markup can be pasted and then evaluated.  The markup is evaluated against the terms.  The results can then be added to the collection of results at [2] so that we have a comprehensive view of what terms are used and how often.

This style of testing is considered "manual" in that  a tester needs to paste in a test file in order to evaluate it.  It is POSSIBLE to automate the execution of these tests via WebDriver.  An example of such automation is included in the repository.


ACCESSIBILITY API MAPPING TESTING

The basic requirement here is that the various A11Y API implementations are evaluated to ensure that the DPub-ARIA roles are reflected correctly via the API mapping and passed to Accessible Technologies.  Spec-Ops and Igalia have recently worked together to develop an automated technique for this testing.  You can read up on how that works at [5], but in essence it queries the AT layer of the platform under test to see if the mappings for each defined role, state, or property are reflected.  It can also do things with events, but that isn't relevant to what we are doing.

There is currently one "test case" with a sub-test for each role.  I may leave it this way, or I may split it out into a single "test case" per role.  That's a design decision, and will not have much impact on the use of the tests.

Once the tests are complete, they can be run automatically *or manually* on the various platforms that have a defined accessibility mapping.  Results will be sent to [2] in JSON and reports generated, as above.

That's about it.  I don't know that we need any assistance with the development of the tests at this time.  As they mature it would be great if people could take a look and provide feedback!

[1] https://github.com/w3c/web-platform-tests

[2] https://github.com/w3c/test-results

[3] https://github.com/w3c/wptreport

[4] https://wiki.spec-ops.io/wiki/DpubTesting

[5] https://spec-ops.github.io/atta-api/index.html

--
Shane McCarron
Projects Manager, Spec-Ops