DPub-ARIA Testing

As many of you know, we are entering a phase where we need to be testing
both vocabulary use and support for the dpub-aria roles via the various AT
APIs.  I have not really communicated with most of you before, so I thought
I would send a long a quick overview of how W3C testing for these sorts of
things works (will work).

tl;dr: There is a w3c test framework and we can do the testing within that
framework, gathering information about implementation support at a central
site W3C uses for these purposes.

TEST FRAMEWORK - or W3C TESTING 101

The W3C has a test framework called "Web Platform Tests" or WPT.  This
framework is a rich environment for exercising components of the web
platform.  Dpub-ARIA is, of course, part of that platform.  If you want to
learn about WPT, check out [1]

Each W3C spec has a top level folder in this framework.  Within that folder
each spec group is responsible for populating and validating the tests.
Each test is called a "test case".  Each "test case" can be automated,
manual, or some other modes.

Once tests are run, they produce a JSON file that is sent as input to the
test results repository [2].  Results in that repo are processed by various
reporting tools, including wptreport [3] to generate the reports that can
help inform transitions from W3C CR to PR to Rec.

The rest of this mail talks about how I plan to use these tools to exercise
implementations of the dpub-aria and dpub-aam specifications.  This will
also be reflected and updated in a wiki at [4].

VOCABULARY TESTING

The basic requirement here is that we examine a bunch of real-world content
to demonstrate use of the terms we have introduced.  My simplistic test for
this includes a single "test case" with a sub-test for each term.  This
"test case" has a textarea into which markup can be pasted and then
evaluated.  The markup is evaluated against the terms.  The results can
then be added to the collection of results at [2] so that we have a
comprehensive view of what terms are used and how often.

This style of testing is considered "manual" in that  a tester needs to
paste in a test file in order to evaluate it.  It is POSSIBLE to automate
the execution of these tests via WebDriver.  An example of such automation
is included in the repository.


ACCESSIBILITY API MAPPING TESTING

The basic requirement here is that the various A11Y API implementations are
evaluated to ensure that the DPub-ARIA roles are reflected correctly via
the API mapping and passed to Accessible Technologies.  Spec-Ops and Igalia
have recently worked together to develop an automated technique for this
testing.  You can read up on how that works at [5], but in essence it
queries the AT layer of the platform under test to see if the mappings for
each defined role, state, or property are reflected.  It can also do things
with events, but that isn't relevant to what we are doing.

There is currently one "test case" with a sub-test for each role.  I may
leave it this way, or I may split it out into a single "test case" per
role.  That's a design decision, and will not have much impact on the use
of the tests.

Once the tests are complete, they can be run automatically *or manually* on
the various platforms that have a defined accessibility mapping.  Results
will be sent to [2] in JSON and reports generated, as above.

That's about it.  I don't know that we need any assistance with the
development of the tests at this time.  As they mature it would be great if
people could take a look and provide feedback!

[1] https://github.com/w3c/web-platform-tests
[2] https://github.com/w3c/test-results
[3] https://github.com/w3c/wptreport
[4] https://wiki.spec-ops.io/wiki/DpubTesting
[5] https://spec-ops.github.io/atta-api/index.html
-- 
Shane McCarron
Projects Manager, Spec-Ops