Re: APA test harness availability

On 2016-01-04 12:40 PM, Léonie Watson wrote:
> Does this mean the test statements are in the test harness, and the test
> cases on Github? Where will the results be recorded and/or reported?

The quick answer is:  the test statements, test cases, and test reports 
are all within the test harness.  Only the test files are located 
outside of the harness.

The longer answer:  The organization of the test harness is a 
hierarchy.  The top level is the testharness page itself, which lists 
tests suites.  The suites are keyed to the various aria documents. For 
example, there is a test suite for the main ARIA 1.1 spec, another for 
Core-AAM 1.1, and so on.

Each suite has three parts.  A set of test cases, a set of test runs, 
and a test report.

Each test case contains testable statements for each accessibility API, 
namely IA2, ATK/AT-SPI, AXAPI, and UIA.  The test case also contains a 
link to the test file, in github.

Each test run contains a table whose rows list:  the name of a tester, 
the date the tests were run, the test suite they ran, the accessibility 
API they tested, and the browser they used.

Each test report contains a table whose main column headers names the 
browser used for the tests.  There are column sub-headers grouped under 
the browser that list which accessibility APIs were tested.  For 
example, FireFox supports both IA2 and ATK/AT-SPI, whereas IE supports 
only UIA.  Each row of the table represents a single test case.  The 
cells going across are, in order, whether the test passed, failed, was 
inconclusive, or was invalid.  At the far right of the table is an 
aggregation result, where the criterion is, if there are two "passes" in 
the row for different browsers, then the aggregated result is a "Pass".  
If the tests passed for some, but not others, there is a "Check" result, 
meaning more testing of that test case is needed to verify the result.  
The last column lists the ultimate result for the test case which is 
either "Pass" or blank.

There are other details inside a test suite, test case, test run, and 
test report, but I'm leaving them out.  I fear I've gone into too much 
detail already.

Hope that helps.

-- 
;;;;joseph.

'Array(16).join("wat" - 1) + " Batman!"'
            - G. Bernhardt -

Received on Monday, 4 January 2016 18:09:21 UTC