- From: Joseph Scheuhammer <clown@alum.mit.edu>
- Date: Mon, 4 Jan 2016 13:08:52 -0500
- To: tink@tink.uk, public-apa@w3.org
On 2016-01-04 12:40 PM, Léonie Watson wrote: > Does this mean the test statements are in the test harness, and the test > cases on Github? Where will the results be recorded and/or reported? The quick answer is: the test statements, test cases, and test reports are all within the test harness. Only the test files are located outside of the harness. The longer answer: The organization of the test harness is a hierarchy. The top level is the testharness page itself, which lists tests suites. The suites are keyed to the various aria documents. For example, there is a test suite for the main ARIA 1.1 spec, another for Core-AAM 1.1, and so on. Each suite has three parts. A set of test cases, a set of test runs, and a test report. Each test case contains testable statements for each accessibility API, namely IA2, ATK/AT-SPI, AXAPI, and UIA. The test case also contains a link to the test file, in github. Each test run contains a table whose rows list: the name of a tester, the date the tests were run, the test suite they ran, the accessibility API they tested, and the browser they used. Each test report contains a table whose main column headers names the browser used for the tests. There are column sub-headers grouped under the browser that list which accessibility APIs were tested. For example, FireFox supports both IA2 and ATK/AT-SPI, whereas IE supports only UIA. Each row of the table represents a single test case. The cells going across are, in order, whether the test passed, failed, was inconclusive, or was invalid. At the far right of the table is an aggregation result, where the criterion is, if there are two "passes" in the row for different browsers, then the aggregated result is a "Pass". If the tests passed for some, but not others, there is a "Check" result, meaning more testing of that test case is needed to verify the result. The last column lists the ultimate result for the test case which is either "Pass" or blank. There are other details inside a test suite, test case, test run, and test report, but I'm leaving them out. I fear I've gone into too much detail already. Hope that helps. -- ;;;;joseph. 'Array(16).join("wat" - 1) + " Batman!"' - G. Bernhardt -
Received on Monday, 4 January 2016 18:09:21 UTC