ARIA-AT CG Working Mode

Tests

Roles

Role	Responsibility
Test developer	Research AT behavior, propose assertions, and develop test plans and test files.
Tester	Manually test AT against authored assertions.
AT implementer	Review assertions, priorities, test results, and fix bugs in AT.
Web developer (or other interested party)	Review assertions, priorities, and test results.
Community Group	Resolve differences in draft test results. Develop consensus on assertions and priorities.
Community Group Chair	Resolve conflicts
ARIA Working Group	Follow up on issues with the ARIA spec, APG, browser implementations, or OS-level accessibility API implementations.

Process

(Test developer) Select an APG design pattern or ARIA state or property to test.
(Test developer) Study the current behavior in AT.
- Create a demo and test it in different browser/screen reader combinations. Document what the behavior is and any differences between browsers or ATs.
- Test different user settings in the AT. Document if changing a user setting affects the behavior of the current test.
(Test developer) Read the relevant parts of the ARIA specification and the APG document to understand the requirements and the intent of the features being tested. Document your understanding and provide links.
(Test developer) Report research findings to the Community Group.
(Test developer) Propose assertions that
- is aligned with the requirements in the ARIA specification
- is aligned with the intent of the ARIA specification and the APG documentation
- is aligned with what ATs already do or what AT implementers are willing to implement
(Test developer) Propose priorities for the assertions (must have, optional, user setting)
(Test developer) Write a test plan
(Test developer) Write the test
(AT implementer, optionally Web developer) Review the proposed assertions and priorities (Timebox: X)
(Tester) Run the test. Each test is run by at least two testers.
(Tester) Report raw test results.
(Community Group) If there are differences between draft results, resolve those differences.
(Community Group) Report draft test results.
(AT implementer, optionally Web developer) Review the draft test results. (Timebox: X)
- Provide feedback if any.
- Mark test results as “reviewed” if they are OK.
- AT implementer may change their implementation and ask the Community Group to re-run the test in a new build/version.
(Community Group) If there are issues in the ARIA specification, the APG documentation, browser implementations, or OS-level accessibility API implementations, then file an issue for the ARIA WG.
(AT implementer) If there is a bug in AT, file the bug for the AT.
(AT implementer) Fix the bug, or provide (late) feedback on the test.

The chairs should seek wide review at a regular cadence.

Conflict resolution

If agreement regarding (1) what is being asserted in a test or (2) the assertion’s priority cannot be reached, it can be resolved as follows:

The test author should evaluate the feedback and make a decision on whether to redo steps 5-13, or respond and keep the test as-is.
If the involved AT implementers agree with each other but disagree with the test author, then:
- The test author should edit the test plan and the test to reflect AT implementer consensus, or escalate to the chairs.

In any case, if agreement still cannot be made, either party can escalate to the chairs.

Escalation to chairs

Escalate by sending an email to the Chairs, or at-mentioning the Chairs in the relevant GitHub issue.

The Chairs attempt to resolve the conflict. If this is not possible, the Chairs can opt for one of the following, as they deem appropriate:

Find a solution that is mutually acceptable.
Choose one behavior based on merit or based on majority implemented behavior in AT.
Raise the issue with the ARIA WG.
Remove the test.

The chairs should respond within X.

Test Runner

The development plan of the test runner is outlined in a separate document.

Provide feedback by reporting an issue in GitHub with “[runner]” prefix in the issue summary.

If there are disagreements that can’t be resolved among the involved parties, the issue can be escalated to the chairs.

GitHub management

TODO - see https://github.com/w3c/aria-at/issues/57