Ensuring we're testing what we need to test

Hey all.

I was looking at some of the testable statements on this page [1], and
have some questions and observations.

Consider the aria-busy="false" on a listbox [2]. The statements
regarding what has changed in 1.1 say:

* Changed aria-busy so that it could be applied to all elements rather
  than limited to live regions;
* Added normative requirement that authors MUST set aria-busy to true
  if changes to a rendered widget would result in that widget missing
  required owned elements during the update process.

The testable statements for ATK are:

* role: ROLE_LIST_BOX
* Selection pattern: return false for all Selection methods that
  provide a means to modify the selection
* STATE-BUSY cleared: <shown>

Note: There are some nits in the above (Selection is an interface, you
cannot show a cleared/removed state). However, for the purpose of this
thread, assume those nits are absent.

My first question: What does the role or the selection interface have to
do with the change we must test for 1.1, namely state busy is now
allowed on a listbox?

It's true that the element in question should have ROLE_LIST_BOX. But
the listbox role is not new to ARIA 1.1, nor is the role in any way
affected by whether or not the element has STATE_BUSY.

I similarly agree that the selection interface should return False when
the AT attempts to change the selection. But that is also not new to
ARIA 1.1. Nor is it (currently) dependent upon whether or not the
element has STATE_BUSY. The reason you cannot change the selection on
this listbox via the accessibility APIs is because it is not a native
select element plus ARIA is one way (can get; can't set).

If the results of the test were STATE_BUSY cleared/absent, ROLE_LIST, I
would argue that we have one of our required implementations (modulo
something I'll come back to in a minute). Yeah, the role is wrong. But
it's wrongness is irrelevant to the 1.1 change for which we need to
demonstrate implementations. And it would almost certainly be wrong even
without aria-busy. And unless it's a regression, it was probably wrong
back in 1.0.

With the above in mind: Should we be limiting our test assertions to
those things which are new to 1.1?

Related to all of this: I mentioned above we'd have one of our required
implementations **modulo something**. Here's what that something is:
Let's say the user agent doesn't support aria-busy on elements which are
not live regions. With that support missing, what would I expect to find
for this test? The very same thing. STATE_BUSY might be absent because
it was removed due to the user agent's ARIA 1.1 implementation. It might
also be absent due to lack of implementation.

In other words, not only is this test testing things that are NOT
relevant to the ARIA 1.1 change, the one ARIA 1.1 change it is testing
is NOT a reliable indication of implementation.

What I, as a consumer of this implementation, would use as a means to
verify implementation on a user agent is the following:

Start test, load file
1. Page has aria-busy="false" on the test element
2. Set aria-busy="true" on that same test element
3. Set aria-busy="false" on that same test element
End test, close file

Item 1 is the existing test. It, in and of itself, tells us nothing. But
when item 2 happens, I would expect two results:

a. STATE_BUSY is present in the state set.
b. an object:state-changed:busy event would be emitted with
   event.detail1 equal to 1 (it's a boolean, but exposed as an int)

When item 3 happens, I would also expect two results:

a. STATE_BUSY has been removed from the state set.
b. an object:state-changed:busy event would be emitted with
   event.detail1 equal to 0

The testable statements in items 2 and 3 tell us what we really need to
know: When STATE_BUSY is absent, it's deliberate; not coincidence. When
aria-busy changes, the user agent is updating things appropriately.

If you don't change the value during the test, you may have an
implementation that sets things correctly initially, but then fails to
update them. Failure to update things is NOT a complete implementation.

Thoughts?
--joanie

[1] https://www.w3.org/wiki/ARIA_1.1_Testable_Statements
[2]
https://www.w3.org/wiki/ARIA_1.1_Testable_Statements#aria-busy.3D.22false.22_on_a_listbox

Received on Friday, 9 September 2016 17:43:47 UTC