Re: Ensuring we're testing what we need to test

Hi Joanie,

I am still working on those particular statements/tests.

:-)

Sent from my iPhone

> On Sep 9, 2016, at 12:43 PM, Joanmarie Diggs <jdiggs@igalia.com> wrote:
> 
> Hey all.
> 
> I was looking at some of the testable statements on this page [1], and
> have some questions and observations.
> 
> Consider the aria-busy="false" on a listbox [2]. The statements
> regarding what has changed in 1.1 say:
> 
> * Changed aria-busy so that it could be applied to all elements rather
>  than limited to live regions;
> * Added normative requirement that authors MUST set aria-busy to true
>  if changes to a rendered widget would result in that widget missing
>  required owned elements during the update process.
> 
> The testable statements for ATK are:
> 
> * role: ROLE_LIST_BOX
> * Selection pattern: return false for all Selection methods that
>  provide a means to modify the selection
> * STATE-BUSY cleared: <shown>
> 
> Note: There are some nits in the above (Selection is an interface, you
> cannot show a cleared/removed state). However, for the purpose of this
> thread, assume those nits are absent.
> 
> My first question: What does the role or the selection interface have to
> do with the change we must test for 1.1, namely state busy is now
> allowed on a listbox?
> 
> It's true that the element in question should have ROLE_LIST_BOX. But
> the listbox role is not new to ARIA 1.1, nor is the role in any way
> affected by whether or not the element has STATE_BUSY.
> 
> I similarly agree that the selection interface should return False when
> the AT attempts to change the selection. But that is also not new to
> ARIA 1.1. Nor is it (currently) dependent upon whether or not the
> element has STATE_BUSY. The reason you cannot change the selection on
> this listbox via the accessibility APIs is because it is not a native
> select element plus ARIA is one way (can get; can't set).
> 
> If the results of the test were STATE_BUSY cleared/absent, ROLE_LIST, I
> would argue that we have one of our required implementations (modulo
> something I'll come back to in a minute). Yeah, the role is wrong. But
> it's wrongness is irrelevant to the 1.1 change for which we need to
> demonstrate implementations. And it would almost certainly be wrong even
> without aria-busy. And unless it's a regression, it was probably wrong
> back in 1.0.
> 
> With the above in mind: Should we be limiting our test assertions to
> those things which are new to 1.1?
> 
> Related to all of this: I mentioned above we'd have one of our required
> implementations **modulo something**. Here's what that something is:
> Let's say the user agent doesn't support aria-busy on elements which are
> not live regions. With that support missing, what would I expect to find
> for this test? The very same thing. STATE_BUSY might be absent because
> it was removed due to the user agent's ARIA 1.1 implementation. It might
> also be absent due to lack of implementation.
> 
> In other words, not only is this test testing things that are NOT
> relevant to the ARIA 1.1 change, the one ARIA 1.1 change it is testing
> is NOT a reliable indication of implementation.
> 
> What I, as a consumer of this implementation, would use as a means to
> verify implementation on a user agent is the following:
> 
> Start test, load file
> 1. Page has aria-busy="false" on the test element
> 2. Set aria-busy="true" on that same test element
> 3. Set aria-busy="false" on that same test element
> End test, close file
> 
> Item 1 is the existing test. It, in and of itself, tells us nothing. But
> when item 2 happens, I would expect two results:
> 
> a. STATE_BUSY is present in the state set.
> b. an object:state-changed:busy event would be emitted with
>   event.detail1 equal to 1 (it's a boolean, but exposed as an int)
> 
> When item 3 happens, I would also expect two results:
> 
> a. STATE_BUSY has been removed from the state set.
> b. an object:state-changed:busy event would be emitted with
>   event.detail1 equal to 0
> 
> The testable statements in items 2 and 3 tell us what we really need to
> know: When STATE_BUSY is absent, it's deliberate; not coincidence. When
> aria-busy changes, the user agent is updating things appropriately.
> 
> If you don't change the value during the test, you may have an
> implementation that sets things correctly initially, but then fails to
> update them. Failure to update things is NOT a complete implementation.
> 
> Thoughts?
> --joanie
> 
> [1] https://www.w3.org/wiki/ARIA_1.1_Testable_Statements
> [2]
> https://www.w3.org/wiki/ARIA_1.1_Testable_Statements#aria-busy.3D.22false.22_on_a_listbox
>

Received on Saturday, 10 September 2016 20:18:37 UTC