Re: Task testing structure

Hi Shawn / all,

After an experiment the last two weeks with "Headings" based on Johns pages
and Bruce his approach I came to the same kind of conclusions as you did
and understand your approach, but I see bears on the road with combining
too much as this creates way too much complexity, takes too much time to
test, and will lack completeness.

Some findings based on the Headings experiment:

- Last week I also had a call with Wilco about using and extending ACT
rules as this seems most appropriate for methods
- Reached the conclusion we need a low, medium, high severity like scoring
as not all fails are as impactful as others
- Next to the fails and severity you also have the criticality for the task
being performed and should probably be judged somehow.
- Adding the functional areas in your testing scoring directly makes
testing undesirably long / complex

So when I read your proposal I see similarities in conclusion and the way
you've tried to fix them, but also so many possible variations and outcomes
that in general this will possibly not be feasible.

Take your " For ACT rules, Link has accessible name
<https://act-rules.github.io/rules/c487ae> applies
<https://act-rules.github.io/rules/c487ae#applicability> to any
HTML... etc" and
lay it next to the heading examples from John, there are so many ways to
fill; them up / in that you'll have a test / decision tree pretty
complicated to work through. Even more if you realize that Johns examples
are not all combinations possible and not technology agnostic. Add to this
mix all different ways they 'might' be implemented in a page as multiple
teams might work on the same feature and you'll end up with testing results
so complex... we need to make it easier step by step.

The same for task based testing. This is a wasps nest to set in stone for
every site I don't see is feasible for us to document this.

What I do see is:

- All should be based on pass /fail as we had
- ACT seems most appropriate to extend for testing (is hard to write the
tests!)
- we need a adjectival scoring for the pass / fails (3 preferred as more
will create confusion, disagree, and complexity we probably don't want)
- A scoring for criticality seems necessary but needs to be clearly scoped
or will get pushback because of the question "who's criticality". This
should probably be the task based scenario.
- Scoring tasks on the same level as pass / fail will probably not work and
suggest to place the tasks to the scope and within that scope do your pass/
fail (with severity and criticallity)
- Scoping of WCAG conformance should be opened up to add more than one
options like: web page, instance based, process or task completion,
component, feature ...

In short this comes down to:

- Create Scope (more possible like 5 screens, 2 tasks, 3 widgets)
- Do the pass / fail based on ACT rules
- Judge severity (Low, Medium, High impact)
- Severity should be added to ACT rules
- Add criticality based on scope
- Criticality might be guided by test work like Johns Heading pages
- Provide scoring based on total issues, severity of issues and criticality
for scope
- Scoring can be as easy as severity and criticality divided by total score
(low = 1, medium = 2, high = 3?!)

The result is not a perfect something, but we need to acknowledge our
strengths and weakness and this is a simple approach, doesn't take a lot
more time than we're used to, is for most part already known by WCAG
testers and gives so much more insight in WCAG testing compared to what we
have now.

Have all documented but here the spreadsheet for headings experiment:

https://docs.google.com/spreadsheets/d/1fAbtL_A6Xhbs5mKpuLuWPYu2QmKwIPdw6GoE6C5K3w0/edit?usp=sharing

Cheers,
Jake



Op ma 27 apr. 2020 om 21:56 schreef John Foliot <john.foliot@deque.com>:

> Hi Shawn,
>
> I think there is a fundamental issue here however that perhaps is being
> overlooked: not all websites are "task-based", and/or not all sites are
> *exclusively* task-based. And is not "playing a game" also a task?
>
> In your previous straw-man example, a decision had been taken that the
> "pizza game" isn't part of the task of ordering a pizza, so you then argue
> it's out of scope. But Accessibility Human Rights legislation doesn't work
> that way: if it's posted on-line for the general population, then it *MUST*
> be in scope for being accessible as well, and so while I can support the
> idea of task-based testing within Silver, I fall significantly short of
> allowing conformance scoping to have the ability to pick and choose what
> they think is critical for all users, and what content they think "disabled
> users" don't need (or want).
>
> And while I'll note that VPATS have a notion of "supporting with
> exceptions", and accept that we're going to see something similar with the
> Silver scoring methods, I personally will strenuously oppose selective
> scoping at the page or site level. There is a world of difference in saying
> "*We've got an 85% Accessibility score, INCLUDING our non-accessible game*"
> versus "*We got to 85% conformance by REMOVING the game from our test
> scope*".
>
> JF
>
> On Mon, Apr 27, 2020 at 2:25 PM Shawn Lauriat <lauriat@google.com> wrote:
>
>> ...now has to go to court and explain why they thought that game wasn't
>>> important for disabled people?
>>
>>
>> Exactly. And now the court and those involved have clear documentation
>> for how the pizza place considered accessibility and can then look at the
>> resulting impact to users. Everyone today with WCAG 2.x's conformance model
>> has that same ability to just declare a path or a page as not a part of
>> what needs to conform for a given conformance claim, so I don't think this
>> concept introduces any new ways for people to get things wrong there.
>>
>> By giving people the ability to define their own scope as groups of user
>> journeys, and users a way to identify gaps in that scope that affect them,
>> I think transparent task-based conformance can better support both sides of
>> that while also offering a structure for test results to have a better
>> chance of expressing the resulting experience for users trying to do things.
>>
>> -Shawn
>>
>> On Mon, Apr 27, 2020 at 2:27 PM John Foliot <john.foliot@deque.com>
>> wrote:
>>
>>> Hi Shawn,
>>>
>>> > Following that example of the pizza place site: they may have left a
>>> pizza game out of their scope of conformance, judging it not a part of
>>> their core offering of pizza. If someone has a problem with the pizza game
>>> and raises that as an issue preventing them from getting pizza, everyone
>>> then has the clear scope that left the game out, and whoever [was] involved
>>> in the decision...
>>>
>>> ...now has to go to court and explain why they thought that game wasn't
>>> important for disabled people?
>>>
>>> *Re: Scoping*
>>> The ability to cherry-pick what is and isn't out of scope is a
>>> dangerous precedent/concept, and will have (I fear) detrimental effects for
>>> persons with disabilities. Why wouldn't a "pizza game" be of interest to
>>> disabled users as well? Why shouldn't they also get to play along? Because
>>> making the pizza game accessible is too hard? - wrong answer...  (I recall
>>> our colleague and friend Victor Tsaran once saying to me - and I paraphrase
>>> - that today it's relatively easy to make sites 'accessible', but he could
>>> hardly wait for the day when they were also "fun" - this from back when he
>>> was still at Yahoo!, and they included an Easter Egg on the Yahoo! site:
>>> https://youtu.be/xXNkP2jU7Pg)
>>>
>>> Selective Accessibility MUST be avoided, not encouraged, and I fear your
>>> use-case is an example of why we shouldn't be leaving scoping to the
>>> content owners (and also demonstrates how easy it will be for uninformed
>>> content creators to miss the forest, because we've got them looking at -
>>> and selecting - specific trees...) Using the same logic, I could also argue
>>> that content in an <aside> isn't really critical to the main content (which
>>> MUST be accessible) - that's why it is an aside - and so any content in an
>>> <aside> is then out of scope? Slippery slope ahead.
>>>
>>> JF
>>>
>>> On Mon, Apr 27, 2020 at 12:36 PM Shawn Lauriat <lauriat@google.com>
>>> wrote:
>>>
>>>> Many good questions!
>>>>
>>>> I'm kinda liking aspects of this approach (ACT Rules format for testing
>>>>> flows), but (of course) I have a critical question: *how do we score
>>>>> something like this?*
>>>>
>>>>
>>>> Honestly, I'd like to think through that as a separate thing to figure
>>>> out from the topic of scoping and task definition, though still heavily
>>>> related. We could end up with any number of scoring systems using the same
>>>> scoping and task definition. Trying to figure them out at the same time
>>>> just introduces too many variables for me.
>>>>
>>>> As I described it to Jeanne recently, I have this kind of thought about
>>>> how we could define scope and tasks, I have a clear-ish sense of how we can
>>>> build up tests for methods, but still have only murky ideas on how we can
>>>> get the two to meet in the middle. We've certainly made some good progress
>>>> on that, but we still definitely have further to go.
>>>>
>>>> Open question: is this a correct interpretation? Does all critical path
>>>>> testing need to start from a common starting point?
>>>>
>>>>
>>>> A really good question, and one that honestly depends on the site or
>>>> app (etc.). For a pizza site, you can link directly to the contact page.
>>>> For an app like Google Docs, you can't really link directly to text
>>>> substitution preferences, so that'd need to come from a more common start
>>>> point. We should help walk people through how to define and include this in
>>>> scope, definitely, as the accessibility of a thing doesn't really matter if
>>>> you can't get access to it in the first place.
>>>>
>>>> Additionally, how do we ensure that all critical path testing is scoped
>>>>> by any given site? (the current scoping proposal leaves it to the
>>>>> site-owner to scope their conformance claims, so leaving out complex or
>>>>> critical flows that are non-conformant could be easily overcome by simply
>>>>> leaving those flows out of the testing scope).
>>>>
>>>>
>>>> I don't think we need to. If we have a clear definition of how to
>>>> scope, and a way for people to transparently declare that scope, we can
>>>> leave the "is this right?" part to those who need to decide it. Following
>>>> that example of the pizza place site: they may have left a pizza game out
>>>> of their scope of conformance, judging it not a part of their core offering
>>>> of pizza. If someone has a problem with the pizza game and raises that as
>>>> an issue preventing them from getting pizza, everyone then has the clear
>>>> scope that left the game out, and whoever involved in the decision as to
>>>> whether the scope should include the game can make it in an informed way.
>>>>
>>>> Thanks,
>>>>
>>>> Shawn
>>>>
>>>> On Mon, Apr 27, 2020 at 12:20 PM John Foliot <john.foliot@deque.com>
>>>> wrote:
>>>>
>>>>> Hi Shawn
>>>>>
>>>>> I'm kinda liking aspects of this approach (ACT Rules format for
>>>>> testing flows), but (of course) I have a critical question: *how do
>>>>> we score something like this*?
>>>>>
>>>>> Each site(1) is going to have "critical paths" but few sites will be
>>>>> sharing the same critical paths. Additionally, some paths or tasks (find
>>>>> hours of operation) are significantly easier to do then others (update my
>>>>> emergency contact information on my companies HR intranet), especially if
>>>>> it pre-supposes that *all* paths start at a site's "homepage" (and/or the
>>>>> outcome or solution to Success Criterion 2.4.5 Multiple Ways - i.e. a
>>>>> sitemap page or search results page).
>>>>>
>>>>> No matter which, it seems to me that testing a critical path needs to
>>>>> start *somewhere*, and for a scalable and repeatable testing regime, about
>>>>> the only thing all sites have in common is a 'homepage', which is something
>>>>> your example already suggests:
>>>>>
>>>>>
>>>>>    1. Load the pizza restaurant's site
>>>>>       1. Possible inputs: found via search engine, hit a bookmark
>>>>>       link, selected from browser's history, etc.
>>>>>       2. *Main page loads* with focus at the top of the screen
>>>>>
>>>>> Open question: is this a correct interpretation? Does all critical
>>>>> path testing need to start from a common starting point?
>>>>>
>>>>> Additionally, how do we ensure that *all *critical path testing is
>>>>> scoped by any given site? (the current scoping proposal leaves it to the
>>>>> site-owner to scope their conformance claims, so leaving out complex or
>>>>> critical flows that are non-conformant could be easily overcome by simply
>>>>> leaving those flows out of the testing scope).
>>>>>
>>>>> JF
>>>>>
>>>>>
>>>>> (1: site being an euphemism for 'online digital activity or presence'
>>>>> - as we need to take XR and other emergent tech into account as well)
>>>>>
>>>>> On Mon, Apr 27, 2020 at 10:28 AM Shawn Lauriat <lauriat@google.com>
>>>>> wrote:
>>>>>
>>>>>> From an email I sent to some ACT folks a little while ago, where I
>>>>>> had tried expressing my thoughts on how we could use the same kind of
>>>>>> structure that ACT has, but as a way of essentially expressing overall
>>>>>> scope as a set of user journeys for task based testing. Hoping this can
>>>>>> help for tomorrow's conversation to have an example written out:
>>>>>>
>>>>>> For ACT rules, Link has accessible name
>>>>>> <https://act-rules.github.io/rules/c487ae> applies
>>>>>> <https://act-rules.github.io/rules/c487ae#applicability> to any HTML
>>>>>> element with the semantic role
>>>>>> <https://act-rules.github.io/rules/c487ae#semantic-role> of link that
>>>>>> is included in the accessibility tree
>>>>>> <https://act-rules.github.io/rules/c487ae#included-in-the-accessibility-tree>
>>>>>> . Link in context is <https://act-rules.github.io/rules/5effbb>
>>>>>> descriptive <https://act-rules.github.io/rules/5effbb> essentially
>>>>>> applies to any element that passes Link has accessible name
>>>>>> <https://act-rules.github.io/rules/c487ae>. In other words:
>>>>>>
>>>>>>    1. For each thing exposed in the accessibility tree as a link
>>>>>>       1. Go through Link has accessible name
>>>>>>       <https://act-rules.github.io/rules/c487ae> steps
>>>>>>       2. For each link that fails, note result
>>>>>>       3. For each link that passes
>>>>>>          1. Go through Link in context is
>>>>>>          <https://act-rules.github.io/rules/5effbb>descriptive
>>>>>>          <https://act-rules.github.io/rules/5effbb> steps
>>>>>>          2. For each link that fails, note result
>>>>>>
>>>>>> For tasks, even if simply in Education & Outreach type documentation,
>>>>>> we could walk people through the process of defining tasks and the steps
>>>>>> within each task similar to how the ACT Rules Format
>>>>>> <https://www.w3.org/TR/act-rules-format/> describes composite rules
>>>>>> and the atomic rules within each composite.
>>>>>>
>>>>>> The scope of a pizza restaurant's site could then have the definition
>>>>>> of a collection of tasks, the level at which they could/would measure
>>>>>> overall conformance:
>>>>>>
>>>>>>    1. Choose what kind of pizza to order from the available options
>>>>>>    2. Find out the hours of operation
>>>>>>    3. Find out how to get to the restaurant to dine in
>>>>>>    4. Contact the restaurant to order delivery
>>>>>>
>>>>>> Each task could consist of atomic actions, typically defined by
>>>>>> design, development, and testing activities. For task 2. Find out the hours
>>>>>> of operation, that could look like:
>>>>>>
>>>>>>    1. Load the pizza restaurant's site
>>>>>>       1. Possible inputs: found via search engine, hit a bookmark
>>>>>>       link, selected from browser's history, etc.
>>>>>>       2. Main page loads with focus at the top of the screen
>>>>>>    2. Navigate to contact page (composite, describes one possible
>>>>>>    path)
>>>>>>       1. Move focus to site navigation menu
>>>>>>       2. Open navigation menu
>>>>>>       3. Move focus to "Contact us" link
>>>>>>       4. Activate link
>>>>>>    3. Navigate to text containing the hours of operation (composite)
>>>>>>       1. Find "Hours of operation" section
>>>>>>       2. Read contents of "Hours of operation" section
>>>>>>
>>>>>> Within the steps of each atomic task bit, we could then run through
>>>>>> the applicability checks for each ACT-type Rule. So Link has
>>>>>> accessible name <https://act-rules.github.io/rules/c487ae> would
>>>>>> apply to all links within the path, but not to a random link in the footer
>>>>>> that has a label that doesn't imply any relation to hours or contact
>>>>>> information.
>>>>>>
>>>>>> I have thoughts about how each of these could work and how we would
>>>>>> define applicability of rules and such based on the tasks, but I think it
>>>>>> would make sense to just start with this higher-level question of whether
>>>>>> we could (or should) have some kind of structured task definition similar
>>>>>> to ACT's current structured rule definition.
>>>>>>
>>>>>> -Shawn
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *John Foliot* | Principal Accessibility Strategist | W3C AC
>>>>> Representative
>>>>> Deque Systems - Accessibility for Good
>>>>> deque.com
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> *John Foliot* | Principal Accessibility Strategist | W3C AC
>>> Representative
>>> Deque Systems - Accessibility for Good
>>> deque.com
>>>
>>>
>>>
>
> --
> *​John Foliot* | Principal Accessibility Strategist | W3C AC
> Representative
> Deque Systems - Accessibility for Good
> deque.com
>
>
>