Re: Task testing structure

John,

While it might "work", I've always proceeded under the assumption that
> scores had to apply to *something*, and whether that's a page, a screen, an
> app, a VR instance, or a 'site' is, I guess, something else we need to be
> clear about.


Very much agreed, and we'll need to figure out the building blocks that
make up tasks for each platform/technology that we provide guidance for,
including the necessary scope of building blocks as they pertain to tasks
(in terms of direct impact, non-interference, etc.).

From my perspective at Deque, it has been my experience that overall most
> site owners want a "site-level" view/report/assessment, and whenever I am
> verbally queried, it is *ALWAYS* at the site level ("How are we doing
> overall?"). Ditto for SPA's and both native and mobile-web content; clients
> come to us looking for that high-level 'review', both from a usability and
> legal-liability perspective.


That matches my experience, regardless of website, app, hardware, VR, etc.,
so trust me, I have every motivation to support that! While I've suggested
that we don't need to require that for a conformance claim, you could still
define a site as a collection of tasks that covers everything that the
site contains. You could also make a claim for part of a site or app, but
that conformance claim would not make a claim for the entire site or app -
just that part that you said it covers. Again: this mirrors what we have in
WCAG today, where it defines conformance at the page level, and even the
description of how you can document an optional conformance claim doesn't
include a requirement that the collection of pages adds up to a full site,
but you still can cover the entire site if you so choose. 99% of the time,
you do.

If "a score" is what we're seeking to create (as opposed to the binary
> pass/fail we have today), then I liken it to baseball: individual players
> (your flows/processes) may have 'stats' that can be measured and reported
> on (RBI's - runs batted in - for example), but for the fans, the most
> important 'score' is the one at the end of each game: did their team (the
> site) win or lose? By how much?


Very much agreed, or as much as my understanding of baseball stats allows!
In particular, you've hit on the two different aspects of conformance that
I wanted to work through and define not just how they each work, but how
the lower granularity builds up into a full story of the higher granularity:

   - Individual player stats
   Parallel: this widget over here really represents the best
   implementation meeting this guideline that I've ever seen!
   - Did the team win or lose and by how much?
   Parallel: even with that one great implementation, the rest of the
   site/app/thing still left these users unable to do what they came to do,
   and boy did it fail badly overall…here's how.

To reach that, I thought through how to measure task scoring by building up
the interactions making up that task, where each of those interactions
would need the scrutiny at the lower level of how each piece and its
surrounding environment meets the building-block-level tests. My email
kicking off this thread describes, at least roughly, how I would think
through that for one particular case (Find out the hours of operation) and
one particular test (Link has accessible name
<https://act-rules.github.io/rules/c487ae>).

-Shawn

On Fri, May 1, 2020 at 3:28 PM John Foliot <john.foliot@deque.com> wrote:

> Shawn wrote:
>
> John wrote:
>
>> If we really are moving away from page-level conformance to 'site-level'
>> conformance "score" (no longer just pass/fail), then I assert our spec
>> needs to be very clear with any exclusion criteria, as well as how the
>> conformance is scoped - in the conformance statement itself.
>
>
> I've proposed a model that wouldn't allow exclusion, as it wouldn't have a
> concept of a whole "site-level" to exclude from, just as WCAG today defines
> conformance claims. Would that model work?
>
>
> Hi Shawn,
>
> While it might "work", I've always proceeded under the assumption that
> scores had to apply to *something*, and whether that's a page, a screen, an
> app, a VR instance, or a 'site' is, I guess, something else we need to be
> clear about. I am also reminded of the following:
>
>  "...Silver needs to expand the scope beyond web to include apps,
> documents, authoring, user agents, wearables, kiosks, IoT, VR, etc. and be
> inclusive of more disabilities."
> (UX Professionals Use of WCAG: Analysis) -
> https://www.w3.org/WAI/GL/task-forces/silver/wiki/Problem_Statements#themes_from_Research_2
> )
>
>
> From my perspective at Deque, it has been my experience that overall most
> site owners want a "site-level" view/report/assessment, and whenever I am
> verbally queried, it is *ALWAYS* at the site level ("How are we doing
> overall?"). Ditto for SPA's and both native and mobile-web content; clients
> come to us looking for that high-level 'review', both from a usability and
> legal-liability perspective.
>
> As I review some of the earlier comments in this thread, even you are
> suggesting a 'site-wide' concept at some level:
>
>    - " The scope of a pizza restaurant's *site *could then have the
>    definition of a collection of tasks, the level at which they could/would measure
>    overall conformance:"
>    - " A really good question, and one that honestly depends on the *site
>    *or app (etc.). "
>
>
> If "a score" is what we're seeking to create (as opposed to the binary
> pass/fail we have today), then I liken it to baseball: *individual
> players* (your flows/processes) may have 'stats' that can be measured and
> reported on (RBI's - runs batted in - for example), but for the fans, the
> most important 'score' is the one at the end of each game: did their
> *team* (the site) win or lose? By how much?
>
> JF
>
> On Fri, May 1, 2020 at 12:15 PM Shawn Lauriat <lauriat@google.com> wrote:
>
>> John wrote:
>>
>>> If we really are moving away from page-level conformance to 'site-level'
>>> conformance "score" (no longer just pass/fail), then I assert our spec
>>> needs to be very clear with any exclusion criteria, as well as how the
>>> conformance is scoped - in the conformance statement itself.
>>
>>
>> I've proposed a model that wouldn't allow exclusion, as it wouldn't have
>> a concept of a whole "site-level" to exclude from, just as WCAG today
>> defines conformance claims. Would that model work?
>>
>> Not every site is 'task' or 'interaction' driven - they may also be 100%
>>> 'push' content (Joe's Blog, with commenting turned off)
>>
>>
>> The task: read Joe's Blog posts or a specific post. Maybe find out how to
>> get in touch with Joe via email. Tasks don't necessarily require
>> interactive controls.
>>
>> Katie wrote:
>>
>>> Please if you are aware of any government regulations that are requiring
>>> WCAG Conformance Claims, I'd love to hear about them.
>>
>>
>> I only know of one off the top of my head, The State of California
>> requires "compliance" with WCAG
>> <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180AB434> (at
>> about the level of readable as John notes for conformance claims, so I've
>> bolded the most applicable bit):
>>
>> 11546.7. (a) Before July 1, 2019, and before July 1 biennially
>> thereafter, the director of each state agency or state entity, as defined
>> in subdivision (e) of Section 11546.1, and each chief information officer
>> appointed under Section 11546.1, shall post on the home page of the state
>> agency’s or state entity’s Internet Web site a signed certification from
>> the state agency’s or state entity’s director and chief information officer
>> that they have determined that *the Internet Web site is in compliance
>> with Sections 7405 and 11135, and the Web Content Accessibility Guidelines
>> 2.0, or a subsequent version, published by the Web Accessibility Initiative
>> of the World Wide Web Consortium at a minimum Level AA success criteria.*
>> (b) The Director of Technology shall create a standard form that each
>> state agency’s or state entity’s chief information officer shall use to
>> determine whether the state agency’s or state entity’s Internet Web site is
>> in compliance with the accessibility standards specified in subdivision (a).
>>
>>
>> -Shawn
>>
>> On Tue, Apr 28, 2020 at 8:12 PM Katie Haritos-Shea <ryladog@gmail.com>
>> wrote:
>>
>>> I'll just note that there are no regulators that I know of who are using
>>> or requiring WCAG Conformance Claims.
>>>
>>> Please if you are aware of any government regulations that are requiring
>>> WCAG Conformance Claims, I'd love to hear about them.
>>>
>>> On Tue, Apr 28, 2020, 6:12 PM John Foliot <john.foliot@deque.com> wrote:
>>>
>>>> Shawn writes:
>>>>
>>>> Potential adapting to:
>>>>
>>>> Conformance Claims (Optional)
>>>> Conformance is defined only for tasks. However, a conformance claim may
>>>> be made to cover one task, a series of tasks, or multiple related tasks.
>>>>
>>>>
>>>> I think (if I better understand your concern, John) this should address
>>>> the case of someone having their conformance claim imply that it covers all
>>>> of example.com or MyAwesomeApp, while only defining and testing a
>>>> subset of tasks wiithin that conformance claim?
>>>>
>>>> Thoughts?
>>>>
>>>> Well, putting aside the fact that I understand you aren't 'happy' with
>>>> the term Task here either... (perhaps "interaction"?) my thoughts are still:
>>>>
>>>>    - Not every site is 'task' or 'interaction' driven - they may also
>>>>    be 100% 'push' content (Joe's Blog, with commenting turned off)
>>>>
>>>>    - The language still feels kind of slippery - it still lets the
>>>>    owner decide what is more critical or important for all end users, but also
>>>>    empowers them to exclude some content from some users without even making
>>>>    an effort. It's akin to me establishing my own speed limit on the highway
>>>>    (which I understand is actually a thing on the Autobans of Germany
>>>>    <https://en.wikipedia.org/wiki/Autobahn>). This will benefit
>>>>    site-owners at the cost to site users I fear.
>>>>
>>>>    - If we really are moving away from page-level conformance to
>>>>    'site-level' conformance "score" (no longer just pass/fail), then I assert
>>>>    our spec needs to be very clear with any exclusion criteria, as well as how
>>>>    the conformance is scoped - in the conformance statement itself.
>>>>
>>>>    - Any "exemption rules" need a real grounding in logic and
>>>>    feasibility - it cannot just be because the site owner doesn't think it's
>>>>    in scope. (I mean, they can think anything they want, but then there is the
>>>>    law...) As I noted on today's call, we're going to need regulators to pick
>>>>    up our specification if we *really* want it to be effective (beyond a
>>>>    collection of Best Practices), and so we need to ensure we are as
>>>>    unambiguous as possible in this regard.
>>>>
>>>> However Shawn, yes, this is my concern: "*...having their conformance
>>>> claim imply that it covers all of example.com <http://example.com/> or
>>>> MyAwesomeApp, while only defining and testing a subset of tasks within that
>>>> conformance claim?*"
>>>>
>>>> Part of the problem is that 'conformance statements' are about as dry
>>>> to read as click-through EULA's, and most site visitors won't know what is
>>>> or isn't in or out of scope. And I cannot think of a more frustrating
>>>> scenario for a PwD than to be told that the section of the site *they* want
>>>> to access - Chuck's Pizza Game(TM) - is out of scope for the
>>>> accessibility conformance thing, so "too bad".
>>>>
>>>> Additionally, as we transition away from black or white (aka Pass/Fail)
>>>> to a more realistic long-tail score (the million shades of gray between
>>>> black and white), the scoring *should* allow for partially but not fully
>>>> conformant content to exist on the site - but at a cost to the overall
>>>> score (i.e. "you're losing points there bud..."). Allowing site owners to
>>>> self-exempt will drive up scores, but do little for end users.
>>>>
>>>> JF
>>>>
>>>> On Tue, Apr 28, 2020 at 4:32 PM Shawn Lauriat <lauriat@google.com>
>>>> wrote:
>>>>
>>>>> Should the standard be defining coverage? Or would it be better to
>>>>>> make the same sort of assumptions as 2.x and leave it to regulators to say
>>>>>> whether a site should cover all tasks, or what the sampling is?
>>>>>
>>>>>
>>>>> Exactly, we could instead define conformance scope as purely per-task
>>>>> and discard the notion of a collection of tasks as a definition of a
>>>>> complete thing. That'd follow the WCAG 2.x per-page model, while allowing
>>>>> people to optionally put several together to declare a larger scope, just
>>>>> like WCAG 2.x does today. It wouldn't have any aspect of conformance itself
>>>>> implying declaration of a full scope of an "entire" thing, but merely a
>>>>> collection of small things. To write it out for comparison:
>>>>>
>>>>> WCAG: 5.3 Conformance Claims (Optional)
>>>>> <https://www.w3.org/TR/WCAG21/#conformance-claims>
>>>>> Conformance is defined only for Web pages. However, a conformance
>>>>> claim may be made to cover one page, a series of pages, or multiple related
>>>>> Web pages.
>>>>>
>>>>>
>>>>> Potential adapting to:
>>>>>
>>>>> Conformance Claims (Optional)
>>>>> Conformance is defined only for tasks. However, a conformance claim
>>>>> may be made to cover one task, a series of tasks, or multiple related tasks.
>>>>>
>>>>>
>>>>> I think (if I better understand your concern, John) this should
>>>>> address the case of someone having their conformance claim imply that it
>>>>> covers all of example.com or MyAwesomeApp, while only defining and
>>>>> testing a subset of tasks wiithin that conformance claim?
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> (Chuck wrote, as I had this drafted:)
>>>>>
>>>>> Shawn mentioned something in the W3C call that didn't get a lot of
>>>>>> attention in call, was some system whereby the author defines the scope,
>>>>>> and that scoping exercise (or process or determination) is open to the
>>>>>> public (or I forget the words...) fully disclosable?  Discoverable?  He
>>>>>> used a word or phrase...
>>>>>
>>>>>
>>>>> "Transparent", which came to mind from the Silver requirement for
>>>>> providing broad support to the regulatory environment
>>>>> <https://w3c.github.io/silver/requirements/index.html#regulatory-environment>.
>>>>> Not necessarily public, but if someone says their thing conforms and then
>>>>> needs to show their conformance claim,
>>>>>
>>>>> -Shawn
>>>>>
>>>>> P.S. Separately from scope itself, we'll definitely need to work out
>>>>> non-interference, what does "task" mean and can we have a better word, etc.
>>>>> The agenda item we didn't get to in today's call.
>>>>>
>>>>> On Tue, Apr 28, 2020 at 4:11 PM Charles Adams <
>>>>> charles.adams@oracle.com> wrote:
>>>>>
>>>>>> Shawn mentioned something in the W3C call that didn't get a lot of
>>>>>> attention in call, was some system whereby the author defines the scope,
>>>>>> and that scoping exercise (or process or determination) is open to the
>>>>>> public (or I forget the words...) fully disclosable?  Discoverable?  He
>>>>>> used a word or phrase...
>>>>>>
>>>>>> I'm going to extend the use case of "Corner Pizza", and the game
>>>>>> that's been mentioned.  I'm going to extend it to two games.
>>>>>>
>>>>>> Game One:  Guess which celebrity most recently visited our shop and
>>>>>> get $1 off your pizza order.
>>>>>>
>>>>>> Game Two:  Web based Pizza Party!  Cook needs to create a series of
>>>>>> pizzas correctly as orders speed up.  This web based game exists only to
>>>>>> generate traffic, and is not necessary to order a pizza or to utilize any
>>>>>> other "Corner Pizza" service.
>>>>>>
>>>>>> In the case of Game One I think it should be scoped in, as there's a
>>>>>> purpose with consumer impact for playing and "winning" the game.
>>>>>>
>>>>>> If the business case for Game Two was just to attract web traffic to
>>>>>> the site, I could see a case for the author wanting to scope that out.  The
>>>>>> second game isn't a required step for ordering a pizza or any other
>>>>>> service.  The author just wants to attract traffic in the hopes that the
>>>>>> players decide to use the accessible ordering app to order something.
>>>>>>
>>>>>> In the extended use case of "Corner Pizza", I would think that Game
>>>>>> One's accessibility SHOULD impact the final score, and I would think that
>>>>>> Game Two's accessibility could be argued to be irrelevant to the final
>>>>>> score.  I also think that if there was some public facing information which
>>>>>> documented that Game Two was scoped out of the score, reviewers would be
>>>>>> able to determine that this "scope exclusion" was not intended to abuse any
>>>>>> conformance model.  IF the content author also tried to scope out Game One,
>>>>>> that could be perceived as an attempt to abuse the conformance model.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Chuck
>>>>>>
>>>>>>
>>>>>> On 4/28/2020 7:42 AM, Alastair Campbell wrote:
>>>>>>
>>>>>> > I also was under the presumption that we were moving from a
>>>>>> "per-page" conformance model to a "site-wide" conformance "score".
>>>>>>
>>>>>>
>>>>>>
>>>>>> I’m not sure about that, but I thought there was a move from page to
>>>>>> task.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Given a move from page to task:
>>>>>>
>>>>>> Should the standard be defining coverage? Or would it be better to
>>>>>> make the same sort of assumptions as 2.x and leave it to regulators to say
>>>>>> whether a site should cover all tasks, or what the sampling is?
>>>>>>
>>>>>> -Alastair
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> *​John Foliot* | Principal Accessibility Strategist | W3C AC
>>>> Representative
>>>> Deque Systems - Accessibility for Good
>>>> deque.com
>>>>
>>>>
>>>>
>
> --
> *​John Foliot* | Principal Accessibility Strategist | W3C AC
> Representative
> Deque Systems - Accessibility for Good
> deque.com
>
>
>