Re: Task testing structure

Shawn wrote:

John wrote:

> If we really are moving away from page-level conformance to 'site-level'
> conformance "score" (no longer just pass/fail), then I assert our spec
> needs to be very clear with any exclusion criteria, as well as how the
> conformance is scoped - in the conformance statement itself.


I've proposed a model that wouldn't allow exclusion, as it wouldn't have a
concept of a whole "site-level" to exclude from, just as WCAG today defines
conformance claims. Would that model work?


Hi Shawn,

While it might "work", I've always proceeded under the assumption that
scores had to apply to *something*, and whether that's a page, a screen, an
app, a VR instance, or a 'site' is, I guess, something else we need to be
clear about. I am also reminded of the following:

 "...Silver needs to expand the scope beyond web to include apps,
documents, authoring, user agents, wearables, kiosks, IoT, VR, etc. and be
inclusive of more disabilities."
(UX Professionals Use of WCAG: Analysis) -
https://www.w3.org/WAI/GL/task-forces/silver/wiki/Problem_Statements#themes_from_Research_2
)


From my perspective at Deque, it has been my experience that overall most
site owners want a "site-level" view/report/assessment, and whenever I am
verbally queried, it is *ALWAYS* at the site level ("How are we doing
overall?"). Ditto for SPA's and both native and mobile-web content; clients
come to us looking for that high-level 'review', both from a usability and
legal-liability perspective.

As I review some of the earlier comments in this thread, even you are
suggesting a 'site-wide' concept at some level:

   - " The scope of a pizza restaurant's *site *could then have the
   definition of a collection of tasks, the level at which they
could/would measure
   overall conformance:"
   - " A really good question, and one that honestly depends on the *site *or
   app (etc.). "


If "a score" is what we're seeking to create (as opposed to the binary
pass/fail we have today), then I liken it to baseball: *individual players*
(your flows/processes) may have 'stats' that can be measured and reported
on (RBI's - runs batted in - for example), but for the fans, the most
important 'score' is the one at the end of each game: did their *team* (the
site) win or lose? By how much?

JF

On Fri, May 1, 2020 at 12:15 PM Shawn Lauriat <lauriat@google.com> wrote:

> John wrote:
>
>> If we really are moving away from page-level conformance to 'site-level'
>> conformance "score" (no longer just pass/fail), then I assert our spec
>> needs to be very clear with any exclusion criteria, as well as how the
>> conformance is scoped - in the conformance statement itself.
>
>
> I've proposed a model that wouldn't allow exclusion, as it wouldn't have a
> concept of a whole "site-level" to exclude from, just as WCAG today defines
> conformance claims. Would that model work?
>
> Not every site is 'task' or 'interaction' driven - they may also be 100%
>> 'push' content (Joe's Blog, with commenting turned off)
>
>
> The task: read Joe's Blog posts or a specific post. Maybe find out how to
> get in touch with Joe via email. Tasks don't necessarily require
> interactive controls.
>
> Katie wrote:
>
>> Please if you are aware of any government regulations that are requiring
>> WCAG Conformance Claims, I'd love to hear about them.
>
>
> I only know of one off the top of my head, The State of California
> requires "compliance" with WCAG
> <https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180AB434> (at
> about the level of readable as John notes for conformance claims, so I've
> bolded the most applicable bit):
>
> 11546.7. (a) Before July 1, 2019, and before July 1 biennially thereafter,
> the director of each state agency or state entity, as defined in
> subdivision (e) of Section 11546.1, and each chief information officer
> appointed under Section 11546.1, shall post on the home page of the state
> agency’s or state entity’s Internet Web site a signed certification from
> the state agency’s or state entity’s director and chief information officer
> that they have determined that *the Internet Web site is in compliance
> with Sections 7405 and 11135, and the Web Content Accessibility Guidelines
> 2.0, or a subsequent version, published by the Web Accessibility Initiative
> of the World Wide Web Consortium at a minimum Level AA success criteria.*
> (b) The Director of Technology shall create a standard form that each
> state agency’s or state entity’s chief information officer shall use to
> determine whether the state agency’s or state entity’s Internet Web site is
> in compliance with the accessibility standards specified in subdivision (a).
>
>
> -Shawn
>
> On Tue, Apr 28, 2020 at 8:12 PM Katie Haritos-Shea <ryladog@gmail.com>
> wrote:
>
>> I'll just note that there are no regulators that I know of who are using
>> or requiring WCAG Conformance Claims.
>>
>> Please if you are aware of any government regulations that are requiring
>> WCAG Conformance Claims, I'd love to hear about them.
>>
>> On Tue, Apr 28, 2020, 6:12 PM John Foliot <john.foliot@deque.com> wrote:
>>
>>> Shawn writes:
>>>
>>> Potential adapting to:
>>>
>>> Conformance Claims (Optional)
>>> Conformance is defined only for tasks. However, a conformance claim may
>>> be made to cover one task, a series of tasks, or multiple related tasks.
>>>
>>>
>>> I think (if I better understand your concern, John) this should address
>>> the case of someone having their conformance claim imply that it covers all
>>> of example.com or MyAwesomeApp, while only defining and testing a
>>> subset of tasks wiithin that conformance claim?
>>>
>>> Thoughts?
>>>
>>> Well, putting aside the fact that I understand you aren't 'happy' with
>>> the term Task here either... (perhaps "interaction"?) my thoughts are still:
>>>
>>>    - Not every site is 'task' or 'interaction' driven - they may also
>>>    be 100% 'push' content (Joe's Blog, with commenting turned off)
>>>
>>>    - The language still feels kind of slippery - it still lets the
>>>    owner decide what is more critical or important for all end users, but also
>>>    empowers them to exclude some content from some users without even making
>>>    an effort. It's akin to me establishing my own speed limit on the highway
>>>    (which I understand is actually a thing on the Autobans of Germany
>>>    <https://en.wikipedia.org/wiki/Autobahn>). This will benefit
>>>    site-owners at the cost to site users I fear.
>>>
>>>    - If we really are moving away from page-level conformance to
>>>    'site-level' conformance "score" (no longer just pass/fail), then I assert
>>>    our spec needs to be very clear with any exclusion criteria, as well as how
>>>    the conformance is scoped - in the conformance statement itself.
>>>
>>>    - Any "exemption rules" need a real grounding in logic and
>>>    feasibility - it cannot just be because the site owner doesn't think it's
>>>    in scope. (I mean, they can think anything they want, but then there is the
>>>    law...) As I noted on today's call, we're going to need regulators to pick
>>>    up our specification if we *really* want it to be effective (beyond a
>>>    collection of Best Practices), and so we need to ensure we are as
>>>    unambiguous as possible in this regard.
>>>
>>> However Shawn, yes, this is my concern: "*...having their conformance
>>> claim imply that it covers all of example.com <http://example.com/> or
>>> MyAwesomeApp, while only defining and testing a subset of tasks within that
>>> conformance claim?*"
>>>
>>> Part of the problem is that 'conformance statements' are about as dry to
>>> read as click-through EULA's, and most site visitors won't know what is or
>>> isn't in or out of scope. And I cannot think of a more frustrating scenario
>>> for a PwD than to be told that the section of the site *they* want to
>>> access - Chuck's Pizza Game(TM) - is out of scope for the accessibility
>>> conformance thing, so "too bad".
>>>
>>> Additionally, as we transition away from black or white (aka Pass/Fail)
>>> to a more realistic long-tail score (the million shades of gray between
>>> black and white), the scoring *should* allow for partially but not fully
>>> conformant content to exist on the site - but at a cost to the overall
>>> score (i.e. "you're losing points there bud..."). Allowing site owners to
>>> self-exempt will drive up scores, but do little for end users.
>>>
>>> JF
>>>
>>> On Tue, Apr 28, 2020 at 4:32 PM Shawn Lauriat <lauriat@google.com>
>>> wrote:
>>>
>>>> Should the standard be defining coverage? Or would it be better to make
>>>>> the same sort of assumptions as 2.x and leave it to regulators to say
>>>>> whether a site should cover all tasks, or what the sampling is?
>>>>
>>>>
>>>> Exactly, we could instead define conformance scope as purely per-task
>>>> and discard the notion of a collection of tasks as a definition of a
>>>> complete thing. That'd follow the WCAG 2.x per-page model, while allowing
>>>> people to optionally put several together to declare a larger scope, just
>>>> like WCAG 2.x does today. It wouldn't have any aspect of conformance itself
>>>> implying declaration of a full scope of an "entire" thing, but merely a
>>>> collection of small things. To write it out for comparison:
>>>>
>>>> WCAG: 5.3 Conformance Claims (Optional)
>>>> <https://www.w3.org/TR/WCAG21/#conformance-claims>
>>>> Conformance is defined only for Web pages. However, a conformance claim
>>>> may be made to cover one page, a series of pages, or multiple related Web
>>>> pages.
>>>>
>>>>
>>>> Potential adapting to:
>>>>
>>>> Conformance Claims (Optional)
>>>> Conformance is defined only for tasks. However, a conformance claim may
>>>> be made to cover one task, a series of tasks, or multiple related tasks.
>>>>
>>>>
>>>> I think (if I better understand your concern, John) this should address
>>>> the case of someone having their conformance claim imply that it covers all
>>>> of example.com or MyAwesomeApp, while only defining and testing a
>>>> subset of tasks wiithin that conformance claim?
>>>>
>>>> Thoughts?
>>>>
>>>> (Chuck wrote, as I had this drafted:)
>>>>
>>>> Shawn mentioned something in the W3C call that didn't get a lot of
>>>>> attention in call, was some system whereby the author defines the scope,
>>>>> and that scoping exercise (or process or determination) is open to the
>>>>> public (or I forget the words...) fully disclosable?  Discoverable?  He
>>>>> used a word or phrase...
>>>>
>>>>
>>>> "Transparent", which came to mind from the Silver requirement for
>>>> providing broad support to the regulatory environment
>>>> <https://w3c.github.io/silver/requirements/index.html#regulatory-environment>.
>>>> Not necessarily public, but if someone says their thing conforms and then
>>>> needs to show their conformance claim,
>>>>
>>>> -Shawn
>>>>
>>>> P.S. Separately from scope itself, we'll definitely need to work out
>>>> non-interference, what does "task" mean and can we have a better word, etc.
>>>> The agenda item we didn't get to in today's call.
>>>>
>>>> On Tue, Apr 28, 2020 at 4:11 PM Charles Adams <charles.adams@oracle.com>
>>>> wrote:
>>>>
>>>>> Shawn mentioned something in the W3C call that didn't get a lot of
>>>>> attention in call, was some system whereby the author defines the scope,
>>>>> and that scoping exercise (or process or determination) is open to the
>>>>> public (or I forget the words...) fully disclosable?  Discoverable?  He
>>>>> used a word or phrase...
>>>>>
>>>>> I'm going to extend the use case of "Corner Pizza", and the game
>>>>> that's been mentioned.  I'm going to extend it to two games.
>>>>>
>>>>> Game One:  Guess which celebrity most recently visited our shop and
>>>>> get $1 off your pizza order.
>>>>>
>>>>> Game Two:  Web based Pizza Party!  Cook needs to create a series of
>>>>> pizzas correctly as orders speed up.  This web based game exists only to
>>>>> generate traffic, and is not necessary to order a pizza or to utilize any
>>>>> other "Corner Pizza" service.
>>>>>
>>>>> In the case of Game One I think it should be scoped in, as there's a
>>>>> purpose with consumer impact for playing and "winning" the game.
>>>>>
>>>>> If the business case for Game Two was just to attract web traffic to
>>>>> the site, I could see a case for the author wanting to scope that out.  The
>>>>> second game isn't a required step for ordering a pizza or any other
>>>>> service.  The author just wants to attract traffic in the hopes that the
>>>>> players decide to use the accessible ordering app to order something.
>>>>>
>>>>> In the extended use case of "Corner Pizza", I would think that Game
>>>>> One's accessibility SHOULD impact the final score, and I would think that
>>>>> Game Two's accessibility could be argued to be irrelevant to the final
>>>>> score.  I also think that if there was some public facing information which
>>>>> documented that Game Two was scoped out of the score, reviewers would be
>>>>> able to determine that this "scope exclusion" was not intended to abuse any
>>>>> conformance model.  IF the content author also tried to scope out Game One,
>>>>> that could be perceived as an attempt to abuse the conformance model.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Chuck
>>>>>
>>>>>
>>>>> On 4/28/2020 7:42 AM, Alastair Campbell wrote:
>>>>>
>>>>> > I also was under the presumption that we were moving from a
>>>>> "per-page" conformance model to a "site-wide" conformance "score".
>>>>>
>>>>>
>>>>>
>>>>> I’m not sure about that, but I thought there was a move from page to
>>>>> task.
>>>>>
>>>>>
>>>>>
>>>>> Given a move from page to task:
>>>>>
>>>>> Should the standard be defining coverage? Or would it be better to
>>>>> make the same sort of assumptions as 2.x and leave it to regulators to say
>>>>> whether a site should cover all tasks, or what the sampling is?
>>>>>
>>>>> -Alastair
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> *​John Foliot* | Principal Accessibility Strategist | W3C AC
>>> Representative
>>> Deque Systems - Accessibility for Good
>>> deque.com
>>>
>>>
>>>

-- 
*​John Foliot* | Principal Accessibility Strategist | W3C AC Representative
Deque Systems - Accessibility for Good
deque.com