Conformance Challenges Update Survey (Closes Tuesday May 26th at midnight EST)

Hello,


Please review the Conformance Challenges Survey
<https://www.w3.org/2002/09/wbs/35422/conformance-challenges-05-21-20/> before
Tuesday's meeting if possible. The survey will close on Tuesday at midnight
Boston time.


We realize this is a quick turnaround request. We have received feedback
from the public and W3C (outside AG) that the status, intent and some
content of the conformance challenges document published on 1 May are not
clear and may be causing some confusion. We would like to clarify the
document as quickly as possible with the working group's consent.
Unfortunately this means a quick review turnaround.

We have focused the proposed changes  to:


   - clarify the draft/work-in-progress status of the document;
   - clarify the scope of this document within the broader context of
   evaluation conformance;
   - clarify the forward-facing focus; and
   - remove absolutist characterizations of the current conformance model.

We ask you to review the changes and answer the survey question. To make
this as easy as possible, there are three ways to do this review:


   - Review  the list of changes at the bottom of this email.
   - Review the proposed changes (scroll to the bottom)
   <https://github.com/w3c/wcag/compare/conformance-challenges-update-ac?expand=1>
   and/or
   - Read the revised draft
   <http://raw.githack.com/w3c/wcag/conformance-challenges-update-ac/conformance-challenges/index.html>


This is our first agenda item for Tuesday's meeting.


Kind Regards,


Rachael

***

*List of Changes with Text*

Abstract

*1. Added an Editor's Note*

Publication as a First Public Working Draft does not imply endorsement by
the W3C Membership. This is a draft document and may be updated, replaced
or obsoleted by other documents at any time. It is inappropriate to cite
this document as other than work in progress.

The intent of this First Public Working Draft by the Accessibility
Guidelines Working Group is to explore how to improve the testability and
page-based conformance verification of the WCAG accessibility guidelines in
order to incorporate improvements into a future version of the guidelines.
This draft is published to obtain public review of the issues identified
and solutions proposed.

*2. Adjusted Intro Paragraph*

*Original:* This document explores how testability and page-based
conformance verification of the WCAG 2.0 and 2.1 accessibility guidelines
are challenging to apply to a broad range of websites and web
applications.  It also explores approaches for mitigating these challenges,
to realize as accessible a site as possible.

*Revised: *This document explores how the page-based conformance
verification of the WCAG 2.0 and 2.1 accessibility guidelines are
challenging to apply to certain websites and web applications. This
document also explores ideas on how future versions of guidelines might
address these challenges. This document focuses primarily on challenges to
large, highly complex, dynamic sites.  Other efforts in WAI are looking at
different aspects of conformance for different types of sites.

*3. Revised Wording*

*Original:* Large and dynamic sites may have too many changing permutations
to validate effectively;

*Revised:* Large and dynamic sites with their changing permutations may be
difficult to validate;

*4. Removed Text*

The purpose of this document is to help understand those challenges more
holistically, and explore approaches to mitigating those challenges, both
so that sites can use these mitigation approaches now, and also so that we
can address the  challenges more fully in future accessibility guidelines
such as WCAG 3.0...

Status of Document

*5. Revised Text*

*Original: *This is a First Public Working Draft by the Accessibility
Guidelines Working Group. This document explores how testability and
page-based conformance verification of the WCAG accessibility guidelines
are challenging to apply to a broad range of websites and web applications.
It also explores approaches for mitigating these challenges, to realize as
accessible a site as possible. This draft is published to obtain public
review of the issues identified and solutions proposed.

*Revised: *This is a Working Draft by the Accessibility Guidelines Working
Group. This document explores how the page-based conformance verification
of the WCAG 2.0 and 2.1 accessibility guidelines are challenging to apply
to certain websites and web applications. It also explores approaches for
mitigating these challenges, which could be incorporated into a future
version of the guidelines. This draft is published to obtain public review
of the issues identified and solutions proposed.

*6. Revised Text (handled via an editor's note until publication)*

*Original: *Publication as a First Public Working Draft does not imply
endorsement by the W3C Membership. This is a draft document and may be
updated, replaced or obsoleted by other documents at any time. It is
inappropriate to cite this document as other than work in progress.

*Revised: *Publication as a Working Draft does not represent a consensus of
the Working Group and does not imply endorsement by the W3C Membership.
This is a draft document and may be updated, replaced or obsoleted by other
documents at any time. It is inappropriate to cite this document as other
than work in progress.

Introduction: Problem Statement

*7. Moved a paragraph up*

While the challenges discussed in this document apply to websites and web
applications broadly, this early version of the document focuses
particularly on situations involving large, dynamic, and complex websites.
There are valid reasons WCAG 2 and related resources have the conformance
model they do; failures to conform to the current model are likely to
present a barrier for people with disabilities. The issues raised in this
document do not mean sites should not strive to conform to WCAG 2. It is
also vital to consider aspects beyond the conformance model that lead to
accessibility issues found during late stage testing, such as the lack of
accessibility prioritization, training, and integration throughout the
design, development, and maintenance processes. A new version of
accessibility guidelines, W3C Accessibility Guidelines 3.0 (WCAG 3.0),
rethinks all aspects of the accessibility guidance and is expressly
chartered to develop a new conformance model that should help to address
these challenges.

*8. Revised Wording*

*Original:* Ensuring that every one of these page updates fully satisfies
all success criteria (as appropriate), especially where expert human review
is required for some criteria, presents a massive scaling problem. Further,
where pages are generated programmatically, finding every last bug related
to that generation may prove challenging, especially when they only arise
from uncommon content scenarios or combinations (and updates to those
algorithms and code happen multiple times per week). Thus, the likelihood
that every last page (out of what might be millions or billions of pages)
can satisfy each and every success criterion 100% of the time is extremely
low.

*Revised:* Ensuring that every one of these page updates fully satisfies
all success criteria (as appropriate), especially where expert human review
is required for some criteria presents a problem for scaling conformance
assessments. Further, where pages are generated programmatically, finding
every bug related to that generation may prove challenging, especially when
they only arise from uncommon content scenarios or combinations (and
updates to those algorithms and code happen multiple times per week). It is
incumbent on websites - especially for large, complex, dynamic websites -
to do everything they can to conform. However, to date, no large, complex
software has been bug free, and it is equally difficult for such sites to
claim conformance with no accessibility defects on any page.

*9. Revised Wording in Conformance Quotes*

*Changed: *"including setting forth that conformance"* to* " including that
conformance is"

*Changed:* "The conformance requirements also set forth" *to*    "the
conformance requirements also state"

*Removed: * ...with text that states:

*10. Revised Wording  *

*Original: *While a useful methodology for providing confidence in either a
prior claim of 100% conformance across a website or as part of an internal
process to help an organization assess their progress toward 100%
conformance, in and of itself it doesn't address the challenges in making
every last aspect of every page conform 100% to every success criterion.

*Revised:* While WCAG-EM provides a practical method for claiming
conformance for a website, it doesn't fully address the challenges in
making every part of every page in a large, dynamic website conform to
every success criterion.


*11. Revised Wording   *

*Original:* Leveraging authoring tools can significantly help with a number
of the challenges with accessibility guidelines conformance and testing,
and in future versions of this document we hope to describe those in more
detail.

*Revised:* Leveraging ATAG-conformant authoring tools can significantly
help with preventing accessibility issues from occurring on a website. A
number of the challenges with conformance could be addressed with authoring
tools, and in future versions of this document we hope to describe those in
more detail.

*12. Revised Wording*

*Original: *The Research and Development Working Group was disbanded in
2015, and the document was never advanced to contain guidance on the
specific non-conformance-based qualities that should be used.

*Revised: *The Research and Development Working Group was not renewed
by W3C Membership
in 2015, though a Research Questions Task Force, under the Accessible
Platform Architecture (APA) Working Group, is able to look at similar types
of issues.

* 13. Revised Wording*

*Original: *One particularly interesting thing this research report
explored are qualities such as the severity of an accessibility barrier and
the time it takes for a site visitor to conduct a task, as an alternative
time it takes for a site visitor to conduct a task, as alternative approach
to conformance-based metrics.

*Revised: *  One particularly interesting thing this research report
explored are qualities such as the severity of an accessibility barrier and
the time it takes for a site visitor to conduct a task, as alternative
measures that could compliment conformance-based metrics.

*14. Revised Wording*

*Original:  *They are doing it through defining a W3C specification,
published as a W3C Recommendation in 2019, Accessibility Conformance
Testing (ACT) Rules Format 1.0, [act-rules-format-1.0] as well as
considering ways to output metrics around what the tests find. This could
be very useful as an alternative to the what the tests find. This could
contribute to an alternative to the conformance model, which requires a
site to have no defects on any page to claim conformance.

*Revised: *They are doing it through defining a W3C specification,
published as a W3C Recommendation in 2019, Accessibility Conformance
Testing (ACT) Rules Format 1.0, [act-rules-format-1.0] as well as
considering ways to output metrics around what the tests find. This could
contribute to an alternative to the conformance model, which requires a
site to have no defects on any page to claim conformance.

*15. Revised Wording*

*Original*:  So, fundamentally, while they are all substantial
contributions to the field of web accessibility, neither WCAG-EM, nor ATAG,
nor ACT, as they stand today, are able to fully address the many challenges
described in this document.

Separately, the phrase <q>substantially conforms to WCAG</q> is coming into
use as one way of conveying the status of a website that is broadly
accessible, but not 100% perfect, given the challenges noted above and
described more fully below. Unfortunately, that phrase has no W3C
definition today, nor does it actually address the accessibility challenges
with testing and conformance themselves.

*Revised: *While they are all substantial contributions to the field of web
accessibility, WCAG-EM, ATAG, and ACT task forces actively address
different types of challenges, and do not fully address the challenges
described in this document.

Separately, the phrase substantially conforms to WCAG is a way of conveying
that a website is broadly accessible, but that not every page conforms to
every requirement. Unfortunately, that phrase has no W3C definition today
and there is no definition or mechanism that sets a level for a site to
meet that would qualify as substantial conformance.

Approaches for Mitigating Challenges with Accessibility Guidelines
Conformance and Testing

*16. Revised Wording*

*Original: *While some approaches may be more applicable to a particular
website design than others; and not all approaches may be appropriate or
practical for a particular website, it is likely that many websites can
utilize at least some of these approaches.  Website authors are encouraged
to utilize as many of these approaches as possible to minimize these
challenges, and maximize the likelihood that all website visitors will be
able to use the site effectively.  Though the challenges described in this
document illustrate that it is not possible for large, complex, and/or
dynamic websites to meet the 100% perfection standard of WCAG 2.x
conformance, mitigation strategies might enable a substantial level of
conformance that enables people with disabilities to effectively utilize
websites with little difficulty.

*Revised: *While some approaches may be more applicable to a particular
website design than others; and not all approaches may be appropriate or
practical for a particular website, it is likely that many websites can
utilize at least some of these approaches. These approaches are proposed as
ways through which these conformance challenges might be able to be
mitigated, while maximizing the likelihood that all website visitors will
be able to use the site effectively. Though the challenges described in
this document illustrate that it is difficult for large, complex, and/or
dynamic websites to ensure a site has no defects on any page to claim
conformance of WCAG 2.x conformance across the whole website, mitigation
strategies may be able to create a substantial level of conformance that
enables people with disabilities to effectively utilize websites with
little difficulty.

*Goals*

*17. Revised Wording*

*Original: *WCAG 2.x conformance model may be difficult or impossible to
apply, and      WCAG 2.x conformance model may be difficult to apply could
lead to more effective conformance models the places where accessibility
conformance verification may present difficulties in scaling, could lead to
more effective conformance models and testing approaches in the future.

*Revised: *A better understanding of the situations in which the WCAG 2.x
conformance model may be difficult to apply could lead to more effective
conformance models and testing approaches in the future.

*18. Revised Wording*

*Original: *Challenges with the conformance model and testing verification
doesn't mean the criteria aren't valid. For example, while requiring human
judgment to validate a page limits testing to sampling of templates, flows,
and top tasks, etc. (see Challenge #1 below), absent that human judgement
it may not be possible to deliver a page that fully conforms to WCAG 2.x.
Similarly, while it may not be possible to ensure that all third party
content is fully accessible (see Challenge #3 below), absent review of that
content by a human sufficiently versed in accessibility it may again not be
possible to deliver pages containing third party content that fully conform
to WCAG 2.x. Human judgement is a core part of much of WCAG 2.x for good
reasons, and the challenges that arise from it important to successfully
grapple with.

*Revised: *Challenges with the conformance model do not in any way
invalidate the criteria. For example, while requiring human judgment to
validate a page limits testing to sampling of templates, flows, and top
tasks, etc. (see Challenge #1 below), without that human judgement it may
not be possible to deliver a page that makes sense to someone with a
disability. Similarly, while it may not be possible to know that all third
party content is fully accessible (see Challenge #3 below), without review
of that content by someone sufficiently versed in accessibility it may not
be possible to be sure that pages containing third party content fully
conform to WCAG 2.x. Human judgement is a core part of much of WCAG 2.x for
good reasons, and the challenges that arise from it are important to
successfully grapple with.

Additional Background

*19. Revised Wording*

*Original: *One of the reasons for publishing this draft document is to
seek additional contributions from the wider web community describing any
additional challenges, or further illustration of challenges in the
existing identified areas below; as well as contributions to the mitigation
approaches described herein, to provide further guidance for addressing
these challenges.  We seek to gain a thorough understanding of the
challenges faced by large, complex, and dynamic websites who are attempting
to provide accessible service to their web site users. It is expected that
a more thorough understanding of these challenges can lead to either a new
conformance model, or an alternative model that is more appropriate for
large, complex, and/or dynamic websites. Ideally, such a model would also
be able to distinguish between websites that are substantially accessible
for most visitors with disabilities most of the time, and websites that are
largely unusable by a significant portion of visitors with a
disability.This document also includes previously published research from
the Silver Task Force and Community Group that was specifically related to
Challenges with Accessibility Guidelines Conformance and Testing.

*Revised:* One of the reasons for publishing this draft document is to seek
additional contributions from the wider web community:

   - Any additional challenges, or further illustration of challenges in
   the existing identified areas below;
   - Contributions to the mitigation approaches, and questions or concerns
   about the mitigation approaches;

We seek to gain a thorough understanding of the challenges faced by large,
complex, and dynamic websites who are attempting to provide accessible
services to their web site users. It is expected that a more thorough
understanding of these challenges can lead to either a new conformance
model, or an alternative model that is more appropriate for large, complex,
and/or dynamic websites (in WCAG 3.0).

This document also includes previously published research from the Silver
Task Force and Community Group that was related to Challenges with
Accessibility Guidelines Conformance and Testing.

*20. Revised Wording*

*Original:* While we have heretofore emphasized systematically collecting a
comprehensive inventory of challenges, we believe our collection is now
sufficiently mature to begin enumerating and considering the various
mitigating approaches that have come to light as a result of this work. We
are publishing this document now to seek the widest possible public comment
and assistance in further cataloging and characterizing both these
challenges and these mitigation approaches, so that this work can become
widely reviewed input into the next major revision of W3C accessibility
guidelines (now chartered by W3C for eventual release as WCAG 3.0 and
currently in early development under the name Silver)

*Revised: *Previous drafts emphasized collecting a comprehensive inventory
of challenges, we believe our collection is now sufficiently mature to
begin enumerating and considering the various mitigating approaches that
have been suggested by various stakeholders. We are publishing this
document now to seek wide review to further catalogue and characterize the
challenges and mitigation approaches, so that this work can become input
into the next major revision of W3C accessibility guidelines (now chartered
by W3C for eventual release as WCAG 3.0).

Key Terms

*21. Revised Wording*

*Original: *Websites that are being updated with new content hundreds of
times an hour, let alone hundreds of times per second or more.

*Revised: *Websites that are constantly being updated with new content,
possibly hundreds of times an hour, or even thousands of times per second.

Challenge 1

*22. Revised Wording*

*Original:* Many existing accessibility success criteria expect informed
human evaluation to ensure that the end users benefit from conformance.

*Revised*:  Many existing accessibility success criteria require an
informed human evaluation to ensure that the human end-users benefit from
conformance.

Challenge 2

*23. Revised Wording*

*Original: * It may not be possible to validate every possible publishing
permutation with a page-level test, each of which can have an impact on
whether that particular rendering of the content at that particular moment
conforms.

*Revised: *It is difficult to validate every possible publishing
permutation with a page-level test, each of which can have an impact on
whether that particular rendering of the content at that particular moment
conforms.

Challenge 3

*24. Revised Wording*

*Orignal: * While the website can provide guidance on how to post content
so that it meets accessibility guidance, it is ultimately up to those third
parties to understand and correctly implement that guidance. And as noted
above, even with automated checking prior to accepting the post, many
Guidelines and Success Criteria expect human validation involvement.

Copyright and similar constraints that restrict the ability to modify or
impose requirements on third party data can also make full page conformance
impossible to assure, e.g. articles that allow reposting but without
modification due to copyright restrictions.

*Revised*:  The relationship to the 3rd party can be that of a user, a
customer, or a professional provider of content such as an advertiser.
While the website can provide guidance on how to post content so that it
meets accessibility guidance, it is ultimately up to those third parties to
understand and correctly implement that guidance. Constraints on page
templates and editing facilities can greatly help minimize accessibility
issues but, even with automated checks prior to accepting the post, some
Success Criteria require human assessment.

Copyright, commercial agreements, and similar constraints that restrict the
ability to modify or impose requirements on third party data can also make
conformance difficult to assure.

Challenge 4

*25. Revised Wording*

*Original: *The core principles, and many of the guidelines, contained in
WCAG 2.x, are broadly applicable outside of the web context. For example,
no matter the technology, information presented to humans needs to be
perceivable by them in order for them to access and use it. At the same
time, some of the specific guidelines and especially some of the individual
success criteria of WCAG 2.x are written specifically for web content and
web technologies, and may be difficult to apply to non-web information and
communications technologies (as set forth in the W3C Note Guidance on
Applying WCAG to non-web Information and Communication Technologies
(WCAG2ICT)). [[wcag2ict]] Furthermore, the state of programmatic test tools
for assessing whether non-web information and communications technologies
meet various WCAG 2.x success criteria varies widely with the type of
non-web document, the operating system, and the user interface toolkits
used to create the non-web software. In no case that we are aware of do
such tools explicitly map the accessibility issued found to specific WCAG
2.x success criteria. Therefore, it is potentially the case that for some
documents or software, it will not be possible to use any programmatic
accessibility evaluation tools for any success criterion — conformance to
each and every success criterion will need human expertise and judgment.

*Revised:* The core principles, and many of the guidelines, contained in
WCAG 2.x, are broadly applicable outside of the web context. For example,
no matter the technology, information presented to humans needs to be
perceivable by them in order for them to access and use it. At the same
time, some of the specific guidelines and especially some of the individual
success criteria of WCAG 2.x are  specifically scoped to web content and
web technologies, and may be difficult to apply to non-web information and
communications technologies (as set forth in the W3C Note Guidance on
Applying WCAG to non-web Information and Communication Technologies
(WCAG2ICT)). [[wcag2ict]] Furthermore, the state of programmatic test tools
for assessing whether non-web information and communications technologies
meet various WCAG 2.x success criteria varies widely with the type of
non-web document, the operating system, and the user interface toolkits
used to create the non-web software. Therefore, it is potentially the case
that for some documents or software, it will not be possible to use any
programmatic accessibility evaluation tools for some success criterion —
conformance to each and every success criterion will need human expertise
and judgment.

Challenges of Conformance as identified from Silver Research

*26. New Wording*
The following items are shown as they were originally presented by the
Silver task force, in future versions of this document they will be
integrated into the challenges.

5.2.4 Human Testable

*27. Revised Wording*
*Original:*  Opportunity: There's an opportunity to make the success
criteria more clear for human auditors and testers. fEducating business
leaders on how the varying levels of conformance apply to their
organization may be useful as well. We can educate about the ways that
people with disabilities use their assistive technology.

*Revised:*  Opportunity: There's an opportunity to improve the success
criteria such that human auditors and testers find the success criteria
more understandable. Educating business leaders on how the varying levels
of conformance apply to their organization may be useful as well. We can
educate about the ways that people with disabilities use their assistive
technology.

Accessibility Supported

*28. Revised Wording*

*Original: *Specific problem: Accessibility supported was never fully
implemented in a way that was clear and useful to developers and testers.
It also requires a harmonious relationship and persistent interoperability
between content technologies and requesting technologies that must be
continuously evaluated as either is updated.

*Revised: * Accessibility supported was implemented in a way that did not
facilitate consistent adoption by developers and testers. It also requires
a harmonious relationship and persistent interoperability between content
technologies and requesting technologies that must be continuously
evaluated as either is updated.

*29. Added Enabling Funder text*

This publication has been funded in part with U.S. Federal funds from the
Health and Human Services, National Institute on Disability, Independent
Living, and Rehabilitation Research (NIDILRR) under contract number
HHSP23301500054C. The content of this publication does not necessarily
reflect the views or policies of the U.S. Department of Health and Human
Services, nor does mention of trade names, commercial products, or
organizations imply endorsement by the U.S. Government.