RE: [EXTERNAL] Conformance Challenges Update Survey (Closes Tuesday May 26th at midnight EST)

Hi Rachael,

Thanks for sharing the conformance challenges information. With all due respect, I think it is unrealistic and inappropriate to demand a response with such a short timeframe considering this is a major US holiday weekend. Company W3C representatives such as myself, may need to consult with others internally before responding to the survey. I thus kindly ask that Tuesday midnight deadline for the survey be extended.

Thx,

ND



From: Rachael Bradley Montgomery <rachael@accessiblecommunity.org>
Sent: Friday, May 22, 2020 5:41 PM
To: WCAG <w3c-wai-gl@w3.org>
Subject: [EXTERNAL] Conformance Challenges Update Survey (Closes Tuesday May 26th at midnight EST)


Hello,



Please review the Conformance Challenges Survey<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.w3.org%2F2002%2F09%2Fwbs%2F35422%2Fconformance-challenges-05-21-20%2F&data=02%7C01%7Cnidogbo%40microsoft.com%7Ca222511372a34e52863a08d7feb22421%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637257913393325112&sdata=4yvxkw1dcJXMrHgTKYB%2BplrpoFmwFEtDVn%2BV0Z46oR8%3D&reserved=0> before Tuesday's meeting if possible. The survey will close on Tuesday at midnight Boston time.



We realize this is a quick turnaround request. We have received feedback from the public and W3C (outside AG) that the status, intent and some content of the conformance challenges document published on 1 May are not clear and may be causing some confusion. We would like to clarify the document as quickly as possible with the working group's consent. Unfortunately this means a quick review turnaround.

We have focused the proposed changes  to:
·  clarify the draft/work-in-progress status of the document;
·  clarify the scope of this document within the broader context of evaluation conformance;
·  clarify the forward-facing focus; and
·  remove absolutist characterizations of the current conformance model.

We ask you to review the changes and answer the survey question. To make this as easy as possible, there are three ways to do this review:

  *   Review  the list of changes at the bottom of this email.
  *   Review the proposed changes (scroll to the bottom) <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fwcag%2Fcompare%2Fconformance-challenges-update-ac%3Fexpand%3D1&data=02%7C01%7Cnidogbo%40microsoft.com%7Ca222511372a34e52863a08d7feb22421%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637257913393335106&sdata=9aJyG54kvP1%2BKIOnZFuf4C4thI9VmFdmuY4zaihUwGc%3D&reserved=0> and/or
  *   Read the revised draft<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fraw.githack.com%2Fw3c%2Fwcag%2Fconformance-challenges-update-ac%2Fconformance-challenges%2Findex.html&data=02%7C01%7Cnidogbo%40microsoft.com%7Ca222511372a34e52863a08d7feb22421%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637257913393345096&sdata=tFRie849FTt%2BfWZkwE9nuI0pF44Mg9dBVvz2DYacqn8%3D&reserved=0>

This is our first agenda item for Tuesday's meeting.



Kind Regards,



Rachael

***

List of Changes with Text

Abstract

1. Added an Editor's Note

Publication as a First Public Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

The intent of this First Public Working Draft by the Accessibility Guidelines Working Group is to explore how to improve the testability and page-based conformance verification of the WCAG accessibility guidelines in order to incorporate improvements into a future version of the guidelines. This draft is published to obtain public review of the issues identified and solutions proposed.

2. Adjusted Intro Paragraph

Original: This document explores how testability and page-based conformance verification of the WCAG 2.0 and 2.1 accessibility guidelines are challenging to apply to a broad range of websites and web applications.  It also explores approaches for mitigating these challenges, to realize as accessible a site as possible.

Revised: This document explores how the page-based conformance verification of the WCAG 2.0 and 2.1 accessibility guidelines are challenging to apply to certain websites and web applications. This document also explores ideas on how future versions of guidelines might address these challenges. This document focuses primarily on challenges to large, highly complex, dynamic sites.  Other efforts in WAI are looking at different aspects of conformance for different types of sites.

3. Revised Wording

Original: Large and dynamic sites may have too many changing permutations to validate effectively;

Revised: Large and dynamic sites with their changing permutations may be difficult to validate;

4. Removed Text

The purpose of this document is to help understand those challenges more holistically, and explore approaches to mitigating those challenges, both so that sites can use these mitigation approaches now, and also so that we can address the  challenges more fully in future accessibility guidelines such as WCAG 3.0...

Status of Document

5. Revised Text

Original: This is a First Public Working Draft by the Accessibility Guidelines Working Group. This document explores how testability and page-based conformance verification of the WCAG accessibility guidelines are challenging to apply to a broad range of websites and web applications. It also explores approaches for mitigating these challenges, to realize as accessible a site as possible. This draft is published to obtain public review of the issues identified and solutions proposed.

Revised: This is a Working Draft by the Accessibility Guidelines Working Group. This document explores how the page-based conformance verification of the WCAG 2.0 and 2.1 accessibility guidelines are challenging to apply to certain websites and web applications. It also explores approaches for mitigating these challenges, which could be incorporated into a future version of the guidelines. This draft is published to obtain public review of the issues identified and solutions proposed.

6. Revised Text (handled via an editor's note until publication)

Original: Publication as a First Public Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Revised: Publication as a Working Draft does not represent a consensus of the Working Group and does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Introduction: Problem Statement

7. Moved a paragraph up

While the challenges discussed in this document apply to websites and web applications broadly, this early version of the document focuses particularly on situations involving large, dynamic, and complex websites. There are valid reasons WCAG 2 and related resources have the conformance model they do; failures to conform to the current model are likely to present a barrier for people with disabilities. The issues raised in this document do not mean sites should not strive to conform to WCAG 2. It is also vital to consider aspects beyond the conformance model that lead to accessibility issues found during late stage testing, such as the lack of accessibility prioritization, training, and integration throughout the design, development, and maintenance processes. A new version of accessibility guidelines, W3C Accessibility Guidelines 3.0 (WCAG 3.0), rethinks all aspects of the accessibility guidance and is expressly chartered to develop a new conformance model that should help to address these challenges.

8. Revised Wording

Original: Ensuring that every one of these page updates fully satisfies all success criteria (as appropriate), especially where expert human review is required for some criteria, presents a massive scaling problem. Further, where pages are generated programmatically, finding every last bug related to that generation may prove challenging, especially when they only arise from uncommon content scenarios or combinations (and updates to those algorithms and code happen multiple times per week). Thus, the likelihood that every last page (out of what might be millions or billions of pages) can satisfy each and every success criterion 100% of the time is extremely low.

Revised: Ensuring that every one of these page updates fully satisfies all success criteria (as appropriate), especially where expert human review is required for some criteria presents a problem for scaling conformance assessments. Further, where pages are generated programmatically, finding every bug related to that generation may prove challenging, especially when they only arise from uncommon content scenarios or combinations (and updates to those algorithms and code happen multiple times per week). It is incumbent on websites - especially for large, complex, dynamic websites - to do everything they can to conform. However, to date, no large, complex software has been bug free, and it is equally difficult for such sites to claim conformance with no accessibility defects on any page.

9. Revised Wording in Conformance Quotes

Changed: "including setting forth that conformance" to " including that conformance is"

Changed: "The conformance requirements also set forth" to    "the conformance requirements also state"

Removed: ...with text that states:

10. Revised Wording

Original: While a useful methodology for providing confidence in either a prior claim of 100% conformance across a website or as part of an internal process to help an organization assess their progress toward 100% conformance, in and of itself it doesn't address the challenges in making every last aspect of every page conform 100% to every success criterion.

Revised: While WCAG-EM provides a practical method for claiming conformance for a website, it doesn't fully address the challenges in making every part of every page in a large, dynamic website conform to every success criterion.

11. Revised Wording

Original: Leveraging authoring tools can significantly help with a number of the challenges with accessibility guidelines conformance and testing, and in future versions of this document we hope to describe those in more detail.

Revised: Leveraging ATAG-conformant authoring tools can significantly help with preventing accessibility issues from occurring on a website. A number of the challenges with conformance could be addressed with authoring tools, and in future versions of this document we hope to describe those in more detail.

12. Revised Wording

Original: The Research and Development Working Group was disbanded in 2015, and the document was never advanced to contain guidance on the specific non-conformance-based qualities that should be used.

Revised: The Research and Development Working Group was not renewed by W3C Membership in 2015, though a Research Questions Task Force, under the Accessible Platform Architecture (APA) Working Group, is able to look at similar types of issues.

 13. Revised Wording

Original: One particularly interesting thing this research report explored are qualities such as the severity of an accessibility barrier and the time it takes for a site visitor to conduct a task, as an alternative time it takes for a site visitor to conduct a task, as alternative approach to conformance-based metrics.

Revised:   One particularly interesting thing this research report explored are qualities such as the severity of an accessibility barrier and the time it takes for a site visitor to conduct a task, as alternative measures that could compliment conformance-based metrics.

14. Revised Wording

Original:  They are doing it through defining a W3C specification, published as a W3C Recommendation in 2019, Accessibility Conformance Testing (ACT) Rules Format 1.0, [act-rules-format-1.0] as well as considering ways to output metrics around what the tests find. This could be very useful as an alternative to the what the tests find. This could contribute to an alternative to the conformance model, which requires a site to have no defects on any page to claim conformance.

Revised: They are doing it through defining a W3C specification, published as a W3C Recommendation in 2019, Accessibility Conformance Testing (ACT) Rules Format 1.0, [act-rules-format-1.0] as well as considering ways to output metrics around what the tests find. This could contribute to an alternative to the conformance model, which requires a site to have no defects on any page to claim conformance.

15. Revised Wording

Original:  So, fundamentally, while they are all substantial contributions to the field of web accessibility, neither WCAG-EM, nor ATAG, nor ACT, as they stand today, are able to fully address the many challenges described in this document.

Separately, the phrase <q>substantially conforms to WCAG</q> is coming into use as one way of conveying the status of a website that is broadly accessible, but not 100% perfect, given the challenges noted above and described more fully below. Unfortunately, that phrase has no W3C definition today, nor does it actually address the accessibility challenges with testing and conformance themselves.

Revised: While they are all substantial contributions to the field of web accessibility, WCAG-EM, ATAG, and ACT task forces actively address different types of challenges, and do not fully address the challenges described in this document.

Separately, the phrase substantially conforms to WCAG is a way of conveying that a website is broadly accessible, but that not every page conforms to every requirement. Unfortunately, that phrase has no W3C definition today and there is no definition or mechanism that sets a level for a site to meet that would qualify as substantial conformance.

Approaches for Mitigating Challenges with Accessibility Guidelines Conformance and Testing

16. Revised Wording

Original: While some approaches may be more applicable to a particular website design than others; and not all approaches may be appropriate or practical for a particular website, it is likely that many websites can utilize at least some of these approaches.  Website authors are encouraged to utilize as many of these approaches as possible to minimize these challenges, and maximize the likelihood that all website visitors will be able to use the site effectively.  Though the challenges described in this document illustrate that it is not possible for large, complex, and/or dynamic websites to meet the 100% perfection standard of WCAG 2.x conformance, mitigation strategies might enable a substantial level of conformance that enables people with disabilities to effectively utilize websites with little difficulty.

Revised: While some approaches may be more applicable to a particular website design than others; and not all approaches may be appropriate or practical for a particular website, it is likely that many websites can utilize at least some of these approaches. These approaches are proposed as ways through which these conformance challenges might be able to be mitigated, while maximizing the likelihood that all website visitors will be able to use the site effectively. Though the challenges described in this document illustrate that it is difficult for large, complex, and/or dynamic websites to ensure a site has no defects on any page to claim conformance of WCAG 2.x conformance across the whole website, mitigation strategies may be able to create a substantial level of conformance that enables people with disabilities to effectively utilize websites with little difficulty.

Goals

17. Revised Wording

Original: WCAG 2.x conformance model may be difficult or impossible to apply, and      WCAG 2.x conformance model may be difficult to apply could lead to more effective conformance models the places where accessibility conformance verification may present difficulties in scaling, could lead to more effective conformance models and testing approaches in the future.

Revised: A better understanding of the situations in which the WCAG 2.x conformance model may be difficult to apply could lead to more effective conformance models and testing approaches in the future.

18. Revised Wording

Original: Challenges with the conformance model and testing verification doesn't mean the criteria aren't valid. For example, while requiring human judgment to validate a page limits testing to sampling of templates, flows, and top tasks, etc. (see Challenge #1 below), absent that human judgement it may not be possible to deliver a page that fully conforms to WCAG 2.x. Similarly, while it may not be possible to ensure that all third party content is fully accessible (see Challenge #3 below), absent review of that content by a human sufficiently versed in accessibility it may again not be possible to deliver pages containing third party content that fully conform to WCAG 2.x. Human judgement is a core part of much of WCAG 2.x for good reasons, and the challenges that arise from it important to successfully grapple with.

Revised: Challenges with the conformance model do not in any way invalidate the criteria. For example, while requiring human judgment to validate a page limits testing to sampling of templates, flows, and top tasks, etc. (see Challenge #1 below), without that human judgement it may not be possible to deliver a page that makes sense to someone with a disability. Similarly, while it may not be possible to know that all third party content is fully accessible (see Challenge #3 below), without review of that content by someone sufficiently versed in accessibility it may not be possible to be sure that pages containing third party content fully conform to WCAG 2.x. Human judgement is a core part of much of WCAG 2.x for good reasons, and the challenges that arise from it are important to successfully grapple with.

Additional Background

19. Revised Wording

Original: One of the reasons for publishing this draft document is to seek additional contributions from the wider web community describing any additional challenges, or further illustration of challenges in the existing identified areas below; as well as contributions to the mitigation approaches described herein, to provide further guidance for addressing these challenges.  We seek to gain a thorough understanding of the challenges faced by large, complex, and dynamic websites who are attempting to provide accessible service to their web site users. It is expected that a more thorough understanding of these challenges can lead to either a new conformance model, or an alternative model that is more appropriate for large, complex, and/or dynamic websites. Ideally, such a model would also be able to distinguish between websites that are substantially accessible for most visitors with disabilities most of the time, and websites that are largely unusable by a significant portion of visitors with a disability.This document also includes previously published research from the Silver Task Force and Community Group that was specifically related to Challenges with Accessibility Guidelines Conformance and Testing.

Revised: One of the reasons for publishing this draft document is to seek additional contributions from the wider web community:
·  Any additional challenges, or further illustration of challenges in the existing identified areas below;
·  Contributions to the mitigation approaches, and questions or concerns about the mitigation approaches;
We seek to gain a thorough understanding of the challenges faced by large, complex, and dynamic websites who are attempting to provide accessible services to their web site users. It is expected that a more thorough understanding of these challenges can lead to either a new conformance model, or an alternative model that is more appropriate for large, complex, and/or dynamic websites (in WCAG 3.0).

This document also includes previously published research from the Silver Task Force and Community Group that was related to Challenges with Accessibility Guidelines Conformance and Testing.

20. Revised Wording

Original: While we have heretofore emphasized systematically collecting a comprehensive inventory of challenges, we believe our collection is now sufficiently mature to begin enumerating and considering the various mitigating approaches that have come to light as a result of this work. We are publishing this document now to seek the widest possible public comment and assistance in further cataloging and characterizing both these challenges and these mitigation approaches, so that this work can become widely reviewed input into the next major revision of W3C accessibility guidelines (now chartered by W3C for eventual release as WCAG 3.0 and currently in early development under the name Silver)

Revised: Previous drafts emphasized collecting a comprehensive inventory of challenges, we believe our collection is now sufficiently mature to begin enumerating and considering the various mitigating approaches that have been suggested by various stakeholders. We are publishing this document now to seek wide review to further catalogue and characterize the challenges and mitigation approaches, so that this work can become input into the next major revision of W3C accessibility guidelines (now chartered by W3C for eventual release as WCAG 3.0).

Key Terms

21. Revised Wording

Original: Websites that are being updated with new content hundreds of times an hour, let alone hundreds of times per second or more.

Revised: Websites that are constantly being updated with new content, possibly hundreds of times an hour, or even thousands of times per second.

Challenge 1

22. Revised Wording

Original: Many existing accessibility success criteria expect informed human evaluation to ensure that the end users benefit from conformance.

Revised:  Many existing accessibility success criteria require an informed human evaluation to ensure that the human end-users benefit from conformance.

Challenge 2

23. Revised Wording

Original:  It may not be possible to validate every possible publishing permutation with a page-level test, each of which can have an impact on whether that particular rendering of the content at that particular moment conforms.

Revised: It is difficult to validate every possible publishing permutation with a page-level test, each of which can have an impact on whether that particular rendering of the content at that particular moment conforms.

Challenge 3

24. Revised Wording

Orignal:  While the website can provide guidance on how to post content so that it meets accessibility guidance, it is ultimately up to those third parties to understand and correctly implement that guidance. And as noted above, even with automated checking prior to accepting the post, many Guidelines and Success Criteria expect human validation involvement.

Copyright and similar constraints that restrict the ability to modify or impose requirements on third party data can also make full page conformance impossible to assure, e.g. articles that allow reposting but without modification due to copyright restrictions.

Revised:  The relationship to the 3rd party can be that of a user, a customer, or a professional provider of content such as an advertiser. While the website can provide guidance on how to post content so that it meets accessibility guidance, it is ultimately up to those third parties to understand and correctly implement that guidance. Constraints on page templates and editing facilities can greatly help minimize accessibility issues but, even with automated checks prior to accepting the post, some Success Criteria require human assessment.

Copyright, commercial agreements, and similar constraints that restrict the ability to modify or impose requirements on third party data can also make conformance difficult to assure.

Challenge 4

25. Revised Wording

Original: The core principles, and many of the guidelines, contained in WCAG 2.x, are broadly applicable outside of the web context. For example, no matter the technology, information presented to humans needs to be perceivable by them in order for them to access and use it. At the same time, some of the specific guidelines and especially some of the individual success criteria of WCAG 2.x are written specifically for web content and web technologies, and may be difficult to apply to non-web information and communications technologies (as set forth in the W3C Note Guidance on Applying WCAG to non-web Information and Communication Technologies (WCAG2ICT)). [[wcag2ict]] Furthermore, the state of programmatic test tools for assessing whether non-web information and communications technologies meet various WCAG 2.x success criteria varies widely with the type of non-web document, the operating system, and the user interface toolkits used to create the non-web software. In no case that we are aware of do such tools explicitly map the accessibility issued found to specific WCAG 2.x success criteria. Therefore, it is potentially the case that for some documents or software, it will not be possible to use any programmatic accessibility evaluation tools for any success criterion — conformance to each and every success criterion will need human expertise and judgment.

Revised: The core principles, and many of the guidelines, contained in WCAG 2.x, are broadly applicable outside of the web context. For example, no matter the technology, information presented to humans needs to be perceivable by them in order for them to access and use it. At the same time, some of the specific guidelines and especially some of the individual success criteria of WCAG 2.x are  specifically scoped to web content and web technologies, and may be difficult to apply to non-web information and communications technologies (as set forth in the W3C Note Guidance on Applying WCAG to non-web Information and Communication Technologies (WCAG2ICT)). [[wcag2ict]] Furthermore, the state of programmatic test tools for assessing whether non-web information and communications technologies meet various WCAG 2.x success criteria varies widely with the type of non-web document, the operating system, and the user interface toolkits used to create the non-web software. Therefore, it is potentially the case that for some documents or software, it will not be possible to use any programmatic accessibility evaluation tools for some success criterion — conformance to each and every success criterion will need human expertise and judgment.

Challenges of Conformance as identified from Silver Research

26. New Wording
The following items are shown as they were originally presented by the Silver task force, in future versions of this document they will be integrated into the challenges.

5.2.4 Human Testable

27. Revised Wording
Original:  Opportunity: There's an opportunity to make the success criteria more clear for human auditors and testers. fEducating business leaders on how the varying levels of conformance apply to their organization may be useful as well. We can educate about the ways that people with disabilities use their assistive technology.

Revised:  Opportunity: There's an opportunity to improve the success criteria such that human auditors and testers find the success criteria more understandable. Educating business leaders on how the varying levels of conformance apply to their organization may be useful as well. We can educate about the ways that people with disabilities use their assistive technology.

Accessibility Supported

28. Revised Wording

Original: Specific problem: Accessibility supported was never fully implemented in a way that was clear and useful to developers and testers. It also requires a harmonious relationship and persistent interoperability between content technologies and requesting technologies that must be continuously evaluated as either is updated.

Revised:  Accessibility supported was implemented in a way that did not facilitate consistent adoption by developers and testers. It also requires a harmonious relationship and persistent interoperability between content technologies and requesting technologies that must be continuously evaluated as either is updated.

29. Added Enabling Funder text

This publication has been funded in part with U.S. Federal funds from the Health and Human Services, National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) under contract number HHSP23301500054C. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.