W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > October 2011

RE: Questions & Review -- Website Accessibility Evaluation Methodology for WCAG 2.0 Requirements

From: Aaron Leventhal <aleventhal@rim.com>
Date: Fri, 14 Oct 2011 13:43:19 -0400
To: "Velleman, Eric" <evelleman@bartimeus.nl>, "public-wai-evaltf@w3.org" <public-wai-evaltf@w3.org>
Message-ID: <600E102A81A3BD42BF087D2397E70692B921EB4E@XCH108CNC.rim.net>
Hi Eric, thanks for taking a look. My replies below (apologies for the length):

> Have you had a look at the new version of the requirements? I think some
> of your concerns may have been covered there. 
Yes, I have now. I don't see that the concerns were addressed.

> Is your comment about "both
> developers & end users benefit when content is made accessible for the
> environments it was designed for" not already covered inside WCAG 2.0? If
> not, we should cover it in the Methodology. 
My read is that WCAG failures and techniques for HTML do not vary for different uses of HTML. However, some are more or less applicable depending on what the HTML is targeted for -- e.g. there are some differences in what techniques make sense for a general desktop site, mobile site, a web app using scripted widgets, apps built with HTML and packaged for an app store on a specific set of devices, etc. Even if 90% of the techniques stay the same, common sense would dictate avoiding any work that is not useful for a given scenario.

As a person developing web accessibility requirements for a large organization, I see some issues with simply pointing to any generic doc describing how to comply with WCAG. 
1. The techniques are non-normative, yet WCAG compliance requires the use of techniques. How can a public policy requirement for general WCAG A or AA compliance be clear enough on what's required? Which specific techniques must be followed?
2. There can sometimes be redundancies between techniques. When is it necessary to use more than one technique in a document? For example, between ARIA landmarks, skip links and headings -- must I have all three? Are two enough?
3. Not all techniques or failure cases are appropriate in all situations. If a given checkpoint or technique can be shown to not affect the users of a product, must it be followed for compliance?
4. Not all possible techniques are listed as the technology evolves, and it's not always useful if policies encourage older techniques at the expense of newer ones. It depends on the situation -- for example, an app written only ARIA environments might use different techniques to provide form instruction & validation information. A strict reading of WCAG might discourage this and encourage the less usable lowest common denominator.
5. Testability -- I want to check off a checkpoint as done, but the automated tests are nonexistent or unreliable. I have 19,000 pages on my site. How do I prove I'm compliant? (Example: G130 -- descriptive headings).
6. Difficulty in cases of 3rd party libraries -- Example: what if we develop a new app using 3rd party widget library X which provides accessibility. We've tested the app with users with disabilities and they like it a lot more than the old static version. But, it is difficult to know exactly what underlying WCAG techniques the 3rd party library uses. How do I show I'm compliant?

Since a target audience includes "Policy makers, project managers, and other decision makers who need a standard." I believe it is vital for these issues to be clarified. Public policies which attempt to simply require WCAG compliance need to answer those questions. It's important that typical common sense practices can be compatible with WCAG compliance.

> But like your remark with
> tolerance metrics, I agree that we should not divert from WCAG 2.0 or
> apply weighting to guidelines, successcriteria etc. This is something we
> covered in the requirements already. How we cover tolerance metrics will
> be an interesting discussion and task when we start with the Methodology
> itself. Hope we can proceed to that soon.
I see that WCAG provides weights in the form of priority 1/2/3 but are there weights for specific techniques? Here's an example. Within 1.3.1 there are many techniques to add structure, and some are much important than others. It's clear that using headings, labels, table markup and lists is generally important. I argue they are far more important than using blockquote/cite, which is listed in the same technique (H49) and therefore currently gets the same priority. However, should blockquote even be required in environments where there are no specific accessibility features enabled by it? Common sense would say no. My point is, a very general tolerance metric is not granular enough to apply common sense to this situation. The new wording of "RQ 09: Consideration for occassional oversight errors" does not address purposeful avoidance where the work is deprioritized as it does not have a pay off. 

Finally, should compliance consider how reliable automated tests for a given technique can be. Some tests are very reliable but others are not useful at all. It's nearly impossible to test whether headings text is descriptive enough.

All of this can consider an effort that wants to get the best real results for the cost & effort. That's why I think end users are a stakeholder which should be named. That helps bring the common sense factor in. A technique that helps users more than another is better, and a technique which doesn't help anyone for the specific situation shouldn't be required for compliance.

> I will read through your comments with the new version of the requirements
> and see if we can answer them better in the text. Please help there if you
> can.

Yes, I've taken a look and don't see that my original comments have yet been addressed.

Thank you,
Aaron

> 
> Kindest regards,
> 
> Eric
> 
> 
> I think the basic concern you express has to be addressed inside the
> Methodology and the current set of requirements leave space for that.
> 
> Van: public-wai-evaltf-request@w3.org [public-wai-evaltf-request@w3.org]
> namens Aaron Leventhal [aleventhal@rim.com]
> Verzonden: woensdag 5 oktober 2011 23:09
> Aan: public-wai-evaltf@w3.org
> Onderwerp: Questions & Review -- Website Accessibility Evaluation
> Methodology for WCAG 2.0 Requirements
> 
> 
> Hi everyone,
> 
> 
> 
> I'm looking at this version:
> 
> http://www.w3.org/WAI/ER/conformance/ED-methodology-20110928.html
> 
> 
> 
> First some comments on a few requirements:
> 
> R02: Tool and browser independent
> 
> Some WCAG 2 techniques depend on specific target user agent scenarios.
> 
> -          Techniques provided to support older technologies and therefore
> won't apply to content not targeted to older software
> 
> -          ARIA/HTML5 techniques which depend on support in the specific
> tools and browsers used by the consumers of the content, but may not be
> sufficient if we assume all content must support older browsers.
> 
> 
> 
> R02 seems to imply that the evaluation criteria must be independent of the
> actual technologies end users employ to consume the content. Shouldn't
> evaluation criteria vary slightly depending on the likely usage scenarios?
> 
> R08. Address the needs of the target audience
> 
> 
> Was the idea of including end user consumers as an indirect target
> audience discussed?  That might go a long way in ensuring the evaluation
> methodologies optimize the end user benefit. There doesn't seem to be much
> sense in requiring lowest common denominator techniques in modern web apps
> targeted only to modern software, where ARIA/HTML5 techniques are
> beneficial to users.
> 
> 
> How will the needs of each audience be identified? I think listing them in
> this document would be a good idea because it could affect the specific
> requirements.
> 
> 
> For example, "Website developers, suppliers, procurers, and owners wishing
> to evaluate during the development process"  -- this audience has a need
> to lower cost or at least maximize the real user benefit for the effort
> spent. That could be at odds with R02 because specific browsers/tools
> cannot be considered in evaluation.
> 
> R14: Include tolerance metrics
> 
> - What potentially informs the tolerance metrics, other than the amount of
> content & number of violations?
> 
> - Are the tolerance metrics intended to allow for a certain # of
> violations for a given amount of content?
> 
> - Can tolerance metrics be based on the WCAG requirement/failure as some
> are more crucial than others? That would seem to create a dependency on
> importance weights for WCAG 2 techniques & failures. Importance weights
> would seem to depend on the type of content and the usage scenarios.
> 
> 
> 
> Recommendations
> 
> 
> 
> *         A stated goal should be to optimize end-user benefits while
> making the most of development resources.
> 
> *         A stated assumption should be that technology and accessibility
> techniques change over time.
> 
> *         A stated requirement could be to consider the breadth of browser
> support and depth of technology used for a given site/project. I'm
> provided examples for reference, although it's too detailed for the
> requirements doc:
> 
> 1.       Broad content maximizes support for older browsers, using only
> the very safest code:
> Appropriate WCAG techniques - safe techniques supporting a broad list of
> ATs including legacy versions
> 
> 2.       Middle ground content is designed or optimized for up-to-date
> browser versions
> Appropriate WCAG techniques should not include items only for older
> ATs/browsers
> 
> 3.       Deep, leading edge web content targets only very recent browsers
> Appropriate WCAG techniques include WAI-ARIA and HTML5. In this case,
> fallback WCAG techniques are only necessary when not all accessible target
> browsers support the necessary markup.
> 
> 4.       Device-specific content is the extreme end  -- takes advantage of
> device-specific features (deep) but has very shallow UA support:
> Appropriate techniques should be restricted to those with potential
> impact. It should not be necessary to use techniques to support features
> which do not exist and are not planned.
> 
> *         A stated requirement should be to consider common content
> development scenarios, such as:
> 
> 1.       Static documentation pages
> 
> 2.       Content that behaves like a native application
> 
> 3.       Content designed for mobile devices only
> 
> *         A stated requirement should be to define tolerance for using new
> techniques as a replacement for traditional techniques (for now, newer
> techniques include HTML5, ARIA and CSS3 -- but in the future, something
> could replace them as well).
> 
> *         A stated requirement should be to define tolerance for ignoring
> legacy techniques
> 
> *         The tolerance requirement should be more specific about
> granularity (by WCAG failure?) or other inputs to tolerance
> 
> *         The tool & browser independence requirement should be more
> specific
> 
> *         The audience section should either define some of needs or
> provide more guidance so that we're sure the requirements  make sense for
> the audience
> 
> 
> 
> Hope these suggestions are useful and not too repetitive .. in summary,
> both developers & end users benefit when content is made accessible for
> the environments it was designed for.
> 
> 
> 
> Thanks!
> 
> Aaron
> 
> Aaron Leventhal
> Senior Product Manager, Accessibility
> Research In Motion (RIM)
> www.blackberry.com/accessibility<http://www.blackberry.com/accessibility>
> 
> ---------------------------------------------------------------------
> This transmission (including any attachments) may contain confidential
> information, privileged material (including material protected by the
> solicitor-client or other applicable privileges), or constitute non-public
> information. Any use of this information by anyone other than the intended
> recipient is prohibited. If you have received this transmission in error,
> please immediately reply to the sender and delete this information from
> your system. Use, dissemination, distribution, or reproduction of this
> transmission by unintended recipients is not authorized and may be
> unlawful.

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
Received on Friday, 14 October 2011 17:43:59 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:12 GMT