W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > October 2011

Re: use-cases and scenarios for conformance evaluation

From: Detlev Fischer <fischer@dias.de>
Date: Thu, 20 Oct 2011 10:08:30 +0200
Message-ID: <4E9FD6FE.1040407@dias.de>
To: public-wai-evaltf@w3.org
Hi Aaron,

you have given us a lot to chew on. I think the use cases are all 
relevant and a timely input to ensure that our website accessibility 
evaluation methodology (for the sake of brevity, WAEM) will be of 
practical relevance. I am not sure however if WAEM (as currenty 
outlined) will be able to tackle these use cases properly since many 
hinge on contextual factors that may be too complex to cover or are at 
least not yet dealt with in the draft requirements.

In many cases, what you write boils down to the question: can WAEM be 
customised / trimmed down if we deal with a particular subset of users 
(e.g. mobile users using up-to-date smartphones), and can the 
relationship between techniques and respective tests be operationalised 
by the methodology (use one or the other, use both, or don't bother with 
a technique if you are dealing with subset X of web content or a limited 
set of UAs).

First of all, basing a claim on a subset of the overall web audience 
seems only permissible to me if the site owner can show that he is in 
control of the audience and its technical setup (UA, AT), which will 
often be the case in intranets. Not so for general web content. If the 
same web app can be used in a variety of contexts, including desktop and 
older mobile UA, these contexts should be fully supported.

If a service *forks* different versions for different UA or types of 
environment, the WAEM might need to make sense of the requirement for 
"conforming alternate version". This is tricky since in many cases, 
content is explicitly not equivalent (i.e. trimmed down for mobile). So 
this might need work to determine the conditions under which offering 
different versions of a service for different UA would justify a 
conformance claim that cuts down on some requirements because they are 
not applicable to the specific variant tested.

I think ways to customise the WAEM according to use cases like the ones 
you have offered might be useful, but it increases complexity a lot. It 
could turn out to be essential to make the resulting test procedure(s) 
usable at all, but I am not sure about that.

The basis for such customisation would need to be the assessment of 
'accessibility support', and this would hark back to a somewhat wobbly 
statement in the section on 'Understanding accessibility support', which 
betrays a certain degree of hand-waving when faced with the complexity 
of web content and use environments:

"The Working Group, therefore, limited itself to defining what 
constituted support and defers the judgment of how much, how many, or 
which AT must support a technology to the community and to entities 
closer to each situation that set requirements for an organization, 
purchase, community, etc."

The topic of defining what a "website" is when defining the scope and 
aplicablitiy of what we are undertaking to put together has been raised 
before. Coming from a hands-on perspective as someone using our own 
web-based test procedure in the evaluation of web sites, I realise that 
on the level of operational procedure, the tester needs something which 
is close enough to the targeted web site or application to provide 
practical guidance for actual checks and the interpretation of results. 
I cannot speak for testing mobile apps since we don't do that currenty 
but I can imagine the procedure might better be different there. It 
would test the same SC, but possibly do it differently.

For now, the working assumption of EVAL TF (meaning the assumption 
seemingly most widely supported / favoured by members of the TF) 
includes two things that are in stark contrast to any need for 
customising or honing the methodology to particular use cases:

(1) the methodology should be *tool-independent* (i.e. avoid referencing 
particular tools such as accessibilty tool bars, Firebug, aChecker or 
the quivalent for mobile environments) - the rationale possibly being 
that there are many ways to perform the prescribed checks.
This is true. The interpretation of results, however, may often be 
contingent on the tool used. Just an example: the mere listing of a web 
page headline hierarchy with WAT shows whether headlines are descriptive 
and whether the hierarchy is OK. It does not show, however, the *content 
below headlines* and therefore masks an assessment of whether the 
headlines have been used corectly to determine the content underneath.
Shadi would say that this level cannot and should not be covered by WAEM 
since this is already in the tests of the WCAG techniques.
There are two issues here:
A: this is only partly true: the tests in techniques mostly do not 
reference tools and therefore allow for different interpretations to the 
extent that different tools can lead to different results.
B: tests in techniques also say nothing about the problem of dealing 
with instance test results that are half way between PASS or FAIL (such 
as a not quite appropriate alt text that is better than nothing but much 
worse than the proper thing).
Start testing any complex site and you find a myriad of these mongrels 
living in limbo between pass and fail.

(2) the methodology should not *replicate* existing tests (i.e., bring 
test descriptions into a concise, organised form in an actual test 
procedure on the level of SC - or part of SC, as for complex SCs like 
1.1.1 and 1.3.1) - the rationale being that we would otherwise duplicate 
content and run into maintenance issues.
The problem then is that a test procedure for *actual testing* ( I think 
of a web app with pages that describe the SC, detail checks and contain 
form elements to select a ranking - pass/fail or some scale - and also 
comment fields), i.e. the procedure *that actually gets used* during 
evalution, would constantly reference (technically link) to external 
resources, such as tests in WCAG techniques. I have gone through all the 
WCAG techniques (bar Flash) in quite some detail and believe me, if 
testers would actually follow through all references and not just pass 
over the links because they think they already know how to do it, the 
methodology will be incredibly burdensome, long-winded, I dare say: 
unusable in the practical evaluation context.
This is why I insist again and again that the actual practical value of 
the methodology would be to provide a *usable synthesis* of the many 
atomic techniques out there in *one* resource (per checkpoint) that 
should be as concise as possible and guide assessment in line with 
techniques out there and their level of accessibility support. ARIA 
landmarks and HTML5 section elements instead of headlines are a case in 
point. If the WAEM cannot tell you (for your chosen scope of audience) 
whether ARIA landmarks are deemed sufficent to meet 2.4.1 Bypass Blocks, 
who will?
Allowing for customisation for different use cases like the ones 
sketched out by Aaron might be a good way forward and improve conciseness.
Our aim must be to produce something that is of proven benefit to 
testers and does not linger largely unused and unnoticed in academic limbo.

Synthesis, customisation, and constant technological change *do* raise 
the issue of maintenance and consistency. I firmly believe that to the 
extent that WAEM will offer a usable, hands-on procedure and not just a 
generic blueprint, it *will need* frequent maintenance as new 
technologies become accessibility supported and replace other 
technologies. It will need a support structure to deal with that 
maintenance issue.

To sum up: I believe the current WAEM requirements do not explicity deal 
with many of the issues raised in Aaron's input, such as accessibility 
support and contextual conformance requirements. What are the conditions 
under which a certain targeted service can claim conformance? Does it 
need a confoming alternate version with full equivalence or just a 
complementary service for another environment? Or is the target audience 
and use environment clearly delineated (e.g. intranet)?)

So much for now - sorry for the log post!


Am 19.10.2011 20:28, schrieb Aaron Leventhal:
> Use scenario:
> Planners, developers, testers at an organization working to remediate a large amount of web content. Organization wants to do a good job for end users, but has a limited budget. The organization needs requirements which do two things:
> * Maximize user benefit -- both for users with disabilities and all users when possible.
> * Minimize unnecessary costs, e.g. "busy-work" or complete rewrites where it doesn't make sense, as those resources should be used for other important work.
> Use cases:
> 1. Determine whether it is necessary to support techniques with no perceivable user impact in order to maintain WCAG conformance, given a known set of supported target browsers for the content.
> 	Example A: What is sufficient for identifying semantics and structure (1.3.1)? Specific example: on a site targeted to mobile browsers where navigation by quotation is not available in ATs, is it necessary to mark the<blockquote>s?
> 	Example B: Parsing errors (4.1.1) apply only to legacy ATs that are not available for many content targets. Is it necessary to conform when all target user agents can consume the content?
> 9. Determine which techniques are acceptable for conformance in a mobile application targeting only user agents on current generation small-screen devices.
> Examples:
> 	Example C: On a mobile site with a very small number of links at the start of each page, is it necessary to provide a mechanism to bypass blocks (2.4.1)?
> 	Example D: Is it necessary to follow SCR21, "Using functions of the Document Object Model (DOM) to add content to a page (Scripting)" -- the technique rejects innerHTML as not part of the spec (it is part of HTML5), and rejects document.write in all cases (should be sufficient if used as part of the page load script).
> 2. (Similar to #1, but for "near zero" rather than "no" user impact) Determine whether a technique with an absurdly low user impact is necessary to implement:
> 	Example: On a mobile site with a very small number of links at the start of each page (e.g. 3), is it necessary to provide a mechanism to bypass blocks (2.4.1)?
> 3. Determine which techniques or sub-techniques are equivalent and can be used in place of each other. Determine when one, more than one, or all of the equivalent techniques should be used.
> 	Example A: Determine which technique(s) are sufficient for bypassing large amounts of content (headings, skip links, ARIA landmarks, high level HTML 5 elements like<article>  etc.)
> 	Example B: the organization is retrofitting accessibility on its website and has data describing what ATs are typical for site visitors. The site is targeted to be conformant by 2014. The organization wants to implement the most usable solution for the target users with disabilities. Determine whether it is preferable to provide form entry guidance (descriptions, error messages, required, invalid, etc.) via HTML4, HTML5 or ARIA.
> 4. Determine whether techniques using modern markup can be used in place of traditional techniques.
> 	Example: For a site targeted to a limited number of browsers, determine the appropriate labeling mechanism for a search field with an inner label.
> 5. Determine whether a documented technique on a site external to W3C is sufficient for conformance.
> 	Example: Document conformance when using role="alert" for error identification (3.3), where the technique was provided on a well-known, highly credentialed accessibility checklist (e.g. in IBM web accessibility checklist).
> 6. When a given technique is not part of WCAG 2.0 techniques, determine how to sufficiently document it and remain conformant.
> 	Example: Document a long description technique which uses aria-describedby, such that it can be used in a conformance statement.
> 7. Determine whether the use of alternate conforming versions is conformant overall.
> 	Example: A video tutorial for email set up provides a link to alternate versions: "Setting Up Email for Screen Reader Users", and "Setting Up Email for Screen Magnifier Users". The alternate versions are clearly more usable than an audio described version where the ATs are running. Determine whether the links to the alternate versions are sufficient for conformance.
> 8. Determine whether techniques used in an app are "accessibility supported".
> 	Example: an organization is faced with making an advanced calendar app accessible, either by adding ARIA markup to it or by writing a new version in static HTML. Writing a new version would be more expensive and have limited applicability. What data (surveys, site analytics, online docs, etc.) would be sufficient to show that the proposed ARIA techniques are accessibility supported and WCAG conformant?
> 9. Determine whether a 3rd party library can be used while conforming to WCAG.
> 	Example A: an organization uses a 3rd party open source JavaScript widget library which was made accessible through a great deal of effort by the industry. The developers of the library claim that every effort has been made, but will not claim conformance to WCAG. The accessibility lead must determine whether to recommend against using the library because of legal requirements for WCAG compliance.
> 	Example B: an organization uses a 3rd party closed source Flash widget library, where every effort has been made for accessibility, but WCAG conformance has not been claimed. Manual testing with a variety of end users with disabilities shows satisfactory results. Is conformance even possible in this scenario?
> 10. Provide requirements to internal/external teams for checkpoints or techniques which do not lend themselves to automated testing, for remediation of a large website.
> 	Example: satisfy conformance for checkpoint 2.4.6 -- "Headings and labels describe topic or purpose", while remediating a 30,000 page website, given that manual checks would be required. Spot checks seem reasonable, but this is using common sense as opposed to conforming to WCAG language. Determine what's appropriate.
> Hope that helps,
> Aaron
>> -----Original Message-----
>> From: public-wai-evaltf-request@w3.org [mailto:public-wai-evaltf-
>> request@w3.org] On Behalf Of Shadi Abou-Zahra
>> Sent: Wednesday, October 19, 2011 11:44 AM
>> To: Eval TF
>> Subject: use-cases and scenarios for conformance evaluation
>> Hi folks,
>> As part of the requirements, what do people think of collecting some
>> different use-cases and scenarios for conformance evaluation?
>> We could use these to have specific examples to design to and also to
>> verify the Methodology. Please send your favorite use-cases and
>> scenarios to the list and I will collect them into a web page.
>> Best,
>>     Shadi
>> --
>> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
>> Activity Lead, W3C/WAI International Program Office
>> Evaluation and Repair Tools Working Group (ERT WG)
>> Research and Development Working Group (RDWG)
> ---------------------------------------------------------------------
> This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

Detlev Fischer PhD
DIAS GmbH - Daten, Informationssysteme und Analysen im Sozialen
Geschäftsführung: Thomas Lilienthal, Michael Zapp

Telefon: +49-40-43 18 75-25
Mobile: +49-157 7-170 73 84
Fax: +49-40-43 18 75-19
E-Mail: fischer@dias.de

Anschrift: Schulterblatt 36, D-20357 Hamburg
Amtsgericht Hamburg HRB 58 167
Geschäftsführer: Thomas Lilienthal, Michael Zapp
Received on Thursday, 20 October 2011 08:09:00 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:12 GMT