W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > October 2011

RE: use-cases and scenarios for conformance evaluation

From: Aaron Leventhal <aleventhal@rim.com>
Date: Thu, 20 Oct 2011 16:18:41 -0400
To: Detlev Fischer <fischer@dias.de>, "public-wai-evaltf@w3.org" <public-wai-evaltf@w3.org>
Message-ID: <600E102A81A3BD42BF087D2397E706920228ACB611@XCH108CNC.rim.net>
Hi Detlev,

Thanks for the detailed reply. I'm glad if my thoughts on WCAG conformance are useful to anyone.

I appreciated your comments about ARIA landmarks:
> If the WAEM cannot tell you (for your chosen scope of audience)
> whether ARIA landmarks are deemed sufficent to meet 2.4.1 Bypass Blocks,
> who will?
Perhaps WCAG's intention was to provide the sufficient/advisory delineation for this.
WCAG says, "Most Success Criteria have multiple sufficient techniques listed. Any of the listed sufficient techniques can be used to meet the Success Criterion."
And then it says:
"In addition to the sufficient techniques, there are a number of advisory techniques that can enhance accessibility, but did not qualify as sufficient techniques because they are not sufficient to meet the full requirements of the Success Criteria, they are not testable, and/or because they are good and effective techniques in some circumstances but not effective or helpful in others."

So, regarding our ARIA landmark use case -- there are currently only 3 ARIA techniques, all advisory. There is no technique for ARIA landmarks, which means we are already guided against using them, even when all target browsers for our content supports them. This is probably because at the time they were currently only "effective in some circumstances". I don't think there are any HTML5 techniques provided at all.

> Synthesis, customisation, and constant technological change *do* raise
> the issue of maintenance and consistency. I firmly believe that to the
> extent that WAEM will offer a usable, hands-on procedure and not just a
> generic blueprint, it *will need* frequent maintenance as new
> technologies become accessibility supported and replace other
> technologies. It will need a support structure to deal with that
> maintenance issue.
Again this may be a WCAG issue. WCAG probably needs to provide more modern techniques, or at least create a stronger potential link with externally developed techniques, indicating what is sufficient. WCAG has two similar concepts required for conformance -- "accessibility supported" and whether a technique is "sufficient". On one hand, the definition of "accessibility supported" is very loose in terms of how many ATs and browsers must be supported. That gives us flexibility regarding the use of newer browsers and ATs. On the other hand, something is not "sufficient" if it is only "effective in some circumstances", which is quite strict in terms of browsers and ATs -- it doesn't even take the context of the content into account.

The context of the web audience and technical setup are key as to what should be considered accessibility supported and sufficient ...

> First of all, basing a claim on a subset of the overall web audience seems only permissible
> to me if the site owner can show that he is in control of the audience and its technical setup
> (UA, AT), which will often be the case in intranets.
> Not so for general web content.
I do need to humbly disagree here with the "not so for general web content" statement. Here are some typical examples where a public site controls the technical setup:
1. Site controls which web app features are available based on browser or capability detection:
    * Gmail only shows the "Chat feature" only in newer browsers. Older browsers hide the link in the sidebar.
    * Google Docs provides this message: "Your browser does not support all features of Google
Docs. If you are having problems, try Google Chrome."
    * JS libraries use capability detection to provide features which gracefully degrade or are unavailable. Developers use the capability detection to roll their own advanced features.
2. Site controls availability of a page, site or web app on a per-browser basis:
    * iCloud unsupported browser message: https://www.icloud.com/unsupported_browser/
    * Google Plus has a really nice message with links:
                https://plus.google.com/not-supported/?ref=/
    * Yahoo mail provides a similar message

These content switches on the public web is both common and necessary. It's simply too expensive and difficult to try to try to support many features in older browsers.

> If a service *forks* different versions for different UA or types of
> environment, the WAEM might need to make sense of the requirement for
> "conforming alternate version". This is tricky ... <snip>
Agreed. However, is WAEM going to be another WAI commentary on the meaning of WCAG, when there is already the "Understanding WCAG" text? Where does "Understanding WCAG" end and WAEM begin?

> I think ways to customise the WAEM according to use cases like the ones
> you have offered might be useful, but it increases complexity a lot.
We might be able to simplify here by asking WCAG WG to consider some changes first:
1. Split out the 3 factors built into the concept of "advisory" techniques. Currently techniques are advisory if they A) do not "meet full requirements of the Success Criteria, B) they are not testable, and/or C) because they are good and effective techniques in some circumstances but not effective or helpful in others.". Each of these needs to be handled separately:
        A. Does not meet full requirements: this is a good reason to not be sufficient. I don't currently know which are like this because this info has been blended into advisory vs. not advisory.
        B. Not testable: define what this means. I have no idea .. manual, provably not automatable, not automated in current tools? Isn't everything testable on a manual level? And, if it's manually tested why can't that be sufficient? Innovations in testing may make some of these statements false. In the end, I think this needs to be removed from WCAG. If anything, it should be part of WAEM.
        C. Effective in some circumstances and not others: this seems to be stricter than accessibility supported -- not sure whether that's because it's broader than user agent choice. Would like to get criteria clarified.
2. Promote advisory techniques which were omitted merely because they were "effective in some circumstances and not others". Instead, allow an intelligent determination similar to the flexible "accessibility supported".
3. Develop a full set of HTML5 and ARIA techniques. Note that HTML5 implicitly includes ARIA -- see http://dev.w3.org/html5/spec/Overview.html#wai-aria. The browsers and ATs supporting these techniques are now widespread, as are the web apps which need them. It's time to start making this stuff official, even with caveats about needing them to be "accessibility supported" or something stricter.
4. Provide guidance on documenting a new sufficient technique, such that it can be used in a conformance claim.
5. Under conformance, provide developers with a way out of implementing anything with zero impact on users. State that any given technique (or sub-technique) does not need to be applied if it provably has zero user impact. This puts the burden on the website developer to show why a technique wasn't implemented. But, it also provides developers with a reasonable alternative to repeatedly making useless changes to code. Developers will need some guidance on this -- what is or isn't zero-impact, what needs to be documented for conformance, etc., but it's fair to allow this. In a way it's just the counterpart to "accessibility supported", requiring about the same amount of thought.

These changes to WCAG would already provide developers of advanced web apps with appropriate conformance options that they don't currently have. And, I hope it simplifies the context questions for WAEM a bit.

Aaron

> -----Original Message-----
> From: public-wai-evaltf-request@w3.org [mailto:public-wai-evaltf-
> request@w3.org] On Behalf Of Detlev Fischer
> Sent: Thursday, October 20, 2011 4:09 AM
> To: public-wai-evaltf@w3.org
> Subject: Re: use-cases and scenarios for conformance evaluation
>
> Hi Aaron,
>
> you have given us a lot to chew on. I think the use cases are all
> relevant and a timely input to ensure that our website accessibility
> evaluation methodology (for the sake of brevity, WAEM) will be of
> practical relevance. I am not sure however if WAEM (as currenty
> outlined) will be able to tackle these use cases properly since many
> hinge on contextual factors that may be too complex to cover or are at
> least not yet dealt with in the draft requirements.
>
> In many cases, what you write boils down to the question: can WAEM be
> customised / trimmed down if we deal with a particular subset of users
> (e.g. mobile users using up-to-date smartphones), and can the
> relationship between techniques and respective tests be operationalised
> by the methodology (use one or the other, use both, or don't bother with
> a technique if you are dealing with subset X of web content or a limited
> set of UAs).
>
> First of all, basing a claim on a subset of the overall web audience
> seems only permissible to me if the site owner can show that he is in
> control of the audience and its technical setup (UA, AT), which will
> often be the case in intranets. Not so for general web content. If the
> same web app can be used in a variety of contexts, including desktop and
> older mobile UA, these contexts should be fully supported.
>
> If a service *forks* different versions for different UA or types of
> environment, the WAEM might need to make sense of the requirement for
> "conforming alternate version". This is tricky since in many cases,
> content is explicitly not equivalent (i.e. trimmed down for mobile). So
> this might need work to determine the conditions under which offering
> different versions of a service for different UA would justify a
> conformance claim that cuts down on some requirements because they are
> not applicable to the specific variant tested.
>
> I think ways to customise the WAEM according to use cases like the ones
> you have offered might be useful, but it increases complexity a lot. It
> could turn out to be essential to make the resulting test procedure(s)
> usable at all, but I am not sure about that.
>
> The basis for such customisation would need to be the assessment of
> 'accessibility support', and this would hark back to a somewhat wobbly
> statement in the section on 'Understanding accessibility support', which
> betrays a certain degree of hand-waving when faced with the complexity
> of web content and use environments:
>
> "The Working Group, therefore, limited itself to defining what
> constituted support and defers the judgment of how much, how many, or
> which AT must support a technology to the community and to entities
> closer to each situation that set requirements for an organization,
> purchase, community, etc."
> http://url.ie/dcmx
>
> The topic of defining what a "website" is when defining the scope and
> aplicablitiy of what we are undertaking to put together has been raised
> before. Coming from a hands-on perspective as someone using our own
> web-based test procedure in the evaluation of web sites, I realise that
> on the level of operational procedure, the tester needs something which
> is close enough to the targeted web site or application to provide
> practical guidance for actual checks and the interpretation of results.
> I cannot speak for testing mobile apps since we don't do that currenty
> but I can imagine the procedure might better be different there. It
> would test the same SC, but possibly do it differently.
>
> For now, the working assumption of EVAL TF (meaning the assumption
> seemingly most widely supported / favoured by members of the TF)
> includes two things that are in stark contrast to any need for
> customising or honing the methodology to particular use cases:
>
> (1) the methodology should be *tool-independent* (i.e. avoid referencing
> particular tools such as accessibilty tool bars, Firebug, aChecker or
> the quivalent for mobile environments) - the rationale possibly being
> that there are many ways to perform the prescribed checks.
> This is true. The interpretation of results, however, may often be
> contingent on the tool used. Just an example: the mere listing of a web
> page headline hierarchy with WAT shows whether headlines are descriptive
> and whether the hierarchy is OK. It does not show, however, the *content
> below headlines* and therefore masks an assessment of whether the
> headlines have been used corectly to determine the content underneath.
> Shadi would say that this level cannot and should not be covered by WAEM
> since this is already in the tests of the WCAG techniques.
> There are two issues here:
> A: this is only partly true: the tests in techniques mostly do not
> reference tools and therefore allow for different interpretations to the
> extent that different tools can lead to different results.
> B: tests in techniques also say nothing about the problem of dealing
> with instance test results that are half way between PASS or FAIL (such
> as a not quite appropriate alt text that is better than nothing but much
> worse than the proper thing).
> Start testing any complex site and you find a myriad of these mongrels
> living in limbo between pass and fail.
>
> (2) the methodology should not *replicate* existing tests (i.e., bring
> test descriptions into a concise, organised form in an actual test
> procedure on the level of SC - or part of SC, as for complex SCs like
> 1.1.1 and 1.3.1) - the rationale being that we would otherwise duplicate
> content and run into maintenance issues.
> The problem then is that a test procedure for *actual testing* ( I think
> of a web app with pages that describe the SC, detail checks and contain
> form elements to select a ranking - pass/fail or some scale - and also
> comment fields), i.e. the procedure *that actually gets used* during
> evalution, would constantly reference (technically link) to external
> resources, such as tests in WCAG techniques. I have gone through all the
> WCAG techniques (bar Flash) in quite some detail and believe me, if
> testers would actually follow through all references and not just pass
> over the links because they think they already know how to do it, the
> methodology will be incredibly burdensome, long-winded, I dare say:
> unusable in the practical evaluation context.
> This is why I insist again and again that the actual practical value of
> the methodology would be to provide a *usable synthesis* of the many
> atomic techniques out there in *one* resource (per checkpoint) that
> should be as concise as possible and guide assessment in line with
> techniques out there and their level of accessibility support. ARIA
> landmarks and HTML5 section elements instead of headlines are a case in
> point. If the WAEM cannot tell you (for your chosen scope of audience)
> whether ARIA landmarks are deemed sufficent to meet 2.4.1 Bypass Blocks,
> who will?
> Allowing for customisation for different use cases like the ones
> sketched out by Aaron might be a good way forward and improve conciseness.
> Our aim must be to produce something that is of proven benefit to
> testers and does not linger largely unused and unnoticed in academic
> limbo.
>
> Synthesis, customisation, and constant technological change *do* raise
> the issue of maintenance and consistency. I firmly believe that to the
> extent that WAEM will offer a usable, hands-on procedure and not just a
> generic blueprint, it *will need* frequent maintenance as new
> technologies become accessibility supported and replace other
> technologies. It will need a support structure to deal with that
> maintenance issue.
>
> To sum up: I believe the current WAEM requirements do not explicity deal
> with many of the issues raised in Aaron's input, such as accessibility
> support and contextual conformance requirements. What are the conditions
> under which a certain targeted service can claim conformance? Does it
> need a confoming alternate version with full equivalence or just a
> complementary service for another environment? Or is the target audience
> and use environment clearly delineated (e.g. intranet)?)
>
> So much for now - sorry for the log post!
>
> Detlev
>
>
> Am 19.10.2011 20:28, schrieb Aaron Leventhal:
> > Use scenario:
> > Planners, developers, testers at an organization working to remediate a
> large amount of web content. Organization wants to do a good job for end
> users, but has a limited budget. The organization needs requirements which
> do two things:
> > * Maximize user benefit -- both for users with disabilities and all
> users when possible.
> > * Minimize unnecessary costs, e.g. "busy-work" or complete rewrites
> where it doesn't make sense, as those resources should be used for other
> important work.
> >
> > Use cases:
> > 1. Determine whether it is necessary to support techniques with no
> perceivable user impact in order to maintain WCAG conformance, given a
> known set of supported target browsers for the content.
> >     Example A: What is sufficient for identifying semantics and
> structure (1.3.1)? Specific example: on a site targeted to mobile browsers
> where navigation by quotation is not available in ATs, is it necessary to
> mark the<blockquote>s?
> >     Example B: Parsing errors (4.1.1) apply only to legacy ATs that are
> not available for many content targets. Is it necessary to conform when
> all target user agents can consume the content?
> > 9. Determine which techniques are acceptable for conformance in a mobile
> application targeting only user agents on current generation small-screen
> devices.
> > Examples:
> >     Example C: On a mobile site with a very small number of links at
> the start of each page, is it necessary to provide a mechanism to bypass
> blocks (2.4.1)?
> >     Example D: Is it necessary to follow SCR21, "Using functions of the
> Document Object Model (DOM) to add content to a page (Scripting)" -- the
> technique rejects innerHTML as not part of the spec (it is part of HTML5),
> and rejects document.write in all cases (should be sufficient if used as
> part of the page load script).
> >
> > 2. (Similar to #1, but for "near zero" rather than "no" user impact)
> Determine whether a technique with an absurdly low user impact is
> necessary to implement:
> >     Example: On a mobile site with a very small number of links at the
> start of each page (e.g. 3), is it necessary to provide a mechanism to
> bypass blocks (2.4.1)?
> >
> > 3. Determine which techniques or sub-techniques are equivalent and can
> be used in place of each other. Determine when one, more than one, or all
> of the equivalent techniques should be used.
> >     Example A: Determine which technique(s) are sufficient for
> bypassing large amounts of content (headings, skip links, ARIA landmarks,
> high level HTML 5 elements like<article>  etc.)
> >     Example B: the organization is retrofitting accessibility on its
> website and has data describing what ATs are typical for site visitors.
> The site is targeted to be conformant by 2014. The organization wants to
> implement the most usable solution for the target users with disabilities.
> Determine whether it is preferable to provide form entry guidance
> (descriptions, error messages, required, invalid, etc.) via HTML4, HTML5
> or ARIA.
> >
> > 4. Determine whether techniques using modern markup can be used in place
> of traditional techniques.
> >     Example: For a site targeted to a limited number of browsers,
> determine the appropriate labeling mechanism for a search field with an
> inner label.
> >
> > 5. Determine whether a documented technique on a site external to W3C is
> sufficient for conformance.
> >     Example: Document conformance when using role="alert" for error
> identification (3.3), where the technique was provided on a well-known,
> highly credentialed accessibility checklist (e.g. in IBM web accessibility
> checklist).
> >
> > 6. When a given technique is not part of WCAG 2.0 techniques, determine
> how to sufficiently document it and remain conformant.
> >     Example: Document a long description technique which uses aria-
> describedby, such that it can be used in a conformance statement.
> >
> > 7. Determine whether the use of alternate conforming versions is
> conformant overall.
> >     Example: A video tutorial for email set up provides a link to
> alternate versions: "Setting Up Email for Screen Reader Users", and
> "Setting Up Email for Screen Magnifier Users". The alternate versions are
> clearly more usable than an audio described version where the ATs are
> running. Determine whether the links to the alternate versions are
> sufficient for conformance.
> >
> > 8. Determine whether techniques used in an app are "accessibility
> supported".
> >     Example: an organization is faced with making an advanced calendar
> app accessible, either by adding ARIA markup to it or by writing a new
> version in static HTML. Writing a new version would be more expensive and
> have limited applicability. What data (surveys, site analytics, online
> docs, etc.) would be sufficient to show that the proposed ARIA techniques
> are accessibility supported and WCAG conformant?
> >
> > 9. Determine whether a 3rd party library can be used while conforming to
> WCAG.
> >     Example A: an organization uses a 3rd party open source JavaScript
> widget library which was made accessible through a great deal of effort by
> the industry. The developers of the library claim that every effort has
> been made, but will not claim conformance to WCAG. The accessibility lead
> must determine whether to recommend against using the library because of
> legal requirements for WCAG compliance.
> >     Example B: an organization uses a 3rd party closed source Flash
> widget library, where every effort has been made for accessibility, but
> WCAG conformance has not been claimed. Manual testing with a variety of
> end users with disabilities shows satisfactory results. Is conformance
> even possible in this scenario?
> >
> > 10. Provide requirements to internal/external teams for checkpoints or
> techniques which do not lend themselves to automated testing, for
> remediation of a large website.
> >     Example: satisfy conformance for checkpoint 2.4.6 -- "Headings and
> labels describe topic or purpose", while remediating a 30,000 page
> website, given that manual checks would be required. Spot checks seem
> reasonable, but this is using common sense as opposed to conforming to
> WCAG language. Determine what's appropriate.
> >
> > Hope that helps,
> >
> > Aaron
> >
> >> -----Original Message-----
> >> From: public-wai-evaltf-request@w3.org [mailto:public-wai-evaltf-
> >> request@w3.org] On Behalf Of Shadi Abou-Zahra
> >> Sent: Wednesday, October 19, 2011 11:44 AM
> >> To: Eval TF
> >> Subject: use-cases and scenarios for conformance evaluation
> >>
> >> Hi folks,
> >>
> >> As part of the requirements, what do people think of collecting some
> >> different use-cases and scenarios for conformance evaluation?
> >>
> >> We could use these to have specific examples to design to and also to
> >> verify the Methodology. Please send your favorite use-cases and
> >> scenarios to the list and I will collect them into a web page.
> >>
> >> Best,
> >>     Shadi
> >>
> >> --
> >> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
> >> Activity Lead, W3C/WAI International Program Office
> >> Evaluation and Repair Tools Working Group (ERT WG)
> >> Research and Development Working Group (RDWG)
> >
> >
> > ---------------------------------------------------------------------
> > This transmission (including any attachments) may contain confidential
> information, privileged material (including material protected by the
> solicitor-client or other applicable privileges), or constitute non-public
> information. Any use of this information by anyone other than the intended
> recipient is prohibited. If you have received this transmission in error,
> please immediately reply to the sender and delete this information from
> your system. Use, dissemination, distribution, or reproduction of this
> transmission by unintended recipients is not authorized and may be
> unlawful.
> >
>
>
> --
> ---------------------------------------------------------------
> Detlev Fischer PhD
> DIAS GmbH - Daten, Informationssysteme und Analysen im Sozialen
> Geschäftsführung: Thomas Lilienthal, Michael Zapp
>
> Telefon: +49-40-43 18 75-25
> Mobile: +49-157 7-170 73 84
> Fax: +49-40-43 18 75-19
> E-Mail: fischer@dias.de
>
> Anschrift: Schulterblatt 36, D-20357 Hamburg
> Amtsgericht Hamburg HRB 58 167
> Geschäftsführer: Thomas Lilienthal, Michael Zapp
> ---------------------------------------------------------------


---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
Received on Thursday, 20 October 2011 20:19:19 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:12 GMT