W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > January 2013

RE: Combining evaluations

From: Katie Haritos-Shea EARTHLINK <ryladog@earthlink.net>
Date: Sun, 27 Jan 2013 11:28:38 -0500
To: "'Velleman, Eric'" <evelleman@bartimeus.nl>, "'David MacDonald'" <david100@sympatico.ca>, "'Roberto Scano \(IWA/HWG\)'" <r.scano@webprofession.com>, <public-wai-evaltf@w3.org>
Message-ID: <07b701cdfcab$5e4d64f0$1ae82ed0$@earthlink.net>

I agree with this approach you have outlined exactly. 

That is what I do in real world testing. 

I identify: 
* who performed the test (if possible, 2 testers using different configurations of technologies below),
* when I tested, 
* what I tested (pages, templates, overlays, widgets) on both pre-identified items and on NOT pre-identified random items, 
* what AT I used (which includes at least 1; screen reader, magnification tool, voice recognition tool), 
* what test tools and methods I used (many [keyboard, DOM inspection, CCA, auto scans, etc.], must be more than 1),
* what OSs I used (aim for more than 1), and
* (if applicable) what device(s) I used (desktop, laptop, tablet, etc.), 
* what browsers I used (must more than 1).

So the outcome is what the conformance results were, on that date, using these specific technologies. The tools/technologies are chosen from reliable stats on those highest in CURRENT use (at the time of the testing).

One should not assume that a test 2 years later would use ANY of the same items (or tester.)

My 3 cents.........:-)

* katie *

-----Original Message-----
From: Velleman, Eric [mailto:evelleman@bartimeus.nl] 
Sent: Sunday, January 27, 2013 9:08 AM
To: David MacDonald; 'Roberto Scano (IWA/HWG)'; public-wai-evaltf@w3.org
Subject: RE: Combining evaluations

Hi David,

Thanks for your reaction. I think you may have pointed to a possible solution, at least for now until we can do a test with what we have. The section you point to has a number of proposed accessibility support statement examples that we could use as inspiration that make this more flexible. 

We could still stick with uniform, but larger than just one tool, user-agent etc. What about if we leave this choice of tools and configurations to the evaluator (and to the evaluation commissioner ..who pays for it) but in return require that the evaluator(s) indicate 'what' he/she/they used and 'where' in the reporting and statement. 

We could use statement(s) like the ones in the link you provide: "This conformance claim meets the accessibility support requirement based on testing content in language(s) of the content with User Agents A, B, and C, and Assistive Technologies X, Y, and Z. This means that we were able to pass all of the success criteria for level A of WCAG 2.0 using these products."

This does have some downsides:
1. This could be misused...
2. It is more difficult to compare the data...
3. Evaluators could use anything to get a single website through an evaluation...

The positive thing is .... this flexible solution is already in the current text. The 'uniform..' as described now in the last editor draft does not limit you from using many user-agents, tools etc. We should add this to the report and statement sections to make it more clear.

Kindest regards,


Van: David MacDonald [david100@sympatico.ca]
Verzonden: zondag 27 januari 2013 13:55
Aan: 'Roberto Scano (IWA/HWG)'; Velleman, Eric; public-wai-evaltf@w3.org
Onderwerp: RE: Combining evaluations

> In WCAG it is not a problem if you are a person with a disability and you have to use multiple user-agents, assistive technology and other tools and configurations to use a single web page.

Since this is a public forum I think we need to correct the record on this above statement. Specifically "In WCAG it is not a problem..." is erroneous. A WCAG discussion of accessibility support in our understanding document is here. I think anyone considering this issue in the EVAL group needs to read this...

I personally don't think it is reasonable to expect a blind person to need two or three screen readers to access a page... on the other hand I don't think it is reasonable to expect companies to only use one person, or even one company, to test an entire site. And in WCAG we found defining an entire "web site" is very difficult, and this was one of our considerations on going to a url page based conformance model.

So my recommendation is to say on a page accessibility support NEEDS to be uniform. On a site (whatever that is), it is ENCOURAGED. However, others in WCAG might disagree with my thinking on single page uniformity... for instance, if the company is using a widget library... things get complicated because the author of the library might be from the page author or from an entirely different company, and might have designed it in an entirely different year... I do not have an answer for this, but I think we have to be realistic about the complexities here.

The big take away items from this WCAG Understanding document are:
-It is a complex issue
-More public discussion is needed
-this WCAG document t is a jumping off point for further public discussion "The Working Group encourages more discussion of this topic in the general forum of society since this lack of generally available yet robust assistive technologies is a problem that affects users, technology developers and authors negatively."

I think we have to be honest and say accessibility support is like the "wild west"... AT companies are small and sometimes differentiate themselves by handling certain problems on the web with different heuristics... it's not unlike the browser problems... except all browsers are free... A few sentences from the WCAG Understanding conformance document are pertinent here.

----start snip---
This topic raises the question of how many or which assistive technologies must support a Web technology in order for that Web technology to be considered "accessibility supported". The WCAG Working group and the W3C do not specify which or how many assistive technologies must support a Web technology in order for it to be classified as accessibility supported. This is a complex topic and one that varies both by environment and by language. There is a need for an external and international dialogue on this topic. Some notes to help in understanding and exploring this topic are:
Accessibility support of Web technologies varies by environment ◦ Web technologies may only need to be supported by those specific user agents and assistive technologies deployed at a company. (These may be older versions of user agents and assistive technologies or the very newest versions).
Content posted to the public Web may need to work with a broader range of user agents and assistive technologies, including older versions.

Accessibility support of Web technologies varies by language (and dialect) ◦ There are different levels of older assistive technologies support in different languages and even countries. Some environments or countries may provide free assistive technologies.

New technologies won't be supported in older assistive technologies ◦ Clearly, a new technology cannot be supported by all past assistive technologies, so requiring that a technology be supported by all assistive technologies is not possible.

Support for a single older assistive technology is usually not sufficient ◦ Support by just one assistive technology (for a given disability) would not usually be enough, especially if most users who need it in order to access content do not have and cannot afford that assistive technology. The exception here would be information distributed to company employees only where they all have one assistive technology (of that type).

Currently assistive technology that is affordable by the general public is often very poor ◦ Creating content that can't be used by the general public with disabilities should be avoided. In many cases, the cost of assistive technologies is too high for users who need it. Also, the capabilities of free or low cost AT is often so poor today that Web content cannot be realistically restricted to this lowest (or even middle) common denominator. This creates a very difficult dilemma that needs to be addressed.

The Working Group, therefore, limited itself to defining what constituted support and defers the judgment of how much, how many, or which AT must support a technology to the community and to entities closer to each situation that set requirements for an organization, purchase, community, etc.

The Working Group encourages more discussion of this topic in the general forum of society since this lack of generally available yet robust assistive technologies is a problem that affects users, technology developers and authors negatively.

----end snip

David MacDonald

CanAdapt Solutions Inc.
  Adapting the web to all users
            Including those with disabilities www.Can-Adapt.com

-----Original Message-----
From: Roberto Scano (IWA/HWG) [mailto:r.scano@webprofession.com]
Sent: January-27-13 2:21 AM
To: 'Velleman, Eric'; public-wai-evaltf@w3.org; david100@sympatico.ca
Subject: R: Combining evaluations

This is one of the problem of WCAG 2.0 that we have also noticed in italian government working group for accessibility. The definition of accessibility supported regarding different assistive technologies in a real big problem, especially if the conformance declaration isn't a voluntary declaration but a law requirement.
I think that evalutation could be do by different people but with the same "baseline" (same technologies for testing), otherwise I think is required that any evalutator must evalutate the same pages for conformance.

Roberto Scano
International Webmasters Association / The HTML Writers Guild http://www.iwanet.org

-----Messaggio originale-----
Da: Velleman, Eric [mailto:evelleman@bartimeus.nl]
Inviato: sabato 26 gennaio 2013 23:23
A: public-wai-evaltf@w3.org; david100@sympatico.ca
Oggetto: Combining evaluations

Dear EvalTF,

Although we mostly agree that "Accessibility support must be uniform throughout a single website", there are a few comments in the latest survey [1]. I would like to discuss this a bit more on the list, specifically a bit more about combining evaluations.

In WCAG it is not a problem if you are a person with a disability and you have to use multiple user-agents, assistive technology and other tools and configurations to use a single web page. We do not change this in WCAG-EM:
If you want to evaluate using multiple  versions of user agents, tools and screenreaders, that is ok. We only say that this should then be uniform throughout the single website evaluation. For the next single website, you can use another configuration, other tools, assistive technology etc.. This may be depending on the evaluation commissioner, the budget, time etc.

The question is: Can we combine two evaluations by two different evaluators (using different configurations for evaluation) of two different parts of a single website into one accessibility evaluation statement for the single website?

Both results of the evaluators are of course valid. but can they be combined if they use different configurations and are about different parts of a single website?

Kindest regards,


[1] <https://www.w3.org/2002/09/wbs/48225/evaltfq7/results#x2586>

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2013.0.2890 / Virus Database: 2639/6052 - Release Date: 01/23/13
Received on Sunday, 27 January 2013 16:29:09 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:16 GMT