Re: EvalTF discussion overview 100% conformance

-----Original Message----- 
From: Velleman, Eric
Sent: Wednesday, February 01, 2012 1:03 AM
To: public-wai-evaltf@w3.org
Subject: EvalTF discussion overview 100% conformance

Dear all,

Please find below an overview of the discussion we had in the previous week. 
I hope this gives a good overview of things we agree on and the questions 
that are open.
Please let me know if I missed something.

--------
Discussion on 100% Conformance
--------

# I think we agree on:

1. A sample only covers a small portion of a website and because we want to 
evaluate conformance to WCAG 2.0, the entire sample should be without 
failures of Success criteria. This means that any failure found leads to 
non-conformance of the website regardless of the impact or barrier.
2. The section on error margin can be deleted. This section was in fact 
about the margin to which extent and under what conditions, we would accept 
success criteria failures in the sample. This margin is now set to 0%.
3. We will add the requirement that the conformance claim should provide a 
“list of success criteria beyond the level of conformance claimed that have 
been met. This information should be provided in a form that users can use, 
preferably machine-readable metadata” (from: 
http://www.w3.org/TR/WCAG/#conformance-claims).
4. We should include the possibility for website owners to fix incidental 
errors without a totally new evaluation being necessary.
5. Leonie adds to the previous: My suggestion is that after the website 
owner has fixed the failed criteria, it isn't only the original sample of 
pages that is tested. Instead it's a combination of pages from the original 
sample and randomly selected new pages. Is that ok?

>>>> Richard - Yes ( 1,2 and 3 are fine, we should combine 4 and 5 to 
>>>> provide a procedure for "re-evaluation".)

# Discussion overview:

Accepting no errors in the sample, indicates (Alistair:) “that for our 
purposes (those of evaluating the whole site) this would mean - making a 
level” A or “AA conformance claim for all pages in the sample (website).  If 
required by the website owner (or other) it could be provided along with a 
list of urls for the web pages on which the additional AAA Success Criteria 
have been achieved“. This is conformant with how WCAG 2.0 looks at this from 
a webpage perspective.

Within this whole website scope, it would be possible to claim conformance 
for A, AA or AAA for separate parts as long as the scope of that claim is 
clear and conforms with the section on conformance claim in WCAG2.0. It is 
proposed that it should also be possible to exclude particular sections of a 
website from the scope. Examples for possible exclusion are: user generated 
content, wiki’s, bulletin boards etc. although they are all not excluded in 
WCAG 2.0 .. Martijn proposes the possibility to divide the whole website 
into subevaluations that together form a whole website.


Sofar, I think we all agree more or less. But then it becomes less obvious:

# Can a WCAG 2.0 website evaluation for which we are drafting this 
Methodology also address particular sections or techniques. We seem to agree 
on: yes.

>>>> Richard - Yes

- For instance (Elle:) A large e-Commerce website with several segmented 
portals catering to different audiences - This company is trying to secure a 
government contract to sell a specific product suite to the state of New 
Jersey. In order to secure this contract, the company must show 
accessibility conformance for the specific product suite portal. Evaluating 
the complete suite is not relevant as they are not selling that. New Jersey 
could then add the conformance claim supplied by the company to their whole 
website conformance claim or just check this addition to their whole 
conformance claim. Is that ok?

- But what about a bank that just wants an evaluation of a part of their 
website where you can buy insurances for people with disabilities. This does 
not include the homepage, the search engine, etc. You need a direct link to 
get there and any button you press on the menu takes you to parts of the 
website that do not have a conformance claim. Is that ok?

# Can the Methodology be used for any sample? Although the Methodology is 
for whole websites, there seems to be a tendency towards: yes. In that case, 
we need to be specific on the requirements to make a conformance claim on 
the basis of just any sample, technique or element. Elle writes: the concept 
of a "full website" is becoming less and less viable to companies in a 
component driven environment of content delivery. But any sample?

- For instance if a website owner already has a conformance claim for his 
whole website and adds a particular section to the website. He wants to 
evaluate that addition and then add it to his conformance claim. This is 
described above. Is that ok?
Kerstin proposes to add requirements with regards to the time between the 
two evaluations, the path etc?

- What if a website owner does an evaluation of the whole website and then 
for the conformance claim, he only chooses to select the pages in the sample 
that satisfy level AA conformance. Is that ok?

- On a large shopping website only the shirts section is level A. The rest 
of the website is not accessible. The website owner wants to claim 
conformance for the shirts section using the WCAG2.0 Evaluation Methodology 
Is that ok?

- And if the whole website is accessible except for the payment part at the 
end (that is a third party)? Can you claim conformance for level A?

>>>> Richard - Yes (On the basis that it is not our job to limit or restrict 
>>>> the use of our methodology - the important thing is to enable clear, 
>>>> accurate and reliable scope and conformance statements)



# Other interesting reading:

http://www.w3.org/TR/2011/WD-ATAG20-20110721/#conf-prog-statement (thanks 
Tim)


# Other discussions:

- Detlev discussed that a recognised weakness of a seal is that users may be 
led to believe that it confirms the a11y of the *entire* site, something 
that is hard to achieve in practice and over time even if the sample tested 
was large. This seems to be covered in the current sample section by using 
the error margin theory related to samples. In UWEM, the stakeholders agreed 
with statistical people that the sample could be representative. To avoid 
fragmentation we should take care that this Methodology can also be used by 
first, second and third party evaluators. This means that is can be used for 
‘labels’ by all those parties. Is that ok?

>>>> Richard -  Yes so long as the sampling includes selected as well as 
>>>> random procedures.

- There was a short discussion on: does the Methodology support first, 
second and third party evaluations. I would say yes. This would be important 
for harmonization. So it should ultimately also support existing labeling 
schemes inside companies and second and third party. Is that ok?

>>>> Richard - Yes

- An important question in this discussion seems to be for whom we are 
making this Methodology. Is it for website owners? Is it cost-benefit 
driven? What is the role of people with disabilities when it comes to the 
conformance claims? Reducing the scope is a great way to reduce the cost. 
But at what cost? It seems important to know for whom the conformance claims 
are made?

>>>> Richard - This question was already answered last year (1.2 - Target 
>>>> Audience)

Wilco writes: “An important reason why there is a need for this methodology 
is so that, if applied to a website, gives an outside party an overview of 
how accessible that website is. This allows the national governments and 
other institutions interested in accessibility to collect information about 
the accessibility of the websites in their region. This information is 
needed to design an adequate response. If a website owner can define their 
own scope, they can skew the results and the outcomes of the evaluation 
become unusable for this kind of analysis.”

>>>> Richard  No - sorry - The website owner (or anyone else)  should be 
>>>> able to define the scope. As long as the scope is defined to our 
>>>> standard (complete process etc - Section 3) and the conformance claim 
>>>> matches!

Kindest regards,

Eric

Received on Wednesday, 1 February 2012 13:26:25 UTC