Re: EvalTF discussion overview 100% conformance

See responses below.

On 1/31/2012 8:03 PM, Velleman, Eric wrote:
> Dear all,
>
> Please find below an overview of the discussion we had in the previous week. I hope this gives a good overview of things we agree on and the questions that are open.
> Please let me know if I missed something.
>
> --------
> Discussion on 100% Conformance
> --------
>
> # I think we agree on:
>
> 1. A sample only covers a small portion of a website and because we want to evaluate conformance to WCAG 2.0, the entire sample should be without failures of Success criteria. This means that any failure found leads to non-conformance of the website regardless of the impact or barrier.
Yes.
> 2. The section on error margin can be deleted. This section was in fact about the margin to which extent and under what conditions, we would accept success criteria failures in the sample. This margin is now set to 0%.
Yes.
> 3. We will add the requirement that the conformance claim should provide a “list of success criteria beyond the level of conformance claimed that have been met. This information should be provided in a form that users can use, preferably machine-readable metadata” (from: http://www.w3.org/TR/WCAG/#conformance-claims).
Yes. It would be helpful to provide an example of "machine-readable 
metadata" so people will know what that is and how to do it.
> 4. We should include the possibility for website owners to fix incidental errors without a totally new evaluation being necessary.
Yes. We should also give examples of incidental errors.
> 5. Leonie adds to the previous: My suggestion is that after the website owner has fixed the failed criteria, it isn't only the original sample of pages that is tested. Instead it's a combination of pages from the original sample and randomly selected new pages. Is that ok?
Yes. Trust, but verify.  :^)
>
> # Discussion overview:
>
> Accepting no errors in the sample, indicates (Alistair:) “that for our purposes (those of evaluating the whole site) this would mean - making a level” A or “AA conformance claim for all pages in the sample (website).  If required by the website owner (or other) it could be provided along with a list of urls for the web pages on which the additional AAA Success Criteria have been achieved“. This is conformant with how WCAG 2.0 looks at this from a webpage perspective.
>
> Within this whole website scope, it would be possible to claim conformance for A, AA or AAA for separate parts as long as the scope of that claim is clear and conforms with the section on conformance claim in WCAG2.0. It is proposed that it should also be possible to exclude particular sections of a website from the scope. Examples for possible exclusion are: user generated content, wiki’s, bulletin boards etc. although they are all not excluded in WCAG 2.0 .. Martijn proposes the possibility to divide the whole website into subevaluations that together form a whole website.
Yes. I would suggest that we provide a set of conformance levels for 
people to use, along with the date of the most recent evaluation:

_100% Compliant, Entire Website--WCAG 2.0 AA_: As of February 2, 2012, 
all pages in the website have been tested and found to conform with WCAG 
2.0 AA checkpoints.

_100% Compliant, Sample of Entire Website--WCAG 2.0 AA_: As of February 
2, 2012, a representative sample of "x" pages has been tested and found 
to be compliant with WCAG 2.0 AA checkpoints. These pages were tested: 
......

_100% Compliant, Section of Website--WCAG 2.0 AA_: As of February 2, 
2012, all pages of the "y" section of the website have been tested and 
found to be compliant with WCAG 2.0 AA checkpoints. The following pages 
were tested: ......
_
100% Compliant, Sample of Website Section--WCAG 2.0 AA_: As of February 
2, 2012, a representative sample of the "y" section of the website has 
been tested and found to be compliant with WCAG 2.0 AA checkpoints. The 
following pages were tested: ......

_Non-compliant, Entire Website--WCAG 2.0 AA_: As of February 2, 2012, 
all pages in the website have been tested and the following pages were 
found not to conform with WCAG 2.0 AA checkpoints:

Additional information about the types of failures could be provided in 
a variety of ways, depending on whether this was a report, a conformance 
claim on a website, etc.

_Non-compliant, Sample_...etc.
>
> Sofar, I think we all agree more or less. But then it becomes less obvious:
>
> # Can a WCAG 2.0 website evaluation for which we are drafting this Methodology also address particular sections or techniques. We seem to agree on: yes.
Yes. See previous comment.
> - For instance (Elle:) A large e-Commerce website with several segmented portals catering to different audiences - This company is trying to secure a government contract to sell a specific product suite to the state of New Jersey. In order to secure this contract, the company must show accessibility conformance for the specific product suite portal. Evaluating the complete suite is not relevant as they are not selling that. New Jersey could then add the conformance claim supplied by the company to their whole website conformance claim or just check this addition to their whole conformance claim. Is that ok?
>
> - But what about a bank that just wants an evaluation of a part of their website where you can buy insurances for people with disabilities. This does not include the homepage, the search engine, etc. You need a direct link to get there and any button you press on the menu takes you to parts of the website that do not have a conformance claim. Is that ok?
>
> # Can the Methodology be used for any sample? Although the Methodology is for whole websites, there seems to be a tendency towards: yes. In that case, we need to be specific on the requirements to make a conformance claim on the basis of just any sample, technique or element. Elle writes: the concept of a "full website" is becoming less and less viable to companies in a component driven environment of content delivery. But any sample?
>
> - For instance if a website owner already has a conformance claim for his whole website and adds a particular section to the website. He wants to evaluate that addition and then add it to his conformance claim. This is described above. Is that ok?
Yes, if he can be certain that previously tested pages have not changed. 
Otherwise, he will have to retest a sample of the original site.
> Kerstin proposes to add requirements with regards to the time between the two evaluations, the path etc?
>
> - What if a website owner does an evaluation of the whole website and then for the conformance claim, he only chooses to select the pages in the sample that satisfy level AA conformance. Is that ok?
In most cases claims will be made for the website as a whole. 
Indications of compliance for sections of a website will probably only 
be made when someone is doing an analysis of a new or revised section. 
Nonetheless, we can prevent misleading claims by requiring that a when a 
public claim for a portion of the website is made it must be accompanied 
by a statement for the website as a whole.
> - On a large shopping website only the shirts section is level A. The rest of the website is not accessible. The website owner wants to claim conformance for the shirts section using the WCAG2.0 Evaluation Methodology Is that ok?
See above.
> - And if the whole website is accessible except for the payment part at the end (that is a third party)? Can you claim conformance for level A?
No. You can claim that sections are in conformance, but must also state 
that the website as a whole is not.
>
> # Other interesting reading:
>
> http://www.w3.org/TR/2011/WD-ATAG20-20110721/#conf-prog-statement (thanks Tim)
>
>
> # Other discussions:
>
> - Detlev discussed that a recognised weakness of a seal is that users may be led to believe that it confirms the a11y of the *entire* site, something that is hard to achieve in practice and over time even if the sample tested was large. This seems to be covered in the current sample section by using the error margin theory related to samples. In UWEM, the stakeholders agreed with statistical people that the sample could be representative. To avoid fragmentation we should take care that this Methodology can also be used by first, second and third party evaluators. This means that is can be used for ‘labels’ by all those parties. Is that ok?
I'm not sure what you mean by "first, second and third party evaluators."
> - There was a short discussion on: does the Methodology support first, second and third party evaluations. I would say yes. This would be important for harmonization. So it should ultimately also support existing labeling schemes inside companies and second and third party. Is that ok?
?
> - An important question in this discussion seems to be for whom we are making this Methodology. Is it for website owners? Is it cost-benefit driven? What is the role of people with disabilities when it comes to the conformance claims? Reducing the scope is a great way to reduce the cost. But at what cost? It seems important to know for whom the conformance claims are made?
I'm not it matters if we ensure that the claims are easy to understand 
and contain sufficient detail. Again, it seems to me that websites will 
be unlikely to make "partial" or "non-conformance" claims. I would guess 
that it will only show up in evaluation reports.
> Wilco writes: “An important reason why there is a need for this methodology is so that, if applied to a website, gives an outside party an overview of how accessible that website is. This allows the national governments and other institutions interested in accessibility to collect information about the accessibility of the websites in their region. This information is needed to design an adequate response. If a website owner can define their own scope, they can skew the results and the outcomes of the evaluation become unusable for this kind of analysis.”
Agree.
> Kindest regards,
>
> Eric
>
>
>