RE: EvalTF discussion overview 100% conformance

My responses are below...
Sarah

 

On 1/31/2012 8:03 PM, Velleman, Eric wrote: 

Dear all,
 
Please find below an overview of the discussion we had in the previous week.
I hope this gives a good overview of things we agree on and the questions
that are open.
Please let me know if I missed something.
 
--------
Discussion on 100% Conformance
--------
 
# I think we agree on:
 
1. A sample only covers a small portion of a website and because we want to
evaluate conformance to WCAG 2.0, the entire sample should be without
failures of Success criteria. This means that any failure found leads to
non-conformance of the website regardless of the impact or barrier.

Yes



2. The section on error margin can be deleted. This section was in fact
about the margin to which extent and under what conditions, we would accept
success criteria failures in the sample. This margin is now set to 0%.

Yes



3. We will add the requirement that the conformance claim should provide a
"list of success criteria beyond the level of conformance claimed that have
been met. This information should be provided in a form that users can use,
preferably machine-readable metadata" (from:
http://www.w3.org/TR/WCAG/#conformance-claims).

Yes

 

4. We should include the possibility for website owners to fix incidental
errors without a totally new evaluation being necessary.

Yes



5. Leonie adds to the previous: My suggestion is that after the website
owner has fixed the failed criteria, it isn't only the original sample of
pages that is tested. Instead it's a combination of pages from the original
sample and randomly selected new pages. Is that ok?

Yes

 
# Discussion overview:
 
Accepting no errors in the sample, indicates (Alistair:) "that for our
purposes (those of evaluating the whole site) this would mean - making a
level" A or "AA conformance claim for all pages in the sample (website).  If
required by the website owner (or other) it could be provided along with a
list of urls for the web pages on which the additional AAA Success Criteria
have been achieved". This is conformant with how WCAG 2.0 looks at this from
a webpage perspective.
 
Within this whole website scope, it would be possible to claim conformance
for A, AA or AAA for separate parts as long as the scope of that claim is
clear and conforms with the section on conformance claim in WCAG2.0. It is
proposed that it should also be possible to exclude particular sections of a
website from the scope. Examples for possible exclusion are: user generated
content, wiki's, bulletin boards etc. although they are all not excluded in
WCAG 2.0 .. Martijn proposes the possibility to divide the whole website
into subevaluations that together form a whole website.
Yes
 
Sofar, I think we all agree more or less. But then it becomes less obvious:
 
# Can a WCAG 2.0 website evaluation for which we are drafting this
Methodology also address particular sections or techniques. We seem to agree
on: yes.

Yes



- For instance (Elle:) A large e-Commerce website with several segmented
portals catering to different audiences - This company is trying to secure a
government contract to sell a specific product suite to the state of New
Jersey. In order to secure this contract, the company must show
accessibility conformance for the specific product suite portal. Evaluating
the complete suite is not relevant as they are not selling that. New Jersey
could then add the conformance claim supplied by the company to their whole
website conformance claim or just check this addition to their whole
conformance claim. Is that ok?
 
- But what about a bank that just wants an evaluation of a part of their
website where you can buy insurances for people with disabilities. This does
not include the homepage, the search engine, etc. You need a direct link to
get there and any button you press on the menu takes you to parts of the
website that do not have a conformance claim. Is that ok?
 
# Can the Methodology be used for any sample? Although the Methodology is
for whole websites, there seems to be a tendency towards: yes. In that case,
we need to be specific on the requirements to make a conformance claim on
the basis of just any sample, technique or element. Elle writes: the concept
of a "full website" is becoming less and less viable to companies in a
component driven environment of content delivery. But any sample?
 
- For instance if a website owner already has a conformance claim for his
whole website and adds a particular section to the website. He wants to
evaluate that addition and then add it to his conformance claim. This is
described above. Is that ok?

Yes



Kerstin proposes to add requirements with regards to the time between the
two evaluations, the path etc?
 
- What if a website owner does an evaluation of the whole website and then
for the conformance claim, he only chooses to select the pages in the sample
that satisfy level AA conformance. Is that ok?

I'm not sure that this would be okay.

 
- On a large shopping website only the shirts section is level A. The rest
of the website is not accessible. The website owner wants to claim
conformance for the shirts section using the WCAG2.0 Evaluation Methodology
Is that ok?

See above.



 
- And if the whole website is accessible except for the payment part at the
end (that is a third party)? Can you claim conformance for level A?

No, because the payment part at the end is essential for completing the
purchasing task.

 
 
# Other interesting reading:
 
http://www.w3.org/TR/2011/WD-ATAG20-20110721/#conf-prog-statement (thanks
Tim)
 
 
# Other discussions:
 
- Detlev discussed that a recognised weakness of a seal is that users may be
led to believe that it confirms the a11y of the *entire* site, something
that is hard to achieve in practice and over time even if the sample tested
was large. This seems to be covered in the current sample section by using
the error margin theory related to samples. In UWEM, the stakeholders agreed
with statistical people that the sample could be representative. To avoid
fragmentation we should take care that this Methodology can also be used by
first, second and third party evaluators. This means that is can be used for
'labels' by all those parties. Is that ok?

If a seal is offered, it would need to be for the whole website. Users may
not notice that the seal is appearing and disappearing as they move around
in a website, but they will assume that the seal applies to the whole site.

 
- There was a short discussion on: does the Methodology support first,
second and third party evaluations. I would say yes. This would be important
for harmonization. So it should ultimately also support existing labeling
schemes inside companies and second and third party. Is that ok?

I'm not sure what this means.



 
- An important question in this discussion seems to be for whom we are
making this Methodology. Is it for website owners? Is it cost-benefit
driven? What is the role of people with disabilities when it comes to the
conformance claims? Reducing the scope is a great way to reduce the cost.
But at what cost? It seems important to know for whom the conformance claims
are made?

I assumed that we are creating this Methodology for website owners. The
conformance claims made by website owners, then, communicate their website's
level of accessibility to persons with disabilities. If we are not setting
up the Methodology specifically to benefit persons with disabilities, it's
not going to have any real impact on website design - and website usability
for persons with disabilities. 



 
Wilco writes: "An important reason why there is a need for this methodology
is so that, if applied to a website, gives an outside party an overview of
how accessible that website is. This allows the national governments and
other institutions interested in accessibility to collect information about
the accessibility of the websites in their region. This information is
needed to design an adequate response. If a website owner can define their
own scope, they can skew the results and the outcomes of the evaluation
become unusable for this kind of analysis."

I agree with Wilco on his assessment.



 
Kindest regards,
 
Eric