W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > February 2012

AW: EvalTF discussion overview 100% conformance

From: Kerstin Probiesch <k.probiesch@googlemail.com>
Date: Thu, 2 Feb 2012 09:49:14 +0100
To: "'Velleman, Eric'" <evelleman@bartimeus.nl>, <public-wai-evaltf@w3.org>
Message-ID: <4f2a4def.d2260e0a.1b52.ffffc6f0@mx.google.com>
Hi all,

and thanks @Eric for this overview of our discussion.

My votes and thoughts

> --------
> Discussion on 100% Conformance
> --------
> # I think we agree on:
> 1. A sample only covers a small portion of a website and because we
> want to evaluate conformance to WCAG 2.0, the entire sample should be
> without failures of Success criteria. This means that any failure found
> leads to non-conformance of the website regardless of the impact or
> barrier.


> 2. The section on error margin can be deleted. This section was in fact
> about the margin to which extent and under what conditions, we would
> accept success criteria failures in the sample. This margin is now set
> to 0%.


> 3. We will add the requirement that the conformance claim should
> provide a "list of success criteria beyond the level of conformance
> claimed that have been met. 

I'm not sure about that. This would mean, that evaluation must include all
SCs (A, AA, AAA). Without evaluating every SC one could not provide such a

> This information should be provided in a
> form that users can use, preferably machine-readable metadata" (from:
> http://www.w3.org/TR/WCAG/#conformance-claims).


> 4. We should include the possibility for website owners to fix
> incidental errors without a totally new evaluation being necessary.


> 5. Leonie adds to the previous: My suggestion is that after the website
> owner has fixed the failed criteria, it isn't only the original sample
> of pages that is tested. Instead it's a combination of pages from the
> original sample and randomly selected new pages. Is that ok?

Yes. But I think that an evaluation shouldn't check just whole pages, but
also X tables, X forms, and so on in both cases: the first evaluation and
the re-evaluation. 

> # Discussion overview:
> Accepting no errors in the sample, indicates (Alistair:) "that for our
> purposes (those of evaluating the whole site) this would mean - making
> a level" A or "AA conformance claim for all pages in the sample
> (website).  If required by the website owner (or other) it could be
> provided along with a list of urls for the web pages on which the
> additional AAA Success Criteria have been achieved". This is conformant
> with how WCAG 2.0 looks at this from a webpage perspective.

Yes. As above: Wouldn't that mean that an evaluation always include AAA SCs?

> Within this whole website scope, it would be possible to claim
> conformance for A, AA or AAA for separate parts as long as the scope of
> that claim is clear and conforms with the section on conformance claim
> in WCAG2.0. 

Parts yes, but not single pages. I still think that users might be confused
upon different claims. But time will tell if is possible to explain it in
way which is clear for the users. Probably it's just a formal issue. Users
want no barriers and the only people who are interested in claims are
evaluators and website owners. 

> It is proposed that it should also be possible to exclude
> particular sections of a website from the scope. Examples for possible
> exclusion are: user generated content, wiki's, bulletin boards etc.
> although they are all not excluded in WCAG 2.0 .. 

Yes. I think all other ways will lead to the situation that no claim could
be made at all. What I'm thinking about is: what shall we do with websites
where the content is user generated in general like YouTube, Wikipedia,
social bookmarking services?

> Martijn proposes the
> possibility to divide the whole website into subevaluations that
> together form a whole website.

I think this needs a bit more discussion about the "how to".

> Sofar, I think we all agree more or less. But then it becomes less
> obvious:
> # Can a WCAG 2.0 website evaluation for which we are drafting this
> Methodology also address particular sections or techniques. We seem to
> agree on: yes.

Just under very strict conditions. I haven't understood "techniques" in this

> - For instance (Elle:) A large e-Commerce website with several
> segmented portals catering to different audiences - This company is
> trying to secure a government contract to sell a specific product suite
> to the state of New Jersey. In order to secure this contract, the
> company must show accessibility conformance for the specific product
> suite portal. Evaluating the complete suite is not relevant as they are
> not selling that. New Jersey could then add the conformance claim
> supplied by the company to their whole website conformance claim or
> just check this addition to their whole conformance claim. Is that ok?

I' still not sure about this. It sounds logical and there might be a need to
exclude all other segmented portals but I fear that accessibility then will
become a subject of economical interests of the website owner or company and
of contracts made.

If a company sells evaluation systems and provide a demo on a subdomain
incl. the possibility to buy it. I think this is ok. But where to draw the

> - But what about a bank that just wants an evaluation of a part of
> their website where you can buy insurances for people with
> disabilities. This does not include the homepage, the search engine,
> etc. You need a direct link to get there and any button you press on
> the menu takes you to parts of the website that do not have a
> conformance claim. Is that ok?

This is very tricky:

- a company sells Electronics & Computers. There is a special subsection or
subdomain with assistive technologies. The company wants to claim just this
subsection/subdomain or
- a book store sells printed books and audiobooks (one section of the page)
and audio books (DAISY) in another section. The company just claims
accessibility for the DAISY-Section (a real existing example). 

I think this is not ok.

> # Can the Methodology be used for any sample? Although the Methodology
> is for whole websites, there seems to be a tendency towards: yes. In
> that case, we need to be specific on the requirements to make a
> conformance claim on the basis of just any sample, technique or
> element. Elle writes: the concept of a "full website" is becoming less
> and less viable to companies in a component driven environment of
> content delivery. But any sample?

As mentioned above: I think the sample shouldn't only include pages but also

claiming techniques: I think this is not ok, because all techniques are
informative ("for information purposes and not required for conformance")
and "test procedures for individual techniques should not be taken as test
procedures for the WCAG 2.0 success criteria overall." 
> - For instance if a website owner already has a conformance claim for
> his whole website and adds a particular section to the website. He
> wants to evaluate that addition and then add it to his conformance
> claim. This is described above. Is that ok?

Yes. The question is wether there should be a shorter evaluation of the
already existing content, in cases where the first evaluation is some time

> Kerstin proposes to add requirements with regards to the time between
> the two evaluations, the path etc?
> - What if a website owner does an evaluation of the whole website and
> then for the conformance claim, he only chooses to select the pages in
> the sample that satisfy level AA conformance. Is that ok?


> - On a large shopping website only the shirts section is level A. The
> rest of the website is not accessible. The website owner wants to claim
> conformance for the shirts section using the WCAG2.0 Evaluation
> Methodology Is that ok?


> - And if the whole website is accessible except for the payment part at
> the end (that is a third party)? Can you claim conformance for level A?


> # Other interesting reading:
> http://www.w3.org/TR/2011/WD-ATAG20-20110721/#conf-prog-statement
> (thanks Tim)
> # Other discussions:
> - Detlev discussed that a recognised weakness of a seal is that users
> may be led to believe that it confirms the a11y of the *entire* site,
> something that is hard to achieve in practice and over time even if the
> sample tested was large. 
> This seems to be covered in the current sample
> section by using the error margin theory related to samples. In UWEM,
> the stakeholders agreed with statistical people that the sample could
> be representative. 

@Eric: I'm a bit confused about "could be representative". Would you give
more information about this issue.

> To avoid fragmentation we should take care that this
> Methodology can also be used by first, second and third party
> evaluators. This means that is can be used for 'labels' by all those
> parties. Is that ok?

If different evaluators are coming to the same results - even if they don't
belong to the same organization - I think there is no problem with using the
evaluation methodology for 'labels'.
> - There was a short discussion on: does the Methodology support first,
> second and third party evaluations. I would say yes. This would be
> important for harmonization. 

Yes. An important point.

> So it should ultimately also support
> existing labeling schemes inside companies and second and third party.
> Is that ok?

I don't know what is meant by this. If existing labeling schemes differ
(considerably) from our Methodology how can our Methodology supports
existing labeling schemes?

> - An important question in this discussion seems to be for whom we are
> making this Methodology. Is it for website owners? Is it cost-benefit
> driven? What is the role of people with disabilities when it comes to
> the conformance claims? Reducing the scope is a great way to reduce the
> cost. But at what cost? It seems important to know for whom the
> conformance claims are made?

Well. One could think that conformance claims are made for users. But I
think if user don't have to fight with barriers at all it is not important
if there is a conformance claim or not. Sometimes conformance claims are
important for other evaluators, or?! ;-). I think every accessibility
consultant has in the minimum one slide with examples for fantastic
conformance claims (A, AA or even AAA), which do not reflect reality at all.

So I think conformance claims are especially important for companies and the
public sector. Reducing the scope is a very understandable interest. But if
we follow those interests accessibility becomes a plaything of economical
considerations. The more one reduces the scope, the less mistakes will be
found and the more barriers users are encountering. At least of course for
the users but with some "roundabouts". 

> Wilco writes: "An important reason why there is a need for this
> methodology is so that, if applied to a website, gives an outside party
> an overview of how accessible that website is. This allows the national
> governments and other institutions interested in accessibility to
> collect information about the accessibility of the websites in their
> region. This information is needed to design an adequate response.


> If a
> website owner can define their own scope, they can skew the results and
> the outcomes of the evaluation become unusable for this kind of
> analysis."

Absolutely and it is likely that it is has also negative consequences for
users also.



> Kindest regards,
> Eric
Received on Thursday, 2 February 2012 08:49:17 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:40:20 UTC