RE: EvalTF discussion overview 100% conformance

My comments are below:

-----Original Message-----
From: Detlev Fischer [mailto:fischer@dias.de] 
Sent: Thursday, February 02, 2012 6:10 AM
To: public-wai-evaltf@w3.org
Subject: Re: EvalTF discussion overview 100% conformance

Hi all,

BTW I can't take part in today's teleconference - we have a conference
running here.

 > --------
 > Discussion on 100% Conformance
 > --------
 >
 > # I think we agree on:
 >
 > 1. A sample only covers a small portion of a website and because we  >
want to evaluate conformance to WCAG 2.0, the entire sample should be  >
without failures of Success criteria. This means that any failure found  >
leads to non-conformance of the website regardless of the impact or  >
barrier.

Detlev: "Yes" seems to be the majority view and I accept that. I have said
more than once that I believe few sites tested will ever achieve 100%
compliance even on level A.
One solution is for evaluation schemes to calculate a "percent of
conformance achieved" value independent of the (probably common) statement
of non-conformance. That way, site owners get some measure for how far they
are off the mark.

KW:  Yes

 > 2. The section on error margin can be deleted. This section was in fact
> about the margin to which extent and under what conditions, we would  >
accept success criteria failures in the sample. This margin is now set  > to
0%.

Detlev: Yes. I think, like Kerstin, that error margin must not be confused
with failure rate. Regarding the 0%, see my comment above.

KW:  Yes

 > 3. We will add the requirement that the conformance claim should  >
provide a "list of success criteria beyond the level of conformance  >
claimed that have been met.

Detlev: Yes. That list should not be seen as exhaustive to prevent haviong
to check conformance levels above the level chosen. But if, say, the testing
of text contrast shows that it is uniformly 7:1 or better, one would know
that it is possible to state "meets AAA-SC 1.4.6"

KW:  Yes.  I think that is a good.  

 > This information should be provided in a  > form that users can use,
preferably machine-readable metadata" (from:
 > http://www.w3.org/TR/WCAG/#conformance-claims).

Detlev: Yes. I agree this could be explained though so that people are not
frightened off by the "machine -readable" bit.

 > 4. We should include the possibility for website owners to fix  >
incidental errors without a totally new evaluation being necessary.

Detlev: Yes. It needs careful thought however what must be done to verify
that incidental errors were remedied. Not surte whether the evaluation
methodology can go to that level of detail..

KW:  Yes, but we need to have them report all changes made as other changes
could have been introduced.

 > 5. Leonie adds to the previous: My suggestion is that after the website
> owner has fixed the failed criteria, it isn't only the original sample  >
of pages that is tested. Instead it's a combination of pages from the  >
original sample and randomly selected new pages. Is that ok?

Detlev: Yes this makes perfect sense.

KW:  Yes, that is a good idea 

 > # Discussion overview:
 >
 > Accepting no errors in the sample, indicates (Alistair:) "that for our  >
purposes (those of evaluating the whole site) this would mean - making  > a
level" A or "AA conformance claim for all pages in the sample  > (website).
If required by the website owner (or other) it could be  > provided along
with a list of urls for the web pages on which the  > additional AAA Success
Criteria have been achieved". This is conformant  > with how WCAG 2.0 looks
at this from a webpage perspective.

Detlev: Yes.

KW:  Yes

 > Within this whole website scope, it would be possible to claim  >
conformance for A AA or AAA for separate parts as long as the scope of  >
that claim is clear and conforms with the section on conformance claim  > >
in WCAG2.0.

Detlev: Yes. For web applications running on just one page, this could even
mean that just one page has a separate conformance claim.

KW:  Yes

 > It is proposed that it should also be possible to exclude
 > particular sections of a website from the scope. Examples for possible
 > exclusion are: user generated content, wiki's, bulletin boards etc.
 > although they are all not excluded in WCAG 2.0 ..

Detlev: Yes. If user-generated content is central to the function of the 
site, these sections should ideally not be excluded, but I believe 
whether such exclusion is possible or not would be established on the 
level of the evaluation scheme, not the methodology. So if a Dutch or 
German evaluation scheme decides that some parts cannot be excluded, 
that's fine. Site owners may still stake their own conformance claim 
excluding those parts, but they won't get the label/seal/certificate of 
that particular scheme.

KW:  Yes.  This is particularly important when they are integrating with
tools that have no alternative and the company has no control over the code.
For example the Facebook Like button.

 > Martijn proposes the
 > possibility to divide the whole website into subevaluations that
 > together form a whole website.

Detlev: Yes.

Kathy:  Yes.  This gets back to my point about user stories and task
evaluations.

 > Sofar, I think we all agree more or less. But then it becomes less
 > obvious:
 >
 > # Can a WCAG 2.0 website evaluation for which we are drafting this
 > Methodology also address particular sections or techniques. We seem to
 > agree on: yes.

Detlev: Yes. I guess with "techniques" you mean basic choices like HTML 
/ Silverlight / Flash?

KW:  Yes


 > - For instance (Elle:) A large e-Commerce website with several
 > segmented portals catering to different audiences - This company is
 > trying to secure a government contract to sell a specific product suite
 > to the state of New Jersey. In order to secure this contract, the
 > company must show accessibility conformance for the specific product
 > suite portal. Evaluating the complete suite is not relevant as they are
 > not selling that. New Jersey could then add the conformance claim
 > supplied by the company to their whole website conformance claim or
 > just check this addition to their whole conformance claim. Is that ok?

Detlev: Yes. Whether an evaluation scheme would go along with that might 
better be a decision outside of the methodology.

KW:  Yes

 > - But what about a bank that just wants an evaluation of a part of
 > their website where you can buy insurances for people with
 > disabilities. This does not include the homepage, the search engine,
 > etc. You need a direct link to get there and any button you press on
 > the menu takes you to parts of the website that do not have a
 > conformance claim. Is that ok?

Detlev: Not sure. This needs more discussion.

KW:  If we are agreeing above that we can exclude parts of the website then
this situation would be allowed.  I think we need to be careful with this
one

 > # Can the Methodology be used for any sample? Although the Methodology
 > is for whole websites, there seems to be a tendency towards: yes. In
 > that case, we need to be specific on the requirements to make a
 > conformance claim on the basis of just any sample, technique or
 > element. Elle writes: the concept of a "full website" is becoming less
 > and less viable to companies in a component driven environment of
 > content delivery. But any sample?

Detlev: See above. OK in principle, but providers of evaluation schemes 
may refuse to test / certify. Site owners are still free to stake their 
claim.

KW:  I think the critical task paths and pages need to be included in the
evaluation

 > - For instance if a website owner already has a conformance claim for
 > his whole website and adds a particular section to the website. He
 > wants to evaluate that addition and then add it to his conformance
 > claim. This is described above. Is that ok?

Detlev: Yes. It is tricky though due to the time element - by that time 
the other stuff may no longer meet the chosen level of conformance.

KW: Yes

 > Kerstin proposes to add requirements with regards to the time between
 > the two evaluations, the path etc?
 >
 > - What if a website owner does an evaluation of the whole website and
 > then for the conformance claim, he only chooses to select the pages in
 > > the sample that satisfy level AA conformance. Is that ok?

Detlev: No.

KW:  No

 > - On a large shopping website only the shirts section is level A. The
 > rest of the website is not accessible. The website owner wants to claim
 > conformance for the shirts section using the WCAG2.0 Evaluation
 > Methodology Is that ok?

Detlev: In principle he can do so but the limits of the claim it must be 
clear. Evaluation scheme providers may still refuse to test such a site.

KW:  Yes.

 > - And if the whole website is accessible except for the payment part at
 > the end (that is a third party)? Can you claim conformance for level A?

Detlev: this would be ruled out by "complete processes" but it is tricky 
and needs more discussion.

KW:  No, this is part of a critical task


 > # Other interesting reading:
 >
 > http://www.w3.org/TR/2011/WD-ATAG20-20110721/#conf-prog-statement
 > (thanks Tim)
 >
 >
 > # Other discussions:
 >
 > - Detlev discussed that a recognised weakness of a seal is that users
 > may be led to believe that it confirms the a11y of the *entire* site,
 > something that is hard to achieve in practice and over time even if the
 > sample tested was large.
 > This seems to be covered in the current sample
 > section by using the error margin theory related to samples. In UWEM,
 > the stakeholders agreed with statistical people that the sample could
 > be representative.

Detlev: see my first comment. I still believe the "strict" approach is a 
bit unrealistic.

KW:  I agree.  The seal should link to the conformance claim document.

 > To avoid fragmentation we should take care that this
 > Methodology can also be used by first, second and third party
 > evaluators. This means that is can be used for 'labels' by all those
 > parties. Is that ok?

Detlev: yes.

KW:  Yes

 > - There was a short discussion on: does the Methodology support first,
 > second and third party evaluations. I would say yes. This would be
 > important for harmonization.

Detlev: yes.

KW:  yes

 > So it should ultimately also support
 > existing labeling schemes inside companies and second and third party.
 > Is that ok?

Deltev: To the extent that there is a clear mapping, yes. Some schemes 
may go beyond WCAG (e.g. include usability aspects), then the 
distinction should be clear what maps onto the WCAG evaluation 
methodology and what goes beyond.

KW:  Yes

 > - An important question in this discussion seems to be for whom we are
 > making this Methodology. Is it for website owners? Is it cost-benefit
 > driven? What is the role of people with disabilities when it comes to
 > the conformance claims? Reducing the scope is a great way to reduce the
 > cost. But at what cost? It seems important to know for whom the
 > conformance claims are made?

Detlev: not quite sure what you are getting at. There is always a 
tradeoff between cost and quality of results.

KW: more discussion is needed on this.  More testing is always better.  If
you test with people with disabilities you will always find things that can
be improved.  

 > Wilco writes: "An important reason why there is a need for this
 > methodology is so that, if applied to a website, gives an outside party
 > an overview of how accessible that website is. This allows the national
 > governments and other institutions interested in accessibility to
 > collect information about the accessibility of the websites in their
 > region. This information is needed to design an adequate response.

Detlev: The grand EU benchmarking schemes have so far only compared the 
automated part of evaluations? But benchmarking will be easier if there 
is a common approach, true.

KW:  yes

 > If a
 > website owner can define their own scope, they can skew the results and
 > the outcomes of the evaluation become unusable for this kind of
 > analysis."

Detlev: Yes - that's why it is important to discuss how rthe claim looks 
like to the user of the site, if it is clear what it applies to. We will 
not be able to prevent all sorts of claims and fantasy seals, however.

KW:  I agree with Detlev.  I think this seal should always link to the
conformance documentation that clearly specifies  what it applies to.

Received on Thursday, 2 February 2012 13:21:44 UTC