RE: Concerns about not covering full website

Hi Alistair, TF,

I think this approach will be getting a little bit too flexible. As I understand, all the work we have done until now is in order to evaluate whole websites. We want "a standardized way to evaluate the conformance of websites to WCAG 2.0." (quoted from the introduction of our draft methodology), the whole draft document focuses on  'websites' instead of web pages. I can understand from an owners point of view that he/she sometimes will want to split up a whole website into several subevaluations, but these subevaluations, in my opinion, will together comprise the entire website, and not leave out some part for any reason.

Some problems arise if we do follow this strategy:

If we only check what the owner of a website wants us to check, this creates the risk that an owner will only have the conforming parts of the website evaluated. An owner of a inaccessible website can then, with a little smart wording, proudly claim conformance (of f.e. a little subsection). An experienced user can differentiate between strong and weak claims, but a normal user can't. This in turn creates two potential pitfalls: lots of 'empty' claims that in practice do not help users, f.e.: "Our homepage conforms to WCAG 2.0", leaving out that none of the underlying pages conform.  Secondly the degradation of those claims that are truly useful to an user "Page X claims to conform (using an 'empty' claim) but I can't use it, so I probably also can't use page Y (that conforms and has a useful claim). Confusion as to what an evaluated conformance claim means, need to be avoided.

Secondly, this will create the possibility to exclude key scenario's and complete paths. Again from the introduction: "The Methodology defines manual and semi-automated methods for selecting representative samples of web pages from websites that include complete processes." I feel we are giving people who want to evaluate conformity to WCAG a guide on how to do a proper, representative evaluation of a website. If we only check pages supplied by a website owner we can not always include key scenario's or complete paths. I think we do not want a web store claiming conformance, whilst it check-out procedure is inaccessible. Or YouTube claiming conformance except for the pages that have video's. To form a statement about conformance, I feel we have to be able to independently evaluate the whole website including whatever it may contain. The resulting conformance statement, if made, will then be of practical use for people with disabilities, as they can trust a conformance claim to be of use for them.

Kind regards,

Martijn Houtepen


________________________________
Van: Alistair Garrison [mailto:alistair.j.garrison@gmail.com]
Verzonden: vrijdag 27 januari 2012 10:31
Aan: Eval TF
CC: Eric Velleman; Detlev Fischer
Onderwerp: Re: Concerns about not covering full website

Dear all,

Due to its many perceived benefits, I have long been a passionate advocate for using a website owner's WCAG 2.0 Conformance Claim as the scope of our evaluation - writing several emails in the past months about this very subject*.  The following summarises my thoughts to date...

I totally support the pros stated by both Detlev and Eric, emphasising that:

"This approach is very flexible and would make it possible for evaluators to verify a WCAG 2.0 claim conformance for more stable websites or fast growing websites, or portions of websites, sections of a website or even single pages in a website - using sampling methods for large websites as per our discussions."

In addition, I strongly believe that:

1) It makes evaluation more reproducible, and more efficient, as we know exactly what forms the scope of our evaluation (i.e. what is defined in the WCAG 2.0 Conformance claim at the time of evaluation, from which samples can be taken as necessary);

2) Once an initial evaluation had been done (scope WCAG 2.0 Conformance claim) we would only ever need to assess new additions to the WCAG 2.0 conformance claim on an on-going basis - hopefully saving people time and money;

3) It tells the public exactly what conforms in the website - a bit more like the VPAT.  If the WCAG 2.0 conformance claim states, for example, that a website's shopping process (defined by urls) conforms to x and y,  a user should know it will be accessible to them;

4) If the website contains any complete processes which back-ends into third party pages - we are in fact prevented from just saying "the whole website" conforms.  A WCAG 2.0 Conformance Claim, on the other hand, could be made for this website, and evaluated, if we choose to use WCAG 2.0 conformance claims as our scope.

5) It supports, even promotes and encourages, detailed WCAG 2.0 conformance claims to be made by the people responsible for entire websites / parts of websites.

It would also mean that the majority of Section 3 could be re-defined simply e.g. "The scope of the evaluation is defined as all urls for which a WCAG 2.0 conformance claim is being made, at the time of the evaluation".

* Referenced emails from Nov / Dec 2011

http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Dec/0004.html
http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Nov/0073.html
<http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Nov/0073.html>http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Nov/0086.html
http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Dec/0023.html

All the best

Alistair
<http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Nov/0073.html>
On 27 Jan 2012, at 01:01, Velleman, Eric wrote:


Hi Detlev, all,

Yes, it was an interesting discussion this afternoon about the scope of an evaluation :-).

This afternoon we discussed the possibility to use the WCAG2.0 Evaluation Methodology not only for full website evaluation, but also for parts of websites. In the current version the evaluation focuses on the full website with a possibility to exclude parts from the scope.  If we want to include evaluating parts of a website (like only the WAI part of W3C or only the BAD website) this should be added in the scope section.

Today a different approach was proposed: We look at it the other way around: the Conformance claim determines the scope.

This approach is very flexible and makes it possible to claim conformance for portions of websites, possibly even for technologies or collections of pages or single pages. This makes the Methodology much more flexible for website owners who are not always interested in evaluation of the full website. This is a change to the requirements document but it will provide support for requests by website owners and evaluators wishing to just look at a specific part of a website.

I would propose that we do a short discussion this week on this approach. Do we leave the scope completely free for the site owner to decide?

Kindest regards,

Eric



________________________________________
Van: Detlev Fischer [fischer@dias.de]
Verzonden: donderdag 26 januari 2012 17:16
Aan: EVAL TF
Onderwerp: Concerns about not covering full website

Hi everyone,

I think a lot of the heat in the disacussion we just had may be down to
a misunderstanding.

Eric, no one says that the evaluation of an entire webiste is not a good
idea or should not be done. If a site owner wants it done, that's fine.

If however, a site owner wants to look at a particular section (that can
be many pages), it should be possible to evaluate just that. It's also a
cost issue. Any conformance claim must make that limited scope perfectly
clear, preferably by binding the evaluation to a set of URLs.

Increasingly, sites have many different sections, like user generated
content. No sane person would claim even A-Level conformance for a wiki
or bulletin board that allows unstructured text input. It would fail SC
1.3.1 in minutes or hours. That's why it is more straightforward to be
able to include things even if they are important processes on a site.

Working out a sampling approach still makes sense even if you look at
parts of a site - and also if you evaluate the whole site.

Regards,
Detlev