RE: Concerns about not covering full website

Hi Alistair,

So this would mean that the conformance claim is already made and not dependent on the evaluation. The Evaluation Methodology would then be to check if conformance claims are valid. Are we not putting the horse behind the wagon this way?
Kindest regards,

Eric


________________________________
Van: Alistair Garrison [alistair.j.garrison@gmail.com]
Verzonden: maandag 30 januari 2012 10:48
Aan: Eval TF
Onderwerp: Re: Concerns about not covering full website

Dear Eval TF,

I'm now almost certain we are wanting the evaluation methodology to deliver the same things - a precise method "for evaluating the conformance of websites, sections of websites, and web applications, regardless of size, to WCAG 2.0" (copied from RQ 01 Scope and target audience of the Methodology - http://www.w3.org/WAI/ER/conformance/ED-methodology-reqs-20111012).

The question I still have is why in the rest of this draft document we have limited ourselves to saying "conformance of entire website", or words to that effect.  With what I have read, I'd go so far as to say that the word 'website' has already become quite a confused term, much, much different from our definition "website - a coherent collection of one or more related web pages that together provide common use or functionality" (again copied from the draft).

My concern is that in using the term 'website' we have lost sight of what we really meant to say - that our methodology should be able to accommodate the evaluation of "a coherent collection of one or more related web pages that together provide common use or functionality" - backed up by our continued use of the phrase 'regardless of size'.

The main reason for suggesting the use of WCAG 2.0 Conformance claims was that they already, seemingly, provide an effective way to precisely define a "coherent collection of one or more related web pages that together provide common use or functionality", and have already been adopted by the main WCAG 2.0 WG into the WCAG 2.0 concept / documentation.

In my opinion, our intensions for the evaluation methodology would be much more precisely captured if we were to change the first line in the scope section of our draft requirements document to possibly read -

"The main goal is to define an internationally harmonised methodology for evaluating WCAG 2.0 Conformance claims (for entire websites, sections of websites, web applications, whole web pages, etc...)."

All the best

Alistair

On 29 Jan 2012, at 21:06, Kerstin Probiesch wrote:

Hi Allistair,

As, only a WCAG 2.0 Conformance claim based
on WCAG 2.0 Conformance evaluation undertaken by an external evaluator
could ever be fully trusted.

No. This is not what I'm trying to say. My intention is to have clear
requirements and give guidelines for controlled testing situations. The more
flexible our methodology is the more - I believe - it is likely that
goodness criteria won't be fulfilled. If the evaluator is external or
internal is not that much important. More important is: reducing the
probability for errors and give guidance for "controlled" test situations.
Every type of error could have an impact on the result and on the
reliability. The more flexible the more errors - including systematic errors
- could come in. As already mentioned: I think some exceptions are necessary
and possible.

Best

Kerstin



All the best

Alistair

On 29 Jan 2012, at 10:28, Kerstin Probiesch wrote:

Hi Elle, all,

probably there is a way for some exceptions. But I strongly disagree
with a
methodology with a scope that is externally defined in general.
Companies
will always choose the most cheapest way for evaluating the
accessibility of
their websites or products. But later they want to sell their
products and
earn money with them. And I'm sure the same is true not only for the
private
sector. Even the public sector will seize every opportunity for
reducing
costs. The best way for doing so is: reducing the scope. What if the
next
website owner says: choose 3 "representative" pages. Than this is the
scope.
An evaluator will choose those pages and the company or any other
website
owner can publish a claim that according to this there is 100%
conformance.
Others will follow, cause it's very cheap, very fast. One can imagine
a lot
of different scenarios and I fear in the worst case we will come out
with a
methodology which depends on economic interests of companies. The
question
is also if a methodology like this evaluates WCAG2 or measure what
companies
and the public sector want to pay? Will there be a correlation with
WCAG2 or
merely a correlation with economic interests? Sorry for these
probably
provocative sentences, but I believe that every negative scenario
which one
can imagine will happen.

Anyway and as said before: I also recognize the necessity for some
exceptions and for some guidelines for special cases but they must be
linked
to strict conditions. They could be placed in a separate chapter of
our
methodology:

# Follow-Up Checks / quality assurance (for website which are already
evaluated according to our methodology)
# Portions of website: Shops (as special and even typical cases for
complete
processes)
# and probably subdomains (but without permission placing the claim
on any
other subdomain or main page and with conditions like: a clear and
unique
separate content which also could be a standalone website)

Best

Kerstin




Von: Elle [mailto:nethermind@gmail.com]
Gesendet: Samstag, 28. Januar 2012 20:36
An: Kerstin Probiesch
Cc: RichardWarren; Alistair Garrison; Martijn Houtepen; Eval TF
Betreff: Re: Concerns about not covering full website

I have an alternate example to consider:

A large e-Commerce website with several segmented portals catering to
different audiences - This company is trying to secure a government
contract
to sell a specific product suite to the state of New Jersey. In order
to
secure this contract, the company must show accessibility conformance
for
the specific product suite portal.

In our current model, the company would be forced to consider the
entire
e-Commerce website in its evaluation, which could be cost prohibitive
and
time consuming compared to the initial goal for securing the
contract.
Additionally, it affects several different business teams within the
company itself, so making the business case for evaluation (internal
resources are always required) and remediation could be impossible.
Additionally, the state of New Jersey may end up penalized, because
companies find it an unreasonable burden to evaluate and change an
entire
enterprise website for a single contract.

In an externally defined scope model, the company would identify with
the
state of New Jersey what the scope of the conformance claim needed to
be,
and likely include areas of the website that were equally utilized by
all
portals.  The company performs the evaluation, remediates the portal
and any
associated applications, and they win the contract. The result would
be that
a portion of the website is now compliant for anyone who uses that
portal,
and the company has a business case that can stand up on the
profitability
of building accessible websites.  Perhaps the next redesign of the
entire
e-Commerce website will consider it a core requirement to meet these
standards.



Respectfully,
Elle



On Sat, Jan 28, 2012 at 3:54 AM, Kerstin Probiesch
<k.probiesch@googlemail.com<mailto:k.probiesch@googlemail.com>> wrote:
Hi all,

I agree with Martijn. The more I think about it I also feel that this
approach is too flexible and I fear that in the end we will not come
out
with a harmonized evaluation methodology but with something we
already have.


Of course there are Organizations/Clients who invest in evaluations
because
they feel that it is important. But I believe the majority don't
invest in
that because of the users, because of the importance of
accessibility. They
invest because they have to and they will choose the most cheapest
way for
doing so and claiming conformance on the base of the less pages as
possible.

I agree that we need some procedures for cases like:

#there was already an successful evaluation with a testing procedure
which
follows our methodology and a website owner included a shop. Will it
be
enough to evaluate just the shop? Does it depends on the time between
the
first evaluation and second of the shop? Are there other changes,
other
additional content?
#fast growing websites

And probably some other well described exceptions: for example a
website
owner wants to claim just a shop. This should include the evaluation
of the
shop itself and the path to the shop.

These cases should be linked to strict defined conditions.

But I disagree with a methodology where a website owner (always)
defines the
scope. Except of course where a website owner don't want to claim
anything.

Best

Kerstin


Von: Elle [mailto:nethermind@gmail.com]
Gesendet: Freitag, 27. Januar 2012 18:53
An: RichardWarren
Cc: Alistair Garrison; Martijn Houtepen; Eval TF
Betreff: Re: Concerns about not covering full website

I also agree.  I've thought about this since our call from a few
pragmatic
and business angles.

While I see the risks that Martin identified, I don't think people
would
invest in auditing and meeting conformance levels on individual pages
just
to spin it for better public opinion. Organizations are either
committed to
accessibility or they're forced to meet requirements due to
regulatory or
litigation reasons.

Either way, that scope is defined by them, not W3C or a methodology.
If
organizations are committed to accessibility, we requiring a full
website
evaluation penalizes any phased efforts. Additionally, the concept of
a
"full website" is becoming less and less viable to companies in a
component
driven environment of content delivery.

I do, however, want to request that we keep the URLs as a requirement
in
defining that scope (and not something smaller within a single page).


Respectfully,
Elle

On Fri, Jan 27, 2012 at 12:12 PM, RichardWarren
<richard.warren@userite.com<mailto:richard.warren@userite.com>>
wrote:
Hi
I agree with Alistair. We are supposed to be developing a methodology
to
help people deliver a reliable and trusted conformance claim (if
appropriate). It is not our job to rewrite WCAG.

Richard

From: Alistair Garrison
Sent: Friday, January 27, 2012 4:03 PM
To: Martijn Houtepen ; Eval TF
Subject: Re: Concerns about not covering full website

Hi Martijn,

I too feel we are giving people who want to evaluate conformity to
WCAG 2.0
a guide on how to do a proper, representative evaluation... the
question I
suppose is why must it be limited to only our definition of a
website?

Can I just clarify something (it being late on a Friday) - are you
saying
that in your opinion the evaluation methodology should primarily
support
independent (e.g. third-party) evaluations of whole websites, like
something
you might need to support some sort of conformance scheme / badge?

I might be missing something, but I'd like to think that if I were a
website
owner I'd be able to check the claimed conformance of my own page,
pages,
sections, sub-domains or entire website using the W3C/WAI WCAG 2.0
Evaluation methodology - and, without having to use a third-party. In
fact,
I think I would raise questions / eyebrows if I couldn't...

I suppose I would also like to think that if I verified my claimed
conformance myself using this Evaluation methodology people would
actually
believe it - without the need to have it independently verified.  We
should
not forget, that there will be many folks out there who have a site
but
cannot afford to get it independently verified...

I'm certain that in this day-and-age a website owner would have more
to
loose, than gain, by using 'smart wording' to inflate their claims. I
would
even go so far as to suggest that the probability of a website owner
making
a WCAG 2.0 Conformance claim for a small part of their site and then
implying its for the entire website (by saying or not saying) would
be small
- especially as the whole point of the WCAG 2.0 Conformance claim is
to make
this situation ever-so-plainly obvious...

Even so, you appear to have questions about the actual worth of WCAG
2.0
Conformance claims (something you might consider raising with the
WCAG 2.0
working group).  Personally, I believe the WCAG 2.0 Conformance Claim
concept to be quite sound... and, I can't really understand why (as
members
of a Task Force of the WCAG 2.0 WG) we wouldn't want to be
encouraging their
use...

The draft methodology is a draft, and as such not set in stone... I
trust we
hopefully have the possibility to change it if the consensus view is
that
the change (using a Conformance Claim to define our scope) is of
value...

All the best...

Alistair

On 27 Jan 2012, at 14:55, Martijn Houtepen wrote:


Hi Alistair, TF,

I think this approach will be getting a little bit too flexible. As I
understand, all the work we have done until now is in order to
evaluate
whole websites. We want “a standardized way to evaluate the
conformance of
websites to WCAG 2.0.” (quoted from the introduction of our draft
methodology), the whole draft document focuses on  ‘websites’ instead
of web
pages. I can understand from an owners point of view that he/she
sometimes
will want to split up a whole website into several subevaluations,
but these
subevaluations, in my opinion, will together comprise the entire
website,
and not leave out some part for any reason.

Some problems arise if we do follow this strategy:

If we only check what the owner of a website wants us to check, this
creates
the risk that an owner will only have the conforming parts of the
website
evaluated. An owner of a inaccessible website can then, with a little
smart
wording, proudly claim conformance (of f.e. a little subsection). An
experienced user can differentiate between strong and weak claims,
but a
normal user can’t. This in turn creates two potential pitfalls: lots
of
‘empty’ claims that in practice do not help users, f.e.: “Our
homepage
conforms to WCAG 2.0”, leaving out that none of the underlying pages
conform.  Secondly the degradation of those claims that are truly
useful to
an user “Page X claims to conform (using an ‘empty’ claim) but I
can’t use
it, so I probably also can’t use page Y (that conforms and has a
useful
claim). Confusion as to what an evaluated conformance claim means,
need to
be avoided.

Secondly, this will create the possibility to exclude key scenario’s
and
complete paths. Again from the introduction: “The Methodology defines
manual
and semi-automated methods for selecting representative samples of
web pages
from websites that include complete processes.” I feel we are giving
people
who want to evaluate conformity to WCAG a guide on how to do a
proper,
representative evaluation of a website. If we only check pages
supplied by a
website owner we can not always include key scenario’s or complete
paths. I
think we do not want a web store claiming conformance, whilst it
check-out
procedure is inaccessible. Or YouTube claiming conformance except for
the
pages that have video’s. To form a statement about conformance, I
feel we
have to be able to independently evaluate the whole website including
whatever it may contain. The resulting conformance statement, if
made, will
then be of practical use for people with disabilities, as they can
trust a
conformance claim to be of use for them.

Kind regards,

Martijn Houtepen


________________________________________
Van: Alistair Garrison [mailto:alistair.j.garrison@gmail.com]
Verzonden: vrijdag 27 januari 2012 10:31
Aan: Eval TF
CC: Eric Velleman; Detlev Fischer
Onderwerp: Re: Concerns about not covering full website

Dear all,

Due to its many perceived benefits, I have long been a passionate
advocate
for using a website owner's WCAG 2.0 Conformance Claim as the scope
of our
evaluation - writing several emails in the past months about this
very
subject*.  The following summarises my thoughts to date...

I totally support the pros stated by both Detlev and Eric,
emphasising that:

"This approach is very flexible and would make it possible for
evaluators to
verify a WCAG 2.0 claim conformance for more stable websites or fast
growing
websites, or portions of websites, sections of a website or even
single
pages in a website - using sampling methods for large websites as per
our
discussions."

In addition, I strongly believe that:

1) It makes evaluation more reproducible, and more efficient, as we
know
exactly what forms the scope of our evaluation (i.e. what is defined
in the
WCAG 2.0 Conformance claim at the time of evaluation, from which
samples can
be taken as necessary);

2) Once an initial evaluation had been done (scope WCAG 2.0
Conformance
claim) we would only ever need to assess new additions to the WCAG
2.0
conformance claim on an on-going basis - hopefully saving people time
and
money;

3) It tells the public exactly what conforms in the website - a bit
more
like the VPAT.  If the WCAG 2.0 conformance claim states, for
example, that
a website's shopping process (defined by urls) conforms to x and y,
a user
should know it will be accessible to them;

4) If the website contains any complete processes which back-ends
into third
party pages - we are in fact prevented from just saying "the whole
website"
conforms.  A WCAG 2.0 Conformance Claim, on the other hand, could be
made
for this website, and evaluated, if we choose to use WCAG 2.0
conformance
claims as our scope.

5) It supports, even promotes and encourages, detailed WCAG 2.0
conformance
claims to be made by the people responsible for entire websites /
parts of
websites.

It would also mean that the majority of Section 3 could be re-defined
simply
e.g. "The scope of the evaluation is defined as all urls for which a
WCAG
2.0 conformance claim is being made, at the time of the evaluation".

* Referenced emails from Nov / Dec 2011

http://lists.w3.org/Archives/Public/public-wai-
evaltf/2011Dec/0004.html
http://lists.w3.org/Archives/Public/public-wai-
evaltf/2011Nov/0073.html
http://lists.w3.org/Archives/Public/public-wai-
evaltf/2011Nov/0086.html
http://lists.w3.org/Archives/Public/public-wai-
evaltf/2011Dec/0023.html

All the best

Alistair
On 27 Jan 2012, at 01:01, Velleman, Eric wrote:

Hi Detlev, all,

Yes, it was an interesting discussion this afternoon about the scope
of an
evaluation :-).

This afternoon we discussed the possibility to use the WCAG2.0
Evaluation
Methodology not only for full website evaluation, but also for parts
of
websites. In the current version the evaluation focuses on the full
website
with a possibility to exclude parts from the scope.  If we want to
include
evaluating parts of a website (like only the WAI part of W3C or only
the BAD
website) this should be added in the scope section.

Today a different approach was proposed: We look at it the other way
around:
the Conformance claim determines the scope.

This approach is very flexible and makes it possible to claim
conformance
for portions of websites, possibly even for technologies or
collections of
pages or single pages. This makes the Methodology much more flexible
for
website owners who are not always interested in evaluation of the
full
website. This is a change to the requirements document but it will
provide
support for requests by website owners and evaluators wishing to just
look
at a specific part of a website.

I would propose that we do a short discussion this week on this
approach. Do
we leave the scope completely free for the site owner to decide?

Kindest regards,

Eric



________________________________________
Van: Detlev Fischer [fischer@dias.de]
Verzonden: donderdag 26 januari 2012 17:16
Aan: EVAL TF
Onderwerp: Concerns about not covering full website

Hi everyone,

I think a lot of the heat in the disacussion we just had may be down
to
a misunderstanding.

Eric, no one says that the evaluation of an entire webiste is not a
good
idea or should not be done. If a site owner wants it done, that's
fine.

If however, a site owner wants to look at a particular section (that
can
be many pages), it should be possible to evaluate just that. It's
also a
cost issue. Any conformance claim must make that limited scope
perfectly
clear, preferably by binding the evaluation to a set of URLs.

Increasingly, sites have many different sections, like user generated
content. No sane person would claim even A-Level conformance for a
wiki
or bulletin board that allows unstructured text input. It would fail
SC
1.3.1 in minutes or hours. That's why it is more straightforward to
be
able to include things even if they are important processes on a
site.

Working out a sampling approach still makes sense even if you look at
parts of a site - and also if you evaluate the whole site.

Regards,
Detlev






--
If you want to build a ship, don't drum up the people to gather wood,
divide
the work, and give orders. Instead, teach them to yearn for the vast
and
endless sea.
- Antoine De Saint-Exupéry, The Little Prince





--
If you want to build a ship, don't drum up the people to gather wood,
divide
the work, and give orders. Instead, teach them to yearn for the vast
and
endless sea.
- Antoine De Saint-Exupéry, The Little Prince

Received on Monday, 30 January 2012 10:55:09 UTC