Re: EvalTF Test-run of WCAG-EM

Eric, all,

My I suggest some slight edits to the text below?  Made in-line

On 4/9/2013 1:46 PM, Velleman, Eric wrote:
> Dear all,
>
> After fruitful discussion on the list, many volunteers and some nominations for websites, we can get started with the first test-run of WCAG-EM. We will discuss this in the telco of April 11. If you have not done so, please indicate if you want to participate. We will discuss the time necessary for the larger and the smaller tasks in the telco.
>
> The general focus of the test-run is to evaluate how WCAG-EM works in practice and help us identify where the document needs more attention. Please remember this is the first test-run. We do not have to delve into extreme detail. We are looking for the big picture, ambiguous things, missing information, missing steps, etc. We are not yet working on fine-tuning the text although every input is valuable. Note that we are evaluating WCAG-EM and not WCAG itself.

I think the "purpose" text below is something we should make questions 
that we ask explicitly of our "beta testers".  We should explicitly seek 
their comments on these topics I think.

>
> Purpose of testing:
>
> - Do we adequately address the target audience?
Do the target audience(s) find the guidance clear, cover the items they 
felt it should?
> - Do we cover websites as defined (including web applications etc.)?
Does the guidance cover...
> - Does it cover very large and very small websites?
Does the guidance cover...
> - Is the text clear and unambiguous
Is the guidance clear...
> - Do two different evaluators end up with comparable results when using it
Do multiple evaluators...?

> - Is there information missing that is important for understanding
Do reviewers find any guidance or information missing from the guidance 
that is important...
> - Is there information missing that is important for the practical use of WCAG-EM
Do reviewers find any guidance or information missing from the guidance 
that is important....
> - Is the document in any way contradicting WCAG
Do reviewers find anything in the guidance that in any way contradicts 
WCAG 2.0
> - More...?

I think the broad & open-ended questions above are good, but that we 
should ask more specific questions.  E.g.:
  - Are the Conformance Evaluation Procedure steps clear, and easy to 
follow?
  - Is the format of the report clear and easy to follow?
    - Is there anything the reviewer feels should be added to one or 
more of the reports?  Removed?
  - Were there parts of the evaluation (following the guidance) that 
were more difficult to accomplish?  If so, which and why?
  - Does the reviewer feel they were able to collect a representative 
sample of the pages of the website (or paths through a web application)?
    - Is the size of the sample developed by the reviewer in line with 
the "typical" sample size the reviewer would have otherwise used (e.g. 
in a contract hired by the owner of the site)

>
> Cautions:
>
> - We are evaluating WCAG-EM and not WCAG.
"The purpose of this exercise is to evaluate WCAG-EM, and not WCAG 2.0"
> - Follow the text in the Methodology as close as possible.
"Please follow..."
> - This is a first evaluation of WCAG-EM
> - We have to be cautious with naming websites on the mailing list. What we say will be there ‘for always’.
I'm not sure how this should be stated to evaluators.  Perhaps "Please 
note: there should be no public discussion of any website names, URLs, 
or characteristics being evaluated."
> - Please replace the name of the websites we use for the evaluation of WCAG-EM by ‘website1’ and ‘website2’. So turn ‘www.w3.org/morehere’ into ‘website1/morehere’. For details about the evaluation, we can use the WBS system.

"Particularly because this is an evaluation of WCAG-EM, it is 
inappropriate to consider any issues found in this evaluation of a 
website (or lack of issues found) as any accurate indication of the 
(in)accessibility of that website."

>
> Plan:
>
> To make things easier for all of us now, we will cut the evaluation of WCAG-EM into parts and apply this to two or three different websites.
>
> Part One (partial test of first three steps):
>   - Test-run the first three steps (sampling). The volunteers will all do the first three steps for the nominated websites.
> - Discuss outcome: Then we will discuss the outcome on the list, in the WBS system and in the Telco.
>
> Part Two (Test of separate steps):
> - We will test separate steps and discuss the outcome. Output of this discussion to be presented in the WBS/list (to be decided) and in the Telco. What went good and where do we need to do extra work.
>
> Part Three (Test of reporting):
> - Using the different outcomes of step 3, suppose we do the evaluation. Try writing one report and discuss with others.

I think it is very important to do a holistic review of WCAG-EM, not 
just a reductionist approach.  I see value in having a discussion after 
each of these three logical parts, but then I think we should do a 
second evaluation (of different sites) that isn't broken down into these 
parts.  However, if we can only do this once, I think it is better to do 
ONLY a holistic review rather than a reductionist one.

Either the key point of doing this is to figure out how "close to done" 
we are.  Or the key purpose is to test the individual parts to improve 
them - to then later do a holistic view.

To that end, I think we should do (at least) two test runs; this one 
being the "unit testing" run.  And to focus on how human beings read our 
document and find it clear, I think for Part Two above we should all use 
the same sample (for any given website) - keeping that constant so we 
can better focus on differences that arise from how human beings read 
and apply our document.  Same with Part Three - to evaluate the 
differences in reporting that arise from the same set of errors.


Peter


>
> Please join in the start of the test-run in the EvalTF telco of 11 april.
> Kindest regards,
>
> Eric Velleman
>
>
>

-- 
Oracle <http://www.oracle.com>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522 <tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94065
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment