Comment on Requirements RO3, R04, proposal to include Validity

Regrets for not being able to participate in today's Teleconference.

Here are my comments to the updated section on requirements:


R03: Unambiguous interpretation

The current draft text is:

"The Methodology itself is unambiguous to people who want to use it. It 
should make it clear to users what they can do if they choose a certain 
evaluation approach in the document."

DF: The second sentence sounds a bit lame. 'Evaluation approach' sounds 
like a synonym for the Methodology we are developing, so "a certain 
Evaluation approach" sounds like one of many, possible approaches. This 
makes the Methodology sound like a broad-church kind of thing: you could 
follow this approach, or that, or something else. That is probably not 
what is intended.
The methodology should be prescriptive even if it leaves the details 
(what tools, what sequence of checks) to evaluation tool implementations.
Another bit of nitpicking: "In the document" also sounds somewhat loose 
- what document? Is this a synonym for the entire methodology or will 
the methodology contain several documents? Or (unlikely), is it the 
document to be evaluated?

And to hark back to the term "unambiguous" (sorry, I can't resist to 
insist): I see a fundamental conflict between keeping the methodology, 
acc. to R02, "Tool and browser independent" and free of concrete 
interpretation advice, and the goal of achieving "unambiguopus 
interpretation".

I agree that resources that will help achieve a valid interpretation can 
resist outside the methodology itself (e.g., a set of cases with 
appropriate judgements/ ratings per SC). But whether an interpretation 
will be unambiguous will depend on the quality of this external set. In 
my view, in cannot be safeguarded within a methodology that just refers 
to outside resouces not in its scope and outside its control.

So I still maintain that R03 is ultimately more than our methodology 
could hope to stem. Therefore, I think it should not appear as 
requirement in this form.


R04: Replicable

The current draft text is:

"Different Web accessibility evaluators using the same methods on the 
website(s) should get the same results."

The comment to R03 above similarly applies to R04: Replicable. I agree 
it should be our *aim to strive for replicability* (like a vanishing 
point) but I believe it will never be achieved in the literal sense.

We had the term 'Reliable' at some point next to 'Replicable'. 
'Reliable' is less deterministic and seems a lot more suited. A test can 
be reliable within tolerances (see R14). To claim Replicability and 
allow for tolerances at the same time seems disingenuous to me.


Proposal: Change to

R03: Valid
R04: Reliable

Validity is perhaps the most important requirement and is it not yet 
stated. There is "R17: Support validity" but this just expresses that 
tests need to be documented so they can be cross-checked.

I think R16 and R17 are actually quite similar and could be folded into

R16: Support independent verification and quality assurance

*Validity* is necessary as an independent requirement somewhere near the 
top of the list. Without achieving it (to a degree, see tolerances), the 
methodology would be pointless.

Validity ultimately needs to be grounded in testing with real users of 
assistive technologies. That does *not* mean that actual AT tests have 
to carried out in each application of the methogology, but that the set 
of resources guiding interpretation needs to be consistent with AT user 
experience (which also implies that it needs to be updated as 
technologies become better supported by UA / AT), so there is a dynamic 
element.

This is what I tried to convey in the last teleconference, but talking 
too much as usual, there wasn't much of this point left in the scribe's 
notes (NOT the fault of the scribe, just too much talk)

Hope you have a fruitful discussion today!
Detlev



Am 29.09.2011 09:09, schrieb Shadi Abou-Zahra:
> Dear All,
>
> Please find the latest draft of the requirements from Eric:
> - <http://www.w3.org/WAI/ER/conformance/ED-methodology-20110928>
>
> Best,
> Shadi
>
>
> On 28.9.2011 21:39, Velleman, Eric wrote:
>> Dear Eval TF,
>>
>> The next teleconference is scheduled for Thursday 29 September 2011 at:
>> * 14:00 to 15:00 UTC
>> * 16:00 to 17:00 Central European Time
>> * 10:00 to 11:00 North American Eastern Time (ET)
>> * 07:00 to 08:00 North American Pacific Time (PT)
>> * 22:00 to 23:00 Western Australia Time
>>
>> Please check the World Clock Meeting Planner to find out the precise
>> date for your own time zone:
>> -<http://www.timeanddate.com/worldclock/meeting.html>
>>
>> The teleconference information is: (Passcode 3825 - "EVAL")
>> * +1.617.761.6200
>> * SIP / VoIP -http://www.w3.org/2006/tools/wiki/Zakim-SIP
>>
>> We also use IRC to support the meeting: (http://irc.w3.org)
>> * IRC server: irc.w3.org
>> * port: 6665
>> * channel: #eval
>>
>>
>> AGENDA:
>>
>> #1. Welcome
>>
>> #2. Requirements
There is a new version available at:
>> <http://www.w3.org/WAI/ER/conformance/ED-methodology-20110924.html>
>> We will discuss this version during this meeting. A lot of changes
>> following our discussion last week and on the list. And some changes
>> proposed by Shadi to make things more clear.
>>
>> #3. Resources related to our Methodology
>> It would be interesting to gather resources about Web Evaluation
>> Methodologies. We made a good start thanks to Tim and . What to do
>> with this information? Shadi proposes to add information to the
>> Benchmarking Web Accessibility Metrics Wiki
>> at:<http://www.w3.org/WAI/RD/wiki/Benchmarking_Web_Accessibility_Metrics>
>>
>> #4. Any other business
>>
>> Regards,
>>
>> Eric
>>
>


-- 
---------------------------------------------------------------
Detlev Fischer PhD
DIAS GmbH - Daten, Informationssysteme und Analysen im Sozialen
Geschäftsführung: Thomas Lilienthal, Michael Zapp

Telefon: +49-40-43 18 75-25
Mobile: +49-157 7-170 73 84
Fax: +49-40-43 18 75-19
E-Mail: fischer@dias.de

Anschrift: Schulterblatt 36, D-20357 Hamburg
Amtsgericht Hamburg HRB 58 167
Geschäftsführer: Thomas Lilienthal, Michael Zapp
---------------------------------------------------------------

Received on Thursday, 29 September 2011 08:18:20 UTC