Re: What is a subject of a test?

On Fri, 01 Apr 2005 23:06:02 +1000, Carlos Iglesias  
<carlos.iglesias@fundacionctic.org> wrote:

>> Having read the WCAG spec a lot of times, it seems that a
>> properly designed tool and an expert evaluator will get the
>> same results nearly all the time... (of course a badly
>> designed tool and a person testing without much idea of how
>> to do it will not :-)

> I absolutely disagree. If another evaluator gets different results in an  
> evaluation, I
> prefer to think that it's due the implicit ambiguity of the WCAG spec,  
> and different
> persons could have different opinions, instead thinking that it's due to  
> the use of a bad
> tool or a possible precarious knowledge of the evaluator.

Both cases are possible, and I have seen both many many times.

>> > What I mean is that if we don't have a clear specification then it
>> > will be open to personal interpretation.
>>
>> Sure. But this is irrelevant to the EARL spec - it is a
>> question of how good a particular spec we are testing against is.
> So, do you think that is the spec we are testing that should explicitly  
> when a test case
> fail, pass, is not tested, not applicable, etc. ?

Yes.

> I think that EARL could have something to do with this. For example:
> ValidityLevel
> Pass - When the test case is tested and passed
> NotApplicable - When there is no subject to be tested

I think there are two different topics at once. I agree completely that  
EARL should explain what its result types mean, and I think that Pass /  
NotApplicable are like you suggest. We went through the same process when  
we created the "Partial" result type, as a subclass of Fail. It means that  
the point is not met, but there is some attempt to meet it - as opposed to  
completely ignoring it.

But the specs themselves still need to be reasonably testable, and EARL  
(and we the designers of EARL) don't necessarily know anything about the  
spec we are testing against. If we finish EARL before WCAG 2 is finished,  
we won't even be able to say anything certain about those tests. So the  
spec used for testing needs to make it clear what passes or doesn't.

>
>> In practical terms this particular ambiguity isn't such a big
>> deal anyway.
>> Most specs are pretty clear on whether a result of Not
>> Applicable is equivalent to a pass in determining overall
>> conformance (as in the case of WCAG specs) or not...
> In practical terms most specs are pretty clear, not all.
>

Right. But EARL isn't going to make it possible to do god interoperable  
testing using a set of tests that people don't agree on anyway...

cheers

Chaals

-- 
Charles McCathieNevile                      Fundacion Sidar
charles@sidar.org   +61 409 134 136    http://www.sidar.org

Received on Friday, 1 April 2005 13:36:55 UTC