Re: Requirements draft - objectivity

Hi all--

I'm not sure what is meant by a controlled test design. Is this the  
same as a test protocol?

Also, when we are talking about objectivity, are we saying that a  
method must lead to an unbiased result, that the reviewer must be  
unbiased, our criteria are not subjective, or all three?

A bit confused.

Mike

On Sep 15, 2011, at 4:24 AM, Kerstin Probiesch <k.probiesch@googlemail.com 
 > wrote:

> Hi Derlev, all,
>
> because one can not be sure about 100 percent objectivity a Test  
> Design should be a controlled test design. In our case - we haven't  
> decided about the Approach - this can happen for example over the  
> amount of pages or the amount of pages per SC. Also with other  
> Deskriptions for Testing Procedures.
>
> Best
>
> Kerstin
>
> Via Mobile
>
> Am 15.09.2011 um 07:39 schrieb Detlev Fischer <fischer@dias.de>:
>
>> Quoting Kerstin Probiesch <k.probiesch@googlemail.com>:
>>
>>> Central question:
>>>
>>> Do we want that a tester can manipulate the results?
>>
>> DF: of course not, but this cannot be ensured by objectivity  
>> (whatever that would mean in practice) but only by some measure of  
>> quality control: a second tester or independent verification of  
>> results (also, verification of the adequacy of the page sample)
>>>
>>> I don't mean the case that something was overlooked but the case  
>>> that something was willingly overlooked. Or the other Way round.
>>
>> DF: Well, if someone wants to distort results there will probably  
>> always ways to do that, I would not start from that assumption. Is  
>> one imperfect or missing alt attributes TRUE or FALSE for SC 1.1.1  
>> applied to the entire page? What about a less than perfect heading  
>> structure? etc, etc. There is, "objectively", always leeway, room  
>> for interpretation, and I think we unfortunately DO need agreement  
>> with reference to cases / examples that set out a model for how  
>> they should be rated.
>>>
>>> If not we need Objectivity as a Requirement. Just Agreement on  
>>> something is not enough.
>>
>> DF: Can you explain what in your view the requirement of  
>> "objectivity" should entail *in practice*, as part of the test  
>> procedure the methodology defines?
>>
>>>
>>> And again: No Objectivity - no standardized methodology.
>>>
>>> Kerstin
>>>
>>>
>>>
>>>
>>>
>>> Via Mobile
>>>
>>> Am 14.09.2011 um 12:09 schrieb Detlev Fischer <fischer@dias.de>:
>>>
>>>> DF: Just one point on objective, objectivity:
>>>> This is not an easy concept - it relies on a proof protocol. For  
>>>> example, you would *map* a page instance tested to a documented  
>>>> inventory of model cases to establish how you should rate it  
>>>> against a particular SC. Often this is easy, but there are many  
>>>> "not ideal" cases to be dealt with.
>>>> So "objective" sounds nice but it does not remove the problem  
>>>> that there will be cases that do not fit the protocol, at which  
>>>> point a human (or group, community) will have to make an informed  
>>>> mapping decision or extend the protocol to include the new  
>>>> instance. I think "agreed interpretation" hits it nicely because  
>>>> there is the community element in it which is quite central to  
>>>> WCAG 2.0 (think of defining accessibility support)
>>>>
>>>> Regards,
>>>> Detlev
>>>>
>>>>>
>>>>> Comment (KP): I understand the Denis' arguments. The more I  
>>>>> think about
>>>>> this: neither "unique interpretation" nor "agreed  
>>>>> interpretation" work very
>>>>> well. I would like to suggest "Objective". Because of the  
>>>>> following reason:
>>>>> It would be one of Criteria for the quality of tests and  
>>>>> includes Execution
>>>>> objectivity, Analysis objectivity and Interpretation  
>>>>> objectivity. If we will
>>>>> have in some cases 100% percent fine, if not we can discuss the  
>>>>> "tolerance".
>>>>> I would suggest:
>>>>>
>>>>> (VC)  I'm still contemplating this one.  I can see both  
>>>>> arguments as plausible.
>>>>> I'm okay with 'objectivity' but think it needs more explanation  
>>>>> i.e. who defines
>>>>> how objective it is?
>>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> ---------------------------------------------------------------
>> Detlev Fischer PhD
>> DIAS GmbH - Daten, Informationssysteme und Analysen im Sozialen
>> Geschäftsführung: Thomas Lilienthal, Michael Zapp
>>
>> Telefon: +49-40-43 18 75-25
>> Mobile: +49-157 7-170 73 84
>> Fax: +49-40-43 18 75-19
>> E-Mail: fischer@dias.de
>>
>> Anschrift: Schulterblatt 36, D-20357 Hamburg
>> Amtsgericht Hamburg HRB 58 167
>> Geschäftsführer: Thomas Lilienthal, Michael Zapp
>> ---------------------------------------------------------------
>>
>

Received on Thursday, 15 September 2011 13:53:40 UTC