Re: Aspects of an evaluator

Alistair,

I think the question of trust is, frankly, outside of the scope of our 
work as well.  Nothing prevents someone from lying - whether or not they 
are independent.  They may make honest mistakes because they lack the 
technical expertise needed to do a good job (again whether independent 
or not).  They may claim their sample is representative but it isn't.  
They may claim something failed when it passed (or vice-versa).

Being independent doesn't prevent any of that.


Regards,

Peter

On 6/21/2012 12:35 PM, Alistair Garrison wrote:
> Hi Peter,
>
> Your point about the single person with the massive website is well 
> made - and moves forcibly against the idea of independence and my 
> early suggested aspects… And, in this context - I can see clearly why 
> you think the question of the independence/inter-dependence of an 
> evaluator from the site being evaluated is outside of the scope of our 
> charter.  But, I'm still not 100% convinced… 99% maybe ;-)
>
> To my mind, the question is about trust - will the public place an 
> equal amount of trust in an evaluation done by a 1st party, as they 
> would a 3rd party?
>
> Well, I think yes they could…
>
> But, in our situation it might only be achieved under certain 
> circumstances…  Seemingly, a number of options exist:
>
> 1) Insist that the whole evaluation report be published, along with 
> the urls tested, procedure, etc...  The public could recreate your 
> tests and confirm your findings; or
>
> 2) Leave the publishing decision to the evaluation commissioner, and 
> instead set requirements for independence on the evaluator.
>
> Note: In our methodology we already state "documentation need not 
> necessarily be public, as disclosure is up to the owner and/or 
> evaluation commissioner" - presumably you would not wish to publish if 
> you have assessed restricted areas of the website (which is a big 
> issue with option 1, amongst others)...
>
> So… We already seem to be looking at the second option - hence the 
> reason I'm not yet 100% convinced that the question of evaluator 
> independence is currently entirely out of scope; or
>
> 3) Insist that some parts of every report are made public - not urls, 
> just procedures (as it should be the case that any similar samples of 
> web pages taken from the defined site will lead to pretty much the 
> same result)…
>
> In light of your point, and the issues with option 1 further 
> investigation into option 3 might be an idea - it would of course mean 
> changes to 5a, and removing the idea of independence from the scope.
>
> Thoughts / comments welcome…
> All the best
>
> Alistair
>
> On 21 Jun 2012, at 20:17, Peter Korn wrote:
>
>> Alistair,
>>
>> If I am the only person involved in creating my own, fair sized 
>> website (too large to feasibly evaluate every single page, being as 
>> it is programmatically generated, etc. etc.), then "self-assessment" 
>> means that I am also the assessor.  I cannot be an ISO 9001:2000 
>> compliant internal auditor.
>>
>> Separate from that example, I don't understand why EvalTF should be 
>> concerning itself with ISO 900x in any way. Looking again at the 
>> Objective portion of the Work Statement 
>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our mandate 
>> is for a technical task (as I understand it): how to select a 
>> representative sample of a site, how to aggregate results into an 
>> overall conformance statement, etc.
>>
>> The question of the independence/inter-dependence of an evaluator 
>> from the site being evaluated is outside of the scope of our charter.
>>
>>
>> Peter
>>
>>
>> On 6/21/2012 11:06 AM, Alistair Garrison wrote:
>>> Hi Peter,
>>>
>>> Supporting 1st party assessment is as important to me as supporting 
>>> 3rd party assessment - which is why I based my proposal on those 
>>> well documented aspects you would look for in an internal auditor 
>>> for ISO 9001:2000.
>>>
>>> Maybe, for clarity, it should have been 'not associated in their day 
>>> to day role with' - I think you have read 'associated' in the same 
>>> light as 'independent'.
>>>
>>> Hope this helps.
>>>
>>> Alistair
>>>
>>> On 21 Jun 2012, at 19:36, Peter Korn wrote:
>>>
>>>> Alistair,
>>>>
>>>> It wasn't clear to me that this was the outcome of our meeting.
>>>>
>>>> Reviewing the EvalTF Work Statement 
>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first sentence of 
>>>> the Objective reads (*/emphasis added/*): "objective of Eval TF is 
>>>> to develop an internationally harmonized methodology for evaluating 
>>>> the conformance of websites to WCAG 2.0,/*that supports different 
>>>> contexts, such as for self-assessment or third-party evaluation*/ 
>>>> of small or larger websites".
>>>>
>>>> If the methodology is to support self-assessment, then it cannot 
>>>> define the evaluator as be different from the 
>>>> developer/maintainer/accessibility-expert for the site.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Peter
>>>>
>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote:
>>>>> Dear all,
>>>>>
>>>>> In today's telecon, we discussed terms like independent when talking about evaluations.  The outcome appeared to be that what was needed was in fact a better definition for 'evaluator'.
>>>>>
>>>>> I'm not going to propose the whole definition for 'evaluator', however, just two aspects which we might consider including in the definition:
>>>>>
>>>>> Aspect 1) (of an evaluator) someone who is not responsible for the accessibility of the website being evaluated.
>>>>> Aspect 2) (of an evaluator) someone who is not associated with developing and maintaining the website or its content.
>>>>>
>>>>> Thoughts…
>>>>>
>>>>> All the best
>>>>>
>>>>> Alistair
>>>>
>>>> -- 
>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>> Peter Korn | Accessibility Principal
>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> 
>>>> Oracle is committed to developing practices and products that help 
>>>> protect the environment
>>>>
>>>>
>>>
>>
>> -- 
>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>> Peter Korn | Accessibility Principal
>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>> 500 Oracle Parkway | Redwood City, CA 94065
>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> Oracle 
>> is committed to developing practices and products that help protect 
>> the environment
>>
>>
>

-- 
Oracle <http://www.oracle.com>
Peter Korn | Accessibility Principal
Phone: +1 650 5069522 <tel:+1%20650%205069522>
500 Oracle Parkway | Redwood City, CA 94065
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment