Re: Aspects of an evaluator

Hi Peter, 

Trust issues aside… I'd be really interested to hear your thoughts on the 3rd option I presented…

All the best 

Alistair 


On 21 Jun 2012, at 22:26, Peter Korn wrote:

> Alistair,
> 
> I think the question of trust is, frankly, outside of the scope of our work as well.  Nothing prevents someone from lying - whether or not they are independent.  They may make honest mistakes because they lack the technical expertise needed to do a good job (again whether independent or not).  They may claim their sample is representative but it isn't.  They may claim something failed when it passed (or vice-versa).  
> 
> Being independent doesn't prevent any of that.
> 
> 
> Regards,
> 
> Peter
> 
> On 6/21/2012 12:35 PM, Alistair Garrison wrote:
>> Hi Peter, 
>> 
>> Your point about the single person with the massive website is well made - and moves forcibly against the idea of independence and my early suggested aspects… And, in this context - I can see clearly why you think the question of the independence/inter-dependence of an evaluator from the site being evaluated is outside of the scope of our charter.  But, I'm still not 100% convinced… 99% maybe ;-)
>> 
>> To my mind, the question is about trust - will the public place an equal amount of trust in an evaluation done by a 1st party, as they would a 3rd party? 
>> 
>> Well, I think yes they could…
>> 
>> But, in our situation it might only be achieved under certain circumstances…  Seemingly, a number of options exist:
>> 
>> 1) Insist that the whole evaluation report be published, along with the urls tested, procedure, etc...  The public could recreate your tests and confirm your findings; or
>> 
>> 2) Leave the publishing decision to the evaluation commissioner, and instead set requirements for independence on the evaluator.
>> 
>>  Note: In our methodology we already state "documentation need not necessarily be public, as disclosure is up to the owner and/or evaluation commissioner" - presumably you would not wish to publish if you have assessed  restricted areas of the website (which is a big issue with option 1, amongst others)...
>> 
>>  So… We already seem to be looking at the second option - hence the reason I'm not yet 100% convinced that the question of evaluator independence is currently entirely out of scope; or
>> 
>> 3) Insist that some parts of every report are made public - not urls, just procedures (as it should be the case that any similar samples of web pages taken from the defined site will lead to pretty much the same result)… 
>> 
>> In light of your point, and the issues with option 1 further investigation into option 3 might be an idea - it would of course mean changes to 5a, and removing the idea of independence from the scope.
>> 
>> Thoughts / comments welcome… 
>>  
>> All the best 
>> 
>> Alistair
>> 
>> On 21 Jun 2012, at 20:17, Peter Korn wrote:
>> 
>>> Alistair,
>>> 
>>> If I am the only person involved in creating my own, fair sized website (too large to feasibly evaluate every single page, being as it is programmatically generated, etc. etc.), then "self-assessment" means that I am also the assessor.  I cannot be an ISO 9001:2000 compliant internal auditor.
>>> 
>>> Separate from that example, I don't understand why EvalTF should be concerning itself with ISO 900x in any way.  Looking again at the Objective portion of the Work Statement, our mandate is for a technical task (as I understand it): how to select a representative sample of a site, how to aggregate results into an overall conformance statement, etc.  
>>> 
>>> The question of the independence/inter-dependence of an evaluator from the site being evaluated is outside of the scope of our charter.
>>> 
>>> 
>>> Peter
>>> 
>>> 
>>> On 6/21/2012 11:06 AM, Alistair Garrison wrote:
>>>> Hi Peter, 
>>>> 
>>>> Supporting 1st party assessment is as important to me as supporting 3rd party assessment - which is why I based my proposal on those well documented aspects you would look for in an internal auditor for ISO 9001:2000.
>>>> 
>>>> Maybe, for clarity, it should have been 'not associated in their day to day role with' - I think you have read 'associated' in the same light as 'independent'.  
>>>> 
>>>> Hope this helps.
>>>> 
>>>> Alistair
>>>> 
>>>> On 21 Jun 2012, at 19:36, Peter Korn wrote:
>>>> 
>>>>> Alistair,
>>>>> 
>>>>> It wasn't clear to me that this was the outcome of our meeting.  
>>>>> 
>>>>> Reviewing the EvalTF Work Statement, the first sentence of the Objective reads (emphasis added): "objective of Eval TF is to develop an internationally harmonized methodology for evaluating the conformance of websites to WCAG 2.0, that supports different contexts, such as for self-assessment or third-party evaluation of small or larger websites".
>>>>> 
>>>>> If the methodology is to support self-assessment, then it cannot define the evaluator as be different from the developer/maintainer/accessibility-expert for the site.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Peter
>>>>> 
>>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote:
>>>>>> Dear all, 
>>>>>> 
>>>>>> In today's telecon, we discussed terms like independent when talking about evaluations.  The outcome appeared to be that what was needed was in fact a better definition for 'evaluator'.  
>>>>>> 
>>>>>> I'm not going to propose the whole definition for 'evaluator', however, just two aspects which we might consider including in the definition:
>>>>>> 
>>>>>> Aspect 1) (of an evaluator) someone who is not responsible for the accessibility of the website being evaluated.
>>>>>> Aspect 2) (of an evaluator) someone who is not associated with developing and maintaining the website or its content.
>>>>>> 
>>>>>> Thoughts…
>>>>>> 
>>>>>> All the best 
>>>>>> 
>>>>>> Alistair
>>>>> 
>>>>> -- 
>>>>> <oracle_sig_logo.gif>
>>>>> Peter Korn | Accessibility Principal
>>>>> Phone: +1 650 5069522 
>>>>> 500 Oracle Parkway | Redwood City, CA 94065 
>>>>> <green-for-email-sig_0.gif> Oracle is committed to developing practices and products that help protect the environment
>>>>> 
>>>>> 
>>>> 
>>> 
>>> -- 
>>> <oracle_sig_logo.gif>
>>> Peter Korn | Accessibility Principal
>>> Phone: +1 650 5069522 
>>> 500 Oracle Parkway | Redwood City, CA 94065 
>>> <green-for-email-sig_0.gif> Oracle is committed to developing practices and products that help protect the environment
>>> 
>>> 
>> 
> 
> -- 
> <oracle_sig_logo.gif>
> Peter Korn | Accessibility Principal
> Phone: +1 650 5069522 
> 500 Oracle Parkway | Redwood City, CA 94065 
> <green-for-email-sig_0.gif> Oracle is committed to developing practices and products that help protect the environment
> 
>