Re: Aspects of an evaluator

Hi Peter, 

Your point about the single person with the massive website is well made - and moves forcibly against the idea of independence and my early suggested aspects… And, in this context - I can see clearly why you think the question of the independence/inter-dependence of an evaluator from the site being evaluated is outside of the scope of our charter.  But, I'm still not 100% convinced… 99% maybe ;-)

To my mind, the question is about trust - will the public place an equal amount of trust in an evaluation done by a 1st party, as they would a 3rd party? 

Well, I think yes they could…

But, in our situation it might only be achieved under certain circumstances…  Seemingly, a number of options exist:

1) Insist that the whole evaluation report be published, along with the urls tested, procedure, etc...  The public could recreate your tests and confirm your findings; or

2) Leave the publishing decision to the evaluation commissioner, and instead set requirements for independence on the evaluator.

	Note: In our methodology we already state "documentation need not necessarily be public, as disclosure is up to the owner and/or evaluation commissioner" - presumably you would not wish to publish if you have assessed 	restricted areas of the website (which is a big issue with option 1, amongst others)...

	So… We already seem to be looking at the second option - hence the reason I'm not yet 100% convinced that the question of evaluator independence is currently entirely out of scope; or

3) Insist that some parts of every report are made public - not urls, just procedures (as it should be the case that any similar samples of web pages taken from the defined site will lead to pretty much the same result)… 

In light of your point, and the issues with option 1 further investigation into option 3 might be an idea - it would of course mean changes to 5a, and removing the idea of independence from the scope.

Thoughts / comments welcome… 
 
All the best 

Alistair

On 21 Jun 2012, at 20:17, Peter Korn wrote:

> Alistair,
> 
> If I am the only person involved in creating my own, fair sized website (too large to feasibly evaluate every single page, being as it is programmatically generated, etc. etc.), then "self-assessment" means that I am also the assessor.  I cannot be an ISO 9001:2000 compliant internal auditor.
> 
> Separate from that example, I don't understand why EvalTF should be concerning itself with ISO 900x in any way.  Looking again at the Objective portion of the Work Statement, our mandate is for a technical task (as I understand it): how to select a representative sample of a site, how to aggregate results into an overall conformance statement, etc.  
> 
> The question of the independence/inter-dependence of an evaluator from the site being evaluated is outside of the scope of our charter.
> 
> 
> Peter
> 
> 
> On 6/21/2012 11:06 AM, Alistair Garrison wrote:
>> Hi Peter, 
>> 
>> Supporting 1st party assessment is as important to me as supporting 3rd party assessment - which is why I based my proposal on those well documented aspects you would look for in an internal auditor for ISO 9001:2000.
>> 
>> Maybe, for clarity, it should have been 'not associated in their day to day role with' - I think you have read 'associated' in the same light as 'independent'.  
>> 
>> Hope this helps.
>> 
>> Alistair
>> 
>> On 21 Jun 2012, at 19:36, Peter Korn wrote:
>> 
>>> Alistair,
>>> 
>>> It wasn't clear to me that this was the outcome of our meeting.  
>>> 
>>> Reviewing the EvalTF Work Statement, the first sentence of the Objective reads (emphasis added): "objective of Eval TF is to develop an internationally harmonized methodology for evaluating the conformance of websites to WCAG 2.0, that supports different contexts, such as for self-assessment or third-party evaluation of small or larger websites".
>>> 
>>> If the methodology is to support self-assessment, then it cannot define the evaluator as be different from the developer/maintainer/accessibility-expert for the site.
>>> 
>>> 
>>> Regards,
>>> 
>>> Peter
>>> 
>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote:
>>>> Dear all, 
>>>> 
>>>> In today's telecon, we discussed terms like independent when talking about evaluations.  The outcome appeared to be that what was needed was in fact a better definition for 'evaluator'.  
>>>> 
>>>> I'm not going to propose the whole definition for 'evaluator', however, just two aspects which we might consider including in the definition:
>>>> 
>>>> Aspect 1) (of an evaluator) someone who is not responsible for the accessibility of the website being evaluated.
>>>> Aspect 2) (of an evaluator) someone who is not associated with developing and maintaining the website or its content.
>>>> 
>>>> Thoughts…
>>>> 
>>>> All the best 
>>>> 
>>>> Alistair
>>> 
>>> -- 
>>> <oracle_sig_logo.gif>
>>> Peter Korn | Accessibility Principal
>>> Phone: +1 650 5069522 
>>> 500 Oracle Parkway | Redwood City, CA 94065 
>>> <green-for-email-sig_0.gif> Oracle is committed to developing practices and products that help protect the environment
>>> 
>>> 
>> 
> 
> -- 
> <oracle_sig_logo.gif>
> Peter Korn | Accessibility Principal
> Phone: +1 650 5069522 
> 500 Oracle Parkway | Redwood City, CA 94065 
> <green-for-email-sig_0.gif> Oracle is committed to developing practices and products that help protect the environment
> 
>