Re: Aspects of an evaluator

Hi Richard, 

We all know what we are doing is for 1st and 3rd Parties - and of course they each have an equal weight.

The debate stems from Eric's questionnaire number 2 - question 6.

DoC_ID_28: Independent evaluator

I would say that I'm hearing most people are now against the proposed action of stating that an evaluator must be independent (me too after Peter's email) - as it would not work in all cases.

All the best 

Alistair

On 22 Jun 2012, at 11:04, RichardWarren wrote:

> If I might summarise our understanding of the role etc. of the evaluator
> 
> 1) Peter has reminded us of our terms of reference and that it includes reference to "different contexts such as self-assessment or third party evaluation"
> 
> 2) Our target audience (Section 1.2) includes -
> Website developers, suppliers, procurers, and owners.
> Web accessibility consultants and evaluation service providers.
> Web accessibility monitoring and benchmarking activities.
> Web accessibility researchers and disability advocates.
> 
> Having established the breadth of our target audience at the beginning we cannot, half-way through our method, suddenly introduce the concept of only allowing "independent"  or "third-party" evaluators.
> 
> As I understand it our task is to deliver a methodology that helps more people to check if websites comply with WCAG so that the web as a whole becomes more accessible.
> 
> The issue of the validity of conformance claims (transparency etc.) should be a separate discussion
> 
> Regards
> Richard
> 
> 
> 
> -----Original Message----- From: Alistair Garrison
> Sent: Friday, June 22, 2012 8:28 AM
> To: kvotis@iti.gr ; Peter Korn ; Eval TF
> Subject: Re: Aspects of an evaluator
> 
> Dear Eval,
> 
> So, if we don't want do set requirements of independence on the evaluator, and don't want to publish the procedures followed (or any parts of an evaluation report) - how do we ensure public trust in the conformance statement being made?
> 
> All the best
> 
> Alistair
> 
> On 22 Jun 2012, at 08:19, kvotis@iti.gr wrote:
> 
>> Dear Alistair,
>> 
>> regarding your following comment:
>> 3) Insist that some parts of every report are made public - not
>> urls, just procedures (as it should be the case that any similar
>> samples of web pages taken from the defined site will lead to pretty
>> much the same result)
>> 
>> 
>> i am not sure about the described procedure and actually regarding  the
>> selection of appropriate parts of Web sites..How someone who is also
>> developer will select which of the parts should be selected?Moreover i
>> agree with Peter regarding the publicity of all evaluation reports. From
>> my point of view this is a difficult procedure while it requires heavy
>> storage and processing mechanisms
>> 
>> regards
>> 
>> Kostas
>> 
>> 
>> 
>> -------------------
>> Dr. Konstantinos Votis
>> Computer Engineer & Informatics,PhD, Msc, MBA
>> Research Associate
>> Informatics and Telematics Institute
>> Centre for Research and Technology Hellas
>> 6th Klm. Charilaou - Thermi Road
>> P.O. BOX 60361 GR - 570 01
>> Thessaloniki – Greece
>> Tel.: +30-2311-257722
>> Fax : +30-2310-474128
>> E-mail : kvotis@iti.gr
>> 
>> 
>> 
>> 
>>> Alistair,
>>> 
>>> I don't see how we can insist that all evaluation reports be public, or
>>> that certain parts of an evaluation report be public.  We can say that
>>> an evaluation report isn't complete unless it has all of the mandatory
>>> parts.  But we can't stop someone from producing a summary of the
>>> report, or excerpting parts of a report, or producing a description of
>>> the report, or...
>>> 
>>> Not unless we require copyright on all reports generated by anyone else
>>> and license it and...  (and even then "fair use" doctrines in many
>>> countries would likely still allow publishing excerpts).
>>> 
>>> 
>>> Peter
>>> 
>>> 
>>> On 6/21/2012 1:31 PM, Alistair Garrison wrote:
>>>> Hi Peter,
>>>> 
>>>> Trust issues aside? I'd be really interested to hear your thoughts on
>>>> the 3rd option I presented?
>>>> 
>>>> All the best
>>>> 
>>>> Alistair
>>>> 
>>>> 
>>>> On 21 Jun 2012, at 22:26, Peter Korn wrote:
>>>> 
>>>>> Alistair,
>>>>> 
>>>>> I think the question of trust is, frankly, outside of the scope of
>>>>> our work as well.  Nothing prevents someone from lying - whether or
>>>>> not they are independent.  They may make honest mistakes because they
>>>>> lack the technical expertise needed to do a good job (again whether
>>>>> independent or not). They may claim their sample is representative
>>>>> but it isn't. They may claim something failed when it passed (or
>>>>> vice-versa).
>>>>> 
>>>>> Being independent doesn't prevent any of that.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Peter
>>>>> 
>>>>> On 6/21/2012 12:35 PM, Alistair Garrison wrote:
>>>>>> Hi Peter,
>>>>>> 
>>>>>> Your point about the single person with the massive website is well
>>>>>> made - and moves forcibly against the idea of independence and my
>>>>>> early suggested aspects? And, in this context - I can see clearly
>>>>>> why you think the question of the independence/inter-dependence of
>>>>>> an evaluator from the site being evaluated is outside of the scope
>>>>>> of our charter.  But, I'm still not 100% convinced? 99% maybe ;-)
>>>>>> 
>>>>>> To my mind, the question is about trust - will the public place an
>>>>>> equal amount of trust in an evaluation done by a 1st party, as they
>>>>>> would a 3rd party?
>>>>>> 
>>>>>> Well, I think yes they could?
>>>>>> 
>>>>>> But, in our situation it might only be achieved under certain
>>>>>> circumstances?  Seemingly, a number of options exist:
>>>>>> 
>>>>>> 1) Insist that the whole evaluation report be published, along with
>>>>>> the urls tested, procedure, etc...  The public could recreate your
>>>>>> tests and confirm your findings; or
>>>>>> 
>>>>>> 2) Leave the publishing decision to the evaluation commissioner, and
>>>>>> instead set requirements for independence on the evaluator.
>>>>>> 
>>>>>> Note: In our methodology we already state "documentation need not
>>>>>> necessarily be public, as disclosure is up to the owner and/or
>>>>>> evaluation commissioner" - presumably you would not wish to publish
>>>>>> if you have assessed restricted areas of the website (which is a big
>>>>>> issue with option 1, amongst others)...
>>>>>> 
>>>>>> So? We already seem to be looking at the second option - hence the
>>>>>> reason I'm not yet 100% convinced that the question of evaluator
>>>>>> independence is currently entirely out of scope; or
>>>>>> 
>>>>>> 3) Insist that some parts of every report are made public - not
>>>>>> urls, just procedures (as it should be the case that any similar
>>>>>> samples of web pages taken from the defined site will lead to pretty
>>>>>> much the same result)?
>>>>>> 
>>>>>> In light of your point, and the issues with option 1 further
>>>>>> investigation into option 3 might be an idea - it would of course
>>>>>> mean changes to 5a, and removing the idea of independence from the
>>>>>> scope.
>>>>>> 
>>>>>> Thoughts / comments welcome?
>>>>>> All the best
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> On 21 Jun 2012, at 20:17, Peter Korn wrote:
>>>>>> 
>>>>>>> Alistair,
>>>>>>> 
>>>>>>> If I am the only person involved in creating my own, fair sized
>>>>>>> website (too large to feasibly evaluate every single page, being as
>>>>>>> it is programmatically generated, etc. etc.), then
>>>>>>> "self-assessment" means that I am also the assessor.  I cannot be
>>>>>>> an ISO 9001:2000 compliant internal auditor.
>>>>>>> 
>>>>>>> Separate from that example, I don't understand why EvalTF should be
>>>>>>> concerning itself with ISO 900x in any way.  Looking again at the
>>>>>>> Objective portion of the Work Statement
>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our
>>>>>>> mandate is for a technical task (as I understand it): how to select
>>>>>>> a representative sample of a site, how to aggregate results into an
>>>>>>> overall conformance statement, etc.
>>>>>>> 
>>>>>>> The question of the independence/inter-dependence of an evaluator
>>>>>>> from the site being evaluated is outside of the scope of our charter.
>>>>>>> 
>>>>>>> 
>>>>>>> Peter
>>>>>>> 
>>>>>>> 
>>>>>>> On 6/21/2012 11:06 AM, Alistair Garrison wrote:
>>>>>>>> Hi Peter,
>>>>>>>> 
>>>>>>>> Supporting 1st party assessment is as important to me as
>>>>>>>> supporting 3rd party assessment - which is why I based my proposal
>>>>>>>> on those well documented aspects you would look for in an internal
>>>>>>>> auditor for ISO 9001:2000.
>>>>>>>> 
>>>>>>>> Maybe, for clarity, it should have been 'not associated in their
>>>>>>>> day to day role with' - I think you have read 'associated' in the
>>>>>>>> same light as 'independent'.
>>>>>>>> 
>>>>>>>> Hope this helps.
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> On 21 Jun 2012, at 19:36, Peter Korn wrote:
>>>>>>>> 
>>>>>>>>> Alistair,
>>>>>>>>> 
>>>>>>>>> It wasn't clear to me that this was the outcome of our meeting.
>>>>>>>>> 
>>>>>>>>> Reviewing the EvalTF Work Statement
>>>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first sentence
>>>>>>>>> of the Objective reads (*/emphasis added/*): "objective of Eval
>>>>>>>>> TF is to develop an internationally harmonized methodology for
>>>>>>>>> evaluating the conformance of websites to WCAG 2.0,/*that
>>>>>>>>> supports different contexts, such as for self-assessment or
>>>>>>>>> third-party evaluation*/ of small or larger websites".
>>>>>>>>> 
>>>>>>>>> If the methodology is to support self-assessment, then it cannot
>>>>>>>>> define the evaluator as be different from the
>>>>>>>>> developer/maintainer/accessibility-expert for the site.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> 
>>>>>>>>> Peter
>>>>>>>>> 
>>>>>>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote:
>>>>>>>>>> Dear all,
>>>>>>>>>> 
>>>>>>>>>> In today's telecon, we discussed terms like independent when
>>>>>>>>>> talking about evaluations.  The outcome appeared to be that what
>>>>>>>>>> was needed was in fact a better definition for 'evaluator'.
>>>>>>>>>> 
>>>>>>>>>> I'm not going to propose the whole definition for 'evaluator',
>>>>>>>>>> however, just two aspects which we might consider including in the
>>>>>>>>>> definition:
>>>>>>>>>> 
>>>>>>>>>> Aspect 1) (of an evaluator) someone who is not responsible for the
>>>>>>>>>> accessibility of the website being evaluated.
>>>>>>>>>> Aspect 2) (of an evaluator) someone who is not associated with
>>>>>>>>>> developing and maintaining the website or its content.
>>>>>>>>>> 
>>>>>>>>>> Thoughts?
>>>>>>>>>> 
>>>>>>>>>> All the best
>>>>>>>>>> 
>>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>>>>>>> Peter Korn | Accessibility Principal
>>>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
>>>>>>>>> Oracle is committed to developing practices and products that
>>>>>>>>> help protect the environment
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>>>>> Peter Korn | Accessibility Principal
>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
>>>>>>> Oracle is committed to developing practices and products that help
>>>>>>> protect the environment
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> --
>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>>> Peter Korn | Accessibility Principal
>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> Oracle
>>>>> is committed to developing practices and products that help protect
>>>>> the environment
>>>>> 
>>>>> 
>>>> 
>>> 
>>> --
>>> Oracle <http://www.oracle.com>
>>> Peter Korn | Accessibility Principal
>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>> 500 Oracle Parkway | Redwood City, CA 94065
>>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
>>> developing practices and products that help protect the environment
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
>> 
> 
>