W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > June 2012

Re: Aspects of an evaluator

From: Alistair Garrison <alistair.j.garrison@gmail.com>
Date: Fri, 22 Jun 2012 09:28:47 +0200
Message-Id: <1AE0E34D-53F5-491B-9623-79E6E26CF400@gmail.com>
To: kvotis@iti.gr, Peter Korn <peter.korn@oracle.com>, Eval TF <public-wai-evaltf@w3.org>
Dear Eval, 

So, if we don't want do set requirements of independence on the evaluator, and don't want to publish the procedures followed (or any parts of an evaluation report) - how do we ensure public trust in the conformance statement being made?

All the best 

Alistair

On 22 Jun 2012, at 08:19, kvotis@iti.gr wrote:

> Dear Alistair,
> 
> regarding your following comment:
> 3) Insist that some parts of every report are made public - not
> urls, just procedures (as it should be the case that any similar
> samples of web pages taken from the defined site will lead to pretty
> much the same result)
> 
> 
> i am not sure about the described procedure and actually regarding  the
> selection of appropriate parts of Web sites..How someone who is also
> developer will select which of the parts should be selected?Moreover i
> agree with Peter regarding the publicity of all evaluation reports. From
> my point of view this is a difficult procedure while it requires heavy
> storage and processing mechanisms
> 
> regards
> 
> Kostas
> 
> 
> 
> -------------------
> Dr. Konstantinos Votis
> Computer Engineer & Informatics,PhD, Msc, MBA
> Research Associate
> Informatics and Telematics Institute
> Centre for Research and Technology Hellas
> 6th Klm. Charilaou - Thermi Road
> P.O. BOX 60361 GR - 570 01
> Thessaloniki &#8211; Greece
> Tel.: +30-2311-257722
> Fax : +30-2310-474128
> E-mail : kvotis@iti.gr
> 
> 
> 
> 
>> Alistair,
>> 
>> I don't see how we can insist that all evaluation reports be public, or
>> that certain parts of an evaluation report be public.  We can say that
>> an evaluation report isn't complete unless it has all of the mandatory
>> parts.  But we can't stop someone from producing a summary of the
>> report, or excerpting parts of a report, or producing a description of
>> the report, or...
>> 
>> Not unless we require copyright on all reports generated by anyone else
>> and license it and...  (and even then "fair use" doctrines in many
>> countries would likely still allow publishing excerpts).
>> 
>> 
>> Peter
>> 
>> 
>> On 6/21/2012 1:31 PM, Alistair Garrison wrote:
>>> Hi Peter,
>>> 
>>> Trust issues aside? I'd be really interested to hear your thoughts on
>>> the 3rd option I presented?
>>> 
>>> All the best
>>> 
>>> Alistair
>>> 
>>> 
>>> On 21 Jun 2012, at 22:26, Peter Korn wrote:
>>> 
>>>> Alistair,
>>>> 
>>>> I think the question of trust is, frankly, outside of the scope of
>>>> our work as well.  Nothing prevents someone from lying - whether or
>>>> not they are independent.  They may make honest mistakes because they
>>>> lack the technical expertise needed to do a good job (again whether
>>>> independent or not). They may claim their sample is representative
>>>> but it isn't. They may claim something failed when it passed (or
>>>> vice-versa).
>>>> 
>>>> Being independent doesn't prevent any of that.
>>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> Peter
>>>> 
>>>> On 6/21/2012 12:35 PM, Alistair Garrison wrote:
>>>>> Hi Peter,
>>>>> 
>>>>> Your point about the single person with the massive website is well
>>>>> made - and moves forcibly against the idea of independence and my
>>>>> early suggested aspects? And, in this context - I can see clearly
>>>>> why you think the question of the independence/inter-dependence of
>>>>> an evaluator from the site being evaluated is outside of the scope
>>>>> of our charter.  But, I'm still not 100% convinced? 99% maybe ;-)
>>>>> 
>>>>> To my mind, the question is about trust - will the public place an
>>>>> equal amount of trust in an evaluation done by a 1st party, as they
>>>>> would a 3rd party?
>>>>> 
>>>>> Well, I think yes they could?
>>>>> 
>>>>> But, in our situation it might only be achieved under certain
>>>>> circumstances?  Seemingly, a number of options exist:
>>>>> 
>>>>> 1) Insist that the whole evaluation report be published, along with
>>>>> the urls tested, procedure, etc...  The public could recreate your
>>>>> tests and confirm your findings; or
>>>>> 
>>>>> 2) Leave the publishing decision to the evaluation commissioner, and
>>>>> instead set requirements for independence on the evaluator.
>>>>> 
>>>>> Note: In our methodology we already state "documentation need not
>>>>> necessarily be public, as disclosure is up to the owner and/or
>>>>> evaluation commissioner" - presumably you would not wish to publish
>>>>> if you have assessed restricted areas of the website (which is a big
>>>>> issue with option 1, amongst others)...
>>>>> 
>>>>> So? We already seem to be looking at the second option - hence the
>>>>> reason I'm not yet 100% convinced that the question of evaluator
>>>>> independence is currently entirely out of scope; or
>>>>> 
>>>>> 3) Insist that some parts of every report are made public - not
>>>>> urls, just procedures (as it should be the case that any similar
>>>>> samples of web pages taken from the defined site will lead to pretty
>>>>> much the same result)?
>>>>> 
>>>>> In light of your point, and the issues with option 1 further
>>>>> investigation into option 3 might be an idea - it would of course
>>>>> mean changes to 5a, and removing the idea of independence from the
>>>>> scope.
>>>>> 
>>>>> Thoughts / comments welcome?
>>>>> All the best
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> On 21 Jun 2012, at 20:17, Peter Korn wrote:
>>>>> 
>>>>>> Alistair,
>>>>>> 
>>>>>> If I am the only person involved in creating my own, fair sized
>>>>>> website (too large to feasibly evaluate every single page, being as
>>>>>> it is programmatically generated, etc. etc.), then
>>>>>> "self-assessment" means that I am also the assessor.  I cannot be
>>>>>> an ISO 9001:2000 compliant internal auditor.
>>>>>> 
>>>>>> Separate from that example, I don't understand why EvalTF should be
>>>>>> concerning itself with ISO 900x in any way.  Looking again at the
>>>>>> Objective portion of the Work Statement
>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our
>>>>>> mandate is for a technical task (as I understand it): how to select
>>>>>> a representative sample of a site, how to aggregate results into an
>>>>>> overall conformance statement, etc.
>>>>>> 
>>>>>> The question of the independence/inter-dependence of an evaluator
>>>>>> from the site being evaluated is outside of the scope of our charter.
>>>>>> 
>>>>>> 
>>>>>> Peter
>>>>>> 
>>>>>> 
>>>>>> On 6/21/2012 11:06 AM, Alistair Garrison wrote:
>>>>>>> Hi Peter,
>>>>>>> 
>>>>>>> Supporting 1st party assessment is as important to me as
>>>>>>> supporting 3rd party assessment - which is why I based my proposal
>>>>>>> on those well documented aspects you would look for in an internal
>>>>>>> auditor for ISO 9001:2000.
>>>>>>> 
>>>>>>> Maybe, for clarity, it should have been 'not associated in their
>>>>>>> day to day role with' - I think you have read 'associated' in the
>>>>>>> same light as 'independent'.
>>>>>>> 
>>>>>>> Hope this helps.
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>>> On 21 Jun 2012, at 19:36, Peter Korn wrote:
>>>>>>> 
>>>>>>>> Alistair,
>>>>>>>> 
>>>>>>>> It wasn't clear to me that this was the outcome of our meeting.
>>>>>>>> 
>>>>>>>> Reviewing the EvalTF Work Statement
>>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first sentence
>>>>>>>> of the Objective reads (*/emphasis added/*): "objective of Eval
>>>>>>>> TF is to develop an internationally harmonized methodology for
>>>>>>>> evaluating the conformance of websites to WCAG 2.0,/*that
>>>>>>>> supports different contexts, such as for self-assessment or
>>>>>>>> third-party evaluation*/ of small or larger websites".
>>>>>>>> 
>>>>>>>> If the methodology is to support self-assessment, then it cannot
>>>>>>>> define the evaluator as be different from the
>>>>>>>> developer/maintainer/accessibility-expert for the site.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Peter
>>>>>>>> 
>>>>>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote:
>>>>>>>>> Dear all,
>>>>>>>>> 
>>>>>>>>> In today's telecon, we discussed terms like independent when
>>>>>>>>> talking about evaluations.  The outcome appeared to be that what
>>>>>>>>> was needed was in fact a better definition for 'evaluator'.
>>>>>>>>> 
>>>>>>>>> I'm not going to propose the whole definition for 'evaluator',
>>>>>>>>> however, just two aspects which we might consider including in the
>>>>>>>>> definition:
>>>>>>>>> 
>>>>>>>>> Aspect 1) (of an evaluator) someone who is not responsible for the
>>>>>>>>> accessibility of the website being evaluated.
>>>>>>>>> Aspect 2) (of an evaluator) someone who is not associated with
>>>>>>>>> developing and maintaining the website or its content.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> 
>>>>>>>>> All the best
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> --
>>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>>>>>> Peter Korn | Accessibility Principal
>>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
>>>>>>>> Oracle is committed to developing practices and products that
>>>>>>>> help protect the environment
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>>>> Peter Korn | Accessibility Principal
>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
>>>>>> Oracle is committed to developing practices and products that help
>>>>>> protect the environment
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>> Peter Korn | Accessibility Principal
>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> Oracle
>>>> is committed to developing practices and products that help protect
>>>> the environment
>>>> 
>>>> 
>>> 
>> 
>> --
>> Oracle <http://www.oracle.com>
>> Peter Korn | Accessibility Principal
>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>> 500 Oracle Parkway | Redwood City, CA 94065
>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
>> developing practices and products that help protect the environment
>> 
>> 
>> 
> 
> 
> 
> 
> 
Received on Friday, 22 June 2012 07:29:21 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:14 GMT