W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > June 2012

Re: Aspects of an evaluator

From: Detlev Fischer <detlev.fischer@testkreis.de>
Date: Fri, 22 Jun 2012 10:14:17 +0200
To: Alistair Garrison <alistair.j.garrison@gmail.com>
Message-Id: <5EAD3EF4-8D21-468E-A5F6-FF3B2DBBBB74@testkreis.de>
Cc: kvotis@iti.gr, Peter Korn <peter.korn@oracle.com>, Eval TF <public-wai-evaltf@w3.org>
THe discussion yesterday has made me think again about the  
requirements for transparency of testing and the independence of  
evaluators.

At bottom, a conformance claim is just that: a claim.

There may not be much trust in undocumented claims but as long as it  
is plain that an evaluation was a non-public self-assessment, there is  
little harm.

If a claim can be replicated with access to the documented sample, it  
will appear stronger. But, as Peter noted yesterday, this says nothing  
about the quality of the claim itself - a non-public self assessment  
may be more thorough and more correct than a documented external  
assessment.

If removing the requirement for independence of evaluator and some  
level of public documentation ensures a greater appeal and uptake of  
WCAG-EM especially in the private sector, this is probably reason  
enough to support it.

Let's not forget that evaluators / evaluating organisations ate free  
to build those layers (documentation, replication/re-test, etc) on top  
of WCAG-EM.

Regards,
Detlev

On 22 Jun 2012, at 09:28, Alistair Garrison wrote:

> Dear Eval,
>
> So, if we don't want do set requirements of independence on the  
> evaluator, and don't want to publish the procedures followed (or any  
> parts of an evaluation report) - how do we ensure public trust in  
> the conformance statement being made?
>
> All the best
>
> Alistair
>
> On 22 Jun 2012, at 08:19, kvotis@iti.gr wrote:
>
>> Dear Alistair,
>>
>> regarding your following comment:
>> 3) Insist that some parts of every report are made public - not
>> urls, just procedures (as it should be the case that any similar
>> samples of web pages taken from the defined site will lead to pretty
>> much the same result)
>>
>>
>> i am not sure about the described procedure and actually regarding   
>> the
>> selection of appropriate parts of Web sites..How someone who is also
>> developer will select which of the parts should be selected? 
>> Moreover i
>> agree with Peter regarding the publicity of all evaluation reports.  
>> From
>> my point of view this is a difficult procedure while it requires  
>> heavy
>> storage and processing mechanisms
>>
>> regards
>>
>> Kostas
>>
>>
>>
>> -------------------
>> Dr. Konstantinos Votis
>> Computer Engineer & Informatics,PhD, Msc, MBA
>> Research Associate
>> Informatics and Telematics Institute
>> Centre for Research and Technology Hellas
>> 6th Klm. Charilaou - Thermi Road
>> P.O. BOX 60361 GR - 570 01
>> Thessaloniki &#8211; Greece
>> Tel.: +30-2311-257722
>> Fax : +30-2310-474128
>> E-mail : kvotis@iti.gr
>>
>>
>>
>>
>>> Alistair,
>>>
>>> I don't see how we can insist that all evaluation reports be  
>>> public, or
>>> that certain parts of an evaluation report be public.  We can say  
>>> that
>>> an evaluation report isn't complete unless it has all of the  
>>> mandatory
>>> parts.  But we can't stop someone from producing a summary of the
>>> report, or excerpting parts of a report, or producing a  
>>> description of
>>> the report, or...
>>>
>>> Not unless we require copyright on all reports generated by anyone  
>>> else
>>> and license it and...  (and even then "fair use" doctrines in many
>>> countries would likely still allow publishing excerpts).
>>>
>>>
>>> Peter
>>>
>>>
>>> On 6/21/2012 1:31 PM, Alistair Garrison wrote:
>>>> Hi Peter,
>>>>
>>>> Trust issues aside? I'd be really interested to hear your  
>>>> thoughts on
>>>> the 3rd option I presented?
>>>>
>>>> All the best
>>>>
>>>> Alistair
>>>>
>>>>
>>>> On 21 Jun 2012, at 22:26, Peter Korn wrote:
>>>>
>>>>> Alistair,
>>>>>
>>>>> I think the question of trust is, frankly, outside of the scope of
>>>>> our work as well.  Nothing prevents someone from lying - whether  
>>>>> or
>>>>> not they are independent.  They may make honest mistakes because  
>>>>> they
>>>>> lack the technical expertise needed to do a good job (again  
>>>>> whether
>>>>> independent or not). They may claim their sample is representative
>>>>> but it isn't. They may claim something failed when it passed (or
>>>>> vice-versa).
>>>>>
>>>>> Being independent doesn't prevent any of that.
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Peter
>>>>>
>>>>> On 6/21/2012 12:35 PM, Alistair Garrison wrote:
>>>>>> Hi Peter,
>>>>>>
>>>>>> Your point about the single person with the massive website is  
>>>>>> well
>>>>>> made - and moves forcibly against the idea of independence and my
>>>>>> early suggested aspects? And, in this context - I can see clearly
>>>>>> why you think the question of the independence/inter-dependence  
>>>>>> of
>>>>>> an evaluator from the site being evaluated is outside of the  
>>>>>> scope
>>>>>> of our charter.  But, I'm still not 100% convinced? 99% maybe ;-)
>>>>>>
>>>>>> To my mind, the question is about trust - will the public place  
>>>>>> an
>>>>>> equal amount of trust in an evaluation done by a 1st party, as  
>>>>>> they
>>>>>> would a 3rd party?
>>>>>>
>>>>>> Well, I think yes they could?
>>>>>>
>>>>>> But, in our situation it might only be achieved under certain
>>>>>> circumstances?  Seemingly, a number of options exist:
>>>>>>
>>>>>> 1) Insist that the whole evaluation report be published, along  
>>>>>> with
>>>>>> the urls tested, procedure, etc...  The public could recreate  
>>>>>> your
>>>>>> tests and confirm your findings; or
>>>>>>
>>>>>> 2) Leave the publishing decision to the evaluation  
>>>>>> commissioner, and
>>>>>> instead set requirements for independence on the evaluator.
>>>>>>
>>>>>> Note: In our methodology we already state "documentation need not
>>>>>> necessarily be public, as disclosure is up to the owner and/or
>>>>>> evaluation commissioner" - presumably you would not wish to  
>>>>>> publish
>>>>>> if you have assessed restricted areas of the website (which is  
>>>>>> a big
>>>>>> issue with option 1, amongst others)...
>>>>>>
>>>>>> So? We already seem to be looking at the second option - hence  
>>>>>> the
>>>>>> reason I'm not yet 100% convinced that the question of evaluator
>>>>>> independence is currently entirely out of scope; or
>>>>>>
>>>>>> 3) Insist that some parts of every report are made public - not
>>>>>> urls, just procedures (as it should be the case that any similar
>>>>>> samples of web pages taken from the defined site will lead to  
>>>>>> pretty
>>>>>> much the same result)?
>>>>>>
>>>>>> In light of your point, and the issues with option 1 further
>>>>>> investigation into option 3 might be an idea - it would of course
>>>>>> mean changes to 5a, and removing the idea of independence from  
>>>>>> the
>>>>>> scope.
>>>>>>
>>>>>> Thoughts / comments welcome?
>>>>>> All the best
>>>>>>
>>>>>> Alistair
>>>>>>
>>>>>> On 21 Jun 2012, at 20:17, Peter Korn wrote:
>>>>>>
>>>>>>> Alistair,
>>>>>>>
>>>>>>> If I am the only person involved in creating my own, fair sized
>>>>>>> website (too large to feasibly evaluate every single page,  
>>>>>>> being as
>>>>>>> it is programmatically generated, etc. etc.), then
>>>>>>> "self-assessment" means that I am also the assessor.  I cannot  
>>>>>>> be
>>>>>>> an ISO 9001:2000 compliant internal auditor.
>>>>>>>
>>>>>>> Separate from that example, I don't understand why EvalTF  
>>>>>>> should be
>>>>>>> concerning itself with ISO 900x in any way.  Looking again at  
>>>>>>> the
>>>>>>> Objective portion of the Work Statement
>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our
>>>>>>> mandate is for a technical task (as I understand it): how to  
>>>>>>> select
>>>>>>> a representative sample of a site, how to aggregate results  
>>>>>>> into an
>>>>>>> overall conformance statement, etc.
>>>>>>>
>>>>>>> The question of the independence/inter-dependence of an  
>>>>>>> evaluator
>>>>>>> from the site being evaluated is outside of the scope of our  
>>>>>>> charter.
>>>>>>>
>>>>>>>
>>>>>>> Peter
>>>>>>>
>>>>>>>
>>>>>>> On 6/21/2012 11:06 AM, Alistair Garrison wrote:
>>>>>>>> Hi Peter,
>>>>>>>>
>>>>>>>> Supporting 1st party assessment is as important to me as
>>>>>>>> supporting 3rd party assessment - which is why I based my  
>>>>>>>> proposal
>>>>>>>> on those well documented aspects you would look for in an  
>>>>>>>> internal
>>>>>>>> auditor for ISO 9001:2000.
>>>>>>>>
>>>>>>>> Maybe, for clarity, it should have been 'not associated in  
>>>>>>>> their
>>>>>>>> day to day role with' - I think you have read 'associated' in  
>>>>>>>> the
>>>>>>>> same light as 'independent'.
>>>>>>>>
>>>>>>>> Hope this helps.
>>>>>>>>
>>>>>>>> Alistair
>>>>>>>>
>>>>>>>> On 21 Jun 2012, at 19:36, Peter Korn wrote:
>>>>>>>>
>>>>>>>>> Alistair,
>>>>>>>>>
>>>>>>>>> It wasn't clear to me that this was the outcome of our  
>>>>>>>>> meeting.
>>>>>>>>>
>>>>>>>>> Reviewing the EvalTF Work Statement
>>>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first  
>>>>>>>>> sentence
>>>>>>>>> of the Objective reads (*/emphasis added/*): "objective of  
>>>>>>>>> Eval
>>>>>>>>> TF is to develop an internationally harmonized methodology for
>>>>>>>>> evaluating the conformance of websites to WCAG 2.0,/*that
>>>>>>>>> supports different contexts, such as for self-assessment or
>>>>>>>>> third-party evaluation*/ of small or larger websites".
>>>>>>>>>
>>>>>>>>> If the methodology is to support self-assessment, then it  
>>>>>>>>> cannot
>>>>>>>>> define the evaluator as be different from the
>>>>>>>>> developer/maintainer/accessibility-expert for the site.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Peter
>>>>>>>>>
>>>>>>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote:
>>>>>>>>>> Dear all,
>>>>>>>>>>
>>>>>>>>>> In today's telecon, we discussed terms like independent when
>>>>>>>>>> talking about evaluations.  The outcome appeared to be that  
>>>>>>>>>> what
>>>>>>>>>> was needed was in fact a better definition for 'evaluator'.
>>>>>>>>>>
>>>>>>>>>> I'm not going to propose the whole definition for  
>>>>>>>>>> 'evaluator',
>>>>>>>>>> however, just two aspects which we might consider including  
>>>>>>>>>> in the
>>>>>>>>>> definition:
>>>>>>>>>>
>>>>>>>>>> Aspect 1) (of an evaluator) someone who is not responsible  
>>>>>>>>>> for the
>>>>>>>>>> accessibility of the website being evaluated.
>>>>>>>>>> Aspect 2) (of an evaluator) someone who is not associated  
>>>>>>>>>> with
>>>>>>>>>> developing and maintaining the website or its content.
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>>
>>>>>>>>>> All the best
>>>>>>>>>>
>>>>>>>>>> Alistair
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>>>>>>> Peter Korn | Accessibility Principal
>>>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
>>>>>>>>> Oracle is committed to developing practices and products that
>>>>>>>>> help protect the environment
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>>>>> Peter Korn | Accessibility Principal
>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
>>>>>>> Oracle is committed to developing practices and products that  
>>>>>>> help
>>>>>>> protect the environment
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/>
>>>>> Peter Korn | Accessibility Principal
>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>>>> 500 Oracle Parkway | Redwood City, CA 94065
>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>  
>>>>> Oracle
>>>>> is committed to developing practices and products that help  
>>>>> protect
>>>>> the environment
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Oracle <http://www.oracle.com>
>>> Peter Korn | Accessibility Principal
>>> Phone: +1 650 5069522 <tel:+1%20650%205069522>
>>> 500 Oracle Parkway | Redwood City, CA 94065
>>> Green Oracle <http://www.oracle.com/commitment> Oracle is  
>>> committed to
>>> developing practices and products that help protect the environment
>>>
>>>
>>>
>>
>>
>>
>>
>>
>

-- 
Detlev Fischer
testkreis - das Accessibility-Team von feld.wald.wiese
c/o feld.wald.wiese
Borselstraße 3-7 (im Hof)
22765 Hamburg

Tel   +49 (0)40 439 10 68-3
Mobil +49 (0)1577 170 73 84
Fax   +49 (0)40 439 10 68-5

http://www.testkreis.de
Beratung, Tests und Schulungen für barrierefreie Websites
Received on Friday, 22 June 2012 08:05:26 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:14 GMT