- From: Detlev Fischer <detlev.fischer@testkreis.de>
- Date: Fri, 22 Jun 2012 10:14:17 +0200
- To: Alistair Garrison <alistair.j.garrison@gmail.com>
- Cc: kvotis@iti.gr, Peter Korn <peter.korn@oracle.com>, Eval TF <public-wai-evaltf@w3.org>
THe discussion yesterday has made me think again about the requirements for transparency of testing and the independence of evaluators. At bottom, a conformance claim is just that: a claim. There may not be much trust in undocumented claims but as long as it is plain that an evaluation was a non-public self-assessment, there is little harm. If a claim can be replicated with access to the documented sample, it will appear stronger. But, as Peter noted yesterday, this says nothing about the quality of the claim itself - a non-public self assessment may be more thorough and more correct than a documented external assessment. If removing the requirement for independence of evaluator and some level of public documentation ensures a greater appeal and uptake of WCAG-EM especially in the private sector, this is probably reason enough to support it. Let's not forget that evaluators / evaluating organisations ate free to build those layers (documentation, replication/re-test, etc) on top of WCAG-EM. Regards, Detlev On 22 Jun 2012, at 09:28, Alistair Garrison wrote: > Dear Eval, > > So, if we don't want do set requirements of independence on the > evaluator, and don't want to publish the procedures followed (or any > parts of an evaluation report) - how do we ensure public trust in > the conformance statement being made? > > All the best > > Alistair > > On 22 Jun 2012, at 08:19, kvotis@iti.gr wrote: > >> Dear Alistair, >> >> regarding your following comment: >> 3) Insist that some parts of every report are made public - not >> urls, just procedures (as it should be the case that any similar >> samples of web pages taken from the defined site will lead to pretty >> much the same result) >> >> >> i am not sure about the described procedure and actually regarding >> the >> selection of appropriate parts of Web sites..How someone who is also >> developer will select which of the parts should be selected? >> Moreover i >> agree with Peter regarding the publicity of all evaluation reports. >> From >> my point of view this is a difficult procedure while it requires >> heavy >> storage and processing mechanisms >> >> regards >> >> Kostas >> >> >> >> ------------------- >> Dr. Konstantinos Votis >> Computer Engineer & Informatics,PhD, Msc, MBA >> Research Associate >> Informatics and Telematics Institute >> Centre for Research and Technology Hellas >> 6th Klm. Charilaou - Thermi Road >> P.O. BOX 60361 GR - 570 01 >> Thessaloniki – Greece >> Tel.: +30-2311-257722 >> Fax : +30-2310-474128 >> E-mail : kvotis@iti.gr >> >> >> >> >>> Alistair, >>> >>> I don't see how we can insist that all evaluation reports be >>> public, or >>> that certain parts of an evaluation report be public. We can say >>> that >>> an evaluation report isn't complete unless it has all of the >>> mandatory >>> parts. But we can't stop someone from producing a summary of the >>> report, or excerpting parts of a report, or producing a >>> description of >>> the report, or... >>> >>> Not unless we require copyright on all reports generated by anyone >>> else >>> and license it and... (and even then "fair use" doctrines in many >>> countries would likely still allow publishing excerpts). >>> >>> >>> Peter >>> >>> >>> On 6/21/2012 1:31 PM, Alistair Garrison wrote: >>>> Hi Peter, >>>> >>>> Trust issues aside? I'd be really interested to hear your >>>> thoughts on >>>> the 3rd option I presented? >>>> >>>> All the best >>>> >>>> Alistair >>>> >>>> >>>> On 21 Jun 2012, at 22:26, Peter Korn wrote: >>>> >>>>> Alistair, >>>>> >>>>> I think the question of trust is, frankly, outside of the scope of >>>>> our work as well. Nothing prevents someone from lying - whether >>>>> or >>>>> not they are independent. They may make honest mistakes because >>>>> they >>>>> lack the technical expertise needed to do a good job (again >>>>> whether >>>>> independent or not). They may claim their sample is representative >>>>> but it isn't. They may claim something failed when it passed (or >>>>> vice-versa). >>>>> >>>>> Being independent doesn't prevent any of that. >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Peter >>>>> >>>>> On 6/21/2012 12:35 PM, Alistair Garrison wrote: >>>>>> Hi Peter, >>>>>> >>>>>> Your point about the single person with the massive website is >>>>>> well >>>>>> made - and moves forcibly against the idea of independence and my >>>>>> early suggested aspects? And, in this context - I can see clearly >>>>>> why you think the question of the independence/inter-dependence >>>>>> of >>>>>> an evaluator from the site being evaluated is outside of the >>>>>> scope >>>>>> of our charter. But, I'm still not 100% convinced? 99% maybe ;-) >>>>>> >>>>>> To my mind, the question is about trust - will the public place >>>>>> an >>>>>> equal amount of trust in an evaluation done by a 1st party, as >>>>>> they >>>>>> would a 3rd party? >>>>>> >>>>>> Well, I think yes they could? >>>>>> >>>>>> But, in our situation it might only be achieved under certain >>>>>> circumstances? Seemingly, a number of options exist: >>>>>> >>>>>> 1) Insist that the whole evaluation report be published, along >>>>>> with >>>>>> the urls tested, procedure, etc... The public could recreate >>>>>> your >>>>>> tests and confirm your findings; or >>>>>> >>>>>> 2) Leave the publishing decision to the evaluation >>>>>> commissioner, and >>>>>> instead set requirements for independence on the evaluator. >>>>>> >>>>>> Note: In our methodology we already state "documentation need not >>>>>> necessarily be public, as disclosure is up to the owner and/or >>>>>> evaluation commissioner" - presumably you would not wish to >>>>>> publish >>>>>> if you have assessed restricted areas of the website (which is >>>>>> a big >>>>>> issue with option 1, amongst others)... >>>>>> >>>>>> So? We already seem to be looking at the second option - hence >>>>>> the >>>>>> reason I'm not yet 100% convinced that the question of evaluator >>>>>> independence is currently entirely out of scope; or >>>>>> >>>>>> 3) Insist that some parts of every report are made public - not >>>>>> urls, just procedures (as it should be the case that any similar >>>>>> samples of web pages taken from the defined site will lead to >>>>>> pretty >>>>>> much the same result)? >>>>>> >>>>>> In light of your point, and the issues with option 1 further >>>>>> investigation into option 3 might be an idea - it would of course >>>>>> mean changes to 5a, and removing the idea of independence from >>>>>> the >>>>>> scope. >>>>>> >>>>>> Thoughts / comments welcome? >>>>>> All the best >>>>>> >>>>>> Alistair >>>>>> >>>>>> On 21 Jun 2012, at 20:17, Peter Korn wrote: >>>>>> >>>>>>> Alistair, >>>>>>> >>>>>>> If I am the only person involved in creating my own, fair sized >>>>>>> website (too large to feasibly evaluate every single page, >>>>>>> being as >>>>>>> it is programmatically generated, etc. etc.), then >>>>>>> "self-assessment" means that I am also the assessor. I cannot >>>>>>> be >>>>>>> an ISO 9001:2000 compliant internal auditor. >>>>>>> >>>>>>> Separate from that example, I don't understand why EvalTF >>>>>>> should be >>>>>>> concerning itself with ISO 900x in any way. Looking again at >>>>>>> the >>>>>>> Objective portion of the Work Statement >>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our >>>>>>> mandate is for a technical task (as I understand it): how to >>>>>>> select >>>>>>> a representative sample of a site, how to aggregate results >>>>>>> into an >>>>>>> overall conformance statement, etc. >>>>>>> >>>>>>> The question of the independence/inter-dependence of an >>>>>>> evaluator >>>>>>> from the site being evaluated is outside of the scope of our >>>>>>> charter. >>>>>>> >>>>>>> >>>>>>> Peter >>>>>>> >>>>>>> >>>>>>> On 6/21/2012 11:06 AM, Alistair Garrison wrote: >>>>>>>> Hi Peter, >>>>>>>> >>>>>>>> Supporting 1st party assessment is as important to me as >>>>>>>> supporting 3rd party assessment - which is why I based my >>>>>>>> proposal >>>>>>>> on those well documented aspects you would look for in an >>>>>>>> internal >>>>>>>> auditor for ISO 9001:2000. >>>>>>>> >>>>>>>> Maybe, for clarity, it should have been 'not associated in >>>>>>>> their >>>>>>>> day to day role with' - I think you have read 'associated' in >>>>>>>> the >>>>>>>> same light as 'independent'. >>>>>>>> >>>>>>>> Hope this helps. >>>>>>>> >>>>>>>> Alistair >>>>>>>> >>>>>>>> On 21 Jun 2012, at 19:36, Peter Korn wrote: >>>>>>>> >>>>>>>>> Alistair, >>>>>>>>> >>>>>>>>> It wasn't clear to me that this was the outcome of our >>>>>>>>> meeting. >>>>>>>>> >>>>>>>>> Reviewing the EvalTF Work Statement >>>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first >>>>>>>>> sentence >>>>>>>>> of the Objective reads (*/emphasis added/*): "objective of >>>>>>>>> Eval >>>>>>>>> TF is to develop an internationally harmonized methodology for >>>>>>>>> evaluating the conformance of websites to WCAG 2.0,/*that >>>>>>>>> supports different contexts, such as for self-assessment or >>>>>>>>> third-party evaluation*/ of small or larger websites". >>>>>>>>> >>>>>>>>> If the methodology is to support self-assessment, then it >>>>>>>>> cannot >>>>>>>>> define the evaluator as be different from the >>>>>>>>> developer/maintainer/accessibility-expert for the site. >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Peter >>>>>>>>> >>>>>>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote: >>>>>>>>>> Dear all, >>>>>>>>>> >>>>>>>>>> In today's telecon, we discussed terms like independent when >>>>>>>>>> talking about evaluations. The outcome appeared to be that >>>>>>>>>> what >>>>>>>>>> was needed was in fact a better definition for 'evaluator'. >>>>>>>>>> >>>>>>>>>> I'm not going to propose the whole definition for >>>>>>>>>> 'evaluator', >>>>>>>>>> however, just two aspects which we might consider including >>>>>>>>>> in the >>>>>>>>>> definition: >>>>>>>>>> >>>>>>>>>> Aspect 1) (of an evaluator) someone who is not responsible >>>>>>>>>> for the >>>>>>>>>> accessibility of the website being evaluated. >>>>>>>>>> Aspect 2) (of an evaluator) someone who is not associated >>>>>>>>>> with >>>>>>>>>> developing and maintaining the website or its content. >>>>>>>>>> >>>>>>>>>> Thoughts? >>>>>>>>>> >>>>>>>>>> All the best >>>>>>>>>> >>>>>>>>>> Alistair >>>>>>>>> >>>>>>>>> -- >>>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>>>>> Peter Korn | Accessibility Principal >>>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>>>>>> Oracle is committed to developing practices and products that >>>>>>>>> help protect the environment >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>>> Peter Korn | Accessibility Principal >>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>>>> Oracle is committed to developing practices and products that >>>>>>> help >>>>>>> protect the environment >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>> Peter Korn | Accessibility Principal >>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>> Oracle >>>>> is committed to developing practices and products that help >>>>> protect >>>>> the environment >>>>> >>>>> >>>> >>> >>> -- >>> Oracle <http://www.oracle.com> >>> Peter Korn | Accessibility Principal >>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>> 500 Oracle Parkway | Redwood City, CA 94065 >>> Green Oracle <http://www.oracle.com/commitment> Oracle is >>> committed to >>> developing practices and products that help protect the environment >>> >>> >>> >> >> >> >> >> > -- Detlev Fischer testkreis - das Accessibility-Team von feld.wald.wiese c/o feld.wald.wiese Borselstraße 3-7 (im Hof) 22765 Hamburg Tel +49 (0)40 439 10 68-3 Mobil +49 (0)1577 170 73 84 Fax +49 (0)40 439 10 68-5 http://www.testkreis.de Beratung, Tests und Schulungen für barrierefreie Websites
Received on Friday, 22 June 2012 08:05:26 UTC