- From: Shadi Abou-Zahra <shadi@w3.org>
- Date: Fri, 22 Jun 2012 10:09:48 +0200
- To: Alistair Garrison <alistair.j.garrison@gmail.com>
- CC: Eval TF <public-wai-evaltf@w3.org>
Who said the evaluation is public in the first place; can't I request a confidential evaluation be done for my website according to WCAG-EM? Is it not up to me to select who carries out an evaluation for my website? I think we may be mixing discussion on certification with the technical process for evaluation. IMO these are two quite separable layers. Regards, Shadi On 22.6.2012 09:28, Alistair Garrison wrote: > Dear Eval, > > So, if we don't want do set requirements of independence on the evaluator, and don't want to publish the procedures followed (or any parts of an evaluation report) - how do we ensure public trust in the conformance statement being made? > > All the best > > Alistair > > On 22 Jun 2012, at 08:19, kvotis@iti.gr wrote: > >> Dear Alistair, >> >> regarding your following comment: >> 3) Insist that some parts of every report are made public - not >> urls, just procedures (as it should be the case that any similar >> samples of web pages taken from the defined site will lead to pretty >> much the same result) >> >> >> i am not sure about the described procedure and actually regarding the >> selection of appropriate parts of Web sites..How someone who is also >> developer will select which of the parts should be selected?Moreover i >> agree with Peter regarding the publicity of all evaluation reports. From >> my point of view this is a difficult procedure while it requires heavy >> storage and processing mechanisms >> >> regards >> >> Kostas >> >> >> >> ------------------- >> Dr. Konstantinos Votis >> Computer Engineer & Informatics,PhD, Msc, MBA >> Research Associate >> Informatics and Telematics Institute >> Centre for Research and Technology Hellas >> 6th Klm. Charilaou - Thermi Road >> P.O. BOX 60361 GR - 570 01 >> Thessaloniki – Greece >> Tel.: +30-2311-257722 >> Fax : +30-2310-474128 >> E-mail : kvotis@iti.gr >> >> >> >> >>> Alistair, >>> >>> I don't see how we can insist that all evaluation reports be public, or >>> that certain parts of an evaluation report be public. We can say that >>> an evaluation report isn't complete unless it has all of the mandatory >>> parts. But we can't stop someone from producing a summary of the >>> report, or excerpting parts of a report, or producing a description of >>> the report, or... >>> >>> Not unless we require copyright on all reports generated by anyone else >>> and license it and... (and even then "fair use" doctrines in many >>> countries would likely still allow publishing excerpts). >>> >>> >>> Peter >>> >>> >>> On 6/21/2012 1:31 PM, Alistair Garrison wrote: >>>> Hi Peter, >>>> >>>> Trust issues aside? I'd be really interested to hear your thoughts on >>>> the 3rd option I presented? >>>> >>>> All the best >>>> >>>> Alistair >>>> >>>> >>>> On 21 Jun 2012, at 22:26, Peter Korn wrote: >>>> >>>>> Alistair, >>>>> >>>>> I think the question of trust is, frankly, outside of the scope of >>>>> our work as well. Nothing prevents someone from lying - whether or >>>>> not they are independent. They may make honest mistakes because they >>>>> lack the technical expertise needed to do a good job (again whether >>>>> independent or not). They may claim their sample is representative >>>>> but it isn't. They may claim something failed when it passed (or >>>>> vice-versa). >>>>> >>>>> Being independent doesn't prevent any of that. >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Peter >>>>> >>>>> On 6/21/2012 12:35 PM, Alistair Garrison wrote: >>>>>> Hi Peter, >>>>>> >>>>>> Your point about the single person with the massive website is well >>>>>> made - and moves forcibly against the idea of independence and my >>>>>> early suggested aspects? And, in this context - I can see clearly >>>>>> why you think the question of the independence/inter-dependence of >>>>>> an evaluator from the site being evaluated is outside of the scope >>>>>> of our charter. But, I'm still not 100% convinced? 99% maybe ;-) >>>>>> >>>>>> To my mind, the question is about trust - will the public place an >>>>>> equal amount of trust in an evaluation done by a 1st party, as they >>>>>> would a 3rd party? >>>>>> >>>>>> Well, I think yes they could? >>>>>> >>>>>> But, in our situation it might only be achieved under certain >>>>>> circumstances? Seemingly, a number of options exist: >>>>>> >>>>>> 1) Insist that the whole evaluation report be published, along with >>>>>> the urls tested, procedure, etc... The public could recreate your >>>>>> tests and confirm your findings; or >>>>>> >>>>>> 2) Leave the publishing decision to the evaluation commissioner, and >>>>>> instead set requirements for independence on the evaluator. >>>>>> >>>>>> Note: In our methodology we already state "documentation need not >>>>>> necessarily be public, as disclosure is up to the owner and/or >>>>>> evaluation commissioner" - presumably you would not wish to publish >>>>>> if you have assessed restricted areas of the website (which is a big >>>>>> issue with option 1, amongst others)... >>>>>> >>>>>> So? We already seem to be looking at the second option - hence the >>>>>> reason I'm not yet 100% convinced that the question of evaluator >>>>>> independence is currently entirely out of scope; or >>>>>> >>>>>> 3) Insist that some parts of every report are made public - not >>>>>> urls, just procedures (as it should be the case that any similar >>>>>> samples of web pages taken from the defined site will lead to pretty >>>>>> much the same result)? >>>>>> >>>>>> In light of your point, and the issues with option 1 further >>>>>> investigation into option 3 might be an idea - it would of course >>>>>> mean changes to 5a, and removing the idea of independence from the >>>>>> scope. >>>>>> >>>>>> Thoughts / comments welcome? >>>>>> All the best >>>>>> >>>>>> Alistair >>>>>> >>>>>> On 21 Jun 2012, at 20:17, Peter Korn wrote: >>>>>> >>>>>>> Alistair, >>>>>>> >>>>>>> If I am the only person involved in creating my own, fair sized >>>>>>> website (too large to feasibly evaluate every single page, being as >>>>>>> it is programmatically generated, etc. etc.), then >>>>>>> "self-assessment" means that I am also the assessor. I cannot be >>>>>>> an ISO 9001:2000 compliant internal auditor. >>>>>>> >>>>>>> Separate from that example, I don't understand why EvalTF should be >>>>>>> concerning itself with ISO 900x in any way. Looking again at the >>>>>>> Objective portion of the Work Statement >>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our >>>>>>> mandate is for a technical task (as I understand it): how to select >>>>>>> a representative sample of a site, how to aggregate results into an >>>>>>> overall conformance statement, etc. >>>>>>> >>>>>>> The question of the independence/inter-dependence of an evaluator >>>>>>> from the site being evaluated is outside of the scope of our charter. >>>>>>> >>>>>>> >>>>>>> Peter >>>>>>> >>>>>>> >>>>>>> On 6/21/2012 11:06 AM, Alistair Garrison wrote: >>>>>>>> Hi Peter, >>>>>>>> >>>>>>>> Supporting 1st party assessment is as important to me as >>>>>>>> supporting 3rd party assessment - which is why I based my proposal >>>>>>>> on those well documented aspects you would look for in an internal >>>>>>>> auditor for ISO 9001:2000. >>>>>>>> >>>>>>>> Maybe, for clarity, it should have been 'not associated in their >>>>>>>> day to day role with' - I think you have read 'associated' in the >>>>>>>> same light as 'independent'. >>>>>>>> >>>>>>>> Hope this helps. >>>>>>>> >>>>>>>> Alistair >>>>>>>> >>>>>>>> On 21 Jun 2012, at 19:36, Peter Korn wrote: >>>>>>>> >>>>>>>>> Alistair, >>>>>>>>> >>>>>>>>> It wasn't clear to me that this was the outcome of our meeting. >>>>>>>>> >>>>>>>>> Reviewing the EvalTF Work Statement >>>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first sentence >>>>>>>>> of the Objective reads (*/emphasis added/*): "objective of Eval >>>>>>>>> TF is to develop an internationally harmonized methodology for >>>>>>>>> evaluating the conformance of websites to WCAG 2.0,/*that >>>>>>>>> supports different contexts, such as for self-assessment or >>>>>>>>> third-party evaluation*/ of small or larger websites". >>>>>>>>> >>>>>>>>> If the methodology is to support self-assessment, then it cannot >>>>>>>>> define the evaluator as be different from the >>>>>>>>> developer/maintainer/accessibility-expert for the site. >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Peter >>>>>>>>> >>>>>>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote: >>>>>>>>>> Dear all, >>>>>>>>>> >>>>>>>>>> In today's telecon, we discussed terms like independent when >>>>>>>>>> talking about evaluations. The outcome appeared to be that what >>>>>>>>>> was needed was in fact a better definition for 'evaluator'. >>>>>>>>>> >>>>>>>>>> I'm not going to propose the whole definition for 'evaluator', >>>>>>>>>> however, just two aspects which we might consider including in the >>>>>>>>>> definition: >>>>>>>>>> >>>>>>>>>> Aspect 1) (of an evaluator) someone who is not responsible for the >>>>>>>>>> accessibility of the website being evaluated. >>>>>>>>>> Aspect 2) (of an evaluator) someone who is not associated with >>>>>>>>>> developing and maintaining the website or its content. >>>>>>>>>> >>>>>>>>>> Thoughts? >>>>>>>>>> >>>>>>>>>> All the best >>>>>>>>>> >>>>>>>>>> Alistair >>>>>>>>> >>>>>>>>> -- >>>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>>>>> Peter Korn | Accessibility Principal >>>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>>>>>> Oracle is committed to developing practices and products that >>>>>>>>> help protect the environment >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>>> Peter Korn | Accessibility Principal >>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>>>> Oracle is committed to developing practices and products that help >>>>>>> protect the environment >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>> Peter Korn | Accessibility Principal >>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> Oracle >>>>> is committed to developing practices and products that help protect >>>>> the environment >>>>> >>>>> >>>> >>> >>> -- >>> Oracle <http://www.oracle.com> >>> Peter Korn | Accessibility Principal >>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>> 500 Oracle Parkway | Redwood City, CA 94065 >>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to >>> developing practices and products that help protect the environment >>> >>> >>> >> >> >> >> >> > > > -- Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Activity Lead, W3C/WAI International Program Office Evaluation and Repair Tools Working Group (ERT WG) Research and Development Working Group (RDWG)
Received on Friday, 22 June 2012 08:10:21 UTC