- From: Alistair Garrison <alistair.j.garrison@gmail.com>
- Date: Fri, 22 Jun 2012 09:28:47 +0200
- To: kvotis@iti.gr, Peter Korn <peter.korn@oracle.com>, Eval TF <public-wai-evaltf@w3.org>
Dear Eval, So, if we don't want do set requirements of independence on the evaluator, and don't want to publish the procedures followed (or any parts of an evaluation report) - how do we ensure public trust in the conformance statement being made? All the best Alistair On 22 Jun 2012, at 08:19, kvotis@iti.gr wrote: > Dear Alistair, > > regarding your following comment: > 3) Insist that some parts of every report are made public - not > urls, just procedures (as it should be the case that any similar > samples of web pages taken from the defined site will lead to pretty > much the same result) > > > i am not sure about the described procedure and actually regarding the > selection of appropriate parts of Web sites..How someone who is also > developer will select which of the parts should be selected?Moreover i > agree with Peter regarding the publicity of all evaluation reports. From > my point of view this is a difficult procedure while it requires heavy > storage and processing mechanisms > > regards > > Kostas > > > > ------------------- > Dr. Konstantinos Votis > Computer Engineer & Informatics,PhD, Msc, MBA > Research Associate > Informatics and Telematics Institute > Centre for Research and Technology Hellas > 6th Klm. Charilaou - Thermi Road > P.O. BOX 60361 GR - 570 01 > Thessaloniki – Greece > Tel.: +30-2311-257722 > Fax : +30-2310-474128 > E-mail : kvotis@iti.gr > > > > >> Alistair, >> >> I don't see how we can insist that all evaluation reports be public, or >> that certain parts of an evaluation report be public. We can say that >> an evaluation report isn't complete unless it has all of the mandatory >> parts. But we can't stop someone from producing a summary of the >> report, or excerpting parts of a report, or producing a description of >> the report, or... >> >> Not unless we require copyright on all reports generated by anyone else >> and license it and... (and even then "fair use" doctrines in many >> countries would likely still allow publishing excerpts). >> >> >> Peter >> >> >> On 6/21/2012 1:31 PM, Alistair Garrison wrote: >>> Hi Peter, >>> >>> Trust issues aside? I'd be really interested to hear your thoughts on >>> the 3rd option I presented? >>> >>> All the best >>> >>> Alistair >>> >>> >>> On 21 Jun 2012, at 22:26, Peter Korn wrote: >>> >>>> Alistair, >>>> >>>> I think the question of trust is, frankly, outside of the scope of >>>> our work as well. Nothing prevents someone from lying - whether or >>>> not they are independent. They may make honest mistakes because they >>>> lack the technical expertise needed to do a good job (again whether >>>> independent or not). They may claim their sample is representative >>>> but it isn't. They may claim something failed when it passed (or >>>> vice-versa). >>>> >>>> Being independent doesn't prevent any of that. >>>> >>>> >>>> Regards, >>>> >>>> Peter >>>> >>>> On 6/21/2012 12:35 PM, Alistair Garrison wrote: >>>>> Hi Peter, >>>>> >>>>> Your point about the single person with the massive website is well >>>>> made - and moves forcibly against the idea of independence and my >>>>> early suggested aspects? And, in this context - I can see clearly >>>>> why you think the question of the independence/inter-dependence of >>>>> an evaluator from the site being evaluated is outside of the scope >>>>> of our charter. But, I'm still not 100% convinced? 99% maybe ;-) >>>>> >>>>> To my mind, the question is about trust - will the public place an >>>>> equal amount of trust in an evaluation done by a 1st party, as they >>>>> would a 3rd party? >>>>> >>>>> Well, I think yes they could? >>>>> >>>>> But, in our situation it might only be achieved under certain >>>>> circumstances? Seemingly, a number of options exist: >>>>> >>>>> 1) Insist that the whole evaluation report be published, along with >>>>> the urls tested, procedure, etc... The public could recreate your >>>>> tests and confirm your findings; or >>>>> >>>>> 2) Leave the publishing decision to the evaluation commissioner, and >>>>> instead set requirements for independence on the evaluator. >>>>> >>>>> Note: In our methodology we already state "documentation need not >>>>> necessarily be public, as disclosure is up to the owner and/or >>>>> evaluation commissioner" - presumably you would not wish to publish >>>>> if you have assessed restricted areas of the website (which is a big >>>>> issue with option 1, amongst others)... >>>>> >>>>> So? We already seem to be looking at the second option - hence the >>>>> reason I'm not yet 100% convinced that the question of evaluator >>>>> independence is currently entirely out of scope; or >>>>> >>>>> 3) Insist that some parts of every report are made public - not >>>>> urls, just procedures (as it should be the case that any similar >>>>> samples of web pages taken from the defined site will lead to pretty >>>>> much the same result)? >>>>> >>>>> In light of your point, and the issues with option 1 further >>>>> investigation into option 3 might be an idea - it would of course >>>>> mean changes to 5a, and removing the idea of independence from the >>>>> scope. >>>>> >>>>> Thoughts / comments welcome? >>>>> All the best >>>>> >>>>> Alistair >>>>> >>>>> On 21 Jun 2012, at 20:17, Peter Korn wrote: >>>>> >>>>>> Alistair, >>>>>> >>>>>> If I am the only person involved in creating my own, fair sized >>>>>> website (too large to feasibly evaluate every single page, being as >>>>>> it is programmatically generated, etc. etc.), then >>>>>> "self-assessment" means that I am also the assessor. I cannot be >>>>>> an ISO 9001:2000 compliant internal auditor. >>>>>> >>>>>> Separate from that example, I don't understand why EvalTF should be >>>>>> concerning itself with ISO 900x in any way. Looking again at the >>>>>> Objective portion of the Work Statement >>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our >>>>>> mandate is for a technical task (as I understand it): how to select >>>>>> a representative sample of a site, how to aggregate results into an >>>>>> overall conformance statement, etc. >>>>>> >>>>>> The question of the independence/inter-dependence of an evaluator >>>>>> from the site being evaluated is outside of the scope of our charter. >>>>>> >>>>>> >>>>>> Peter >>>>>> >>>>>> >>>>>> On 6/21/2012 11:06 AM, Alistair Garrison wrote: >>>>>>> Hi Peter, >>>>>>> >>>>>>> Supporting 1st party assessment is as important to me as >>>>>>> supporting 3rd party assessment - which is why I based my proposal >>>>>>> on those well documented aspects you would look for in an internal >>>>>>> auditor for ISO 9001:2000. >>>>>>> >>>>>>> Maybe, for clarity, it should have been 'not associated in their >>>>>>> day to day role with' - I think you have read 'associated' in the >>>>>>> same light as 'independent'. >>>>>>> >>>>>>> Hope this helps. >>>>>>> >>>>>>> Alistair >>>>>>> >>>>>>> On 21 Jun 2012, at 19:36, Peter Korn wrote: >>>>>>> >>>>>>>> Alistair, >>>>>>>> >>>>>>>> It wasn't clear to me that this was the outcome of our meeting. >>>>>>>> >>>>>>>> Reviewing the EvalTF Work Statement >>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first sentence >>>>>>>> of the Objective reads (*/emphasis added/*): "objective of Eval >>>>>>>> TF is to develop an internationally harmonized methodology for >>>>>>>> evaluating the conformance of websites to WCAG 2.0,/*that >>>>>>>> supports different contexts, such as for self-assessment or >>>>>>>> third-party evaluation*/ of small or larger websites". >>>>>>>> >>>>>>>> If the methodology is to support self-assessment, then it cannot >>>>>>>> define the evaluator as be different from the >>>>>>>> developer/maintainer/accessibility-expert for the site. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Peter >>>>>>>> >>>>>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote: >>>>>>>>> Dear all, >>>>>>>>> >>>>>>>>> In today's telecon, we discussed terms like independent when >>>>>>>>> talking about evaluations. The outcome appeared to be that what >>>>>>>>> was needed was in fact a better definition for 'evaluator'. >>>>>>>>> >>>>>>>>> I'm not going to propose the whole definition for 'evaluator', >>>>>>>>> however, just two aspects which we might consider including in the >>>>>>>>> definition: >>>>>>>>> >>>>>>>>> Aspect 1) (of an evaluator) someone who is not responsible for the >>>>>>>>> accessibility of the website being evaluated. >>>>>>>>> Aspect 2) (of an evaluator) someone who is not associated with >>>>>>>>> developing and maintaining the website or its content. >>>>>>>>> >>>>>>>>> Thoughts? >>>>>>>>> >>>>>>>>> All the best >>>>>>>>> >>>>>>>>> Alistair >>>>>>>> >>>>>>>> -- >>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>>>> Peter Korn | Accessibility Principal >>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>>>>> Oracle is committed to developing practices and products that >>>>>>>> help protect the environment >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>> Peter Korn | Accessibility Principal >>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>>> Oracle is committed to developing practices and products that help >>>>>> protect the environment >>>>>> >>>>>> >>>>> >>>> >>>> -- >>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>> Peter Korn | Accessibility Principal >>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> Oracle >>>> is committed to developing practices and products that help protect >>>> the environment >>>> >>>> >>> >> >> -- >> Oracle <http://www.oracle.com> >> Peter Korn | Accessibility Principal >> Phone: +1 650 5069522 <tel:+1%20650%205069522> >> 500 Oracle Parkway | Redwood City, CA 94065 >> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to >> developing practices and products that help protect the environment >> >> >> > > > > >
Received on Friday, 22 June 2012 07:29:21 UTC