- From: Shadi Abou-Zahra <shadi@w3.org>
- Date: Fri, 22 Jun 2012 12:58:34 +0200
- To: public-wai-evaltf@w3.org
Hi All, As per my survey response [1], I think this discussion is initiated from a misunderstanding. The initial comment [2] says: [[ on 3 step1 section : some time defining the scope with the evaluation commissioner can be not as good as this section suggest it because most of the time he doesn't know anything about accessibility and about what is important for user to be evaluated. Furthermore, he can also include in the scope some specific pages where the development team has concentrated his effort to improve accessibility ]] Now, in Section 3 Step 1 [3] the Evaluator and Evaluation Commissioner decide the *scope* of the evaluation (= what the commissioner wants evaluated). The *individual pages* are identified and selected by the Evaluator _alone_ in Step 2 [4] and Step 3 [5] of that section. The Evaluator is also defined as the "entity responsible for carrying out the evaluation" [6]. I do not see a reason to believe that WCAG-EM is biased towards Evaluation Commissioners or that we need to redefine Evaluator based on this comment alone. Some of the other discussion that emerged from this comments seems to relate to providing an accessibility statement, which would actually be Section 3 Step 5.b [7]. Some of the discussion also went into trust and responsibility which, to me, is more related to certification schemes rather than to the evaluation procedure. It would be useful to have an issue description and proposal for each issue to focus the discussion. [1] <https://www.w3.org/2002/09/wbs/48225/evaltfq2/results#x2613> [2] <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2012Apr/0002> [3] <http://www.w3.org/TR/WCAG-EM/#step1> [4] <http://www.w3.org/TR/WCAG-EM/#step2> [5] <http://www.w3.org/TR/WCAG-EM/#step3> [6] <http://www.w3.org/TR/WCAG-EM/#evaluator> [7] <http://www.w3.org/TR/WCAG-EM/#step5b> Regards, Shadi On 22.6.2012 11:21, Alistair Garrison wrote: > Hi Richard, > > We all know what we are doing is for 1st and 3rd Parties - and of course they each have an equal weight. > > The debate stems from Eric's questionnaire number 2 - question 6. > > DoC_ID_28: Independent evaluator > > I would say that I'm hearing most people are now against the proposed action of stating that an evaluator must be independent (me too after Peter's email) - as it would not work in all cases. > > All the best > > Alistair > > On 22 Jun 2012, at 11:04, RichardWarren wrote: > >> If I might summarise our understanding of the role etc. of the evaluator >> >> 1) Peter has reminded us of our terms of reference and that it includes reference to "different contexts such as self-assessment or third party evaluation" >> >> 2) Our target audience (Section 1.2) includes - >> Website developers, suppliers, procurers, and owners. >> Web accessibility consultants and evaluation service providers. >> Web accessibility monitoring and benchmarking activities. >> Web accessibility researchers and disability advocates. >> >> Having established the breadth of our target audience at the beginning we cannot, half-way through our method, suddenly introduce the concept of only allowing "independent" or "third-party" evaluators. >> >> As I understand it our task is to deliver a methodology that helps more people to check if websites comply with WCAG so that the web as a whole becomes more accessible. >> >> The issue of the validity of conformance claims (transparency etc.) should be a separate discussion >> >> Regards >> Richard >> >> >> >> -----Original Message----- From: Alistair Garrison >> Sent: Friday, June 22, 2012 8:28 AM >> To: kvotis@iti.gr ; Peter Korn ; Eval TF >> Subject: Re: Aspects of an evaluator >> >> Dear Eval, >> >> So, if we don't want do set requirements of independence on the evaluator, and don't want to publish the procedures followed (or any parts of an evaluation report) - how do we ensure public trust in the conformance statement being made? >> >> All the best >> >> Alistair >> >> On 22 Jun 2012, at 08:19, kvotis@iti.gr wrote: >> >>> Dear Alistair, >>> >>> regarding your following comment: >>> 3) Insist that some parts of every report are made public - not >>> urls, just procedures (as it should be the case that any similar >>> samples of web pages taken from the defined site will lead to pretty >>> much the same result) >>> >>> >>> i am not sure about the described procedure and actually regarding the >>> selection of appropriate parts of Web sites..How someone who is also >>> developer will select which of the parts should be selected?Moreover i >>> agree with Peter regarding the publicity of all evaluation reports. From >>> my point of view this is a difficult procedure while it requires heavy >>> storage and processing mechanisms >>> >>> regards >>> >>> Kostas >>> >>> >>> >>> ------------------- >>> Dr. Konstantinos Votis >>> Computer Engineer & Informatics,PhD, Msc, MBA >>> Research Associate >>> Informatics and Telematics Institute >>> Centre for Research and Technology Hellas >>> 6th Klm. Charilaou - Thermi Road >>> P.O. BOX 60361 GR - 570 01 >>> Thessaloniki – Greece >>> Tel.: +30-2311-257722 >>> Fax : +30-2310-474128 >>> E-mail : kvotis@iti.gr >>> >>> >>> >>> >>>> Alistair, >>>> >>>> I don't see how we can insist that all evaluation reports be public, or >>>> that certain parts of an evaluation report be public. We can say that >>>> an evaluation report isn't complete unless it has all of the mandatory >>>> parts. But we can't stop someone from producing a summary of the >>>> report, or excerpting parts of a report, or producing a description of >>>> the report, or... >>>> >>>> Not unless we require copyright on all reports generated by anyone else >>>> and license it and... (and even then "fair use" doctrines in many >>>> countries would likely still allow publishing excerpts). >>>> >>>> >>>> Peter >>>> >>>> >>>> On 6/21/2012 1:31 PM, Alistair Garrison wrote: >>>>> Hi Peter, >>>>> >>>>> Trust issues aside? I'd be really interested to hear your thoughts on >>>>> the 3rd option I presented? >>>>> >>>>> All the best >>>>> >>>>> Alistair >>>>> >>>>> >>>>> On 21 Jun 2012, at 22:26, Peter Korn wrote: >>>>> >>>>>> Alistair, >>>>>> >>>>>> I think the question of trust is, frankly, outside of the scope of >>>>>> our work as well. Nothing prevents someone from lying - whether or >>>>>> not they are independent. They may make honest mistakes because they >>>>>> lack the technical expertise needed to do a good job (again whether >>>>>> independent or not). They may claim their sample is representative >>>>>> but it isn't. They may claim something failed when it passed (or >>>>>> vice-versa). >>>>>> >>>>>> Being independent doesn't prevent any of that. >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Peter >>>>>> >>>>>> On 6/21/2012 12:35 PM, Alistair Garrison wrote: >>>>>>> Hi Peter, >>>>>>> >>>>>>> Your point about the single person with the massive website is well >>>>>>> made - and moves forcibly against the idea of independence and my >>>>>>> early suggested aspects? And, in this context - I can see clearly >>>>>>> why you think the question of the independence/inter-dependence of >>>>>>> an evaluator from the site being evaluated is outside of the scope >>>>>>> of our charter. But, I'm still not 100% convinced? 99% maybe ;-) >>>>>>> >>>>>>> To my mind, the question is about trust - will the public place an >>>>>>> equal amount of trust in an evaluation done by a 1st party, as they >>>>>>> would a 3rd party? >>>>>>> >>>>>>> Well, I think yes they could? >>>>>>> >>>>>>> But, in our situation it might only be achieved under certain >>>>>>> circumstances? Seemingly, a number of options exist: >>>>>>> >>>>>>> 1) Insist that the whole evaluation report be published, along with >>>>>>> the urls tested, procedure, etc... The public could recreate your >>>>>>> tests and confirm your findings; or >>>>>>> >>>>>>> 2) Leave the publishing decision to the evaluation commissioner, and >>>>>>> instead set requirements for independence on the evaluator. >>>>>>> >>>>>>> Note: In our methodology we already state "documentation need not >>>>>>> necessarily be public, as disclosure is up to the owner and/or >>>>>>> evaluation commissioner" - presumably you would not wish to publish >>>>>>> if you have assessed restricted areas of the website (which is a big >>>>>>> issue with option 1, amongst others)... >>>>>>> >>>>>>> So? We already seem to be looking at the second option - hence the >>>>>>> reason I'm not yet 100% convinced that the question of evaluator >>>>>>> independence is currently entirely out of scope; or >>>>>>> >>>>>>> 3) Insist that some parts of every report are made public - not >>>>>>> urls, just procedures (as it should be the case that any similar >>>>>>> samples of web pages taken from the defined site will lead to pretty >>>>>>> much the same result)? >>>>>>> >>>>>>> In light of your point, and the issues with option 1 further >>>>>>> investigation into option 3 might be an idea - it would of course >>>>>>> mean changes to 5a, and removing the idea of independence from the >>>>>>> scope. >>>>>>> >>>>>>> Thoughts / comments welcome? >>>>>>> All the best >>>>>>> >>>>>>> Alistair >>>>>>> >>>>>>> On 21 Jun 2012, at 20:17, Peter Korn wrote: >>>>>>> >>>>>>>> Alistair, >>>>>>>> >>>>>>>> If I am the only person involved in creating my own, fair sized >>>>>>>> website (too large to feasibly evaluate every single page, being as >>>>>>>> it is programmatically generated, etc. etc.), then >>>>>>>> "self-assessment" means that I am also the assessor. I cannot be >>>>>>>> an ISO 9001:2000 compliant internal auditor. >>>>>>>> >>>>>>>> Separate from that example, I don't understand why EvalTF should be >>>>>>>> concerning itself with ISO 900x in any way. Looking again at the >>>>>>>> Objective portion of the Work Statement >>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws#objectives>, our >>>>>>>> mandate is for a technical task (as I understand it): how to select >>>>>>>> a representative sample of a site, how to aggregate results into an >>>>>>>> overall conformance statement, etc. >>>>>>>> >>>>>>>> The question of the independence/inter-dependence of an evaluator >>>>>>>> from the site being evaluated is outside of the scope of our charter. >>>>>>>> >>>>>>>> >>>>>>>> Peter >>>>>>>> >>>>>>>> >>>>>>>> On 6/21/2012 11:06 AM, Alistair Garrison wrote: >>>>>>>>> Hi Peter, >>>>>>>>> >>>>>>>>> Supporting 1st party assessment is as important to me as >>>>>>>>> supporting 3rd party assessment - which is why I based my proposal >>>>>>>>> on those well documented aspects you would look for in an internal >>>>>>>>> auditor for ISO 9001:2000. >>>>>>>>> >>>>>>>>> Maybe, for clarity, it should have been 'not associated in their >>>>>>>>> day to day role with' - I think you have read 'associated' in the >>>>>>>>> same light as 'independent'. >>>>>>>>> >>>>>>>>> Hope this helps. >>>>>>>>> >>>>>>>>> Alistair >>>>>>>>> >>>>>>>>> On 21 Jun 2012, at 19:36, Peter Korn wrote: >>>>>>>>> >>>>>>>>>> Alistair, >>>>>>>>>> >>>>>>>>>> It wasn't clear to me that this was the outcome of our meeting. >>>>>>>>>> >>>>>>>>>> Reviewing the EvalTF Work Statement >>>>>>>>>> <http://www.w3.org/WAI/ER/2011/eval/eval-ws>, the first sentence >>>>>>>>>> of the Objective reads (*/emphasis added/*): "objective of Eval >>>>>>>>>> TF is to develop an internationally harmonized methodology for >>>>>>>>>> evaluating the conformance of websites to WCAG 2.0,/*that >>>>>>>>>> supports different contexts, such as for self-assessment or >>>>>>>>>> third-party evaluation*/ of small or larger websites". >>>>>>>>>> >>>>>>>>>> If the methodology is to support self-assessment, then it cannot >>>>>>>>>> define the evaluator as be different from the >>>>>>>>>> developer/maintainer/accessibility-expert for the site. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Peter >>>>>>>>>> >>>>>>>>>> On 6/21/2012 10:02 AM, Alistair Garrison wrote: >>>>>>>>>>> Dear all, >>>>>>>>>>> >>>>>>>>>>> In today's telecon, we discussed terms like independent when >>>>>>>>>>> talking about evaluations. The outcome appeared to be that what >>>>>>>>>>> was needed was in fact a better definition for 'evaluator'. >>>>>>>>>>> >>>>>>>>>>> I'm not going to propose the whole definition for 'evaluator', >>>>>>>>>>> however, just two aspects which we might consider including in the >>>>>>>>>>> definition: >>>>>>>>>>> >>>>>>>>>>> Aspect 1) (of an evaluator) someone who is not responsible for the >>>>>>>>>>> accessibility of the website being evaluated. >>>>>>>>>>> Aspect 2) (of an evaluator) someone who is not associated with >>>>>>>>>>> developing and maintaining the website or its content. >>>>>>>>>>> >>>>>>>>>>> Thoughts? >>>>>>>>>>> >>>>>>>>>>> All the best >>>>>>>>>>> >>>>>>>>>>> Alistair >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>>>>>> Peter Korn | Accessibility Principal >>>>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>>>>>>> Oracle is committed to developing practices and products that >>>>>>>>>> help protect the environment >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>>>> Peter Korn | Accessibility Principal >>>>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> >>>>>>>> Oracle is committed to developing practices and products that help >>>>>>>> protect the environment >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> <oracle_sig_logo.gif> <http://www.oracle.com/> >>>>>> Peter Korn | Accessibility Principal >>>>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> Oracle >>>>>> is committed to developing practices and products that help protect >>>>>> the environment >>>>>> >>>>>> >>>>> >>>> >>>> -- >>>> Oracle <http://www.oracle.com> >>>> Peter Korn | Accessibility Principal >>>> Phone: +1 650 5069522 <tel:+1%20650%205069522> >>>> 500 Oracle Parkway | Redwood City, CA 94065 >>>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to >>>> developing practices and products that help protect the environment >>>> >>>> >>>> >>> >>> >>> >>> >>> >> >> > > -- Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Activity Lead, W3C/WAI International Program Office Evaluation and Repair Tools Working Group (ERT WG) Research and Development Working Group (RDWG)
Received on Friday, 22 June 2012 10:59:02 UTC