Re: Comments to Editor's draft 2012-02-09 (an Observer awakes...)

Hi Loic--

Thanks for all the thoughtful input. I had a couple of questions--please 
see below. I'd encourage the rest of the group to look over Loic's 
comments on sampling and the evaluation process--Loic's comments show 
how helpful it is to have a fresh set of eyes look things over!

Mike Elledge

On 2/20/2012 6:16 PM, Loïc Martínez Normand wrote:
> Dear all,
>
> Let me first introduce myself. My name is Loïc Martínez and I teach at 
> the Technical University of Madrid (Spain). I've been researching in 
> the field of accessibility since 1995 and I am president of the Sidar 
> Foundation (that is represented in the EVAL-TF by Emmanuelle 
> Gutierrez). I also actively participate in standardization activities 
> in the field of ICT accessibiltiy in Spain (AENOR), Europe (CEN, ETSI) 
> and Internationally (ISO and ISO/IEC).
>
> I was invited by Shadi to actively participate in EVAL-TF but I was 
> unable to commit the required amount of hours, so I have been a (very) 
> quiet observer since the beginning of your work.
>
> Last week I was finally able to spend some time on EVAL-TF issues when 
> travelling to the WAI-ACT open meeting and I have reviewed the latest 
> editor draft of  the Website Accessibility Evaluation Methodology for 
> WCAG 2.0.
>
> I have decided to split my comments into three emails to facilitate 
> threading in the mailing list. In this first email I will post some 
> general and editorial comments. In two following emails I will post my 
> views on sampling (chapter 4) and the evaluation process (chapter 5).
>
> I sincerely hope that my comments will be useful in your future work.
>
> General comments
>
>     * [Abstract] Different contexts should also include summative
>       (i.e. at he end of the,process, such as conformity assessment)
>       and formative (I.e. during development, like usability texting)
>       evaluations. For conformity assessment, only two results are
>       possible (pass, no pass). In formative evaluation other values
>       are possible, such as accessibility metrics. Thus, WCAG-EM
>       should cover both types of results.
>
I think we've come to the conclusion that our emphasis is on a 
methodology that will lead to conformity assessment, even if we're 
evaluating a sub-part of a website.
>
>     * [1.4. Equivalent results] This definition lacks rigour. What is
>       a high correlation degree? I am not good at statistics, but some
>       objective threshold can surely be defined...
>
Perhaps a less technical term should be used than "high correlation." I 
think the intent here is to recognize that there my be more than one way 
to get an answer, but the answer has to be consistent and repeatable.
>
>     * [2.1] If the methodology proposes to use review teams, then it
>       should provide guidance on how to perform evaluation by teams:
>       how to split the evaluation, how to combine the results of
>       several evaluators, how to grade evaluators performance...
>
You make a good point; are your concerns answered by the "Using Combined 
Expertise" article?
>
>     * [2.2] When persons with disability evaluate web sites, they are
>       not able to evaluate all success criteria. For instance, a blind
>       person cannot evaluate colour contrast. Thus, the methodology
>       should provide guidance about which portion of WCAG can  one
>       person evaluate depending on disability...
>
Great suggestion. We should also mention that not all evaluation tools 
are fully accessible, which will further affect the participation of 
persons with disabilities.
>
>     * [3. Paragraph 1] One critical aspect of the scope of the
>       evaluation is the concept of "accessibility supported". The
>       decision of what is accessibility supported should be part of
>       the scope of the evaluation and affects the sampling and the
>       evaluation process.
>
Can you explain this some more? I'm not sure I understand what you mean 
by "accessibility supported."
>
>     * [3. Paragraph 1] This methodology should be much more specific
>       to be useful. Concerning scope, the methodology should mandate
>       particular forms of defining the scope of the evaluation.. It is
>       the only way to facilitate interchange of results.
>
I believe we are going to circle back to this and include examples...
>
>     * [3.1] Currently this paragraph is confusing. I think that it
>       should explain what is a complete process and that in many cases
>       some steps of a complete process can be out of control for the
>       website owner (I.e. payment subsystems). Because of that in some
>       cases the evaluators could chose not to consider full processes.
>
I'm not sure I agree. I think processes should be evaluated in their 
entirety, even if a portion of the process is outside the control of the 
owner. Since any claims of conformance must take into account the entire 
process shouldn't the evaluation?
>
> Editorial comments
>
>     * [1.4 Web page] Some consistency is needed. Is it "web page" or
>       "webpage"? Is it "web site" or "website"? Please unify.
>
Yes.
>
>     * [3. Paragraph 2. Word "Office"] Why uppercase? Are you thinking
>       about a particular office application or suite?
>
This should probably be "Microsoft Office."
>
>     * [4. Paragraph 1. Last sentence] Editorial comment. This last
>       sentence is confusing and needs rewriting.
>
The term "resource" is confusing and something we've talked about 
replacing which I think will help.
> Best regards,
> Loïc