Comments to Editor's draft 2012-02-09 (an Observer awakes...)

Dear all,

Let me first introduce myself. My name is Loïc Martínez and I teach at the
Technical University of Madrid (Spain). I've been researching in the field
of accessibility since 1995 and I am president of the Sidar Foundation
(that is represented in the EVAL-TF by Emmanuelle Gutierrez). I also
actively participate in standardization activities in the field of ICT
accessibiltiy in Spain (AENOR), Europe (CEN, ETSI) and Internationally (ISO
and ISO/IEC).

I was invited by Shadi to actively participate in EVAL-TF but I was unable
to commit the required amount of hours, so I have been a (very) quiet
observer since the beginning of your work.

Last week I was finally able to spend some time on EVAL-TF issues when
travelling to the WAI-ACT open meeting and I have reviewed the latest
editor draft of  the Website Accessibility Evaluation Methodology for WCAG
2.0.

I have decided to split my comments into three emails to facilitate
threading in the mailing list. In this first email I will post some general
and editorial comments. In two following emails I will post my views on
sampling (chapter 4) and the evaluation process (chapter 5).

I sincerely hope that my comments will be useful in your future work.

General comments

   - [Abstract] Different contexts should also include summative (i.e. at
   he end of the,process, such as conformity assessment) and formative (I.e..
   during development, like usability texting) evaluations. For conformity
   assessment, only two results are possible (pass, no pass). In formative
   evaluation other values are possible, such as accessibility metrics. Thus,
   WCAG-EM should cover both types of results.
   - [1.4. Equivalent results] This definition lacks rigour. What is a high
   correlation degree? I am not good at statistics, but some objective
   threshold can surely be defined...
   - [2.1] If the methodology proposes to use review teams, then it should
   provide guidance on how to perform evaluation by teams: how to split the
   evaluation, how to combine the results of several evaluators, how to grade
   evaluators performance...
   - [2.2] When persons with disability evaluate web sites, they are not
   able to evaluate all success criteria. For instance, a blind person cannot
   evaluate colour contrast. Thus, the methodology should provide guidance
   about which portion of WCAG can  one person evaluate depending on
   disability...
   - [3. Paragraph 1] One critical aspect of the scope of the evaluation is
   the concept of "accessibility supported". The decision of what is
   accessibility supported should be part of the scope of the evaluation and
   affects the sampling and the evaluation process.
   - [3. Paragraph 1] This methodology should be much more specific to be
   useful. Concerning scope, the methodology should mandate particular forms
   of defining the scope of the evaluation.. It is the only way to facilitate
   interchange of results.
   - [3.1] Currently this paragraph is confusing. I think that it should
   explain what is a complete process and that in many cases some steps of a
   complete process can be out of control for the website owner (I.e. payment
   subsystems). Because of that in some cases the evaluators could chose not
   to consider full processes.

Editorial comments

   - [1.4 Web page] Some consistency is needed. Is it "web page" or
   "webpage"? Is it "web site" or "website"? Please unify.
   - [3. Paragraph 2. Word "Office"] Why uppercase? Are you thinking about
   a particular office application or suite?
   - [4. Paragraph 1. Last sentence] Editorial comment. This last sentence
   is confusing and needs rewriting.

Best regards,
Loïc