W3C home > Mailing lists > Public > public-wcag-em-comments@w3.org > April 2013

Review of WCAG-EM

From: Loïc Martínez Normand <loic@fi.upm.es>
Date: Mon, 15 Apr 2013 23:41:42 +0200
Message-ID: <CAJpUyz=GzPGqHOpPRu-9ubc_QLd_jYqC-j9gwBiHoGvQsFwNLw@mail.gmail.com>
To: public-wcag-em-comments@w3.org
Dear members of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF).

In the last weeks I've been reviewing the latest draft (26 February 2013)
of the Website Accessibility Conformance Evaluation Methodology (WCAG-EM)

In this message I provide all my detailed comments, structured according to
the table of contents of WCAG-EM. Before my detailed comments I would like
to raise several issues:

   - The TF has done an incredible amount of work to prepare the current
   draft of WCAG-EM and I haven't had as much time as I would have liked to
   for preparing my comments, so some of them may look rude. My apologies in
   - You already know that I'm an observer of the Eval TF. Don't hesitate
   to ask me about any clarification about my comments. They may be too
   cryptic in some cases.I will do my best to reply to your questions.
   - It is highly probable that my comments will raise issues where you
   already have reached consensus. It is not my intention to reopen debates or
   break that consensus.

My detailed comments are below. Best regards,

*Introduction* (http://www.w3.org/TR/WCAG-EM/#introduction)

   - *Terms and definitions* (http://www.w3.org/TR/WCAG-EM/#terms)
      - *Complete processes*. First, I think it should be singular
      (complete process). In addition, the current text is not a
definition as it
      just quotes text from WCAG 2.0. I suggest the following text as
      "Complete process: sequence of steps that need to be completed
in order to
      accomplish an activity".
      - *Common functionality*. Given the definition (functionality that if
      removes *fundamentally* changes the purpose of the website), the term
      "common" is not strong enough. I believe that the term should
emphasize the
      essential or fundamental character of this functionality that is, by
      definition, more important than non-fundamental functionality. I suggest
      one of the following:
         - Critical functionality (taken from note 2)
         - Fundamental functionality (taken from the definition)
         - Purpose-defining functionality (again, based on the definition)
      - *Common web pages*. I think that this definition (web pages that
      are relevant to the entire website) lacks precision. I think that the
      concept that defines these pages is that their underlying
functionality is
      not specific for a given website, but is general to many web sites. Maybe
      they could be named "general-purpose web pages" and then the definition
      could be "web pages whose purpose is general and not specific to the
      website they belong". But I am not completely sure about this.
      - *(new definitions)* Given their relevance to WCAG-EM, I suggest
      adding two definitions from WCAG that are essential to the process of
      conformance evaluation: "conformance" and "satisfies a success
      This last one means that a success criterion is satisfied (passes) unless
      it fails and to me has great implications in the evaluation of
      accessibility: a success criterion passes for a page unless it is
      demonstrated that it fails.
         - *conformance*: satisfying all the requirements of a given
         standard, guideline or specification
         - *satisfies a success criterion*: the success criterion does not
         evaluate to 'false' when applied to the page

*Using this methodology* (http://www.w3.org/TR/WCAG-EM/#usage)

   - *Scope of applicability* (http://www.w3.org/TR/WCAG-EM/#applicability)
      - *Particular types of websites* (
         - *Website with separable areas*.
            - First of all I think that it would be relevant to talk about
            "independence" between the areas. The first sentence could
be "In some
            cases websites may have clearly separable areas where
using one area does
            not require or depend on using another area of the website".
            - Then the example could be modified into something different:
            "An example could be a service company website with three
areas: a public
            area for information purposes, a customer area for
providing services to
            existing customers and an employee area for providing
services to people
            working in the company".
            - The last sentence can be maintained.
            - The result of applying all my proposals would be: "In some
            cases websites may have clearly separable areas where
using one area does
            not require or depend on using another area of the
website. An example
            could be a service company website with three areas: a
public area for
            information purposes, a customer area for providing
services to existing
            customers and an employee area for providing services to
people working in
            the company. Such areas can be considered as individual
websites rather
            than sub-sites for the purpose of this document."
         - *Evaluation tools (Optional)* (
      - I don't understand why this section is labelled as "optional". It
      just describes the relationship between the methodology and evaluation
      tools. I suggest removing the "(optional)".
   - *Review teams (Optional)* (http://www.w3.org/TR/WCAG-EM/#teams)
      - I don't understand why this section is labelled as "optional". This
      section "Using this methodology" does not seem to me a place to define
      "requirements" for users of the methodology (and in fact the previous
      sections - scope of applicability, required expertise, evaluation tools -
      don't require anything). The "methodology requirements" do
really start in
      the next level-1 section ("Conformance Evaluation Procedure").
      - I suggest removing the "(optional").
      - In addition the wording of the paragraph could be modified to avoid
      references to "requirements" by deleting the third sentence: "The
      methodology defined by this document can be carried out by an individual
      evaluator with the skills described in section Required
Expertise. However,
      using the combined expertise of review teams provides better coverage for
      the required skills and helps identify accessibility barriers more
      effectively. Specific guidance is provided in Using Combined Expertise to
      Evaluate Web Accessibility."
   - *Involving users (Optional)* (http://www.w3.org/TR/WCAG-EM/#users).
      - I don't understand why this section is labelled as "optional". I
      suggest making the same changes: removing the "(optional)" and
deleting the
      third sentence.

*Conformance evaluation procedure* (http://www.w3.org/TR/WCAG-EM/#procedure)

   - One issue with terminology: I think there is lack of consistence while
   using the words "stages" and "steps". I suggest using "stage" for the
   high-level elements (1, 2, ..) and "steps" for the inner levels (1a, 1b,
   - The description of the diagram (paragraph below the diagram) seems to
   be incomplete. It seems to me that there are back arrows from each step to
   any previous step: that is, for instance, from step 5 to steps 4, 3, 2 and
   1. Am I right interpreting the diagram?
      - If I am right then the description has to be changed: "The diagram
      depicts each of the five steps defined in this section with arrows forth
      between each of two consecutive steps: 1. Define the Evaluation Scope; 2.
      Explore the Target Website; 3. Select a Representative Sample;
4. Audit the
      Selected Sample and 5. Report the Evaluation Findings. There are also
      arrows back from each step to any of its previous steps."
      - If I am wrong then the diagram should be changed to avoid this
   - *Step 1: Define the Evaluation Scope* (
      - *Step 1.a: Define the Scope of the Website* (
         - Paragraph 1. Second sentence. The wording of this sentence is
         strange. I understand that WCAG-EM cannot use normative
language (shall,
         should), but the use of "may" doesn't work for me. What about
"This scope
         definition will follow the terms established in section Scope of
      - *Step 1.b: Define the Goal of the Evaluation* (
         - The terminology is confusing. The name of the step is "define
         the goal of the evaluation", whereas the content talks about "types of
         evaluation". These two should be aligned. Either the title of
the step is
         changed into "Define the type of evaluation" or the content
of the step is
         changed to describe different goals instead of evaluation types.
      - *Step 1.d: Define the Techniques and Failures to be Used (Optional)*
         - I strongly believe that this step does not belong to this stage
         (1. Define evaluation scope) but to the next one (2. Explore
the target
         website). More precisely, I think it should be renumbered as
step 2e. The
         reason is that, to me, one can only define techniques and
failures once the
         functionality, the types of web pages and the technologies
relied upon are
         - If this step is moved to become 2e, then the last paragraph
         should be revisited and changed accordingly. For instance, the second
         sentence ("This definition...") refers to the exploration stage.
      - *Step 2: Explore the Target Website* (
      - *Step 2.b: Identify Common Functionality of the Website* (
         - As I have suggested alternatives for "common functionality"
         (such as "critical functionality", "fundamental functionality" or
         "purpose-defining functionality" this step should be modified to be
         consistent with the new term.
      - *Step 2.d: Identify Web Technologies Relied Upon* (
         - There is a problem with the use of "relied upon" in this step.
         According to the definition of technologies that are "relied
upon" what it
         means is that the content would not conform WCAG if those
technologies were
         not supported. But the description of step 2.d talks about
         relied upon to provide the website", which is a different concept. My
         suggestion is either to avoid the use of "relied upon" in
this step or to
         use it related to WCAG conformance and not to the overall website.
         - My preferred option is the first one: to avoid the use of
         "relied upon" and use the term "technologies used" instead.
         - If the second option is selected then this step would require
         two "substeps": (1) to identify all the technologies that are used to
         provide the website and (2) to identify which of those
technologies are
         relied upon for accessibility purposes.
      - *Step 3: Select a Representative Sample* (
      - I have on general comment on the "sampling" stage. I have not seen
      any guidance related to how to measure the confidence level based on
      parameters of the sample. But this confidence level is essential when
      declaring the results of the evaluation of the website as a whole. Some
      guidance should be included in the steps below.
      - *Step 3.e: Include a randomly selected sample* (
         - I agree that a random sample can act as a verification indicator
         of the results found in the structured sample, but I don't
really agree
         that a random sample can be used to increase the confidence
in the results
         of the evaluation of the structured sample.
         - The reason is that the structured sample and the random sample
         are two different samples and my understanding from my (low)
knowledge on
         statistics is that these two different samples cannot be
combined into a
         single sample to derive a global result for the website with
an increased
         confidence level.
         - So I think that this step should be focused on using the random
         sample as a "control sample" to see whether the results of
the structured
         sample are replicated in the random sample.
         - Another possible use of the random sample would be to increase
         the probability of finding new accessibility errors that were
not present
         in the structured sample.
         - In addition I would like to know the reason for the numbers
         used. Why 5%? Why a minimum of 5 pages? What if the website
only contained
         10 pages and 6 of them were already part of the structured
sample? It would
         be great to have some reasoning explaining how the
probability of finding
         new errors could increase by increasing the percentage of
pages selected in
         the random sample.
      - *Step 3.f: Eliminate Redundancies in the Sample (Optional)* (
         - I don't agree with this step being optional. I cannot think of
         any good reason to maintain duplicated pages in the sample. That would
         decrease the relevance of the sample and the confidence level of the
         result, wouldn't it? I think that this step should be mandatory.
      - *Step 4: Audit the Selected Sample* (
      - General comment about step 4: some of the content of this step
      seems to me to be out of scope. WCAG-EM is about testing conformance to
      WCAG 2.0, that is, to determine whether a web site conforms to the
      technical requirements of WCAG 2.0, which are the 61 success criteria and
      the 5 conformance requirements. WCAG-EM should not deal with
other types of
      accessibility evaluations, such as user testing or the concept of
      "accessibility-in-use" as defined in current research. So the first
      substep, as 4.a (check for use cases), seems to me out of scope.
      - What I expected as content of step 4 is guidance about:
         - how to evaluate each success criterion for each element of a web
         page of the sample, including guidance about the best
approach: inspection,
         testing by experts, testing by users...
         - how to combine the results obtained for each element in order to
         get a result for the page,
         - how to evaluate the 5 conformance requirements once the result
         of each SC is known for the page,
         - how to combine the results obtained for each page to get the
         global result for the site (that then will be reported in step 5).
      - Note 1.
         - The first note is confusing. If there are repetitive elements on
         every web page it makes sense to evaluate them only once, and
then record
         and reuse the evaluation results. It is clear that an
evaluator should not
         evaluate several times the same repetitive elements but it is also
         important to note that the result assigned after the first
evaluation has
         to be used when evaluating each page where the repetitive
elements appear!
         - My understanding of the second sentence of this note is that it
         implies that the repetitive elements are only evaluated once and their
         result is only used once... I'm probably misreading it, but I
think that
         the text would benefit from some rewrite.
         - In addition I don't see the point of referring back to step 1.b
         in this note. What would be the difference on dealing with repetitive
         elements depending of the type of expected report?
         - My proposal for that second sentence would be:
            - "An evaluator does not need to repeatedly identify successes
            and failures in meeting the conformance target for these repetitive
            elements on every web page. The success and failures for
such repetitive
            elements can be identified once and then be reused in all
the pages those
            elements appear."
         - Note 2.
         - I think that the wording of this note is confusing. It should
         clearly explain that, according to WCAG 2.0, "no matching
content for SC"
         is really "SC is satisfied" with no question. It should also
explain that
         the "not applicable" could be used as a "description" of the
result but not
         as a result.
         - My proposal:
            - According to WCAG 2.0, Success Criteria to which there is no
            matching content are satisfied. In such cases, an
evaluator may use text
            such as "Not applicable" as a description associated to the outcome
            "satisfied", to denote the particular situation where a
success criterion
            was satisfied because no relevant content was applicable.
      - *Step 4.a: Check for the Broadest Variety of Use Cases* (
         - The title of misleading, because the requirement 4.a is about
         checking the conformance requirements, not about use cases.
         - I suggest changing the title: "Step 4.a: Check the conformance
         - I find that this step has gaps in the content I expected it to
         have. In my opinion checking for the 5 conformance
requirements is not a
         straightforward task and more guidance is needed. I would like to have
         guidance outlining a process such as:
            - a) Look for failures (as published)
            - b) If no failures are found, check if any SC fails
            - c) If no SC fails and a "in-depth analysis" has been selected
            in step 1.b, then determines which techniques have been
used to conform to
            each SC
            - ...
         - My proposed process is based on the definition of "satisfies a
         SC" in WCAG: "the success criterion does not evaluate to
'false'", Given
         that definition, it is not really needed to find/determine
the techniques.
         It should be enough to be sure that no SC fails.
         - All the text about scenarios, use cases, personas and user
         involvement is, to me, out of scope when dealing about testing for
         conformance with a standard. These type of things are
essential during the
         development process when following a user-centred approach,
but I believe
         that they don't provide added value in the context of checking the
         conformance of WCAG 2.0. If this text about scenarios and
users is to be
         kept, I'd suggest it being transformed into a note.
      - *Step 4.b: Assess Accessibility Support for Features* (
         - First, this should be linked to step 2.d (identify web
         technologies relied upon). The first "cut" for features that have
         accessibility support is to look for the ones using
technologies identified
         in step 2.b.
         - Second, and according to my comment for step 4.a, I think this
         could be optional as it is only needed when the evaluation report will
         contain detailed information about how each SC is met (see my outlined
         process, step c)).
      - *Setp 4.c: Use Techniques and Failures Where Possible (Optional)* (
         - I agree that the use of techniques is optional, but I
         fundamentally disagree in the use of failures. Even if
failures are part of
         an informative document it is 100% sure that if one of those
failures is
         found, then the corresponding SC fails for the page being evaluated. I
         cannot imagine any exception to that idea.
         - As I already explained above, given the definition of
         "satisfying a SC", the concept of "failures" is an essential tool for
         evaluators and should not be optional
         - So I propose dividing this into two:
            - 4.c.1 Use failures (mandatory)
            - 4.c.2 Use techniques (optional)
         - If this is accepted then the text of these two steps should be
         rewritten to fit the new structure.
      - *Step 4.d: Archive Web Pages for Reference (Optional)* (
         - I think that it would be good to have an explanation of the
         reasons to make this step optional. My first idea when
reading the title of
         the step was to ask for it to be mandatory, as I thought that
any evaluator
         should really archive the pages being evaluated (at least the
URI and data
         required to be able to re-evaluate the page if needed). But
then I read the
         details of this step and I tend to agree that all this information is
         optional. This is why I think that some clarification is needed.
      - *Step 4.e: Record Software Tools and Methods Used* (Optional) (
         - I think that this step should not be optional. I agree that it
         is not always necessary to report on this information but
evaluators should
         always record the information so it can be accessed if needed.
      - *Step 5: Report the Evaluation Findings* (
      - *Step 5.a: Provide Documentation for Each Step* (
         - In the list of information documented in the report there is one
         reference missing: step 3.f (optional), that should be listed
with all the
         other sub-steps of step 3.
         - I think that the approach for documented failures in the
         detailed report could be similar to the one in the basic
report, but on a
         page by page basis. That is, if a success criterion fails,
then at least
         one example is provided in the page. My proposal for
replacing the last
            - "Where failures in meeting WCAG 2.0 Success Criteria on a web
            page are identified, at least one examples is provided for
each identified
            failire in the page."
         - *Step 5.c: Provide a Performance Score (Optional)* (
         - I appreciate the effort spent in providing some guidance in this
         difficult area of web accessibility reporting. The three
options  are well
         explained with details on their sensitiveness.
         - This area of accessibility metrics is still under research and I
         think that WCAG-EM should at least acknowledge the W3C report (
         http://www.w3.org/TR/accessibility-metrics-report/) that was
         produced after a symposium organized in December 2011 (
      - *Appendix C: Example Reports* (http://www.w3.org/TR/WCAG-EM/#reports
      - I have no comments as the review note says that this section is to
      be updated to better align with step 5.

Loïc Martínez-Normand
DLSIIS. Facultad de Informática
Universidad Politécnica de Madrid
Campus de Montegancedo
28660 Boadilla del Monte
e-mail: loic@fi.upm.es
tfno: +34 91 336 74 11
Received on Monday, 15 April 2013 22:08:13 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 19:41:54 UTC