Comments to WCAG-EM Editors Draft 29 November 2013 by Samuel Martín (ERT WG)

Abstract

Comment #1:

• priority: strong
• location: at the end of the abstract
• suggested revision: Add a sentence skimming over the contents of the document, similar to this sentence (which has been removed since the previous working draft), but more concise and adapted to the current version: “It provides guidance on defining parameters for the evaluation scope; on exploring and understanding websites including their key features and functionality; on sampling representative web pages from websites where it is not feasible to evaluate all web pages of a website; on evaluating the conformance of these selected web pages to the target level of WCAG 2.0 conformance; and on documenting and reporting the findings from such evaluation.”
• rationale: The current version of the abstract is motivating and sounds rather good. However, a summary can provide a quick overview of what the reader may expect to find in the document. This can be useful to introduce the first-time or casual reader to WCAG-EM, to have it quoted in external documentation, etc.

Introduction

Comment #2:

• priority: N/A
• location: definition list
• current wording: “Core Functionality”
• suggested revision: none, I would just express my support to the current wording. It still appears as an open issue from a comment by myself to the previous WD

Comment #3:

• priority: mild
• location: Definition list
• current wording: Defined terms are capitalized using “Title Case”.
• suggested revision: Use just an uppercase initial in the first word
• rationale: Common orthographic conventions. On the other hand, WCAG 2.0 uses lowercase in the defined terms, because these terms then appear in lowercase throughout the document, that would be another option.

Comment #4:

• priority: mild
• location: definition of “Web Page States”
• current wording: defined term is “Web Page States”
• suggested revision: “Web page state” (in singular)

Comment #5:

• priority: mild
• location: definition of “Web Page States”
• current wording: “Some web page states are ancilliary or treated similarly to individual web pages”
• suggested revision: remove “ancilliary”
• rationale: “ancillary” does not provide any additional information to the sentence “some web page states are treated similarly to individual web pages”

Comment #6:

• priority: mild
• location: definition of “Web Page States”
• current wording: “ancilliary”
• suggested revision: “ancillary”
• rationale: in case the comment #5 above is not accepted, there is still a typo

Using this methodology

Comment #7:

• priority: moderate
• location: under “Involving Users (Optional)”
• current wording: “people with disabilities and people with aging-related impairments”
• suggested revision: Introduce “aging-related impairments” somewhere else, and clarify its relation to people with disabilities.
• rationale: There is a single appearance of “aging” throughout the document, it suddenly appears, while “disability(ies)” is consistently mentioned 20 times. I understand the reference to people with disabilities encompasses people with aging-related impairments, but that should be made clear to the reader.

Scope of applicability

Comment #8:

• priority: moderate
• location: “Examples of Websites” inset.
• current wording: “an area for each department within the organization” has been removed since the previous Working Draft.
• suggested revision: Add another example corresponding to the website of an organization department.
• rationale:  The case where the responsibility for content publishing of each department is distributed, is typical and relevant enough to be mentioned among the list of examples (even though is later dealt with in detail, in the discussion on websites with separable areas).

Comment #9:

• priority: strong
• location: “Example of Website Enclosure” inset
• current wording: “This includes (...) forms for online payments, and discussion boards, including when such parts originate from third-party sources.”
• suggested revision: Add information to the Principle of Website Enclosure, so as to explain websites also enclose some third-party contents needed to abide by the “full pages” and “complete process” requirements.
• rationale: The sentence quoted from example implies this, but it should appear as well in the “prescriptive” part above, where the Principle of Website Enclosure is defined

Comment #10:

• priority: moderate
• location: “Example of Website Enclosure” and “Particular Types of Websites”
• suggested revision: Exchange the order of these two subsections.
• rationale: the content of “Example of Website Enclosure” relies on the definition of “Website with separable areas” (included in “Particular Types of Websites”)

Comment #11:

• priority: moderate
• location: under “Particular Types of Websites”
• current wording: “Web applications will typically require (...) larger web page samples to reflect the different types of content, functionality, and processes.”
• suggested revision: Check whether this is substantiated or just assumed.
• rationale:  In my understanding, the amount of “different types of content, functionality and processes” does not depend on being a web application per se. On the contrary, if the same functions are offered through a static site and a dynamic web application, the same complexity should be expected. Besides, even some web applications are more homogeneous than static web sites, in that all the contents are generated from the same templates and scripts, and then a smaller sample is enough.
  Step 3 talks about “How interactive the content is”, which is a much more objective foundation to determine that a sample should be larger, rather than just relying on the fact the functionality is provided through a web application.

Comment #12:

• priority: mild
• location: Website in Multiple Versions
• current wording: “are each independent of one another in usage, where using one version does not require or depend on using another version of the website”
• suggested revision: “are independent of one another in use, that is, using one version does not require or...”
• rationale: clarify the wording

Comment #13:

• priority: moderate
• location: “Self-Assessment of Conformance” under “Particular Evaluation Contexts”
• current wording: “In this context evaluators often have easier access to the website developers and maintainers, the development environment and authoring tools, and the materials used for development”
• suggested revision: Add “... and the hosting platform where the website is deployed”.
• rationale: Being able to “get into the kitchen” eases accessibility evaluation. Not only is it a matter of snooping around development processes, but it also facilitates access to later stages. E.g. self-assessment may ease accessing restricted areas (which is unrelated to accessing development).

Comment #14:

• priority: strong
• location: “Third-Party Assessment of Conformance” under “Particular Evaluation Contexts”
• current wording: “Third-party evaluators typically use pure black-box testing and … Therefore it is more difficult, often impossible, to make conformance claims based on such an evaluation alone.”
• suggested revision: Revise content (see rationale)
• rationale: First, black-box testing means ‘no access to the internals of the software’, instead of ‘no access to the development environment (and authoring tools)’. Second, in the specific context of website evaluation accessibility, third-party evaluators usually do have access to a large part of the source code (e.g. HTML, CSS, Javascript), which even sometimes represents all the source code (e.g. in static websites, where there is no server-side scripting environment). Thus these tests are rather grey-box (or even white-box). Third, WCAG success criteria are defined from the perspective of the end-user, so black-box testing should not be in principle excluded as a way to test conformance. For instance, the evaluator does not need to know whether device-independent scripting events have been used -i.e. black box-, they just need to check the functionality is available from a keyboard -white box-.

Comment #15:

• priority: strong
• location: “Example of Web site enclosure”, and “Evaluating Third-Party Content” under “Particular Evaluation Contexts”
• current wording: in “Example of Web site enclosure”, it is said that the scope of evaluation includes “aggregated and embedded content (...) including when such parts originate from third-party sources”. Later on, in “Particular Evaluation Contexts”, it is said that “third-party content is not under the control of the website or web service providers (...) non-conforming content needs to be clearly identified in the web pages in which it appears”.
• suggested revision/rationale: Distinguish between these two uses of “third-party content”. One refers to content integrated from external sources, but which is part of the scope, so as to consider complete processes. The other refers to user-generated contents, which can be either repaired or excluded from the scope through a statement of partial conformance.
• rationale: the same term “third-party” is used for two different concepts that require two different approaches, this may confuse the readers.

Comment #16:

• priority: mild
• location: bullet list under “Re-Running Website Evaluation”
• suggested revision: add links from each item to the relevant steps
• rationale: In the previous working draft, the phrases “as per step x.x” appeared at the end of each item. They have been removed to make the structure clearer, but at the same time the whole section has been shifted and now appears before the methodology steps. With the current wording, “exemplar web page instances” are mentioned before they are explained in the methodology.

Comment #17:

• priority: moderate
• location: Particular Evaluation Contexts
• suggested revision: Add an explanation of the security and privacy issues that may be involved when following this methodology. At least, I have identified these two potential issues:

• Evaluators being granted access to restricted areas (Note in the introduction of Step 2), especially if evaluators are external to the organization.
• Recording the evaluation specifics (Step 5.b), which may generate potentially private information (including copies or screenshots the contents themselves, internal paths and templates, etc.), that should be subject to suitable management policies.

• rationale: Security and privacy issues may appear when following this methodology, and evaluators should be aware of that, so as to take whatever measures (out of the scope of WCAG-EM) they consider. Different kinds of standards usually add a section on potential security risks. In any case, not sure if it fits in “Particular Evaluation Contexts”, or they should be added directly to Step 2 and Step 5.b respectively.

Step 1

Comment #18:

• priority: moderate
• location: introduction to “Evaluation procedure”, introduction to Step 1, introduction to Step 2.
• current wording: Introductory paragraphs to these sections talk several times about “stages”
• suggested revision: replace with “steps”
• rationale: “steps” is used elsewhere in the document. [Other appearances of the word “stage” should be kept, as they don’t refer to the steps of the methodology.]

Comment #19:

• priority: strong
• location: figure in the introduction of “Evaluation Procedure”
• current wording: in the figure, each step can only progress to the immediately next one, or go backwards.
• suggested revision: make step 5 cover the rest of the steps in parallel.
• rationale: Step 5 happens in parallel to the others, instead of taking place after the others (as the figure is currently implying). Even the introductory text from step 5 reads “While evaluation findings are reported at the end of the process, recording them is carried out throughout the evaluation process to ensure reliable outcomes.” and then step 5.1 deals with the documentation of each of the previous steps.

Comment #20:

• priority: strong
• location: Step 1.c: Define an Accessibility Support Baseline 
• current wording: baseline
• suggested revision: Ask for public feedback on the concept of baseline as it is used in WCAG-EM.
• rationale: The concept of “baseline” has been historically subject to much contention during the development of WCAG 2.0, until it was finally dismissed. In WCAG-EM, it is used in a different context, but with several aspects in common. Risks exist, such as evaluating against an artificially restricted subset of user agents that support the accessibility features. Later on, in Step 4, it is not clear enough whether all the user agents (including assistive technologies) included in the chosen baseline should be employed to audit websites.
  I understand the concept may be necessary here, but it should be treated with caution.

Comment #21:

• priority: mild
• location: Step 1.e: Define Additional Evaluation Requirements (Optional)
• suggested revision: add a new item to the bullet list: “Adherence to specific documentation or reporting templates.”
• rationale: Additional requirements may also be targeting non-functional process constraints. One of the most typical examples is the integration of accessibility evaluation within a larger (e.g. corporate QA) process, whose documentation system must be used.

Step 2

Comment #22:

• priority: mild
• location: Step 2.a: Identify Common Web Pages of the Website
• current wording: The last paragraph has been removed since the previous working draft “This step also helps understand key aspects of the website, such as the navigation and overall structure of the website.”
• suggested revision: Re-add it as a “Note”:
• rationale: Identifying common web pages has a value by itself, but it is also helpful to gain an overall vision of the website organization at this early stage of the evaluation.

Comment #23:

• priority: mild
• location: Step 2.b: Identify Core Functionality of the Website
• current wording: “Identify an initial list of core functionality”
• suggested revision: “... core functionalities”
• rationale: grammatical agreement (“list of” + plural).

Comment #24:

• priority: moderate
• location: Step 2.b: Identify Core Functionality of the Website
• current wording: “it might be less easy to identify that it also has a currency conversion function that is core to the particular context of the online shop”
• suggested revision: Replace the currency converter example, maybe with “... that it also has a personalization function to customize options such as currency and country”. Or just choose another functionality for the example (maybe my specific suggestion would also fall under step 2.e)
• rationale: A currency converter as that one would not be presented as “core” in that context. Indeed, there is a bullet list that develops the example within the same step, but the currency converter is not even listed there among the core functionalities.

Comment #25:

• priority: mild
• location: Step 2.b: Identify Core Functionality of the Website
• current wording: “For example, an online shop is (...) to that online shop”. “on the website, for example: [...follows a bullet list with three items]”.
• suggested revision: Separate the sentences dealing with the example, and move them to an inset.
• rationale: Clarify what is “prescriptive” and what is an example. Keep coherence with other examples presented as insets throughout the document.

Comment #26:

• priority: mild
• location: Step 2.c: Identify the Variety of Web Page Types
• current wording: “Web pages that are created using different templates and by different authors (if this is known to the evaluator);”
• suggested revision: split into two items: “Web pages that are created using different templates (if this is known to the evaluator);” and “Web pages that are authored by different people (if this is known to the evaluator);” plus add “Web pages that are created using different coding styles”.
• rationale: These are two different points and concepts. Sometimes it will be possible to distinguish according to one criteria but not the other, or vice versa. In addition, variety of coding styles should also be mentioned, to be consistent with the list later presented in step 3. The variety of coding styles is always discernible by the evaluators, whether they know the reason (e.g. different authors) or not

Comment #27:

• priority: mild
• location: Step 2.d: Identify Web Technologies Relied Upon
• current wording: “auxiliary web technologies such as Java, JavaScript and WAI-ARIA”
• suggested revision: remove Java
• rationale: as of December 2014, it does not seem client-side Java is used enough as an auxiliary web technology to deserve being mentioned in such a short list.

Comment #28:

• priority: mild
• location: Step 2.d: Identify Web Technologies Relied Upon
• current wording: “as well as specific web technologies such as SMIL and SVG”
• suggested revision: add PDF
• rationale: relevance of PDF; plus consistency with the introduction (where HTML and PDF are the only examples of technologies mentioned).

Comment #29:

• priority: moderate
• location: Step 2.e: Identify Other Relevant Web Pages
• current wording: “Some websites include web pages and web page states that are specifically relevant for people with disabilities and accessibility of the website. (...) This includes: (...) Particularly popular web pages and those that are commonly used as entry points to the website;”
• suggested revision: either change the first sentence, so that it does not specifically refer to people with disabilities, or move the mentioned bullet item to its own step 2.f.
• rationale: The presented example (popular web pages and common entry points) is neither “specifically relevant for people with disabilities” nor included in “some websites”. On the contrary, they are relevant for everybody, and they are included in any website (each will have their own popular pages and common entry points, but all will have some). It makes perfect sense that they are included in the sample, but they do not fit under the current heading. Another option is to move them to a different step.

Step 3

Comment #30:

• priority: mild
• location: Step 3 (introduction)
• current wording: typcially
• suggested revision: typically
• rationale: typo

Comment #31:

• priority: moderate
• location: Step 3 (introduction)
• current wording: “websites with content that is rich in functionality require larger samples to cover the different tasks that can be carried out on a website and the different states that individual web pages can have;”
• suggested revision: “websites with content that is rich in interaction require larger samples to cover the functions provided by a website and the different states that individual web pages can have;”
• rationale: what is relevant for rich web applications is that they have rich interactivity, not whether they provide complex or simpler functions. Likewise, it is not a matter of being able to carry out many tasks (this is more related to the size of the site), but a matter of covering the whole site (same as in a static site).

Comment #32:

• priority: mild
• location: Step 3 (introduction)
• current wording: “content that is aggregated from different sources or that is processed during runtime”
• suggested revision: clarify the meaning of “during runtime”
• rationale: It is not clear whether it means the content is processed (on the server) upon user accessed, or whether it is processed (on either the server or the client) during/after user access.

Comment #33:

• priority: mild
• location: Step 3 (introduction)
• current wording: “ websites that are available in different versions, are served according to users and their preferences, and adapt to access devices require larger samples to cover these different situations;”
• suggested revision: “... or adapt to…”
• rationale: express alternatives, not restrictions

Comment #34:

• priority: mild
• location: Step 3.c: Include Exemplar Instances of Web Pages
• current wording: distinctinstance
• suggested revision: a space is missing in between
• rationale: typo

Comment #35:

• priority: mild
• location: Step 3.c: Include Exemplar Instances of Web Pages
• current wording: “instance(s)” of web pages (three times within the step)
• suggested revision: Consider defining an “instance” above.
• rationale: in Step 2.c, it is said that “The outcome of this step is a list of the different types (as opposed to specific instances) of web pages and web page states”, but then in Step 3.c. it is not clear what “exemplar instances” refer to.

Step 5

Comment #36:

• priority: moderate
• location: Note at the end of  Step 5.a: Document the Outcomes of Each Step
• current wording: “The outcomes of Step 4: Audit the Selected Sample can be aggregated over the entire sample”
• suggested revision: Add before “Depending on the desired granularity of the report documentation, …”
• rationale: Make it clear aggregation is not a mere choice, but it responds to more general requirements of the evaluation process.

Comment #37:

• priority: moderate
• location: Step 5.d: Provide a Performance Score (Optional)
• current wording: Some sentences have been removed since the previous working draft, including this one “The type of scoring system used has to be indicated along with the score whenever such a score is provided.”
• suggested revision: Re-add it
• rationale: The measure of a magnitude needs the units into which it is expressed so as to make sense; and the measurement procedure followed so as to be reproducible.

Comment #38:

• priority: mild
• location: Step 5.d: Provide a Performance Score (Optional), per web page
• current wording: “During the evaluation mark any WCAG 2.0 Success Criteria that are satisfied for each of the web pages”
• suggested revision: “During the evaluation mark each WCAG 2.0 Success Criteria that is satisfied by each of the web pages”
• rationale: grammar (doesn’t sound ok, yet my suggestion is still dubious)

Comment #39:

• priority: moderate
• location: Step 5.d: Provide a Performance Score (Optional), per web page
• current wording: “and divide this by the sum of all applicable WCAG 2.0 Success Criteria across all web pages and web page states in the selected sample.”
• suggested revision: Make the meaning of the sentence more accurate.
• rationale: The first-time reader may interpret that “the sum of criteria across all web pages” means the number of applicable criteria, not that number multiplied (I mean, roughly) by the number of web pages.

Comment #40:

• priority: mild
• location: Step 5.d: Provide a Performance Score (Optional), per instance
• current wording: “average ratio over all WCAG 2.0 Success Criteria occurrences” … “determine each possible occurrence of WCAG 2.0 Success Criteria”
• suggested revision: replace “occurrence” with “examination”.
• rationale: success criteria do not occur, they are applied, examined, tested, etc.

Comment #41:

• priority: strong
• location: Step 5.d: Provide a Performance Score (Optional): per web page and per instance
• current wording: “applicable” or “apply to the content”, referring to Success Criteria, is used in this step with two meanings: a) a Success Criterion is applicable as determined by the target conformance level (defined at Step 1.b), b) a Success Criterion is applicable because there are some elements on the web page which are concerned by the criterion and whose conformance to the it must be examined.
• suggested revision: distinguish between these two different meanings
• rationale: readers can be confused by that. Indeed, I got myself confused before as shown in a previous comment where I interpreted the later meaning which made me think the specific definition of the metric was not sensible.

Comment #42:

• priority: strong
• location: Step 5.d: Provide a Performance Score (Optional)
• current wording: “Performance scores are not intended for inter-site comparisons.”
• suggested revision: add strong emphasis to the sentence.
• rationale: Even though the document states these are just the approaches provided by the methodology (thus, WCAG-EM does not claim this is the ultimate methodology), Eval TF should be aware that the scoring proposed herein may soon become the “default” scoring system, as the public will recognize it as sanctioned by the W3C WAI. Thus, the goal of the performance score should be made clear-cut. Otherwise, we risk everybody will start establishing site rankings based on the WCAG-EM performance score (what they will do, anyway).

Comment #43:

• priority: strong
• location: Step 5.d: Provide a Performance Score (Optional): per web page and per instance
• current wording: N/A
• suggested revision: Keep asking for public feedback on scoring.
• rationale: performance scoring being still under discussion, also to deal with the next comment.

Comment #44:

• priority: strong
• location: Step 5.d: Provide a Performance Score (Optional): per web page and per instance
• current wording:
• suggested revision: conformance scores should differentiate between Success Criteria that are met just because they are not applicable, and those that are met because they are applicable *and* succeed.
• rationale: I understand my original comment regarding performance scores was addressed in the previous working draft through the editor’s note to get public feedback (and I understand performance scoring is still under discussion from the latest minutes and comments). I would like to reassert the original content of the comment, and specifically the problems mentioned there, namely: less skew through counting Success Criteria which are not examined and non-monotonicity.
  For convenience, I am reproducing here the relevant example: a website with ten pages, two of them (pages 1 and 2) have some time-based contents (audio, multimedia and animations), the others do not. A per-webpage score is applied. Pages 1 and 2 fail only at each of the 7 criteria affecting time-based contents; the other 18 criteria pass for the 10 pages. The per-webpage score would yield (18*2+25*8)/(25*10) = 0.944, although it systematically fails at time-based criteria. Now consider audio and animations are added to the other eight web pages, passing all the time-based (and other) accessibility criteria, but the original pages are not repaired yet. The site would seem proportionally "more accessible", as we now only have an occasional oversight of the time-based criteria; however, the score would be the same. This could even disincentivize adding new accessible content. On the other hand, had we not considered these time-based criteria, the original website would have had (18*2+18*8)/(25*2+18*8) = 0.928, which would improve to 0.944 when new, time-based yet accessible content is added.

Comment #45:

• priority: moderate
• location: N/A
• current wording: Appendix C Example Reports has been removed
• suggested revision: Add a report example at some point
• rationale: Provide users with applied examples of the methodology. I understand this maybe will be addressed in future versions.