[wbs] response to 'Approval for draft publication of WCAG-EM'

The following answers have been successfully submitted to 'Approval for
draft publication of WCAG-EM' (public) for Detlev Fischer.


---------------------------------
Abstract
----



 * ( ) accept this section as draft
 * (x) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 priority: mild
I agree to reinstate the section Michael refers to, with a minor change.

Current wording (actually wording of previous version quoted by Michael): 
"...on evaluating the conformance of these selected web pages to the target
level of WCAG 2.0 conformance"
We have conformance twice here.

Suggested revision: 
"...on evaluating the conformance of these selected web pages to the
targeted level of WCAG 2.0"

Actually, there is practically no guidance regarding the evaluation of
content bar a blanket referral to the Techniques, so perhaps this part
should be deleted?


---------------------------------
Introduction
----



 * ( ) accept this section as draft
 * (x) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 Cf. Loretta's wording suggestion: "The methodolody relies on evaluating
against techniques for meeting WCAG 2.0 success criteria, such as the
Techniques for WCAG 2.0 documented by W3C/WAI, but does not require this or
any other specific set of techniques."

Would we want to require that techniques used for evaluation need to be
documented somewhere in the public domain? If so, I would suggest:

"The methodolody relies on evaluating against publicly documented
techniques for meeting WCAG 2.0 success criteria, such as the Techniques
for WCAG 2.0 documented by W3C/WAI, but does not require this or any other
specific set of techniques." 

--------------
Suggestion by Kerstin Probiesch to remove section "Review teams": 
This section is optional and clearly states that evaluations can be carried
out by individuals. Even experienced evaluators can miss things, so a
second pair of eyes is often beneficial. I see no need to remove the
section. Perhaps change wording instead of deletion?

Current wording: 
"While not required for using this methodology, the use of review teams is
highly recommended when performing an evaluation of a website."

Suggested wording:
While not required for using this methodology, the use of review teams is
often beneficial when performing an evaluation of a website.

--------------
Priority: high

Location: Terms and Definitions, Entry Web Page States

Current wording: 
"Some web page states are ancilliary or treated similarly to individual web
pages in the context of this methodology."

Suggested revision: 
"In the context of this methodology, web page states can be treated as
ancilliary to pages (i.e., recorded as additional state of a page in a page
sample) or as individual web pages."

Rationale: Current wording is rather fuzzy. I admit that the content in
brackets may go beyond what you would expect in the definition of a term
but "ancilliary to pages" alone does not indicate that states should be
recorded with their base page during exploration and sampling.



---------------------------------
Using This Methodology
----



 * ( ) accept this section as draft
 * ( ) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 


---------------------------------
Scope of Applicability
----



 * ( ) accept this section as draft
 * ( ) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 


---------------------------------
Step 1: Define the Evaluation Scope
----



 * ( ) accept this section as draft
 * ( ) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 


---------------------------------
Step 2: Explore the Target Website
----



 * ( ) accept this section as draft
 * ( ) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 


---------------------------------
Step 3: Select a Representative Sample
----



 * ( ) accept this section as draft
 * ( ) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 Priority: medium
Location: Step 3.a: Include Common Web Pages of the Website

Methodology Requirement 3.a: Include all common web pages, including web
page states, into the selected sample.

Current wording: "Include all common web pages and web page states that
were identified in Step 2.a: Identify Common Web Pages of the Website into
the selected sample for evaluation."

Suggested revision: "Include all common web pages and web page states that
were identified in Step 2.a: Identify Common Web Pages of the Website into
the selected sample for evaluation. Web page states and the method to call
them up can be recorded together with the base page. Alternatively, web
page states can constitute a separate page."

Rationale: I think we should make it clear to evaluators that they will
often need to call up additional states to be evaluated, and the idea of
replicability requires that the way to call up states is documented. I am
not happy with my own wording though, so maybe someone else has a better
suggestion?



---------------------------------
Step 4: Audit the Selected Sample
----



 * ( ) accept this section as draft
 * ( ) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 


---------------------------------
Step 5: Record the Evaluation Findings
----



 * ( ) accept this section as draft
 * (x) accept this section as draft with the following suggestions
 * ( ) I do not accept this section as draft
 * ( ) I abstain (not vote)
 I add a reply to Gregg's suggestion to drop "Step 5.d: Provide a
Performance Score (Optional)" altogether.

I agree with Gregg that scoring may not adequately represent the overall
practical accessibility of content, especially in edge case examples he
provides.

Scores fulfil another important role, however. In EVAL TF, several
participants have emphasised that site owners / developers commissioning
accessibility testing are keen to be able to record and track where
problems (SC Failures) exist in order to manage the a11y part of their
development, prioritise remediation, and check whether things have improved
when re-testing. In *this* application context of WCAG-EM, the question
whether the site as a whole will ultimately conform is irrelevant (we know
that most likely it won't, ever). Still, it is important to engage in the
real-world development process in a meaningful way. 

For cases such as Gregg's first example, the site-wide score 86 out of 87
would indeed poorly reflect the site's practical accessibility, but for a
developer, it shows that there is but one particular issue that needs
attention. A page based score in the second example with few minor issues
distributed widely over SCs would arrive at a very good average SC values
across the sample (say 0.99, where 1.0 would be conformance for all pages
in the sample). Again, the developer would know that the site is basically
in good shape, but lots af small issues still need to be addressed. Useful
information indeed.

For scores to adequately reflect the actual accessibility of the site
tested, we would need to include the criticality of issues. Lacking that,
scores will neverthelsss often be instructive from a site development point
of view. 
In my view, the section should stay (it is optional, after all). Perhaps we
need to point out what the resulting scores can be used for, and that they
may not reflect the overall practical accessibility of the site tested.

==========

Priority: high
Location: Step 5.d: Provide a Performance Score (Optional)

Suggested revision: This optional step suggests three different scoring
approaches: per web site, per web page, and per instance. The first two are
pretty uncontroversial. The third approach suggests a score calculation
based on instances on the page "for which each WCAG 2.0 Success Criterion
is applicable". 

In the last Eval TF Telco, there was a consensus decision (with one
abstention) to remove this third type of scoring method:
http://www.w3.org/2013/12/05-eval-minutes.html

In a future version of WCAG-EM, we might express that there are different
ways of calculating scores. If references are given, these might point to
the approach that is currently being proposed (or anything that has emerged
from work on UWEM 2 ?), the Accessibility Priority Tool used by Roger
Hudson and others, or some other additional, informative graded scoring
approach that can reflect the criticality of failed instances. 

These answers were last modified on 12 December 2013 at 12:55:49 U.T.C.
by Detlev Fischer

Answers to this questionnaire can be set and changed at
https://www.w3.org/2002/09/wbs/1/WCAG-EM-20131129/ until 2013-12-17.

 Regards,

 The Automatic WBS Mailer

Received on Thursday, 12 December 2013 12:57:07 UTC