W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > October 2012

WCAG-EM comments. Section 3.3 Select a Representative Sample

From: Ramón Corominas <rcorominas@technosite.es>
Date: Wed, 17 Oct 2012 11:55:58 +0200
Message-ID: <507E80AE.1010507@technosite.es>
To: Eval TF <public-wai-evaltf@w3.org>
Dear Eval TF,

Find below my comments for section 3.3 "as it is". I'm conscious that 
this section will probably change in the near future.

3.3 Step 3: Select a Representative Sample

[WCAG-EM] "Depending on the size and complexity of the website, the size 
of this sample will vary.

It also depends on the goal and the purpose of the analysis. Basic 
evaluation, pre-sales, multi-site comparisons... All of them can reduce 
the sample size.

In general, the sampling process is very webpage-oriented. Some steps 
need additional guidance for sampling functionalities/tasks in web apps.

3.3.1 Step 3.a: Include Common Web Pages of the Website

[WCAG-EM] "All common web pages, including the common states of these 
web pages for web applications, must be part of the selected sample."

"All" sounds too strict and in many cases will not offer better 
evaluation results. Many of the common web pages may be almost 
identical, sharing design, structure, technologies, etc., so it makes no 
sense to evaluate all of them. For web applications this can be even 
worse (for example, in an online text editor with only one web page, 
should we test every toolbar button and every function?)

Maybe we can include a final step: "Step 3.e Filter the sample", so we 
can adjust the sample size by removing (some) pages that are almost 

3.3.2 Step 3.b: Include Exemplar Instances of Web Pages

In principle I agree, but I'm unsure if, in practice, this can lead to 
too many redundancies.

3.3.3 Step 3.c: Include Other Relevant Web Pages

Since "common web pages" are already defined as "relevant", I would 
change this title to something more specific: "Include Web Pages that 
are Relevant for People with Disabilities". The word "Other" should be 
omitted, because some pf these pages may be already in the sample for 
other reasons.

In addition, some of the described web pages should be already in the 
sample, as per 3.3.1 and the definition of "common web pages": site 
help, entry pages. Although it should already be included, the "contact" 
page is also very relevant for PwD, so it should be explicitly mentioned.

For web apps, some relevant options could be "help", 
"settings/preferences/options", "shortcuts"...

3.3.4 Step 3.d: Include Complete Processes in the Sample

[Ed] I would suppress "in the sample" (as in the other steps)

This step seems to be page-oriented only. With the current definition of 
"web page" that is used within the methodology (including "states" of a 
page), a process can also be a "task" performed in a web app. The 
sampling methodology should guide on how to select processes/tasks in 
web apps. I would probably separate the "page-oriented sampling" from 
the "functionality-oriented" one, including an additional step, or at 
least clarifying the method for web apps.

Indeed, I would not use a "state of a web page" as a synonym of "web 
page", since it introduces more confusion about the concepts. Instead, I 
would explicitly mention that the sample should cover web pages AND 
functionalities/tasks performed in a web page/app.

Lastly, as mentioned above, I would add another optional step aimed to 
filter/refine the sample to eliminate unnecessary redundancies. For 
example, we had to evaluate a website where the starting sample was 
almost 70 pages, but most of them shared the same structure or 
functionalities, so the final sample was about 25.


3.3.5 Step 3.e Filter the Sample to Eliminate Unnecessary Redundancy 

"Once a complete sample has been selected according to the above steps, 
the evaluator may re-explore the sample to detect web pages whose 
structure, design, technologies, etc. are similar to those of other web 
pages in the sample. The evaluator may then exclude web pages from the 
sample, provided that the filtered sample is still representative of the 
whole website.

This filtering must not contradict Step 3.b: Include Exemplar Instances 
of Web Pages."

Maybe additional requirements can be added to "pages that cannot be 
excluded" such as "home", "accessibility", "sitemap", "search results" 
or others, but I think that the filtering step is necessary for most 
manual evaluations. Of course, this step should not contradict any other 
steps, including the amount/percentage of random pages in the sample.

Kind regards,

Ramón Corominas
Accessibility specialist
Technosite - Fundación ONCE
+34 91 121 0330
Received on Wednesday, 17 October 2012 09:56:42 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:40:23 UTC