Comments on WCAG-EM Public Working Draft version 26 February 2013

Complete Processes: This is a vague notion b/c it relies on “process”
which is itself not defined, and difficult to define

I would try to redefine it in terms of goals that users try or may want
to try to achieve. All pages (from a starting point) that have to be
visited so that a certain goal, out of a given list of goals/use cases,
have to be audited.

Templates: The auditor has not enough info regarding which templates
were used, if any.

Website: self enclosed? the web is by definition not self enclosed

Web page: this does not say anything of what can be inside this
container. For something like gmail, I need to test many different
states in which the “page” can be. How does the methodology help me
decide which states to consider?

Conformance Evaluation Procedure: in a usability report one usually
starts by saying which method was used. Here you are assuming that there
are more than 1 method, but you really do not list/present them.

Step 1.a: second “where possible, it is recommended to use
formalizations such as regular expressions or ..” – this looks to me
useless. Why is a list of page titles and their URI’s not sufficient?

Step 3.e: why 5? why 5%? Where does it come from?
How does one select a random sample? Literally that would mean to first
list all (ALL!) the pages and then using a random number generator,
select the sample. A variation could be a random stratified sample,
where pages are first grouped into categories (like product description
pages) and a random sample is selected from each group. Another way is a
sample generated through random walks in the site (i.e. randomly
selecting one link among the available and not yet visited ones)
Notice that even a pure random sample of pages has to cope with
“complete process”. And hence has to be extended.
The size of the sample depends on the kind of results that are needed.
For statistical significance, the size depends on the spread/variance of
the measured variable. Hence size cannot be decided a priori. Other
auditing goals might call for other criteria, like a coverage criterion
that aims at capturing all the possible types of problems.

Step 4: Introduction: Very general question: in reality what do people
do? For example the Germans that follow the BITV, what methodology do
they employ?
Or the Knowbility.org people that run the accessibility rallies?
Is this methodology connected to those?
Because there’s no bibliography in this document I think it is not. And
it should not be such.

Step 4.a: if one enumerates the use cases then the sample of pages may
change, as per “complete process” requirement. Hence this step should
come first.

Step 5.a (Basic Report): Rubin, in this book on usability testing, gives
a neat description of different goals that usability investigations
might have. Here something similar could be done.

Step 5.c: Performance score? Who’s performance? In which sense?
Are you talking of accessibility metrics, or expected performance by
users belonging to certain categories and trying to achieve certain goals?




-- 

  Giorgio Brajnik
______________________________________________________________________
Dip. di Matematica e Informatica | voice: +39 (0432) 55.8445
Università di Udine | fax: +39 (0432) 55.8499
Via delle Scienze, 206 | email: brajnik@uniud.it
Loc. Rizzi -- 33100 Udine -- ITALY | http://www.dimi.uniud.it/giorgio

Received on Tuesday, 2 July 2013 10:28:45 UTC