W3C home > Mailing lists > Public > public-wai-evaltf@w3.org > January 2012

RE: Sampling - number of participants in usability studies

From: Sarah Swierenga <sswieren@msu.edu>
Date: Wed, 4 Jan 2012 14:43:12 -0500
To: "'Velleman, Eric'" <evelleman@bartimeus.nl>, "'Eval TF'" <public-wai-evaltf@w3.org>
Cc: "'Vivienne CONWAY'" <v.conway@ecu.edu.au>, "'RichardWarren'" <richard.warren@userite.com>
Message-ID: <00b701cccb19$17f40230$47dc0690$@edu>
Hello,
When we were discussing targeted page sampling during the December
teleconference I mentioned that there is some research literature regarding
sample size for usability evaluations, which are typically qualitative in
nature. Here are links to a few recent articles that may be helpful as we
decide on the number of pages that need a full/manual review:

Hwang, W., and Slavendy, G. (2010, May). Number of People Required for
Usability Evaluation: the 102 rule. Communications of the ACM, 53(5),
130-133. Available online at:
http://cacm.acm.org/magazines/2010/5/87254-number-of-people-required-for-usa
bility-evaluation/comments 

Macefield, R. (2009). How To Specify the Participant Group Size for
Usability Studies: A Practitioner’s Guide. Journal of Usability Studies,
5(1), 34-45. Available online at:
http://www.upassoc.org/upa_publications/jus/2009november/macefield1.html 

Molich, R. (2010, May). A Commentary of “How To Specify the Participant
Group Size for Usability Studies: A Practitioner’s Guide” by Macefield.
Journal of Usability Studies, 5(3), 124-128. Available online at:
http://www.upassoc.org/upa_publications/jus/2010may/molich2.html


Best regards,
Sarah

Sarah J. Swierenga, PhD, CPE
Director, Usability/Accessibility Research and Consulting (UARC)
University Outreach and Engagement
Professor by Courtesy
Dept. of Telecommunication, Information Studies and Media
College of Communication Arts and Sciences
Michigan State University
93 Kellogg Center
East Lansing, MI 48824-1022
Telephone: (517) 353-8977
Fax: (517) 432-9541
E-mail: sswieren@msu.edu
WWW: usability.msu.edu


-----Original Message-----
From: Vivienne CONWAY [mailto:v.conway@ecu.edu.au] 
Sent: Thursday, December 15, 2011 9:41 AM
To: Velleman, Eric; RichardWarren; Eval TF
Subject: RE: Sampling

HI all

Unless I'm missing something, if we are talking about random sampling
methods, how do we make sure they include those 'complete processes'.  Do we
look at doing random sampling, plus complete processes, plus core elements
of the website (website purpose).?


Regards

Vivienne L. Conway, B.IT(Hons)
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au
v.conway@webkeyit.com
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual
or entity named above. If you are not the intended recipient, you are
notified that any dissemination, distribution or copying of this email is
strictly prohibited. If you have received this email in error, please notify
me immediately by return email or telephone and destroy the original
message.
________________________________________
From: Velleman, Eric [evelleman@bartimeus.nl]
Sent: Thursday, 15 December 2011 7:35 PM
To: RichardWarren; Eval TF
Subject: RE: Sampling

Hi,

Yes, agree, the evaluation will need to specify the resources that have been
evaluated.

If the evaluation needs to be replicable and allow synchronous or
asynchronous comparisons (like monitoring) the evaluation sample must be
generated by a uniform random procedure that is partly described by Richard
in an earlier mail (see bottom of this message). Partly, because the
situation for our uniform random procedure is a bit more complicated than
with WCAG 1.0. There are some additional factors at work here that are
described in the Scope section like accessibility support and use of
different technologies and more. This is covered in WCAG 2.0 like also
described by Alistair in an earlier mail but we will have to check if that
is enough for the purpose of the evaluation report.

Question: Can we make a list of what should minimally be in the core
resource list (if available in the scope of the Website that is being
evaluated)? We will discuss the size of the sample later.

Using Richards list I come to:

Home Page,
Site Map,
Section landing pages (is there a maximum?)
Any sub-section landing pages (usually linked to from the section landing
pages)
Forms
Data tables
Multimedia (maybe we have to be more specific here)

While reading, the following additions seem interesting to add:
Help resource
Contact information resource
Search and extended search resources including resulting resources
Distinct web technology pages (...)
Pages with other programming languages
CSS alternatives for mobile, (more..)
Frames (are they still used?)

Also:
Resources representative of each category of resources having a
substantially distinct “look and feel” (typically representative of distinct
underlying site “templates”) (if identifiable).
Resources describing accessibility features and / or the accessibility
policy of the site (if any).


The resource list as a whole should, as far as possible, collectively
address all the applicable sampling objectives within the scope of the
evaluation.

Kindest regards,

Eric




________________________________________
Van: RichardWarren [richard.warren@userite.com]
Verzonden: donderdag 15 december 2011 11:09
Aan: Eval TF
Onderwerp: Re: Sampling

Hi
Eric is correct that we are evaluating (and therefore sampling) at a moment
in time.

I presume that we will include documentation of pages visited etc. so that
the process can be audited (we normally keep a checked list from the site
map). This check list can be just as useful for Vivienne's monitoring.

Richard

-----Original Message-----
From: Vivienne CONWAY
Sent: Thursday, December 15, 2011 2:11 AM
To: Velleman, Eric ; Boland Jr, Frederick E. ; RichardWarren ; Eval TF
Subject: RE: Sampling

Hi all

For my current research project, I have used a targetted sampling method.
As well as scanning the whole site with automated tools to check for trends,
I have chosen 5 pages from each website to check manually according to the
WCAG 2.0 guidelines.  These pages reflect as many WCAG 2.0 checkpoints as
possible: home page, contact us page (or something with a form), page with
photos, page with multimedia, and a page describing their services or other
page with some complexity.

As I check these websites repeatedly, I will check the same pages to see
their progression (or degeneration) over time.


Regards

Vivienne L. Conway, B.IT(Hons)
PhD Candidate & Sessional Lecturer, Edith Cowan University, Perth, W.A.
Director, Web Key IT Pty Ltd.
v.conway@ecu.edu.au
v.conway@webkeyit.com
Mob: 0415 383 673

This email is confidential and intended only for the use of the individual
or entity named above. If you are not the intended recipient, you are
notified that any dissemination, distribution or copying of this email is
strictly prohibited. If you have received this email in error, please notify
me immediately by return email or telephone and destroy the original
message.
________________________________________
From: Velleman, Eric [evelleman@bartimeus.nl]
Sent: Thursday, 15 December 2011 7:22 AM
To: Boland Jr, Frederick E.; RichardWarren; Eval TF
Subject: RE: Sampling

Frederick,

your remark made me think of time lapse sampling:

It could be that a website/webpage changes over time on purpose. For
example: it could show images and text related to the time of the day or the
feeling of the owner. If the owner feels bad at 8 am, then the page is dark
but after his first cup of coffee, the page looks bright and happy...
In that case it could be interesting to sample the same page at different
times but I do not know how to put this into the methodology, maybe in a
footnote?

Eric

________________________________
Van: Boland Jr, Frederick E. [frederick.boland@nist.gov]
Verzonden: woensdag 14 december 2011 21:48
Aan: RichardWarren; Eval TF
Onderwerp: RE: Sampling

We may also need to sample over time, since a site’s pages/content may
change over time, which could affect WCAG2.0 conformance and/or resultant
accessibility of the site.. or provide a date/time (required component) of
evaluation for pages/site..  Questions that may arise in this regard are:
how often to sample, etc. (for example,  do we just want to sample when
there are major content changes, or do we just want to sample at regular
intervals regardless of any perceived changes, or do we want to apply
different strategies for different parts of a site?)

Thanks Tim Boland

From: RichardWarren [mailto:richard.warren@userite.com]
Sent: Wednesday, December 14, 2011 2:32 PM
To: Eval TF
Subject: Sampling

Hi,

To reduce the effort of evaluating a large site I suggest that there are
three methods of sampling the site’s pages/content which can be coupled with
a barrier identification technique to avoid constant repetition.

The three sampling techniques for manual evaluation are  :-

1) Random sampling  - selecting a number of pages at random. This can be
done by making a random selection from the site map, or to take every tenth
(or other suitable number) of links from the site map.

2) Structure sampling – selecting the higher level structural pages such as
Home Page, Site Map, section landing pages (usually linked from the Home
page within the main navigation bar), any sub-section landing pages (usually
linked to from the section landing pages, plus (if not already found) a
sample of pages containing elements such as forms, data tables and
multi-media.

3) Task orientated sampling – Completing the key tasks on the site required
to meet the site’s purpose. This might include tasks such as to source
certain information, place an order or participate in a discussion.

Barrier identification reduces effort further by noting examples of common
failures in technique employed within the site and once identified and
commented on we can ignore further occurrences within that section.

We use a combination of all three sampling methods. We start with 2
(structured sample) to explore the site and obtain an overview of its’
purpose etc. Next we attempt the key tasks.  Then we do a random sample
(skipping any pages already sampled).

Richard

This e-mail is confidential. If you are not the intended recipient you must
not disclose or use the information contained within. If you have received
it in error please return it to the sender via reply e-mail and delete any
record of it from your system. The information contained within is not the
opinion of Edith Cowan University in general and the University accepts no
liability for the accuracy of the information provided.

CRICOS IPC 00279B

This e-mail is confidential. If you are not the intended recipient you must
not disclose or use the information contained within. If you have received
it in error please return it to the sender via reply e-mail and delete any
record of it from your system. The information contained within is not the
opinion of Edith Cowan University in general and the University accepts no
liability for the accuracy of the information provided.

CRICOS IPC 00279B
Received on Wednesday, 4 January 2012 19:43:43 GMT

This archive was generated by hypermail 2.3.1 : Friday, 8 March 2013 15:52:12 GMT