W3C home > Mailing lists > Public > public-wsc-wg@w3.org > November 2007

Re: ACTION-330 Requirements for usability testing for conformace

From: Serge Egelman <egelman@cs.cmu.edu>
Date: Mon, 12 Nov 2007 16:08:50 -0500
Message-ID: <4738C0E2.3020807@cs.cmu.edu>
To: Ian Fette <ifette@google.com>
CC: W3C WSC Public <public-wsc-wg@w3.org>

I think that since these recommendations are to prevent attacks, all the 
proposals should be tested in attack scenarios.  In this case there are 
two areas that need to be covered: does the user understand what the 
feature is conveying, and can the user be fooled.

serge

Ian Fette wrote:
> ACTION-330 is about requiring usability tests for conformance, 
> specifically whether we can make any recommendations on how to conduct 
> required usability testing. My fear is that we are going to get into a 
> situation like the following:
> 
> We end up having a section of the recommendation saying "Do not show X 
> in the section of the chrome intended to convey trust information" or 
> "Inform the user of X", where usability testing is required to configure 
> out whether the user thought a particular part of chrome conveys trust 
> decisions, or user testing is done to figure out whether the user was 
> actually informed.
> 
> The person doing the testing then has to design an experiment to test 
> this feature. The person doing the testing has an incentive to construct 
> a test where they will do well (to achieve conformance). You can imagine 
> someone therefore constructing an experiment in which the user is shown 
> help pages first, or given a manual and 1/2 hour to read it, or some 
> other non-realistic setting. This would likely produce a different 
> result than an experiment where the user simply dives right in to using 
> the product.
> 
> You could also imagine less sinister ways to skew the results. For 
> instance, testing "whether the user was informed" - Someone could decide 
> to sit a user down for a half hour, have them go through a few sites 
> (some of which produce notifications), and then see that the user 
> watched the notices. Another person may say "Well, they notice the 
> dialogs now because this is the first time they're using the product, 
> but after a while they might just ignore them" and instead do a 30-day 
> study, and see that the results on day 30 are very different than a 
> 30-minute user study.
> 
> Hence, my main concern is that we are going to require usability testing 
> for conformance, and the way the test is constructed will be the primary 
> factor in whether an implementation appears "usable". As such, I think 
> we would have to lay out very clear guidelines on how the usability 
> testing should be done (basically specifying the experimental design), 
> which seems fraught with peril given how different implementations might 
> be and might become over time, or we would have to take a huge leap of 
> faith. Personally, my preference would be to avoid requiring [in the 
> MUST sense] usability testing for conformance in general, and instead 
> come up with good guidelines for how a usability test SHOULD be 
> conducted to address these issues.
> 
> I believe this fulfills my requirements for ACTION-330.
> 
> -Ian
> 

-- 
/*
PhD Candidate
Vice President for External Affairs, Graduate Student Assembly
Carnegie Mellon University

Legislative Concerns Chair
National Association of Graduate-Professional Students
*/
Received on Monday, 12 November 2007 21:09:20 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 5 February 2008 03:52:53 GMT