- From: Ian Fette <ifette@google.com>
- Date: Fri, 9 Nov 2007 13:16:10 -0800
- To: "W3C WSC Public" <public-wsc-wg@w3.org>
- Message-ID: <bbeaa26f0711091316n3cca6608tab818385514c0b19@mail.gmail.com>
ACTION-330 is about requiring usability tests for conformance, specifically whether we can make any recommendations on how to conduct required usability testing. My fear is that we are going to get into a situation like the following: We end up having a section of the recommendation saying "Do not show X in the section of the chrome intended to convey trust information" or "Inform the user of X", where usability testing is required to configure out whether the user thought a particular part of chrome conveys trust decisions, or user testing is done to figure out whether the user was actually informed. The person doing the testing then has to design an experiment to test this feature. The person doing the testing has an incentive to construct a test where they will do well (to achieve conformance). You can imagine someone therefore constructing an experiment in which the user is shown help pages first, or given a manual and 1/2 hour to read it, or some other non-realistic setting. This would likely produce a different result than an experiment where the user simply dives right in to using the product. You could also imagine less sinister ways to skew the results. For instance, testing "whether the user was informed" - Someone could decide to sit a user down for a half hour, have them go through a few sites (some of which produce notifications), and then see that the user watched the notices. Another person may say "Well, they notice the dialogs now because this is the first time they're using the product, but after a while they might just ignore them" and instead do a 30-day study, and see that the results on day 30 are very different than a 30-minute user study. Hence, my main concern is that we are going to require usability testing for conformance, and the way the test is constructed will be the primary factor in whether an implementation appears "usable". As such, I think we would have to lay out very clear guidelines on how the usability testing should be done (basically specifying the experimental design), which seems fraught with peril given how different implementations might be and might become over time, or we would have to take a huge leap of faith. Personally, my preference would be to avoid requiring [in the MUST sense] usability testing for conformance in general, and instead come up with good guidelines for how a usability test SHOULD be conducted to address these issues. I believe this fulfills my requirements for ACTION-330. -Ian
Received on Friday, 9 November 2007 21:16:33 UTC