Re: Section 6 - User Test Verification

Thanks Maritza. 

All, from a process and schedule point of view (keep reading folks!), my 
assumption (as laid out in assumptions) is that we'll do the user test 
verification where testing usually occurs, between Candidate 
Recommendations and Proposed Recommendations. Right now the schedule gives 
us three months. So we have to either scope the testing to fit, or grow 
the time estimate. Now is the time for us to put a stake in the ground on 
what testing we will do. Can this fit in the timeframe? My experience is 
that the hardest part to control, time wise, is lining up the 
participants. 

          Mez

Mary Ellen Zurko, STSM, IBM Lotus CTO Office       (t/l 333-6389)
Lotus/WPLC Security Strategy and Patent Innovation Architect




Maritza Johnson <maritzaj@cs.columbia.edu> 
Sent by: public-wsc-wg-request@w3.org
01/10/2007 06:27 PM

To
W3 Work Group <public-wsc-wg@w3.org>
cc

Subject
Re: Section 6 - User Test Verification







I just added a cleaned up draft of this to the wiki if anyone wants to 
make changes or additions.

http://www.w3.org/2006/WSC/wiki/NoteUserTestVerification

- Maritza



On Dec 26, 2006, at 3:27 AM, Maritza Johnson wrote:


Agreeing with what a few others have been saying, I think we need to lay 
out who the user is.

My suggestion is to take each use case and consider what security 
information is relevant to different levels of users given the task in the 
use case. With this in mind, when making our recommendation we should 
consider if the amount of effort necessary to extract the information is 
appropriate according to  the level of detail/experience of the user.

When I say depending on the task or the user's level of experience, I'm 
trying to include cases when someone has the need to be concerned about 
who issued a certificate, or some other detail that probably shouldn't be 
displayed to everyone, but should be available on demand.


The user test verification should evaluate if the meaning of the security 
cue is clear to the user, if the necessary information is available given 
a reasonable amount of effort according to the task and the user's level 
of experience, and if the meaning can be determined with little to no 
training.


One method of user test verification would involve participants 
representative of each of the user groups defined as closely as possible, 
asking them to perform the tasks in the use cases, and asking questions 
afterwards to gather feedback about how the felt about the security 
information they were provided with. The format would be similar to an 
in-lab study followed by a questionnaire.

Another way to gather feedback more directly would be to present 
participants with the security cues in the context of certain tasks, 
without asking them to carry out any tasks. The format would be more like 
an interview, or a focus group. The participant would be shown screen 
shots or something similar, and asked if the information presented was 
enough to make a security decision.

With either of these methods questions to ask the participants would 
include things like: Do you understand the information the cue portrays? 
What does the cue suggest (to help indicate sources of confusion)? Do you 
feel this information is easily accessible? Do you think enough relevant 
information is displayed? 

The task-based method is slightly more flexible and can be used to gather 
feedback that may be more representative of how users might react in a 
natural setting. It's possible to structure the study in a way that 
distracts the focus from being purely on browsing securely ( which is the 
case when the user is browsing at home). Similar to the user study on 
phishing toolbars, the task-based user study can also be adjusted so 
training is given halfway through the session so feedback can be gathered 
about the effects of training on results.

Also, the task-based method can be modified to gauge whether or not a user 
can be easily tricked into believing a spoofed security cue. Possible 
spoofs or changes to the security cues could be presented to the 
participants in the tasks to see if they fall for the fake cues. The study 
would then be similar to the study in Why Phishing Works.





- Maritza

http://www.cs.columbia.edu/~maritzaj/





- Maritza

http://www.cs.columbia.edu/~maritzaj/