Re: Section 6 - User Test Verification

Agreeing with what a few others have been saying, I think we need to  
lay out who the user is.

My suggestion is to take each use case and consider what security  
information is relevant to different levels of users given the task  
in the use case. With this in mind, when making our recommendation we  
should consider if the amount of effort necessary to extract the  
information is appropriate according to  the level of detail/ 
experience of the user.

When I say depending on the task or the user's level of experience,  
I'm trying to include cases when someone has the need to be concerned  
about who issued a certificate, or some other detail that probably  
shouldn't be displayed to everyone, but should be available on demand.


The user test verification should evaluate if the meaning of the  
security cue is clear to the user, if the necessary information is  
available given a reasonable amount of effort according to the task  
and the user's level of experience, and if the meaning can be  
determined with little to no training.


One method of user test verification would involve participants  
representative of each of the user groups defined as closely as  
possible, asking them to perform the tasks in the use cases, and  
asking questions afterwards to gather feedback about how the felt  
about the security information they were provided with. The format  
would be similar to an in-lab study followed by a questionnaire.

Another way to gather feedback more directly would be to present  
participants with the security cues in the context of certain tasks,  
without asking them to carry out any tasks. The format would be more  
like an interview, or a focus group. The participant would be shown  
screen shots or something similar, and asked if the information  
presented was enough to make a security decision.

With either of these methods questions to ask the participants would  
include things like: Do you understand the information the cue  
portrays? What does the cue suggest (to help indicate sources of  
confusion)? Do you feel this information is easily accessible? Do you  
think enough relevant information is displayed?

The task-based method is slightly more flexible and can be used to  
gather feedback that may be more representative of how users might  
react in a natural setting. It's possible to structure the study in a  
way that distracts the focus from being purely on browsing securely  
( which is the case when the user is browsing at home). Similar to  
the user study on phishing toolbars, the task-based user study can  
also be adjusted so training is given halfway through the session so  
feedback can be gathered about the effects of training on results.

Also, the task-based method can be modified to gauge whether or not a  
user can be easily tricked into believing a spoofed security cue.  
Possible spoofs or changes to the security cues could be presented to  
the participants in the tasks to see if they fall for the fake cues.  
The study would then be similar to the study in Why Phishing Works.





- Maritza

http://www.cs.columbia.edu/~maritzaj/