Re: Section 6 - User Test Verification

I just added a cleaned up draft of this to the wiki if anyone wants  
to make changes or additions.

http://www.w3.org/2006/WSC/wiki/NoteUserTestVerification

- Maritza



On Dec 26, 2006, at 3:27 AM, Maritza Johnson wrote:

>
> Agreeing with what a few others have been saying, I think we need  
> to lay out who the user is.
>
> My suggestion is to take each use case and consider what security  
> information is relevant to different levels of users given the task  
> in the use case. With this in mind, when making our recommendation  
> we should consider if the amount of effort necessary to extract the  
> information is appropriate according to  the level of detail/ 
> experience of the user.
>
> When I say depending on the task or the user's level of experience,  
> I'm trying to include cases when someone has the need to be  
> concerned about who issued a certificate, or some other detail that  
> probably shouldn't be displayed to everyone, but should be  
> available on demand.
>
>
> The user test verification should evaluate if the meaning of the  
> security cue is clear to the user, if the necessary information is  
> available given a reasonable amount of effort according to the task  
> and the user's level of experience, and if the meaning can be  
> determined with little to no training.
>
>
> One method of user test verification would involve participants  
> representative of each of the user groups defined as closely as  
> possible, asking them to perform the tasks in the use cases, and  
> asking questions afterwards to gather feedback about how the felt  
> about the security information they were provided with. The format  
> would be similar to an in-lab study followed by a questionnaire.
>
> Another way to gather feedback more directly would be to present  
> participants with the security cues in the context of certain  
> tasks, without asking them to carry out any tasks. The format would  
> be more like an interview, or a focus group. The participant would  
> be shown screen shots or something similar, and asked if the  
> information presented was enough to make a security decision.
>
> With either of these methods questions to ask the participants  
> would include things like: Do you understand the information the  
> cue portrays? What does the cue suggest (to help indicate sources  
> of confusion)? Do you feel this information is easily accessible?  
> Do you think enough relevant information is displayed?
>
> The task-based method is slightly more flexible and can be used to  
> gather feedback that may be more representative of how users might  
> react in a natural setting. It's possible to structure the study in  
> a way that distracts the focus from being purely on browsing  
> securely ( which is the case when the user is browsing at home).  
> Similar to the user study on phishing toolbars, the task-based user  
> study can also be adjusted so training is given halfway through the  
> session so feedback can be gathered about the effects of training  
> on results.
>
> Also, the task-based method can be modified to gauge whether or not  
> a user can be easily tricked into believing a spoofed security cue.  
> Possible spoofs or changes to the security cues could be presented  
> to the participants in the tasks to see if they fall for the fake  
> cues. The study would then be similar to the study in Why Phishing  
> Works.
>
>
>
>
>
> - Maritza
>
> http://www.cs.columbia.edu/~maritzaj/
>
>



- Maritza

http://www.cs.columbia.edu/~maritzaj/