[Content Security Policy] Usability?

Gaz has made a couple of comments about the usability of content 
security policy based on gut feelings about it, but it occurs to me that 
we could probably conduct some informal usability testing (e.g. buy a 
friend a beer and ask them to try something) to see if Gaz's vague sense 
of unease is something that hits other people too, and whether that 
impacts their ability to use CSP in the wild.

I get the sense that it's hard to debate the usability without more data.

Here's a really rough study design:

----

Target user: a web developer, preferably with some security knowledge to 
save time explaining (not the best possible test, but a good start)

Tasks:
1. Familiarize yourself with CSP.  You can take as long as you need to 
read the documentation, but please try to keep rough track of how long 
you spend on this task. (provide links)

2. Attempt to construct a policy for http://www.mozilla.org/ [or another 
website which already has a CSP policy... bugzilla?]  Please try to keep 
track of how long this takes.

[  Other sites that might be worth testing:
    a) A toy site with easy behaviour [maybe used as a warm-up?]
    b) A site you have worked on [something with which the user is 
reasonably familiar]
    c) http://www.cnn.com/ [or some other complex, popular site which 
was not designed with CSP in mind. CNN might actually be too complex for 
a short study.]]

Survey:
[After familiarization with CSP]
1. How challenging did you find it to learn CSP? [scale of 1-7]
2. How long did you take to familiarize yourself with CSP? [time ranges]
3. Please provide any additional comments you might have (e.g. Was 
anything especially confusing?  What about easy?)

[Repeat for all sites attempted]
1. What is the policy you created?
2. How challenging was creating a policy for site (a)? [1-7]
3. How long did you take to create a policy for site (a)? [time ranges]
4. How confident are you in your policy's ability to mitigate attacks? [1-7]
5. Please provide any additional comments you might have (e.g. did you 
encounter any problems?)

[Debriefing survey after user has attempted at least one site]
1. Would you use CSP on your sites? [1-7]
2. Why or why not?
3. Please provide any additional comments you might have (e.g. What did 
you like about CSP? What did you dislike?)

-----



Does that sound reasonable?  I'd love suggestions from those more 
experienced in usability study design, but this could be ok for a first 
pass where we grab a few folk and try to see what other questions we 
should be asking.  We can see if people actually do create policy with 
weird syntax because they mis-understand, or whether it's actually 
fairly easy for new users to create policies.

If we could choose a good site or two to start with, maybe we could find 
a few users and give it a try?

  Terri

Received on Monday, 31 January 2011 09:16:03 UTC