Re: ACTION-301: Usability review of Identity Signal

On 26-Oct-07, at 12:34 PM, Rachna Dhamija wrote:

> On 10/26/07, Mary Ellen Zurko <>  
> wrote:
>> I don't see why it is (and I expect kind and informative responses to
>> naivete :-). The testing of understandability of visual icons goes  
>> much
>> further back than usability testing around user attacks. I would  
>> expect that
>> kind of UT would be the most appropriate.
>>         Mez

Yeah, my apologies.  I guess what I meant was that much of our Shared  
Bookmarks consists of tests which specifically address the  
quantitative fitness of various UIs as defense mechanisms against  
phishing.  To the extent that "statistically significant improvements  
to attack resiliency" represented any kind of bar which must be met,  
I was cautioning that I didn't think that would be appropriate here.   
Of course there are a wealth of methodologies that can yield  
interesting results, and I'm all for pursuing them.

> To do this, you need to define what you mean by "understanding
> identity".  What exactly do you want users to know?  E.g. "when a user
> visits the Bank X website, they understand that they are at Bank X and
> not Y", or "when they visit site A that does not have an EV
> certificate they understand that a third party has not verified the
> identity of the site".  Your standard might be higher e.g. "they might
> be suspicious" in some circumstances or be able to verify the identity
> in a phishing attack that spoofs Larry (I know this is not your goal).
>  Once you define the goals, we can ask users to use the interface and
> then test them or interview them to see if your goals were met.

So, the goal of the identity signal is to provide a stronger cue for  
site identity than what currently exists in chrome (i.e. url,  
padlock, etc.)  Practically speaking, I think that means that for  
entities with which the user has an existing relationship (stores  
with brick and mortar presence, government institutions, online shops  
with which they have prior history) the identity signal should help  
them to understand whether a given web site belongs to/is operated by  
that entity.  You raise an interesting point too, that it would be  
good to know how they perceived the lack of such validation.

> We can do this in a lab, by distributing the client to users and then
> interviewing them, or you could instrument the client. Obviously, you
> can get more accurate answers to behavior questions (e.g. do users
> discover Larry on their own?) if you have a long term study with an
> instrumented client.  However, if you have questions about what users
> *understand*, there is nothing that beats the kind of data you can get
> by showing users the interface and interviewing them face to face.
> Computer scientists really discount the value of this methodology, and
> I think our designs suffer for it.

I don't know if I'm being lumped in with the computer scientists  
there or not - my training is in cognitive psychology - but I  
absolutely agree that interview data is hard to beat in terms of  
understanding mental models.  It's hampered here by the short  
timeframe somewhat - self-report is going to have trouble reflecting  
the long-term habit forming that might or might not develop around  
the UI, but I do think the information would still be quite valuable.

If I'm reading _Why Phishing Works_ correctly, more than 75% of users  
included the address bar in their judgements about site legitimacy.   
Since the study was concerned with phishing, it's not clear how well  
the numbers would carry over to other contexts, but nevertheless, it  
makes me personally more inclined to look for interview data about  
users' understanding than to pursue other methodologies that might be  
suggested (e.g. eye tracking data).  Again though, I defer to those  
who do this for a living - do I understand you correctly, Rachna, to  
be suggesting that goals stated in terms of user understanding are  
suitably tested by an interview setup like the one you describe?



Johnathan Nightingale
Human Shield

Received on Monday, 29 October 2007 13:27:38 UTC