W3C home > Mailing lists > Public > public-wsc-wg@w3.org > May 2008

Re: Eye tracking study of Firefox EV indicators

From: Johnathan Nightingale <johnath@mozilla.com>
Date: Tue, 27 May 2008 09:26:25 -0400
Message-Id: <4B4DA1C8-7766-4135-B481-D7728D1B85F6@mozilla.com>
To: W3C WSC Public <public-wsc-wg@w3.org>
It's quite possible to come down too hard on this study too, though.

Jennifer and her advisors spoke with me several times during the  
planning phases for this study, and I underlined some of the concerns  
I had with a sort of naive approach ("Put users in front of an  
unfamiliar browser, see if the EV indicator reduces phishing.") I am  
confident they understood what I was getting at, and the design of  
this study reflects some of that - going for more appropriate measures  
like user confidence and willingness to transact.  I actually think  
that is a significant improvement over some of the previous work in  
the area (though obviously this is not the first study to use those  
measures.)

I do agree that a novel interface element that seems visually out of  
place will artificially inflate user attention.  I also think that the  
study is critically hampered by testing users who are unfamiliar with  
the interface when its role is to serve as ambient, contextual cueing  
for users who *are* familiar with the interface.  But this isn't  
really news to Jennifer or her advisors either, and they make a point  
of calling it out in the limitations section.

Basically, I think this is progress in terms of academia working with  
browser authors to develop studies we can all agree on.  I think, as  
Mike points out elsewhere, that the eye tracking data is interesting  
in itself (That users might spend fully 10% of their time looking at  
chrome is surprising to me, but might be accounted for by the  
relatively disjoint lab browsing sessions, rather than seamless  
navigation between 20 sites as a normal user might do.)  I think that  
using things like willingness to transact is a more useful approach  
for this kind of work (not purely commercial transactions either, it  
would be interesting to know if experienced users behave differently  
interacting with government organizations).

I think we all know that *something* happens with EV treatment - I'm  
reasonably confident that Paypal didn't *invent* the drop in  
abandonment rates among IE7 users when they switched to EV, even if  
their trumpeting of it can be seen mostly as a marketing ploy, and  
even if reasonable minds can disagree about whether it's a good  
thing.  I think studies like this are part of a collective move  
towards gathering the kind of data that we can more immediately  
recognize, understand, and benefit from ("Ah yes, they're using the  
Dhamija-Close methodology here") -- even if we're not quite there yet.

I'll get off my soap box now, I'm getting dizzy.

Cheers,

Johnathan

On 22-May-08, at 7:46 PM, Ian Fette wrote:

> No offense, and not to be blunt, but this study looks... less than  
> stellar. :( They're testing Firefox 3 beta 1, which, IIRC, didn't  
> even display the site name in the URL bar for EV sites. Then they  
> stick in some crap indicator that looks so god awful and totally out  
> of place with the Firefox UI it's no wonder that people look at it  
> and say "ah ha! people look at it." and claim that the base Firefox  
> 3 browser fails. Lovely.
>
> Sure, you can drop in a dork-o-meter that is totally out of line  
> with the rest of the UI and people will look at it. But is that a  
> good idea? I still don't know what takeaway points I'm supposed to  
> get out of here :( They made something look awful, claimed a number  
> of people looked at it... and?
>
> On Thu, May 22, 2008 at 4:32 PM, Rachna Dhamija  
> <rachna.w3c@gmail.com> wrote:
> Hi all,
>
> Jennifer Sobey and her colleagues at Carleton University published a  
> paper that our group should be aware of:
> http://www.scs.carleton.ca/research/tech_reports/index.php?Abstract=tr-08-10_0010&Year=2008
>
> They conducted an eye tracking study of people using Firefox 3 and  
> observed whether people noticed EV indicators.  They conclude that  
> the new indicators are ineffective because none of the participants  
> noticed them or discovered the clickable regions that reveal site  
> identity details.  The study tested the Firefox 3 Beta 1 release,  
> but the results are still relevant to the interface in the current  
> release.
>
> They also experimented with a new interface for an "identity  
> confidence meter", which is similar to some of the interfaces that  
> we've discussed in the Web Security Score proposal.
>
> The authors welcome our comments on the study.
>
> Rachna

---
Johnathan Nightingale
Human Shield
johnath@mozilla.com
Received on Tuesday, 27 May 2008 13:27:32 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 27 May 2008 13:27:33 GMT