Re: Eye tracking study of Firefox EV indicators

It's quite possible to come down too hard on this study too, though.

Jennifer and her advisors spoke with me several times during the  
planning phases for this study, and I underlined some of the concerns  
I had with a sort of naive approach ("Put users in front of an  
unfamiliar browser, see if the EV indicator reduces phishing.") I am  
confident they understood what I was getting at, and the design of  
this study reflects some of that - going for more appropriate measures  
like user confidence and willingness to transact.  I actually think  
that is a significant improvement over some of the previous work in  
the area (though obviously this is not the first study to use those  
measures.)

I do agree that a novel interface element that seems visually out of  
place will artificially inflate user attention.  I also think that the  
study is critically hampered by testing users who are unfamiliar with  
the interface when its role is to serve as ambient, contextual cueing  
for users who *are* familiar with the interface.  But this isn't  
really news to Jennifer or her advisors either, and they make a point  
of calling it out in the limitations section.

Basically, I think this is progress in terms of academia working with  
browser authors to develop studies we can all agree on.  I think, as  
Mike points out elsewhere, that the eye tracking data is interesting  
in itself (That users might spend fully 10% of their time looking at  
chrome is surprising to me, but might be accounted for by the  
relatively disjoint lab browsing sessions, rather than seamless  
navigation between 20 sites as a normal user might do.)  I think that  
using things like willingness to transact is a more useful approach  
for this kind of work (not purely commercial transactions either, it  
would be interesting to know if experienced users behave differently  
interacting with government organizations).

I think we all know that *something* happens with EV treatment - I'm  
reasonably confident that Paypal didn't *invent* the drop in  
abandonment rates among IE7 users when they switched to EV, even if  
their trumpeting of it can be seen mostly as a marketing ploy, and  
even if reasonable minds can disagree about whether it's a good  
thing.  I think studies like this are part of a collective move  
towards gathering the kind of data that we can more immediately  
recognize, understand, and benefit from ("Ah yes, they're using the  
Dhamija-Close methodology here") -- even if we're not quite there yet.

I'll get off my soap box now, I'm getting dizzy.

Cheers,

Johnathan

On 22-May-08, at 7:46 PM, Ian Fette wrote:

> No offense, and not to be blunt, but this study looks... less than  
> stellar. :( They're testing Firefox 3 beta 1, which, IIRC, didn't  
> even display the site name in the URL bar for EV sites. Then they  
> stick in some crap indicator that looks so god awful and totally out  
> of place with the Firefox UI it's no wonder that people look at it  
> and say "ah ha! people look at it." and claim that the base Firefox  
> 3 browser fails. Lovely.
>
> Sure, you can drop in a dork-o-meter that is totally out of line  
> with the rest of the UI and people will look at it. But is that a  
> good idea? I still don't know what takeaway points I'm supposed to  
> get out of here :( They made something look awful, claimed a number  
> of people looked at it... and?
>
> On Thu, May 22, 2008 at 4:32 PM, Rachna Dhamija  
> <rachna.w3c@gmail.com> wrote:
> Hi all,
>
> Jennifer Sobey and her colleagues at Carleton University published a  
> paper that our group should be aware of:
> http://www.scs.carleton.ca/research/tech_reports/index.php?Abstract=tr-08-10_0010&Year=2008
>
> They conducted an eye tracking study of people using Firefox 3 and  
> observed whether people noticed EV indicators.  They conclude that  
> the new indicators are ineffective because none of the participants  
> noticed them or discovered the clickable regions that reveal site  
> identity details.  The study tested the Firefox 3 Beta 1 release,  
> but the results are still relevant to the interface in the current  
> release.
>
> They also experimented with a new interface for an "identity  
> confidence meter", which is similar to some of the interfaces that  
> we've discussed in the Web Security Score proposal.
>
> The authors welcome our comments on the study.
>
> Rachna

---
Johnathan Nightingale
Human Shield
johnath@mozilla.com