Re: first cut usability walk through

On 2007-08-01 13:11:50 -0400, Serge Egelman wrote:

>> - Identity Signal, Page Security Score, and the EV part of the
>>   proposals are pretty much focused on the same topic --
>>   passive indicators, and when to show them.  However, we have
>>   no language in the proposals so far that would usefully tell
>>   us what these indicators would look like.

> We don't need to know what the specific indicators look like if
> the underlying concepts are flawed.  This is what this study
> examines.  If users ignore the most flashy passive indicators,
> then using any type of passive indicator is a nonstarter.

I'd beg to differ.

There are at least two different concepts around:

- Teach people that a specific symbol (color, padlock, etc) has a
  certain meaning.

- Teach people to look in a specific place for certain information,
  e.g., jurisdiction of a company.

In the first case, we essentially teach people about new metaphors.
This is good if we want them to act quickly, but it needs teaching
-- see street signals.

In the second case, we teach them to find stuff expressed in a way
that is expressed in their own language.  This is good if we want
people to get information without much teaching, but might have
different implications on their attention.

I'm seeing these two concepts overlap in most of the proposals; yet,
they're vastly different if you consider communication to humans.
I'd therefore not be surprised if you'd find that, e.g., either
all-text or all-symbol signals were ineffective, but could find some
useful mix.

I actually wonder to what extent that kind of question has been
studied for traffic signs (to give just one example), where the mix
is actually different depending on countries and cultures.

>>   Working on an editor's draft for what the rec track document
>>   might look like, one question is what attributes about the
>>   issuer and subject would actually be displayed in the
>>   identity signal, and under what conditions.

> I'm not sure this matters for the purpose of testing.  If we're
> just displaying identity information, we shouldn't see any
> statistically significant results based on what type of
> information is displayed. We're testing if users will notice
> *any* information gleaned from the certificates and displayed
> passively.

Are you testing whether they are *noticing* the information, or
whether they are *acting* on that information in the desired way?
These are different things.

(Think of the "the padlock is intriguing" kind of responses in some
of the user studies.)

>> - The proposed experiment for EV doesn't actually check whether
>>   people understand the indicator; it rather checks whether the
>>   absence of these indicators can be used as a hook to social
>>   engineer users into subverting the integrity of their
>>   browser. That's a somewhat different question.
> 
> How is that a different question?  

The proposed experiment involved an "install a plug-in to see a
green bar" style interaction, if I recall correctly; it therefore
also tests malware installation paths, and creates a risk of false
positives if, e.g., people got irritated by the plug-in part.

Therefore, the proposal during the call to not throw malware into
the mix for this particular experiment, but just flash up a
JavaScript powered idiot-box style dialogue that says something like
this...

	To guarantee the security and safety of your online banking
	experience, Bank of Suburbia's web site is protected with an
	Extended Validation certificate from foo-bar, the most
	trusted name in eCommerce.
	
	To experience our Extended Validation Enhanced online
	banking services, please acknowledge this Security Advisory
	by clicking "OK".

	Online banking with Bank of Suburbia is secure, and
	protected by foo-bar's Extended Validation Guarantee.

	[ OK ]

(Replace foo-bar by some known name, maybe remove the cable news
pun.  And yes, I do find my energy to find an effective
social-engineering dialogue a bit scary. ;-)

The much broader question is of course what kinds of success
criteria for possible material we'll come up with as far as user
studies and the like are concerned...

Regards,
-- 
Thomas Roessler, W3C  <tlr@w3.org>

Received on Friday, 3 August 2007 12:22:01 UTC