W3C home > Mailing lists > Public > public-wsc-wg@w3.org > November 2007

Re: ACTION-335 logotypes and ISSUE-96 discussion

From: Serge Egelman <egelman@cs.cmu.edu>
Date: Tue, 13 Nov 2007 18:39:53 -0500
Message-ID: <473A35C9.2060609@cs.cmu.edu>
To: Ian Fette <ifette@google.com>
CC: Dan Schutzer <dan.schutzer@fstc.org>, "Hallam-Baker, Phillip" <pbaker@verisign.com>, W3C WSC Public <public-wsc-wg@w3.org>



Ian Fette wrote:
> Disagree.
> 
> While I don't agree with all of Phil's points, there was one that I 
> definitely agree with that Serge seems to have glossed over. That would 
> be the point about whether you're testing the user over a half-hour in a 
> lab, or a longer (30+ day) field-study in their natural environment. 
> Phil's point was that anything new and disruptive is likely to show a 
> strong effect in the short-term, but over the long-term the effect may 
> be drastically different (including causing people to stop using the 
> product). This is a very good point, and I think that if possible we 
> should aim to do a longer field-study as opposed to a 30m in-lab study.
> 

This is something that can be examined later.  There are pros and cons 
for both types of study designs.  But then again, if your only argument 
is that the effect is stronger in the lab study, I'm not sure how that's 
a problem.  If the users can be easily fooled in the best case scenario 
(or they simply don't notice/trust the indicators), then it's likely 
this effect will only be stronger in the wild.  Of course playing arm 
chair quarterback here isn't going to do much good.  We need an actual 
study design before we can critique it.  To do that we need to first 
figure out which questions we want the study to answer.

> As for "testing them in a perfect world" - I have no idea why this is a 
> good experiment to run, because we know that we will never be operating 
> in a perfect world. I'm not saying we should test in a world with zero 
> adoption, but rather I'm saying that we should try to figure out (guess) 
> what /reasonable/ adoption is, and test in that world. We already know 
> that there are some sites that are not adopting EV because of the cost 
> model. I'm sure someone is more knowledgeable about the specifics than 
> I, but my understanding is that, for instance, Google could not buy one 
> EV certificate for google.com <http://google.com> and use it across all 
> of our numerous servers, rather we would have to pay some increased 
> (large) fee based on number of servers. (Also, does EV support wildcard 
> certs?). Given that, you can come up with a list of companies for which 
> EV would be very expensive and likely not adopted (eBay?), and test with 
> the assumption that those sites won't adopt. What does that do to the 
> overall model?

This is kind of a no-brainer.  The system will not work when there's 
very low adoption.  Users will go to a website, see no EV certificate 
indicator, and just assume that the site never had one.  That's because 
this is the norm.  Thus, if we assume everyone adopts EV, we can test 
the rate of success in the best case.  If the system is not successful 
in the best case scenario, then we know it won't be successful in 
conditions worse than that.  Likewise, if we test the system under 
"reasonable" circumstances (forgetting for a second that "reasonable" is 
completely subjective and we're unlikely to agree on that definition) 
and it's a failure or success, the validity of the study comes into 
question because someone will invariably ask, "well, what happens when 
everyone adopts EV?"

If we're going to agree that a minority of sites will adopt EV, I'm not 
convinced we should be making recommendations about it to begin with. 
Users will be required to remember which websites are EV-enabled and 
which ones aren't, and that's a completely ridiculous assumption to 
make.  Even if (again, best case scenario) they can make this 
distinction, no doubt they'll still interact with websites that are not 
EV-enabled.  These websites will remain targets for attack.

> 
> Finally, I'm extremely concerned about the attitude of "Well, it works 
> in lab studies, so let's mandate it, vendors be damned." I understand 
> the desire not to be seen as being beholden to the desires of browser 
> manufacturers, but on the other hand, I have a very real desire not to 
> be seen as floating around in la-la land, disconnected from reality. If 
> something is going to cause people not to adopt a product, a vendor is 
> not going to implement it, regardless of any mandates from W3C. There is 
> a very real risk of steering ourselves towards irrelevancy. Without 
> getting into too many politics, that's why WHATWG was formed, and 
> provides a good bit of background for the current HTML5 /realpolitik/. I 
> don't want to see us go the way of XForms 2.

Who said if it works in lab studies we'll automatically recommend it?  I 
see the lab studies as a way for weeding things out.  If they perform 
well in lab studies, obviously we'll need to conduct further tests.  Of 
course, this doesn't work if only three of us are doing the lab studies 
on our own time...

serge

> 
> My $0.02 x 3 (== 0.03)
> 
> On Nov 13, 2007 8:51 AM, Dan Schutzer < dan.schutzer@fstc.org 
> <mailto:dan.schutzer@fstc.org>> wrote:
> 
>     agreed
> 
>     -----Original Message-----
>     From: public-wsc-wg-request@w3.org
>     <mailto:public-wsc-wg-request@w3.org>
>     [mailto:public-wsc-wg-request@w3.org
>     <mailto:public-wsc-wg-request@w3.org>] On
>     Behalf Of Serge Egelman
>     Sent: Tuesday, November 13, 2007 11:23 AM
>     To: Hallam-Baker, Phillip
>     Cc: Ian Fette; W3C WSC Public
>     Subject: Re: ACTION-335 logotypes and ISSUE-96 discussion
> 
> 
>     This is irrelevant for our purposes.  If we test them and find that in a
>     perfect world they don't work, then this is moot.  If we test them and
>     find that they're effective, then we make a recommendation, and it's out
>     of our hands.  At that point the application vendors aren't in
>     compliance.
> 
>     serge
> 
>     Hallam-Baker, Phillip wrote:
>      > I have never had the slightest difficulty selling the idea of
>     logotypes
>      > to customers. The problem is purely on the application side. The
>     logos
>      > have no value unless they are displayed.
>      >
>      > So we risk a chicken and egg situation where the application side
>     people
>      > refuse to do anything about implementation until they are assured
>     that
>      > there will be 100% adoption by the site owners which is not going to
>      > happen until there are applications to present the logos.
>      >
>      > Someone has to make the first move, we cannot gate the scope of
>     what we
>      > will consider by requiring an assurance of total adoption by any
>     market
>      > participant.
>      >
>      >
>     ------------------------------------------------------------------------
> 
>      > *From:* public-wsc-wg-request@w3.org
>     <mailto:public-wsc-wg-request@w3.org> on behalf of Ian Fette
>      > *Sent:* Fri 09/11/2007 4:49 PM
>      > *To:* W3C WSC Public
>      > *Subject:* ACTION-335 logotypes and ISSUE-96 discussion
>      >
>      > This action (ACTION-335) was to provide discussion topics for
>     ISSUE-96.
>      > I only really have one point, and I will try to state it more clearly
>      > than at the meeting.
>      >
>      > To me, the effectiveness of any of the logotype proposals (or the EV
>      > proposals, for that matter) depends greatly upon the adoption of
>     these
>      > technologies by sites. We can do really cool flashy things when
>     we get
>      > an EV cert, or an EV-cert with a logo, but right now the only two
>     sites
>      > I can find using an EV cert are PayPal and VeriSign. Therefore, I
>     wonder
>      > how habituated people would become in practice, if they never (or
>      > rarely) saw the EV/logotype interface stuff in use.
>      >
>      > My proposal is that any usability testing of the EV and/or logotype
>      > things in the spec not only reflect how users would behave in a land
>      > where everyone is using EV-certs and life is happy, but rather
>     also test
>      > a more realistic case. That is, look at what the adoption is
>     presently
>      > and/or what we can reasonably expect it to be at time of last
>     call, and
>      > do usability testing in an environment that reflects that
>     adoption rate
>      > - i.e. some percentage of sites using EV certs, some percentage also
>      > using logos, and another percentage still using "normal" SSL
>     certs. My
>      > worry is that we may be thinking "EV certs will solve X,Y, and
>     Z", but
>      > that may only be the case if users are used to seeing them on the
>      > majority of sites, and should that not end up being the case, we
>     need to
>      > look at the usability and benefit in that scenario as well.
>      >
>      > I think this is what the ACTION wanted, i.e. for me to state this
>     point
>      > more explicitly. I am going to therefore assume that my work on this
>      > action is complete, unless I hear otherwise.
>      >
>      > -Ian
> 
>     --
>     /*
>     PhD Candidate
>     Vice President for External Affairs, Graduate Student Assembly
>     Carnegie Mellon University
> 
>     Legislative Concerns Chair
>     National Association of Graduate-Professional Students
>     */
> 
> 
> 

-- 
/*
PhD Candidate
Vice President for External Affairs, Graduate Student Assembly
Carnegie Mellon University

Legislative Concerns Chair
National Association of Graduate-Professional Students
*/
Received on Tuesday, 13 November 2007 23:40:45 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 5 February 2008 03:52:53 GMT