RE: ACTION-335 logotypes and ISSUE-96 discussion

What you are saying might be sumarized as

1) Lab tests may be sufficient to reject a technology. I agree here, but mostly on the acceptabilty side, if users refuse to use a system in the lab you can be pretty sure they won't use it in production.

Noticing the indicata is likely to change over time, whether it increases or decreases is going to require large n studies. I expect it will go both ways.


2) We need a deployment strategy.

Absolutely, preaching to the choir here. 

But your deployent strategy is faulty. I don't need to prove that a proposal provides ANY security advantage whatsoever before getting to a threshold level where it is effective. All I need to do is to prove that it is likely to get to that threshold level.

The adoption incentive for early adopters does not have to be the adoption incentive for late adopters. And in point of fact the people we need to deploy are motivated not by security but by concern for security.

The incentive for merchants to buy an EV certificate is that the number of shopping carts abandonded reduces. In the case of Overstock.com they report a reduction of 8.6%. That is a huge incentive for deployment and more than justifies the modest cost of a quality certificate.




 

> -----Original Message-----
> From: Serge Egelman [mailto:egelman@cs.cmu.edu] 
> Sent: Tuesday, November 13, 2007 6:40 PM
> To: Ian Fette
> Cc: Dan Schutzer; Hallam-Baker, Phillip; W3C WSC Public
> Subject: Re: ACTION-335 logotypes and ISSUE-96 discussion
> 
> 
> 
> Ian Fette wrote:
> > Disagree.
> > 
> > While I don't agree with all of Phil's points, there was one that I 
> > definitely agree with that Serge seems to have glossed over. That 
> > would be the point about whether you're testing the user over a 
> > half-hour in a lab, or a longer (30+ day) field-study in 
> their natural environment.
> > Phil's point was that anything new and disruptive is likely 
> to show a 
> > strong effect in the short-term, but over the long-term the 
> effect may 
> > be drastically different (including causing people to stop 
> using the 
> > product). This is a very good point, and I think that if 
> possible we 
> > should aim to do a longer field-study as opposed to a 30m 
> in-lab study.
> > 
> 
> This is something that can be examined later.  There are pros 
> and cons 
> for both types of study designs.  But then again, if your 
> only argument 
> is that the effect is stronger in the lab study, I'm not sure 
> how that's 
> a problem.  If the users can be easily fooled in the best 
> case scenario 
> (or they simply don't notice/trust the indicators), then it's likely 
> this effect will only be stronger in the wild.  Of course playing arm 
> chair quarterback here isn't going to do much good.  We need 
> an actual 
> study design before we can critique it.  To do that we need to first 
> figure out which questions we want the study to answer.
> 
> > As for "testing them in a perfect world" - I have no idea 
> why this is a 
> > good experiment to run, because we know that we will never 
> be operating 
> > in a perfect world. I'm not saying we should test in a 
> world with zero 
> > adoption, but rather I'm saying that we should try to 
> figure out (guess) 
> > what /reasonable/ adoption is, and test in that world. We 
> already know 
> > that there are some sites that are not adopting EV because 
> of the cost 
> > model. I'm sure someone is more knowledgeable about the 
> specifics than 
> > I, but my understanding is that, for instance, Google could 
> not buy one 
> > EV certificate for google.com <http://google.com> and use 
> it across all 
> > of our numerous servers, rather we would have to pay some increased 
> > (large) fee based on number of servers. (Also, does EV 
> support wildcard 
> > certs?). Given that, you can come up with a list of 
> companies for which 
> > EV would be very expensive and likely not adopted (eBay?), 
> and test with 
> > the assumption that those sites won't adopt. What does that 
> do to the 
> > overall model?
> 
> This is kind of a no-brainer.  The system will not work when there's 
> very low adoption.  Users will go to a website, see no EV certificate 
> indicator, and just assume that the site never had one.  
> That's because 
> this is the norm.  Thus, if we assume everyone adopts EV, we can test 
> the rate of success in the best case.  If the system is not 
> successful 
> in the best case scenario, then we know it won't be successful in 
> conditions worse than that.  Likewise, if we test the system under 
> "reasonable" circumstances (forgetting for a second that 
> "reasonable" is 
> completely subjective and we're unlikely to agree on that definition) 
> and it's a failure or success, the validity of the study comes into 
> question because someone will invariably ask, "well, what 
> happens when 
> everyone adopts EV?"
> 
> If we're going to agree that a minority of sites will adopt 
> EV, I'm not 
> convinced we should be making recommendations about it to begin with. 
> Users will be required to remember which websites are EV-enabled and 
> which ones aren't, and that's a completely ridiculous assumption to 
> make.  Even if (again, best case scenario) they can make this 
> distinction, no doubt they'll still interact with websites 
> that are not 
> EV-enabled.  These websites will remain targets for attack.
> 
> > 
> > Finally, I'm extremely concerned about the attitude of 
> "Well, it works 
> > in lab studies, so let's mandate it, vendors be damned." I 
> understand 
> > the desire not to be seen as being beholden to the desires 
> of browser 
> > manufacturers, but on the other hand, I have a very real 
> desire not to 
> > be seen as floating around in la-la land, disconnected from 
> reality. If 
> > something is going to cause people not to adopt a product, 
> a vendor is 
> > not going to implement it, regardless of any mandates from 
> W3C. There is 
> > a very real risk of steering ourselves towards irrelevancy. Without 
> > getting into too many politics, that's why WHATWG was formed, and 
> > provides a good bit of background for the current HTML5 
> /realpolitik/. I 
> > don't want to see us go the way of XForms 2.
> 
> Who said if it works in lab studies we'll automatically 
> recommend it?  I 
> see the lab studies as a way for weeding things out.  If they perform 
> well in lab studies, obviously we'll need to conduct further 
> tests.  Of 
> course, this doesn't work if only three of us are doing the 
> lab studies 
> on our own time...
> 
> serge
> 
> > 
> > My $0.02 x 3 (== £0.03)
> > 
> > On Nov 13, 2007 8:51 AM, Dan Schutzer < dan.schutzer@fstc.org 
> > <mailto:dan.schutzer@fstc.org>> wrote:
> > 
> >     agreed
> > 
> >     -----Original Message-----
> >     From: public-wsc-wg-request@w3.org
> >     <mailto:public-wsc-wg-request@w3.org>
> >     [mailto:public-wsc-wg-request@w3.org
> >     <mailto:public-wsc-wg-request@w3.org>] On
> >     Behalf Of Serge Egelman
> >     Sent: Tuesday, November 13, 2007 11:23 AM
> >     To: Hallam-Baker, Phillip
> >     Cc: Ian Fette; W3C WSC Public
> >     Subject: Re: ACTION-335 logotypes and ISSUE-96 discussion
> > 
> > 
> >     This is irrelevant for our purposes.  If we test them 
> and find that in a
> >     perfect world they don't work, then this is moot.  If 
> we test them and
> >     find that they're effective, then we make a 
> recommendation, and it's out
> >     of our hands.  At that point the application vendors aren't in
> >     compliance.
> > 
> >     serge
> > 
> >     Hallam-Baker, Phillip wrote:
> >      > I have never had the slightest difficulty selling the idea of
> >     logotypes
> >      > to customers. The problem is purely on the 
> application side. The
> >     logos
> >      > have no value unless they are displayed.
> >      >
> >      > So we risk a chicken and egg situation where the 
> application side
> >     people
> >      > refuse to do anything about implementation until 
> they are assured
> >     that
> >      > there will be 100% adoption by the site owners which 
> is not going to
> >      > happen until there are applications to present the logos.
> >      >
> >      > Someone has to make the first move, we cannot gate 
> the scope of
> >     what we
> >      > will consider by requiring an assurance of total 
> adoption by any
> >     market
> >      > participant.
> >      >
> >      >
> >     
> --------------------------------------------------------------
> ----------
> > 
> >      > *From:* public-wsc-wg-request@w3.org
> >     <mailto:public-wsc-wg-request@w3.org> on behalf of Ian Fette
> >      > *Sent:* Fri 09/11/2007 4:49 PM
> >      > *To:* W3C WSC Public
> >      > *Subject:* ACTION-335 logotypes and ISSUE-96 discussion
> >      >
> >      > This action (ACTION-335) was to provide discussion topics for
> >     ISSUE-96.
> >      > I only really have one point, and I will try to 
> state it more clearly
> >      > than at the meeting.
> >      >
> >      > To me, the effectiveness of any of the logotype 
> proposals (or the EV
> >      > proposals, for that matter) depends greatly upon the 
> adoption of
> >     these
> >      > technologies by sites. We can do really cool flashy 
> things when
> >     we get
> >      > an EV cert, or an EV-cert with a logo, but right now 
> the only two
> >     sites
> >      > I can find using an EV cert are PayPal and VeriSign. 
> Therefore, I
> >     wonder
> >      > how habituated people would become in practice, if 
> they never (or
> >      > rarely) saw the EV/logotype interface stuff in use.
> >      >
> >      > My proposal is that any usability testing of the EV 
> and/or logotype
> >      > things in the spec not only reflect how users would 
> behave in a land
> >      > where everyone is using EV-certs and life is happy, 
> but rather
> >     also test
> >      > a more realistic case. That is, look at what the adoption is
> >     presently
> >      > and/or what we can reasonably expect it to be at time of last
> >     call, and
> >      > do usability testing in an environment that reflects that
> >     adoption rate
> >      > - i.e. some percentage of sites using EV certs, some 
> percentage also
> >      > using logos, and another percentage still using "normal" SSL
> >     certs. My
> >      > worry is that we may be thinking "EV certs will 
> solve X,Y, and
> >     Z", but
> >      > that may only be the case if users are used to 
> seeing them on the
> >      > majority of sites, and should that not end up being 
> the case, we
> >     need to
> >      > look at the usability and benefit in that scenario as well.
> >      >
> >      > I think this is what the ACTION wanted, i.e. for me 
> to state this
> >     point
> >      > more explicitly. I am going to therefore assume that 
> my work on this
> >      > action is complete, unless I hear otherwise.
> >      >
> >      > -Ian
> > 
> >     --
> >     /*
> >     PhD Candidate
> >     Vice President for External Affairs, Graduate Student Assembly
> >     Carnegie Mellon University
> > 
> >     Legislative Concerns Chair
> >     National Association of Graduate-Professional Students
> >     */
> > 
> > 
> > 
> 
> -- 
> /*
> PhD Candidate
> Vice President for External Affairs, Graduate Student Assembly
> Carnegie Mellon University
> 
> Legislative Concerns Chair
> National Association of Graduate-Professional Students
> */
>

Received on Wednesday, 14 November 2007 14:43:47 UTC