Re: ISSUE-117 (serge): Eliminating Faulty Recommendations [All]

I agree completely.  With this particular recommendation our goal should 
be to improve how things are done, not recommend the status quo.

Also, one issue I wanted to add is the role of the shared bookmarks. 
I'm not sure how many people have bothered to read them, but there's a 
lot in there which arguably has already examined some of the proposals. 
    I think that each proposal author should go through there and 
explain why the prior work either supports their proposals, or that 
their proposals are original and have never been examined (i.e. the 
underlying techniques).

The main point I wanted to make with this issue is that if there's prior 
literature showing certain techniques to be defunct, we shouldn't waste 
our time testing, much less recommending them.

serge

michael.mccormick@wellsfargo.com wrote:
> OK, I'm cc-ing the group list.  Thanks, Mike
> 
> ------------------------------------------------------------------------
> *From:* Mary Ellen Zurko [mailto:Mary_Ellen_Zurko@notesdev.ibm.com]
> *Sent:* Monday, November 12, 2007 7:01 AM
> *To:* McCormick, Mike
> *Subject:* RE: ISSUE-117 (serge): Eliminating Faulty Recommendations [All]
> 
> 
> You should start that discussion on the group's list.
> 
>           Mez
> 
> 
> From: 	<michael.mccormick@wellsfargo.com>
> To: 	<Mary_Ellen_Zurko@notesdev.ibm.com>
> Date: 	11/09/2007 11:33 PM
> Subject: 	RE: ISSUE-117 (serge): Eliminating Faulty Recommendations [All]
> 
> 
> ------------------------------------------------------------------------
> 
> Criteria 2, at least as phrased below, concerns me.  I don't feel WSC 
> should be constrained from making a recommendation just because a 
> particular community may resist adopting it.  Our guidance on favicons 
> is a case in point.  I'm skeptical browsers will adopt that 
> recommendation any time soon but it's still the right thing to do.  If 
> browser manufacturers could always be counted on to do the right things 
> for security on their own, then initiatives like WSC would be less 
> necessary.  Criteria 2 could also reinforce a perception among some 
> skeptics that W3C is beholden to certain web technology vendors and 
> gives their needs priority over those of other industries or the broader 
> user community.  Just my $.02.  Mike
> 
> ------------------------------------------------------------------------
> *From:* public-wsc-wg-request@w3.org 
> [mailto:public-wsc-wg-request@w3.org] *On Behalf Of *Mary Ellen Zurko*
> Sent:* Friday, November 09, 2007 2:56 PM*
> To:* Web Security Context Working Group WG*
> Subject:* Re: ISSUE-117 (serge): Eliminating Faulty Recommendations [All]
> 
> 
> Our discussions on baseline success criteria and ISSUE-112 at the f2f 
> provided the input I needed to respond to this. _
> __http://www.w3.org/2006/WSC/wiki/SuccessBaseline_
> [also see the minutes from November 6 on the topic of  ISSUE-112, 
> currently members only]
> 
> I would argue to eliminate any recommendation that we believe we could 
> not get buy in for (and that we did not believe in the future of uptake 
> of) from the appropriate community (browsers, web app developers, web 
> site administrators, users) (see criteria 2).
> 
> I would also argue to eliminate any recommendation that neither captured 
> current best practice (criteria 3) nor had WG consensus that it would be 
> demonstrably better at aiding trust decisions than the state before the 
> WG started (criteria 4).
> 
> The last line of this issue seems to ask about the place of prior user 
> studies and literature in this process. I see them feeding into criteria 
> 4. For any of our recommendations, anyone can challenge whether or not 
> they help in aiding trust decisions. Prior user studies and literature 
> may be the reason why (or part of the reason why). We discuss it, 
> including any other information or data on the topic, then see what 
> group consensus is. Other sorts of data may be brought to bear on the 
> topic; see _
> __http://www.w3.org/TR/wsc-usecases/#process_
> 
> I bring up the ISSUE-112 here as well because I do not want anyone 
> wasting time doing any user studies if the results will be discounted by 
> the group during discussions. That would be unfair and disrespectful. My 
> advice is that for any user study done specifically for this group, we 
> specify ahead of time what we're doing, what sort of outcomes might be 
> expected, and how that should influence our recommendation. We then 
> discuss _that_ and get group consensus on the trajectory and impact of a 
> user study before actually running it. If we can run this process with 
> something modest soon, it can helpfully provide input to anything more 
> resource intensive we do later, and see if that's a reasonable way to 
> integrate them into our work.
> 
> As a side note, since I consider myself an actual expert (for some value 
> of expert) on the topic of usable security, I'm likely to want to read 
> the data on prior user studies and literature that people cite. I've 
> tried to stay on top of our bookmarks, and will continue to try to stay 
> on top of citations used in discussion. I find it deeply irksome when 
> there's a reference that I can't get to (the ACM Portal being the 
> canonical example for me). That doesn't mean that citations there won't 
> have impact, just the way deployment or product experience that is based 
> on data not directly available to all of us have impact. It means it 
> will be subject to the same sort of engagement by wg members, to try to 
> understand and reason about it in the WG context.
> 
> Any other takers on this issue before I put it on a meeting agenda? 
> Additional ideas, expectations, suggestions, assumptions, presumptions?
> 
> 
> From: 	Web Security Context Working Group Issue Tracker 
> <sysbot+tracker@w3.org>
> To: 	public-wsc-wg@w3.org
> Date: 	10/08/2007 12:56 PM
> Subject: 	ISSUE-117 (serge): Eliminating Faulty Recommendations [All]
> 
> 
> 
> ------------------------------------------------------------------------
> 
> 
> 
> 
> 
> ISSUE-117 (serge): Eliminating Faulty Recommendations [All]
> _
> __http://www.w3.org/2006/WSC/track/issues/_
> 
> Raised by: Serge Egelman
> On product: All
> 
> At what point can we say that a recommendation is unlikely to work and 
> should be removed from consideration?
> 
> For some of these we obviously need user studies to see how effective 
> the techniques are.  However, if prior user studies and literature have 
> already tested similar concepts, it would be a waste of our time and 
> resources to test them again.
> 
> 
> 
> 
> 
> 
> 
> 

-- 
/*
PhD Candidate
Vice President for External Affairs, Graduate Student Assembly
Carnegie Mellon University

Legislative Concerns Chair
National Association of Graduate-Professional Students
*/

Received on Monday, 12 November 2007 21:01:28 UTC