Re: ISSUE-117 (serge): Eliminating Faulty Recommendations [All]

Late apologies for missing the call on this ... I've been running a  
user study for another project the past two weeks and my schedule's  
been dictated by when people can show up for sessions (I know, lame  
excuse, but it's tough to recruit participants). The work is related  
to our group and I plan to share the results as soon as I hear back  
from Steve about when and how much I can say.


> I bring up the ISSUE-112 here as well because I do not want anyone  
> wasting time doing any user studies if the results will be  
> discounted by the group during discussions. That would be unfair  
> and disrespectful. My advice is that for any user study done  
> specifically for this group, we specify ahead of time what we're  
> doing, what sort of outcomes might be expected, and how that should  
> influence our recommendation. We then discuss _that_ and get group  
> consensus on the trajectory and impact of a user study before  
> actually running it. If we can run this process with something  
> modest soon, it can helpfully provide input to anything more  
> resource intensive we do later, and see if that's a reasonable way  
> to integrate them into our work.


Reading over the minutes about this issue again, I noticed Serge has  
an action item to begin going through the recommendations, I guess to  
do a more thorough job of what we started in July, maybe to produce  
tangible action items out of the process?


In agreement with what Mez said above and in response to Tim's  
followup on my comments about Browser Lockdown <http://www.w3.org/ 
2006/WSC/track/actions/305 >, I've given myself an action item  
<http://www.w3.org/2006/WSC/track/actions/345> to come up with a user  
study design that will begin to evaluate the unknowns for the  
recommendation.

I'll write up a study that will mostly consist of a questionnaire  
with the goal of collecting data on whether or not users can be  
relied on the choose the right mode given a certain task. It will  
also be designed to gather feedback on how users might react to being  
blocked from accessing a site.

I'll address the points Mez has made above the best I can and post it  
to the list in hopes of generating useful discussion on refining the  
study.

Mez -- is this what you had in mind for moving forward?


-- Maritza