Re: ACTION-331 Work toward worked example of usability testing for conformance

My original thought on this issue wasn't that we would recommend  
usability evaluation for every change a developer makes as a result  
of our other recommendations, but I was thinking more in terms of a  
developer implementing the larger recommendations we were making  
where a concrete implementation isn't specified in our document. But  
after the discussion at the F2F, I guess we might need to address  
both cases.


Reply Clipped from Ian's email on ACTION-330:

> Hence, my main concern is that we are going to require usability  
> testing for conformance, and the way the test is constructed will  
> be the primary factor in whether an implementation appears  
> "usable". As such, I think we would have to lay out very clear  
> guidelines on how the usability testing should be done (basically  
> specifying the experimental design), which seems fraught with peril  
> given how different implementations might be and might become over  
> time, or we would have to take a huge leap of faith. Personally, my  
> preference would be to avoid requiring [in the MUST sense]  
> usability testing for conformance in general, and instead come up  
> with good guidelines for how a usability test SHOULD be conducted  
> to address these issues.


Point taken. I agree that the design of a study can have a strong  
effect on the outcome, and I also agree that it would be a huge task  
to attempt to design a study for each of our recommendations and for  
every possible way it might be implemented.

But I also think we should specify usability requirements for  
anything we recommend where the implementation can be left to  
interpretation. My choice of words so far may have distracted from  
the point I was trying to make -- we should give usability  
requirements, which doesn't necessarily imply we provide a user study  
design for proving the usability requirements are met.


I'm not sure this issue can be fully addressed until:
	1)  We've decided what recommendations we're making.
	2) We have conducted some of our own usability evaluations.

Having the final set of recommendations will be useful because we can  
go through the finished document and identify where we need to  
specify usability requirements. Some started to do this at the F2F,  
so we might talk about either doing this in one shot, or doing it for  
the existing text and incrementally as new text is added.

Waiting until we complete some of our own user studies will be useful  
because each one will be an exercise in specifying usability  
requirements. They're what we'll be testing, so we can't run a study  
without doing that first anyway.

I imagine the usability requirements will be inspired mostly by our  
charter, what we know of the status quo, and prior usability studies.  
I can't remember our group ever talking about usability requirements  
specifically -- maybe we should start by using some of the examples  
people pointed out in the F2F?



-- Maritza

Received on Tuesday, 20 November 2007 17:15:30 UTC