Re: Questions to be asking ourselves if/when we use our draft methodology to evaluate some websites

Hi Peter, All,

I agree with all the points raised.

Initially we had discussed that participants of the Task Force to test 
drive the methodology as part of their usual evaluation work and give 
feedback on how it worked. This way we would get real feedback on how 
the methodology works in practice without disclosing any website.

Also, I believe the plan was to have this test drive period after the 
next public Working Draft scheduled for December 2012 (!). I think we 
have several known issues that probably need to be resolved before we 
start test driving, and I also think that the time at TPAC will be too 
limited to do such comprehensive test driving as is necessary.

Let's discuss this further in the coming weeks.

Regards,
   Shadi


On 21.9.2012 19:58, Peter Korn wrote:
> Hi gang,
>
> Yesterday on the EvalTF call we started discussing the idea of using the time at
> TPAC to put our methodology through a trial run by applying it to some websites.
>
> I really like this idea.  It was incredibly helpful when a number of industry
> members did this during the TEITAC process, applying drafts of the TEITAC
> report/regulations we were considering to a number of real-life products.
>
> I think, however, if we do this we need to be very careful and thoughtful in how
> we approach it.
>
> First, I think we need to make very clear to everyone involved & everyone we
> publish the results to that we are testing OUR draft evaluation methodology, and
> NOT the websites we are looking at.  The point is to figure out how well our
> methodology works, and NOT to criticize/critique websites.
>
> Second, I think we need to have a set of questions in mind about the document -
> things in the draft that we are particularly looking at and evaluating.
>
> Third, I think we need to have a sufficiently broad range of sites, to better
> evaluate our work in the "real world" diversity out there.
>
> To that third end, I think the kinds of sites we should be testing include:
>
>   1. A relatively homogenous site (e.g. an on-line newspaper where most pages
>      look largely the same)
>   2. A very heterogeneous site (with a mixture of page styles, content types, etc.)
>   3. A site where much of the content comes from elsewhere (3rd party content
>      where the site owner has little/no control over the 3rd party content)
>   4. Web applications, including specifically single-page web apps that have lots
>      of different "screens" shown
>
>
> Returning to the second item, I think the key questions we should be asking
> about our document include:
>
>   1. Does the methodology cover the situations arising from all of these websites?
>   2. Is there significant functionality / parts of these sites we aren't reaching
>      (statistical methodology)?
>   3. Do all of the required parts make sense as required, the optional parts as
>      optional? (and do all of the optional parts for all sites as part of
>      evaluating them)
>   4. Does the notion of templates make sense?
>   5. Are the report(s) that we generate useful?  If so, useful to all of the
>      potential "customers" of the methodology, or just some?  What changes to the
>      reports might we make to make them more useful?
>   6. Do our sampling methods (as they may have developed by TPAC) cover enough of
>      the website to be reliable?
>
>
> Going back to the first point - I think we should seek volunteer sites if
> possible, again making clear the purpose of our work.  I think we should NOT
> publish the problems found - though that may be a challenge when it comes to
> wanting to review the report results.  If so, we should keep the sites anonymous
> in anything we publish, unless the site owners give permission for that.  We do
> not want to even give the impression that the W3C is in the business of
> evaluating others' websites.  Which also reminds me - perhaps we might run this
> on the W3C site (again after checking in with W3C management); that might be
> something publishable...
>
>
> Regards,
>
> Peter
>
> --
> Oracle <http://www.oracle.com>
> Peter Korn | Accessibility Principal
> Phone: +1 650 5069522 <tel:+1%20650%205069522>
> 500 Oracle Parkway | Redwood City, CA 94065
> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
> developing practices and products that help protect the environment
>

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Activity Lead, W3C/WAI International Program Office
Evaluation and Repair Tools Working Group (ERT WG)
Research and Development Working Group (RDWG)

Received on Friday, 21 September 2012 18:53:26 UTC