A testing tool - MUTAT coming alive

[Bcc cross-posted. Follow up to www-qa@w3.org please]

Hi folks

the MUTAT test harness [1] was originally developed by Nadia Heninger in 2002
as a general purpose test harness. Briefly, it takes a simple description of
a set of tests, such as SpecGL [2] and uses it to generate an interface for a
user to record results [3]. These results are collected, and offered as EARL
[4]. The tool is able to store these results as Annotea [5] annotations, so
they can be queried and searched later, or modified.

I have started working on updating the tool. My primary goals are to document
the code so someone with only minimal knowledge of PERL, RDF, and all the
rest can work on the code or just use the tool, to update it so that it uses
the latest version of EARL, to provide a human-friendly output option (an
HTML document), and if I get time to make it accept results from other
EARL-producing tools (examples include Hera [6] and the W3C Validator [7])
and incorporate those to streamline the evaluation process.

I also plan to develop a sample test set for the Authoring Tool Accessibility
Guidelines (I expect to work on that next week), as a practical use case.

There is some stuff to look at and try out. I am in the process of working on
the EARL it produces, so that could be messy for the next little while, but
the basis already works something like what I have in mind.

All comments are welcome. And if anyone is an expert in PERL and has time to
look through the code, I am having real trouble tracking down a bug which I
will describe in a following email.

[1] http://www.w3.org/QA/Tools/MUTAT is the developer page. There is a demo
      at http://www.w3.org/QA/Tools/bin/start
[2] http://www.w3.org/TR/qaframe-specgl - the test file describing it is at
      http://www.w3.org/QA/Tools/MUTAT/specgl.n3
[3] Follow the demo: http://www.w3.org/QA/Tools/MUTAT/bin/start
[4] http://www.w3.org/TR/EARL10 is the spec. There is a general introduction
      at http://www.w3.org/2001/sw/Europe/talks/031202-earl and a
      presentation of how to write EARL at
      http://www.w3.org/2001/sw/Europe/talks/200311-earl/all
[5] http://www.w3.org/2001/Annotea
[6] http://www.sidar.org/hera - an accessibility testing tool I have had a
      small hand in designing and use a lot. It inherited lots of ideas from
      MUTAT.
[7] http://validator.w3.org/docs/users.html#Interpret is where the earl
      option is described.

cheers

Chaals

Charles McCathieNevile             http://www.w3.org/People/Charles
tel: +61 409 134 136                 fax(france): +33 4 92 38 78 22
Post:    21 Mitchell street, FOOTSCRAY Vic 3011, Australia    or
 W3C, 2004 Route des Lucioles, 06902 Sophia Antipolis Cedex, France

Received on Thursday, 13 January 2005 17:28:10 UTC