W3C home > Mailing lists > Public > www-qa-wg@w3.org > March 2005

Re: TestFAQ: request for input

From: Dominique HazaŽl-Massieux <dom@w3.org>
Date: Tue, 08 Mar 2005 10:31:12 +0100
To: Patrick Curran <Patrick.Curran@sun.com>
Cc: QAWG <www-qa-wg@w3.org>
Message-Id: <1110274273.11665.197.camel@stratustier>
Le lundi 07 mars 2005 ŗ 12:14 -0800, Patrick Curran a ťcrit :
> I'm looking for examples that will help to make the Test FAQ more
> concrete and realistic. The attached draft contains entries in red
> requesting input. Please mail me with references that help to provide
> examples for any of these requests.

>  For examples of such guidelines, see the CSS Test Authoring
> Guidelines and the Submission Procedure for XSLT/XPath Test Suites.
> Other examples of test-authoring guidelines or submission procedures? 

The SVG Test suite manual has some detailed authoring guidelines:
So does the DOM Test Suite:
and the HTML 4.01 one:
http://www.w3.org/MarkUp/Test/HTML401/current/htmltestdocumentation.html#testsuiteformat (although their usage of the term "test suite" instead of "test case" is confusing)
XKMS 2.0 does too:
and OWL:

> Define a process to manage contributions. Review submissions to ensure
> that they are appropriate and correct. Keep track of who submitted
> what, and of the 'state' that a particular test is in (submitted,
> reviewed, accepted, returned for revision, rejected, etc.) A test-case
> management system [@@ example @@] can help with this task.
> Examples of test review processes or test-case management systems?

The SVG TS manual has test review guidelines that were (supposedly) used
to filter test cases:

The OWL Test Suite documents its creation/approval/modification process:

The XForms Test Suite has an XForms-based test cases management system:
http://www.w3.org/MarkUp/Forms/Test/maintain.html (I haven't tried it

(I know the Voice Browser WG is/has been using a more sophisticated
system, but I can't find any reference to it just now)

> Ask test developers to give priority to the areas of the spec where
> coverage is most needed - don't just leave it up to them to develop
> whatever they want. (This can also help to avoid duplication of
> effort.) Note that this implies the creation and maintenance of some
> kind of 'coverage map' (see the next question for more on this).
> Example of a WG that guides test contributors, telling them where
> tests are most needed?

I haven't found such a type of guidelines in my review of W3C test

> 5. How many tests are enough?
> Whether or not you define coverage goals in advance, it is always
> helpful to provide some kind of coverage report  with your test suite.
> This could be as simple as a mapping of tests to areas of the
> specification, or a more detailed report providing counts and averages
> of the number of tests associated with different areas. Such reports
> can help the users of your test suite understand its strengths and
> weaknesses
> Examples of WGs publishing coverage numbers?

The XForms Test Suite doesn't give coverage number per se, but asserts
that it has covered all the test assertions defined in the Specs:
http://www.w3.org/MarkUp/Forms/Test/ (and
http://www.w3.org/MarkUp/Forms/Test/ImplementationReport.html )

The HTML 4.01 Test Suite has also a list of test assertions, and shows
which assertions have a matching test case, e.g.:

(there are probably more examples available if needed)

> 6. How should tests report their results?
> Some WGs have defined RDF formats for collecting and processing test
> results, and there are a number of XSLT style sheets that can be used
> to format results in an attractive way [@@ provide links to examples
> @@].
> Examples of style-sheets and test results publication (implementation
> reports)?

The RDF and OWL test suites used an innovative way to collect test
results on the Web:
as explained/documented in

The XKMS WG has been using WBS to collect test results, see:

The Voice Browser WG has used an XML format to collect test results:

EARL is another example of test results format (but is mentioned earlier
in the text, so may not need to be repeated).

> 7. Do I really have to worry about all that legal stuff?
> Need links to W3C licenses..

Document license:
Software license:
QA Handbook on this topic:

> 8. How should I package and publish my tests?
> Examples of real test suites (containing docs, harness, etc.)?

I guess I'd rather have you say which of the W3C Test Suites you
consider to be real test suites; they are all linked from the QA Matrix:

Maybe the SVG test suite? It has a documentation, a test harness, is
packaged as a zip file:

> 9. What should the test documentation cover?
> Example of good test suite documentation?

Same here; I expect the SVG, CSS and DOM Test Suites documentations to
be good candidates.

> 10. Should I automate test execution?
> Examples of automated test suites, and/or of tests published with
> metadata allowing others to automate?

Maybe you could simply link to

> 11. Once I publish my tests, I'm done, right?
> Sorry, no. Test suites must evolve over time
> Examples of WGs that have released multiple versions of their test
> suite?

The SVG WG released 3 versions of its test suite for SVG 1.0:
The CSS WG maintains the full lists of its test suites releases:
The OWL, RDF, and SOAP WGs published their test suites as technical
reports, so they are "naturally" versionned (using the
previous/this/latest versions links part of each technical report).

> 12. How should I handle bugs in my test suite?
> Pointer to Bugzilla - example of a WG using it (us?)...

The XML Query WG is using bugzilla to track its test-related issues, so
that's a perfect fit. It is Member-only, but I don't think that's a big
deal as long as you mention it in the text:

> 13. Should test results be published?
> Example of a WG encouraging/supporting publication of test results?

Hmm... I'm not sure we need this; at least, I don't really know what I
should be looking for as an example.

> 14. Should we implement a branding or certification program?
> Example of a WG encouraging/supporting a certification or logo program
> ("this page validates....")?

I guess the HTML WG is the obvious example:
And so is the WCAG WG:

Dominique HazaŽl-Massieux - http://www.w3.org/People/Dom/

Received on Tuesday, 8 March 2005 09:31:15 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:19 GMT