Re: TestFAQ: request for input

Le lundi 07 mars 2005 à 12:14 -0800, Patrick Curran a écrit :
> I'm looking for examples that will help to make the Test FAQ more
> concrete and realistic. The attached draft contains entries in red
> requesting input. Please mail me with references that help to provide
> examples for any of these requests.

>  For examples of such guidelines, see the CSS Test Authoring
> Guidelines and the Submission Procedure for XSLT/XPath Test Suites.
> 
> Other examples of test-authoring guidelines or submission procedures? 

The SVG Test suite manual has some detailed authoring guidelines:
http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#HowtoDoIt
So does the DOM Test Suite:
http://www.w3.org/DOM/Test/Documents/DOMTSML.html
and the HTML 4.01 one:
http://www.w3.org/MarkUp/Test/HTML401/current/htmltestdocumentation.html#testsuiteformat (although their usage of the term "test suite" instead of "test case" is confusing)
XKMS 2.0 does too:
http://www.w3.org/2001/XKMS/Drafts/test-suite/#Contributi
and OWL:
http://www.w3.org/TR/owl-test/#style

> Define a process to manage contributions. Review submissions to ensure
> that they are appropriate and correct. Keep track of who submitted
> what, and of the 'state' that a particular test is in (submitted,
> reviewed, accepted, returned for revision, rejected, etc.) A test-case
> management system [@@ example @@] can help with this task.
> 
> Examples of test review processes or test-case management systems?

The SVG TS manual has test review guidelines that were (supposedly) used
to filter test cases:
http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#TestReviewGuidelines

The OWL Test Suite documents its creation/approval/modification process:
http://www.w3.org/TR/owl-test/#testProcess

The XForms Test Suite has an XForms-based test cases management system:
http://www.w3.org/MarkUp/Forms/Test/maintain.html (I haven't tried it
though)

(I know the Voice Browser WG is/has been using a more sophisticated
system, but I can't find any reference to it just now)

> Ask test developers to give priority to the areas of the spec where
> coverage is most needed - don't just leave it up to them to develop
> whatever they want. (This can also help to avoid duplication of
> effort.) Note that this implies the creation and maintenance of some
> kind of 'coverage map' (see the next question for more on this).
> 
> Example of a WG that guides test contributors, telling them where
> tests are most needed?

I haven't found such a type of guidelines in my review of W3C test
suites...

> 5. How many tests are enough?
> Whether or not you define coverage goals in advance, it is always
> helpful to provide some kind of coverage report  with your test suite.
> This could be as simple as a mapping of tests to areas of the
> specification, or a more detailed report providing counts and averages
> of the number of tests associated with different areas. Such reports
> can help the users of your test suite understand its strengths and
> weaknesses
> 
> Examples of WGs publishing coverage numbers?

The XForms Test Suite doesn't give coverage number per se, but asserts
that it has covered all the test assertions defined in the Specs:
http://www.w3.org/MarkUp/Forms/Test/ (and
http://www.w3.org/MarkUp/Forms/Test/ImplementationReport.html )

The HTML 4.01 Test Suite has also a list of test assertions, and shows
which assertions have a matching test case, e.g.:
http://www.w3.org/MarkUp/Test/HTML401/current/assertions/assertions_section05.html

(there are probably more examples available if needed)


> 6. How should tests report their results?
> Some WGs have defined RDF formats for collecting and processing test
> results, and there are a number of XSLT style sheets that can be used
> to format results in an attractive way [@@ provide links to examples
> @@].
> 
> Examples of style-sheets and test results publication (implementation
> reports)?

The RDF and OWL test suites used an innovative way to collect test
results on the Web:
http://www.w3.org/2003/08/owl-systems/test-results-out#about 
as explained/documented in
http://lists.w3.org/Archives/Public/www-qa/2003Sep/0038.html
http://lists.w3.org/Archives/Public/www-qa-wg/2004Jan/0011.html

The XKMS WG has been using WBS to collect test results, see:
http://www.w3.org/2002/09/wbs/1/XKMS-WG-CR-TEST-SUITE/

The Voice Browser WG has used an XML format to collect test results:
http://www.w3.org/Voice/2003/ir/voicexml20-ir.html#participate

EARL is another example of test results format (but is mentioned earlier
in the text, so may not need to be repeated).

> 7. Do I really have to worry about all that legal stuff?
> Need links to W3C licenses..

Document license:
http://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231
Software license:
http://www.w3.org/Consortium/Legal/copyright-software
QA Handbook on this topic:
http://www.w3.org/TR/qa-handbook/#IANAL

> 8. How should I package and publish my tests?
> Examples of real test suites (containing docs, harness, etc.)?

I guess I'd rather have you say which of the W3C Test Suites you
consider to be real test suites; they are all linked from the QA Matrix:
http://www.w3.org/QA/TheMatrix

Maybe the SVG test suite? It has a documentation, a test harness, is
packaged as a zip file:
http://www.w3.org/Graphics/SVG/Test/

> 9. What should the test documentation cover?
> Example of good test suite documentation?

Same here; I expect the SVG, CSS and DOM Test Suites documentations to
be good candidates.

> 10. Should I automate test execution?
> Examples of automated test suites, and/or of tests published with
> metadata allowing others to automate?

Maybe you could simply link to
http://esw.w3.org/topic/TestCaseMetadata

> 11. Once I publish my tests, I'm done, right?
> Sorry, no. Test suites must evolve over time
> Examples of WGs that have released multiple versions of their test
> suite?

The SVG WG released 3 versions of its test suite for SVG 1.0:
http://www.w3.org/Graphics/SVG/Test/Overview-10.html
The CSS WG maintains the full lists of its test suites releases:
http://www.w3.org/Style/CSS/Test/
The OWL, RDF, and SOAP WGs published their test suites as technical
reports, so they are "naturally" versionned (using the
previous/this/latest versions links part of each technical report).

> 12. How should I handle bugs in my test suite?
> Pointer to Bugzilla - example of a WG using it (us?)...

The XML Query WG is using bugzilla to track its test-related issues, so
that's a perfect fit. It is Member-only, but I don't think that's a big
deal as long as you mention it in the text:
http://www.w3.org/Member/bugzilla/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__all__&product=XML+Query+Working+Group&content=

> 13. Should test results be published?
> Example of a WG encouraging/supporting publication of test results?

Hmm... I'm not sure we need this; at least, I don't really know what I
should be looking for as an example.

> 14. Should we implement a branding or certification program?
> Example of a WG encouraging/supporting a certification or logo program
> ("this page validates....")?

I guess the HTML WG is the obvious example:
http://www.w3.org/MarkUp/#validation
And so is the WCAG WG:
http://www.w3.org/WAI/WCAG1-Conformance.html

Dom
-- 
Dominique Hazaël-Massieux - http://www.w3.org/People/Dom/
W3C/ERCIM
mailto:dom@w3.org