Observations about subsetting the test suite

In [1], Patrick Curran wrote:
>However, many "tests" distributed by W3C would perhaps more
>accurately be called "data files", since they are fragments of XML,
>for example. These are fed to a parser or processor, and the output is
>compared to what is expected.

>Nevertheless, I think this is what we want....
>In each case, we must consider the answer when the "testsuite"
>is actually an incomplete set of tests.

It always is! We must assume three things:
A. New tests will be contributed in the future
B. Existing tests will be found to be invalid
C. Errata will cause new Dimensions of Variability (DoV)

>Is a licensee permitted to:
>(1) create a derivative testsuite from a W3C testsuite?
>(1.1) by adding extra tests to it?

Yes, at arm's length, as others have noted. In particular, if a test
lab has submitted conformance tests for inclusion in the test suite,
they should be able to use them as extras while awaiting acceptance
of the cases into the suite. This will encourage more submissions.

>(1.2) by removing some tests from it?

This is a potential problem area. Unless a spec has no DoV other than
extensions, the proper operation of the testing software will "remove"
tests in the sense of excluding them from application against a product
that does not implement the features under test. However, we want very
much to have the issuer of the test suite (W3C or approved authority)
control such exclusions. In general, the test lab may not exclude tests
on their own judgment.

>...(1.4) by adding extra documents or files to it?

I see a (natural) language issue here. If the suite does not currently
include (say) Japanese-language text, and the spec anticipates that
Japanese norms can be supported, but the spec sets conformance notions
that would apply to Japanese, the test lab should be able to compose
Japanese equivalents of data files provided in other languages. There
need to be guidelines here, though, such as requiring the lab to offer
the data files to the W3C if they attempt to make any conformance
claims using them. This derives from the general principle that any
test materials should be available for other labs to attempt to run the
same tests against the same products.

>...(3) execute only a portion of a W3C test suite (for example,
>excluding some tests from the run on the grounds that they have been
>determined to be invalid)?

My preference is that the test lab can only annotate case-by-case
*results* like this, and offer summaries under tight rules. Example
rule: if the lab wishes to summarize "we consider N cases invalid",
they must point to and freely provide a published statement that
gives their explanation for each test case. In other words, they must
run all the tests, then quibble about the tests in their report of
the outcome.

Of course, (3) above is different from W3C-approved subsetting of the
test suite on a per-product basis to match DoV choices implemented in
each product (1.2 above).
.................David Marston

_____
[1] http://lists.w3.org/Archives/Public/www-qa-wg/2003Feb/0103.html

Received on Thursday, 27 February 2003 14:09:05 UTC