RE: Observations about subsetting the test suite

>Yes, at arm's length, as others have noted. In particular, if a test
lab >has submitted conformance tests for inclusion in the test suite,
they >should be able to use them as extras while awaiting acceptance of
the cases >into the suite. 

I would add:
...but clearly separate test cases that are part of the official W3C
test suite and tests that are added.

Every addition, annotation not part of the official original W3C test
suite, should be ideally done in a separate folder. That does not
prevent the lab's harness to assemble the annotations/additions and
report them together with the main suite. Again good policy would be to
explicitly outline what came from where.


Agree on sub-setting remark.


> -----Original Message-----
> From: David Marston/Cambridge/IBM [mailto:david_marston@us.ibm.com]
> Sent: Thursday, February 27, 2003 11:08 AM
> To: www-qa-wg@w3.org
> Cc: Joseph Reagle
> Subject: Observations about subsetting the test suite
> 
> 
> 
> 
> 
> 
> In [1], Patrick Curran wrote:
> >However, many "tests" distributed by W3C would perhaps more
> >accurately be called "data files", since they are fragments of XML,
> >for example. These are fed to a parser or processor, and the output
is
> >compared to what is expected.
> 
> >Nevertheless, I think this is what we want....
> >In each case, we must consider the answer when the "testsuite"
> >is actually an incomplete set of tests.
> 
> It always is! We must assume three things:
> A. New tests will be contributed in the future
> B. Existing tests will be found to be invalid
> C. Errata will cause new Dimensions of Variability (DoV)
> 
> >Is a licensee permitted to:
> >(1) create a derivative testsuite from a W3C testsuite?
> >(1.1) by adding extra tests to it?
> 
> Yes, at arm's length, as others have noted. In particular, if a test
> lab has submitted conformance tests for inclusion in the test suite,
> they should be able to use them as extras while awaiting acceptance
> of the cases into the suite. This will encourage more submissions.
> 
> >(1.2) by removing some tests from it?
> 
> This is a potential problem area. Unless a spec has no DoV other than
> extensions, the proper operation of the testing software will "remove"
> tests in the sense of excluding them from application against a
product
> that does not implement the features under test. However, we want very
> much to have the issuer of the test suite (W3C or approved authority)
> control such exclusions. In general, the test lab may not exclude
tests
> on their own judgment.
> 
> >...(1.4) by adding extra documents or files to it?
> 
> I see a (natural) language issue here. If the suite does not currently
> include (say) Japanese-language text, and the spec anticipates that
> Japanese norms can be supported, but the spec sets conformance notions
> that would apply to Japanese, the test lab should be able to compose
> Japanese equivalents of data files provided in other languages. There
> need to be guidelines here, though, such as requiring the lab to offer
> the data files to the W3C if they attempt to make any conformance
> claims using them. This derives from the general principle that any
> test materials should be available for other labs to attempt to run
the
> same tests against the same products.
> 
> >...(3) execute only a portion of a W3C test suite (for example,
> >excluding some tests from the run on the grounds that they have been
> >determined to be invalid)?
> 
> My preference is that the test lab can only annotate case-by-case
> *results* like this, and offer summaries under tight rules. Example
> rule: if the lab wishes to summarize "we consider N cases invalid",
> they must point to and freely provide a published statement that
> gives their explanation for each test case. In other words, they must
> run all the tests, then quibble about the tests in their report of
> the outcome.
> 
> Of course, (3) above is different from W3C-approved subsetting of the
> test suite on a per-product basis to match DoV choices implemented in
> each product (1.2 above).
> .................David Marston
> 
> _____
> [1] http://lists.w3.org/Archives/Public/www-qa-wg/2003Feb/0103.html

Received on Thursday, 27 February 2003 14:53:07 UTC