Re: Discussion of test case submission from KDE

This is a personal reply, not expressing the opinion of any group.
However, my opinion is open-sourced and others are welcome to share
this opinion. :)

Regarding this point:
>There is a risk of that some tests duplicate tests in XQTS. It is a
>very large job(and error prone in several senses) to check this
>manually. Perhaps one could write a tool which opens all queries,
>removes the initial comment and then compares the tests for finding
>duplicates. Creating such a tool would hopefully be useful with other
>submissions as well.

Duplicate tests are not really a risk. By ignoring the possibility of
duplicates, the task force reduces the risk that some edge case will
go untested. There is no requirement to count individual cases as
single points, equal in weight or otherwise, in some sort of scoring
system. If a processor-under-test fails some test cases, you care about
which ones but not how many. If a processor-under-test passes all test
cases, you still can't declare it "100% conformant" because other tests
that didn't get written might have uncovered non-conformant results.

Duplicate test cases might waste someone's time, but they don't cause
a conformance-assessment problem. The time and effort of test case
writers is a scarce resource; right now it's better to have them
thinking about new cases rather than trying to identify duplicates.
.................David Marston

Received on Monday, 17 April 2006 15:39:37 UTC