Re: Discussion of test case submission from KDE

On Monday 17 April 2006 15:37, david_marston@us.ibm.com wrote:

Hi David,

> This is a personal reply, not expressing the opinion of any group.
> However, my opinion is open-sourced and others are welcome to share
> this opinion. :)
>
> Regarding this point:
> >There is a risk of that some tests duplicate tests in XQTS. It is a
> >very large job(and error prone in several senses) to check this
> >manually. Perhaps one could write a tool which opens all queries,
> >removes the initial comment and then compares the tests for finding
> >duplicates. Creating such a tool would hopefully be useful with other
> >submissions as well.
>
> Duplicate tests are not really a risk. By ignoring the possibility of
> duplicates, the task force reduces the risk that some edge case will
> go untested. There is no requirement to count individual cases as
> single points, equal in weight or otherwise, in some sort of scoring
> system. If a processor-under-test fails some test cases, you care about
> which ones but not how many. If a processor-under-test passes all test
> cases, you still can't declare it "100% conformant" because other tests
> that didn't get written might have uncovered non-conformant results.
>
> Duplicate test cases might waste someone's time, but they don't cause
> a conformance-assessment problem. The time and effort of test case
> writers is a scarce resource; right now it's better to have them
> thinking about new cases rather than trying to identify duplicates.

Sensible words(but hey, I'm just jumping on the open source wagon here).

I also think not trying to sort out duplicates is the way to go, for several 
reasons. My only concern is if it was manually attempted to sort out 
duplicates, because it could potentially introduce regressions. However, I am 
positive towards an automated comparison on the byte level, since I don't see 
how that could go wrong.

And yes, a little modesty cannot hurt. I don't represent or speak for KDE, I 
am only one of the developers.


Regards,

		Frans

Received on Thursday, 20 April 2006 14:10:41 UTC