Re: test suite distinctions [was: Re: Feedback on "The Matrix"]

On Thu, 28 Feb 2002, Rob Lanphier wrote:

> Let me quote from the charter:
> "The main objective of the Working Group is to foster the development of
> usable and useful test suites endorsed by the W3C, which share a common
> look and feel"
> 
> There's a lot of qualitative statements there.  How is the W3C
> supposed to promote this if it can't actually cull out the good
> from the bad?

You promote usable and useful test suites. You also promote test
suites which share a common look and feel. There is no need to develop
a rating system to satisfy this statement in the Charter -- the
Charter does not say "promote THE BEST test suite and demote all
others".

[ BTW, the charter has a practical conflict between fostering the
development of usable and useful test suites and fostering the
development of test suites which share a common look and feel. This
conflict would have to be resolved on a case-by-case basis, I guess:
"This tool is so usable and useful, but it does not meet W3C Look and
Feel Recommendation 1.13; what do we do now?". ]

> The fact of the matter is that working groups are going to have to
> produce test suites, and those test suites are going to have to
> live up to standards.

If W3C must have meta-level standards with obvious requirements, fine.
Such a standard would not hurt (or help) much. If you are going to
introduce a rating system, then you may be hurting those working
groups that develop better tools but receive worse scores due to your
necessarily imperfect rating algorithm.

Rating is only useful when you have to select the "best" product among
several candidates. I do not see why QA WG should be concerned with
such selections: If a working group produces two test suites, W3C
should promote both or ask the WG to nominate a "winner". I do not see
a good reason for comparing test materials produced by different WG.
Is there such a reason?

Alex.

Received on Thursday, 28 February 2002 16:39:32 UTC