Re: test suite distinctions [was: Re: Feedback on "The Matrix"]

On Thu, 28 Feb 2002, Lofton Henderson wrote:

> At 09:43 AM 2/28/2002 -0700, Alex Rousskov wrote:
> >On Thu, 28 Feb 2002, Lofton Henderson wrote:
> >
> > > Associated WAI-like checklists will allow scoring of processes, specs,
> > > materials according to checklists, and rating of the target with WAI-like
> > > conformance levels (A, AA, AAA).  I'm not sure that I'm understanding the
> > > suggestions in this thread.  Is it suggested that W3C should:
> > >
> > > 1.) not produce such goodness-rating specs/tools?
> > > 2.) produce them but don't, ourselves (W3C), apply them and publish 
> > results?
> > > 3.) something else?
> >
> >I think it is (3) because the context of this thread is test suites
> >(and their goodness-rating) rather than documents (and their
> >goodness-rating). The original proposal was to rate the quality of all
> >test suites listed in the Matrix. This is different from what you seem
> >to be talking about (rating the quality of the documents that those
> >test suite may have been based on).
> 
> Sorry for being unclear previously.  The Framework documents will indeed 
> address the goodness of test suites, and have associated 
> checklists.  That's what I meant by "Test Materials", which is our 
> catch-all phrase for test suites and test tools (to include validators, etc).

Oh, I see now. Let me explain:

The original proposal by Tantek Celik was to rate specs/tools based on
W3C affiliation ("being hosted at w3.org"). I replied saying that W3C
affiliation is not sufficiently related to quality of a test suite
and, hence, has little utility for testers. I argued that rating
quality of a test suite might be useful but is going to be very
controversial. Al Gilman seemed to support my opinion with a couple of
real-life examples.


If the QA WG manages to produce a checklist that helps to measure a
quality of a "test material" in a useful, objective, comprehensive,
and uncontroversial way, then W3C should, of course, promote such a
checklist and publish its application to available "test materials"!

Personally, I doubt it is possible to produce such a checklist. My
doubts are based on my inability to suggest such a checklist for test
tools I am familiar with. They are too different in scope,
functionality, and intended audience to be rated using common
criteria.

Your checklist is, essentially, a test. It is very difficult to agree
on a common test for any given class of products. Agreeing on a common
useful/objective/etc test for all test suites, validators, etc. would
be a miracle!

To summarize: My position is that it is a waste of time to work on
such a checklist, but if the group manages to produce such a
checklist, it should be used to the fullest extent possible.

Alex.

Received on Thursday, 28 February 2002 12:40:21 UTC