W3C home > Mailing lists > Public > www-qa@w3.org > February 2002

Re: test suite distinctions [was: Re: Feedback on "The Matrix"]

From: Rob Lanphier <robla@real.com>
Date: Thu, 28 Feb 2002 10:39:22 -0800 (Pacific Standard Time)
To: Alex Rousskov <rousskov@measurement-factory.com>
cc: Lofton Henderson <lofton@rockynet.com>, Al Gilman <asgilman@iamdigex.net>, "www-qa@w3.org" <www-qa@w3.org>
Message-ID: <Pine.WNT.4.43.0202281026420.254-100000@robla350.dev.prognet.com>
On Thu, 28 Feb 2002, Alex Rousskov wrote:
> Personally, I doubt it is possible to produce [a checklist to
> objectively rate test suites]. My
> doubts are based on my inability to suggest such a checklist for test
> tools I am familiar with. They are too different in scope,
> functionality, and intended audience to be rated using common
> criteria.

I was going to object to this point, but in writing my response, I find
myself in the middle. I think Alex is right in saying that this group
won't be able to construct a checklist that is safe to mechanically apply
to test suites to determine their conformance level.  There's going to be
qualitative judgements made.

That said, the W3C makes qualitative judgements all of the time, any time
a specification is promoted to "Recommendation" status.  So, there may
need to be various levels of blessing of test suites, and a process for
getting those test suites blessed.  Part of the process would most likely
involve measuring the test suite against a checklist, but there should be
a process to ensure that we arrive at a consensus judgement regarding the
satisfaction of that checklist.

Rob
Received on Thursday, 28 February 2002 13:45:55 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:13:58 GMT