W3C home > Mailing lists > Public > www-qa-wg@w3.org > January 2003

Re: OpsGL CP1.1 commitment table

From: David Marston/Cambridge/IBM <david_marston@us.ibm.com>
Date: Sun, 12 Jan 2003 21:29:16 -0500
To: www-qa-wg@w3.org
Message-ID: <OF8E65E9CA.53725DE4-ON85256CAD.000B40AF-85256CAD.000DBB9A@lotus.com>





>1.) that the columns (C) of the table were inextricably combined in
>  satisfying a given level;
>2.) that the levels (rows, R) were nested, and someone could in
>  principle satisfy e.g. R3C1 without satisfying R1C1 (ignoring the
>  "in addition to..." clause, of course);
>3.) that the conformance requirements were fuzzy.

I agree that (1) and (2) can be retained. I notice that since the
original version, the notion of having an inventory of test cases
that should exist has now been replaced (I guess) by test assertions
and that those assertions now fall into the documentation column.
Meanwhile, the test materials column has been simplified to discuss
only test cases, not any harness to run them.

One thing needs to be addressed about the interlock between the two
columns: at Level 2, the test materials are "not necessarily
complete and thorough" with respect to the spec, but *are* they
complete and thorough with respect to the test assertions that have
been identified so far? I think not, meaning that you have two
separate sets, neither complete, but the test cases could be less
complete than the assertions. Do you care whether there is a case
for every assertion (and combination of assertions)? If so, there
might need to be more levels.

Regarding Level 2 assertions: I think it's not much of a failure
to begin collecting test cases without assertions. If tests will
be written, they will influence the spec verbiage, including the
assertions.

The big chasm seems to be between Levels 4 and 5, since I gather
that Level 5 is where we start to think about completeness. If
that's the intent, then the word "complete" should be there, or its
equivalent that satisfies the critics. But QAWG needs to think about
whether (attempted) completeness is too hard to tie to Priority 2.
.................David Marston
Received on Sunday, 12 January 2003 21:30:38 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:12 GMT