Re: OpsGL CP1.1 commitment table

David,

Thanks for your comments...

At 09:29 PM 1/12/03 -0500, you wrote:
>[...]
> >1.) that the columns (C) of the table were inextricably combined in
> >  satisfying a given level;
> >2.) that the levels (rows, R) were nested, and someone could in
> >  principle satisfy e.g. R3C1 without satisfying R1C1 (ignoring the
> >  "in addition to..." clause, of course);
> >3.) that the conformance requirements were fuzzy.
>
>I agree that (1) and (2) can be retained. I notice that since the
>original version,

...by which, do you mean your original email, from which this is all derived?

[1] http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html

Wording like the current wording goes back at least to the 20020515 
publication.  The previous one (Feb 2002, FPWD) is fairly different.

>the notion of having an inventory of test cases
>that should exist has now been replaced (I guess) by test assertions
>and that those assertions now fall into the documentation column.
>Meanwhile, the test materials column has been simplified to discuss
>only test cases, not any harness to run them.
>
>One thing needs to be addressed about the interlock between the two
>columns: at Level 2, the test materials are "not necessarily
>complete and thorough" with respect to the spec, but *are* they
>complete and thorough with respect to the test assertions that have
>been identified so far? I think not, meaning that you have two
>separate sets, neither complete, but the test cases could be less
>complete than the assertions.

We left it as is (i.e., 20021220 with my proposed clarifications) at 
telecon today.  This seems in keeping with your recommendation.

>Do you care whether there is a case
>for every assertion (and combination of assertions)? If so, there
>might need to be more levels.

At the lower levels, we do not care (this is a "default" decision today, as 
we didn't discuss or endorse it explicitly).  We defined "complete TS" to 
be:  at least one test case associated with each identifiable 
*requirement*.  (We get into difficulties about the priority of the TA 
checkpoints, the completeness of the TA list, etc, if we try to tie it TC's 
to TA's.)


>Regarding Level 2 assertions: I think it's not much of a failure
>to begin collecting test cases without assertions. If tests will
>be written, they will influence the spec verbiage, including the
>assertions.

I agree.  SVG typically does this.


>The big chasm seems to be between Levels 4 and 5, since I gather
>that Level 5 is where we start to think about completeness. If
>that's the intent, then the word "complete" should be there, or its
>equivalent that satisfies the critics. But QAWG needs to think about
>whether (attempted) completeness is too hard to tie to Priority 2.

Here is an interesting point.  CP14.1 (& 14.2) of SpecGL don't mention the 
word "complete".  Until it does, we should not assume "complete" at level 
5.  "Complete" for the list of test assertions has the same problems of 
definition as "complete TS".

-Lofton.

Received on Monday, 13 January 2003 18:28:08 UTC