Re: RDF Core test driven development and QA Test Doc

I know Karl has proposed (previous message in this thread) that the 
test-driven development of OWL specifications means, by definition, that 
you pass the checkpoint we're talking about.

Without addressing at that view, I'd like to look at this from a different 
angle, starting with a clarification.  Again, these are my own views, as 
the QAWG has not yet taken up the question(s) you raise.  (But will do so 
in a near-future telecon.)

At 09:07 PM 1/6/04 +0000, Jeremy Carroll wrote:

>[...]
>Test GL:
> >> [[
> >> Checkpoint 1.3. Analyze the structure of the specification, partition
> >> it as appropriate, and determine and document the testing approach to
> >> be used for the test suite as a whole and for each partition.
> >> [Priority 1]
> >> ]]
>
>This checkpoint defines method - the testing approach is determined as a 
>result of analysing the specification.
>That might not be what the QAWG intends but that is what the current WD 
>and editors draft both say.

We have been trying to move away from defining the method or process, which 
was prevalent in the earliest drafts.  Instead, we are trying to state the 
requirements in terms of testable characteristics of the test suite (TS).

Note that the normative content of CP1.3 is not in the statement quoted 
above, but rather it is contained in the Conformance Requirements section:
"The scope, goal, and purpose of the test suite as a whole, and where 
appropriate of each logical 'partition' of the test suite, and the mapping 
between such partitions and sections of the specification, must be 
identified and documented"
[Note.  This WG-only draft is raw, and "must" should be read as "MUST" -- 
i.e., that's a normative must above.]

So we've improved the ConfReqs -- the only normative part of the checkpoint 
-- so they can now be read as testable conditions on a conforming 
conformance test suite.  [We should continue to align the CPs' statements, 
so they don't misleadingly suggest "process", and we should fiddle with the 
wording of the Rationale, for same reason.]

You then said...

>(So while the checkpoint above could be written more declaratively e.g.
>"There should be a systematic relationship between the organization of the 
>test suite and the sections of the specification", this would still have 
>been a problem for RDF Core, in my view. Such a change, while an 
>improvement, does not go far enough to get my support).

This is getting close to the ConfReqs for CP1.3, which are the normative 
parts of the checkpoint.  Except notice that CP1.3 ConfReqs only requires 
that the relationship be documented, whatever it is, whereas your proposed 
rewording seems to require a systematic relationship.  (Since you said, 
"still a problem", I'm assuming that you're thinking that we intend to 
require that the structures be parallel, isomorphic, or something like 
that.  IMO, CP1.3 does not require that, but encourages its consideration 
by requiring documentation.)

IMO, for the OWL TS to pass CP1.3 would be trivial, if indeed it doesn't 
already pass (as Karl claims) -- it would be a small addition (not even 
necessarily Normative), to the OWL TS document.  Maybe it's even already 
there?  (I read the document, but don't remember all the details).

In other words, while we think about such questions as whether the OWL TS 
is within scope of various QAF requirements, there is nothing about the 
test-driven development process that would preclude simple passing of CP1.3.

Notice also that I used "conformance test suite" above.  This plays into 
your later comment:

>[...]
>I tend to agree with Lofton's suggestion that the OWL and RDF TS are not 
>TS as envisaged by the QAF.

It is my own view that "Conformance TS" is intended everywhere in QAF where 
simply "TS" is used (or almost everywhere).  This has been my own working 
assumption all along.  The OWL TS document says, "the OWL TS is not a 
conformance TS".

I know that some others in QAWG also have the same working assumption, that 
"TS" is synonymous with "Conformance TS".  It is strongly suggested in 
places in the QAF, e.g., in QAF Introduction.  If this were to turn out to 
be the QAWG consensus (stay tuned about that), then the following might be 
more a simple editorial issue (clarification of original intent) than a 
substantive problem...

>This problem is made worse by
>
>http://www.w3.org/TR/2003/CR-qaframe-ops-20030922/guidelines-chapter#Ck-TM-plans-in-charter
>[[
>for any Test Materials that it plans to produce or adopt, the WG MUST 
>define its commitment level to QA Framework: Test Guidelines -- A, AA, or AAA.
>]]

We (QAWG) plan to look at these questions soon.

-Lofton.

[1]