W3C home > Mailing lists > Public > www-qa-wg@w3.org > October 2003

Addressing Sandra's comments on TCDL

From: <david_marston@us.ibm.com>
Date: Tue, 28 Oct 2003 11:50:41 -0500
To: www-qa-wg@w3.org
Message-ID: <OFAB37C823.26FAD2DC-ON85256DCD.005731F6@lotus.com>

I've numbered Sandra's questions for easy cross-reference.

>1. If this document is to be considered complementary material and 
>practices to the QA guidelines, so far, the relationship to SpecGl is 
>captured but the relationship to TestGl is not (Would it be captured
>later?)

TCDL is most or all of the metadata from Checkpoint 3.2 of TestGL. For
now, I can make a general reference to TestGL, in case TCDL gets
published first. TestGL should refer to TCDL somewhere, but it is more
general and can defer to TCDL for the specifics.

>2. It is imperative for this document as well as TestGL that a 
>definition of test material be included in the QA glossary.

Fine with me. I expect TCDL to use QA Glossary terms where it can.

>3. Are you referring to the test developers when using WG?

I try to think of the WG and and test developer/contributors as two
separate parties. If there is a passage where you think I said one
and meant the other, please let me know, but check answers 4 and 5
below for more on this topic. If I should refer to "Test Task Force",
it means a subcommittee of the WG.

>4. The use of this document by the WG (test developer) seems under 
>estimated in the document.

This comment is hard to parse in light of what I said in answer 3.
Nevertheless, let me sketch out three ways in which test case authors
(including non-WG contributors) might get their test cases cataloged
using TCDL:
A. Author must write all the TCDL for every case;
B. Author submits case, WG writes TCDL for it (unlikely);
C. Author writes a mandatory part of the TCDL, WG adds more metadata.
Note that even for scenario A, the WG must enumerate values for some
items before accepting submissions. For example, the WG defines a set
of result-comparison techniques and identifiers for each. (TCDL 1.0
probably doesn't define those but just give some guiding examples.
TCDL 2.0 may be able to go further, if QAWG has gathered some
comparitor tools that it favors.) Then the test case developers are
told that each test case must designate a result-comparison technique
from the enumerated set.

The reference to "mandatory part" in scenario C indicates that the WG
would require contributors to provide those bits of metadata that they
know best, such as the inputs and outputs. The WG would finish off the
entry for each test case with data that they want to control very
precisely across all submissions. (Note how this meshes with TestGL
checkpoint 3.2.)

>5. The document could also be used as means of capturing testcase 
>contributions by other test developers.

Absolutely! It is not really designed to track commitments, but see
answer 7 for more about that. As I pointed out in answer 4, any
submission of test cases probably should be accompanied by the TCDL
for those cases, to be merged and possibly supplemented by the WG.

>6. The document could also support packaging the test materials for 
>public release including documentation.

Indeed, the main purpose of the TCDL is to catalog what is in the
public release. If there are documentation files, do we want them
cataloged in TCDL, too? I could invent an element for that. The
original idea was that a ReadMe.txt file points to all the docs, and
points to whatever file (e.g., catalog.xml) contains the TCDL.

>7. In section 1.1 paragraph three, additional information was 
>provided to expand on maintenance activities but no information was 
>provided to expand on test material development.

If I were to say a little more about the development side, it would
be in the context that the set of status codes for a test case is
specified elsewhere. My use of statuses such as "Approved" is for
expository purposes, and I should add a disclaimer about that. It
would be possible for the WG to define statuses such as "Needed" and
"Promised" to reflect test cases that are not yet submitted. If they
did that, and if they used TCDL with the Status-Tracking Module, then
they could write TCDL entries before they even had the test cases
being described and use the pre-submission status codes. Section 1.4
should probably state that TCDL is not intended to be a full-blown
tracking system for development. In particular, I don't think TCDL
should be used to store the name of the person/company that promised
to develop the test, priority for completion, nor any deadlines.
(If QAWG wants that stuff, let me know ASAP.)

>8. Recommend including a list of mandatory/optional elements and 
>attributes allowed in a TCDL document.

I will list all the elements and discuss extensibility. Some of them
will be mandatory for any test regime, while others will be specified
by the WG as mandatory or optional. As mentioned in answer 4 above,
each WG will have to enumerate some lists. (Examples: citations, roles
for inputs and outputs, discretionary items, scenarios, comparison
techniques, etc.)

>9. Strongly suggest keeping the document simple.

I'll do the best I can. Like most such documents, simplicity gets a
lower priority than precision and completeness of coverage.
.................David Marston
Received on Tuesday, 28 October 2003 11:50:58 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:14 GMT