- From: <david_marston@us.ibm.com>
- Date: Tue, 28 Oct 2003 11:50:41 -0500
- To: www-qa-wg@w3.org
I've numbered Sandra's questions for easy cross-reference. >1. If this document is to be considered complementary material and >practices to the QA guidelines, so far, the relationship to SpecGl is >captured but the relationship to TestGl is not (Would it be captured >later?) TCDL is most or all of the metadata from Checkpoint 3.2 of TestGL. For now, I can make a general reference to TestGL, in case TCDL gets published first. TestGL should refer to TCDL somewhere, but it is more general and can defer to TCDL for the specifics. >2. It is imperative for this document as well as TestGL that a >definition of test material be included in the QA glossary. Fine with me. I expect TCDL to use QA Glossary terms where it can. >3. Are you referring to the test developers when using WG? I try to think of the WG and and test developer/contributors as two separate parties. If there is a passage where you think I said one and meant the other, please let me know, but check answers 4 and 5 below for more on this topic. If I should refer to "Test Task Force", it means a subcommittee of the WG. >4. The use of this document by the WG (test developer) seems under >estimated in the document. This comment is hard to parse in light of what I said in answer 3. Nevertheless, let me sketch out three ways in which test case authors (including non-WG contributors) might get their test cases cataloged using TCDL: A. Author must write all the TCDL for every case; B. Author submits case, WG writes TCDL for it (unlikely); C. Author writes a mandatory part of the TCDL, WG adds more metadata. Note that even for scenario A, the WG must enumerate values for some items before accepting submissions. For example, the WG defines a set of result-comparison techniques and identifiers for each. (TCDL 1.0 probably doesn't define those but just give some guiding examples. TCDL 2.0 may be able to go further, if QAWG has gathered some comparitor tools that it favors.) Then the test case developers are told that each test case must designate a result-comparison technique from the enumerated set. The reference to "mandatory part" in scenario C indicates that the WG would require contributors to provide those bits of metadata that they know best, such as the inputs and outputs. The WG would finish off the entry for each test case with data that they want to control very precisely across all submissions. (Note how this meshes with TestGL checkpoint 3.2.) >5. The document could also be used as means of capturing testcase >contributions by other test developers. Absolutely! It is not really designed to track commitments, but see answer 7 for more about that. As I pointed out in answer 4, any submission of test cases probably should be accompanied by the TCDL for those cases, to be merged and possibly supplemented by the WG. >6. The document could also support packaging the test materials for >public release including documentation. Indeed, the main purpose of the TCDL is to catalog what is in the public release. If there are documentation files, do we want them cataloged in TCDL, too? I could invent an element for that. The original idea was that a ReadMe.txt file points to all the docs, and points to whatever file (e.g., catalog.xml) contains the TCDL. >7. In section 1.1 paragraph three, additional information was >provided to expand on maintenance activities but no information was >provided to expand on test material development. If I were to say a little more about the development side, it would be in the context that the set of status codes for a test case is specified elsewhere. My use of statuses such as "Approved" is for expository purposes, and I should add a disclaimer about that. It would be possible for the WG to define statuses such as "Needed" and "Promised" to reflect test cases that are not yet submitted. If they did that, and if they used TCDL with the Status-Tracking Module, then they could write TCDL entries before they even had the test cases being described and use the pre-submission status codes. Section 1.4 should probably state that TCDL is not intended to be a full-blown tracking system for development. In particular, I don't think TCDL should be used to store the name of the person/company that promised to develop the test, priority for completion, nor any deadlines. (If QAWG wants that stuff, let me know ASAP.) >8. Recommend including a list of mandatory/optional elements and >attributes allowed in a TCDL document. I will list all the elements and discuss extensibility. Some of them will be mandatory for any test regime, while others will be specified by the WG as mandatory or optional. As mentioned in answer 4 above, each WG will have to enumerate some lists. (Examples: citations, roles for inputs and outputs, discretionary items, scenarios, comparison techniques, etc.) >9. Strongly suggest keeping the document simple. I'll do the best I can. Like most such documents, simplicity gets a lower priority than precision and completeness of coverage. .................David Marston
Received on Tuesday, 28 October 2003 11:50:58 UTC