Proposal: Class Of Product is test materials (in later discussion we also realized that we must include test metadata, else we will not be able to address any of the "process issues"). Audience is (in priority order) test developers, WG members, and users of tests.

DD and PC hoped there would be no need to define test metadata as a separate COP; they would rather define 'test materials' broadly enough to cover these metadata. On the other hand, including in the term "test materials" a test plan and the results of executing that plan to test those test materials (as in checkpoint 5.4) seems to stretch the definition too far (not to mention introduce a confusing element of recursion).

• help the user detect if the implementation is conformant or not
• help the implementation developer improve his product
• help the specification writer improve his specification

[MS] I don't see where these have anything to do with whether or not a "waterfall model" was used. [PC response] the first bullet in the list below does (test first, develop spec later)...

[PC] need a better definition of strategies for development. For example:

• to clarify whether a proposed feature in spec is implementable
• to verify whether implementations of spec are conformant
• to verify whether implementations of spec are interoperable

We must stress that we don't necessarily require a "waterfall" process, and that the requirements described in the GL can be applied on a recursive basis; we should actually encourage this type of strategy (without requiring it) in this section.

[MS] I'd like to add the use case that good QA (and this guideline) will result in good test materials that will improve Q/A and thus, result in less bugs, lower maintenance costs, etc. In other words, this use case illustrates voluntary use of tests as a carrot, rather than the stick as in certification, to improve software and save money.

[PC] This seems more like a general rationale for following these guidelines rather than a use-case. We should certainly incorporate this somewhere.

Proposal: We need a short, concise way of defining the "waterfall model". Once we've done that, all we need to say is that this guideline does not imply any one type of model. (We also need to ensure that we do not imply a waterfall model in the wording of guidelines and checkpoints.)

References:

• http://asd-www.larc.nasa.gov/barkstrom/public/The_Standard_Waterfall_Model_For_Systems_Development.htm
• http://www.ctg.albany.edu/publications/reports/survey_of_sysdev?chapter=5
• http://www.convergsoft.com/contents/methodology_sdlc.htm

• testing lab/certification authority needs to test products; the spec and the products already exist, only needs to check if the products conform
• implementations are begun before spec is finished; tests are needed to check if they conform
• WG needs feedback on its specification, and uses test cases as a way to get this feedback
• WG uses test cases as a way to explore new features (eg in OWL, SVG, CSS)
• comparisons of actual state of implementations independantly of conformance [example of interop testing]

Reviewing the TA definition in specGL, it appears we also want to move some of the verbiage of SpecGL GL10 to the definition in specGL glossary
and QA-Glossary.

The assertions section of TestGL should point to the definition, address why they are important, how they can be extracted/derived automatically
or not, address what makes a good/useful TA. Needs to emphasize again that there might be a feedback loop between the test assertions extraction and the spec development.

Checkpoint 2.1: Rewrite checkpoint to: Provide a list of test assertions. Change to priority 1. Add some verbiage to the discussion to include examples of derived assertions.

Checkpoint 2.2. The discussion for this checkpoint should include more detail including the possibility of extracting more than one assertion from the same quoted text. It is up to the test developer to decide the mechanism for assigning a unique assertion-ID. The metadata in this checkpoint should be tied with the metadata in checkpoint 3.2.

Replace location for the text from which the assertion was derived.

[PC] we've discussed assertions several times, and don't seem to be able to agree on whether a specific assertion list is required, or whether
it's sufficient to define the "test purpose" in metadata (explaining that assertions are a really good way to do this). Will re-open this as an email discussion, referencing relevant SpecGL Issues.

Proposal: Point to the class of product listed in SpecGl rather than duplicate the list here.

[MS] points out that the COP for SpecGL is "specification", in particular W3C Technical Reports.

[PC] We must be referring to Section 2.2.of SpecGL ("Specification category and class of product") where we do classify different types of spec. (Should that section header really include the term "class of product"?)

Proposal: [MS] and [PC] believe that it's unnecessary to formally define this term. The intuitive understanding most readers will have should suffice.

[PC] Alex will need to document this - we have no minutes.

[PC] I don't understand this statement. Metadata is associated with testcases.

In the conformance requirement the third bullet should go back to assertions. The fifth bullet is too ambiguous, it must be clarified.

In the discussion section:

• The third bullet should go back to assertion as an optional metadata
• Fourth bullet should also go back to assertions but as mandated
• Fifth bullet should go back to assertion but rewording conformance level to degree of conformance
• A new bullet should be added to include conditional tests as an optional metadata.

[PC] see Issue #008

[MS] The distinction between tagging assertions with metadata and associating metadata with test materials is confusing to me. Since assertions ultimately result in test materials why do we need the metadata in two places? Wouldn’t it be simpler (and less confusing) if we just associated the metadata with either the assertions or the test materials, but not both?

[PC] There may be many test-cases associated with a single (complex) assertion. Under these circumstances, attributes like "optional", or "specific to version x.y of the spec" belong with the assertion (so that they aren't duplicated). On the other hand, attributes like "input arguments/data for test" and "expected results of test" clearly must be associated with tests. Continue to discuss this in the context of our ongoing discussion of assertions.

Proposal:

[PC] The assertion list? This is covered elsewhere.

[MS] This requirement should also be modified to say “at a minimum . . . “must” be, rather than “should”. Since this is a minimum list, it’s an absolute requirement, thus “must”.

Description: This addresses process - inappropriate for TestGL.

[PC] have we captured this "good practice" somewhere in OpsGL?

[PC] how is this better than what we currently have ("The process for executing tests must be well defined and documented")? Note also that "well defined" is untestable. The essence of this checkpoint is: "tell me how to execute the tests. Do so unambiguously, so that if someone else follows your instructions they will get the same results as I do". (See Issue #023.)

Alternate suggestion (preferred after discussion): materials must document where test results are not expected to be reproducible and repeatable, and to explain why. If this info is specific to a particular test (as opposed to a group of tests or even the entire test suite) it should be contained in the test
metadata.

Include in the rationale discussion of the order in which tests must be run (if indeed order is important). We must also ensure that every test that should be run is run, and that those that should be excluded from the test run are excluded.

Description: Meeting minutes said define in "glossary". Also in definitions section?

The real definitions are from ISO 5725-1:1994/Technical Corrigendum 1, Published 1998-02-15, "Accuracy (trueness and precision) of measurement methods and results Part 1: General principles and definitions" and ISO 5725-2:1994, "Accuracy (trueness and precision) of measurement methods and results Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method."

"3.13 repeatability: Precision under repeatability conditions.

3.14 repeatability conditions: Conditions where independent test results are obtained with the same method on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time.

3.17 reproducibility: Precision under reproducibility conditions.

3.18 reproducibility conditions: Conditions where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment."

[PC] what's the difference between "platform-independent" and "cross-platform"? (Are we really trying to say "don't write to a particular platform, but don't feel obliged to write for all platforms"?)

[PC] I'm not sure what the last two sentences mean.

Description: Ideally, this means that the WG (or whatever body will in the end endorse the use of the test suite) approves the tests in the test suite
as well as test generation mechanisms (if applicable).

[PC] Once again, we're straying into OpsGL territory by addressing process. Can we sidestep this by requiring that test metadata include the results of reviewing? [MS] agrees.

[DD] Agreed, and I propose to reword to "make tests reviewable" (implies coming up with a scheme for reviewing, which is not process, but tech details.

[PC] Still sounds like process to me. We must focus on the output (results of review) and not on how that output is produced (the "scheme for reviewing").

Description: Need to spell out the parts that make up a test suite, as well as what parts are optional. Also need defintions of the terms used (for example "test harness", "test case").

[MS] This sounds like process to me. I’m not sure it should be a ckpt.

[DD] I think this is as much process as providing test materials to begin with, that is, not much. Packaging is maybe an unsuccessfully picked word, but it boils down to someone having to delimit relevant tests (used to test conformance) from irrelevant ones.

[PC] It's easy to reword the checkpoint to focus on the output (the package) rather than the process (the packaging). The essence of the checkpoint is that a website containing a bunch of links, some of which point to tests, other to docs, others to 'metadata' does not constitute a test suite. Until everything is packaged together test execution runs are unlikely to be reproducible and repeatable. (See also new issue 049 below.)

Description: This needs to be either stressed as either the final proof that this is the real thing (being endorsed by the WG or similar body) or alternatively as one more step in making sure the test suite is designed to be of as high quality as possible (without making reference to it being officially approved).

[PC] Again, this checkpoint addresses process (the WG's activities) rather than the test materials. Require that a test plan (and/or test results) be published along with the test suite?

[MS] Also, as this ckpt now stands, it cannot be tested. (How ironic for a ckpt to test the test suite). The ckpt should require a test plan and test results, as PC suggests.

[DD] Again, Ops since it requires something from the WG, but essential in order to be able to use the test materials for conformance. Include it in the workflow of providing test materials for it to be testable (as a checkpoint). I stress the need for it to be clearly delimited as it is THE activity (I think) that discriminates"officially accepted" test materials from non-accepted ones.

Description: I still think we should have two separate checkpoints for test materials and test suite, respectively, as these are two separate things and quite
different from one another (for example, tests are related to specifications, test suites are not).

[PC] Two separate checkpoints for doing what? Testing the test materials and the test suite?

Description: Describe the possible ways in which feedback is given: mailing list for example requires readers that can act on it. Emphasize that feedback should be acted upon.

[PC] Another process-related checkpoint. Relatively easy to reword as a requirement to define and publish a feedback mechanism.

[MS] The important point is not to describe ways feedback can be given, but to obtain feedback, somehow, some way. The feedback mechanism is not important, just the results. Can we just require that feedback solicitation and feedback results be published? [DD] agrees

Description: Are we here speaking of the automated result reporting in the test suite, or of something separate? Since we indicate automation in the second paragraph of the introduction, we may want to specify that.

[MS] Again, this is a process requirement. Does it belong here?

[PC] Shouldn't have used the "process" word. The blurb under the guideline and the discussion text for checkpoints 6.1 and 6.2 make it clear that we're talking about the mechanisms that the tests use to report their results to the person executing them. If this isn't done in a clear, consistent, and understandable manner it will not be possible to accurately determine which tests have passed and which tests have failed, once again defeating the goal that test runs be reproducible and repeatable.

Description: Again, if automated, this relates more to the technical issues about the framework than the process as such (we want to allow for different
processes to lead to the same result, namely uniform results reporting).

[PC] whether or not we have automation we must still define and document a process for reporting results.

[MS] Again, this is process.

[PC] We can get around this in the same way as with 'review the test results' or 'gather feedback'. Focus on the output and not the process.

[DD, commenting earlier, but really addressing this point] Propose to change wording to "in the absence of existing mechanisms for reporting test results, create one and package together with test suite"

Description:

[PC] Why?

Description: Test Suite is all the pieces of materials needed, wrapped up together. As written not testable. Make a minimal list of what MUST be provided, including: user documentation, IPR, test harness if supplied, referenced output if defined. Need to make sure that ‘test suite’ is understood. Test suite is the package (sum) of all the components needed to test an implementation. Test materials are the components that make up the test suite. (These terms should be defined.)

Description: Management system is the object. David Marston to help define the 2 objects of TestGL. In discussion, include the frequency of applying the test plan and testing.

[PC] ???

Description: Need to add definitions for the terms. Reword. Remove Cannot Tell. In discussion, reference that terms came from EARL. Can you map these to other states or must you use these ‘states’? If these apply, then MUST use them. These are states, we have definitions of the states, definitions are normative, not providing labels for the states, if state applies, use it. Recommend that if English, use these labels. Change status to outcome.

[MS] This ckpt, and others in Guideline 6 use “should” in describing the ckpt. The “should” should (or must - I’m going insane) be changed to “must”. Checkpoints should not leave any wiggle room. A better way to describe the ckpt is with an active verb (e.g., Report test status, Report diagnostic info.)

Description: "automate the system"

[PC] ???

Description: Test suites must be explicitly released (rather than "dribbled out" by constantly updating a website), and must be versioned. If they're not, test runs cannot be deterministic or repeatable and the results of different test runs cannot be compared.