Use cases
Conformance requirements: The test suite must define its scope, goal, and intended purpose.
Rationale: When writing test suites it is critical to understand their primary purpose and scope. The scope describes the areas covered by the test suite, thereby indicating the applicability of the test suite, the motivation and objectives, and coverage. For example, the goals, intended purpose, and coverage of tests for a CR document may be different than those for a Recommendation.
Discussion:
Conformance requirements: All specifications referenced by the specification under test must be identified. The extent to which these specifications can be assumed to be tested, or must be tested by the conformance test suite for this specification, must be stated.
Rationale: Many specifications make reference to and/or depend on other specifications. It is important to determine the extent to which the specification under test can assume that referenced specifications have already been conformance-tested.
Discussion:
Conformance requirements: The scope, goal, and purpose of the test suite as a whole, and where appropriate of each logical 'partition' of the test suite, and the mapping between such partitions and sections of the specification, must be identified and documented
Rationale: Different areas of the specification under test may require different testing approaches (for example, low-level API testing as opposed to higher-level user-scenario testing). The test-suite documentation should explain any partitioning of the specification (and other referenced specifications) and the testing approach adopted for each partition.
Discussion:
As recommended in the QA Specification Guidelines, an important prerequisite of test development is the identification of test assertions from the specification.
Conformance requirements: Test assertions within or derived from the specification must be identified and documented.
Rationale: Conformance testing involves the verification of normative statements within specifications. If such statements ("test assertions") cannot be unambiguously identified, the validity of the corresponding tests will be in dispute.
Discussion: In some cases assertions can be directly identified within the specification (that is, by identifying text from within the specification). In other cases assertions may be derived from the specification, perhaps automatically.
Conformance requirements: The Working Group must define the required set of metadata that will be associated with test assertions. This must include at least the following data:
Rationale: It must be possible to uniquely identify assertions, and to map them to a particular location, or to particular text, within the specification.
Note that the metadata defined in this checkpoint is not exhaustive; Working Groups are encouraged to define additional metadata as appropriate.
Discussion:
[ED NOTE: These checkpoints overlap with 10.1 and 10.2 from SpecGL ("Provide test assertions" and "Provide a mapping between the specification and the test assertions list"). Is it appropriate to duplicate them here?]
When test materials are submitted to the Working Group they will need to be cataloged, reviewed, and tested before being they can be documented, packaged, and published.
In order to facilitate these activities it is advisable to define and document a process for managing test materials. This process should address how test materials are identified, where they will be stored, the kind of metadata that will be associated with them, and how they can be selected and filtered according to various criteria.
Conformance requirements: The Working Group must define and document the process it will adopt for managing test materials.
Rationale: A well-defined mechanism for managing test materials will facilitate the process of reviewing, testing, documenting, and packaging them for public release.
Discussion:
Conformance requirements: The Working Group must define the required set of metadata that will be associated with test materials. This must include at least the following data:
When the Working Group requests test submissions, it must also request that the appropriate metadata be supplied.
Rationale: Well defined test metadata will allow tests to be filtered and selected (whether at 'build time', while a collection of tests is being constructed, or at 'run time' when they are executed), will facilitate the test development and review processes, and is essential for the definition of a test execution process (see Guideline 4).
Discussion: Some Working Groups may choose to create some test metadata elements even before test development begins, as an aid to planning and managing the test development process. For example, it may be helpful to provide the test purpose and description as a form of test-specification. Note that the metadata defined in this checkpoint is not exhaustive; Working Groups are encouraged to define additional metadata as appropriate. Some examples are:
[ED NOTE: The term "test materials" seems clumsy in this context; the majority of metadata elements refer to "tests").]
Conformance requirement: The test materials management process must provide coverage data. At a minimum, the percentage of assertions for which at least one test-case exists should be calculated and published.
Rationale: In order to thoroughly test an implementation, each area of functionality must be systematically tested. Test developers and the Working Group must know what has been covered, what still needs to be covered and what is not applicable. A mapping of test assertions to test cases is the best way to track this.
Discussion: Note that while the coverage metric specified above provides a measurement of breadth of coverage, it says nothing about the extent to which assertions are thoroughly tested (depth of coverage).
Conformance requirements: The test materials management process must include an issue-tracking system.
Rationale: If a high-quality test suite is to developed it is important to methodically record and track problems and issues that arise during test development, testing, and use. For example, the test review process may generate a variety of issues (whether the test is necessary, appropriate, or correct), while after publication users of the test suite may assert that a particular test is incorrect. Such issues must be tracked, and their resolution recorded.
Discussion:
Conformance requirements: The test materials management process must be automated.
Rationale: Automation of the test materials management process, perhaps by providing a web-based interface to a database backend, will simplify the process of organizing, selecting, and filtering test materials.
Discussion:
It is important that test execution be a repeatable and deterministic process.
Conformance requirements: The process for executing tests must be well defined and documented.
Rationale: The test-execution process should be repeatable and deterministic; different test execution runs on the same system, even if performed by different testers, should return identical results. This can only be ensured by explicitly specifying the selection of tests to be run, the mechanisms for invoking them, and the way in which test results should be interpreted.
Discussion:
[ED NOTE: This checkpoint overlaps with 5.1 from OpsGL ("Ensure test materials are documented and usable for their intended purposes").]
Conformance requirements: Test execution should be automated in a cross-platform manner. The automation system must support running a subset of tests based on various selection criteria.
Rationale: If feasible, automating the test execution process is the best way to ensure that it is repeatable and deterministic, as required by Checkpoint 4.1. If the test execution process is automated, this should be done in a cross-platform manner, so that all implementers may take advantage of the automation.
Discussion:
Conformance requirements: If the test execution process is automated, the automation system must also capture test results.
Rationale: If tests are executed automatically by some kind of test harness it will be a relatively simple matter to design the harness to capture test results for later processing by a results-reporting system.
Discussion:
However test materials are developed (this is typically performed in a distributed manner by vendors who are interested in the specific technology) they must be reviewed, tested, documented, and packaged together with other materials (documentation, data files, perhaps a test harness) into a "test suite", which itself must be tested before it is finally released.
Conformance requirements: The test materials must be reviewed to ensure that they meet the submission requirements. The status of the review must be recorded in the test materials management system, as discussed in Checkpoint 3.2 above.
Rationale: Working Groups usually state requirements for the submission of test materials. These will typically include a request that certain metadata be provided with the test materials (see Checkpoint 3.2 above), and may include additional requirements, for example that assertion lists and coverage information be provided. Thorough review of the submitted materials will help to ensure that these additional requirements have been met.
Discussion:
Conformance requirements: The Working Group must prepare documentation that explains how the test materials are to be used.
Rationale: Guideline 4 requires that the test execution process be well defined and deterministic. This will typically require the creation of some end-user documentation that explains to testers how to use the test materials.
Conformance requirements: The test materials must be packaged together with all applicable user documentation, licenses, data files, tools, and test harness into a comprehensive test suite.
Rationale: Guideline 4 requires that the test execution process be well defined and deterministic. This would be difficult to achieve if it were left to the users of the test materials to determine which materials are required, and how to combine them.
Discussion: A simple tar or zip file, containing all relevant test materials and documentation, should be sufficient to meet the requirements of this checkpoint.
Conformance requirements: The Working Group must create and publish a test plan for testing the test materials and the test suite as a whole. The results of executing this test plan must be published.
Rationale: If tests are buggy or incorrect this will be noticed eventually, and will lead to their exclusion from the test suite. Similarly, if the test suite as a whole has problems (if it is difficult to install, or has inaccurate documentation) then these problems will be reported by users. It is preferable to discover and correct these problems before the test suite is released rather than to wait for users to discover them while testing their implementations. The test materials must therefore be tested to ensure that they function correctly, and that they correctly test what they claim to test, and the test suite as a whole must be tested to ensure that it is usable and that it performs as designed.
Discussion:
[ED NOTE: The previous draft had two separate checkpoints for testing: one for testing the "test materials" and another for testing the "test suite". These have been combined into this checkpoint.]
[ED NOTE: For ExTech - beta-test the test suite before release.]
Conformance requirements: The Working Group must encourage users of the test materials to provide feedback, and should incorporate this feedback into the test materials as development proceeds.
Rationale: test suites can reach high quality only if they themselves are thoroughly tested. This testing process should include real-world testing by users and potential users of the test materials.
Discussion:
A well defined and consistent mechanism for reporting test results is essential to ensure that the test materials are easy to use and that it is possible to compare the results of executing the tests on different implementations. This will make easier to compare implementation feature-sets to determine interoperability. It also provides data that supports the process of identifying overall and feature-specific levels of support among implementations.
As with test-materials management and test execution it is beneficial, but by no means required, that some level of automation be applied.
Conformance requirements: Tests must report their execution status in a consistent manner. At a minimum, tests must report the following statuses.
Rationale: If tests do not report their status, or if they do so in an inconsistent manner, it will be difficult or even impossible to unambiguously determine the results of a test execution run.
Discussion: It is not necessary for tests to automatically report status for this checkpoint to be met. It would be sufficient, in the case of manually executed tests, for the test execution procedure to unambiguously define how the person executing the tests should determine the test execution status.
Conformance requirements: When tests fail, they must provide diagnostic information to assist the implementer in determining the source of the problem.
Rationale: In order to improve an implementation the developers must know what is failing and how it is failing in order to fix it. The more details that are provided about the failure, the easier it is for the developers to locate and fix the problem.
Conformance requirements: The process for reporting test results must be defined and documented.
Rationale: A well defined and standardized process for reporting test results will facilitate and enable the comparison and publication of different test results. Working Groups must define a standard form for results reporting, and make the necessary style sheets available to implementers. This will facilitate their publication on the web.
Discussion:
Conformance requirements: The results reporting framework must support filtering on the same metadata used in the Test Management System.
Rationale: Results should be filterable based on criteria just as test cases should be. This allows for implementors to focus on tests and the results of tests that relate to the particular target criteria of the implementation, while being able to filter out tests and the results of tests that are not applicable to a given implementation.
Discussion: While this kind of filtering is certainly valuable, in many cases it will be more efficient to filter at 'build time' or 'runtime' rather than during the analysis of test results.
Conformance requirements: The results reporting system should be automated.
Rationale: As with the test materials management system, some degree of automation is desirable. If the test materials management system has been automated, it is likely that a similar mechanism could be used for results reporting, and desirable that the two systems be linked or integrated.
Discussion: