Conformance requirements: The test suite must define its scope, goal, and intended purpose.
Rationale: When writing test suites it is critical to understand their primary purpose and scope. The scope describes the areas covered by the test suite, thereby indicating the applicability of the test suite, the motivation and objectives, and coverage. For example, the goals, intended purpose, and coverage of tests for a CR document may be different than those for a Recommendation.
Conformance requirements: All specifications referenced by the specification under test must be identified. The extent to which these specifications can be assumed to be tested, or must be tested by the conformance test suite for this specification, must be stated.
Rationale: Many specifications make reference to and/or depend on other specifications. It is important to determine the extent to which the specification under test can assume that referenced specifications have already been conformance-tested.
Conformance requirements: The scope, goal, and purpose of the test suite as a whole, and where appropriate of each logical 'partition' of the test suite, and the mapping between such partitions and sections of the specification, must be identified and documented
Rationale: Different areas of the specification under test may require different testing approaches (for example, low-level API testing as opposed to higher-level user-scenario testing). The test-suite documentation should explain any partitioning of the specification (and other referenced specifications) and the testing approach adopted for each partition.
As recommended in the QA Specification Guidelines, an important prerequisite of test development is the identification of test assertions from the specification. Once assertions have been identified they should be tagged with attributes that will enable test developers, the test management system, the test execution framework, and the results reporting process to make useful distinctions between groups of tests.
Conformance requirements: Test assertions within or derived from the specification must be identified and documented.
Rationale: Conformance testing involves the verification of normative statements within specifications. If such statements ("test assertions") cannot be unambiguously identified, the validity of the corresponding tests will be in dispute.
Conformance requirements: The Working Group must define the required set of metadata that will be associated with test assertions. This must include at least the following data:
Rationale: Well-defined metadata enables test developers, the test materials management system, the test execution framework, and the results reporting process to make useful distinctions between groups of tests.
Conformance requirements: Additional metadata may be associated with test assertions. This should include at least the following data:
Rationale: In addition to the essential data defined in CP 2.2, additional metadata can aid in planning for test development.
[ED NOTE: If we can't come up with more 'priority 2' metadata, then this checkpoint should be removed. Put the statement that the metadata list is not exhaustive, and a suggestion that additional metadata is OK (use development priority as an example) into the Discussion for 2.3.]
Note that the metadata defined in CPs 2.2 and 2.3 is not exhaustive; Working Goups are encouraged to define additional metadata as appropriate.
When test materials are submitted to the Working Group they will need to be cataloged, reviewed, and tested before being they can be documented, packaged, and published.
In order to facilitate these activities it is advisable to define and document a process for managing test materials. This process should address how test materials are identified, where they will be stored, the kind of metadata that will be associated with them, and how they can be selected and filtered according to various criteria.
Conformance requirements: The Working Group must define and document the process it will adopt for managing test materials.
Rationale: A well-defined mechanism for managing test materials will facilitate the process of reviewing, testing, documenting, and packaging them for public release.
Conformance requirements: The Working Group must define the required set of metadata that will be associated with test materials. This must include at least the following data:
When the Working Group requests test submissions, it must also request that the appropriate metadate be supplied.
Rationale: Well defined test metadata will allow tests to be filtered and selected (whether at 'build time', while a collection of tests is being constructed, or at 'run time' when they are executed), will facilitate the test review process, and is essential for the definition of a test execution process (see Guideline 4).
Conformance requirement: The test materials management process must provide coverage data.
Rationale: In order to thoroughly test an implementation, each area of functionality must be systematically tested. Test developers and the Working Group must know what has been covered, what still needs to be covered and what is not applicable. A mapping of test assertions to test cases is the best way to track this.
Conformance requirements: The test materials management process must include bug-tracking data.
Rationale: If a high-quality test suite is to developed it is important to methodically record and track bugs in test materials.
Conformance requirements: The test materials management process should be automated.
Rationale: Automation of the test materials management process, perhaps by providing a web-based interface to a database backend, will simplify the process of organizing, selecting, and filtering test materials.
It is important that test execution be a repeatable and deterministic process.
Conformance requirements: The process for executing tests must be well defined and documented.
Rationale: The test-execution process should be repeatable and deterministic; different test execution runs on the same system, even if performed by different testers, should return identical results. This can only be ensured by explicitly specifying the selection of tests to be run, the mechanisms for invoking them, and the way in which test results should be interpreted.
[ED NOTE: This checkpoint overlaps with 5.1 from OpsGL ("Ensure test materials are documented and usable for their intended purposes").]
Conformance requirements: Test execution should be automated in a cross-platform manner. The automation system must support running a subset of tests based on various selection criteria.
Rationale: If feasible, automating the test execution process is the best way to ensure that it is repeatable and deterministic, as required by Checkpoint 4.1. If the test execution process is automated, this should be done in a cross-platform manner, so that all implementors may take advantage of the automation.
Conformance requirements: If the test execution process is automated, the automation system must also capture test results.
Rationale: If tests are executed automatically by some kind of test harness it will be a relatively simple matter to design the harness to capture test results for later processing by a results-reporting system.
[ED NOTE: is 4.3 implicit in 4.1 and 4.2?]
However test materials are developed (this is typically performed in a distributed manner by vendors who are interested in the specific technology) they must be reviewed, tested, documented, and packaged together with other materials (documentation, data files, perhaps a test harness) into a "test suite", which itself must be tested before it is finally released.
Conformance requirements: The test materials must be reviewed to ensure that they meet the submission requirements.
Rationale: Working Groups usually state requirements for the submission of test materials. These will typically include a request that certain metadata be provided with the test materials (see Checkpoint 3.2 above), and may include additional requirements, for example that assertion lists and coverage information be provided. Thorough review of the submitted materials will help to ensure that these additional requirements have been met.
Conformance requirements: The test materials must be tested to ensure that they are usable, and that they correctly test what they claim to test.
Rationale: If tests are buggy or incorrect this will be noticed eventually, and will lead to their exclusion from the test suite. It is preferable to discover this before they are released than to wait for users of the test materials to discover this while testing their implementations.
Conformance requirements: The Working Group must prepare documentation that explains how the test materials are to be used.
Rationale: Guideline 4 requires that the test execution process be well defined and deterministic. This will typically require the creation of some end-user documentation that explains to testers how to use the test materials.
Conformance requirements: Test materials must be packaged together with all applicable user documentation, licenses, data files, tools, and test harness into a comprehensive test suite.
Rationale: Guideline 4 requires that the test execution process be well defined and deterministic. This would be difficult to achieve if it were left to the users of the test materials to determine which materials are required, and how to combine them.
Conformance requirements: The test suite must be tested before release.
Rationale: testing the test suite as a whole is likely to flush out integration problems, and will help to improve the quality and usablity of the test suite.
[For ExTech: beta-test the test suite before release.]
A well defined and consistent mechanism for reporting test results is essential to ensure that the test materials are easy to use and that it is possible to compare the results of executing the tests on different implementations. This will make easier to compare implementation feature-sets to determine interoperability. It also provides data that supports the process of identifying overall and feature-specific levels of support among implementations.
As with test-materials management and test execution it is beneficial, but by no means required, that some level of automation be applied.
Conformance requirements: Tests must report their execution status in a consistent manner. At a minumum, tests should report whether they passed, failed, or whether the results were inconclusive.
[ED NOTE: do we need to add more states?]
Rationale: If tests do not report their status, or if they do so in an inconsistent manner, it will be difficult or even impossible to unambiguously determine the results of a test execution run.
Conformance requirements: When tests fail, they must provide diagnostic information to assist the implementor in determining the source of the problem.
Rationale: In order to improve an implementation the developers must know what is failing and how it is failing in order to fix it. The more details that are provided about the failure, the easier it is for the developers to locate and fix the problem.
Conformance requirements: The process for reporting test results must be defined and documented.
Rationale: A well defined and standardized process for reporting test results will facilitate and enable the comparison and publication of different test results. Working Groups must define a standard form for results reporting, and make the necessary style sheets available to implementors. This will facilitate their publication on the web.
Conformance requirements: The results reporting framework must support filtering on the same metadata used in the Test Management System.
Rationale: Results should be filterable based on criteria just as test cases should be. This allows for implementors to focus on tests and the results of tests that relate to the particular target criteria of the implementation, while being able to filter out tests and the results of tests that are not applicable to a given implementation.
[ED NOTE: While this kind of filtering is certainly valuable, in many cases it will be more efficient to filter at 'build time' or 'runtime' rather than during the analysis of test results.]
Conformance requirements: The results reporting system should be automated.
Rationale: As with the test materials management system, some degree of automation is desirable. If the test materials management system has been automated, it is likely that a similar mechanism could be used for results reporting, and desirable that the two systems be linked or integrated.
[ED NOTE: shouldn't this be in OpsGL rather than here?]
Conformance testing is the ultimate goal of QA activities, and real-world use of test materials is essential to ensure that they meet their design goals and are of high quality.
Conformance requirements: The Working Group must create, publish, and implement a plan to engage vendors of implementations to participate in conformance testing activities.
Rationale: Conformance testing promotes higher quality and more interoperable implementations.
[For EXTech: A common practice is to support public discussion group dedicated to the test suite and organize face-to-face meetings for vendors and other interested parties.]
Conformance requirements: The Working Group must encourage users of the test materials to provide feedback, and should incorporate this feedback into the test materials as development proceeeds.
Rationale: test suites can reach high quality only if they themselves are thoroughly tested. This testing process should include real-world testing by users and potential users of the test materials.
[ED NOTE: the previous checkpoint 6.2 ("Encourage Vendors to publish test results") has been removed, since this seems identical to OpsGL checkpoint 6.5 "Address the publication of test results for products".]