---

Status

This is the [DATE] QA Test Material Process Document.

Comments on this document are invited and are to be sent to the QA WG mailing list www-qa-wg@w3.org. An archive is available at http://lists.w3.org/Archives/Public/www-qa-wg/.

@@ wg list or ig list? or do we create another (probably not likely needed).@@

This document has been produced as part of the W3C QA Working Group. The authors of this document are the QA WG members.

Table of contents

• 1. Introduction
• 2. Requirements
• 3. Parties involved
• 4. QA-related Communication
• 5. Developmental Framework
• 6. Conformance
• 7. Licenses
• 8. Publication
• 9. Test Materials Maintenance

Introduction

This document describes the QA process for ensuring that a comprehensive conformance Checklist is available for the Qa Working Group Guidelines. It provides information about the WG's quality-related logistical and communications setup, contribution mechanism, licensing policies, publication and maintenance plans, as well as any other critical processes and operational details needed to produce and maintain the Checklist. It supplements the W3C Process document.

The Checklists produced by the W3C QA WG will be known as QA WG Guildlines Checklist.

Requirements

1. The Checklist should be developed in a public framework. The development of this framework itself is a central area of interest. The QAWG welcomes the participation of interested parties in developing the Checklist.
2. A standard approach to the Checklist IPR which meets the needs of both contributors and users should be established.
3. The Checklist is intended to be used as a tool to aid implementors in defining and impleming qa processes. Validation and certification of these implementations are outside the scope of this work. The Checklist will be provided for information and help only. However, we intend to produce as comprehensive, functional, and general checkpoints as possible.
4. The Checklist should be available in a common environment so they can be accessed publicly and shared among developers and users.
5. The Checklist will be hosted on the W3C site once developed. It may also after development be mirrored at various sites in order to simplify access and enhance availability to the community.

Parties involved

The Checklist will be produced in a public framework with contributions from developers and companies in the community. The Checklist will be coordinated and supervised by the QA WG. The representatives will use resources from their organisations as well as invite individuals and companies (through representatives) to allow for a larger number of people to get involved.

QA-related Communication

[For OpsGL checkpoint CP4.4]

The main channel of communication for the Checklists will be the existing QA WG mailing list www-qa-wg@w3.org. An archive is available at http://lists.w3.org/Archives/Public/www-qa-wg/

@@ Should we/are we going to do this..@@

The public web page for QA information is named /Test/ and is available at: [http://www.w3.org/WGnameTest/]. This page includes links to this process document, the Checklists and information about the Checklist including:

• an overview or introduction to the Checklist
• releases
• how to participate
• how to use the Checklist
• Checklist principles
• acknowledgements

Development Framework

[For OpsGL checkpoint CP4.5]

The Checklist will be produced in a public framework with contributions from members of the QA WG, W3C, or the community at large. The framework will provide a stable mechanism for submission of test materials, saving information about those checkpoints, version handling, use of the checkpoints, and so forth.

Test material contributions

[For OpsGL checkpoint CP5.2]

Contributions should be submitted to the Checklist via the QA WG mailing list: www-qa-wg@w3.org.

In order to simplify for individuals and companies the production of the test material and, if they so wish, contribute with checkpoints of their own, we propose the following process for submitting checkpoints:

1. It is acknowledged (presumably by the moderators/reviewers of the test materials) that the test has been received. This should be done via email directly to the submitter, and a copy should be sent to the mailing list. The test receives the "Received"; status.
2. The test together with the necessary documentation is saved for future reference.

Receipt and review of test contributions

[For OpsGL checkpoint CP5.4]

The test suite's moderators evaluate the test materials included in a contribution according to the following criteria.

1. Relevance for the particular implementation of QA Processes. It can here be decided that the particular checkpoint is irrelevant to the Checklist since it tests a recomondation that is not in the specification.
2. Part of the guidelines being tested. If there are other, previously submitted checkpoints that are functionally equivalent and test the same part of the guidelines, this test should either be kept for archive purposes but not published or should replace the previously submitted test, in which case the previous item would be archived.
3. Quality of code and documentation. The Checkpoint must be parseable and verifiable.
4. Scenarios that underly the particular Checkpoint and its intended scope.
5. Development guide for the Checklist, once developed. The existence of these guidelines will be communicated through the QA WG mailing list. @@ should this be the WG list @@

If a given test judged suitable for inclusin in the Checklist according to these criteria, the test is added to the Checklist with status "Accepted".

Tests that are judged to be inappropriate for publication are reurned to the contributor. Such checkpoints are archived and not included in the Checklist.

Test status and review procedures

[For OpsGL checkpoint CP8.3 (?)]

Once a Checkpoint is included into the Checklist, it becomes subject to the dispute and review process outlined below.

At any time, any member of the community may dispute the validity of any test in the Checklist. This is done by raising the issue on the QA WG mailing list. If the dispute is felt to be valid either by the community and/or the Checklist Task Force, the status of the test is changed to "disputed", and the test is passed on to the WG for detailed review. The outcome of the review will result in one of the following states.

1. Error Free. The test and its accompanying documentation have undergone review by the WG and have been found to be free of errors and relevant to the specification.
2. Disputed: Checkpoint error. The validity of the test has been questions by one or more members of the community. A consensus exists in the community or the Checklist Task Force agrees (or both) that the test is in error. The Checklist Task Force has submitted the test and the associated dispute to the WG for detailed review, along with a suggested resolution.
3. Disputed: spec error. As a result of running the test, one or more members of the community has questioned the consistency, clarity, etc. of the part(s) of the specification addressed by the test. A consensus exists in the community, or the Checklist Task Force aggress (or both) that a problem exists with the specification. The Checklist Task Force has submitted the test and the associated dispute to the WG's erratum process.

Documentation

Each contribution to the Checklist must be fully documented, and conform to the criteria for test submissions, which is available at the Checklist website. The documentation provides such information as:

• Name of submitter, date of creation, and, if appropriate, organization;
• Suggested ID for each test (this may be changed by the moderator to comply with the identification mechanism of the technical platform);
• Part(s) of the specification being tested. Areas in the specification being tested should be suitably indicated by either a link to the relevant part of the specification or a text annotation to the relevant part of the specification;
• The pass/fail criteria of the test;
• If appropriate, a textual annotation containing remarks on the test in human readable format, for example any scenario which prompted the writing of the test.

Status of the test suite according to the above

1. The Checklist should consist only of submitted, reviewed and stable checkpoints.
2. The Checklist for all levels will also consist of documentation on the checkpoints it contains, scenarios that have served as a base for them, commented code, and possibly comments from the editors.

It is proposed that the W3C QA WG representative act as moderator and controller for incoming checkpoints. The Checklist moderator is chosen by the QA WG. All checkpoints should be kept for archive purposes, whether they get published or not.

Conformance

[For OpsGL checkpoints CP4.6, CP6.4]

The Checklist aims, as explained above, at helping implementers to write specifcation that conform to the QA WG Guidelines. In no way are these conformance checkpoints in the sense of providing companies or institutions with certification or branding of QA support. The only claim that could be made is that a particular implementation is conformant to a particular version of the Checklist. There are two cases of results of running the Checklist:

1. The implementation fails to pass the Checklist. In this case it can be asserted that the implementation fails to meet the relevant Guidelines.
2. The implementation passes the test suite. In this case all that can be asserted is that the implementation is conformant to that particular version of the Checklist.

Licenses

[For OpsGL checkpoints CP5.3, CP6.2]

Grant of License for Contributions under the W3C Software License

The Contributor hereby makes certain materials, as described below (the "Materials";), available for use in supporting the World Wide Web Consortium (W3C). The Contributor hereby grants to MIT, ERCIM, and Keio (the "W3C Hosts";) a perpetual, nonexclusive, royalty-free, world-wide right and license to use, copy, modify, create derivatives of and distribute the Materials, in whole or in part, solely in connection with the W3C, and to allow others to do the same. W3C Hosts will publish and distribute the Materials, or any modifications or derivatives thereof, pursuant to the W3C Software License (20021231), and/or the W3C Document License (20021231) as modified from time to time.

The Contributor represents that she/he has all rights necessary to contribute the Materials, and that use of the Materials as contemplated herein by W3C Hosts does not violate any copyright, patent, trademark, or contractual obligations.

The Contributor agrees that any derivatives of the Materials created in connection with the W3C shall be owned by the W3C Hosts. Any publication or distribution of the Materials or any derivative thereof, will retain attribution of authorship to the Contributor. Whenever modifications are made to the Materials, this fact, and the nature of the modifications, will be clearly identified in the distributed version thereof. The W3C Hosts make no commitment to support or distribute the Materials.

Test material license

The Checklist produced by the QA WG Working Group will be distributed under the W3C Document License.

Publication

Publication of test material

The Checklist will be published on the W3C QA WG website, available from: [http://www.w3.org/[WGname]/Test/].

The Checklist will be published after thorough review by the WG and the community, and with the consensus of the WG.

The Checklist will contain the conformance disclaimer as stated above.

Publication of test results

[For OpsGL checkpoint CP6.5]

The QA WG encourages the publication of test results. To facilitate publication of results, the WG will make available on its webspace, a place to publish test results. Only test results that have been supplied by the developer of the tested implementation will be considered for publication.

The WG will also consider a 'members only' webspace to publish developer submitted test results. This restricted publication of results may be used for situations in which the specification and/or checkpoints are being debugged. For example, when the specification is in CR and a set of checkpoints are being used to assess the features of the specification by determining what is being implemented.

Test Material Maintenance

[For OpsGL checkpoints CP8.2, CP8.3]

The WG is committed to maintaining the quality and relevance of the Checklist. This will be done by ensuring that the test materials are consistent with any erratum and are updated as needed to reflect the current versions and status of the specification. The procedures describe above will be used to accept new contributions as well as to handle checkpoints that are in dispute.