W3C

The QA Handbook

W3C Working Draft 10 May 2004

5. Acquiring test materials

Story: An external organization built a test suite for ABC 1.0. The ABC Working Group had no test suite, no effort in progress, and no resources to staff a from-scratch effort. The external organization had no resources to continue maintaining the test suite. With QA staff moderating, several months of sorting the details led to a win-win agreement to transfer the ABC 1.0 test suite to the ABC working group. The test suite got a secure repository within W3C, was published for public use, and was given adequate maintenance resources.

Principle: The Working Group can save valuable time and a ton of work by acquiring a test suite, but be ready to address and resolve many of the same issues as build-your-own scenarios.

The three main issues big-three problems are:

The IPR and staff issues are similar in concept to what the Working Group faces when building if builds build the test materials TM, but probably lesser in degree. A pre-transfer quality assessment might seem unique to the acquire option, but the actual steps will probably look similar to a test case contribution/review process in a build-your-own scenario (see QA Framework: Test Guidelines [QAF-TEST]).

Good Practice: Do a quality assessment of proposed candidate test materials before going any further.

Basic things that a good quality assessment might cover would for example include:

A more comprehensive list of things that an assessment process could cover might include:

QA Framework: Test Guidelines [QAF-TEST] deals with this topic in much more details detail, including (planned) templates and assessment aids.

Examples:

Related:

[TBD. There was a recent www-qa dialog about "testing a test suite". Might make an interesting reference.]

Good Practice: Ensure there are sufficient (enough?) adequate staff resources to support the transferred test materials.

A Test Materials ( test materials TM) manager is still needed, but total staff resources ought to be considerably less than build-your-own scenarios. With luck, the original test materials TM manager of the external test materials TM source might become a WG member after the test materials are transferred.

Good Practice: Sort out IPR issues with the external party that produced the test materials.

How can I do it?

This is a virtualization of an actual transfer scenario that QA helped to moderate. It could serve as a checklist of steps to consider for Working Group's taking the acquire route.

Legend: EG the external group or entity; QAWG the QA Working Group; test materials TM the test materials; WG the Working Group acquiring the test materials.

  1. Before the transfer, WG with the help of QAWG:
    • performs an assessment of the quality of candidate test materials TM (by WG, QAWG)
    • identifies and commits to a set of test-related deliverables from the candidate test materials TM. These could be: releases, appeal/challenge processes, maintenance plan, submission/review process, Web site, mail list, etc. (by WG)
    • identifies sufficient staffing, including at least a test materials TM manager. Recommendation: recruit the test materials TM manager from the EG (if one exists) to become a WG member after the transfer. (by WG)
    • makes the decision to seek/accept the transfer. (by WG)
    • (potentially) initiates Charter amendment (by WG, QAWG may consult), if the test materials TM acquisition doesn't fit within the current Charter.
  2. During the transfer:
    • EG and W3C reach agreement to transfer the test materials TM (by WG, QAWG, EG)
    • WG and EG perform the actual transfer of the test materials TM, WG creates an initial repository (by WG, EG)
  3. After transfer, initial test development/framework process setup:
    • WG appoints a test materials TM manager.
    • The test materials TM manager creates a QA Process Document for WG (by WG, test materials TM manager, QAWG may consult)
    • set up the test materials TM home page, a test materials TM issues mailing list (by WG, test materials TM manager)
    • determine the appropriate W3C IPR license (by WG, QAWG)
  4. First W3C public release of the new test materials TM:
    • make any needed enhancements prior to the first public release: fix known/reported errors, produce documentation (by WG), W3C license labelling, etc.
    • announce the first public release of test materials TM (by test materials TM moderator, Communications Group)
    • joint W3C/EG press release (by WG, QAWG, Communications Group, EG)
  5. After the first public release, the test materials TM enter the maintenance phase (see QA Framework: Test Guidelines [QAF-TEST].