High-level goal (why TestGL?)

Define principles & practices to support the creation of useful and usable conformance test-suites.

Principle #1: Users of the test suite must understand what is tested, how extensively it's tested, what is not tested

They need to know whether this test suite applies to them, the extent to which they can rely on it, and where they might need to focus additional testing efforts.

This implies:

• The scope and goals of the test suite are documented
• Coverage information is provided (it must be possible to map tests back to the spec)
  • Assertion lists are a good way to do this

Principle #2: Users of the test suite must understand how to interpret the test results

Implementation developers need to understand where their implementation is deficient, and what they can do to fix it.

This implies:

• Tests should report their results in a meaningful and consistent manner
• To the extent that this is possible, tests should report what went wrong (what they were expecting, and what happened), as an aid to debugging the problem

Principle #3: Test execution must be repeatable and reproducible

If it's not, I cannot rely on the results, and I cannot compare my results with anyone else's.

This implies:

• At a minimum:
  • components of a particular revision of test suite must be unambiguously identified
  • package them together, and publish with a version number
    • don't simply point people to a directory that contains a bunch of stuff, and is randomly updated
  • it must be possible to determine what tests must be executed for a particular implementation
    • filter tests based on features implemented
  • documentation must clearly specify how to execute tests, and how to interpret results
• Ideally:
  • provide a test-harness and supporting tools, libraries, framework
  • or provide sufficient metadata and documentation to allow a test harness to be constructed

Potential components of test suite

• Tests (may be "object", or may be "source" that must be processed into "object")
• Test materials metadata (must define - see previous draft) sufficient to allow (define operations/functions)
  • For example: filtering by supported feature/level of spec, module, etc.
• Test harness (optional)
• Supporting libraries, tools (optional)
• Documentation
• Conformance requirements

Principle #4: Test suites must evolve over time

• To meet needs of changing specs (revisions)
• To improve quality/coverage
• To fix bugs found during development, testing, or use of the test suite

This implies:

• Must plan for multiple releases of the test suite
• Each release must be versioned (people must know what version they're using)
• Must be able to accept andrespond to bug reports
  • Against individual tests, the docs, the harness, etc.
  • Possible responses to bug reports:
    • Patch existing test suite by
      • excluding broken tests from the test suite
      • creating and distributing alternate tests
      • updating docs, harness, framework, etc.
    • Release an updated version of the entire test suite
    • Re-releasing the entire test-suite, even if the changes are minor, might be the simplest and least confusing way to release updates

Operational guidelines

Treat test development like product development - it is (or should be) a formal engineering process.

For the highest-quality test suite:

• Resources must be effectively applied (you'll never have enough!)
• Understand your scope and goals: what do you want to test and how
  • What kind of tests to be developed
• What kind of metadata to be supplied along with tests
• Apply resources in the most effective manner
  • Focus on developing tests in areas where they will be most useful/effective
    • Where it's most likely that implementations will be non-conformant
    • Where the consequences of non-conformance would be greatest (eg, breaking interoperability)
  • Avoid duplication of resources
  • You can't just say to people "develop whatever tests you want" - must guide them/allocate areas where work is needed
• Development process must be managed
• Provide guidelines to test developers
• Submissions must be reviewed
• Issues must be tracked in a formal manner
• Test materials must be formally tested (by you, by your users)
  • You need a test plan for testing your test materials
  • You need a beta-test and feedback process
  • Bugs must be logged, tracked, and addressed