Eliminating overlap between OpsGL and TestGL

Dimitris and I met last night, and discussed the overlap between OpsGL 
and TestGL. We decided that at a minimum the following checkpoints 
should be dropped from OpsGL since we will be covering them adequately 
in TestGL:

3.2 Support spec versioning/errata in QA deliverables
5.2 Ensure test materials are usable for intended purpose
6.3 Include conformance verification disclaimer in test materials
8.1 Specify update procedure to track new spec versions/errata
8.2 Identify procedure for test validity appeals

I was thinking things over later, and am now inclining to the position 
that even more material should be transferred from Ops to Tech. Here's why:

Ops should focus specifically (and only) on WG-organizational issues 
(resources, communications, W3C procedural issues, etc.) "Operational 
guidelines' that address the process of developing test materials should 
be incorporated into a non-normative section in TestGL (similarly, the 
process of developing specs should be covered in SpecGL). In 
anticipation of taking this approach, here's a brief outline of what the 
"ops guidelines for test development" might cover:

Treat test development like product development - it is (should be) a 
formal process
For the highest-quality test suite:
  Resources must be effectively applied (you'll never have enough!)
    Understand your scope and goals: what do you want to test and how
    What kind of tests to be developed
    What kind of metadata to be supplied along with tests
    Apply resources in the most effective manner
      Focus on developing tests in areas where they will be most 
useful/effective
        Where it's most likely that implementations will be non-conformant
        Where the consequences of non-conformance would be greatest (eg, 
breaking interoperability)
      Avoid duplication of resources
      You can't just say to people "develop whatever tests you want" - 
must guide them/allocate areas
  Development process must be managed
  Provide guidelines to test developers
  Submissions must be reviewed
  Issues must be tracked in a formal manner
  Test materials must be formally tested (by you, by your users)
    You need a test plan for testing your test materials
    You need a beta-test and feedback process
  Bugs must be logged, tracked, and addressed

Incorporating this material into TestGL would allow us to delete the 
following additional checkpoints from OpsGL:

2.1 Address where and how conformance test materials will be produced
5.1 Define a framework for test materials development
5.3 Define a contribution process
5.5 Define review procedures for submitted test materials

Received on Wednesday, 3 March 2004 03:23:59 UTC