Re: Outline for TestGL-Lite

Maybe I shouldn't have lumped two "requirements" together. I agree that 
the 'diagnostic' information is merely "nice to have". The first half - 
"test developers need to understand where their implementations are 
deficient" - is, I think, an absolute requirement (albeit trivial, since 
it translates to "report test results clearly and unambiguously").

On the other hand, I'm trying to extend a little beyond pure 
conformance, and to address the "useful and usable" aspect of the TestGL 
charter...

Lynne Rosenthal wrote:

> I'm not sure I agree with the Principle 2: that  "test developers need 
> to understand where their implementations are deficient and what they 
> can do to fix it...To the extent that this is possible, tests should 
> report what went wrong (what they were expecting, and what happened), 
> as an aid to debugging the problem"
>
>  For conformance tests, this is nice to have, but not necessary to 
> determine conformance.  In the purest sense, a conformance test does 
> not provide information beyond the pass/fail indication and what we 
> are calling test metadata (description or purpose, test requirement 
> and/or traceability back to the spec).  In fact, many test suites 
> (within and external to W3C) do only this much. 
>
> I think providing extra information as to what went wrong is nice, but 
> not necessary for conformance tests nor always possible (as you 
> indicate).  So, I would categorize this as a good practice (i.e., 
> recommended).

Received on Wednesday, 3 March 2004 11:46:49 UTC