W3C home > Mailing lists > Public > www-qa-wg@w3.org > March 2004

Re: Outline for TestGL-Lite

From: Patrick Curran <Patrick.Curran@Sun.COM>
Date: Wed, 03 Mar 2004 17:46:33 +0100
To: Lynne Rosenthal <lynne.rosenthal@nist.gov>
Cc: QAWG <www-qa-wg@w3.org>
Message-id: <40460BE9.7080700@sun.com>

Maybe I shouldn't have lumped two "requirements" together. I agree that 
the 'diagnostic' information is merely "nice to have". The first half - 
"test developers need to understand where their implementations are 
deficient" - is, I think, an absolute requirement (albeit trivial, since 
it translates to "report test results clearly and unambiguously").

On the other hand, I'm trying to extend a little beyond pure 
conformance, and to address the "useful and usable" aspect of the TestGL 
charter...

Lynne Rosenthal wrote:

> I'm not sure I agree with the Principle 2: that  "test developers need 
> to understand where their implementations are deficient and what they 
> can do to fix it...To the extent that this is possible, tests should 
> report what went wrong (what they were expecting, and what happened), 
> as an aid to debugging the problem"
>
>  For conformance tests, this is nice to have, but not necessary to 
> determine conformance.  In the purest sense, a conformance test does 
> not provide information beyond the pass/fail indication and what we 
> are calling test metadata (description or purpose, test requirement 
> and/or traceability back to the spec).  In fact, many test suites 
> (within and external to W3C) do only this much. 
>
> I think providing extra information as to what went wrong is nice, but 
> not necessary for conformance tests nor always possible (as you 
> indicate).  So, I would categorize this as a good practice (i.e., 
> recommended).
Received on Wednesday, 3 March 2004 11:46:49 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:15 GMT