Re: Outline for TestGL-Lite

I'm not sure I agree with the Principle 2: that  "test developers need to 
understand where their implementations are deficient and what they can do 
to fix it...To the extent that this is possible, tests should report what 
went wrong (what they were expecting, and what happened), as an aid to 
debugging the problem"

  For conformance tests, this is nice to have, but not necessary to 
determine conformance.  In the purest sense, a conformance test does not 
provide information beyond the pass/fail indication and what we are calling 
test metadata (description or purpose, test requirement and/or traceability 
back to the spec).  In fact, many test suites (within and external to W3C) 
do only this much.

I think providing extra information as to what went wrong is nice, but not 
necessary for conformance tests nor always possible (as you indicate).  So, 
I would categorize this as a good practice (i.e., recommended).


>
>Principle #2: Users of the test suite must understand how to interpret the 
>test results
>
>
>
>Implementation developers need to understand where their implementation is 
>deficient, and what they can do to fix it.
>
>This implies:
>    * Tests should report their results in a meaningful and consistent manner
>    * To the extent that this is possible, tests should report what went 
> wrong (what they were expecting, and what happened), as an aid to 
> debugging the problem

regards
Lynne