Re: TestGL: survey questions

Le lun 24/11/2003 à 09:06, Vanitha Venkatraman a écrit :
> Please review before the meeting.

A few comments inline to make it easier during the meeting. Thanks for
starting this.

> Introduction
> The TestGL document defines a set of common guidelines for conformance
> test materials for W3C specifications. It is one in a family of 
> Framework documents of the Quality Assurance (QA) Activity, which
> includes the other existing or in-progress specifications QA
> Framework: Introduction,QA Framework: Operational Guidelines, and QA
> Framework: Specification Guidelines. 

> The TestGL:Survey Questions is a survey document that has a list of
> questions to be answered by Working Groups, Spec Authors and is part
>  of an outreach program to create "QA Awareness" amongst the
> development community. 

I would avoid speaking of a "TestGL survey", but rather of "Tests
practices survey", or something along that line. The core of the survey
is not really on TestGL, and it might rebutes people thinking they
cannot answer if they haven't read testGL.
Otherwise, I would put the 2 preceding paragraphs the other way aroud,
that is for presenting the survey, and then giving a bit more context on
why the WG is doing this survey.

A bit more context on who should answer it, in what circumstances would
be helpful too.

> TestGL Survey Questions
> 1. Is there a scope, goal and intended purpose for the testsuite ? 
> 2. Do you associate your tests with a specification ? 

1 & 2 seem to assume that the responders have read TestGL and are aware
of its guidelines; I would not assume that; maybe asking more contextual
questions would be better? (what WG are you working in? What Test
material development have you participated to?) And then asking what is
the purpose of the TS (interop/conformance), and the specification
associated with it.

> 3. What approach do you take while developing the tests ? 

Sounds good; maybe adding some guidance on the type of answer with a few
examples?

> 4. Could there be metadata associated with the tests ? 

What kind of metadata do you associate with your test cases? [+ a few
examples]

> 5. Have you identified a process for managing test materials ? If so,
automated or manual ? 

Is there a process...

> 6. Are you planning any test coverage ? 

Does your test suite development have a planned coverage of the
specification?

> 7. Are you planning to track test issues if they arise ? 

Do you have a process to track issues on test cases? If so, what?

> 8. How do you plan to excute the tests ? 

The question is not clear to me...

> 9. Do you have a results reporting mechanism ? If so, automated or
> manual ?
> 10. Do all the tests report results uniformally ? 
> 11. Does your test reporting provide diagnostic information ? 

Good (a bit more explanation on "uniformity", "diagnostic" might help)

> 12. Does you test system support filtering of test results ? 

What kind of filtering (if any) are available on the test results? [+
examples]

> 13. Do you provide documentation with the testsuite ? 
> 14. Do you have a review process for test materials before they are
> released ? 
> 15. How do you deliver the test materials - are they in one package ?
> 16. Do you have a customer feedback process ? 

Good (maybe replace "deliver" by "release" or "publish", and "customer"
by "user").

Thanks again!

Dom
-- 
Dominique Hazaël-Massieux - http://www.w3.org/People/Dom/
W3C/ERCIM
mailto:dom@w3.org