RE: Checkpoint 1.1 Table

Hi, Mark, David,
Here are some arguments why the current requirement makes sense:
1) We want this Checkpoint to be Pri1 - essentially required for
everybody. If we raise a requirement level, we have to drop the priority
to 2, because the existing WGs like XSLT, XSD, XML and others do not
have resources allocated initially to do regular tests reviews. I like
the requirement of having some tests better then option of not having
any test suite. 

2) The model of composing test suite without inside-WG review is
actually viable. The way you operate in this case is:

WG assembles a test collection from contributions and then ask vendors
to publish their results against this collection. This generates open
discussion why the results differs and vendors motivate submitters to
update their tests. Eventually standard and the test collection align to
each other. The only thing that is required from the WG in this case is
to drive publishing tests and results and point out to submitters the
discrepancy in the results. 

I agree that in the correct test result will not be available in timely
manner, but from the benefit/workload perspective for most active
standards this seem to be the golden mean.

Re:test assertions 
- partially agree with you. I'd suggest we add what you suggested to the
Level 2 and simply swap current Level 5 and Level 6. Note, I agree with
Dave that complete set of test assertions is an advanced requirement at
least for existing specs like XSLT or XSD - it will take another 2 years
to produce. But the complete test suite is by definition complete only
when it covers all assertions - I don't know of any complete test suite
produced so far:)

Re: use-cases. 
I think it is as basic notion as a "test" so should be in our Glossary.
Spec development starts from identification of the use-cases - formal
description of the user scenarios. The difference between "test suite"
and "set of use-cases" is that the latter has no upper boundary, it is
an origin for the spec, not a consequence. Hence I would not attempt to
define the word "numerous".   


My suggestions:
1) Add to level 2 what you suggested below
2) Under testability of the specification, level 3, reference definition
of the "use case".
3) Leave the Chkpt unchanged. 
4) Swap Level 5 and Level 6.

Thanks
-----Original Message-----
From: Mark Skall [mailto:mark.skall@nist.gov] 
Sent: Monday, March 11, 2002 8:46 AM
To: www-qa-wg@w3.org
Subject: Checkpoint 1.1 Table


To all,

As promised in Friday's telcon, I reviewed the table associated with 
Checkpoint 1.1.  With the current checkpoint, this is what we require: a
WG 
commitment that a test suite (with no quality control) will exist prior
to 
Recommendation and that the WG aims to have numerous normative use cases
in 
the body of the Recommendation.

These are the problems I see with the current checkpoint and table:

1) We've asked the WG to commit to a test suite (in Level 2), but we
don't 
ask for a commitment to review the test suite until level 4.  Since
we're 
the Q/A group, I think any requirement to produce something (a test
suite), 
with no provisions to ensure adequate quality, is not appropriate.

2) Although we've asked for commitment to the existence of a test suite
in 
level 2, we do not ask for test assertions (as am addendum) until level 
6.  Thus, we could have a test suite developed before the issuance of
test 
assertions.

3) Level 3 asks for the WG to aim to have numerous normative use cases
in 
the Rec.  The word "numerous" is vague and unverifiable.  Additionally,
I'm 
not sure what is meant by use cases at all, in this context.

I propose the following solutions:

1) Add to level 2, in the testability of the specification column,
"Working 
Group commits to develop a set of test assertions, not necessarily 
complete, before beginning development of a test suite."

2) Under testability of the specification, level 3, quantify "numerous 
normative use cases" (i.e., one use case per ???).  In addition, explain

what is meant by "use cases", in this context.  If neither of the above
can 
be done, I suggest removing this requirement.

3) Change Checkpoint 1.1 to "plan to have at least a commitment to Q/A 
level four."  This way, we can guarantee that the test suite will be at 
least of minimal quality (it will have been reviewed and there will be a

process in place for establishing and maintaining test case 
contributions).  Without review and a plan for test cases, all we've
asked 
for in our level 2 requirement is something called a test suite, which 
could be completely garbage.  A bad test suite is much worse than no
test 
suite at all.

4) Add to level 5, in the testability of the specification column, "In 
addition to the commitment for the previous level, insists on a complete

set of test assertions before completing test suite."

In summary, I think the table and checkpoint need some work.  The above 
thoughts are my preliminary ideas, but there may be better ways to
address 
the problems.

Mark




****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division Information
Technology Laboratory National Institute of Standards and Technology
(NIST) 100 Bureau Drive, Stop 8970 Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
****************************************************************

Received on Wednesday, 13 March 2002 01:50:00 UTC