RE: Checkpoint 1.1 Table

Hi Kirill, Mark --

I suggest that we discuss Mark's earlier proposal for modification to the 
1.1 table, and Kirill's later proposal (below), at the telcon, as part of 
the on-going issue#55 processing.


-Lofton.

At 10:49 PM 3/12/02 -0800, you wrote:
>Hi, Mark, David,
>Here are some arguments why the current requirement makes sense:
>1) We want this Checkpoint to be Pri1 - essentially required for
>everybody. If we raise a requirement level, we have to drop the priority
>to 2, because the existing WGs like XSLT, XSD, XML and others do not
>have resources allocated initially to do regular tests reviews. I like
>the requirement of having some tests better then option of not having
>any test suite.
>
>2) The model of composing test suite without inside-WG review is
>actually viable. The way you operate in this case is:
>
>WG assembles a test collection from contributions and then ask vendors
>to publish their results against this collection. This generates open
>discussion why the results differs and vendors motivate submitters to
>update their tests. Eventually standard and the test collection align to
>each other. The only thing that is required from the WG in this case is
>to drive publishing tests and results and point out to submitters the
>discrepancy in the results.
>
>I agree that in the correct test result will not be available in timely
>manner, but from the benefit/workload perspective for most active
>standards this seem to be the golden mean.
>
>Re:test assertions
>- partially agree with you. I'd suggest we add what you suggested to the
>Level 2 and simply swap current Level 5 and Level 6. Note, I agree with
>Dave that complete set of test assertions is an advanced requirement at
>least for existing specs like XSLT or XSD - it will take another 2 years
>to produce. But the complete test suite is by definition complete only
>when it covers all assertions - I don't know of any complete test suite
>produced so far:)
>
>Re: use-cases.
>I think it is as basic notion as a "test" so should be in our Glossary.
>Spec development starts from identification of the use-cases - formal
>description of the user scenarios. The difference between "test suite"
>and "set of use-cases" is that the latter has no upper boundary, it is
>an origin for the spec, not a consequence. Hence I would not attempt to
>define the word "numerous".
>
>
>My suggestions:
>1) Add to level 2 what you suggested below
>2) Under testability of the specification, level 3, reference definition
>of the "use case".
>3) Leave the Chkpt unchanged.
>4) Swap Level 5 and Level 6.
>
>Thanks
>-----Original Message-----
>From: Mark Skall [mailto:mark.skall@nist.gov]
>Sent: Monday, March 11, 2002 8:46 AM
>To: www-qa-wg@w3.org
>Subject: Checkpoint 1.1 Table
>
>
>To all,
>
>As promised in Friday's telcon, I reviewed the table associated with
>Checkpoint 1.1.  With the current checkpoint, this is what we require: a
>WG
>commitment that a test suite (with no quality control) will exist prior
>to
>Recommendation and that the WG aims to have numerous normative use cases
>in
>the body of the Recommendation.
>
>These are the problems I see with the current checkpoint and table:
>
>1) We've asked the WG to commit to a test suite (in Level 2), but we
>don't
>ask for a commitment to review the test suite until level 4.  Since
>we're
>the Q/A group, I think any requirement to produce something (a test
>suite),
>with no provisions to ensure adequate quality, is not appropriate.
>
>2) Although we've asked for commitment to the existence of a test suite
>in
>level 2, we do not ask for test assertions (as am addendum) until level
>6.  Thus, we could have a test suite developed before the issuance of
>test
>assertions.
>
>3) Level 3 asks for the WG to aim to have numerous normative use cases
>in
>the Rec.  The word "numerous" is vague and unverifiable.  Additionally,
>I'm
>not sure what is meant by use cases at all, in this context.
>
>I propose the following solutions:
>
>1) Add to level 2, in the testability of the specification column,
>"Working
>Group commits to develop a set of test assertions, not necessarily
>complete, before beginning development of a test suite."
>
>2) Under testability of the specification, level 3, quantify "numerous
>normative use cases" (i.e., one use case per ???).  In addition, explain
>
>what is meant by "use cases", in this context.  If neither of the above
>can
>be done, I suggest removing this requirement.
>
>3) Change Checkpoint 1.1 to "plan to have at least a commitment to Q/A
>level four."  This way, we can guarantee that the test suite will be at
>least of minimal quality (it will have been reviewed and there will be a
>
>process in place for establishing and maintaining test case
>contributions).  Without review and a plan for test cases, all we've
>asked
>for in our level 2 requirement is something called a test suite, which
>could be completely garbage.  A bad test suite is much worse than no
>test
>suite at all.
>
>4) Add to level 5, in the testability of the specification column, "In
>addition to the commitment for the previous level, insists on a complete
>
>set of test assertions before completing test suite."
>
>In summary, I think the table and checkpoint need some work.  The above
>thoughts are my preliminary ideas, but there may be better ways to
>address
>the problems.
>
>Mark
>
>
>
>
>****************************************************************
>Mark Skall
>Chief, Software Diagnostics and Conformance Testing Division Information
>Technology Laboratory National Institute of Standards and Technology
>(NIST) 100 Bureau Drive, Stop 8970 Gaithersburg, MD 20899-8970
>
>Voice: 301-975-3262
>Fax:   301-590-9174
>Email: skall@nist.gov
>****************************************************************

Received on Wednesday, 13 March 2002 20:48:19 UTC