Re: new draft "Framework: Introduction"

Daniel,

Thanks for your careful reading and comments...

At 02:32 PM 1/25/2002 +0100, Daniel Dardailler wrote:

>My comments on the Intro doc.

My responses in-line.



>First, a spell check is needed, including on words like principal
>instead of principles.

Fixed.  (A complete spell check will be done before publication.)


> >     4. "QA Framework: Technical Guidelines" -- detailed technical
> >        guidelines for test materials, i.e., for building test suites (TS)
> >        and test tools themselves
>
>I'd like to see another name for this one.
>
>Technical Guidelines is going to be easily confused with the
>Techniques documents for Checkpoint/Guidelines that we're going to use
>in the Process/Op and Spec guidelines.

I agree with this, and was going to propose something like "Test Materials 
Guidelines".  (But see below.)


>In fact, I'm not even sure we're going to use the same prioritized
>checkpoint approach for this fourth piece, it's probably going to be a
>resource oriented document, so I propose we call it "Tools Resources".
>

I'll open a new issue about this (#48).  It is not clear cut, as there 
definitely are testable assertions that we can (and probably should) make 
about test materials.  E.g., "Every test case shall be traceable to one or 
more statements in the specification.  [P1]."

>In even more fact, I wonder if this should be counted as part of the
>framework, see more below.
>
> >    The individual conformance test materials efforts of the Working
> >    Groups have in several cases been inspired and on target, in others
> >    have been perfunctory or rudimentary
>
>My "shakespeare english alarm" went on for the word perfunctory :-)
>(i.e. if I need to look in the dictionary for a word, it probably
>means half of the audience will have to)

Okay, I'll rework this (I think Lynne has suggested new language here, also).


> >    Nor do quality
> >    practices comprise a mandate imposed on the Working Groups from
> >    without, notwithstanding that higher expectations are evolving for the
> >    Working Group QA deliverables and for the quality of the products of
> >    the WGs in general.
>
>same alarm, but not on isolate vocabulary, just on sentence you have
>to read three times to understand

Okay, will fix.

>
> >    More specific goals include:
> >      * to encourage the employment of quality practices in the
> >        development of the specifications from their inception through
> >        their deployment;
> >      * to foster a common look-and-feel for test suites, validation
> >        tools, harnesses, and results reporting;
> >      * to encourage development and use of a common, shared set of tools
> >        and methods for building test materials.
>
>I'd put the second point last.

Done.


> > 2. QA resources
> >
> >    A variety of resources are provided by the QA Activity, including:
> >      * common QA Glossary [59][QA-GLOSSARY];
> >      * QA Taxonomy description [60][TAXONOMY];
> >      * "The Matrix" [61][MATRIX], which is a definitive survey of
> >        existing QA practice;
> >      * Notes (white papers) on QA-related topics;
> >      * this QA Framework document family;
> >      * the Issues List of the QA WG [62][ISSUES-LIST];
> >      * expertise and consultancy;
> >      * technical assets.
>
>I'd put those two first
>    * "The Matrix" [61][MATRIX], which is a definitive survey of
>         existing QA practice;
>    * this QA Framework document family;

Done.


>I wonder in which way
>    * technical assets.
>
> >   2.2 Technical assets
> >
> >    While the initial resources that the QA Working Group offers to other
> >    WGs focuses heavily on the Framework documents and available
> >    consulting expertise, it is intended that ongoing development within
> >    QA Activity will result in a collection of commonly useful technical
> >    assets. Some anticipated offerings include:
> >      * XML-based test case description languages (TCDL) that can be
> >        applied or adapted to diverse test suite projects.
> >      * Scripts and transformation stylesheets to generate commonly needed
> >        TS materials, harnesses, and documents from TCDLs.
> >      * Grammars and methods for granular construction of specifications,
> >        as well as tools and methods for management of testable assertions
> >        and even (in some cases) automatic generation of tests.
> >      * Templates for setting up project TS process documents, ../Test Web
> >        sites, and other conformance project logistics.
> >      * Standardized and routine procedures (including forms and scripts)
> >        for WGs to manage the process of (external) test case submission,
> >        review, and integration.
> >      * Results reporting languages (e.g., EARL), plus methodologies and
> >        specific tools for working with them.
>
>
>is different from the planned 4th framework piece on tools ?

Personal opinion.  There are in fact some Gd-ck conformance requirements 
that we might want to place on "conforming test materials" (see above).  I 
have opened an issue (#48) about this, for telcon discussion.


>
> >   3.2 Document orientation & structure
> >
> >    The Framework documents will utilize the style of guidelines and
> >    verifiable checkpoints,
>
>I'd say "structuring guidelines and verifiable checkpoints, so people
>don't think guidelines are also verifiables.

Done.


> >      * and, for each of the Framework's guideline documents, having an
> >        associated "Examples & Techniques" document.
>
>"accompanying techniques" is the usual wording in WAI

I changed "associated" to "accompanying".  (Is that what you meant, or 
replace the longer phrase?)

>
> >      * "QA Framework: Technical Guidelines"
> >           + "Technical Techniques and Examples"
>
>"Technical Techniques" sounds really weird.

Completely agree.  If we keep these in the Frm family as Gd-Ck, I'd say 
"Test Materials Guidelines" and "...Examples & Techniques".

>One more reason I think to consider this part as a simple resource
>listing maintained off the Framework track.

Issue #48.

>
> >    This document family, which will be comprised of two documents, one
> >    with principals and guidelines, the other with examples and
>
>principles

Fixed.


> >    to existing QA work, serves as a more detailed set of guidelines as
> >    far as principal and technical questions are concerned. It should
>
>principle

Fixed.


> >     3.5.5 Technical Guidelines
> >
> >    These two documents deal exclusively with questions related to
> >      * Tools used to produce conformance test suites
> >      * Technologies used in doing so (RDF, XSLT, DTD/Schema etc)
> >      * Building or using harnesses for running the TS and generating
> >        relevant results
> >      * Principles and guidelines for general test suite design, test case
> >        content, reporting etc.
> >      * Principles and guidelines for issue tracking systems in TS
> >        production, matrices of existing, expected and wanted tests and so
> >        forth
> >      * Generation of more than one language binding, where applicable
> >      * Using assertions to produce relevant tests that take into account
> >        the particulars of the specific technology tested
>
>As I said, I think this one is very similar to the technical assets
>set of resources we're going to maintain.

Personal opinion.  I agree that's part of it.  But I wonder, if we get turn 
this into a "Test Materials Resource Guide", where do we put stuff like 
good-practice conformance requirements on test materials?


>e.g. WAI has guidelines specs and a Resource directory at
>    http://www.w3.org/WAI/Resources/
>
> >     4.1.1 For the overall Framework
> >
> >    QA is properly an intergral part of the efforts of each Working Group.
> >    The QA WG charter succinctly summarizes the point that has been made
> >    from the earliest W3C conformance papers (@@link to tbl/dd paper in
> >    member space?@@):
>
>I think we should just use the charter pointer, not the member private
>one.

Done.


> >    As presently scoped, there will be seven documents in the Framework
>
>let's not forget to change that if we decide to drop the 4th set of
>docs (tools + technique for tools, which doesn't look relevant btw
>since tools are already techniques kind of things)

Right.

Thanks,
-Lofton.

Received on Sunday, 27 January 2002 22:01:58 UTC