Re: [www-qa-wg] <none>

Some in-line responses to these points:

At 09:17 PM 12/1/2002 -0700, Lofton Henderson wrote:

>Here are some thoughts and opinions on the SpecGL topics for Monday...
>
>At 12:34 PM 11/29/02 -0500, lsr@nist.gov wrote:
>>[...]
>>IMPORTANT:  There are 2 aspects to consider here:
>>1. changes that need to be made to the CP to clarify the CP. (labeled For CP)
>>2. changes we need to make to the SpecGL so that it conforms to the SpecGL
>>(based on Mark’s review [1]).  (Labeled, For SpecGL)
>>
>>QUESTIONS and ISSUES:
>>
>>General (applies to many CPs):  When a CP doesn’t apply, should a
>>specification be required to indicate somewhere, that xxx doesn’t apply?


I believe, strongly, that it must.  If not, determining conformance to the 
SpecGL is almost impossible.  You can check the whole spec, but you're 
never sure if compliance to this one requirement is hidden somewhere (this 
is from personal experience).

>>This
>>could be done in the ICS. Should CPs include a discussion of what happens if
>>the CP doesn’t apply?
>
>This is a tricky question. We should look at those ckpts that seem 
>difficult to apply to a GL document (esp. SpecGL), and ask "why?" (i.e., 
>why is this CP difficult to apply to SpecGL?). Some possible answers to 
>the "why?" question:
>
>** we misunderstand the nature of GL specs;
>
>** or, we are not abstracting their properties correctly;
>
>** or, some ckpts are inherently N/A to something like SpecGL (in which 
>case, we might need a generalized "N/A" escape clause -- we may have 
>failed to anticipate and appreciate other N/A situations);
>
>** or, we have worded the ckpt incorrectly so that it only *appears* to be 
>N/A.
>
>
>>2.1 Identify all classes of products
>>What are the classes of products?  According to the Scope, the SpecGL applies
>>to Specification. However, we go on to distinguish between existing specs and
>>new specs.  Also Conformance Clause addresses WG specifications.
>>What is the class of products?  1 class= specification or 2 classes: new
>>specs and old specs.
>>Clarification:  In the scope we define specifications to be W3C Technical
>>supports.  Do Technical Reports include Recommendations and Guidelines or
>>just Recommendations?
>
>My interpretation.  Scope includes and is limited to new Technical 
>Reports.  The reference to "WG specifications" in the Conformance section 
>is carelessness (we fail the "consistent terminology" CP!).  From a W3C 
>Process perspective, there is nothing called "Guidelines".  In WAI, the 
>guidelines documents are Recommendations (or on the way to being Recs).
>
>There is one class:  new TRs.  The talk about old specifications is 
>informative discussion.  Or more precisely, "old TRs are out of scope", 
>although it may be useful to look at them from the perspective of SpecGL 
>in order to understand their conformance characteristics, strengths and 
>weaknesses (e.g., to fix problems in their next edition or version).
>
>>Proposal:
>>For SpecGL, specifically state in the Scope, the class of product that this
>>document targets is xxxx.
>
>I agree that we should be more careful to define class of product (CoP) 
>and category of SpecGL, in SpecGL.  I disagree with Mark's assertion that 
>we necessarily need to add "specification" to the enumerated list(s).  It 
>depends on how widely useful a "specification" CoP is.  Is SpecGL likely 
>to be the only (Rec, or Note, or whatever) that addresses the 
>"specification" CoP?  If so, then we should not add it to the enumerated 
>list.  We should use the "...else define your CoP" variation of the 
>requirement.


Your assertion that we shouldn't include specifications as a class of 
product argues that we shouldn't expect SpecGL to conform to itself.  If 
there is even one use of this class, it should be included. If we don't 
treat conforming specs as a class equal to the others, why should we expect 
specs to conform?


>>13.2 Distinguish normative and informative text. (David and Mark listed this)
>>SpecGL specifically identifies non-normative references, but otherwise it
>>isn’t clear.  Is it sufficient to only label informative items and by
>>default everything else is normative? Do we want editors to label all
>>sections, explain in Scope what is or isn’t normative?
>>Proposal: For CP, do we need to add anything here or cover this in ExTech?
>
>I think what we need is a better common understanding of what we mean by 
>normative and informative.


That's the whole point. We shouldn't need to have a common 
understanding.  SpecGL should be precise on this point so that we all have 
the exact same detailed understanding.

>  A fairly stringent view would be -- (almost) the only normative bits are 
> the test assertions.  I.e., in SpecGL, the bits in the "to fulfill" 
> sections (which are marked-up as class="TA", i.e., these are the test 
> assertions).
>
>I said "almost", because I guess stuff like the ch.3 Conformance section 
>are normative also.  But in the GL and the CP text, it is the test 
>assertions that are clearly normative.  Since these are somewhat terse and 
>abbreviated, they are often followed by supplemental clarifying 
>verbiage.  This is the tough stuff to decide -- is that verbiage 
>informative, or is it normative?
>
>>For SpecGL, in Scope add a general statement that examples and the
>>rationale are informative unless otherwise indicated and that the GL, CP,
>>and ‘to fulfill…’ are normative.
>
>IMO, this is the right approach.
>
>If that is agreed, then we need to sort out whether we consider the "gray 
>stuff" -- clarification of terse normative language and test requirements 
>-- to be normative or informative.
>
>
>>Is the ExTech a normative or informative reference?  (see also 10.3 below)
>>Proposal: For SpecGL, add it to the appropriate reference list?
>
>Extech is entirely informative.  OpsET (OpsGL Extech) is fairly explicit 
>about this -- no conformance requirements, all conformance requirements 
>are in OpsGL -- although it is not explicitly stated in OpsGL.


To conform to SpecGL, it is required that all associated references be 
categorized as normative or informative.  It needs to be explicit there.

>>10.3 Make normative reference to specifications on which the current
>>specification depends.
>>Does the SpecGL depend on the ExTech?  It does to satisfy CP 1.3 Provide
>>examples.  So, is ExTech normative or informative?
>
>Informative.
>
>
>>14.1 Provide test assertions (for SpecGL conformance to SpecGL)
>>Issue 99: Scope of definition of test assertion.
>>What are the test assertions for the SpecGL?  The CP or the ‘to
>>fulfill…’?  Is it ‘derived from the spec’s requirements? Can they all be
>>measures/tested? (e.t., 13.3). Test assertion defined as a statement of
>>behavior, action or condition that can be measured or tested.  It is
>>derived from the specification’s requirements.
>>Issue 99: Does the definition preclude expressing a test assertion in a
>>formal language without words and phrases rather than in sentences?  What
>>is definition in Glossary?
>>Is there a mapping between the spec and test assertion?
>>Proposal:  ?
>
>We have clearly stated in Issue List resolutions that the CP statement is 
>a "title" for the CP.  The normative requirements -- i.e., the test 
>assertions -- are contained within the body of the CP verbiage.  SpecGL 
>and OpsGL have isolated the test assertions into the "to fulfill" markup 
>sections, using RFC2119 terminology.  Those are the test assertions of 
>those two GL documents.


It doesn't matter what we say in an issue resolution.  SpecGL conformance 
must stand alone.  We are asking people to conform to SpecGL, not to our 
issue resolutions.   According to my reading of SpecGL, those are not test 
assertions.

>I agree that there are some document conventions and verbiage 
>clarification to be addressed here, but the parts (i.e., the TAs) are in 
>place and even marked up as such (class="TA").
>
>
>>12.1 Provide an Implementation Conformance Statement proforma
>>Is the Checklist for the SpecGL the ICS?
>>Proposal; use the Checklist as the ICS, make necessary adjustments.  Editor
>>(and Lofton?) action.
>
>The checklist is an ICS.  Although it is not labelled as 
>such.  Question.  Is an ICS not only required to be present, but also 
>required to be labelled as "ICS"?  I.e., the requirement is "...MUST have 
>an ICS" (SpecGL meets this).  Should it say instead, "...MUST have an ICS, 
>and SHOULD label it as an ICS"?
>
>>1.4  Provide an example for each formal description
>>What is a formal description? Definition is unclear.
>>Does SpecGL provide this?
>
>Sandra suggested a definition and it is now in CP 1.4.
>
>It should be added to section 6, "Definitions".
>
>
>>13. 3 Use consistent terminology
>>Can this be verified?  What exactly is required? Should this CP be removed
>>and left to Manual of Style and Susan L to worry about?
>
>No, because Style Manual conformance is not required for publication.
>
>Let's try to make it verifiable.  "...MUST NOT use different terms or 
>phrases for the same concept [ed.  w/o an accompanying specific reference 
>to the principal term or phrase?]"
>
>
>>GL8: Discretionary Items and
>>13.4 Provide a fast way to find conformance information (David and Sandra
>>revies)
>>Issues of navigation and discovery were under-represented in the
>>checkpoints.  For example, discretionary items are only detectable by a
>>thorough reading of the entire document.
>>Proposal: ?
>>
>>1.1 Define the scope of the specification
>>Clearly there is a scope section.  But much of the wording is vague and
>>imprecise, e.g., ‘it is not expected…’
>>Proposal: editor action to clean up
>>
>>CP 1.2 Provide User Scenarios
>>CP 1.3 Provide examples
>>Proposal (Karl): For CP change to Use Scenarios
>>For SpecGL: Include in ExTech, link is already provided.
>
>I doubt ET will have User Scenarios (or Use Scenarios).  This was an issue 
>a little while ago.  When I rewrote the Introduction, I had a placeholder 
>subsection for "User Scenarios", with the disclaimer that there was no 
>time to write it before 8-nov publication.  So we decided to drop it for 
>8-nov and complete it later.
>
>In other words, it is to-be-done.
>
>
>>2.3 Identify which of the categories of object are specified in the
>>document as a whole
>>Unclear what ‘types of objects’ are and what object SpecGL would be.  List
>>given in GL 2 are called ‘categories’.  Also need to define it, not just
>>give an example of what it is.
>>Proposal: For CP, change ‘types of objects’ to ‘categories’, also add
>>category of ‘guideline’ to the list.
>
>I don't think we need to necessarily add to the list.  It is acceptable 
>for SpecGL to define its own Category and CoP(s).  So let's decide if we 
>want to add a class-of-one to the lists, or take the latter (self-define) 
>approach.  I favor the latter.
>
>
>>3.1 Specify any universal requirements for minimum functionality
>>Unclear what is meant by this requirements.  For the SpecGL, is Level A
>>considered the minimum functionality?  Does a specification (in our case
>>SpecGL) need to explicitly say ‘Level A is the minimal requirement for all
>>specifications’ .
>
>I think that is sensible -- we intend that all specs should achieve at 
>least degree A conformance.
>
>
>>3.2 Identify strict conformance requirements
>>SpecGL doesn’t explicitly specify this, although it is implied that SpecGL
>>doesn’t do strict conformance (see conformance claim wording section 3.2, “
>>…meets at least all degree X conformance requirements’.
>>
>>9.1 Indicate if extensions are disallowed
>>9.2 Indicate if extensions are allowed
>>Is there a way to combine these?
>>Proposal: For CP, Lofton proposal [find email]
>>For SpecGL, need to add explicit words to address the itemized list.  Not
>>have use cases or project requirements, so can’t do last MUST. Editor action
>>
>>   9.3 Prevent extensions from contradicting the specification
>>Proposal: For SpecGL, Editor action to explicitly state.
>
>(...strict conformance, extensions ... this is tricky for SpecGL.  I don't 
>have a clear understanding of it yet.)
>
>
>>11.2 Provide specific wording of the claim
>>Make sure all MUST items are in included in Section 3.2.
>>Should the example at end of Section 3.2 move to ExTech
>
>No, I think it is useful here.  We do include short examples in SpecGL 
>where they help comprehension.  SpecET is for more extensive examples and 
>possible strategies.
>
>>For SpecGL conformance to SpecGL, not all the items are satisfied.
>>Proposal: For CP and For SpecGL, Editor action in Section 3.2, add version
>>and date to conformance claim. Also add them to the list.
>
>Btw, I disagree with Mark's comment that specific wording is not 
>provided.  Looking at the 3 bullets, the specific wording is what follows 
>the first COLON on each bullet.  Lack of specific wording is not why 
>SpecGL fail the checkpoint.  It fails because it omits some required bits, 
>which Mark pointed out -- it need more bullets (and wording), to 
>require:  identification of the subject of claim, version of SpecGL, and 
>date of claim.
>
>-Lofton.

****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
****************************************************************

Received on Monday, 2 December 2002 09:44:38 UTC