QAWG - QA Process document

Good evening folks,

I've finished polishing off our comments on the Template and I've 
finished a first rough draft of the QA Process document of us.
I still think it needs work and there are a few sections that I don't 
think are truly relevant to our particular 'test materials' that can 
probably be edited out still but over all it's not too bad.

I've attached the Process Document Draft to this mail.
The comments on the Template are below.
Peter

First off, a few general comments.
After much more discussion, our group of 4 came to some consensus that 
yes,
what we "Test" is specification, WG organization and process and the
way in which test materials are produced. The way we do this is by using
the checklist documents to verify each checkpoint. Once this is done one
can then state a level of compliance to our guidelines. The closest
analogy I can think of is the WAI check list when one applies 
checkpoints to
web pages to determine the level of WAI compliance.
So... our (the WG) current test materials that we have produced so far 
are
the checkpoint documents that are associated with each Guideline 
document.

That being said, trying to apply this template to us (our working group)
was a bit more tricky that using this same template for a more 
conventual
type of test materials. It also lead to our having to edit a bit more 
of the
process document than just the fill in the blanks.

We found when filling this out, that in cases where it didn't make much 
sense,
if you replaced our Checklist with say, "XML Test Suite" things often 
were
much clearer. Again this is an artifact due to our some what odd test 
materials.

(A third source of some confusion here as well was the some what self
referential aspect of this in terms of our filling it out for our 
group. )

Now then on to the comments.

- First of, "test material" is referred to in multiple ways in the 
document
	including but not limited to "test material" and "test material type".
	We could not determine a difference in concept from context nor could 
we
	come up with an example that needed both.

- The language in Rec 1 seems odd. Especially the second sentence.

	"The Checklist should be developed in a public framework.
	The development of this framework itself is a central area of
	interest. The QAWG welcomes the participation of interested
	parties in developing the Checklist."

- The third sentence in Rec 3 also seems a bit odd

- We found a number of uses of "test materials" and
"test suite" that did not use the "fill-in" markup.

- DocumentFramework should be Document Framework

- In Test materials contribution section in the a/b choice rather than
saying tests or series..  say contributions.

- In Test Materials contribution section there is a missing 
<span></span>
around address.
	above]</span> mailing list: [address].</p>

- We found the first item in the list under Test Materials Contribution 
to be
some what redundant. At least in our case as any submissions would 
already
go to the list as it is.

   	The test or series of tests, get submitted to the framework. 
Submitter
     should also send a notification to the mailing list that will be 
set up
     for the Checklist to indicate that he/she has submitted a test to 
the
     class="fill-this-in">[test material name] framework.

- There is a typo in section Receipt and review of test contributions
"according tot he" should be "according to the"

- In Receipt and Review section list item 1
  it currently says:
  "since it tests a feature that is not in the specification"

  should it say something more like:
  "since it tests a recommendation that is not in the specification"
  as 'feature' is not a universally applicable term.

- In Receipt and Review section list item 3:
	This item makes the assumption that the test cases will be written
	in some xml based language.
	How ever can we make this assumption? What about languages like
	rdf that don't use an xml syntax?
- this same section (item 3) also has a hanging ']</span>' on it.

- In Receipt and Review section list item 4:
	"<li><em>Scenarios</em> that underlay the particular test layout and 
its
     intended scope.</li>"
     We found the use of "test layout" to be out of place in the 
document.
     it is the only place that this term is used. We believe that it 
could
     just as easily use language that's more standard to the document.

- In Receipt and Review section list item 5:
	The point of this checkpoint is valid, in that if the submitted test 
doesn't follow
	the Development Guide it should be rejected, how ever the notion of 
the Development
	Guide is introduced here (in this document). We feel that it should be 
listed
	earlier in the document as well. perhaps in the requirements, that the 
test
	materials be developed in accordance with the Development Guide.

- Typo in last sentence of Receipt and Review section:
	is: "publication arereurned to"
	should be: "publication are returned to"
	
- In Test status and review procedures Section:
	In the second paragraph it reads:
	"the status of the test is changed to reflect the fact that its
	validity has been disputed"
	
	Earlier we say that "status is changed to "accepted"
	we should use the same verbiage here like: "state changed to 
"disputed""
	
- In Test status and review procedures Section:
	The last sentence of the second paragraph has very odd wording.

- In Test status and review procedures Section:
	second paragraph uses term "Task Force" in terms of the maintainers of 
the test
	materials. This is another concept that is first introduced in this 
context.
	This too would work better if the concept of appointing a "Test Task 
Force" was
	done at an earlier step.

- In Test status and review procedures Section in the first item of the 
list.
	The label "stable" may not be the best word for case 1 as it doesn't
	really reflect what is meant. "Error Free" or "Valid" would work 
better.
	
- In Test status and review procedures Section in the third item of the 
list.
	It reads: "questioned the correctness, consistency, clarity, etc"
     we think that correctness does not belong.

- In Test status and review procedures Section in the second and third 
item of the list.
     "A consensus exists in the community"
     What community? The w3c, the 'whole internet' the WG (as this is 
specifically
     talking about the task force.)
     We are also fairly uncomfortable with the notion that the task 
force can be
     over ridden by the (not clearly defined) community.
     Either the taskforce agrees or disagrees and the case accepted or 
not.
     Unless "community" means WG in which case this might make sense...

- In "Section Status of the test suite according to the above"
	We feel that the wording of this section heading could be improved 
quite a lot.
	We assume this means that if you conform to all of the above. or 
you've done/are
	doing all of the above.

- In "Section Status of the test suite according to the above" second 
paragraph
	It reads:
		It is proposed that the W3C WG representative act as moderator and 
controller
		for incoming tests. The moderator is chosen by the [WG name] WG. All 
tests
		should be kept for archive purposes, whether they get published or 
not.
	much of this is repeated from above. The new information is that a 
moderator
	should be chosen. This is another case where it may be beneficial to 
introduce
	the concept of a Moderator earlier in the document.

This is already covered above to some degree. But with moderator.

- Both the Documentation and "See above" sections do not have 
references back to
     a GL document.