Fwd: Minutes of June 14 Q/A Meeting

QAWG --

Today's (special) telcon topic is: Test Guidelines outline [1] review.

Below attached please find the draft minutes from Friday, where we started 
discussion of the Test Guidelines outline [1] in the afternoon.

-Lofton.

[1] http://www.w3.org/QA/WG/2002/06/qaframe-test-0612.html

>X-Sender: skall@mailserver.nist.gov
>X-Mailer: QUALCOMM Windows Eudora Version 5.0
>Date: Mon, 17 Jun 2002 17:43:21 -0400
>To: karl@w3.org
>From: Mark Skall <mark.skall@nist.gov>
>Subject: Minutes of June 14 Q/A Meeting
>Cc: lofton@rockynet.com, sandra Martinez <sandra.martinez@nist.gov>,
>         lsr@nist.gov
>X-RCPT-TO: <lofton@rockynet.com>
>
>Karl,
>
>Below are the minutes for the morning (Sandra is the scribe) and afternoon 
>(I'm the scribe).
>
>Please post these on the QAWG home page.
>
>Thanks.
>
>Mark
>
>
>
>
>QA Working Group Face-To-Face Meeting (AM)
>Montreal, Canada
>Friday, 14-June-2002
>--
>Scribe: Sandra I. Martinez
>
>Attendees:
>Karl Dubost (W3C, WG co-chair)
>Dominique Hazael-Massieux (W3C - Webmaster)
>Lofton Henderson (CGMO - WG co-chair)
>Sandra Martinez (NIST)
>Mark Skall (NIST)
>Olivier Thereaux (W3C - systems)
>Dimitris Dimitriadis (Ontologicon)
>Kirill Gavrylyuk (Microsoft)
>
>Regrets:
>(LR) Lynne Rosenthal (NIST - IG co-chair)
>
>
>Summary of New Action Items:
>AI-2002-06-14-1: Karl will contact Peter Fawcett to recruit him to help 
>with the Week in QA.
>AI-2002-06-14-2: Dimitris will propose how to develop a testing group 
>activity due August 1st.
>AI-2002-06-14-3: Lofton will prepare slides for project review.
>Previous Telcon Minutes: 
>http://lists.w3.org/Archives/Public/www-qa/2002Jun/0007.html
>
>
>Lofton reviewed yesterday's meeting.
>
>Karl will contact Peter Fawcett to recruit him to help with the Week in QA.
>
>Dimitris will propose how to develop a testing group activity due August 1st.
>
>On Thursday June 20, Lofton will present a project review to a W3C team 
>where he will discuss accomplishments, activities, issues, etc. One of the 
>issues he will present is whether the QA guidelines are to become 
>mandatory or not. The WG agreed that some degree of being mandatory should 
>be required when the documents reach a certain level.  It was also agreed 
>that the issue about being mandatory should be brought up at the next 
>chair's meeting in the fall.
>
>Each WG member will apply each of the three wg guidelines to a working 
>group, specification or test suite. The purpose is for each working group 
>member to gain experience in applying our guidelines to real example and 
>to improve the guidelines. While this is being done, each wg member, as a 
>courtesy, should communicate this effort to each WG chair.
>
>An extended discussion concerning issue 67 (What should be the scope of 
>the framework ex/tech parts?) took place. The consensus was that we need 
>case studies, as well as specific enumeration of techniques that satisfy 
>the checkpoints (a virtual working group). The discussion proceeded to 
>look at how to do this in a timely manner. It was agreed that the best way 
>to get this accomplished is for each wg member do a case study, then an 
>enumeration of techniques that satisfy the checkpoints, for one of our 
>documents (rather than case studies for three documents).
>
>Issue 68 was discussed. This issue pertains to whether the three ex/tech 
>parts should be separate standalone reports or the second part of each 
>guideline. No consensus was reached.
>
>Dimitris presented his proposal for Taggable Test Assertion. This proposal 
>discusses writing testable assertions in the spec itself using a detail 
>markup language (XMLspec.). This will allow the specification to be 
>machine processable.  Advantages of this approach include:
>1.      The elimination of ambiguities.
>
>2.      Faster development of tests.
>3.      The ability to identify discrepancies in the spec while developing 
>tests, those enabling test and spec development to proceed in parallel.
>
>If this proposal is adopted, another question concerns which version would 
>be normative  the XML or the HTML version. Right now the HTML version is 
>normative but if can show that the two versions are equivalent it should 
>not matter. If the XML version becomes normative, the test suite becomes 
>normative. It was recommended that the proposal should be presented in 
>small increments rather than as one monolithic proposal.
>
>
>
>
>QA Working Group Face-To-Face Meeting (PM)
>Montreal, Canada
>Friday, 14-June-2002
>--
>Scribe: Mark Skall
>
>Attendees:
>Karl Dubost (W3C, WG co-chair)
>Dominique Hazael-Massieux (W3C - Webmaster)
>Lofton Henderson (CGMO - WG co-chair)
>Sandra Martinez (NIST)
>Mark Skall (NIST)
>Olivier Thereaux (W3C - systems)
>Dimitris Dimitriadis (Ontologicon)
>Kirill Gavrylyuk (Microsoft)
>
>Regrets:
>(LR) Lynne Rosenthal (NIST - IG co-chair)
>
>
>Summary of New Action Items:
>AI-2002-06-14-4: Dimitris will provide boiler plate for test assertion 
>questionnaire by June 21.
>AI-2002-06-14-5: Dimitris will send e-mail to wg with the list of items 
>that came up from the thread on testable assertions by June 21.
>AI-2002-06-14-6: Lofton will arrange next telcom bridge for the special 
>meeting on Thursday.
>AI-2002-06-14-7: Karl will add a statement at beginning of glossary that 
>any comments/suggested changes to the glossary must be accompanied by 
>specific wording.
>Previous Telcon Minutes: 
>http://lists.w3.org/Archives/Public/www-qa/2002Jun/0007.html
>
>The wg continued the discussion about the Testable Assertion proposal. It 
>was agreed upon that a cost benefit analysis should be developed. This 
>will be helpful in convincing the wg chairs that the proposal will result 
>in real benefits to overall test and specification development effort.  A 
>simple questionnaire will be developed to survey the working group chairs 
>to determine what they are doing, why they are doing it and how well that 
>works.
>
>Kirill presented a proposed outlined for the Test Materials Guideline 
>document. The document consisted of seven guidelines: 1) Analyze the 
>specification(s); 2) Define areas for testing; 3) Choose the testing 
>methodology; 4) Provide the test automation and framework; 5) Provide the 
>results reporting framework; 6) Organize tests development; and 7) Conduct 
>testing. A general Working Group comment, not related to any specific 
>guideline had to do with adding checkpoint/guidelines on automatic 
>generation of tests. Specific comments/clarification for each guideline follow:
>
>Guideline 1
>Checkpoint 1.2: Clarification  testable assertion are identified for all 
>requirements plus non-required statements (i.e. shalls, shoulds, and mays)
>Checkpoint 1.3: The test assertions that are part of the conformance 
>criteria (i.e. the shalls) are determined in this checkpoint.
>Checkpoint 1.5: Change vague to intentional and unintentional ambiguities. 
>We also want to add a separate checkpoint for contradictory behavior.
>
>Guideline 2:
>Checkpoint 2.2: Change from priority 2 to priority 1.
>Add another checkpoint for combination of test (molecular test).
>A discussion ensued about when to stop testing. It was decided that 
>priorities could be used to control when to stop.
>
>The special telcom on June 20 will continue discussion on the test 
>guideline document.
>
>The next face to face in Tokyo will be a three-day meeting, October 8-10, 
>2002.
>
>Issue 65 (Level of detail in the QA glossary) was discussed. It was agreed 
>that some definitions should be short, but that other items might need to 
>be expanded. The glossary editors will modify some of the definitions.  It 
>was also agreed that any comments on the glossary must be accompanied by 
>specific suggested wording for any modification of the definitions to be 
>considered.  A statement to this effect should be added at the beginning 
>of the glossary document.
>
>Issue 49 (Global license for use on distribution of test materials) was 
>discussed and Kirill presented a sample license to be reviewed. Working 
>Group members will review the sample license for content.
>
>
>****************************************************************
>Mark Skall
>Chief, Software Diagnostics and Conformance Testing Division
>Information Technology Laboratory
>National Institute of Standards and Technology (NIST)
>100 Bureau Drive, Stop 8970
>Gaithersburg, MD 20899-8970
>
>Voice: 301-975-3262
>Fax:   301-590-9174
>Email: skall@nist.gov
>****************************************************************