- From: Lofton Henderson <lofton@rockynet.com>
- Date: Mon, 10 Mar 2003 10:27:35 -0700
- To: Sandra Martinez <sandra.martinez@nist.gov>
- Cc: www-qa-wg@w3.org
[Attn David and Patrick...] At 10:22 AM 3/10/03 -0500, Sandra Martinez wrote: >Included are the draft minutes for Thursday morning. Noticed that there >are two >due dates missing in the action list. > >[...] >Summary of New Action Items: > >A-2003-03-07-1: Action to Peter to send comments to the form for OpsGL. >By Tuesday, March 10. >A-2003-03-07-2: Action to Lynne to include Ian Jacob’s comments into the >form for SpecGL. By Tuesday, March 10. >A-2002-03-07-3: Action to Patrick to review to QAWG charter. By end of >next week >A-2002-03-07-4: Action to Lofton Updating the QA PD template and message >to chair list March 20 >A-2002-03-07-5: Action to David to draft resolution to issue 23. David, would you propose a date for this please? >A-2002-03-07-6: Action to Lynne to write checkpoint to reflect resolution >of issue 107 march 20 >A-2002-03-07-7: Action to Patrick to circulate a set of suggested list of >attributes that will lead to collapsing the checkpoints in GL 1. Patrick, would you propose a date for this please? (Is it done already?) >A-2003-03-07-8: Action to Peter to combine GL 2 and 3. By Friday, March 14. >A-2003-03-07-9: Action to Patrick to restructure of the guideline 4. For Olivier's benefit, below shows that the date is "two weeks from Monday" (April 24). All for now, -Lofton. >Peter initiated the meeting providing a brief summary of the results from >the break out team’s review of the Process document using the process >document template. The activity resulted in a set of comments that are >going to be incorporate in the OpsGL comments form. Patrick took this action. >Lynne will incorporate Ian Jacobs’s comments into the SpecGL comments form. > >As a result of this review, Lofton recommended that a review of the >charter should also be done. Patrick took the action of performing the review. > >TestGL issues resolution > >Issue 13: Is inter-standard or multi-standard interoperability within the >QA scope? Issue raised in email thread (DM, 2001-11-07, 2nd pgph). > MS: There are no requirements for interoperability. We need > specific checkpoints. >LR: The requirements for interoperability are being addressed in other >specifications. If they are represented in those specifications they are >going to be tested. > > LF: This is an old issue, just noticed the e-mail that was generated. > > DM: The intention of this issue was related to the pipeline. > > MS: There is a problem with the way we are capturing the issues. >LR: This type of information should not be included in the TestGl at this >point. We need to keep this document simple. > >KD Agree. The TestGl needs to be simple and this will not prevent writing >a separate document to address clarification items. > >PC Also agree to keep the document simple. Put some words in the document >about extensibility/optionality. Propose to address this in the SpecGl, >not in TestGl. > >LH : Propose to cut and paste clarification into the body of the >issue. Also propose to go with Lynne’s proposal. >The issue is closed. > >Issue 79: Should Test Guidelines address how to write a good test suite? > > LH: A similar issue on SpecGL. >LR: This deals with the comment I raised about providing an overview of >the test development process, addressing general concepts and the test >development process. >PC: Are you referring to the question of what is conformance testing? >LH: This kind of information should not be part of this document. The >intent of this document is not to be a tutorial. >MS: It should be part of the scope. >KD: Recommend modeling the introduction after the WCAG introduction. A >primer should make it easier to read the specification, not understand >(explain) the specification. >DM: QA framework introduction could be used. >LR: Close the issue. The scope statement should include general >clarification for the document. >Issue closed. >Issue 107: Differentiate between test materials for CR and those for Recs. >LR: Explained her concern on TestGl not addressing a test suite >development based on the state of the specification. Writing a test suite >for CR might have a different objective than writing a tests for a >recommendation. For example: for CR the objective may be to test the >specification, whereas a test suite for a Recommendation targets >interoperability of the implementations. The TestGl is not allowing >flexibility for CR. >DM: Verbiage should be included in the document to cover these >areas. The test suite development is a constant process of checking the >specification, one thing that may occur, is that a test case identifies a >problem in the specification. This could be an example if identifying >problems in a CR. >SM: The process of developing test cases for a test suite should not >change; the working group is responsible for identifying the areas to test >based on a particular state in the specification development process. >LR: I agree, but the way this document is written it does not allow for >CR. Something should be included in the introduction or scope. I am just >not happy with the structure as it stands right now. >LH: A checkpoint will be appropriate to address this issue. >PC: It is a matter of coverage and completeness. Coverage is barely >defined in the TestGl. >LR: It is up to the working group to determine the coverage. I am not >particular on how is done, usually test suites for CR are smaller and more >focused. >DM: WG delivers only one test suite. Sounds like you are talking about >different test suites. Recommend adding a checkpoint. >LR: A WG may have multiple test suites for different parts of a >specification or for different applications of the specification. The >introduction or scope should also include something about the fact that a >working group might have multiple test suites for a particular specification. >LH Propose to accept Lynne’s proposal of adding a checkpoint to cover >this issue. The components of what needs to be address should included >in the checkpoint >Issue Closed. Lynne will write a checkpoint to for resolution to this issue. > Issue 23: Should tests of MAY/SHOULD assertions be part of a > conformance suite? >This issue was discussed at the f2f March meeting, after more discussion >it was decided that Mark Skall will raise the issue in SpecGL to nail down >the circumstances and David will to try to draft a resolution. >Issue closed. > >Outreach report >David Marston presented the results of the XQuery and XSL outreach. There >were two areas that generated discussion; reporting results and test >assertions. It was recommended to have an example of a disclaimer or a >boiler plate to capture the results. In regards to the test assertions a >discussion on completeness was generated. Tagging was believed to raise >concerns, some words are more important than others. David added that if >this is an issue for some people it could be something we need to recheck. >The concern was centered on the side of the test assertions that ties to >the specification, no much was discussed in the area of the test >assertions leading to test cases. Other than this, the group was basically >asking clarifying questions. >PC: It is a complex process to identify assertions, it is a major task. >DM: Complexity of expression and recursion can complicate things even more. >Lynne’s comments on TestGL. >Comment 1: Prior to the Guideline section, provide an overview of the >test development process, addressing general concepts and the deliverable >path, i.e., spec test assertions test cases reporting maintenance. If >possible the guidelines should follow this progression. Also, describe the >key aspects of a test suite: traceability, verdict criteria, >self-explanatory, valid, short/atomic. >This comment was discussed during the issues section (79) it was agreed >that the scope statement should include general clarification. >Comments 2: The importance and need for traceability back to the spec must >be clearly conveyed. >The comment was accepted by the group. >Comment 3: Make clear that there is no order to satisfying the CPs, as >long as at the end of the day, they are satisfied. >This comment is making reference to the order of satisfying the >checkpoint. Lynne added that it is when the test suite is finished, that >all the checkpoints need to be satisfied. She further recommended that we >should make that statement up front. The group agreed. > >Comment 4: Although we can’t assume that the OpsGL and SpecGL were >followed, but if they were, some of the CPs may already be satisfied we >should provide a table cross-referencing where this occurs. >Lynne explained that we can not assume that a checkpoint was already done. >The editors should include a note to make a table for cross-reference. >Given the status of document a note should be included indicating that it >was already done. >LH : This table is not trustworthy until LC. >Comment 5: Since we advocate developing test materials for CR, we should >explain that test materials for CR may serve a different purpose and have >different coverage than test materials for Rec and this is O.K. > Already discussed as part of issue 107. >Comment 6: Can we incorporate some of the ideas from other WGs SVG and CSS >have good test documentation, explaining not only how to build tests, but some >of the test suite principles. > Lofton recommended there be a list of possible discrete > deliverables in TTF. The group agreed. >Comment 7: Although the formatting changes necessary for the GL and CPs >will help, try to keep it simple. > Lynne added that she had a difficult time with the order of the > guideline. We should adopt the logical process from the tester. Organize > it better. >PF: This is what GL 1 does, probably needs to be made more explicit. >Changing the wording of the guideline will help. >Comment 8: Suggest a separate Guideline for Test Results and >Reporting. It should also mention something about EARL. >LR: All this stuff got merged at the end. It should stand out, there are >also too many check points. >DM: What kind of action you think is expected from the results? >LH: Providing a result management tool, then include an example in >Ex/Tech. Also, the guideline should not mention EARL. >LR: There is more to say at a higher level related to the sorting and >reporting capabilities in ck 5.2. A checkpoint should suggest reporting of >result, something to make reporting consistent across implementations. >KD: More of describing what to do and how is organized. >LR: Also have a concern with the word framework in the title of guideline >5, it seems to be focusing on recording a framework. >Comment 9: CP 1.1 Identify the target set of specifications being >tested. How extensive does this set need to be? For example, DOM must use >valid components of other specifications, so would it be necessary to list >all those specifications? Is this target set limited to those specifications >that are explicitly being tested rather than those specs that are utilized? > >Everyone agreed that the language need to be worked out and the checkpoint >need to be formatted the same way the SpecGL and OpsGL was done. The whole >guideline needs to be restructured. >Comment 10: CP1.5, 1.6, 1.7, 1.8, 1.9: Related to discretionary choices. >These CPs are related and breaking them into separate CPs has resulted in >confusion as to the difference between them What is the difference between >“defined ambiguously” (cp1.7) and “contradictory behaviors” {1.9)? Suggest >combining, as: Identify behavior: undefined, ambiguous, and contradictory. > >Patrick will circulate set suggested list of attributes that will lead to >combining the checkpoints. Lofton expressed concern about the priorities, >but David commented that after combining the checkpoint will become a >priority one. Patrick added that the way this is going to be presented is >not imposing is providing choices. >Comment 11: CP 2.1 Document the structure for the test suite and GL3 >Document the testing methodology. What is the difference between these? I >think there is a difference, but my simple mind is having trouble making >sense of all this. > > Patrick agreed that the distinction is not clear and suggested combining > GL-2 and 3 and speaking more in term of the language. If not combined > they should be reversed in order. >Peter took the action to combine the Guidelines. >Comment 12: CP 3.2 Identify publicly available testing techniques. What >does ‘testing techniques’ mean? How is this different from test automation >and framework? How much of a search for these techniques needs to be done? >Comment is moot due to previous action. >Comment 13: CP 4.1 List available test frameworks and applicable >automation and justify why new frameworks are needed….. >a) Define these terms and their scope. How is this different from 3.2? >b) Why must a justification be given who is the justification for? This >may be adding extra work on the WG with minimal if any benefit. > >Lynne added that the default is not to provide automated frameworks and >that we are being unrealistic. Patrick suggested that all the automation >checkpoints need to be collapsed into one checkpoint; he added that if we >are going to do automation we need to provide good guidelines. Lynne also >suggested using the ideas from the VML presentation to include test case >management. >The group agreed to restructure GL 4 to include automation and test case >management. >Comment 13- 17 >These comments are going to be covered by the restructure of the guideline >4, mapping the process from beginning to end by looking at the structure. >Patrick took this action. Action due two weeks from Monday. >Comment 18: CP 5.2 Ensure the ease of use for results reporting. >Demonstrate results reporting has sorting and filtering. Why is this P1? >Although nice to have, why is sorting and filtering required? > >Comment already discussed. > > > >At 07:25 AM 3/10/2003 -0700, you wrote: >>Hi, >> >>Thanks for sending this, Sandra. >> >>I noticed that it did not get relayed by the www-qa-wg@w3.org list >>manager, and it is not in the archive. I suspect that the reason is that >>your name looks a little different than normal >>(sandra.martinez@nist.gov), and the list manager didn't recognize you. >> >>Could you please resend? Also, if you can convert to HTML, that would be >>better. Olivier can't or doesn't want to handle .DOC. I think Lynne has >>a resource for doing this (which is why I copied her). >> >>Thanks, >>-Lofton. >> >> >> >>>X-Authentication-Warning: calendar.nist.gov: nobody set sender to >>>martines@nist.gov using -f >>>Date: Sun, 9 Mar 2003 20:39:52 -0500 >>>From: martines@nist.gov >>>To: Lofton Henderson <lofton@rockynet.com> >>>Cc: www-qa-wg@w3.org >>>Subject: Re: raw draft minutes >>>User-Agent: Internet Messaging Program (IMP) 2.3.7-cvs >>>X-RCPT-TO: <lofton@rockynet.com> >>> >>>Included is the draft minutes for Thursday morning. Noticed that there >>>are two >>>due dates missing in the action list. >>> >>> >>>Sandra >>> >>>Quoting Lofton Henderson <lofton@rockynet.com>: >>> >>> > >>> > It would be useful (to me, at least) to have access to raw draft minutes >>> > from Boston (4 half days). Could you minute takers please send them >>> to the >>> > >>> > WG list? >>> > >>> > Thanks, >>> > -Lofton. >>> > >>> > >>> > >> > >Sandra I. Martinez >National Institute of Standards and Technology >100 Bureau Drive, Stop 8970, >Gaithersburg, Md. 20899 > >(301) 975-3579 >sandra.martinez@nist.gov > >
Received on Monday, 10 March 2003 15:17:04 UTC