- From: Lofton Henderson <lofton@rockynet.com>
- Date: Wed, 02 Oct 2002 17:52:30 -0600
- To: www-qa-wg@w3.org
- Message-Id: <5.1.0.14.2.20021002173855.0367b9b0@rockynet.com>
(Forwarded to WG by Lofton, at Lynne's request, because of current W3C-NIST email problem...) ************************************ NIST has reviewed the TestGL document and provides the following comments. We realize that some of these comments may be moot, due to last week's special telcon. These comments were complied from all the NIST participants (Sandra, Mark, and Lynne) as well as the XML Technologies Conformance Testing Team (Mary Brady, John Tebbutt who is building Schema tests, Rick Rivello, who is building DOM tests, and Sandra, who builds XML tests.) General Comments 1. This document heavily relies on test assertions. The relationship between this document and SpecGL GL15, Include test assertions, should be highlighted. 2. The assumption is that you have good test assertions. 3. Many of the terms are confusing (see comments below for specifics). We need to make sure they are used consistently among our documents (e.g., Is scenarios and use cases used the same way here as in the SpecGL?) 4. Do we need to let people know that the guidelines don’t need to be don’t sequentially and in fact will probably be done in parallel and in a circular manner, adding and refining as you go 5. Many of the ckpts that require test assertions to be grouped, defined, extracted will be accomplished by tagging. Should we say something about this that we don’t necessarily expect physical lists of these things or do we? Specific Comments 1. Ckpt 1.1: Create a list of all the specifications used or referenced. a) Although this is a reasonable ckpt, in reality people don’t work this way rather it is often done mentally and complied as one proceeds in test development b) Where should this list exist? Is it already part of the specification as in the example (XQuery), in which case, what is required of the test developer to meet this ckpt? 2. Ckpt 1.2: Define the target set of specifications that are being tested How extensive does this set need to be? For example, DOM must use valid components of other specifications, so would it be necessary to list all those specifications? Is this target set limited to those specifications that are explicitly being tested rather than those specifications that get implicitly tested? 3. Ckpt 1.3: Extract test assertions from the target set of specifications. Note that the test assertions may not always be human readable or be what we traditionally think of as test assertions. For example, Schema test development for datatypes is able to go directly from the specification to automatically generating tests. The schema for the Schema spec identifies the datatype and max/min, using this XML, the tests are generated. Note, this works for datatypes which are static. A different approach may be necessary for developing the dynamic tests. 4. Ckpt 1.4 Group test assertions by levels, profiles, modules, etc a) Actually, test assertions should be grouped by what makes sense, e.g., grouped by interface. It would be helpful to explain the rationale/benefit for this ckpt e.g., grouping helps to easily display or to pull out sections of the test suite that a user wants run. b) The grouping most probably will be done via tags, since many of the test assertions will apply to multiple groups and can be applied to a particular group as needed. Should we say this? c) Many people may be unfamiliar with Degree of Variability so, need to link to its discussion. Or, can we remove this notion? Does it add to the understanding and/or implementation or verification of this checkpoint? 5. Ckpt 1.5: Define those test assertions that are part of conformance criteria These should have already been defined in SpecGL and had to be defined in order to accomplish ckpts 1.3 and 1.4. Suggest changing ‘Define’ to ‘Identifying’ or ‘catagorize’ since this ckpt’s objective is to categorize (group) the test assertions according to the conformance criteria. 6. Ckpt 1.7: Identify optional behaviors in the specification a) Suggest removing the sentence beginning with “But…”. Consider choosing the Java binding for DOM, as currently written, this ckpt implies you must do all of Java, when the spec says only need to do a part of Java. b) There are several cases in which you want to develop tests for optional behaviors: For specs in CR want tests for options to evaluate the options and implementer’s use of the options; DOM has optional modules, however once you pick a module there is required functionality. 7. Ckpts 1.8, 1.9, 1.11 a) These are all related and breaking it into separate ckpts has resulted in confusion as to the difference between them. Suggest combine them into 1 ckpt, as follows: Identify behavior: undefined, ambiguous, and contradictory. b) What is the difference between 1.8 and 1.9 does it matter if something is undefined explicitly vs by mistake? Note that tests are typically not written for explicitly undefined behaviors. It is true that identifying undefined behaviors that were left undefined by mistake is helpful feedback to the spec developers c) What is the difference between contradictory behaviors (1.11) and ambiguous behaviors (1.8)? If something is contradictory, isn’t it ambiguous? 8. Ckpt 1.13: List user scenarios for the specification. Suggest lowering the Priority. Are these the same scenarios that were developed for SpecGL? It can also help in picking the data values for the executable tests or for building the xml document that is ‘acted’ upon (e.g., by DOM or XSL-FO tests) 9.Ckpt 2.1 Define target areas for testing. a) You need to know this, prior to doing ckpt 1.2. Hence the suggestion to make it clear that these ckpts don’t need to be done sequentially. b) Didn’t this already do this with test assertions? c) The Schema example categorizes according to the spec structure however, in ckpt 1.4 we don’t mention this as a possible way to group Test Assertions. Is there a reason for having this promote ‘match the specification areas’ whereas ckpt 1.4 promotes the DoV 10. Ckpt 2.2 For each testing area, produce a set of sample testing scenarios What is meant by testing scenarios? How is this different from user scenarios (ckpt 1.13 and SpecGL scenarios) or are these the same? 11. Ckpt 3.1 For each test area identify applicable testing approach In reality, you use whatever works this may be a combination of approaches, dependent on the behavior you are testing. So, in general, there may be 1 major approach, but for a few behaviors the approach doesn’t work and you need to attack it with another approach. You could make this a new test area, but this can be confusing and add excess overhead to have additional groups for a new test area to accommodate an different test approach. For example, in VRML, we needed to test the lighting model differently than how we tested the rest of VRML (eg, test a box with color vs how color is applied) 12. Ckpt 3.2 Identify publicly available testing techniques…. What does ‘testing techniques’ mean? Want to ensure that it is understood that you don’t need to do an exhaustive study 13: Ckpt 4.1 Review available test framework automation and adopt existing… Thought this was what ckpt 3.2 was asking. What is the difference? 14. ckpt 4.2: Ensure the framework and automation are platform independent. Demostrate on 3 platforms. a) Is it clear what ‘platform’ means that it is the computer? Does it also include an operating system e.g, is a PC with windows, the same platform as a PC with Linux? b)This implies that automation is required. Some test suites don’t include automation (harness?) examples: XML test suite and Schema test suites. c) Why 3 platforms. 15. ckpt 4.3 Ensure the framework and automation are applicable to any product/content… Again, why 3 products/content. Also, why isn’t this Priority the same as ckpt 4.2? 16. ckpt 4.4 Ensure the framework makes it easy to add tests for any of the spec areas. What is ‘easy’ 17. ckpt 4.5 Ensure the ease of use for the test automation This is a judgment as to ‘easy of use’ and ‘easily used’. It is important to understand who the audience is for using the test automation and build and document the automation accordingly. What is important is to document how it can be used. Also recognize that if people can’t figure out how to run the tests, they won’t use them. 18. ckpt 4.10 Ensure the results verification is product independent… Is this addressing test results? Not always have 3 different products, especially if the tests are being developed early-on in parallel with the specification. We should make it clear that reporting is required (P1 designation) and is part of the automation - which will help to ensure consistency of reporting. Note that the original XML test suite, which didn’t have a harness, and thus didn’t have consistent reporting. 19. ckpt 4.13. Ensure the framework allows specification coverage to be measured This is a test matrix, right? Example is the XML test suite. 20. ckpt 5.2 Ensure results reports is platform independent. Demonstrate on 3 platforms How is this different from ckpt 4.10 21. ckpt 5.4 Ensure the ease of use for results reporting. Demonstrate that the results reporting has sorting and filtering capabilities. Why is this a P1? Why require sorting and filtering. 22. ckpt 5.8: demonstrate how the results reporting allows for history/storing analysis comments WGs shouldn’t need to store and track this. This is a vendor problem 23. ckpt 6.3 Start with atomic tests first, according to priorities defined in ck2.5. Believe that it should be according to priorities defined in ckpt 6.1. Adjust Priority level of this ckpt and ckpt 6.1 24. ckpt 6.5 Condust regular specification coverage analysis… Need to define coverage analysis. 25. ckpt 7.1 A WG must publicly encourage conformance testing among vendors. Organize at least one F2F.… You don’t need to have a F2F to accomplish this, especially at a P1 you can do this via a telcon. Also recommend doing some type of public announcement to encourage use and comments of the tests.
Received on Wednesday, 2 October 2002 19:51:07 UTC