NIST comments on TestGL

(Forwarded to WG by Lofton, at Lynne's request, because of current W3C-NIST 
email problem...)

************************************
NIST has reviewed the TestGL document and provides the following 
comments.  We realize that some of these comments may be moot, due to last 
week's special telcon.  These comments were complied from all the NIST 
participants (Sandra, Mark, and Lynne) as well as the XML Technologies 
Conformance Testing Team (Mary Brady, John Tebbutt who is building Schema 
tests, Rick Rivello, who is building DOM tests, and Sandra, who builds XML 
tests.)

General Comments
1.      This document heavily relies on test assertions.  The relationship 
between this document and SpecGL GL15, Include test assertions, should be 
highlighted.
2.      The assumption is that you have good test assertions.
3.      Many of the terms are confusing (see comments below for 
specifics).  We need to make sure they are used consistently among our 
documents (e.g., Is scenarios and use cases used the same way here as in 
the SpecGL?)
4.      Do we need to let people know that the guidelines don’t need to be 
don’t sequentially  and in fact will probably be done in parallel and in a 
circular manner, adding and refining as you go
5.      Many of the ckpts that require test assertions to be grouped, 
defined, extracted  will be accomplished by tagging.  Should we say 
something about this  that we don’t necessarily expect physical lists of 
these things or do we?

Specific Comments
1.      Ckpt 1.1:  Create a list of all the specifications used or referenced.
a) Although this is a reasonable ckpt, in reality people don’t work this 
way  rather it is often done mentally and complied as one proceeds in test 
development

b) Where should this list exist?  Is it already part of the specification 
as in the example (XQuery), in which case, what is required of the test 
developer to meet this ckpt?
2.      Ckpt 1.2: Define the target set of specifications that are being 
tested
How extensive does this set need to be?  For example, DOM must use valid 
components of other specifications, so would it be necessary to list all 
those specifications?  Is this target set limited to those specifications 
that are explicitly being tested rather than those specifications that get 
implicitly tested?
3.      Ckpt 1.3: Extract test assertions from the target set of 
specifications.
Note that the test assertions may not always be human readable or be what 
we traditionally think of as test assertions.  For example, Schema test 
development for datatypes is able to go directly from the specification to 
automatically generating tests.  The schema for the Schema spec identifies 
the datatype and max/min, using this XML, the tests are generated.  Note, 
this works for datatypes which are static.  A different approach may be 
necessary for developing the dynamic tests.
4.      Ckpt 1.4 Group test assertions by levels, profiles, modules, etc
a) Actually, test assertions should be grouped by what makes sense, e.g., 
grouped by interface.  It would be helpful to explain the rationale/benefit 
for this ckpt  e.g., grouping helps to easily display or to pull out 
sections of the test suite that a user wants run.

b) The grouping most probably will be done via tags, since many of the test 
assertions will apply to multiple groups and can be applied to a particular 
group as needed.  Should we say this?

c) Many people may be unfamiliar with Degree of Variability  so, need to 
link to its discussion.  Or, can we remove this notion?  Does it add to the 
understanding and/or implementation or verification of this checkpoint?
5.      Ckpt 1.5:  Define those test assertions that are part of 
conformance criteria
These should have already been defined in SpecGL and had to be defined in 
order to accomplish ckpts 1.3 and 1.4.  Suggest changing ‘Define’ to 
‘Identifying’ or ‘catagorize’ since this ckpt’s objective is to categorize 
(group) the test assertions according to the conformance criteria.
6.      Ckpt 1.7: Identify optional behaviors in the specification
a) Suggest removing the sentence beginning with “But…”. Consider choosing 
the Java binding for DOM, as currently written, this ckpt implies you must 
do all of Java, when the spec says only need to do a part of Java.

b) There are several cases in which you want to develop tests for optional 
behaviors:
For specs in CR  want tests for options to evaluate the options and 
implementer’s use of the options; DOM has optional modules, however once 
you pick a module there is required functionality.
7.      Ckpts 1.8, 1.9, 1.11
a) These are all related and breaking it into separate ckpts has resulted 
in confusion as to the difference between them.  Suggest combine them into 
1 ckpt, as follows:  Identify behavior: undefined, ambiguous, and 
contradictory.

b) What is the difference between 1.8 and 1.9  does it matter if something 
is undefined explicitly vs by mistake?  Note that tests are typically not 
written for explicitly undefined behaviors.  It is true that identifying 
undefined behaviors that were left undefined by mistake is helpful feedback 
to the spec developers

c) What is the difference between contradictory behaviors (1.11) and 
ambiguous behaviors (1.8)?  If something is contradictory, isn’t it 
ambiguous?
8.      Ckpt 1.13:  List user scenarios for the specification.
Suggest lowering the Priority.  Are these the same scenarios that were 
developed for SpecGL?   It can also help in picking the data values for the 
executable tests or for building the xml document that is ‘acted’ upon 
(e.g., by DOM or XSL-FO tests)

9.Ckpt 2.1 Define target areas for testing.
a) You need to know this, prior to doing ckpt 1.2.  Hence the suggestion to 
make it clear that these ckpts don’t need to be done sequentially.

b) Didn’t this already do this with test assertions?

c) The Schema example categorizes according to the spec structure  however, 
in ckpt 1.4 we don’t mention this as a possible way to group Test 
Assertions.  Is there a reason for having this promote ‘match the 
specification areas’ whereas ckpt 1.4 promotes the DoV
10.     Ckpt 2.2 For each testing area, produce a set of sample testing 
scenarios
What is meant by testing scenarios? How is this different from user 
scenarios (ckpt 1.13 and SpecGL scenarios) or are these the same?
11.     Ckpt 3.1 For each test area identify applicable testing approach
In reality, you use whatever works  this may be a combination of 
approaches, dependent on the behavior you are testing. So, in general, 
there may be 1 major approach, but for a few behaviors the approach doesn’t 
work and you need to attack it with another approach.  You could make this 
a new test area, but this can be confusing and add excess overhead to have 
additional groups for a new test area to accommodate an different test 
approach.  For example, in VRML, we needed to test the lighting model 
differently than how we tested the rest of VRML (eg, test a box with color 
vs how color is applied)
12.     Ckpt 3.2 Identify publicly available testing techniques….
What does ‘testing techniques’ mean? Want to ensure that it is understood 
that you don’t need to do an exhaustive study

13: Ckpt 4.1 Review available test framework automation and adopt existing…
Thought this was what ckpt 3.2 was asking.  What is the difference?
14.     ckpt 4.2:  Ensure the framework and automation are platform 
independent.  Demostrate on 3 platforms.
a) Is it clear what ‘platform’ means  that it is the computer?  Does it 
also include an operating system  e.g, is a PC with windows, the same 
platform as a PC with Linux?

b)This implies that automation is required.  Some test suites don’t include 
automation (harness?)  examples: XML test suite and Schema test suites.

c) Why 3 platforms.
15.     ckpt 4.3 Ensure the framework and automation are applicable to any 
product/content…
Again, why 3 products/content.  Also, why isn’t this Priority the same as 
ckpt 4.2?
16.     ckpt 4.4 Ensure the framework makes it easy to add tests for any of 
the spec areas.
What is ‘easy’
17.     ckpt 4.5 Ensure the ease of use for the test automation
This is a judgment as to ‘easy of use’ and ‘easily used’.  It is important 
to understand who the audience is for using the test automation and build 
and document the automation accordingly. What is important is to document 
how it can be used.  Also recognize that if people can’t figure out how to 
run the tests, they won’t use them.
18.     ckpt 4.10 Ensure the results verification is product independent…
Is this addressing test results?  Not always have 3 different products, 
especially if the tests are being developed early-on in parallel with the 
specification.  We should make it clear that reporting is required (P1 
designation) and is part of the automation  - which will help to ensure 
consistency of reporting.  Note that the original XML test suite, which 
didn’t have a harness, and thus didn’t have consistent reporting.
19.     ckpt 4.13.  Ensure the framework allows specification coverage to 
be measured
This is a test matrix, right?  Example is the XML test suite.
20.     ckpt 5.2 Ensure results reports is platform 
independent.  Demonstrate on 3 platforms
How is this different from ckpt 4.10
21.     ckpt 5.4 Ensure the ease of use for results reporting.  Demonstrate 
that the results reporting has sorting and filtering capabilities.
Why is this a P1?  Why require sorting and filtering.
22.     ckpt 5.8: demonstrate how the results reporting allows for 
history/storing analysis comments
WGs shouldn’t need to store and track this.  This is a vendor problem
23.     ckpt 6.3 Start with atomic tests first, according to priorities 
defined in ck2.5.
Believe that it should be according to priorities defined in ckpt 
6.1.  Adjust Priority level of this ckpt and ckpt 6.1
24.     ckpt 6.5  Condust regular specification coverage analysis…
Need to define coverage analysis.
25.     ckpt 7.1 A WG must publicly encourage conformance testing among 
vendors.  Organize at least one F2F.…
You don’t need to have a F2F to accomplish this, especially at a P1  you 
can do this via a telcon.  Also recommend doing some type of public 
announcement to encourage use and comments of the tests.