TCDL Review (AI-20030618-9 completed)

All,

Here follows my review of the TCDL WG note 
(http://www.w3.org/QA/WG/2003/10/tcdl-20031013.html).

Outline of the review:

General remarks
	- Where applicable, review in the light of similar work having been 
done already (in my case, DOM)
	- General remarks on specifics introduced by using a particular markup 
for test case materials

Specific remarks follow the WG note by number, I will only for 
simplicity paste the relevant test where needed.

---

General remarks:
When we started the DOM TS group, we played with the idea of creating 
an XML language to not only do what TCDL is designed to do, but also 
represent the tests themselves. This would mean coming up with a 
pseudo-language to represent the DOM's functionality. Due to lack of 
time, we chose to go for the more straight forward and simple way of 
actually using DOM methods and attributes as the building blocks of the 
DOM TS markup (using the DOM specific names for methos and attributes 
as building blocks of the DOM TS Test markup), and then use XSLT to 
transform the test case descriptions into executable code (ECMA and 
Java, primarily, since those are the two "official" DOM bindings, but 
people also tried to write XSLT for C++ and Python). This allows for 
people to stay close to their specification and does not introduce 
another, perhaps unnecessary step between test representation and test 
suite running. It also allowed us to include some basie metadata into 
the test cases themselves.

The markup includes some metadata (creator, date, intended purpose 
which could be expressed and browsed, citation capabilities -- pointer 
to relevant part of the DOM specification --, and test name. These 
tests are stored in a CVS, each in a folder belonging to a structure 
representing a particular level (1, 2, 3). The way the test suites are 
built is by using a build tool (ANT) which lets you define tasks that 
can span file systems for input and serve that to an XML/XSLT process 
to generate output, group output and zip it, create TOC, specification 
coverage tables and documentation, just be prescribing what the build 
script ranges over. I note this as one easy way to create different 
test suites (that can reflect different versions, errata, second 
editions, or what have you), is to let the script range over particular 
files in order to group the right things together. Typically, a test 
for a level 1 will be there for the level 2 as well. This not always 
being the case, however, we let the build script clearly describe where 
a test belongs. This feature is adressed in TCDL as well, but with a 
more complex mechanism (using the version-add and version-drop 
elements/attributes) which allows for a detailed way to build a test 
suite. However, it is not clear how a test, having been written only 
for a particular version or range of versions, can be allowed to be 
part of, say, level 1, level 3, but not level 2 SE, if there is one. On 
that note, I think it is better to treat test cases as atoms and let 
some other mechanism sort out the grouping, using external parameters 
(external to the test metadata, I mean). If TCDL were this external 
mechanism only, it'd be fine.

Also, I find it fairly hard to convince people that have yet to 
implement XML Spec to use a particular markup for metadata, especially 
if most functions that TCDL intends to offer can be achieved by using, 
say, CVS/build script/HTML tables for reporting and citing. You all 
know that I am a friend of granular markup (my work in the DOM TS, 
editing/contributing to SpecGL and TestGL goes to show this) but the 
practical side is that it is really difficult to have people using 
particular markup. I hope to be proven wrong at some point.

The strong point of TCDL is as an interface between (any) test case 
markup and (any) test suite running and result framework. There it 
really makes sense to require that a WG uses TCDL at least to port 
results from the running part of the process to the result part. To be 
clear, what I mean is that it makes sense to let people use whatever 
test running mechanism they choose, any test case management system 
(which, by default, includes even the most basic kind of metadata) and 
any kind of results reporting (perhaps EARL), as long as you can gather 
information from one end and port it to another, in particular when 
wanting to include/exclude specific tests from test suites and generate 
output to try out your implementation.

However, any TCDL-like language should not be used to do what other 
things can do already (file systems, CVS, mailing lists, build scripts, 
documentation creation tools or what have you), since that just places 
an already existing burden in a different place. Its strong points 
should be pursued and implemented, but I would opt for letting things 
do what they are intended to do and add value elsewhere.

Specific remarks:
- In case a WG chooses to use TCDL for, say, test case versioning, what 
happens to the replication of information concerning, for example, test 
creation date, which may be present in the test case file system?

- Could descriptive strings be part of the test itself and be pointed 
to (an included in any test suite) by the framework (to create an easy 
to browse table) instead of having to give the descriptive string as 
part of the metadata surrounding the test? This way, you can have the 
same descriptive string for many test cases (by just including it in 
each instance), which TCDL does not allow, I think (except of the same 
TCDL item can be used to group many instances of similar tests.

- In case we allow for the actual test markup to be anything the WG 
chooses, TCDL could easily be used as a wrapper around tests. In that 
sense, containing all relevant metadata, I see it as part of the 
build/suite management system, which on the other hand I feel that WG's 
should be free to choose (if you can maintain atomic tests and build 
suites using other tools, it should be fine, as long as suite building, 
reporting and versioning issues are streamlined).

- 3.4, supported version: I think the model used in the WG note is 
fine, but one thing missing is that you need to manually denote, in 
each test, for what versions of a specification it is applicable. 
Another way to achieve the same result is to allow for an external 
building script to iterate over particular sets of tests, perhaps by 
flag (accepted/pending/xxx) or by just browsing whatever file system 
contains the tests (Level 1, Level 2.1, Level 2.2 but not level 2.3 and 
so forth). It is easier to let test descriptions be referenced than 
including relevant information in the test itself. For example, it is 
difficult for the same test to be part of the test suite for any level 
1, not 2, 3, but not 4 (I'd be surprised to see any such test, but you 
never know).

- 5.4, relationship to EARL: are the keywords "NotApplicable", 
"UnableToRun" etc. part of TCDL? If so, where are they included? At the 
test level?

In general, I conclude that the TCDL is well worth pursuing, as part of 
an (as yet non-existant) test running framework and reporting 
tool/solution. Already having paid the price of trying to achieve part 
of the TCDL goals in lost development time, my gut feeling is that it 
will simply be too complex for what it provides (even if it is a lot). 
Some of the functions, in particular managing sets of test cases to 
make up test suites, are fairly easy to implement using a simple build 
script (as the afore mentioned ANT tool does by letting you create 
particular tasks that are run by the build process before output). Some 
other functions could be captured by whatever test case management 
solution each WG decides to implement (DOM chose CVS and for versioning 
purposes [of the test cases] it works just fine; you can even include 
switches in the build script to discriminate between different versions 
of each test).

If we decide to allocate resources, I'd be more than happy to help, 
since TCDL adresses issues that are vital to the success of any test 
suite. David, please pass any future version by me, or even better, 
include me in the loop so that we can work together.

Excuses up front for any misunderstandings.

This closes my action item AI-20030618-9.

Best,

/Dimitris

Received on Friday, 31 October 2003 14:49:34 UTC