- From: Dimitris Dimitriadis <dimitris@ontologicon.com>
- Date: Fri, 31 Oct 2003 21:46:27 +0200
- To: www-qa-wg@w3.org
- Cc: david_marston@us.ibm.com
All, Here follows my review of the TCDL WG note (http://www.w3.org/QA/WG/2003/10/tcdl-20031013.html). Outline of the review: General remarks - Where applicable, review in the light of similar work having been done already (in my case, DOM) - General remarks on specifics introduced by using a particular markup for test case materials Specific remarks follow the WG note by number, I will only for simplicity paste the relevant test where needed. --- General remarks: When we started the DOM TS group, we played with the idea of creating an XML language to not only do what TCDL is designed to do, but also represent the tests themselves. This would mean coming up with a pseudo-language to represent the DOM's functionality. Due to lack of time, we chose to go for the more straight forward and simple way of actually using DOM methods and attributes as the building blocks of the DOM TS markup (using the DOM specific names for methos and attributes as building blocks of the DOM TS Test markup), and then use XSLT to transform the test case descriptions into executable code (ECMA and Java, primarily, since those are the two "official" DOM bindings, but people also tried to write XSLT for C++ and Python). This allows for people to stay close to their specification and does not introduce another, perhaps unnecessary step between test representation and test suite running. It also allowed us to include some basie metadata into the test cases themselves. The markup includes some metadata (creator, date, intended purpose which could be expressed and browsed, citation capabilities -- pointer to relevant part of the DOM specification --, and test name. These tests are stored in a CVS, each in a folder belonging to a structure representing a particular level (1, 2, 3). The way the test suites are built is by using a build tool (ANT) which lets you define tasks that can span file systems for input and serve that to an XML/XSLT process to generate output, group output and zip it, create TOC, specification coverage tables and documentation, just be prescribing what the build script ranges over. I note this as one easy way to create different test suites (that can reflect different versions, errata, second editions, or what have you), is to let the script range over particular files in order to group the right things together. Typically, a test for a level 1 will be there for the level 2 as well. This not always being the case, however, we let the build script clearly describe where a test belongs. This feature is adressed in TCDL as well, but with a more complex mechanism (using the version-add and version-drop elements/attributes) which allows for a detailed way to build a test suite. However, it is not clear how a test, having been written only for a particular version or range of versions, can be allowed to be part of, say, level 1, level 3, but not level 2 SE, if there is one. On that note, I think it is better to treat test cases as atoms and let some other mechanism sort out the grouping, using external parameters (external to the test metadata, I mean). If TCDL were this external mechanism only, it'd be fine. Also, I find it fairly hard to convince people that have yet to implement XML Spec to use a particular markup for metadata, especially if most functions that TCDL intends to offer can be achieved by using, say, CVS/build script/HTML tables for reporting and citing. You all know that I am a friend of granular markup (my work in the DOM TS, editing/contributing to SpecGL and TestGL goes to show this) but the practical side is that it is really difficult to have people using particular markup. I hope to be proven wrong at some point. The strong point of TCDL is as an interface between (any) test case markup and (any) test suite running and result framework. There it really makes sense to require that a WG uses TCDL at least to port results from the running part of the process to the result part. To be clear, what I mean is that it makes sense to let people use whatever test running mechanism they choose, any test case management system (which, by default, includes even the most basic kind of metadata) and any kind of results reporting (perhaps EARL), as long as you can gather information from one end and port it to another, in particular when wanting to include/exclude specific tests from test suites and generate output to try out your implementation. However, any TCDL-like language should not be used to do what other things can do already (file systems, CVS, mailing lists, build scripts, documentation creation tools or what have you), since that just places an already existing burden in a different place. Its strong points should be pursued and implemented, but I would opt for letting things do what they are intended to do and add value elsewhere. Specific remarks: - In case a WG chooses to use TCDL for, say, test case versioning, what happens to the replication of information concerning, for example, test creation date, which may be present in the test case file system? - Could descriptive strings be part of the test itself and be pointed to (an included in any test suite) by the framework (to create an easy to browse table) instead of having to give the descriptive string as part of the metadata surrounding the test? This way, you can have the same descriptive string for many test cases (by just including it in each instance), which TCDL does not allow, I think (except of the same TCDL item can be used to group many instances of similar tests. - In case we allow for the actual test markup to be anything the WG chooses, TCDL could easily be used as a wrapper around tests. In that sense, containing all relevant metadata, I see it as part of the build/suite management system, which on the other hand I feel that WG's should be free to choose (if you can maintain atomic tests and build suites using other tools, it should be fine, as long as suite building, reporting and versioning issues are streamlined). - 3.4, supported version: I think the model used in the WG note is fine, but one thing missing is that you need to manually denote, in each test, for what versions of a specification it is applicable. Another way to achieve the same result is to allow for an external building script to iterate over particular sets of tests, perhaps by flag (accepted/pending/xxx) or by just browsing whatever file system contains the tests (Level 1, Level 2.1, Level 2.2 but not level 2.3 and so forth). It is easier to let test descriptions be referenced than including relevant information in the test itself. For example, it is difficult for the same test to be part of the test suite for any level 1, not 2, 3, but not 4 (I'd be surprised to see any such test, but you never know). - 5.4, relationship to EARL: are the keywords "NotApplicable", "UnableToRun" etc. part of TCDL? If so, where are they included? At the test level? In general, I conclude that the TCDL is well worth pursuing, as part of an (as yet non-existant) test running framework and reporting tool/solution. Already having paid the price of trying to achieve part of the TCDL goals in lost development time, my gut feeling is that it will simply be too complex for what it provides (even if it is a lot). Some of the functions, in particular managing sets of test cases to make up test suites, are fairly easy to implement using a simple build script (as the afore mentioned ANT tool does by letting you create particular tasks that are run by the build process before output). Some other functions could be captured by whatever test case management solution each WG decides to implement (DOM chose CVS and for versioning purposes [of the test cases] it works just fine; you can even include switches in the build script to discriminate between different versions of each test). If we decide to allocate resources, I'd be more than happy to help, since TCDL adresses issues that are vital to the success of any test suite. David, please pass any future version by me, or even better, include me in the loop so that we can work together. Excuses up front for any misunderstandings. This closes my action item AI-20030618-9. Best, /Dimitris
Received on Friday, 31 October 2003 14:49:34 UTC