Re: Test Suite Principles

Brad,

Thanks for the proposal. Below you will find a section with general 
comments and then comments on each principle, inlined.

--
General comments
--
There is already a mailing list for questions regarding the DOM Test 
Suites (www-dom-ts@w3.org). Also, there are a number of resources 
explaining the architecture and rationale of the DOM Test Suites at 
http://www.w3.org. The DOM TS was launched in March 2001.

The DOM TS (Document Object Model Test Suites) is a publically 
developed, DOM WG endorsed framework that aims at testing DOM 
conformance using simple frameowrks for running tests. These tests are 
expressed in a language-neutral XML format (since DOM is an API and 
should not be biased toward any particular language binding), and two 
bindings (ECMA and Java, the same bindings as you find in the DOM 
specifications) are generated using XSLT stylesheets. Finally, there is 
a schema (in both XML Schema and DTD form) generated from the 
specifications themselves used to validate the test descriptions.

On a more general note, most of the issues you touch on are relevant to 
each Test Suite produced in the W3C. Therefore, I'd suggest that you 
email directly to the W3C QA IG list, since that WG (of which I am a 
member and lead editor) deals with these exact issues.

--
Replies to your suggested principles (inlined below).
--

Again, thanks for your input.

Kind regards,

/Dimitris Dimitriadis, DOM WG invited expert, DOM TS coordinator and 
lead.

On Wednesday, August 21, 2002, at 02:56  AM, Brad Pettit wrote:

>
> Following is a Microsoft proposal of suggested
> principals for DOM test suite development:
> -----------------------------------------------
>
> Test Suite Principles
>
> These DOM test suite principles are based mostly on the W3C CSS test 
> suite principles 
> <http://www.w3.org/Style/CSS/Test/testsuitedocumentation.html> with 
> some additions.
>
> 1. User agent compatibility
> A test suite should be compatible with as many existing user agents as 
> possible. This can be achieved by limiting the requirements to a small 
> set of features commonly supported by most user agents. This also has 
> the added benefit of being more foregiving of user agents in 
> development which may be lacking some features of the technology being 
> tested.

[dd] The DOM TS Group has from the start on aimed at covering as many 
applications as possible. One problem we have found, however, is that we 
have not received user agent specific help in order to ascertain 
compatibility with the frameworks that have been developed. Also, we've 
found that it is difficult to find a small set of features that are 
common to all browsers, especially as those are not entirely 
DOM-compliant. In particular, we found that we did not receive necessary 
feedback from implementors in time, thus limiting their ability to be 
tested in a fair and straightforward manner. Please contact me for 
pointers to the relevant archived discussions.

> 2. Platform compatibility
> A test suite should be be compatible with as many existing platforms as 
> possible. This can be helped by eliminating usage of libraries and 
> features not explicitly required by the technology that run only on 
> specific platforms. File names should use standard ascii characters and 
> be limited to 31 characters (including any file extensions such as 
> .html) for compatibility with most operating systems. It should be 
> possible to not only run a test suite on a server, but to download to 
> and produce correct results on any supported platform.
>
[dd] Except for the limit imposed on the length of file names, I fully 
agree. This has been a design aim from the start and is still the case.

> 3. Same methodology
> All user agents should be tested using the same testing methodology. 
> This helps ensure that the same features are tested, and that they are 
> tested in the same way. Testing features in the same way also helps to 
> test interoperability, which is important for all W3C technologies.
>
[dd] Pending on a clear definition of 'testing methodology' (something 
which the W3C QA WG is looking into) I agree.

> 4. Valid tests
> Tests should be written using correct syntax, logic, and usage of 
> features being tested, unless specifically testing for error 
> conditions. Expected results should be verified by checking the 
> appropriate specification. In addition, all pages in the test suite 
> should pass the W3C Validator.
>
[dd] This is achieved by using the DOM TS ML format (in XML), something 
which in your original posting is mentioned as a non-requirement.

> 5. Based on testable statements
> Test cases must be based upon and traceable to testable statements in 
> the specification being tested.
>
[dd] This applies only if there are testable statements in the 
specification. Being the lead editor of the parts of the document family 
that the Quality Assurance WG produces focusing on testable assertions 
and using grammars that allow for this functionality, I'd very much 
appreciate your comments on the QA mailing list (www-qa@w3.org), as this 
is an issue that requires endorsement by the entire W3C and its Working 
Groups.

> 6. Self describing results
> Test cases should contain a description in prose of what is expected. 
> Test cases should at a minimum describe correct behavior. Error 
> handling test cases should at a minimum describe incorrect or 
> unexpected behavior.
>
[dd] The results from running the DOM TS are self-explanatory in that 
they (in most cases) require a particular output, to which the actual 
output is compared. This can then be represented in tabular form, for 
example.

> 7. Atomic tests
> A test suite should contain atomic (meaning, testing a single feature 
> at a time) test cases. Some properties have little or no effect by 
> themselves, and therefore may require simple compound tests at a 
> minimum.
>
> 8. Incremental difficulty
> Test suites should begin with simpler test cases and proceed to more 
> complex test cases. This makes sense both from a test case authoring 
> perspective, since simpler test cases are easier to author, and from a 
> technology implementer's perspective, since simpler test cases are 
> easier to implement.

[dd] All fundamental interfaces are given the same status; all are 
tested and required for conformance.

> 9. Avoid side-effects
> Since there may be more than one test case per page, test cases should 
> not cause side-effects that will affect the results of any other test 
> cases. Nor should test cases rely on the results of any other test 
> cases. It should be possible to delete or change the order of test 
> cases without affecting the results of any of the test cases.
>
> 10. Feature coverage
> A test suite should contain at least one test case for each explicit 
> feature of a technology.
>
> 11. Reuse
> Test suites should attempt to reuse test cases from previous test 
> suites which also satisfy the requirements listed in these principles.
>
[dd] This is done in the DOM TS, where for example DOM Level 1 Core test 
cases are used (where applicable) in the DOM TS Level 2 Core.

> 12. Simple format
> Simplicity is always a good design principle. In the case of a test 
> suite, keeping the format simple helps minimize unintended effects from 
> the format itself. In addition, a simple format also makes it easier to 
> author the tests.
>
> 13. Ease of authoring
> To reduce the effective cost of producing test suites, and therefore 
> hopefully increase the number of test suites that are produced, it is 
> only logical to adopt the principle of ease of authoring. Not wanting 
> to depend on any particular tools, platform, or device, the test suite 
> format should be easily authorable with only the use of a simple text 
> editor.
>
> 14. Something rather than nothing
> It was the experience of the CSS working group that the CSS1 Test Suite 
> and the many compliant and interoperable implementations that followed 
> clearly demonstrated that a simple test suite is far better than none.
>
> 15. Sooner rather than later
> Corollary: a simple test suite available within a few months is more 
> useful than a more complex test suite a year or more from now.
>
> 16. Test the technology being tested, not other technologies
> When writing a test suite for a particular technology, it is often 
> necessary to use other technologies in the construction of that test 
> suite. Test suite authors must be careful while testing one technology 
> to avoid testing other technologies. Those other technologies should be 
> authored to be maximally compatible in a way that is most forgiving to 
> the various user agents which will be using the test suite.
>
[dd] In the DOM case, it is difficult if not undoable to not test 
compliance with HTML and XML, to name a few, as DOM rests on those 
techologies to a high degree.

> 17. File handling
> The following general guidelines are provided for both test pages and 
> navigation pages.
>    * use the HTML 4.01 Transitional document type for HTML files
>    * use the XHTML 1.0 Transitional document type for XHTML files
>    * use the file extension .html for HTML files
>    * use the file extension .xml for XML files
>    * use the file extension .xhtml for XHTML files
>    * use the mime type "text/html" for HTML files
>    * use the mime type "text/xml" for XML files
>    * use the mime type "application/xhtml+xml" for XHTML files
>
> 18. Ease of use
> Ease of use is always a good design principle. A test suite that is 
> easy to learn and use may be used more often than one that is not. 
> Features that make a test suite easy to use include some of the 
> following: logical organization, good documentation, simple navigation 
> system, clean visual interface, and readable results. Similarity to 
> existing test suites speeds the learning process.
>
[dd] Please download the existing DOM TS from the main page 
(http://www.w3.org/test) and evaluate in order for us to make it better. 
We look forward to your comments.

> 19. Non-requirements
> Finally, as with any set of principles and requirements, there are some 
> non-requirements. These non-requirements are exactly that - they are 
> features not required of a test suite. They may be desirable, or even 
> included, yet they are not necessary to produce a test suite.
>    * automation - the ability to automatically run a test suite
>    * use of a meta-format for generating multiple variants
>
>
[dd] The DOM TS already does the first in part and the second in full. 
This will be the case in the future as well.

Received on Wednesday, 21 August 2002 06:51:21 UTC