Test Suite Principles

Following is a Microsoft proposal of suggested
principals for DOM test suite development:
-----------------------------------------------

Test Suite Principles

These DOM test suite principles are based mostly on the W3C CSS test suite principles <http://www.w3.org/Style/CSS/Test/testsuitedocumentation.html> with some additions. 

1. User agent compatibility
A test suite should be compatible with as many existing user agents as possible. This can be achieved by limiting the requirements to a small set of features commonly supported by most user agents. This also has the added benefit of being more foregiving of user agents in development which may be lacking some features of the technology being tested.

2. Platform compatibility
A test suite should be be compatible with as many existing platforms as possible. This can be helped by eliminating usage of libraries and features not explicitly required by the technology that run only on specific platforms. File names should use standard ascii characters and be limited to 31 characters (including any file extensions such as .html) for compatibility with most operating systems. It should be possible to not only run a test suite on a server, but to download to and produce correct results on any supported platform. 

3. Same methodology
All user agents should be tested using the same testing methodology. This helps ensure that the same features are tested, and that they are tested in the same way. Testing features in the same way also helps to test interoperability, which is important for all W3C technologies. 

4. Valid tests
Tests should be written using correct syntax, logic, and usage of features being tested, unless specifically testing for error conditions. Expected results should be verified by checking the appropriate specification. In addition, all pages in the test suite should pass the W3C Validator. 

5. Based on testable statements
Test cases must be based upon and traceable to testable statements in the specification being tested. 

6. Self describing results
Test cases should contain a description in prose of what is expected. Test cases should at a minimum describe correct behavior. Error handling test cases should at a minimum describe incorrect or unexpected behavior. 

7. Atomic tests
A test suite should contain atomic (meaning, testing a single feature at a time) test cases. Some properties have little or no effect by themselves, and therefore may require simple compound tests at a minimum. 

8. Incremental difficulty
Test suites should begin with simpler test cases and proceed to more complex test cases. This makes sense both from a test case authoring perspective, since simpler test cases are easier to author, and from a technology implementer's perspective, since simpler test cases are easier to implement. 

9. Avoid side-effects
Since there may be more than one test case per page, test cases should not cause side-effects that will affect the results of any other test cases. Nor should test cases rely on the results of any other test cases. It should be possible to delete or change the order of test cases without affecting the results of any of the test cases. 

10. Feature coverage
A test suite should contain at least one test case for each explicit feature of a technology. 

11. Reuse
Test suites should attempt to reuse test cases from previous test suites which also satisfy the requirements listed in these principles. 

12. Simple format
Simplicity is always a good design principle. In the case of a test suite, keeping the format simple helps minimize unintended effects from the format itself. In addition, a simple format also makes it easier to author the tests. 

13. Ease of authoring
To reduce the effective cost of producing test suites, and therefore hopefully increase the number of test suites that are produced, it is only logical to adopt the principle of ease of authoring. Not wanting to depend on any particular tools, platform, or device, the test suite format should be easily authorable with only the use of a simple text editor. 

14. Something rather than nothing
It was the experience of the CSS working group that the CSS1 Test Suite and the many compliant and interoperable implementations that followed clearly demonstrated that a simple test suite is far better than none. 

15. Sooner rather than later
Corollary: a simple test suite available within a few months is more useful than a more complex test suite a year or more from now. 

16. Test the technology being tested, not other technologies
When writing a test suite for a particular technology, it is often necessary to use other technologies in the construction of that test suite. Test suite authors must be careful while testing one technology to avoid testing other technologies. Those other technologies should be authored to be maximally compatible in a way that is most forgiving to the various user agents which will be using the test suite. 

17. File handling
The following general guidelines are provided for both test pages and navigation pages. 
   * use the HTML 4.01 Transitional document type for HTML files 
   * use the XHTML 1.0 Transitional document type for XHTML files 
   * use the file extension .html for HTML files 
   * use the file extension .xml for XML files 
   * use the file extension .xhtml for XHTML files 
   * use the mime type "text/html" for HTML files 
   * use the mime type "text/xml" for XML files 
   * use the mime type "application/xhtml+xml" for XHTML files 

18. Ease of use
Ease of use is always a good design principle. A test suite that is easy to learn and use may be used more often than one that is not. Features that make a test suite easy to use include some of the following: logical organization, good documentation, simple navigation system, clean visual interface, and readable results. Similarity to existing test suites speeds the learning process. 

19. Non-requirements
Finally, as with any set of principles and requirements, there are some non-requirements. These non-requirements are exactly that - they are features not required of a test suite. They may be desirable, or even included, yet they are not necessary to produce a test suite. 
   * automation - the ability to automatically run a test suite 
   * use of a meta-format for generating multiple variants

Received on Tuesday, 20 August 2002 19:56:59 UTC