Argument for keeping 'Kind of Test' or something similar

With many people getting involved with creating tests for the CSS 2.1 test suite we will quickly be inundated with many tests and it will become very difficult to quickly identify what test are targeting what 'type' of testing. I think keeping some form of 'Kind of test' around would be beneficial to help track UA progress in areas and to help make decisions on what work still needs to get done for the test suite. Here is my proposal...

I know the old definition of the 'kind of tests' were vague if not outright confusing in some cases and I agree that the current ones should go away. However I still believe there is a use for the 'kind of test' marking. Or maybe a new name, 'test category'.

The IE testing team has come up with a definition that I think is really useful in identifying tests and helps in narrowing down what kind of testing you want to address at any one time. I think these additions would help quickly identify tests within a large test suite.

Here are some of the ideas we came up with.

Simple: This is a dramatically simple test basically a single feature test, or property value test. This is verifying if 1 element (one of the standard elements) with 1 CSS property and one of its respective values (features) works. Properties that have enumerated values would have as many simple cases as there are enumerated values. Properties that are <length>'s, <percentage>'s, <number>'s, etc. would have to have their boundary conditions checked (min, min-1, min+1, max, max-1, max+1). Pro: there are a finite number of tests that can be created and used to measure support; quality at a low level can also be measured. Con: there are a lot of tests
A question is what are the boundaries for numbers? It is defined in the DOM specs but does that apply to CSS? Indirectly maybe. The CSS spec only has a line stating that values should be truncated to the nearest supported value. In this case I think the boundary conditions are testing that the truncation happens for that property value on that specific property.

Rules: These tests are specifically testing a specific rule straight from a line in the text of the CSS spec. Section 8.3.1 Margin collapsing is a good example of cases that would be considered rules cases along with the items in section 10.3.8 Absolutely positioned, replaced elements. Pro: There are a finite number of tests than can be created and used to measure support, quality at a medium level can be measured. Con: there are many rules in the spec and some are hard to identify as rules.

Complex: Complex or scenario cases really contain all the other cases that don't fall into the other 2 categories. This would be cases that have multiple CSS properties applied together or are testing multiple rules together. Pro: Covers everything else. Con: there is no way to measure support accurately because this remaining space is basically infinite. Quality can only be measured by the cases that exist.

I think we can define/refine these definitions further if we think these make sense. I know there are open issues around shorthand properties in the simple cases that need to be clarified as well.

--
Thanks,
Arron Eicholz

Received on Wednesday, 27 June 2007 19:21:45 UTC