Re: t0805-c5520-brdr-b-01-e.htm

At 12:58 PM 06/07/2005 +0000, Ian Hickson wrote:
>This would be a valid concern for a Web page. This is not a Web page, it's
>a test case. Test cases are designed specifically to catch bugs.

A big advantage of the W3 tests is that they are not only designed 
specifically to catch bugs, but are designed to catch specific bugs. A huge 
problem with, for example, Acid2, is that it's more a political statement 
than a real test. If something isn't right, it can take a lot of digging 
into the code to figure out what. With the W3 tests, by contrast, I can 
generally see that a border test failed and say hey, here's a problem with 
borders.

Generally.

If you want to test the code set interpretation, then by all means provide 
a code set test. I'm all for good tests, as should be indicated by the fact 
that I've gone through a large set of the W3 tests (stopped only when the 
browser was failing so consistently that it made no sense to continue in 
that area) with a browser for which I'm merely a casual user (it's not even 
my primary browser, and I have nothing invested except that I find it a 
useful tool), to try to encourage full conformity with CSS2.1 rather than 
merely political conformity via Acid2. A little bit of undesigned code set 
testing sprinkled through various CSS tests doesn't say much about code set 
conformity and dilutes the intent of the tests.

Good test design should elicit information about failures, not just the 
fact of failure. When you're writing a borders test, you should design it 
so that a failure indicates a problem with borders. A test that only shows 
that the tested software has a failure somewhere isn't very useful. It's 
like a test of a large accounting package which emits the result "company 
is out of balance by 1.23". Yes, you need such system tests (like Acid2), 
but most of your testing should be done at a level which provides more 
useful information. The W3 tests are much better than Acid2 in this respect 
-- they don't always tell me explicitly what the problem is, but they are 
divided into units which make it relatively easy to investigate a discrepancy.

I'm very concerned that Acid2 will do more harm than good -- you see it 
already, in the popular arena, passing Acid2 is being considered equivalent 
to conforming to CSS2.1. But Acid2 is only a small system test, not an 
exhaustive test and not even a very useful test. I know that one of the 
three browsers claiming to pass Acid2 also fails the W3 tests miserably, 
and I expect that the others probably fail too. That's how I ended up here.

So I urge you to keep the W3 tests focused on providing information about 
bugs, and not let them turn into mini-Acid2 bug dens.

Edward Reid
Art Works by Melynda Reid: http://paleo.org

Received on Tuesday, 7 June 2005 14:46:00 UTC