Re: t0805-c5520-brdr-b-01-e.htm

On Tue, 7 Jun 2005, Edward Reid wrote:
> 
> At 12:58 PM 06/07/2005 +0000, Ian Hickson wrote:
> > This would be a valid concern for a Web page. This is not a Web page, it's
> > a test case. Test cases are designed specifically to catch bugs.
> 
> A big advantage of the W3 tests is that they are not only designed
> specifically to catch bugs, but are designed to catch specific bugs.

That's impossible. You cannot predict what the bugs will be. The best test 
cases are those that catch all kinds of bugs that were never even 
considered when the tests were created.


> A huge problem with, for example, Acid2, is that it's more a political 
> statement than a real test.

Acid2 isn't a QA testcase. (Nor is Acid1, which was a W3C test, FWIW.) 
Acid tests are general indicators of UA quality, designed so as to be fun 
for enterprising UA developers to work with. They are basically complex 
puzzles aimed at programmers. They are also designed to make pretty 
pictures on news articles, so as to encourage user agent vendors to work 
towards supporting them.

However, they are not tests in the true sense.

In any case the test in question here is not in the same bucket as Acid2, 
by any stretch!


> If you want to test the code set interpretation, then by all means provide a
> code set test.

I assume you mean "if you want to test UTF-8 support, then my all means 
provide a UTF-8 test".

However, I fail to see what the alternative is.

If we change the test so that instead of UTF-8 it uses XML entities, then 
all we've done is changed the test from relying on one core requirement of 
a lower-level technology to relying on another core requirement of a 
lower-level technology. Why is that better?


> Good test design should elicit information about failures, not just the 
> fact of failure. When you're writing a borders test, you should design 
> it so that a failure indicates a problem with borders.

I disagree. It is fundamentally impossible to write a test that will catch 
errors in one particular area only.

A CSS test originally designed to catch border bugs could, quite easily, 
catch bugs in any number of other parts of the spec. For example it could 
catch bugs in CSS parsing, CSS cascade, CSS computation, CSS inheritance, 
in the box model, in layout, in painting, in reflow, or even in margin 
collapsing. And it could quite easily catch bugs in other implementations 
of specifications altogether, for example catching an error in the 
handling of UTF-8 (which is a CSS prerequisite).


> A test that only shows that the tested software has a failure somewhere 
> isn't very useful.

Yes, it is. You start with the fact that the test renders incorrectly, 
then you spend about 10 minutes working out where the bug is, and then you 
pass the bug to the engineers. There is no other realistic way to do test 
development in CSS. (I say this as someone who has been working on CSS 
test development professionally for some five years now.)


> It's like a test of a large accounting package which emits the result 
> "company is out of balance by 1.23".

Oh, no, not at all, because you only have the small testcase to work from 
in determining the cause, not a huge test (like the acid tests).


> I'm very concerned that Acid2 will do more harm than good -- you see it 
> already, in the popular arena, passing Acid2 is being considered 
> equivalent to conforming to CSS2.1.

As Acid1 did with CSS1, yeah. I don't really see how this e-mail turned 
into a rant about Acid2, though, since I didn't mention Acid2 in my 
response to your comment about UTF-8 characters. :-)

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Received on Tuesday, 7 June 2005 15:10:38 UTC