Re: CSS 2.1 test suite feedback: test feedback

On Oct 3, 2010, at 12:19 pm, Gérard Talbot wrote:

>> I've been through about 60% of the HTML4 tests in the 20100917 suite,
> and have some feedback.
>> This message describes issue with individual those tests with issues
> that I've covered so far.
>> content-type-000 and content-type-001 need to indicate that they are
> only valid when tested over http with the appropriate .htaccess
>> settings.
> 
> Hello Simon,
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/content-type-000.htm
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/content-type-001.htm
> 
> content-type-000 is (RC2) and was (RC1) declaring the http flag; I
> deliberately left a comment in the testcase wrt this.
> 
> content-type-001 does not rely on any .htaccess setting.
> 
> [Addendum: Opera 10.62 passes
> http://test.csswg.org/suites/css2.1/20101001/html4/content-type-000.htm
> and
> http://test.csswg.org/suites/css2.1/20101001/html4/content-type-001.htm
> but fails
> http://www.gtalbot.org/BrowserBugsSection/css21testsuite/content-type-000.html
> and
> http://www.gtalbot.org/BrowserBugsSection/css21testsuite/content-type-001.html
> so these 2 still requires investigation.

WebKit also fails content-type-001. Its behavior may depend on how the OS or
user agent maps file extensions to MIME types.

>> font-size-123 is not self-documenting.
> 
> font-size-123 has been corrected:
> http://lists.w3.org/Archives/Public/public-css-testsuite/2010Sep/0169.html
> 
>> first-letter-nested-006 is not self-documenting.
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/first-letter-nested-006.htm
> 
> I agree with you and I would add that since the color of first letter is
> going to decide in that testcase, then font-size should be bigger. Green
> is considered as a dard color; so constrast with black is not as great
> as with a light color.

As Boris pointed out, this is a ref test, so it doesn't have to be self-documenting.

> Whenever possible, testcases should comply with
> http://www.w3.org/TR/AERT#color-contrast
> and with usability and accessibility guidelines wrt contrast/brightness
> [Robert Hess MSFT], Jakob Nielsen [useit.com], Aries Arditi [Lighthouse
> International], etc.)
> 
> 
>> numbers-units-015 is ambiguous: does it mean any two of the boxes?
>> units-002, units-003: it would be clearer to say "the same height as"
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/units-002.htm
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/units-003.htm
> 
> units-002.htm:
> I disagree. The testcase absolutely needs to check the vertical
> alignment of bottom and top of all 3 rectangles: the 2 "É" and the img
> will (should!) be all using the ascent space and will (should!) be
> "sitting" on the baseline: they will not be using the descent space. 0.8
> mult 250px = height of image.
> 
> Same thing for units-003.htm and with descent space (below baseline).
> 
> Those units-002 and units-003 really must be comparing ascent space and
> descent space: so vertical alignment of bottom and top of all 3
> rectangles must be compared and are the decisive goals/purposes of those
> testcases.

So in Safari, the orange box is taller, but the bottoms of the boxes are aligned.
If the text said "they must be the same height, and aligned", this would be
easier to judge than "exactly aligned with their top and bottom" (since you can't
be exactly aligned at top and bottom and not be the same height).

>> matching-brackets-002: failure mode is red on red
>> matching-brackets-003: is red background a fail?
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/matching-brackets-002.htm
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/matching-brackets-003.htm
> 
> Maybe both those testcases should state at the beginning that there must
> be no red anywhere.

Agreed
> 
>> counters-order-001: takes too much effort to score.
>> at-charset-001 et al: should clarify what two lines means.
>> active-selector-002: confusing
>> active-selector-004: ambiguous
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/active-selector-004.htm
> 
> My neighbour would rightfully, I think, ask me "What is activating a
> text?" and/or "How do you activate a text?".

Agreed; the text should say "activate (for example, click and hold)" or similar.

> 
>> dom-hover-001: clarify that mouse should not move
>> cascade-precedence-001: ambiguous: vertical or horizontal center?
> 
> Simon, I have to rethink anyway all those cascade-precedence-00* because
> webkit browsers use "text-align: auto" and mozilla browsers use
> "-moz-center-or-inherit" (and not text-align).

Does that make the tests invalid?

>> first-line-inherit-001 is not self-documenting.
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/first-line-inherit-001.htm
> 
> I agree with you.

But it's a ref test, so that's OK.

>> first-letter-non-punctuation-001 ambiguous
> 
> http://test.csswg.org/suites/css2.1/20101001/html4/first-letter-non-punctuation-001.htm
> 
> has been fixed (less ambiguous) in RC2 but now, such first character
> (־) is very small: so, not easy to see, verify, confirm that it is
> effectively green and not black. Here, I would be for making the
> font-size much bigger.

The wording change is an improvement. Agreed on the difficulty caused by small glyphs.

> 
>> floats-126 (and similar): says to resize the window, but not what
> should
>> happen when you do.
> 
> What would you propose to improve such test? I'm not saying that I do
> not agree with you here.

"All floats should be the same size. This should remain true when you resize window so that floats wrap in one or two columns."

Simon