Re: Microsoft contributing test cases to the CSS WG

On Wednesday 2008-03-05 09:26 -0800, Chris Wilson wrote:
> The test cases are published now at http://samples.msdn.microsoft.com/csstestpages/default.htm.

Great to see them published!


So now, for some details:


It also might be good if your test suite page mentioned that the
tests assume a logical resolution of 96dpi; that's a non-obvious
requirement in addition to Ahem.  Many users may already meet it,
but many won't.  (A number of the border and margin tests depend on
this; I switched to 96dpi at that point so I didn't notice all such
tests.)

I *think* the CSS 2.1 test suite documents this assumption
somewhere, but I couldn't find it when I looked just now.


Comments on the tests (I didn't look closely at tests that Firefox
passed, so these are just comments on wording or comments on tests
where Firefox failed but I think Firefox is correct):

I think Chapter_4/escaped-newline-001.htm is wrong (the spec
requires a result that causes the test to fail) in that it assumes
that an escaped newline should cause a newline.  The entire test
*should* show up on one line, since the escaped newline is just
whitespace, and there's no white-space: pre.  See:
http://www.w3.org/TR/CSS21/generate.html#content
which says:
# Authors may include newlines in the generated content by writing
# the "\A" escape sequence in one of the strings after the 'content'
# property. This inserted line break is still subject to the
# 'white-space' property.

In Chapter_4/quoted-string-001.htm and
Chapter_4/quoted-string-002.htm I think the wording could be
improved, since 'Text' appears at the beginning of the line, not
before it.

In Chapter_5/link-visted-selector-001.htm, it's not clear why the
test says "(you may need to click the page after you click "Link")".
Though I'm also not completely sure of the validity of the test; I
don't know if we define whether foo.html#anchor is considered
visited once you've visited foo.html (and I'm not sure that we
should define it).  The test also doesn't work very well if the
tests are run twice; it seems like :link and :visited ought to be
tested separately so that the test works more than once.

Chapter_8/margin-inline-001.htm (and 002.htm) gives two pass
conditions that seem to me to be exactly the same thing.

Chapter_12/before-after-002.htm wants "PASS PASS" but actually
produces "PASSPASS" (no space).

Chapter_12/content-white-space-001.htm is incorrect for the same
reason as Chapter_4/escaped-newline-001.htm (above).

The pass condition in
Chapter_14/Properties/Background-repeat/background-repeat-004.htm is
worded such that I thought it failed even though it passed (since
there should be green underneath the blue box in addition to right
of it).

Chapter_14/background-339.htm has a "pass-if-red", which is a bad
idea.

Same for Chapter_14/background-color-023.htm, where "Red box" is
also ambiguous (it makes me expect a red border, not a red
background).

Chapter_14/background-position-041.htm is incorrect (the spec
requires a result that causes the test to fail) because "top
50%" is not a valid value for background-position.  When the
keywords are mixed with percentages or lengths, they are required to
be in the correct position (i.e., "50% top" is the same as "50% 0%",
but "top 50%" is invalid).  See the grammar in
http://www.w3.org/TR/CSS21/colors.html#propdef-background-position
and the sentence:
# If at least one value is not a keyword, then the first value
# represents the horizontal position and the second represents the
# vertical position.

The following tests are incorrect for the same reason:
Chapter_14/background-position-042.htm
Chapter_14/background-position-045.htm
Chapter_14/background-position-046.htm

Chapter_16/text-decoration-008.htm has a typo: "within in" should be
"within it".

Chapter_16/white-space-007.htm and Chapter_16/white-space-011.htm
need to make both sentences in the same font if we're supposed to
check that the spacing is the same.  It probably also should specify
monospace as a fallback for those who don't have "Courier New".

Chapter_16/white-space-009.htm is puzzling in two ways:
 * the "[nowrap]" attribute should be nowrap
 * there's no width on a container that would force anything to wrap
but I'm also not really sure if the behavior of nowrap on td is in
the scope of a CSS test suite.

Likewise, I don't understand how Chapter_16/white-space-010.htm is
effectively testing pre-line when the text has no linebreaks in it.

Chapter_16/white-space-011.htm should probably test a wrap attribute
rather than a [wrap] attribute.  And again, there's no container
width here to force anything to wrap.  (I think both of those
comments may apply to other tests here that I didn't look at the
source of as well.)

Why don't the white-space tests using the Ahem font and requiring
the box be less than 3em wide (tests 13-21) require that it is
exactly 2em wide, which it should be (at least in some of them; I
only looked at one or two)?

I think Chapter_16/white-space-024.htm needs to specify
line-height:1.0 .  Even given the metrics of the Ahem font, multiple
interpretations of line-height:normal are allowed.

white-space tests Chapter_16/white-space-025.htm and 026 ought to
require the box be 5em wide (rather than just say greater than 4em),
I think (although, again, I didn't look all that closely).

The first group of Outline tests (starting from
Chapter_18/Properties/Outline/outline-000.htm) could perhaps use a
bigger margin on the page so the outline doesn't run off the edge.

The description of the pass condition of
Chapter_18/Properties/Outline-style/outline-style-007.htm should
probably be different from the description for
Chapter_18/Properties/Outline-style/outline-style-006.htm


As a broader issue, some of the pass conditions are more complicated
to verify (and in some cases also less precise) than we'd like.
(For example, tests like
http://samples.msdn.microsoft.com/csstestpages/Chapter_8/margin-shorthand-003.htm
.) I'm not sure if we'd want to take tests with complex pass
conditions if they duplicate tests that we already have in the test
suite.


The Firefox failures I ran into that seemed valid (although I'm not
100% sure yet):

 * at least on Linux, we get an incorrect ex-height out of the Ahem
   font (or at least different from the 0.5em that your tests
   expect; I think we get 1em), so we fail almost all of the ex-unit
   related tests

 * we fail Chapter_4/invalid-decl-at-rule-001.htm (not sure why yet)

 * we fail Chapter_16/text-decoration-010.htm and
   Chapter_16/text-decoration-012.htm and
   Chapter_16/text-decoration-013.htm because of
   https://bugzilla.mozilla.org/show_bug.cgi?id=403526

 * there may be a valid Firefox line-height:normal bug in
   Chapter_16/white-space-024.htm even though the test is wrong

 * we don't support pre-line (white-space-028)

-David

-- 
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/

Received on Wednesday, 5 March 2008 19:35:15 UTC