W3C home > Mailing lists > Public > public-css-testsuite@w3.org > September 2010

Re: Tests with difficult/confusing pass condition/wording

From: Gérard Talbot <css21testsuite@gtalbot.org>
Date: Wed, 8 Sep 2010 13:57:33 -0700
Message-ID: <0b22d2b39c44216bbcb8b7793da49ab1.squirrel@cp3.shieldhost.com>
To: "public-css-testsuite@w3.org" <public-css-testsuite@w3.org>
Cc: "Řyvind Stenhaug" <oyvinds@opera.com>
> Hi,
> the following lists a number of tests that were found to be confusing
with
> regard to determining a test result.
> (Authors mentioned: CSS1 Test Suite Contributors, Elika J. Etemad, Gérard
> Talbot, Hilbrand Edskes, Ian Hickson, James Hopkins, Microsoft,
Richard
> Ishida)


Hello Řyvind :)


> ==== CSS1 Test Suite Contributors, Ian Hickson ====
> http://test.csswg.org/suites/css2.1/20100815/html4/c414-flt-ln-002.htm
* The description is too complicated, and it's difficult to tell
easily
> if
> things are really level.

One way to tell is by highlighting the line. And then we realize that
the small coloured boxes ([A][B][C][D]) tops are not perfectly flush
with their correspondent big coloured boxes.

> Maybe this can be split up into multiple
> simpler
> tests?


http://test.csswg.org/suites/css2.1/20100815/html4/c414-flt-ln-003.htm
looks like one but it's not a less complex testcase.

I examined
http://test.csswg.org/suites/css2.1/20100815/html4/c414-flt-ln-002.htm
a bit and I do not see an easy, quick, clear, clean and reliable way to
improve the testcase right now.

Odd.. we discussed this c414-flt-ln-002.htm test before:

http://lists.w3.org/Archives/Public/public-css-testsuite/2010Jul/0044.html

but no one seem to have noticed that the small vs big coloured boxes are
not perfectly lined up, flush at their respective top.


> * Says "unless there is not enough room for the big box to fit on the
small box's line after the small box" - but in fact some of the big
boxes
> are /left/-floated
> * "big coloured boxes should be level with the top of..." (assuming
it's
> meant that the top border edges of the two boxes should be level) -
not
> necessarily true. The float's top may not be higher than the top of
the
> span's linebox, so the TC seems to assume that 1.3em lineheight is
exactly
> enough space for a medium border.

That's probably where I would start trying to correct, improve the
testcase. Is there a need to have a wrapping border around each of those
small boxes?

The inline boxes bleed out of the line box; also line-height 1.3 (==
20.8px) means that fractional pixel issues can or will happen.

> http://test.csswg.org/suites/css2.1/20100815/html4/c44-ln-box-003.htm
Wording can be confusing. "just the green outline of an empty box"
might
> be slightly better

> http://test.csswg.org/suites/css2.1/20100815/html4/c525-font-wt-000.htm
http://test.csswg.org/suites/css2.1/20100815/html4/c527-font-10.htm
This assumes that there is a "very light" weight available for the
default
> font or that a tester knows which faces are available and that using
the
> lightest one is fine

Regarding that c525-font-wt-000.htm test, one idea would be to provide a
link of downloadable fonts (which should be useful or recommendable) for
the test.

Another idea is to make the text font-size much bigger (say 64px) to
give the tester an honest chance to better see and compare. When using
DejaVu Sans font face and with a big font-size, I can see that some
browsers fail that test ... (or that the test is not correct? I haven't
examined the test...it's not a simple one)

I have mentioned
http://test.csswg.org/suites/css2.1/20100815/html4/c527-font-10.htm as a
questionable test in
http://lists.w3.org/Archives/Public/public-css-testsuite/2010Aug/0000.html


> ==== Gérard Talbot ====
> http://test.csswg.org/suites/css2.1/20100815/html4/separated-borders-model-007.htm
Uses the term "separated borders", would perhaps be better to describe
expected appearance more directly.

I agree. I need to word the expected results differently... or to redo
the testcase so that the wording of expected results do not pose
understanding or interpretation issues... need to think this over..

> Moreover, this test seems to depend
> on
> the HTML 'rules' attribute mapping to/interacting with CSS rules in a
certain unspecified way (Opera currently applies the horizontal rules
to
> the <tr>s for instance, haven't investigated what makes Fx/IE fail it;
HTML5 attempts to specify the presentational hints but seems
> unfinished/broken)

This particular matter has been discussed here:

[CSS21] rules="all" attribute specification and border-collapse property
http://lists.w3.org/Archives/Public/www-style/2009May/0007.html

and a "complete set of testcases and agreement on what they should look
like" is available here:

http://fantasai.inkedblade.net/markup/tests/table-frame-rules/


> ==== Gérard Talbot, Hilbrand Edskes ====
> http://test.csswg.org/suites/css2.1/20100815/html4/min-height-106.htm
It's not at all clear from the wording that the green square should be
covered by the scrollbar (which basically means that it will look like
a
> non-square rectangle)

I think I understand what you mean. Since the horizontal scrollbar
should occupy and use a portion of the height of the green square, then
it no longer makes it a real/perfect green square.

I will modify (after consulting with Hilbrand) the testcase accordingly
so that expected results are more reliable, formal, clear.

Right now, I think we should just overlap a perfect 200px by 200px green
square (with no horizontal scrollbar) over (on top of)
#test-red-overlapped.

> ==== Gérard Talbot, James Hopkins ====
> http://test.csswg.org/suites/css2.1/20100815/html4/max-height-105.htm
(Probably correct per the current spec, but none of the browsers I
tried
> show a square in this case

Řyvind, I'm not sure I understand you here.

so this might change, see thread starting
at
> <http://lists.w3.org/Archives/Public/www-style/2010Aug/0541.html>)

I'll get back to you on this one later.

==== Ian Hickson ====
> http://test.csswg.org/suites/css2.1/20100815/html4/background-root-020.htm
There are no "links below". Also, the last two sentences should be
rewritten to not require an understanding of the specification

Agreed.

> http://test.csswg.org/suites/css2.1/20100815/html4/quotes-035.htm
http://test.csswg.org/suites/css2.1/20100815/html4/quotes-036.htm
These are missing a proper pass condition

http://test.csswg.org/suites/css2.1/20100701/html4/quotes-036.htm

was in Arron's list of testcases

http://lists.w3.org/Archives/Public/public-css-testsuite/2010Jul/0041.html

Not sure if quotes-035 was also mentioned elsewhere as well.

It was discussed in

http://lists.w3.org/Archives/Public/public-css-testsuite/2010Jul/0069.html



> http://test.csswg.org/suites/css2.1/20100815/html4/tables-001.htm This
test could use a description/an explicit pass condition

Agreed.

> ==== Microsoft ====
> http://test.csswg.org/suites/css2.1/20100815/html4/table-visual-layout-024.htm
This kind of pass condition is confusing in general (when does the
test
> not pass?).

As worded, the test can not fail. I think the goal is to verify that the
testcase does not make the application crash or create a infinite loop
(hanging).


> In this particular case the "may" is actually also
> incorrect;
> the assert refers to a note, but the note links to a normative
section,
> so
> overflow should be expected. (Of course this assumes that the span's
content is wider than 3em when rendered with the default font,
probably
> a
> safe assumption though using Ahem or non-textual content seems better)

http://test.csswg.org/suites/css2.1/20100815/html4/min-height-080.htm
http://test.csswg.org/suites/css2.1/20100815/html4/max-height-080.htm
See also
> http://lists.w3.org/Archives/Public/public-css-testsuite/2009Sep/0014.html
- to an observer not familiar with the spec's terminology, it is not
obvious that a thick line counts as a "box"

Agreed. I propose "thick black bar across the page" or "thick black
horizontal line across the page" or "black ribbon across the page"
instead as a replacement.

regards, Gérard
-- 
Contributions to the CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/

CSS 2.1 test suite (beta 3; August 15th 2010):
http://test.csswg.org/suites/css2.1/20100815/html4/toc.html

CSS 2.1 test suite contributors:
http://test.csswg.org/source/contributors/
Received on Wednesday, 8 September 2010 20:58:10 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 20 September 2010 17:52:03 GMT