Re: Test suite submissions

On Tue, 11 Mar 2003, Etan Wexler wrote:
>
> Far from minding, I would welcome the incorporation. It was and is my intent
> to submit the tests for use in W3C test suites.

Excellent!


> Since we're on the topic now, what is the procedure for this?

There is no procedure yet. I'm supposed to be converting the CSS1 test
suite to the CSS2.1 test suite format (a non-trivial task, and I have been
concentrating on other work recently in any case). Once that is done, we
will have some sample tests to show submitters what we expect.


> I wrote to Bert Bos off-list and he told me that you were administering
> the test suite list but were not yet ready for submissions. So what do
> we do in the meantime?

The best thing to do if you really want to work on tests is to convert any
tests into the format of the CSS2.1 test suite. The key points are:

   * Making the tests simple to use (green for pass, red for fail, or
     other techniques mentioned in the guidelines [1]).

   * Rename the tests so they use the 2.1 file naming scheme.

   * Convert the tests to XHTML1.1 as per the 2.1 template.

   * Look up the relevant <link rel="help"> links (there may be several
     per test, although don't mention things like the cascade unless that
     is really what is being tested) and include them at the top.

   * Make sure the tests are valid.

Some examples:

   http://www.kitchenquest.com/css-test/attaching/unknown-element.html
   This test is invalid HTML4, and only HTML4 UAs would be expected to
   find the stylesheet. Therefore it is unclear if this test is correct.

   http://www.kitchenquest.com/css-test/fonts/font-family/block.html
   This test is very hard to use. One has to actually read the text in
   order to establish what the expected results are. A better version of
   that test would maybe use the Ahem [2] font to cause the text to become
   a big block when failed (instead of serif), and have the text simply
   say "PASS" otherwise. Then the test could be prefixed by a sentence
   briefly saying that the next line should say "PASS".

   http://www.kitchenquest.com/css-test/text/text-transform/ls-word-in-2-elements.html
   Again, this test could be easier to use with careful use of the Ahem
   font. Here, the key thing to use would be that the "p" and "P"
   characters in Ahem are different (the first is a 0.2 x 1.0 rectangle
   under the baseline, the second is a 1.0 x 1.0 square) so that, with
   careful use of backgrounds, one can make the second lower case letter
   show green if lowercase and cover the background if uppercase.
   Alternatively, simply having two boxes with the top one being the
   test case and the bottom being the control case, with different styles
   but, in a compliant UA, the same rendering, would be easy to use too.
   That would typically be introduced by an unstyled paragraph reading
   "The following two blocks should look identical".

   http://www.kitchenquest.com/css-test/placement/float/each-side.html
   This is a test that is hard to use quickly as well. This is a perfect
   case where having two boxes, one containing the test and one containing
   a control, would work well.

Don't let this list discourage you, though, I found the tests in this test
suite to be of very good quality.

In general, tests that fail in existing UAs are more useful than those
that pass in every UA under the sun. That isn't meant to discourage people
from writing tests that pass in every UA, as it is important to have good
coverage, even of reliably implemented features, but I find that writing
tests in an exploratory fashion, that is, editing a test until it fails,
generally results in more bugs being found than simply going through each
property writing one test each. Apologies for the huge run on sentence.


Incidentally, dynamic tests, using DOM manipulation, are very welcome.
Beware, though, that such tests have to be able to automatically detect if
they are being run in an XHTML or HTML environment and automatically adapt
to use the right DOM APIs (*NS for XHTML, the DOM1 APIs otherwise).

Generally, avoiding the CSSOM APIs is recommended as they are not widely
implemented and may be deprecated in favour of more useful APIs in the
near to medium future.


> And what do we do when there are hundreds of test pages for submission?

Jump with joy! If possible, providing a text file with the full URIs to
each of the tests, one per line, will greatly help with the creation of
the test suite.

Do not worry about creating harnesses, we (the working group) intend to
take care of that.

-- References --
[1] http://www.w3.org/Style/CSS/Test/guidelines.html
[2] http://www.hixie.ch/resources/fonts/

Thanks for wanting to help!
-- 
Ian Hickson                                      )\._.,--....,'``.    fL
"meow"                                          /,   _.. \   _\  ;`._ ,.
http://index.hixie.ch/                         `._.-(,_..'--(,_..'`-.;.'

Received on Tuesday, 11 March 2003 17:44:47 UTC