Re: Please check out I18N test suite

On Tue, 9 Dec 2003, Richard Ishida wrote:
>
> I have been converting our i18n test pages to use the same model as the
> CSS tests.  I'd be grateful if you could look at two of the tests
> quickly so that I can ensure I'm heading in the right direction.

Sorry it took me so long to reply. I hope my comments are still useful.

Overall these are very good tests. I imagine the CSS working group might
want to borrow some of them for the CSS2.1 test suite when we start
collecting tests for that. :-)


> The tests I have converted so far are:
>
> http://www.w3.org/International/tests/test-css-lang.html

Now redirects to
   http://www.w3.org/International/tests/sec-css-lang-1.html

The main change I would suggest on this page would be to remove the parts
of the tests that explain what is going on, e.g. "The css says: *:lang(es)
{ color:green; }" and "The lang and xml:lang attributes are set to "es".".

Also, the tests that should "remain black" shouldn't -- if the test page
has tests that should go green, then they should _all_ go green. The best
thing to do there is make two pages, one for the "positive" tests, and one
for the "negative" tests. In the negative tests, you style them by default
green, then make the rules make them red if they match.

Finally, note that if a page is sent as text/html, it is HTML, whether or
not the DOCTYPE claims otherwise, and UAs will therefore (correctly) not
interpret xml:lang attributes. Thus the test that checks that xml:lang is
applied is wrong as long as the test is sent as text/html.

Same comments apply to -2 and -3.


> http://www.w3.org/International/tests/test-bidi-blocks.html

Redirects to:
   http://www.w3.org/International/tests/sec-dir-1.html

This test seems pointless -- if the user agent can do whatever it wants
and still claim compliance, why test it? (Note that no specification
requires that the UA use scrollbars, even.)


| http://www.w3.org/International/tests/sec-dir-2.html

Remove text like "The blocks here all inherit RTL directionality from the
<html> element.", or at least comment it out. Most QA engineers don't
care, or understand, the tests -- they just want to see if the UA passes
or fails. Once the test is found to fail, then they will always look at
the source -- often, at least with tests I normally work with, it turns
out the test is wrong. Thus the test's claims can never be trusted. :-)

Similarly, for the tests on this page, remove the sample markup -- if the
tester wants to know the markup, he'll look at the source.

The final thing on this test page is that while reference renderings are
quite useful, on my system they are radically different from the actual
rendering, due to differences in fonts, anti-aliasing, and so forth. (And
since I don't read arabic, I actualy am having trouble working out if the
characters are the same.)

Instead of a reference image, it might be better to have a second way of
rendering the same content, for example using bidi overrides or
left-to-right tables with no borders, with each span in its own cell (a
commonly used way of showing the expected results of bidi rendering).

Note, as an aside, that the alt texts of the images are very poor -- when
read on an aural browser, the tests would sound like this: "The following
should look like this. blablaW3Cblabla. The following should look like
this. 1 2 3 blablaW3Cblabla blablaW3Cblabla blablaW3Cblabla". That's
probably not what you intended! :-) The alt text you give is in fact
better suited for the title attribute.

The same comments apply to -3 and -4.


| http://www.w3.org/International/tests/sec-inline-bidi-1.html

Drop the <h2> headers and anything with class=notes, for the same reasons
as given for the previous tests. The notes about using an alternative to
reference rendering images and better alt text apply here too.

The test that says "Check the text between quotes is in the same order."
should just be "Check looks the same (apart from font differences)." as
for the others, since that applies here too. Same with "Check that all
characters run in the same direction.", that should just be the same as
the other tests. The exclamation mark ones make sense, since it is
sensible to draw attention to the likely problem in those cases. However,
I would suggest making two test pages for this, one for the cases where
the tests should just look the same, and one for the exclamation mark
cases. That would make it easier to run through the tests quickly.


| http://www.w3.org/International/tests/sec-text-transform-1.html

As per earlier comments, I would suggest removing most of the introduction
bit (or putting it in comments) and the class=notes bits and <h2> headers.

It would probably make sense to split this test into one test per block of
codepoints.

It might also be easier to use this test if the characters were side by
side, like this:

   a  a
   b  b
   c  c
   d  d

...etc. For example, using a table. This might make the test long though
(maybe several groups of columns?).


| http://www.w3.org/International/tests/sec-list-style-type-1.html

Split these into one test page per block.

In general, as you might guess, I'm going to suggest removing most of the
explanatory text in these tests. Specifically, removing the "one, should
show" bits is probably best, so that the tests just look like:

   Each line should show the same text twice.

    a. a
    b. b
      a. a
      b. b
      c. c
    d. c
    e. d

(Oops! That UA has a bug, as is obvious from the mismatched letters at
the end.)

The cases that are undefined are probably not overly useful in a testcase,
since they don't test anything.

| http://www.w3.org/International/tests/sec-idn-1.html
| http://www.w3.org/International/tests/sec-idn-2.html

These tests are good, except I would encourage you to use a destination
page more like this:

   http://www.hixie.ch/tests/adhoc/html/flow/object/pass.html

That makes it easier to determine what is going on for the tester.


| http://www.w3.org/International/tests/sec-utf8-signature-1.html

The second control should say "These characters should display as
described: A-tilde, copyright sign ..." instead of saying what they should
_not_ look like. :-)

The "Additional test", as written, is not a test, but what I call a "demo"
(it doesn't say what _should_ happen, it merely demonstrates what _does_
happen). Since the behaviour is well defined for that test, I think it
should define what _should_ happen (namely, three "random" characters
should appear on the first line).

(And, as for the other tests, I would generally recommend cutting back on
the descriptive text.)


| http://www.w3.org/International/tests/sec-ruby-markup-1.html
| http://www.w3.org/International/tests/sec-ruby-markup-2.html

No new comments beyond those that are mentioned for earlier tests.


I hope that helps. Let me know if you have any questions.

Cheers,
-- 
Ian Hickson                                      )\._.,--....,'``.    fL
U+1047E                                         /,   _.. \   _\  ;`._ ,.
http://index.hixie.ch/                         `._.-(,_..'--(,_..'`-.;.'

Received on Monday, 29 December 2003 20:09:38 UTC