Re: Work on the i18n test suite

Hello Richard,

Looks like lot of exciting and important work! Some comments inline.

On 2011/12/14 15:21, Richard Ishida wrote:

> I18N TEST HOME
>
> The main test home page for the i18n Activity at the W3C is here:
> http://www.w3.org/International/tests/
>
> In the list on this page you can see a section related to selectors,
> with a list of subtopics. The links from this page are grouped by topic,
> rather than by spec.
>
> (Other sections still need some work.)
>
>
>
> TESTS IN THE FRAMEWORK
>
> The home page for the test framework is here
> http://w3c-test.org/framework/. If you look down the list of test
> suites, you'll see one called 'i18n Selectors'. Each test suite is
> associated with a specification - in this case, Selectors Level 3, by
> the CSS Working Group.

So are the tests essentially in two places (www.w3.org/International and 
w3c-test.org)? Even if these are just pointers, it looks like an 
enormous amount of work to keep things syncronized.

Another question: Why w3c-test.org? Why not test.w3.org? The later would 
make clear that this is belonging to W3C. The former leaves a lot of 
room for guessing.

>  From this page you can run the tests in the test suite and score
> results or you can look at the results so far. It's pretty intuitive.
>
> Here is the Review Results page for the Selectors spec:
> http://w3c-test.org/framework/review/selectors-level3/

I had a look at 
http://w3c-test.org/framework/details/i18n-html4/character-encoding-024/. Lots 
of failures, so I'd like to know what's being tested, but there is not 
the faintest hint about it on that page (not even the test number is in 
a prominent place).

Very similar situation at
http://w3c-test.org/framework/results/i18n-html4/. It would be extremely 
helpful to have a column for exploratory tests, and a column for some 
keywords or a short description of the test. That may send better 
messages to developers, may make it much easier to figure out what 
changes to the spec may lead to better conformance, and so on.

> And here, the Enter Data page:
> http://w3c-test.org/framework/suite/selectors-level3/
>
> (One thing that doesn't yet work is the 'view results by section' feature.)
>
> Note that each test tests an assertion, which is listed on the test
> page. Some of these assertions are labelled 'exploratory', ie. they
> don't test features described in a specification but explore what
> happens if you try such and such.
>
> It is easy to coast through the tests and click buttons to indicate
> whether the test passed, failed, or was unclear. The framework
> automatically works out which browser, platform, etc. you are using,
> most of the time (and you can tell it, if it doesn't). At the moment you
> have to be careful when entering data, since I don't know how to remove
> mistakes.
>
> The test results submission feature of the W3C Test Framework provides a
> simple and effective way to record test results. The results display
> feature provides a powerful way to view the data for a range of browsers
> and browser versions.

Does the framework have a way to switch from one column per rendering 
engine (helpful e.g. for CSS) to one column per browser (there are quite 
some areas where Chrome and Safari go totally different ways).

> In addition, having the tests work in the framework makes it possible
> for other groups to include our tests into their test suites. For
> example a test in the i18n-selectors test suite can be included by
> reference into the official CSS Working Group's test suite.

That's really great.

> RESULTS SUMMARY PAGES
>
> Alongside the subtopic ':lang with lang' on the i18n test home page is a
> link called 'Results'. These results pages provide two things based on a
> recent snapshot of the results from the test framework:
>
> a. a brief prose summary of the results and perhaps some conclusions for
> the tests in a specific topic or range of topics
>
> b. a set of tables showing, for each test, the assertion, and a snapshot
> of the test results, and some useful links. The title of a test
> (left-most column) links to a page where you can run the test and submit
> results. The test id (right-most column) links to a page that shows
> details about all the browsers and browser versions so far tested, and
> the results.
>
> Here is the results page for normalization related tests in the
> Selectors spec:
> http://www.w3.org/International/tests/html-css/normalization/results-selector-normalization

This looks very good. With a bit of work, however, most of it could be 
provided by the framework directly, without snapshots.

Also, the test results are grouped under articles. It might be better to 
have a separate category, There are links at the bottom to other "testy" 
stuff, but I didn't see one at the top, where I expected one. Another 
idea would be to have the comments as a wiki (integrated with the tables 
in the same page). But I'm not an expert in wikis at all.

> And here all the other tests:
> http://www.w3.org/International/tests/html-css/selectors/results-css-lang
>
> These summary pages provide a handy way to get an overview of browser
> behaviour related to a particular topic, but also group and document the
> tests in a way that makes it easier to understand the results.
>
>
>
> LISTS OF TESTS
>
> Alongside the subtopic ':lang with lang' is another link, called 'List'.

There are e.g. the following two tests:

[lang="es"], lang="ES"
A lang= value will match a lang attribute value regardless of case 
differences.

[lang="es"], lang="ES" (xml)
[Exploratory test] A lang= value will match a lang attribute value 
regardless of case differences in a page served as XML.

If there are two tests like this, it would be good to have information 
in both (and not only one of them) about what's different. So presumably 
the first one should say "in a page served as HTML".

Actually, "served as HTML" or "served as XML" is also somewhat 
ambiguous, it's not clear whether this is about MIME type (text/html vs. 
application/xhtml+xml) or whether it's XML declaration/doctype,... related.

> This takes you to a list of tests related to that topic that is
> particularly useful for the person building the test suites, but also
> allows you to run the tests in various different formats, should you
> wish to.
>
> Here is the list page for the Selectors tests:
> www.w3.org/International/tests/html-css/list-selectors
>
>
>
> FUTURE STUFF
>
> The framework supports automatic gathering of results, when appropriate
> for a particular test, using javascript - so you don't even have to
> click through such tests. I have created (or redesigned many of) the
> tests so that when time allows I can add this feature to the tests that
> don't need human intervention. This will make recording results much
> faster.
>
> The W3C is also looking into enabling automatic ref tests (ie. tests
> that compare two files to see whether they are exactly the same).
> Widespread use of this approach is still some way off, however, since
> browsers need to add support for the needed functionality, and even
> then, it will only be recent versions of browsers that support it.
>
>
>
> CREATING TESTS
>
> Unlike most of the other tests in the framework, our tests are generated
> using PHP rather than based on individual files. The framework was not
> originally designed for this and still has some limitations which need
> to be addressed.
>
> A group of related tests is described in a data.php file.
>
> Generating the tests from a data file containing all the relevant
> information for each test makes it much easier to maintain the tests,
> and makes it possible to generate the test data in different ways for
> the various pages and test contexts that we use. In addition, it makes
> it easy to produce any format (html4, html5, xhtml1.1, etc.) from a
> single set of data.
>
> There is some documentation about how to create test data at
> http://www.w3.org/International/tests/html-css/instructions.html
>
>
>
>
> I hope that soon some of you may be interested in contributing or
> running tests, particularly on browsers on things such as mobile
> devices, tvs, etc. will be able to contribute results quickly and
> easily, and all will benefit from the ability to see what is supported
> by which browser.

I (or some of my students) may be interested if I can write them here in 
Ruby and send you the necessary files. I can imagine creating PHP from 
Ruby if that helps you, though.

Regards,    Martin.

> Please let me know if you are interested in recording tests using the
> framework or contributing tests.
>
>
> RI
>
>
>

Received on Thursday, 15 December 2011 07:05:05 UTC