Work on the i18n test suite

I have been working hard recently on converting the i18n tests to work 
with the new W3C Test Framework. This should make it much easier to 
gather results and to store them, but also to make tests available to 
other groups. I'd like the folks working on the bidi tests for html5 to 
consider using this approach.

The work is not yet finished, and there are still bugs to be sorted out 
with the new framework, but we are getting to a point where you can see 
some results and you can get a good idea of where things are heading.

I have already uploaded quite a few tests to the framework and 
re-engineered several of our i18n-related tests to integrate with the 
framework.

In the rest of this email I will give you a quick peep at what I'm 
aiming for. I'll use a set of tests related to the Selectors Level 3 
specification as an example.


I18N TEST HOME

The main test home page for the i18n Activity at the W3C is here:
     http://www.w3.org/International/tests/

In the list on this page you can see a section related to selectors, 
with a list of subtopics. The links from this page are grouped by topic, 
rather than by spec.

(Other sections still need some work.)



TESTS IN THE FRAMEWORK

The home page for the test framework is here 
http://w3c-test.org/framework/.  If you look down the list of test 
suites, you'll see one called 'i18n Selectors'. Each test suite is 
associated with a specification - in this case, Selectors Level 3, by 
the CSS Working Group.

 From this page you can run the tests in the test suite and score 
results or you can look at the results so far. It's pretty intuitive.

Here is the Review Results page for the Selectors spec: 
http://w3c-test.org/framework/review/selectors-level3/
And here, the Enter Data page: 
http://w3c-test.org/framework/suite/selectors-level3/

(One thing that doesn't yet work is the 'view results by section' feature.)

Note that each test tests an assertion, which is listed on the test 
page. Some of these assertions are labelled 'exploratory', ie. they 
don't test features described in a specification but explore what 
happens if you try such and such.

It is easy to coast through the tests and click buttons to indicate 
whether the test passed, failed, or was unclear. The framework 
automatically works out which browser, platform, etc. you are using, 
most of the time (and you can tell it, if it doesn't). At the moment you 
have to be careful when entering data, since I don't know how to remove 
mistakes.

The test results submission feature of the W3C Test Framework provides a 
simple and effective way to record test results. The results display 
feature provides a powerful way to view the data for a range of browsers 
and browser versions.

In addition, having the tests work in the framework makes it possible 
for other groups to include our tests into their test suites.  For 
example a test in the i18n-selectors test suite can be included by 
reference into the official CSS Working Group's test suite.



RESULTS SUMMARY PAGES

Alongside the subtopic ':lang with lang' on the i18n test home page is a 
link called 'Results'. These results pages provide two things based on a 
recent snapshot of the results from the test framework:

a. a brief prose summary of the results and perhaps some conclusions for 
the tests in a specific topic or range of topics

b. a set of tables showing, for each test, the assertion, and a snapshot 
of the test results, and some useful links. The title of a test 
(left-most column) links to a page where you can run the test and submit 
results. The  test id (right-most column) links to a page that shows 
details about all the browsers and browser versions so far tested, and 
the results.

Here is the results page for normalization related tests in the 
Selectors spec: 
http://www.w3.org/International/tests/html-css/normalization/results-selector-normalization
And here all the other tests: 
http://www.w3.org/International/tests/html-css/selectors/results-css-lang

These summary pages provide a handy way to get an overview of browser 
behaviour related to a particular topic, but also group and document the 
tests in a way that makes it easier to understand the results.



LISTS OF TESTS

Alongside the subtopic ':lang with lang' is another link, called 'List'. 
This takes you to a list of tests related to that topic that is 
particularly useful for the person building the test suites, but also 
allows you to run the tests in various different formats, should you 
wish to.

Here is the list page for the Selectors tests: 
www.w3.org/International/tests/html-css/list-selectors



FUTURE STUFF

The framework supports automatic gathering of results, when appropriate 
for a particular test, using javascript - so you don't even have to 
click through such tests.  I have created (or redesigned many of) the 
tests so that when time allows I can add this feature to the tests that 
don't need human intervention. This will make recording results much faster.

The W3C is also looking into enabling automatic ref tests (ie. tests 
that compare two files to see whether they are exactly the same). 
Widespread use of this approach is still some way off, however, since 
browsers need to add support for the needed functionality, and even 
then, it will only be recent versions of browsers that support it.



CREATING TESTS

Unlike most of the other tests in the framework, our tests are generated 
using PHP rather than based on individual files. The framework was not 
originally designed for this and still has some limitations which need 
to be addressed.

A group of related tests is described in a data.php file.

Generating the tests from a data file containing all the relevant 
information for each test makes it much easier to maintain the tests, 
and makes it possible to generate the test data in different ways for 
the various pages and test contexts that we use. In addition, it makes 
it easy to produce any format (html4, html5, xhtml1.1, etc.) from a 
single set of data.

There is some documentation about how to create test data at 
http://www.w3.org/International/tests/html-css/instructions.html




I hope that soon some of you may be interested in contributing or 
running tests, particularly on browsers on things such as mobile 
devices, tvs, etc. will be able to contribute results quickly and 
easily, and all will benefit from the ability to see what is supported 
by which browser.

Please let me know if you are interested in recording tests using the 
framework or contributing tests.


RI



-- 
Richard Ishida
Internationalization Activity Lead
W3C (World Wide Web Consortium)

http://www.w3.org/International/
http://rishida.net/

Received on Wednesday, 14 December 2011 06:22:22 UTC