Re: Work on the i18n test suite

Hi Martin,

On 15/12/2011 07:04, "Martin J. Dürst" wrote:
> Hello Richard,
>
> Looks like lot of exciting and important work! Some comments inline.
>
> On 2011/12/14 15:21, Richard Ishida wrote:
>
...

>> TESTS IN THE FRAMEWORK
>>
>> The home page for the test framework is here
>> http://w3c-test.org/framework/. If you look down the list of test
>> suites, you'll see one called 'i18n Selectors'. Each test suite is
>> associated with a specification - in this case, Selectors Level 3, by
>> the CSS Working Group.
>
> So are the tests essentially in two places (www.w3.org/International and
> w3c-test.org)? Even if these are just pointers, it looks like an
> enormous amount of work to keep things syncronized.

That was one of the key things I've been working to avoid. There is a 
data.php file for each topic-oriented group of test, and it is the sole 
repository for all of the code that you see anywhere for a given test.

This has been one of the things that I've had to work at to be able to 
use the W3C Framework, since there hadn't been any such generated tests 
before.

The input to the Framework is just a list of tests with some metadata, 
which I generate from that data.php file with a small script.  The 
majority of the results pages, and the lists of links on the list pages 
are also automatically generated from the same data.php file. As are the 
tests themselves, whether on the framework, or running under /International.

Not only that, but having a bunch of related tests in a single data.php 
file really helps when you want to make changes to the tests (especially 
editorial ones, where you can often just search and replace throughout 
the file). No messing around with gazillions of separate file edits.

>
> Another question: Why w3c-test.org? Why not test.w3.org? The later would
> make clear that this is belonging to W3C. The former leaves a lot of
> room for guessing.

That's a question for Philippe and Mike.

>
>> From this page you can run the tests in the test suite and score
>> results or you can look at the results so far. It's pretty intuitive.
>>
>> Here is the Review Results page for the Selectors spec:
>> http://w3c-test.org/framework/review/selectors-level3/
>
> I had a look at
> http://w3c-test.org/framework/details/i18n-html4/character-encoding-024/. Lots
> of failures, so I'd like to know what's being tested, but there is not
> the faintest hint about it on that page (not even the test number is in
> a prominent place).
>
> Very similar situation at
> http://w3c-test.org/framework/results/i18n-html4/. It would be extremely
> helpful to have a column for exploratory tests, and a column for some
> keywords or a short description of the test. That may send better
> messages to developers, may make it much easier to figure out what
> changes to the spec may lead to better conformance, and so on.

Yes, I agree.  That's one reason I created my results pages. You can 
click through to find the test itself, but that's a very unwieldy 
process.  The results pages show the title and the assertion, and group 
things in helpful ways.

Another reason i do the results pages is that the order of the tests in 
the full view, and even the by-section view when it is fixed, will not 
necessarily be the most helpful if ordered alphabetically. I have put in 
a bug asking for per-manifest sorting of tests, although with insertions 
even that can be sub-par.

By the way, my hope is that eventually I will be able to suck the latest 
summary information out of the test framework and into the results page 
using AJAX, but I think there will need to be some code changes to the 
framework to make that easy.

...

>> The test results submission feature of the W3C Test Framework provides a
>> simple and effective way to record test results. The results display
>> feature provides a powerful way to view the data for a range of browsers
>> and browser versions.
>
> Does the framework have a way to switch from one column per rendering
> engine (helpful e.g. for CSS) to one column per browser (there are quite
> some areas where Chrome and Safari go totally different ways).

Not that I'm aware of. I would also like that.  I saw someone else on 
the Testing IG ask for separation of Chrome and Safari too.

...
>> Here is the results page for normalization related tests in the
>> Selectors spec:
>> http://www.w3.org/International/tests/html-css/normalization/results-selector-normalization
>>
>
> This looks very good. With a bit of work, however, most of it could be
> provided by the framework directly, without snapshots.

Yep. See above where I made the same comment.

> Also, the test results are grouped under articles. It might be better to
> have a separate category,

Not clear what you mean by that.


> There are links at the bottom to other "testy"
> stuff, but I didn't see one at the top, where I expected one.

Yes, I've been planning to address that.

> Another
> idea would be to have the comments as a wiki (integrated with the tables
> in the same page). But I'm not an expert in wikis at all.

Hmm. One big obstacle to that for me is the hassle in creating wiki 
pages, and how to do ajaxy stuff and customise sufficently. Or do you 
mean have some contenteditable fields in the page so others can write 
results into the articles?

...> There are e.g. the following two tests:
>
> [lang="es"], lang="ES"
> A lang= value will match a lang attribute value regardless of case
> differences.
>
> [lang="es"], lang="ES" (xml)
> [Exploratory test] A lang= value will match a lang attribute value
> regardless of case differences in a page served as XML.
>
> If there are two tests like this, it would be good to have information
> in both (and not only one of them) about what's different. So presumably
> the first one should say "in a page served as HTML".
>
> Actually, "served as HTML" or "served as XML" is also somewhat
> ambiguous, it's not clear whether this is about MIME type (text/html vs.
> application/xhtml+xml) or whether it's XML declaration/doctype,... related.

Yes, the labelling is also something on my todo list.

...

> I (or some of my students) may be interested if I can write them here in
> Ruby and send you the necessary files. I can imagine creating PHP from
> Ruby if that helps you, though.

Actually, no programming is needed to create a test (unless we come up 
with some new-fangled approach for a particular set of tests). I just 
need the appropriate values filled in in a few variables and a PHP-coded 
array, following the documented conventions, which is the data.php file. 
It's a data-driven program. It would of course, save a lot of trouble to 
create it using the standard PHP code for array declarations, though.

Here's a sample:


=============
$firsttest = '001';
$filename = 'ruby';

$author="Richard Ishida";
$authorhref = "mailto:ishida@w3.org";

$localstyling = "/* the CSS below is not part of the test */\n.test { 
font-size: 150%; padding: 10px; }\nimg { margin: 10px; }\n"; 


# Markup: Simple ruby

$test['001']=array(
'title'=>'basic rendering, with rb',
'assert'=>'When simple ruby markup is used with the rb tag, the ruby 
text appears above the base text at approximately half the font size.',
'prerequisites'=>"A Japanese font.",
'instructions'=>'<p class="instructions"><span 
class="instructionsymbol"><img src="images/same.png" alt="same" 
/></span> Test passes if the text matches the reference graphic.</p>',
'test'=>'<div class="test">これは<ruby><rb>日本語</rb><rt>にほんご</rt> 
</ruby>です。</div>',
'ref'=>'<img src="ruby/support/ruby-markup1.gif" alt="reference graphic" 
/>',
'notes'=>"<p>The Ruby Annotation spec <a 
href='/TR/ruby/#presentation'>recommends non-normatively</a> that, in 
the absence of other style information, simple ruby markup should place 
the ruby text before (ie. above in this case) the base text. This is 
also the <a href='/TR/css3-ruby/#rubypos'>initial value</a> for the CSS3 
Ruby Module draft.</p><p><span class='orangesymbolexpl'>=</span> means 
you should check that the test and reference graphic look the same, 
except for font differences.</p>",
'related'=>'list-ruby#markup_complex',
'x11'=>'tbd',
'next'=>"001a"
);

$test['001a']=array(
'title'=>'basic rendering, with rb',
'assert'=>'[Exploratory test] When simple ruby markup is used with the 
rb tag, the ruby text appears above the base text at approximately half 
the font size.',
'prerequisites'=>"A Japanese font.",
'instructions'=>'<p class="instructions"><span 
class="instructionsymbol"><img src="images/same.png" alt="same" 
/></span> Test passes if the text matches the reference graphic.</p>',
'test'=>'<div class="test">これは<ruby><rb>日本語</rb><rt>にほんご</rt> 
</ruby>です。</div>',
'ref'=>'<img src="ruby/support/ruby-markup1.gif" alt="reference graphic" 
/>',
'notes'=>"<p>The Ruby Annotation spec <a 
href='/TR/ruby/#presentation'>recommends non-normatively</a> that, in 
the absence of other style information, simple ruby markup should place 
the ruby text before (ie. above in this case) the base text. This is 
also the <a href='/TR/css3-ruby/#rubypos'>initial value</a> for the CSS3 
Ruby Module draft.</p><p><span class='orangesymbolexpl'>=</span> means 
you should check that the test and reference graphic look the same, 
except for font differences.</p>",
'related'=>'list-ruby#markup_complex',
'h4'=>'tbd',
'h5'=>'#the-ruby-element',
'xh'=>'tbd',
'xx'=>'tbd',
'x5'=>'#the-ruby-element',
'next'=>"002"
);
================


Before you create the tests, though, talk to me about (1) what tests you 
are planning to create (eg. a list of assertions - so that we don't 
duplicate work), (2) how to design the tests so that they map to 
appropriate formats, exploratory, etc., and (3) how to format the test 
data so that it fits with the generation program.


Cheers,
RI

-- 
Richard Ishida
Internationalization Activity Lead
W3C (World Wide Web Consortium)

http://www.w3.org/International/
http://rishida.net/

Received on Thursday, 15 December 2011 13:08:38 UTC