Re: ACTION-139 options for test suite design

Hi Yves, Dom, all,

2012/7/30 Yves Savourel <ysavourel@enlaso.com>

> Hi Dom, all,
>
> The overall plan looks fine to me.
>
> I would just stress that the sooner we get a standard output the better.
> And mid august is very good :)
>
>
> > 1) We plan to provide an initial static test suite, like that
> > provided for 1.0, (http://www.w3.org/International/its/tests/)
> > but after this test suite is up and running we’ll look at
> > building a more dynamic, forward facing, validator,
> > like provided at http://validator.w3.org/i18n-checker/
> > (Short term goal test-suite, long term validator.)
>
> Interesting. I suppose one could easily validate the ITS syntax, but how
> would you validate that a tool processes the input as expected?
>

That's not the purpose fo the long term validator - I think. It is not
meant to replace the test suite, but to show people that they are using the
right syntax - like with the HTML  / CSS validator.

So I understand your questions below, but since the validator is not meant
to replace the test suite, they might not be applicable.

Now about the test suite: I think (Dom, correct me if I'm wrong) that the
orientation on the ITS 1.0 test suite that Dom is proposing is something
that will happen for the initial version of the test suite, to make sure we
have some tests available. That doesn't exclude further tests for the
real-life production tools that you are mentioning.

Dom, it might make sense to discuss a mini example here, before you set
things in stone for the ITS 1.0 test suite based part of the test suite:
given this input
Example 37: The
Terminology<http://www.w3.org/International/multilingualweb/lt/drafts/its20/its20.html#terminology>
data
category expressed locally in HTML5

<!DOCTYPE html>
<html lang="en">
 <head>
  <meta charset="utf-8"/>
  <title>Terminology test: default</title>
 </head>
 <body>
  <p>We need a new <span its-term="yes">motherboard</span>
  </p>
 </body>
</html>

Or this one

<book
  xmlns:its="http://www.w3.org/2005/11/its"
  its:version="2.0">
 <head>...</head>
 <body> ...  <p>And he said: you need a
   new <quote
     its:term="yes">motherboard</quote>
  </p> ...
  </body>
</book>



how will "your" test suite output look like?


Felix


>
> You could develop your own processor obviously, but then since you can't
> control what another processor outputs, how could you compare your results
> with the output of the tested tool? Would the input of the validator be the
> same output format used in the test suit? Just wondering how far real-life
> production tools would be willing to go to validate their results.
>
> Cheers,
> -yves
>
>
>
>
> From: Dominic Jones [mailto:Dominic.Jones@scss.tcd.ie]
> Sent: Friday, July 27, 2012 1:27 PM
> To: public-multilingualweb-lt-tests@w3.org
> Cc: Multilingual Web LT Public List
> Subject: Fwd: ACTION-139 options for test suite design
>
> (Moving this conversation and thread over to a new mailing list
> public-multilingualweb-lt-tests@w3.org This list is specifically for
> publishing all input examples, expected outputs and developments in test
> suite design.)
>
>
>
> Hi Yves, Felix, Others...
>
> I wanted to email the group to update on test-suite design, before
> addressing Yves comments. For those unaware: an important note regarding
> the ITS 1.0 test suite is that a gold standard input and expected output is
> provided for each data category. Implementers execute against the gold
> standard and in each case this is compared to the expected output.
>
> In terms of the 2.0 test suite I have some key points to propose and would
> like your feedback:
>
> 1)    We plan to provide an initial static test suite, like that provided
> for 1.0, (http://www.w3.org/International/its/tests/) but after this test
> suite is up and running we’ll look at building a more dynamic, forward
> facing, validator, like provided at http://validator.w3.org/i18n-checker/(Short term goal test-suite, long term validator.)
>
> 2)    The 1.0 data categories and tests will carry over into 2.0. All 1.0
> categories (excluding Ruby and Directionality) will have new HTML5 gold
> standard input and expected outputs published as part of the new 2.0 test
> suite. These will be driven from the spec document, using an iterative
> process.
>
> 3)    For the middle of August XML and HTML5 gold standard input and
> expected output will be provided for the new 2.0 categories (Domain, Locale
> Filter and External Resource). These will be combined with the updated 1.0
> data categories to form the new ITS 2.0 test suite, published and hosted.
>
> 4)    At the Prague meeting we’ll get update commitments from implementers
> as to the categories they are willing to test and start rolling out testing
> against implementations.
>
> Point 4 leads nicely to Yves email about the previous manual comparison of
> expected output against each implementation. It’s a very valid point
> especially given that 2.0 has more data categories and more
> implementations. I’d like to shelve this for now as we’re (over the month)
> primarily looking at gold standard input / expected output design. However
> at the end of August I’d like to look specifically at how we compare and
> simplify the output of various implementations against our expected output.
>
> I hope that clears things up a little, please provide your thoughts.
>
> Dom.
>
>
>
> --
> Dominic Jones | Research Assistant
> KDEG, Trinity College Dublin, Ireland.
> Work: + 353 (0) 1896 8426
> Mobile: + 353 (0) 879259719
> http://www.scss.tcd.ie/dominic.jones/
>
>
>
>
> Begin forwarded message:
>
>
> Resent-From: public-multilingualweb-lt@w3.org
> From: Felix Sasaki <fsasaki@w3.org>
> Subject: Re: ACTION-139 options for test suite design
> Date: 27 July 2012 10:55:21 IST
> To: Yves Savourel <ysavourel@enlaso.com>
> Cc: public-multilingualweb-lt@w3.org
>
> +1. The nodelist-with-its-information.xml is based on what we did for the
> ITS 1.0 test suite, but making this simpler sounds like a good plan.
>
> Felix
> 2012/7/27 Yves Savourel <ysavourel@enlaso.com>
> Hi Felix, Dave, all,
>
> Just a few notes about testing:
>
> I was starting to look at producing output similar to Felix's
> nodelist-with-its-information.xml and I was wondering if outputting
> something that is more easily comparable would not be simpler.
>
> You need to use special tools to compare two XML documents (to ignore
> whitepace, etc.), while maybe a simple tab-delimited list of nodes and the
> expected info (outputType, output) can be very easily compared. Just a
> thought.
>
> Another one: we should probably sort the list of the attribute nodes on
> output as different engine will give you different order.
>
> Cheers,
> -ys
>
>
>
>
>
>
>
> --
> Felix Sasaki
> DFKI / W3C Fellow
>
>
>
>
>


-- 
Felix Sasaki
DFKI / W3C Fellow

Received on Tuesday, 31 July 2012 06:45:10 UTC