Re: ACTION-139 options for test suite design

Hi Felix. 

Yeah we think this is a good idea ;) But we don't recon we'll get round to this until after we publish the initial test-suite (13th Aug). We will look back at this for when we roll out testing against multiple implementations. Aiming for this feature by the September meeting. 

Is this ok?

Dom
--
Dominic Jones | Research Assistant 
KDEG, Trinity College Dublin, Ireland.
Work: + 353 (0) 1896 8426 
Mobile: + 353 (0) 879259719 
http://www.scss.tcd.ie/dominic.jones/





On 1 Aug 2012, at 16:02, Felix Sasaki <fsasaki@w3.org> wrote:

> Hi Dom,
> 
> I think it wouldn't be much effort to convert the XML output to the tab separated format that Yves mentioned, e.g. your example below to
> 
> /book   its:term="no"
> ...
> 
> /book/body[1]/p[1]/quote[1] its:term="yes"
> 
> 
> What do you think?
> 
> Felix
> 
> 2012/7/31 Dominic Jones <Dominic.Jones@scss.tcd.ie>
> Felix, Yves, All…
> 
> Thanks for your input, a few comments below:
> 
> Validator --> I see this is forward facing outreach tool which will allow for syntax checking of input ITS 2.0 tagged XML or HTML5. I am not proposing that we replace the test suite with any kind of validation service we will only look at providing an (additional) validation service for those using ITS 2.0.
> 
> Gold Standard vs. Internally Developed Tests --> Please do not see our gold standard usage of the data categories as being things you should be waiting for to test your own implementations. Please use your own examples, sourced from the specification. It is expected that once the test-suite is complete it can be tested against a number of already mature implementations. We may need to then conduct some bug fixing but the test suite and the implementations it is tested against should develop in parallel and without any direct dependancies. In summary: expect our test files for use with your implementations at a later date but use your own tests for now.
> 
> Test Files --> The sample XML file provided by Felix has been passed through the 1.0 (soon to be 2.0) validator and the output provided as a list of nodes. The output from the test suite used against various implementations will be a list of nodes from the input XML file and the application of the ITS data category against each of those terms. So for example below it can be seen that the only node identified as being terminology related is the "motherboard" node.
> 
> Source input file:
> 
> <book
>   xmlns:its="http://www.w3.org/2005/11/its"
>   its:version="2.0">
>  <head>...</head>
>  <body> ...  <p>And he said: you need a
>    new <quote
>      its:term="yes">motherboard</quote>
>   </p> ...
>   </body>
> </book>
> 
> Output file:
> 
> <?xml version="1.0" encoding="utf-8"?>
> <nodeList xmlns:its="http://www.w3.org/2005/11/its"
>           xmlns:datc="http://example.com/datacats">
>    <nodeList datacat="terminology">
>       <node path="/book" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/@its:version" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/head[1]" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/body[1]" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/body[1]/p[1]" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/body[1]/p[1]/quote[1]" outputType="new-value-local">
>          <output its:term="yes"/>
>       </node>
>       <node path="/book/body[1]/p[1]/quote[1]/@its:term" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>    </nodeList>
> </nodeList>
> 
> Development of test files -->  We plan to publish an initial version of the 2.0 test-suite by mid-august. For now my colleague, and new member of the WG, Leroy Finn (welcome) will be posting updated ITS1.0 data categories in HTML5 (making them 2.0 examples) to the public-multilingualweb-lt-tests@w3.org list, expect an individual email for each category. This is the forum for your feedback on the design of these new test examples. Next week (beginning 6th of August) we're looking at 2.0 tests.
> 
> We're about to allocate quite a bit of dev time on this, so if you have concerns, comments or see issues please let me know.
> 
> Apologies for the long mail…
> 
> Dom.
> 
> 
> 
> 
> 
> --
> Dominic Jones | Research Assistant
> KDEG, Trinity College Dublin, Ireland.
> Work: + 353 (0) 1896 8426
> Mobile: + 353 (0) 879259719
> http://www.scss.tcd.ie/dominic.jones/
> 
> 
> 
> 
> 
> On 31 Jul 2012, at 07:44, Felix Sasaki <fsasaki@w3.org> wrote:
> 
> > Hi Yves, Dom, all,
> >
> > 2012/7/30 Yves Savourel <ysavourel@enlaso.com>
> > Hi Dom, all,
> >
> > The overall plan looks fine to me.
> >
> > I would just stress that the sooner we get a standard output the better. And mid august is very good :)
> >
> >
> > > 1) We plan to provide an initial static test suite, like that
> > > provided for 1.0, (http://www.w3.org/International/its/tests/)
> > > but after this test suite is up and running we’ll look at
> > > building a more dynamic, forward facing, validator,
> > > like provided at http://validator.w3.org/i18n-checker/
> > > (Short term goal test-suite, long term validator.)
> >
> > Interesting. I suppose one could easily validate the ITS syntax, but how would you validate that a tool processes the input as expected?
> >
> > That's not the purpose fo the long term validator - I think. It is not meant to replace the test suite, but to show people that they are using the right syntax - like with the HTML  / CSS validator.
> >
> > So I understand your questions below, but since the validator is not meant to replace the test suite, they might not be applicable.
> >
> > Now about the test suite: I think (Dom, correct me if I'm wrong) that the orientation on the ITS 1.0 test suite that Dom is proposing is something that will happen for the initial version of the test suite, to make sure we have some tests available. That doesn't exclude further tests for the real-life production tools that you are mentioning.
> >
> > Dom, it might make sense to discuss a mini example here, before you set things in stone for the ITS 1.0 test suite based part of the test suite: given this input
> > Example 37: The Terminology data category expressed locally in HTML5
> > <!DOCTYPE html>
> > <html lang="en">
> >  <head>
> >   <meta charset="utf-8"/>
> >   <title>Terminology test: default</title>
> >  </head>
> >  <body>
> >   <p>We need a new <span its-term="yes">motherboard</span>
> >   </p>
> >  </body>
> > </html>
> >
> > Or this one
> > <book
> >   xmlns:its="
> > http://www.w3.org/2005/11/its
> > "
> >   its:version="2.0">
> >  <head>...</head>
> >  <body> ...  <p>And he said: you need a
> >    new <quote
> >      its:term="yes">motherboard</quote>
> >   </p> ...
> >   </body>
> > </book>
> >
> >
> >
> > how will "your" test suite output look like?
> >
> >
> > Felix
> >
> >
> > You could develop your own processor obviously, but then since you can't control what another processor outputs, how could you compare your results with the output of the tested tool? Would the input of the validator be the same output format used in the test suit? Just wondering how far real-life production tools would be willing to go to validate their results.
> >
> > Cheers,
> > -yves
> >
> >
> >
> >
> > From: Dominic Jones [mailto:Dominic.Jones@scss.tcd.ie]
> > Sent: Friday, July 27, 2012 1:27 PM
> > To: public-multilingualweb-lt-tests@w3.org
> > Cc: Multilingual Web LT Public List
> > Subject: Fwd: ACTION-139 options for test suite design
> >
> > (Moving this conversation and thread over to a new mailing list public-multilingualweb-lt-tests@w3.org This list is specifically for publishing all input examples, expected outputs and developments in test suite design.)
> >
> >
> >
> > Hi Yves, Felix, Others...
> >
> > I wanted to email the group to update on test-suite design, before addressing Yves comments. For those unaware: an important note regarding the ITS 1.0 test suite is that a gold standard input and expected output is provided for each data category. Implementers execute against the gold standard and in each case this is compared to the expected output.
> >
> > In terms of the 2.0 test suite I have some key points to propose and would like your feedback:
> >
> > 1)    We plan to provide an initial static test suite, like that provided for 1.0, (http://www.w3.org/International/its/tests/) but after this test suite is up and running we’ll look at building a more dynamic, forward facing, validator, like provided at http://validator.w3.org/i18n-checker/ (Short term goal test-suite, long term validator.)
> >
> > 2)    The 1.0 data categories and tests will carry over into 2.0. All 1.0 categories (excluding Ruby and Directionality) will have new HTML5 gold standard input and expected outputs published as part of the new 2.0 test suite. These will be driven from the spec document, using an iterative process.
> >
> > 3)    For the middle of August XML and HTML5 gold standard input and expected output will be provided for the new 2.0 categories (Domain, Locale Filter and External Resource). These will be combined with the updated 1.0 data categories to form the new ITS 2.0 test suite, published and hosted.
> >
> > 4)    At the Prague meeting we’ll get update commitments from implementers as to the categories they are willing to test and start rolling out testing against implementations.
> >
> > Point 4 leads nicely to Yves email about the previous manual comparison of expected output against each implementation. It’s a very valid point especially given that 2.0 has more data categories and more implementations. I’d like to shelve this for now as we’re (over the month) primarily looking at gold standard input / expected output design. However at the end of August I’d like to look specifically at how we compare and simplify the output of various implementations against our expected output.
> >
> > I hope that clears things up a little, please provide your thoughts.
> >
> > Dom.
> >
> >
> >
> > --
> > Dominic Jones | Research Assistant
> > KDEG, Trinity College Dublin, Ireland.
> > Work: + 353 (0) 1896 8426
> > Mobile: + 353 (0) 879259719
> > http://www.scss.tcd.ie/dominic.jones/
> >
> >
> >
> >
> > Begin forwarded message:
> >
> >
> > Resent-From: public-multilingualweb-lt@w3.org
> > From: Felix Sasaki <fsasaki@w3.org>
> > Subject: Re: ACTION-139 options for test suite design
> > Date: 27 July 2012 10:55:21 IST
> > To: Yves Savourel <ysavourel@enlaso.com>
> > Cc: public-multilingualweb-lt@w3.org
> >
> > +1. The nodelist-with-its-information.xml is based on what we did for the ITS 1.0 test suite, but making this simpler sounds like a good plan.
> >
> > Felix
> > 2012/7/27 Yves Savourel <ysavourel@enlaso.com>
> > Hi Felix, Dave, all,
> >
> > Just a few notes about testing:
> >
> > I was starting to look at producing output similar to Felix's nodelist-with-its-information.xml and I was wondering if outputting something that is more easily comparable would not be simpler.
> >
> > You need to use special tools to compare two XML documents (to ignore whitepace, etc.), while maybe a simple tab-delimited list of nodes and the expected info (outputType, output) can be very easily compared. Just a thought.
> >
> > Another one: we should probably sort the list of the attribute nodes on output as different engine will give you different order.
> >
> > Cheers,
> > -ys
> >
> >
> >
> >
> >
> >
> >
> > --
> > Felix Sasaki
> > DFKI / W3C Fellow
> >
> >
> >
> >
> >
> >
> >
> > --
> > Felix Sasaki
> > DFKI / W3C Fellow
> >
> 
> 
> 
> 
> 
> -- 
> Felix Sasaki
> DFKI / W3C Fellow
>

Received on Thursday, 2 August 2012 14:03:59 UTC