Re: ACTION-139 options for test suite design

Hi Dom,

I think it wouldn't be much effort to convert the XML output to the tab
separated format that Yves mentioned, e.g. your example below to

/book   its:term="no"
...

/book/body[1]/p[1]/quote[1] its:term="yes"

What do you think?

Felix

2012/7/31 Dominic Jones <Dominic.Jones@scss.tcd.ie>

> Felix, Yves, All…
>
> Thanks for your input, a few comments below:
>
> Validator --> I see this is forward facing outreach tool which will allow
> for syntax checking of input ITS 2.0 tagged XML or HTML5. I am not
> proposing that we replace the test suite with any kind of validation
> service we will only look at providing an (additional) validation service
> for those using ITS 2.0.
>
> Gold Standard vs. Internally Developed Tests --> Please do not see our
> gold standard usage of the data categories as being things you should be
> waiting for to test your own implementations. Please use your own examples,
> sourced from the specification. It is expected that once the test-suite is
> complete it can be tested against a number of already mature
> implementations. We may need to then conduct some bug fixing but the test
> suite and the implementations it is tested against should develop in
> parallel and without any direct dependancies. In summary: expect our test
> files for use with your implementations at a later date but use your own
> tests for now.
>
> Test Files --> The sample XML file provided by Felix has been passed
> through the 1.0 (soon to be 2.0) validator and the output provided as a
> list of nodes. The output from the test suite used against various
> implementations will be a list of nodes from the input XML file and the
> application of the ITS data category against each of those terms. So for
> example below it can be seen that the only node identified as being
> terminology related is the "motherboard" node.
>
> Source input file:
>
> <book
>   xmlns:its="http://www.w3.org/2005/11/its"
>   its:version="2.0">
>  <head>...</head>
>  <body> ...  <p>And he said: you need a
>    new <quote
>      its:term="yes">motherboard</quote>
>   </p> ...
>   </body>
> </book>
>
> Output file:
>
> <?xml version="1.0" encoding="utf-8"?>
> <nodeList xmlns:its="http://www.w3.org/2005/11/its"
>           xmlns:datc="http://example.com/datacats">
>    <nodeList datacat="terminology">
>       <node path="/book" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/@its:version" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/head[1]" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/body[1]" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/body[1]/p[1]" outputType="default-value">
>          <output its:term="no"/>
>       </node>
>       <node path="/book/body[1]/p[1]/quote[1]"
> outputType="new-value-local">
>          <output its:term="yes"/>
>       </node>
>       <node path="/book/body[1]/p[1]/quote[1]/@its:term"
> outputType="default-value">
>          <output its:term="no"/>
>       </node>
>    </nodeList>
> </nodeList>
>
> Development of test files -->  We plan to publish an initial version of
> the 2.0 test-suite by mid-august. For now my colleague, and new member of
> the WG, Leroy Finn (welcome) will be posting updated ITS1.0 data categories
> in HTML5 (making them 2.0 examples) to the
> public-multilingualweb-lt-tests@w3.org list, expect an individual email
> for each category. This is the forum for your feedback on the design of
> these new test examples. Next week (beginning 6th of August) we're looking
> at 2.0 tests.
>
> We're about to allocate quite a bit of dev time on this, so if you have
> concerns, comments or see issues please let me know.
>
> Apologies for the long mail…
>
> Dom.
>
>
>
>
>
> --
> Dominic Jones | Research Assistant
> KDEG, Trinity College Dublin, Ireland.
> Work: + 353 (0) 1896 8426
> Mobile: + 353 (0) 879259719
> http://www.scss.tcd.ie/dominic.jones/
>
>
>
>
>
> On 31 Jul 2012, at 07:44, Felix Sasaki <fsasaki@w3.org> wrote:
>
> > Hi Yves, Dom, all,
> >
> > 2012/7/30 Yves Savourel <ysavourel@enlaso.com>
> > Hi Dom, all,
> >
> > The overall plan looks fine to me.
> >
> > I would just stress that the sooner we get a standard output the better.
> And mid august is very good :)
> >
> >
> > > 1) We plan to provide an initial static test suite, like that
> > > provided for 1.0, (http://www.w3.org/International/its/tests/)
> > > but after this test suite is up and running we’ll look at
> > > building a more dynamic, forward facing, validator,
> > > like provided at http://validator.w3.org/i18n-checker/
> > > (Short term goal test-suite, long term validator.)
> >
> > Interesting. I suppose one could easily validate the ITS syntax, but how
> would you validate that a tool processes the input as expected?
> >
> > That's not the purpose fo the long term validator - I think. It is not
> meant to replace the test suite, but to show people that they are using the
> right syntax - like with the HTML  / CSS validator.
> >
> > So I understand your questions below, but since the validator is not
> meant to replace the test suite, they might not be applicable.
> >
> > Now about the test suite: I think (Dom, correct me if I'm wrong) that
> the orientation on the ITS 1.0 test suite that Dom is proposing is
> something that will happen for the initial version of the test suite, to
> make sure we have some tests available. That doesn't exclude further tests
> for the real-life production tools that you are mentioning.
> >
> > Dom, it might make sense to discuss a mini example here, before you set
> things in stone for the ITS 1.0 test suite based part of the test suite:
> given this input
> > Example 37: The Terminology data category expressed locally in HTML5
> > <!DOCTYPE html>
> > <html lang="en">
> >  <head>
> >   <meta charset="utf-8"/>
> >   <title>Terminology test: default</title>
> >  </head>
> >  <body>
> >   <p>We need a new <span its-term="yes">motherboard</span>
> >   </p>
> >  </body>
> > </html>
> >
> > Or this one
> > <book
> >   xmlns:its="
> > http://www.w3.org/2005/11/its
> > "
> >   its:version="2.0">
> >  <head>...</head>
> >  <body> ...  <p>And he said: you need a
> >    new <quote
> >      its:term="yes">motherboard</quote>
> >   </p> ...
> >   </body>
> > </book>
> >
> >
> >
> > how will "your" test suite output look like?
> >
> >
> > Felix
> >
> >
> > You could develop your own processor obviously, but then since you can't
> control what another processor outputs, how could you compare your results
> with the output of the tested tool? Would the input of the validator be the
> same output format used in the test suit? Just wondering how far real-life
> production tools would be willing to go to validate their results.
> >
> > Cheers,
> > -yves
> >
> >
> >
> >
> > From: Dominic Jones [mailto:Dominic.Jones@scss.tcd.ie]
> > Sent: Friday, July 27, 2012 1:27 PM
> > To: public-multilingualweb-lt-tests@w3.org
> > Cc: Multilingual Web LT Public List
> > Subject: Fwd: ACTION-139 options for test suite design
> >
> > (Moving this conversation and thread over to a new mailing list
> public-multilingualweb-lt-tests@w3.org This list is specifically for
> publishing all input examples, expected outputs and developments in test
> suite design.)
> >
> >
> >
> > Hi Yves, Felix, Others...
> >
> > I wanted to email the group to update on test-suite design, before
> addressing Yves comments. For those unaware: an important note regarding
> the ITS 1.0 test suite is that a gold standard input and expected output is
> provided for each data category. Implementers execute against the gold
> standard and in each case this is compared to the expected output.
> >
> > In terms of the 2.0 test suite I have some key points to propose and
> would like your feedback:
> >
> > 1)    We plan to provide an initial static test suite, like that
> provided for 1.0, (http://www.w3.org/International/its/tests/) but after
> this test suite is up and running we’ll look at building a more dynamic,
> forward facing, validator, like provided at
> http://validator.w3.org/i18n-checker/ (Short term goal test-suite, long
> term validator.)
> >
> > 2)    The 1.0 data categories and tests will carry over into 2.0. All
> 1.0 categories (excluding Ruby and Directionality) will have new HTML5 gold
> standard input and expected outputs published as part of the new 2.0 test
> suite. These will be driven from the spec document, using an iterative
> process.
> >
> > 3)    For the middle of August XML and HTML5 gold standard input and
> expected output will be provided for the new 2.0 categories (Domain, Locale
> Filter and External Resource). These will be combined with the updated 1.0
> data categories to form the new ITS 2.0 test suite, published and hosted.
> >
> > 4)    At the Prague meeting we’ll get update commitments from
> implementers as to the categories they are willing to test and start
> rolling out testing against implementations.
> >
> > Point 4 leads nicely to Yves email about the previous manual comparison
> of expected output against each implementation. It’s a very valid point
> especially given that 2.0 has more data categories and more
> implementations. I’d like to shelve this for now as we’re (over the month)
> primarily looking at gold standard input / expected output design. However
> at the end of August I’d like to look specifically at how we compare and
> simplify the output of various implementations against our expected output.
> >
> > I hope that clears things up a little, please provide your thoughts.
> >
> > Dom.
> >
> >
> >
> > --
> > Dominic Jones | Research Assistant
> > KDEG, Trinity College Dublin, Ireland.
> > Work: + 353 (0) 1896 8426
> > Mobile: + 353 (0) 879259719
> > http://www.scss.tcd.ie/dominic.jones/
> >
> >
> >
> >
> > Begin forwarded message:
> >
> >
> > Resent-From: public-multilingualweb-lt@w3.org
> > From: Felix Sasaki <fsasaki@w3.org>
> > Subject: Re: ACTION-139 options for test suite design
> > Date: 27 July 2012 10:55:21 IST
> > To: Yves Savourel <ysavourel@enlaso.com>
> > Cc: public-multilingualweb-lt@w3.org
> >
> > +1. The nodelist-with-its-information.xml is based on what we did for
> the ITS 1.0 test suite, but making this simpler sounds like a good plan.
> >
> > Felix
> > 2012/7/27 Yves Savourel <ysavourel@enlaso.com>
> > Hi Felix, Dave, all,
> >
> > Just a few notes about testing:
> >
> > I was starting to look at producing output similar to Felix's
> nodelist-with-its-information.xml and I was wondering if outputting
> something that is more easily comparable would not be simpler.
> >
> > You need to use special tools to compare two XML documents (to ignore
> whitepace, etc.), while maybe a simple tab-delimited list of nodes and the
> expected info (outputType, output) can be very easily compared. Just a
> thought.
> >
> > Another one: we should probably sort the list of the attribute nodes on
> output as different engine will give you different order.
> >
> > Cheers,
> > -ys
> >
> >
> >
> >
> >
> >
> >
> > --
> > Felix Sasaki
> > DFKI / W3C Fellow
> >
> >
> >
> >
> >
> >
> >
> > --
> > Felix Sasaki
> > DFKI / W3C Fellow
> >
>
>
>


-- 
Felix Sasaki
DFKI / W3C Fellow

Received on Wednesday, 1 August 2012 15:03:22 UTC