Re: ACTION-139 options for test suite design from Felix Sasaki on 2012-08-02 (public-multilingualweb-lt-tests@w3.org from August 2012)

From: Felix Sasaki <fsasaki@w3.org>
Date: Thu, 2 Aug 2012 17:16:14 +0200
To: Dominic Jones <Dominic.Jones@scss.tcd.ie>
Cc: "public-multilingualweb-lt-tests@w3.org" <public-multilingualweb-lt-tests@w3.org>, Leroy Finn <finnle@tcd.ie>, Yves Savourel <ysavourel@enlaso.com>, Multilingual Web LT Public List <public-multilingualweb-lt@w3.org>
Message-ID: <CAL58czp8XSmcTYoS+4TuwF4m6BxA_kDjNkDJRmuzkF0YHVP7uQ@mail.gmail.com>
2012/8/2 Dominic Jones <Dominic.Jones@scss.tcd.ie>

> Hi Felix.
>
> Yeah we think this is a good idea ;) But we don't recon we'll get round to
> this until after we publish the initial test-suite (13th Aug). We will look
> back at this for when we roll out testing against multiple implementations.
> Aiming for this feature by the September meeting.
>
> Is this ok?
>

Perfect, thanks, Dom.

Felix




>
> Dom
> --
> Dominic Jones | Research Assistant
> KDEG, Trinity College Dublin, Ireland.
> Work: + 353 (0) 1896 8426
> Mobile: + 353 (0) 879259719
> http://www.scss.tcd.ie/dominic.jones/
>
>
>
>
>
> On 1 Aug 2012, at 16:02, Felix Sasaki <fsasaki@w3.org> wrote:
>
> > Hi Dom,
> >
> > I think it wouldn't be much effort to convert the XML output to the tab
> separated format that Yves mentioned, e.g. your example below to
> >
> > /book   its:term="no"
> > ...
> >
> > /book/body[1]/p[1]/quote[1] its:term="yes"
> >
> >
> > What do you think?
> >
> > Felix
> >
> > 2012/7/31 Dominic Jones <Dominic.Jones@scss.tcd.ie>
> > Felix, Yves, All…
> >
> > Thanks for your input, a few comments below:
> >
> > Validator --> I see this is forward facing outreach tool which will
> allow for syntax checking of input ITS 2.0 tagged XML or HTML5. I am not
> proposing that we replace the test suite with any kind of validation
> service we will only look at providing an (additional) validation service
> for those using ITS 2.0.
> >
> > Gold Standard vs. Internally Developed Tests --> Please do not see our
> gold standard usage of the data categories as being things you should be
> waiting for to test your own implementations. Please use your own examples,
> sourced from the specification. It is expected that once the test-suite is
> complete it can be tested against a number of already mature
> implementations. We may need to then conduct some bug fixing but the test
> suite and the implementations it is tested against should develop in
> parallel and without any direct dependancies. In summary: expect our test
> files for use with your implementations at a later date but use your own
> tests for now.
> >
> > Test Files --> The sample XML file provided by Felix has been passed
> through the 1.0 (soon to be 2.0) validator and the output provided as a
> list of nodes. The output from the test suite used against various
> implementations will be a list of nodes from the input XML file and the
> application of the ITS data category against each of those terms. So for
> example below it can be seen that the only node identified as being
> terminology related is the "motherboard" node.
> >
> > Source input file:
> >
> > <book
> >   xmlns:its="http://www.w3.org/2005/11/its"
> >   its:version="2.0">
> >  <head>...</head>
> >  <body> ...  <p>And he said: you need a
> >    new <quote
> >      its:term="yes">motherboard</quote>
> >   </p> ...
> >   </body>
> > </book>
> >
> > Output file:
> >
> > <?xml version="1.0" encoding="utf-8"?>
> > <nodeList xmlns:its="http://www.w3.org/2005/11/its"
> >           xmlns:datc="http://example.com/datacats">
> >    <nodeList datacat="terminology">
> >       <node path="/book" outputType="default-value">
> >          <output its:term="no"/>
> >       </node>
> >       <node path="/book/@its:version" outputType="default-value">
> >          <output its:term="no"/>
> >       </node>
> >       <node path="/book/head[1]" outputType="default-value">
> >          <output its:term="no"/>
> >       </node>
> >       <node path="/book/body[1]" outputType="default-value">
> >          <output its:term="no"/>
> >       </node>
> >       <node path="/book/body[1]/p[1]" outputType="default-value">
> >          <output its:term="no"/>
> >       </node>
> >       <node path="/book/body[1]/p[1]/quote[1]"
> outputType="new-value-local">
> >          <output its:term="yes"/>
> >       </node>
> >       <node path="/book/body[1]/p[1]/quote[1]/@its:term"
> outputType="default-value">
> >          <output its:term="no"/>
> >       </node>
> >    </nodeList>
> > </nodeList>
> >
> > Development of test files -->  We plan to publish an initial version of
> the 2.0 test-suite by mid-august. For now my colleague, and new member of
> the WG, Leroy Finn (welcome) will be posting updated ITS1.0 data categories
> in HTML5 (making them 2.0 examples) to the
> public-multilingualweb-lt-tests@w3.org list, expect an individual email
> for each category. This is the forum for your feedback on the design of
> these new test examples. Next week (beginning 6th of August) we're looking
> at 2.0 tests.
> >
> > We're about to allocate quite a bit of dev time on this, so if you have
> concerns, comments or see issues please let me know.
> >
> > Apologies for the long mail…
> >
> > Dom.
> >
> >
> >
> >
> >
> > --
> > Dominic Jones | Research Assistant
> > KDEG, Trinity College Dublin, Ireland.
> > Work: + 353 (0) 1896 8426
> > Mobile: + 353 (0) 879259719
> > http://www.scss.tcd.ie/dominic.jones/
> >
> >
> >
> >
> >
> > On 31 Jul 2012, at 07:44, Felix Sasaki <fsasaki@w3.org> wrote:
> >
> > > Hi Yves, Dom, all,
> > >
> > > 2012/7/30 Yves Savourel <ysavourel@enlaso.com>
> > > Hi Dom, all,
> > >
> > > The overall plan looks fine to me.
> > >
> > > I would just stress that the sooner we get a standard output the
> better. And mid august is very good :)
> > >
> > >
> > > > 1) We plan to provide an initial static test suite, like that
> > > > provided for 1.0, (http://www.w3.org/International/its/tests/)
> > > > but after this test suite is up and running we’ll look at
> > > > building a more dynamic, forward facing, validator,
> > > > like provided at http://validator.w3.org/i18n-checker/
> > > > (Short term goal test-suite, long term validator.)
> > >
> > > Interesting. I suppose one could easily validate the ITS syntax, but
> how would you validate that a tool processes the input as expected?
> > >
> > > That's not the purpose fo the long term validator - I think. It is not
> meant to replace the test suite, but to show people that they are using the
> right syntax - like with the HTML  / CSS validator.
> > >
> > > So I understand your questions below, but since the validator is not
> meant to replace the test suite, they might not be applicable.
> > >
> > > Now about the test suite: I think (Dom, correct me if I'm wrong) that
> the orientation on the ITS 1.0 test suite that Dom is proposing is
> something that will happen for the initial version of the test suite, to
> make sure we have some tests available. That doesn't exclude further tests
> for the real-life production tools that you are mentioning.
> > >
> > > Dom, it might make sense to discuss a mini example here, before you
> set things in stone for the ITS 1.0 test suite based part of the test
> suite: given this input
> > > Example 37: The Terminology data category expressed locally in HTML5
> > > <!DOCTYPE html>
> > > <html lang="en">
> > >  <head>
> > >   <meta charset="utf-8"/>
> > >   <title>Terminology test: default</title>
> > >  </head>
> > >  <body>
> > >   <p>We need a new <span its-term="yes">motherboard</span>
> > >   </p>
> > >  </body>
> > > </html>
> > >
> > > Or this one
> > > <book
> > >   xmlns:its="
> > > http://www.w3.org/2005/11/its
> > > "
> > >   its:version="2.0">
> > >  <head>...</head>
> > >  <body> ...  <p>And he said: you need a
> > >    new <quote
> > >      its:term="yes">motherboard</quote>
> > >   </p> ...
> > >   </body>
> > > </book>
> > >
> > >
> > >
> > > how will "your" test suite output look like?
> > >
> > >
> > > Felix
> > >
> > >
> > > You could develop your own processor obviously, but then since you
> can't control what another processor outputs, how could you compare your
> results with the output of the tested tool? Would the input of the
> validator be the same output format used in the test suit? Just wondering
> how far real-life production tools would be willing to go to validate their
> results.
> > >
> > > Cheers,
> > > -yves
> > >
> > >
> > >
> > >
> > > From: Dominic Jones [mailto:Dominic.Jones@scss.tcd.ie]
> > > Sent: Friday, July 27, 2012 1:27 PM
> > > To: public-multilingualweb-lt-tests@w3.org
> > > Cc: Multilingual Web LT Public List
> > > Subject: Fwd: ACTION-139 options for test suite design
> > >
> > > (Moving this conversation and thread over to a new mailing list
> public-multilingualweb-lt-tests@w3.org This list is specifically for
> publishing all input examples, expected outputs and developments in test
> suite design.)
> > >
> > >
> > >
> > > Hi Yves, Felix, Others...
> > >
> > > I wanted to email the group to update on test-suite design, before
> addressing Yves comments. For those unaware: an important note regarding
> the ITS 1.0 test suite is that a gold standard input and expected output is
> provided for each data category. Implementers execute against the gold
> standard and in each case this is compared to the expected output.
> > >
> > > In terms of the 2.0 test suite I have some key points to propose and
> would like your feedback:
> > >
> > > 1)    We plan to provide an initial static test suite, like that
> provided for 1.0, (http://www.w3.org/International/its/tests/) but after
> this test suite is up and running we’ll look at building a more dynamic,
> forward facing, validator, like provided at
> http://validator.w3.org/i18n-checker/ (Short term goal test-suite, long
> term validator.)
> > >
> > > 2)    The 1.0 data categories and tests will carry over into 2.0. All
> 1.0 categories (excluding Ruby and Directionality) will have new HTML5 gold
> standard input and expected outputs published as part of the new 2.0 test
> suite. These will be driven from the spec document, using an iterative
> process.
> > >
> > > 3)    For the middle of August XML and HTML5 gold standard input and
> expected output will be provided for the new 2.0 categories (Domain, Locale
> Filter and External Resource). These will be combined with the updated 1.0
> data categories to form the new ITS 2.0 test suite, published and hosted.
> > >
> > > 4)    At the Prague meeting we’ll get update commitments from
> implementers as to the categories they are willing to test and start
> rolling out testing against implementations.
> > >
> > > Point 4 leads nicely to Yves email about the previous manual
> comparison of expected output against each implementation. It’s a very
> valid point especially given that 2.0 has more data categories and more
> implementations. I’d like to shelve this for now as we’re (over the month)
> primarily looking at gold standard input / expected output design. However
> at the end of August I’d like to look specifically at how we compare and
> simplify the output of various implementations against our expected output.
> > >
> > > I hope that clears things up a little, please provide your thoughts.
> > >
> > > Dom.
> > >
> > >
> > >
> > > --
> > > Dominic Jones | Research Assistant
> > > KDEG, Trinity College Dublin, Ireland.
> > > Work: + 353 (0) 1896 8426
> > > Mobile: + 353 (0) 879259719
> > > http://www.scss.tcd.ie/dominic.jones/
> > >
> > >
> > >
> > >
> > > Begin forwarded message:
> > >
> > >
> > > Resent-From: public-multilingualweb-lt@w3.org
> > > From: Felix Sasaki <fsasaki@w3.org>
> > > Subject: Re: ACTION-139 options for test suite design
> > > Date: 27 July 2012 10:55:21 IST
> > > To: Yves Savourel <ysavourel@enlaso.com>
> > > Cc: public-multilingualweb-lt@w3.org
> > >
> > > +1. The nodelist-with-its-information.xml is based on what we did for
> the ITS 1.0 test suite, but making this simpler sounds like a good plan.
> > >
> > > Felix
> > > 2012/7/27 Yves Savourel <ysavourel@enlaso.com>
> > > Hi Felix, Dave, all,
> > >
> > > Just a few notes about testing:
> > >
> > > I was starting to look at producing output similar to Felix's
> nodelist-with-its-information.xml and I was wondering if outputting
> something that is more easily comparable would not be simpler.
> > >
> > > You need to use special tools to compare two XML documents (to ignore
> whitepace, etc.), while maybe a simple tab-delimited list of nodes and the
> expected info (outputType, output) can be very easily compared. Just a
> thought.
> > >
> > > Another one: we should probably sort the list of the attribute nodes
> on output as different engine will give you different order.
> > >
> > > Cheers,
> > > -ys
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Felix Sasaki
> > > DFKI / W3C Fellow
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Felix Sasaki
> > > DFKI / W3C Fellow
> > >
> >
> >
> >
> >
> >
> > --
> > Felix Sasaki
> > DFKI / W3C Fellow
> >
>
>


-- 
Felix Sasaki
DFKI / W3C Fellow
Received on Thursday, 2 August 2012 15:16:45 UTC