- From: Felix Sasaki <fsasaki@w3.org>
- Date: Thu, 2 Aug 2012 17:16:14 +0200
- To: Dominic Jones <Dominic.Jones@scss.tcd.ie>
- Cc: "public-multilingualweb-lt-tests@w3.org" <public-multilingualweb-lt-tests@w3.org>, Leroy Finn <finnle@tcd.ie>, Yves Savourel <ysavourel@enlaso.com>, Multilingual Web LT Public List <public-multilingualweb-lt@w3.org>
- Message-ID: <CAL58czp8XSmcTYoS+4TuwF4m6BxA_kDjNkDJRmuzkF0YHVP7uQ@mail.gmail.com>
2012/8/2 Dominic Jones <Dominic.Jones@scss.tcd.ie> > Hi Felix. > > Yeah we think this is a good idea ;) But we don't recon we'll get round to > this until after we publish the initial test-suite (13th Aug). We will look > back at this for when we roll out testing against multiple implementations. > Aiming for this feature by the September meeting. > > Is this ok? > Perfect, thanks, Dom. Felix > > Dom > -- > Dominic Jones | Research Assistant > KDEG, Trinity College Dublin, Ireland. > Work: + 353 (0) 1896 8426 > Mobile: + 353 (0) 879259719 > http://www.scss.tcd.ie/dominic.jones/ > > > > > > On 1 Aug 2012, at 16:02, Felix Sasaki <fsasaki@w3.org> wrote: > > > Hi Dom, > > > > I think it wouldn't be much effort to convert the XML output to the tab > separated format that Yves mentioned, e.g. your example below to > > > > /book its:term="no" > > ... > > > > /book/body[1]/p[1]/quote[1] its:term="yes" > > > > > > What do you think? > > > > Felix > > > > 2012/7/31 Dominic Jones <Dominic.Jones@scss.tcd.ie> > > Felix, Yves, All… > > > > Thanks for your input, a few comments below: > > > > Validator --> I see this is forward facing outreach tool which will > allow for syntax checking of input ITS 2.0 tagged XML or HTML5. I am not > proposing that we replace the test suite with any kind of validation > service we will only look at providing an (additional) validation service > for those using ITS 2.0. > > > > Gold Standard vs. Internally Developed Tests --> Please do not see our > gold standard usage of the data categories as being things you should be > waiting for to test your own implementations. Please use your own examples, > sourced from the specification. It is expected that once the test-suite is > complete it can be tested against a number of already mature > implementations. We may need to then conduct some bug fixing but the test > suite and the implementations it is tested against should develop in > parallel and without any direct dependancies. In summary: expect our test > files for use with your implementations at a later date but use your own > tests for now. > > > > Test Files --> The sample XML file provided by Felix has been passed > through the 1.0 (soon to be 2.0) validator and the output provided as a > list of nodes. The output from the test suite used against various > implementations will be a list of nodes from the input XML file and the > application of the ITS data category against each of those terms. So for > example below it can be seen that the only node identified as being > terminology related is the "motherboard" node. > > > > Source input file: > > > > <book > > xmlns:its="http://www.w3.org/2005/11/its" > > its:version="2.0"> > > <head>...</head> > > <body> ... <p>And he said: you need a > > new <quote > > its:term="yes">motherboard</quote> > > </p> ... > > </body> > > </book> > > > > Output file: > > > > <?xml version="1.0" encoding="utf-8"?> > > <nodeList xmlns:its="http://www.w3.org/2005/11/its" > > xmlns:datc="http://example.com/datacats"> > > <nodeList datacat="terminology"> > > <node path="/book" outputType="default-value"> > > <output its:term="no"/> > > </node> > > <node path="/book/@its:version" outputType="default-value"> > > <output its:term="no"/> > > </node> > > <node path="/book/head[1]" outputType="default-value"> > > <output its:term="no"/> > > </node> > > <node path="/book/body[1]" outputType="default-value"> > > <output its:term="no"/> > > </node> > > <node path="/book/body[1]/p[1]" outputType="default-value"> > > <output its:term="no"/> > > </node> > > <node path="/book/body[1]/p[1]/quote[1]" > outputType="new-value-local"> > > <output its:term="yes"/> > > </node> > > <node path="/book/body[1]/p[1]/quote[1]/@its:term" > outputType="default-value"> > > <output its:term="no"/> > > </node> > > </nodeList> > > </nodeList> > > > > Development of test files --> We plan to publish an initial version of > the 2.0 test-suite by mid-august. For now my colleague, and new member of > the WG, Leroy Finn (welcome) will be posting updated ITS1.0 data categories > in HTML5 (making them 2.0 examples) to the > public-multilingualweb-lt-tests@w3.org list, expect an individual email > for each category. This is the forum for your feedback on the design of > these new test examples. Next week (beginning 6th of August) we're looking > at 2.0 tests. > > > > We're about to allocate quite a bit of dev time on this, so if you have > concerns, comments or see issues please let me know. > > > > Apologies for the long mail… > > > > Dom. > > > > > > > > > > > > -- > > Dominic Jones | Research Assistant > > KDEG, Trinity College Dublin, Ireland. > > Work: + 353 (0) 1896 8426 > > Mobile: + 353 (0) 879259719 > > http://www.scss.tcd.ie/dominic.jones/ > > > > > > > > > > > > On 31 Jul 2012, at 07:44, Felix Sasaki <fsasaki@w3.org> wrote: > > > > > Hi Yves, Dom, all, > > > > > > 2012/7/30 Yves Savourel <ysavourel@enlaso.com> > > > Hi Dom, all, > > > > > > The overall plan looks fine to me. > > > > > > I would just stress that the sooner we get a standard output the > better. And mid august is very good :) > > > > > > > > > > 1) We plan to provide an initial static test suite, like that > > > > provided for 1.0, (http://www.w3.org/International/its/tests/) > > > > but after this test suite is up and running we’ll look at > > > > building a more dynamic, forward facing, validator, > > > > like provided at http://validator.w3.org/i18n-checker/ > > > > (Short term goal test-suite, long term validator.) > > > > > > Interesting. I suppose one could easily validate the ITS syntax, but > how would you validate that a tool processes the input as expected? > > > > > > That's not the purpose fo the long term validator - I think. It is not > meant to replace the test suite, but to show people that they are using the > right syntax - like with the HTML / CSS validator. > > > > > > So I understand your questions below, but since the validator is not > meant to replace the test suite, they might not be applicable. > > > > > > Now about the test suite: I think (Dom, correct me if I'm wrong) that > the orientation on the ITS 1.0 test suite that Dom is proposing is > something that will happen for the initial version of the test suite, to > make sure we have some tests available. That doesn't exclude further tests > for the real-life production tools that you are mentioning. > > > > > > Dom, it might make sense to discuss a mini example here, before you > set things in stone for the ITS 1.0 test suite based part of the test > suite: given this input > > > Example 37: The Terminology data category expressed locally in HTML5 > > > <!DOCTYPE html> > > > <html lang="en"> > > > <head> > > > <meta charset="utf-8"/> > > > <title>Terminology test: default</title> > > > </head> > > > <body> > > > <p>We need a new <span its-term="yes">motherboard</span> > > > </p> > > > </body> > > > </html> > > > > > > Or this one > > > <book > > > xmlns:its=" > > > http://www.w3.org/2005/11/its > > > " > > > its:version="2.0"> > > > <head>...</head> > > > <body> ... <p>And he said: you need a > > > new <quote > > > its:term="yes">motherboard</quote> > > > </p> ... > > > </body> > > > </book> > > > > > > > > > > > > how will "your" test suite output look like? > > > > > > > > > Felix > > > > > > > > > You could develop your own processor obviously, but then since you > can't control what another processor outputs, how could you compare your > results with the output of the tested tool? Would the input of the > validator be the same output format used in the test suit? Just wondering > how far real-life production tools would be willing to go to validate their > results. > > > > > > Cheers, > > > -yves > > > > > > > > > > > > > > > From: Dominic Jones [mailto:Dominic.Jones@scss.tcd.ie] > > > Sent: Friday, July 27, 2012 1:27 PM > > > To: public-multilingualweb-lt-tests@w3.org > > > Cc: Multilingual Web LT Public List > > > Subject: Fwd: ACTION-139 options for test suite design > > > > > > (Moving this conversation and thread over to a new mailing list > public-multilingualweb-lt-tests@w3.org This list is specifically for > publishing all input examples, expected outputs and developments in test > suite design.) > > > > > > > > > > > > Hi Yves, Felix, Others... > > > > > > I wanted to email the group to update on test-suite design, before > addressing Yves comments. For those unaware: an important note regarding > the ITS 1.0 test suite is that a gold standard input and expected output is > provided for each data category. Implementers execute against the gold > standard and in each case this is compared to the expected output. > > > > > > In terms of the 2.0 test suite I have some key points to propose and > would like your feedback: > > > > > > 1) We plan to provide an initial static test suite, like that > provided for 1.0, (http://www.w3.org/International/its/tests/) but after > this test suite is up and running we’ll look at building a more dynamic, > forward facing, validator, like provided at > http://validator.w3.org/i18n-checker/ (Short term goal test-suite, long > term validator.) > > > > > > 2) The 1.0 data categories and tests will carry over into 2.0. All > 1.0 categories (excluding Ruby and Directionality) will have new HTML5 gold > standard input and expected outputs published as part of the new 2.0 test > suite. These will be driven from the spec document, using an iterative > process. > > > > > > 3) For the middle of August XML and HTML5 gold standard input and > expected output will be provided for the new 2.0 categories (Domain, Locale > Filter and External Resource). These will be combined with the updated 1.0 > data categories to form the new ITS 2.0 test suite, published and hosted. > > > > > > 4) At the Prague meeting we’ll get update commitments from > implementers as to the categories they are willing to test and start > rolling out testing against implementations. > > > > > > Point 4 leads nicely to Yves email about the previous manual > comparison of expected output against each implementation. It’s a very > valid point especially given that 2.0 has more data categories and more > implementations. I’d like to shelve this for now as we’re (over the month) > primarily looking at gold standard input / expected output design. However > at the end of August I’d like to look specifically at how we compare and > simplify the output of various implementations against our expected output. > > > > > > I hope that clears things up a little, please provide your thoughts. > > > > > > Dom. > > > > > > > > > > > > -- > > > Dominic Jones | Research Assistant > > > KDEG, Trinity College Dublin, Ireland. > > > Work: + 353 (0) 1896 8426 > > > Mobile: + 353 (0) 879259719 > > > http://www.scss.tcd.ie/dominic.jones/ > > > > > > > > > > > > > > > Begin forwarded message: > > > > > > > > > Resent-From: public-multilingualweb-lt@w3.org > > > From: Felix Sasaki <fsasaki@w3.org> > > > Subject: Re: ACTION-139 options for test suite design > > > Date: 27 July 2012 10:55:21 IST > > > To: Yves Savourel <ysavourel@enlaso.com> > > > Cc: public-multilingualweb-lt@w3.org > > > > > > +1. The nodelist-with-its-information.xml is based on what we did for > the ITS 1.0 test suite, but making this simpler sounds like a good plan. > > > > > > Felix > > > 2012/7/27 Yves Savourel <ysavourel@enlaso.com> > > > Hi Felix, Dave, all, > > > > > > Just a few notes about testing: > > > > > > I was starting to look at producing output similar to Felix's > nodelist-with-its-information.xml and I was wondering if outputting > something that is more easily comparable would not be simpler. > > > > > > You need to use special tools to compare two XML documents (to ignore > whitepace, etc.), while maybe a simple tab-delimited list of nodes and the > expected info (outputType, output) can be very easily compared. Just a > thought. > > > > > > Another one: we should probably sort the list of the attribute nodes > on output as different engine will give you different order. > > > > > > Cheers, > > > -ys > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Felix Sasaki > > > DFKI / W3C Fellow > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Felix Sasaki > > > DFKI / W3C Fellow > > > > > > > > > > > > > > > -- > > Felix Sasaki > > DFKI / W3C Fellow > > > > -- Felix Sasaki DFKI / W3C Fellow
Received on Thursday, 2 August 2012 15:16:44 UTC