Re: Merging and improving the Turtle test suite(s)

On Feb 26, 2013, at 2:50 PM, David Robillard <d@drobilla.net> wrote:

> On Tue, 2013-02-26 at 21:27 +0000, Andy Seaborne wrote:
>> 
>> On 26/02/13 21:13, David Robillard wrote:
>>> The Turtle test suite situation is currently a bit of a mess.  There's
>>> the tests-ttl suite, the coverage suite, the old test suite from the
>>> team submission [1], and some additions scattered about various
>>> implementations.  Each of these needs to be run in subtly different
>>> ways.
>> 
>> Could you say more?  (I run them all the same way)
> 
> Just minor things, but the issues I encountered were:
> 
> * The required base URIs are different, one seems to be http://example/
> (which isn't even really valid), the other is
> https://dvcs.w3.org/hg/rdf/raw-file/default/rdf-turtle/coverage/tests,
> and the old one is http://www.w3.org/2001/sw/DataAccess/tests/

Note that the submission tests are historical, and aren't used for the rollup EARL report.

Also the README at < https://dvcs.w3.org/hg/rdf/raw-file/default/rdf-turtle/tests-ttl/README> references <http://www.w3.org/2011/rdf-wg/wiki/Turtle_Test_Suite>, which says that tests should all be run with an assumed base of <http://example/base/>. This could probably be more prominent, perhaps in a dc:comment within the Manifest.

Gregg

> * The old manifest is a bit different (it lacks test types)
> 
> * To generate an EARL report with the correct test URIs,
> tests-ttl/manifest.ttl must be parsed with a different base URI than the
> tests themselves. 
> 
> * Strict byte-by-byte testing is prevented, the blank node stuff is
> different for each, triple order is weird sometimes, and so on.  It is
> important to be able to test parsers and serialisers directly without
> involving stores, blank node logic, etc. (some implementations are
> modular and the syntax component does not include such things at all).
> Partially addressed by my previous patches.
> 
> I was less than happy with it being now required to parse the manifest
> to run the tests, since with the old suite one could simply run the tool
> on test-foo.ttl and check that the output matches test-foo.out, but
> since many new tests have the same output, this is reasonable.  Might be
> nice to consistently use good-foo, bad-foo, etc. anyway, but this is not
> important.
> 
>>> In order to do this, the licensing issues of test-ttl/manifest.ttl
>>> brought up by Dave Beckett [2] will need to be resolved,and perhaps
>>> test-ttl/LICENSE is a problem as well.  Otherwise I see no barriers (and
>>> licensing problems for things like this is silly, really)
>> 
>> The LICENSE file is the W3C Software License.
>> 
>> http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231
>> 
>> what's the problem with that?
> 
> I don't pay much attention to W3C bureaucracy and have no idea if there
> is a problem with that or not, but...
> 
>> (note that the conformance test suite should be the W3C test suite 
>> license which has different provisions)
> 
> ... it sounds like the entire suite should be under this license, so
> stuff under another one is a problem of a sort if it is to be included
> (I don't personally care about the licensing at all as long as it is
> Free Software compatible).
> 
> This is mostly just minor housekeeping stuff, it's just a bit of a mess.
> It's not really clear what the distinction between these different
> suites are, or why several exist at all.  I'm guessing this is just
> historical and only remains that way because nobody has done the work,
> so I am volunteering to.
> 
> If there is good reason for them being separate, that's fine, I can fix
> them up independently much like the patches I have already sent to this
> list.  None of this should significantly affect implementations that can
> already run the tests, but it will make life easier for more to do so.
> 
> As far as non-superficial issues go, the only really important one is
> that the current tests do not adequately cover the language.
> 
> -dr
>

Received on Tuesday, 26 February 2013 23:42:53 UTC