Re: Merging and improving the Turtle test suite(s)

On Tue, 2013-02-26 at 21:27 +0000, Andy Seaborne wrote:
> 
> On 26/02/13 21:13, David Robillard wrote:
> > The Turtle test suite situation is currently a bit of a mess.  There's
> > the tests-ttl suite, the coverage suite, the old test suite from the
> > team submission [1], and some additions scattered about various
> > implementations.  Each of these needs to be run in subtly different
> > ways.
> 
> Could you say more?  (I run them all the same way)

Just minor things, but the issues I encountered were:

 * The required base URIs are different, one seems to be http://example/
(which isn't even really valid), the other is
https://dvcs.w3.org/hg/rdf/raw-file/default/rdf-turtle/coverage/tests,
and the old one is http://www.w3.org/2001/sw/DataAccess/tests/

 * The old manifest is a bit different (it lacks test types)

 * To generate an EARL report with the correct test URIs,
tests-ttl/manifest.ttl must be parsed with a different base URI than the
tests themselves. 

 * Strict byte-by-byte testing is prevented, the blank node stuff is
different for each, triple order is weird sometimes, and so on.  It is
important to be able to test parsers and serialisers directly without
involving stores, blank node logic, etc. (some implementations are
modular and the syntax component does not include such things at all).
Partially addressed by my previous patches.

I was less than happy with it being now required to parse the manifest
to run the tests, since with the old suite one could simply run the tool
on test-foo.ttl and check that the output matches test-foo.out, but
since many new tests have the same output, this is reasonable.  Might be
nice to consistently use good-foo, bad-foo, etc. anyway, but this is not
important.

> > In order to do this, the licensing issues of test-ttl/manifest.ttl
> > brought up by Dave Beckett [2] will need to be resolved,and perhaps
> > test-ttl/LICENSE is a problem as well.  Otherwise I see no barriers (and
> > licensing problems for things like this is silly, really)
> 
> The LICENSE file is the W3C Software License.
> 
> http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231
> 
> what's the problem with that?

I don't pay much attention to W3C bureaucracy and have no idea if there
is a problem with that or not, but...

> (note that the conformance test suite should be the W3C test suite 
> license which has different provisions)

... it sounds like the entire suite should be under this license, so
stuff under another one is a problem of a sort if it is to be included
(I don't personally care about the licensing at all as long as it is
Free Software compatible).

This is mostly just minor housekeeping stuff, it's just a bit of a mess.
It's not really clear what the distinction between these different
suites are, or why several exist at all.  I'm guessing this is just
historical and only remains that way because nobody has done the work,
so I am volunteering to.

If there is good reason for them being separate, that's fine, I can fix
them up independently much like the patches I have already sent to this
list.  None of this should significantly affect implementations that can
already run the tests, but it will make life easier for more to do so.

As far as non-superficial issues go, the only really important one is
that the current tests do not adequately cover the language.

-dr