Re: json-ld test suite

On Apr 5, 2013, at 5:06 AM, Sandro Hawke <sandro@w3.org> wrote:

> On 04/04/2013 06:40 PM, Markus Lanthaler wrote:
>> On Friday, April 05, 2013 12:08 AM, Sandro Hawke wrote:
>> 
>>>> Yeah.. The only other thing I'm aware of is the link to the test
>>>> suite. Currently it points to http://json-ld.org/test-suite/. Shall
>>>> I change that to http://www.w3.org/2013/json-ld-tests/ as we've
>>>> discussed? I just checked. It's not setup yet.
>>> I think so.    Before setting it up, I was waiting to hear feedback on
>>> the URL (eg json-ld/test-suite).  If I proxy that to
>>> https://github.com/json-ld/json-ld.org/tree/master/test-suite we get
>>> individual test URLs that look like:
>>> 
>>> http://www.w3.org/2013/json-ld-tests/tests/compact-0001-context.jsonld
>>> 
>>> I'm wondering about getting that "/tests" out of there.   Hard to do,
>>> though.   Whatever.
>> I assumed that we will need to freeze the test suite anyway at some point.
>> So it shouldn't be a problem to eliminate that unnecessary sub-directory.
>> 
>> We can change that redirection/proxy at any point, right? So probably we'll
>> just create another directory official-w3c-tests or something and move
>> everything that was approved there. For the time being, using what we
>> currently have should be enough and allows me to change the link.
> 
> Turns out I can't really proxy for github because it does redirects, or something.   So, instead I made a landing page which explains the situation (for now).   I put it on the wiki, so feel free to edit.
> 
>> 
>>> Also, I see the test suite says it's maintained as a free-for-all.
>>> That's probably not appropriate, at least while we're going through CR
>>> (or an LC that might also be a CR).
>> Right. I think it makes sense to have an official, frozen test suite (under
>> the req. W3C license) and a place where we collect new tests even after
>> going to REC.
>> 
>> 
>>> I suggest that we start to add to the manifests entries like
>>>      "approved": "http://www.w3.org/2013/04/04-rdf-wg-irc#T21-59-26"
>>> 
>>> ... for any test which has been approved by the RDF WG.    (which might
>>> be a whole bunch at once.)
>> That's another option but it might become problematic when tests are
>> changed/extended etc. I prefer to freeze the official test and to not touch
>> them anymore :-)
> 
> Well, history suggests we will have to keep touching them.
> 
> There's no need to ever modify a test in place though; just approve/reject them.
> 
> Some test suites also have EXTRACREDIT tests, for optional features.
> 
>>> This way we don't really need much access control -- anyone can see if
>>> the test has been modified since being approved, and check whether it
>>> was actually approved by following that link.
>>> 
>>> Then, to exit CR, we'll need people to be passing "approved" tests.
>>> This should be explained in the README.
>>> 
>>> We should also tell people how to submit their test results, and where
>>> the table of everyone's test results is.
>> Gregg is already working on that:
>> 
>> https://github.com/json-ld/json-ld.org/issues/229
> 
> I assumed he would.   :-)
> 
> Other things I observe about the test suite that might be problems:
> 
>   - there's no way to test for extra triples being generated by JSON->RDF

The toRdf tests look for an exact match of N-Quads output. Why do you say there's no way to test for extra triples?

(There was an earlier version which used SPARQL to test for output, and this would not have found extra triples, unless they were specifically coded into the query, but we abandoned that some time ago).

>   - there's no way to test RDF->JSON

<http://json-ld.org/test-suite/tests/fromRdf-manifest.jsonld> includes several tests from RDF (N-Quads) to JSON-LD. However, it seems to have been removed from the main manifest at some point. I don't know why, it should be added back IMO.

>   - there are frame and normalize tests, but I have no idea what those are (since that's not in the spec any more)

These should be removed from the main manifest.

Gregg

> Yes?
> 
>         -- Sandro
> 
>> 
>>> Yeah, let's add that text and see how it goes.
>> OK, cool.
>> 
>> 
>> Cheers,
>> Markus
>> 
>> 
>> 
>> --
>> Markus Lanthaler
>> @markuslanthaler
>> 
>> 
> 
>

Received on Friday, 5 April 2013 15:00:05 UTC