Re: Test manifest/vocabulary updates

On Aug 25, 2013, at 4:02 AM, Markus Lanthaler <markus.lanthaler@gmx.net> wrote:

> On Saturday, August 24, 2013 11:47 PM, Gregg Kellogg wrote:
>> I updated the test-suite instructions with this, and modified the
>> vocabulary. The vocabulary's maintained in
>> http://json-ld.org/test-suite/vocab.ttl. There is a small ruby 
>> script to generate vocab.jsonld and vocab.html (in HTML+RDFa) from
>> this. You can see the HTML+RDFa version at
>> http://json-ld.org/test-suite/vocab. Between the two documents, developers
>> should have everything they need to know to create a test runner to run
>> the test suite.
> 
> Great work Gregg. I have a couple of comments but I think it's easier to
> discuss those things directly in the commits. Thus I added them there [1].
> 
> 
>> As the runner should not have any a-priori knowledge of the test
>> manifests, we can probably simplify the test organization. I'd like
>> to eliminate error-expand-manifest.jsonld and fold that test into the
>> expand-manifest.jsonld. It also uses an @type of jld:ApiErrorTest, but
>> this is redundant with jld:NegativeEvaluationTest, which already expects
>> an error result.
> 
> While I fully agree that the runner should not have any a-priori knowledge
> of the test manifests, I think keeping them separate simplifies the creation
> of test runners a whole lot. You don't need to generalize the test methods
> in that case but can keep them very short, a couple of lines in most cases
> (here's how I do it [2]). It also makes it much simpler to run just a subset
> of the test suite. I would thus prefer to keep them separate.

It's a bit arbitrary to say that error cases should be treated differently than non-error cases; particularly, when we also have combinations of errors and options. Already, we'd have to duplicate the manifests for error-compact, error-flatten, error-framing, error-fromRdf, error-toRdf, in addition to error-expand. That's a lot of different test manifests, when each manifest is really just trying to test a particular API. I suggest we continue to use the consolidated tests so that, for example, all of the "Expand" functionality is tested in an expand-manifest. This would leave expand- compact- flatten-, toRdf- and fromRdf-manifests, in addition to framing- and normalize-; that seems to me like a reasonable breakdown to keep things separated to functional areas. If you would really like to keep error and options separate, perhaps a single error-manifest which tests for errors across all the API methods (and to/from RDF), and manifests for different options. What level of granularity do you feel is important?

Doing it this way, means that you can have a single running that just runs over all of the manifests, and recursively over the included tests. My own runner uses a common helper to do the test runs [3]; I keep different files for different API methods, mostly for development purposes, but could easily consolidate them all into a single runner.

>> Next task is to identify tests; I suggest that we create manifest entries
>> for any MUST clause not already having a test (if we were really
> ambitions,
>> we'd annotate each test with what normative requirement it is testing).
> 
> Yeah. I would also suggest to go through the algorithms and write a test for
> every error that is thrown.
> 
> 
>> If we take a divide and conquer method, we should be able to get through
>> these in the next week or so; certainly, prior to entering CR.
> 
> I'm quite busy at the moment but I'll certainly find the time to write some.
> Just tell me what subset of the spec I should work on.

I'm pretty busy too, unfortunately. We need to look at the following:

Expand: errors, with options for base, and useDocumentLoader
Compact: errors, with options for base, compactArrays, expandContext, and useDocumentLoader
Flatten: errors, with options for base, compactArrays, expandContext, and useDocumentLoader
ToRDF: errors, with options for produceGeneralizedRDF
FromRDF: errors, with options for useNativeTypes

Perhaps you could work on Expand and/or Compact. Dave might work on Flatten, and I could work on To/From RDF.

> [1] https://github.com/json-ld/json-ld.org/commit/73f2c63f0c37f04f46bda17da3f6ac
> fe321b282a
> [2] https://github.com/lanthaler/JsonLD/blob/master/Test/JsonLDTestSuiteTest.php#L69
[3] https://github.com/ruby-rdf/json-ld/blob/feature/CR/spec/suite_helper.rb#L123

Gregg

> --
> Markus Lanthaler
> @markuslanthaler
> 
>

Received on Sunday, 25 August 2013 17:35:32 UTC