Re: Test manifest/vocabulary updates

On Aug 25, 2013, at 10:35 AM, Gregg Kellogg <gregg@greggkellogg.com> wrote:

> On Aug 25, 2013, at 4:02 AM, Markus Lanthaler <markus.lanthaler@gmx.net> wrote:
> 
>> On Saturday, August 24, 2013 11:47 PM, Gregg Kellogg wrote:
>>> I updated the test-suite instructions with this, and modified the
>>> vocabulary. The vocabulary's maintained in
>>> http://json-ld.org/test-suite/vocab.ttl. There is a small ruby 
>>> script to generate vocab.jsonld and vocab.html (in HTML+RDFa) from
>>> this. You can see the HTML+RDFa version at
>>> http://json-ld.org/test-suite/vocab. Between the two documents, developers
>>> should have everything they need to know to create a test runner to run
>>> the test suite.
>> 
>> Great work Gregg. I have a couple of comments but I think it's easier to
>> discuss those things directly in the commits. Thus I added them there [1].

These have been addressed in fa235739cf515078dfef7ba87c7832d0bfb689f0.

Gregg

>>> As the runner should not have any a-priori knowledge of the test
>>> manifests, we can probably simplify the test organization. I'd like
>>> to eliminate error-expand-manifest.jsonld and fold that test into the
>>> expand-manifest.jsonld. It also uses an @type of jld:ApiErrorTest, but
>>> this is redundant with jld:NegativeEvaluationTest, which already expects
>>> an error result.
>> 
>> While I fully agree that the runner should not have any a-priori knowledge
>> of the test manifests, I think keeping them separate simplifies the creation
>> of test runners a whole lot. You don't need to generalize the test methods
>> in that case but can keep them very short, a couple of lines in most cases
>> (here's how I do it [2]). It also makes it much simpler to run just a subset
>> of the test suite. I would thus prefer to keep them separate.
> 
> It's a bit arbitrary to say that error cases should be treated differently than non-error cases; particularly, when we also have combinations of errors and options. Already, we'd have to duplicate the manifests for error-compact, error-flatten, error-framing, error-fromRdf, error-toRdf, in addition to error-expand. That's a lot of different test manifests, when each manifest is really just trying to test a particular API. I suggest we continue to use the consolidated tests so that, for example, all of the "Expand" functionality is tested in an expand-manifest. This would leave expand- compact- flatten-, toRdf- and fromRdf-manifests, in addition to framing- and normalize-; that seems to me like a reasonable breakdown to keep things separated to functional areas. If you would really like to keep error and options separate, perhaps a single error-manifest which tests for errors across all the API methods (and to/from RDF), and manifests for different options. What level of granularity do you feel is important?
> 
> Doing it this way, means that you can have a single running that just runs over all of the manifests, and recursively over the included tests. My own runner uses a common helper to do the test runs [3]; I keep different files for different API methods, mostly for development purposes, but could easily consolidate them all into a single runner.
> 
>>> Next task is to identify tests; I suggest that we create manifest entries
>>> for any MUST clause not already having a test (if we were really
>> ambitions,
>>> we'd annotate each test with what normative requirement it is testing).
>> 
>> Yeah. I would also suggest to go through the algorithms and write a test for
>> every error that is thrown.
>> 
>> 
>>> If we take a divide and conquer method, we should be able to get through
>>> these in the next week or so; certainly, prior to entering CR.
>> 
>> I'm quite busy at the moment but I'll certainly find the time to write some.
>> Just tell me what subset of the spec I should work on.
> 
> I'm pretty busy too, unfortunately. We need to look at the following:
> 
> Expand: errors, with options for base, and useDocumentLoader
> Compact: errors, with options for base, compactArrays, expandContext, and useDocumentLoader
> Flatten: errors, with options for base, compactArrays, expandContext, and useDocumentLoader
> ToRDF: errors, with options for produceGeneralizedRDF
> FromRDF: errors, with options for useNativeTypes
> 
> Perhaps you could work on Expand and/or Compact. Dave might work on Flatten, and I could work on To/From RDF.
> 
>> [1] https://github.com/json-ld/json-ld.org/commit/73f2c63f0c37f04f46bda17da3f6ac
>> fe321b282a
>> [2] https://github.com/lanthaler/JsonLD/blob/master/Test/JsonLDTestSuiteTest.php#L69
> [3] https://github.com/ruby-rdf/json-ld/blob/feature/CR/spec/suite_helper.rb#L123
> 
> Gregg
> 
>> --
>> Markus Lanthaler
>> @markuslanthaler
>> 
>> 
>

Received on Sunday, 25 August 2013 18:06:06 UTC