Test manifest/vocabulary updates

On Aug 23, 2013, at 12:42 PM, Dave Longley <dlongley@digitalbazaar.com> wrote:

> On 08/23/2013 04:51 AM, Markus Lanthaler wrote:
>> On Friday, August 23, 2013 1:32 AM, Gregg Kellogg wrote:
>>> We could create an "option" property, which takes a set of objects to
>>> define the option name and value to be passed in. Values are typically
>>> strings, but it may be useful to pass in arrays or hashes as values
>>> too; however, this would loose an RDF interpretation of the value, so
>>> we might not want to go there. This might look like the following:
>>> 
>> [...]
>>>           "@id": "#txxx",
>>>           "@type": ["jld:PositiveEvaluationTest", "jld:CompactTest"],
>> [...]
>>>           "option": [{
>>>             "name": "base",
>>>             "value": "http://example/"
>>>           }],
>> Wouldn't it be simpler to have a single JSON object containing all the
>> options/flags? Something like
>> 
>>   "options": {
>>     "base": "http://example.com",
>>     "compactArrays": "true"
>>   }
>> 
>> I think that would simplify the implementation of test runners as well as
>> make the test manifests more concise.
> 
> +1

I updated the test-suite instructions with this, and modified the vocabulary. The vocabulary's maintained in http://json-ld.org/test-suite/vocab.ttl. There is a small ruby script to generate vocab.jsonld and vocab.html (in HTML+RDFa) from this. You can see the HTML+RDFa version at http://json-ld.org/test-suite/vocab. Between the two documents, developers should have everything they need to know to create a test runner to run the test suite.

As the runner should not have any a-priori knowledge of the test manifests, we can probably simplify the test organization. I'd like to eliminate error-expand-manifest.jsonld and fold that test into the expand-manifest.jsonld. It also uses an @type of jld:ApiErrorTest, but this is redundant with jld:NegativeEvaluationTest, which already expects an error result.

Next task is to identify tests; I suggest that we create manifest entries for any MUST clause not already having a test (if we were really ambitions, we'd annotate each test with what normative requirement it is testing). If we take a divide and conquer method, we should be able to get through these in the next week or so; certainly, prior to entering CR.

Gregg
>>> Options are taken from the JsonLdOptions set, except for the RDF tests,
>>> which have their own option space (produceGeneralizedRDF and
>>> useNativeTypes)
>>> 
>>> Other things we may want to test for error cases:
>>> * Negative Syntax Tests
>>> * All raised errors are detected
>>> 
>>> Note that it is common for negative tests to not be required for an
>>> implementation to pass, as these represent reasonable areas of
>>> extension. We could add a "processingMode" option to each such test set
>>> to "json-ld-1.0" to override this.
>> I would say that in the json-ld-1.0 processingMode every processor has to
>> raise exactly the same errors and return the same error codes to be
>> conformant. That's exactly the reason why we introduced processing modes.
>> Otherwise applications on using a processor can't rely on the fact that
>> errors are reported in a consistent manner.
>> 
>> 
>>> Testing the documentLoader option clearly requires a function, but
>>> could be the function name to be evaluated.
>> That's indeed a tricky one.. I'm wondering whether we should make that
>> feature optional!? In any case we need to test remote context resolution.
> 
> I have some fairly old tests (and only a few) in jsonld.js that do remote context resolution; I don't know how useful it is to look at them: https://github.com/digitalbazaar/jsonld.js/blob/master/tests/manifest.jsonld
> 
> I don't think the feature should be optional. We could provide a function name and then describe what the function must do/provide example code -- which, I assume, is what Gregg was suggesting. We could also define the test function as one that just returns test contexts from an in-memory map.
> 
> -- 
> Dave Longley
> CTO
> Digital Bazaar, Inc.