Re: Testing API options

On Aug 23, 2013, at 4:04 PM, Gregg Kellogg <gregg@greggkellogg.com> wrote:

> On Aug 23, 2013, at 12:42 PM, Dave Longley <dlongley@digitalbazaar.com> wrote:
> 
>> On 08/23/2013 04:51 AM, Markus Lanthaler wrote:
>>> On Friday, August 23, 2013 1:32 AM, Gregg Kellogg wrote:
>>>> We could create an "option" property, which takes a set of objects to
>>>> define the option name and value to be passed in. Values are typically
>>>> strings, but it may be useful to pass in arrays or hashes as values
>>>> too; however, this would loose an RDF interpretation of the value, so
>>>> we might not want to go there. This might look like the following:
>>>> 
>>> [...]
>>>>           "@id": "#txxx",
>>>>           "@type": ["jld:PositiveEvaluationTest", "jld:CompactTest"],
>>> [...]
>>>>           "option": [{
>>>>             "name": "base",
>>>>             "value": "http://example/"
>>>>           }],
>>> Wouldn't it be simpler to have a single JSON object containing all the
>>> options/flags? Something like
>>> 
>>>   "options": {
>>>     "base": "http://example.com",
>>>     "compactArrays": "true"
>>>   }
>>> 
>>> I think that would simplify the implementation of test runners as well as
>>> make the test manifests more concise.
>> 
>> +1
> 
> Okay, there's few enough options that defining a term for each isn't too much of a problem.

I updated the test-suite context at <http://json-ld.org/test-suite/context.jsonld>. I also created two vocabulary definitions (Turtle and JSON-LD) at vocab.ttl and vocab.jsonld. As the vocabulary is referenced using <http://json-ld.org/test-suite/vocab#> it would be good form to have some content-negotiation rewrite rules in the .htaccess file; I attempted to create such rewrite rules, but it doesn't seem to be properly detecting text/ttl and application/ld+json; perhaps someone more familiar with Apache can take a look. Right now, it returns a stub HTML doccument, were we can document using HTML+RDFa.

Gregg
>>>> Options are taken from the JsonLdOptions set, except for the RDF tests,
>>>> which have their own option space (produceGeneralizedRDF and
>>>> useNativeTypes)
>>>> 
>>>> Other things we may want to test for error cases:
>>>> * Negative Syntax Tests
>>>> * All raised errors are detected
>>>> 
>>>> Note that it is common for negative tests to not be required for an
>>>> implementation to pass, as these represent reasonable areas of
>>>> extension. We could add a "processingMode" option to each such test set
>>>> to "json-ld-1.0" to override this.
>>> I would say that in the json-ld-1.0 processingMode every processor has to
>>> raise exactly the same errors and return the same error codes to be
>>> conformant. That's exactly the reason why we introduced processing modes.
>>> Otherwise applications on using a processor can't rely on the fact that
>>> errors are reported in a consistent manner.
> We should probably just say that in test documentation that the processingMode json-ld-1.0 is assumed to be present for every test.
>>>> Testing the documentLoader option clearly requires a function, but
>>>> could be the function name to be evaluated.
>>> That's indeed a tricky one.. I'm wondering whether we should make that
>>> feature optional!? In any case we need to test remote context resolution.
>> 
>> I have some fairly old tests (and only a few) in jsonld.js that do remote context resolution; I don't know how useful it is to look at them: https://github.com/digitalbazaar/jsonld.js/blob/master/tests/manifest.jsonld
> 
> All tests that reference a context (all the Compact tests, for examples) effectively are testing remote context resolution already.
> 
>> I don't think the feature should be optional. We could provide a function name and then describe what the function must do/provide example code -- which, I assume, is what Gregg was suggesting. We could also define the test function as one that just returns test contexts from an in-memory map.
> 
> Probably the way to do it is to define the behavior of a callback that must be used to respond to such a callback. It should take into account arguments and options passed to the callback so that they can be reflected back in some meaningful way to get the results. This sounds like a separate test manifest just to test the callback behavior. Maybe this also includes catching an exception raised in the callback handler to generate the appropriate exception in an error manifest.
> 
> Gregg
>> -- 
>> Dave Longley
>> CTO
>> Digital Bazaar, Inc.
>