Re: Testing API options

On Aug 23, 2013, at 12:42 PM, Dave Longley <> wrote:

> On 08/23/2013 04:51 AM, Markus Lanthaler wrote:
>> On Friday, August 23, 2013 1:32 AM, Gregg Kellogg wrote:
>>> We could create an "option" property, which takes a set of objects to
>>> define the option name and value to be passed in. Values are typically
>>> strings, but it may be useful to pass in arrays or hashes as values
>>> too; however, this would loose an RDF interpretation of the value, so
>>> we might not want to go there. This might look like the following:
>> [...]
>>>           "@id": "#txxx",
>>>           "@type": ["jld:PositiveEvaluationTest", "jld:CompactTest"],
>> [...]
>>>           "option": [{
>>>             "name": "base",
>>>             "value": "http://example/"
>>>           }],
>> Wouldn't it be simpler to have a single JSON object containing all the
>> options/flags? Something like
>>   "options": {
>>     "base": "",
>>     "compactArrays": "true"
>>   }
>> I think that would simplify the implementation of test runners as well as
>> make the test manifests more concise.
> +1

Okay, there's few enough options that defining a term for each isn't too much of a problem.
>>> Options are taken from the JsonLdOptions set, except for the RDF tests,
>>> which have their own option space (produceGeneralizedRDF and
>>> useNativeTypes)
>>> Other things we may want to test for error cases:
>>> * Negative Syntax Tests
>>> * All raised errors are detected
>>> Note that it is common for negative tests to not be required for an
>>> implementation to pass, as these represent reasonable areas of
>>> extension. We could add a "processingMode" option to each such test set
>>> to "json-ld-1.0" to override this.
>> I would say that in the json-ld-1.0 processingMode every processor has to
>> raise exactly the same errors and return the same error codes to be
>> conformant. That's exactly the reason why we introduced processing modes.
>> Otherwise applications on using a processor can't rely on the fact that
>> errors are reported in a consistent manner.
We should probably just say that in test documentation that the processingMode json-ld-1.0 is assumed to be present for every test.
>>> Testing the documentLoader option clearly requires a function, but
>>> could be the function name to be evaluated.
>> That's indeed a tricky one.. I'm wondering whether we should make that
>> feature optional!? In any case we need to test remote context resolution.
> I have some fairly old tests (and only a few) in jsonld.js that do remote context resolution; I don't know how useful it is to look at them:

All tests that reference a context (all the Compact tests, for examples) effectively are testing remote context resolution already.

> I don't think the feature should be optional. We could provide a function name and then describe what the function must do/provide example code -- which, I assume, is what Gregg was suggesting. We could also define the test function as one that just returns test contexts from an in-memory map.

Probably the way to do it is to define the behavior of a callback that must be used to respond to such a callback. It should take into account arguments and options passed to the callback so that they can be reflected back in some meaningful way to get the results. This sounds like a separate test manifest just to test the callback behavior. Maybe this also includes catching an exception raised in the callback handler to generate the appropriate exception in an error manifest.

> -- 
> Dave Longley
> Digital Bazaar, Inc.

Received on Friday, 23 August 2013 23:04:48 UTC