Re: Testing API options

On Aug 23, 2013, at 12:42 PM, Dave Longley <dlongley@digitalbazaar.com> wrote:

> On 08/23/2013 04:51 AM, Markus Lanthaler wrote:
>> On Friday, August 23, 2013 1:32 AM, Gregg Kellogg wrote:
>>> We could create an "option" property, which takes a set of objects to
>>> define the option name and value to be passed in. Values are typically
>>> strings, but it may be useful to pass in arrays or hashes as values
>>> too; however, this would loose an RDF interpretation of the value, so
>>> we might not want to go there. This might look like the following:
>>> 
>> [...]
>>>           "@id": "#txxx",
>>>           "@type": ["jld:PositiveEvaluationTest", "jld:CompactTest"],
>> [...]
>>>           "option": [{
>>>             "name": "base",
>>>             "value": "http://example/"
>>>           }],
>> Wouldn't it be simpler to have a single JSON object containing all the
>> options/flags? Something like
>> 
>>   "options": {
>>     "base": "http://example.com",
>>     "compactArrays": "true"
>>   }
>> 
>> I think that would simplify the implementation of test runners as well as
>> make the test manifests more concise.
> 
> +1

Okay, there's few enough options that defining a term for each isn't too much of a problem.
>> 
>> 
>>> Options are taken from the JsonLdOptions set, except for the RDF tests,
>>> which have their own option space (produceGeneralizedRDF and
>>> useNativeTypes)
>>> 
>>> Other things we may want to test for error cases:
>>> * Negative Syntax Tests
>>> * All raised errors are detected
>>> 
>>> Note that it is common for negative tests to not be required for an
>>> implementation to pass, as these represent reasonable areas of
>>> extension. We could add a "processingMode" option to each such test set
>>> to "json-ld-1.0" to override this.
>> I would say that in the json-ld-1.0 processingMode every processor has to
>> raise exactly the same errors and return the same error codes to be
>> conformant. That's exactly the reason why we introduced processing modes.
>> Otherwise applications on using a processor can't rely on the fact that
>> errors are reported in a consistent manner.
We should probably just say that in test documentation that the processingMode json-ld-1.0 is assumed to be present for every test.
>>> Testing the documentLoader option clearly requires a function, but
>>> could be the function name to be evaluated.
>> That's indeed a tricky one.. I'm wondering whether we should make that
>> feature optional!? In any case we need to test remote context resolution.
> 
> I have some fairly old tests (and only a few) in jsonld.js that do remote context resolution; I don't know how useful it is to look at them: https://github.com/digitalbazaar/jsonld.js/blob/master/tests/manifest.jsonld

All tests that reference a context (all the Compact tests, for examples) effectively are testing remote context resolution already.

> I don't think the feature should be optional. We could provide a function name and then describe what the function must do/provide example code -- which, I assume, is what Gregg was suggesting. We could also define the test function as one that just returns test contexts from an in-memory map.

Probably the way to do it is to define the behavior of a callback that must be used to respond to such a callback. It should take into account arguments and options passed to the callback so that they can be reflected back in some meaningful way to get the results. This sounds like a separate test manifest just to test the callback behavior. Maybe this also includes catching an exception raised in the callback handler to generate the appropriate exception in an error manifest.

Gregg
> -- 
> Dave Longley
> CTO
> Digital Bazaar, Inc.