Re: Testing API options

On 08/23/2013 04:51 AM, Markus Lanthaler wrote:
> On Friday, August 23, 2013 1:32 AM, Gregg Kellogg wrote:
>> We could create an "option" property, which takes a set of objects to
>> define the option name and value to be passed in. Values are typically
>> strings, but it may be useful to pass in arrays or hashes as values
>> too; however, this would loose an RDF interpretation of the value, so
>> we might not want to go there. This might look like the following:
>>
> [...]
>>            "@id": "#txxx",
>>            "@type": ["jld:PositiveEvaluationTest", "jld:CompactTest"],
> [...]
>>            "option": [{
>>              "name": "base",
>>              "value": "http://example/"
>>            }],
> Wouldn't it be simpler to have a single JSON object containing all the
> options/flags? Something like
>
>    "options": {
>      "base": "http://example.com",
>      "compactArrays": "true"
>    }
>
> I think that would simplify the implementation of test runners as well as
> make the test manifests more concise.

+1

>
>
>> Options are taken from the JsonLdOptions set, except for the RDF tests,
>> which have their own option space (produceGeneralizedRDF and
>> useNativeTypes)
>>
>> Other things we may want to test for error cases:
>> * Negative Syntax Tests
>> * All raised errors are detected
>>
>> Note that it is common for negative tests to not be required for an
>> implementation to pass, as these represent reasonable areas of
>> extension. We could add a "processingMode" option to each such test set
>> to "json-ld-1.0" to override this.
> I would say that in the json-ld-1.0 processingMode every processor has to
> raise exactly the same errors and return the same error codes to be
> conformant. That's exactly the reason why we introduced processing modes.
> Otherwise applications on using a processor can't rely on the fact that
> errors are reported in a consistent manner.
>
>
>> Testing the documentLoader option clearly requires a function, but
>> could be the function name to be evaluated.
> That's indeed a tricky one.. I'm wondering whether we should make that
> feature optional!? In any case we need to test remote context resolution.

I have some fairly old tests (and only a few) in jsonld.js that do 
remote context resolution; I don't know how useful it is to look at 
them: 
https://github.com/digitalbazaar/jsonld.js/blob/master/tests/manifest.jsonld

I don't think the feature should be optional. We could provide a 
function name and then describe what the function must do/provide 
example code -- which, I assume, is what Gregg was suggesting. We could 
also define the test function as one that just returns test contexts 
from an in-memory map.

-- 
Dave Longley
CTO
Digital Bazaar, Inc.