Re: Suggestions for test cases framework

Thanks

Two suggestions:

- About the results format, I agree that a single true/false is not a good
idea, however, there may be some tests that want to detect just if some
data validate or not without looking at further details.

Also, although successful validation could be described as an empty list of
errors, I think it would be better to be more explicit. Furthermore, in
case that the data is valid it may be worth to check the list of
nodes/shapes that have been detected by the validation process.

Taking into account the previous possibilities, I propose to extend a
little bit the results associated with mf:result as follows:

1.- Test whose result is valid (without the need to provide more details)

mf:result [a sht:Valid ]

2.- Test whose result is not valid:

mf:result [ a sht:NotValid ]

3.- Test whose result is not valid result and that describes the list of
errors that must be generated:

mf:result [ a sht:NotValid;
                 sht:errors [ sht:error
                                     [ a sh:Error;
                                       sh:root ...
                                     ];
                                     [ a sh:Error;
                                      ...
                                    ]
                                ] ;
]

4.- Data whose result is valid and where we provide a list of node/shape
associations

mf:result [ a sht:Valid;
                sht:shapeAssociation
                           [ sht:node ...;
                             sht:shape ...
                           ];
                sht:shapeAssociation
                          [ sht:node ...;
                            sht:shape ...
                         ];
 ...
]

2.- Another suggestion that appeared in the last meeting was to include in
the tests a reference to the part of the spec that the test was testing. Of
course, one problem is that the spec document is not fixed yet, but once it
is fixed, we should add to any test a property:

sht:specRef <URI>

where <URI> could be an id in the Spec. In this way, we could facilitate
the generation of coverage reports.

The updated manifest file is here:

https://github.com/w3c/data-shapes/blob/gh-pages/data-shapes-test-suite/manifest.shc

Best regards, Jose Labra

On Tue, May 26, 2015 at 4:52 AM, Holger Knublauch <holger@topquadrant.com>
wrote:

> FYI I have added 17 test cases covering various SHACL features, grouped
> into Core and SPARQL:
>
>
> https://github.com/w3c/data-shapes/commit/2833fc2728dfd7816d85a54bdead249f430cf56d
>
> For the Java engine that I am working on, I have implemented a test
> framework for the provided format, and didn't encounter any issues so far.
>
> Thanks
> Holger
>
>
>
> On 5/22/2015 10:04, Holger Knublauch wrote:
>
>> Thanks to Jose and Dimitris to get the test suite started. A few
>> suggestions
>>
>> - sht:schema is probably not the best name - what about sht:shapesGraph
>> (and sht:dataGraph)?
>> - sht:schema-format and shr:data-format should be optional - can be
>> derived from file suffix
>> - If SHACL is expressed in itself then many sht:WellFormedSchema test
>> cases can be handled with sht:Validate
>> - ms:result cannot only be true or false. Some tests produce multiple
>> constraint violations, the violations may be warnings only, and we need to
>> be able to verify the error details. While true may be sufficient for some
>> tests and implementations, we will need the ability to point at a graph
>> with results, or (even better) allow the expected results to be stated
>> inline, e.g.
>>
>> mf:result [
>>     a sh:Error ;
>>     sh:subject ex:JohnDoe ;
>> ] ... (multiple values allowed)
>>
>> In those tests, sh:message should usually be optional.
>>
>> Thanks
>> Holger
>>
>>
>
>


-- 
-- Jose Labra