Re: Suggestions for test cases framework

On 5/26/15 6:39 PM, Jose Emilio Labra Gayo wrote:
> Thanks
>
> Two suggestions:
>
> - About the results format, I agree that a single true/false is not a 
> good idea, however, there may be some tests that want to detect just 
> if some data validate or not without looking at further details.

Actually I think true/false is still OK, as an alternative to sh:Result 
values.

>
> Also, although successful validation could be described as an empty 
> list of errors, I think it would be better to be more explicit. 
> Furthermore, in case that the data is valid it may be worth to check 
> the list of nodes/shapes that have been detected by the validation 
> process.
>
> Taking into account the previous possibilities, I propose to extend a 
> little bit the results associated with mf:result as follows:
>
> 1.- Test whose result is valid (without the need to provide more details)
>
> mf:result [a sht:Valid ]
>
> 2.- Test whose result is not valid:
>
> mf:result [ a sht:NotValid ]

Sounds like an overkill to me. true/false would have been better, 
because easier to distinguish from result triples. Simply having no 
value for mf:result would be equivalent to true (i.e. valid).

>
> 3.- Test whose result is not valid result and that describes the list 
> of errors that must be generated:
>
> mf:result [ a sht:NotValid;
>                  sht:errors [ sht:error
>                                      [ a sh:Error;
> sh:root ...
>                                      ];
>                                      [ a sh:Error;
>                                       ...
>                                     ]
>                                 ] ;
> ]
>
> 4.- Data whose result is valid and where we provide a list of 
> node/shape associations
>
> mf:result [ a sht:Valid;
>                 sht:shapeAssociation
>                            [ sht:node ...;
>                              sht:shape ...
>                            ];
>                 sht:shapeAssociation
>                           [ sht:node ...;
>                             sht:shape ...
>                          ];
>  ...
> ]

I think if you want to capture node/shape associations then this should 
be independent of the mf:result, because it can apply to some nodes even 
if other don't validate. I'd suggest this becomes its own top-level 
property, but it should be optional, just like the result details are 
optional. Not all engines may support all these features.

>
> 2.- Another suggestion that appeared in the last meeting was to 
> include in the tests a reference to the part of the spec that the test 
> was testing. Of course, one problem is that the spec document is not 
> fixed yet, but once it is fixed, we should add to any test a property:
>
> sht:specRef <URI>
>
> where <URI> could be an id in the Spec. In this way, we could 
> facilitate the generation of coverage reports.

That sounds good.

Holger


>
> The updated manifest file is here:
>
> https://github.com/w3c/data-shapes/blob/gh-pages/data-shapes-test-suite/manifest.shc
>
> Best regards, Jose Labra
>
> On Tue, May 26, 2015 at 4:52 AM, Holger Knublauch 
> <holger@topquadrant.com <mailto:holger@topquadrant.com>> wrote:
>
>     FYI I have added 17 test cases covering various SHACL features,
>     grouped into Core and SPARQL:
>
>     https://github.com/w3c/data-shapes/commit/2833fc2728dfd7816d85a54bdead249f430cf56d
>
>     For the Java engine that I am working on, I have implemented a
>     test framework for the provided format, and didn't encounter any
>     issues so far.
>
>     Thanks
>     Holger
>
>
>
>     On 5/22/2015 10:04, Holger Knublauch wrote:
>
>         Thanks to Jose and Dimitris to get the test suite started. A
>         few suggestions
>
>         - sht:schema is probably not the best name - what about
>         sht:shapesGraph (and sht:dataGraph)?
>         - sht:schema-format and shr:data-format should be optional -
>         can be derived from file suffix
>         - If SHACL is expressed in itself then many
>         sht:WellFormedSchema test cases can be handled with sht:Validate
>         - ms:result cannot only be true or false. Some tests produce
>         multiple constraint violations, the violations may be warnings
>         only, and we need to be able to verify the error details.
>         While true may be sufficient for some tests and
>         implementations, we will need the ability to point at a graph
>         with results, or (even better) allow the expected results to
>         be stated inline, e.g.
>
>         mf:result [
>             a sh:Error ;
>             sh:subject ex:JohnDoe ;
>         ] ... (multiple values allowed)
>
>         In those tests, sh:message should usually be optional.
>
>         Thanks
>         Holger
>
>
>
>
>
>
> -- 
> -- Jose Labra
>