W3C home > Mailing lists > Public > public-data-shapes-wg@w3.org > September 2016

Re: Test cases framework

From: Holger Knublauch <holger@topquadrant.com>
Date: Thu, 29 Sep 2016 20:02:08 +1000
To: Jose Emilio Labra Gayo <jelabra@gmail.com>
Cc: "public-data-shapes-wg@w3.org" <public-data-shapes-wg@w3.org>
Message-ID: <1c2027ec-14f1-3fb1-2b1b-e8c9fb4a421a@topquadrant.com>
On 29/09/2016 19:47, Jose Emilio Labra Gayo wrote:
>
> On Thu, Sep 29, 2016 at 2:10 AM, Holger Knublauch 
> <holger@topquadrant.com <mailto:holger@topquadrant.com>> wrote:
>
>     (Moved thread back to mailing list, I hope that's OK)
>
>
> I was hoping that we could try to solve this without adding more noise 
> to the WG mailing list.
>
> By your email, can I assume that you want to change the test-suite 
> format that has already been accepted by the WG?

The work on our test case framework received very little attention for 
the last year or so. So I believe it's a valid question to at least take 
another fresh look.

>
> If that is the case, you should probably raise a issue proposing that 
> change and justifying the rationale for it.
>
> You are proposing to remove manifest files because they are difficult 
> to maintain.
>
> However, although I partially agree that they may be difficult to 
> maintain, I think manifest files help separate concerns and reuse 
> shapes graphs and instance data. Also, manifest files are a common 
> practice in most of the W3c specifications. For example, both RDF and 
> SPARQL specifications define their test-suites using manifest files 
> [1], [2]. Why should we reinvent something new when there is an 
> accepted practice on this?
>
> Your example: http://datashapes.org/testcases.html#ValidationTestCase 
> lacks information about which nodes are going to be validated and 
> against which shapes they are going to be validated. Where is that 
> information?

The tests simply validate all target nodes based on the usual SHACL 
target mechanism. In this particular case, ex:TestShape is both a class 
and a sh:Shape, which means that all its instances will be validated 
against that shape. In other tests, I am using things like sh:targetNode 
and sh:targetClass.

>
> In that example, it appears that two violation erros must be reported 
> for "ex:InvalidResource1", but where is it declared that the SHACL 
> processor must validate "ex:invalidResource1" against "ex:TestShape"?
>
> Also, in that example, "ex:ValidResource" seems to be a valid 
> resource...where is it declared that it must be validated against 
> "ex:TestShape"?
>
> I would assume that the information about those validations should be 
> declared in the manifest file.

See above, no manifest file necessary.

>
> In fact, I would have defined the shapes graph in a different file as 
> well as the instance data so they could be reused for other tests.

Yes, this can be achieved via owl:imports, again using standard mechanisms.

Admittedly, a downside of my current architecture is that the assumption 
is that shapes graph == data graph. This is not always the case and 
would need to be refined by introducing another property that would be 
attached to the TestCase instance.

Look, I don't want to start longish debates. I am fine either way. It's 
more important that we actually have test cases.

Holger


>
> Best regards, Jose Labra
>
>
>
>
>
>
>
> [1] https://www.w3.org/TR/rdf11-testcases/
> [2] http://www.w3.org/2009/sparql/docs/tests/README.html
>
>
>     On 29/09/2016 6:28, Jose Emilio Labra Gayo wrote:
>>     As I said yesterday during the WG call, I have been working on a
>>     translator that could map part of the shex test-suite to SHACL
>>     with the goal to add the resulting tests to the test-suite.
>>
>>     One problem that I am finding is that I am not sure if all of you
>>     agree on the expected result of what is a test-suite.
>>
>>     My expectation was to have a battery of tests following the
>>     format described here:
>>
>>     http://w3c.github.io/data-shapes/data-shapes-test-suite/
>>     <http://w3c.github.io/data-shapes/data-shapes-test-suite/>
>>
>>     That format was proposed and approved in a WG meeting some time
>>     ago. Do you agree with it now?
>>
>>     I am asking because in the meantime, I have seen several tests
>>     added to the repository:
>>
>>     https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes-test-suite/tests
>>     <https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes-test-suite/tests>
>>
>>     which don't follow that format.
>>
>>     I have also seen this page:
>>
>>     http://datashapes.org/testcases.html
>>     <http://datashapes.org/testcases.html>
>>
>>     but I am not sure how does it fit here...Holger, are you
>>     proposing that we use that tool to generate the tests? What would
>>     be the role of that tool? (I am asking before installing it).
>
>     The link above shows a tool but is primarily about an RDF data
>     model/ontology to represent test cases. An example of this format is
>
>     http://datashapes.org/testcases.html#ValidationTestCase
>     <http://datashapes.org/testcases.html#ValidationTestCase>
>
>     This format is more compact than the one that we had proposed in
>     the WG last year. In particular it no longer requires the overhead
>     of the manifest metadata - it will simply execute all test cases
>     in a given folder. I am rather negative about the design to have
>     the expected validation results in the separate manifest file -
>     this will lead to a maintenance nightmare.
>
>     But otherwise it is very similar. If your starting point is your
>     own format (from ShEx) anyway, then potentially it shouldn't
>     matter too much to which format you write your converter for?
>
>     Overall I would appreciate other opinions on whether the format
>     from datashapes.org <http://datashapes.org> could be an option or
>     not. As said before, I am too biased because I wrote it myself.
>
>     Holger
>
>
>
>>
>>     In principle, if everyone agrees with the proposed format or
>>     something similar, I volunteer to add more tests and to
>>     edit/clean the contents of the repository, but I would like to
>>     know, if we are all on the same page here.
>>
>>     Dimitris, are you also still interested in working on the
>>     test-suite? If not, maybe Mark could work with me here (I would
>>     prefer not to be alone in this task)
>>
>>     Best regards, Jose Labra
>>     PS. I prefer to send this email without using the mailing list to
>>     avoid adding more noise.
>>
>>
>>
>>
>>
>>     On Wed, Sep 28, 2016 at 1:29 AM, Holger Knublauch
>>     <holger@topquadrant.com <mailto:holger@topquadrant.com>> wrote:
>>
>>         Answering Mark, the process that I am using is already
>>         functional for everyone with tool support from TopBraid
>>         Composer (Free Edition) 5.2. Just place the test files into
>>         the workspace, open the Test Cases view and press the run
>>         button as described in http://datashapes.org/testcases.html
>>         <http://datashapes.org/testcases.html>
>>
>>         While I personally find this tool support and format very
>>         convenient (including the ability to create new tests with a
>>         single button click), I am of course too biased to promote or
>>         push this format further. However, it should be easy to write
>>         translators between the officially proposed test format and
>>         the one in the dash namespace. So if the WG officially
>>         decides on the format, I will provide such a translator
>>         because I'll need it myself.
>>
>>         Holger
>>
>>
>>
>>         On 28/09/2016 6:49, Jose Emilio Labra Gayo wrote:
>>>         During today's meeting we were talking about the test-suite.
>>>
>>>         My goal is to work on the test-suite once the spec seems to
>>>         be more stable. I am now working in a translator that will
>>>         convert part of the ShEx test-suite to SHACL. I hope to be
>>>         able to provide this new battery of tests in the next 2/3 weeks
>>>
>>>         In principle, I was considering to maintain the structure of
>>>         the test-suite that had been approved by the WG some time ago.
>>>
>>>         I think it would be better to keep that structure and to add
>>>         more tests following it...although if the WG considers that
>>>         we should use a different approach, maybe we should discuss
>>>         it before doing more work on it.
>>>
>>>         Best regards, Jose Labra
>>>
>>>
>>>         On Tue, Sep 27, 2016 at 5:51 PM, mark <markh@metarelate.net
>>>         <mailto:markh@metarelate.net>> wrote:
>>>
>>>             I haven't seen any further discussion on this topic.
>>>
>>>             I think it is a valuable thing to have a w3 maintained
>>>             set of
>>>             validation cases
>>>
>>>             If no-one is actively using
>>>             https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes-test-suite
>>>             <https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes-test-suite>
>>>             I would support retiring this (onto a branch, so it's
>>>             not lost) and
>>>             putting the current set of test cases from TopQuadrant
>>>             into the W3 space
>>>
>>>             I am happy to help administer this if it is a useful
>>>             approach.
>>>
>>>             Holger: If I stepped in to help, would you be happy to
>>>             start from your
>>>             current test cases and link your development flow into
>>>             W3 test cases
>>>             resources once it is functional?
>>>
>>>             Are there other implementations that we should consider
>>>             contacting
>>>             directly to encourage them to use a new shared set of
>>>             test resources
>>>             who would help drive the development and maintenance of
>>>             these?
>>>
>>>             What would working group members make of this approach?
>>>
>>>             all the best
>>>             mark
>>>
>>>
>>>             On Mon, 5 Sep 2016 15:33:52 +1000
>>>             Holger Knublauch <holger@topquadrant.com
>>>             <mailto:holger@topquadrant.com>> wrote:
>>>
>>>             > Just to point out some work related to test cases for
>>>             SHACL.
>>>             >
>>>             > The current library of (53) test cases that I am using
>>>             to validate
>>>             > the TopBraid SHACL API can be found at
>>>             >
>>>             >
>>>             https://github.com/TopQuadrant/shacl/tree/master/src/test/resources/sh/tests
>>>             <https://github.com/TopQuadrant/shacl/tree/master/src/test/resources/sh/tests>
>>>             >
>>>             > The vocabulary, including tool support, is described here:
>>>             >
>>>             > http://datashapes.org/testcases.html
>>>             <http://datashapes.org/testcases.html>
>>>             >
>>>             > Who else has any test cases that are tracking the spec?
>>>             >
>>>             > The work on an official test cases format has been
>>>             dormant:
>>>             >
>>>             >
>>>             https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes-test-suite
>>>             <https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes-test-suite>
>>>             >
>>>             > Is anyone still "driving" that format and the
>>>             corresponding tests?
>>>             >
>>>             > Thanks,
>>>             > Holger
>>>             >
>>>             >
>>>
>>>
>>>
>>>
>>>
>>>         -- 
>>>         -- Jose Labra
>>>
>>
>>
>>
>>
>>     -- 
>>     -- Jose Labra
>>
>
>
>
>
> -- 
> -- Jose Labra
>
Received on Thursday, 29 September 2016 10:02:51 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 19:30:36 UTC