Re: Test cases framework

I was hoping that we could try to solve this without adding more noise to
the WG mailing list.

By your email, can I assume that you want to change the test-suite format
that has already been accepted by the WG?

If that is the case, you should probably raise a issue proposing that
change and justifying the rationale for it.

You are proposing to remove manifest files because they are difficult to
maintain.

However, although I partially agree that they may be difficult to maintain,
I think manifest files help separate concerns and reuse shapes graphs and
instance data. Also, manifest files are a common practice in most of the
W3c specifications. For example, both RDF and SPARQL specifications define
their test-suites using manifest files [1], [2]. Why should we reinvent
something new when there is an accepted practice on this?

Your example: http://datashapes.org/testcases.html#ValidationTestCase lacks
information about which nodes are going to be validated and against which
shapes they are going to be validated. Where is that information?

In that example, it appears that two violation erros must be reported for
"ex:InvalidResource1", but where is it declared that the SHACL processor
must validate "ex:invalidResource1" against "ex:TestShape"?

Also, in that example, "ex:ValidResource" seems to be a valid
resource...where is it declared that it must be validated against
"ex:TestShape"?

I would assume that the information about those validations should be
declared in the manifest file.

In fact, I would have defined the shapes graph in a different file as well
as the instance data so they could be reused for other tests.

Best regards, Jose Labra

[1] https://www.w3.org/TR/rdf11-testcases/
[2] http://www.w3.org/2009/sparql/docs/tests/README.html

On Thu, Sep 29, 2016 at 2:10 AM, Holger Knublauch <holger@topquadrant.com>
wrote:

> (Moved thread back to mailing list, I hope that's OK)
>
> On 29/09/2016 6:28, Jose Emilio Labra Gayo wrote:
>
> As I said yesterday during the WG call, I have been working on a
> translator that could map part of the shex test-suite to SHACL with the
> goal to add the resulting tests to the test-suite.
>
> One problem that I am finding is that I am not sure if all of you agree on
> the expected result of what is a test-suite.
>
> My expectation was to have a battery of tests following the format
> described here:
>
> http://w3c.github.io/data-shapes/data-shapes-test-suite/
>
> That format was proposed and approved in a WG meeting some time ago. Do
> you agree with it now?
>
> I am asking because in the meantime, I have seen several tests added to
> the repository:
>
> https://github.com/w3c/data-shapes/tree/gh-pages/data-
> shapes-test-suite/tests
>
> which don't follow that format.
>
> I have also seen this page:
>
> http://datashapes.org/testcases.html
>
> but I am not sure how does it fit here...Holger, are you proposing that we
> use that tool to generate the tests? What would be the role of that tool?
> (I am asking before installing it).
>
>
> The link above shows a tool but is primarily about an RDF data
> model/ontology to represent test cases. An example of this format is
>
>     http://datashapes.org/testcases.html#ValidationTestCase
>
> This format is more compact than the one that we had proposed in the WG
> last year. In particular it no longer requires the overhead of the manifest
> metadata - it will simply execute all test cases in a given folder. I am
> rather negative about the design to have the expected validation results in
> the separate manifest file - this will lead to a maintenance nightmare.
>
> But otherwise it is very similar. If your starting point is your own
> format (from ShEx) anyway, then potentially it shouldn't matter too much to
> which format you write your converter for?
>
> Overall I would appreciate other opinions on whether the format from
> datashapes.org could be an option or not. As said before, I am too biased
> because I wrote it myself.
>
> Holger
>
>
>
>
> In principle, if everyone agrees with the proposed format or something
> similar, I volunteer to add more tests and to edit/clean the contents of
> the repository, but I would like to know, if we are all on the same page
> here.
>
> Dimitris, are you also still interested in working on the test-suite? If
> not, maybe Mark could work with me here (I would prefer not to be alone in
> this task)
>
> Best regards, Jose Labra
> PS. I prefer to send this email without using the mailing list to avoid
> adding more noise.
>
>
>
>
>
> On Wed, Sep 28, 2016 at 1:29 AM, Holger Knublauch <holger@topquadrant.com>
> wrote:
>
>> Answering Mark, the process that I am using is already functional for
>> everyone with tool support from TopBraid Composer (Free Edition) 5.2. Just
>> place the test files into the workspace, open the Test Cases view and press
>> the run button as described in http://datashapes.org/testcases.html
>>
>> While I personally find this tool support and format very convenient
>> (including the ability to create new tests with a single button click), I
>> am of course too biased to promote or push this format further. However, it
>> should be easy to write translators between the officially proposed test
>> format and the one in the dash namespace. So if the WG officially decides
>> on the format, I will provide such a translator because I'll need it myself.
>>
>> Holger
>>
>>
>>
>> On 28/09/2016 6:49, Jose Emilio Labra Gayo wrote:
>>
>> During today's meeting we were talking about the test-suite.
>>
>> My goal is to work on the test-suite once the spec seems to be more
>> stable. I am now working in a translator that will convert part of the ShEx
>> test-suite to SHACL. I hope to be able to provide this new battery of tests
>> in the next 2/3 weeks
>>
>> In principle, I was considering to maintain the structure of the
>> test-suite that had been approved by the WG some time ago.
>>
>> I think it would be better to keep that structure and to add more tests
>> following it...although if the WG considers that we should use a different
>> approach, maybe we should discuss it before doing more work on it.
>>
>> Best regards, Jose Labra
>>
>>
>> On Tue, Sep 27, 2016 at 5:51 PM, mark <markh@metarelate.net> wrote:
>>
>>> I haven't seen any further discussion on this topic.
>>>
>>> I think it is a valuable thing to have a w3 maintained set of
>>> validation cases
>>>
>>> If no-one is actively using
>>> https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes-test-suite
>>> I would support retiring this (onto a branch, so it's not lost) and
>>> putting the current set of test cases from TopQuadrant into the W3 space
>>>
>>> I am happy to help administer this if it is a useful approach.
>>>
>>> Holger: If I stepped in to help, would you be happy to start from your
>>> current test cases and link your development flow into W3 test cases
>>> resources once it is functional?
>>>
>>> Are there other implementations that we should consider contacting
>>> directly to encourage them to use a new shared set of test resources
>>> who would help drive the development and maintenance of these?
>>>
>>> What would working group members make of this approach?
>>>
>>> all the best
>>> mark
>>>
>>>
>>> On Mon, 5 Sep 2016 15:33:52 +1000
>>> Holger Knublauch <holger@topquadrant.com> wrote:
>>>
>>> > Just to point out some work related to test cases for SHACL.
>>> >
>>> > The current library of (53) test cases that I am using to validate
>>> > the TopBraid SHACL API can be found at
>>> >
>>> > https://github.com/TopQuadrant/shacl/tree/master/src/test/re
>>> sources/sh/tests
>>> >
>>> > The vocabulary, including tool support, is described here:
>>> >
>>> >      http://datashapes.org/testcases.html
>>> >
>>> > Who else has any test cases that are tracking the spec?
>>> >
>>> > The work on an official test cases format has been dormant:
>>> >
>>> > https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes
>>> -test-suite
>>> >
>>> > Is anyone still "driving" that format and the corresponding tests?
>>> >
>>> > Thanks,
>>> > Holger
>>> >
>>> >
>>>
>>>
>>>
>>
>>
>> --
>> -- Jose Labra
>>
>>
>>
>
>
> --
> -- Jose Labra
>
>
>


-- 
Saludos, Labra