Re: paper that Jose was talking about today

The benchmarking process may be of interest yet then there is no need to 
print tables of actual numbers, which (surprise!) show that your ShEx 
implementation is 20 times faster than my current SHACL prototype. Of 
course I could make mine orders of magnitude faster by hard-coding the 
core language instead of turning them into many small SPARQL queries. 
The paper is comparing apples with oranges.

Anyway, you are free to publish whatever you want, like we are free to 
post our own propaganda on the web. I find it just deplorable that this 
is labeled academic research.

Holger


On 16/12/2015 10:56 AM, Eric Prud'hommeaux wrote:
> * Holger Knublauch <holger@topquadrant.com> [2015-12-16 10:25+0100]
>> I am disappointed that this paper includes performance comparisons
>> including my experimental prototype. I made it very clear that the
>> current implementation is not optimized at all, so what is the point
>> of printing this? The academic value of this comparison is nil.
> The paper explicitly states that it was testing the validator on
> "preliminary implementations":
>
> Abstract: [[
>    We then performed some preliminary experiments comparing performance
>    of two validation engines based on Shape Expressions and SHACL
>    respectively against the proposed benchmark.
> ]]
>
> Section 8: [[
>    While these results are calculated using early betas of both SHACL and
>    ShEx, it demonstrates how wiGen can be used for evaluation of
>    validation tools and algorithms.
> ]]
>
> It goes on to describe how the tool hilights negative performace
> behavior of ShEx (not SHACL) [[
>    The results show for instance that the ShEx implementation's
>    calculation time grows considerably when validating many Datasets,
>    perhaps because the shape is recursive (validating Datasets requires
>    validating Observations which in turn validates other Datasets).
> ]]
>
> and further describes how the tool will be useful [[
>    The wiGen tool can be scripted to explore many relevent parameters:
>    size of the validation graph, number of nodes to be validated,
>    interrelations between nodes in recursive shapes.  This will permit
>    principled desing choices in language developement and tool selection
>    and ultimately contribute to improved quality in Linked Data.
> ]]
>
> concluding with one more emphasis that these are preliminary results [[
>    The take-home message from this very preliminary evaluation is that,
>    while the performance figures still leave much to be desired,
>    reasonable performance is definitely reachable using either the ShEx
>    or SHACL approach.
> ]]
>
> It seems very clear from this that a benchmarking tool is useful,
> both to users and developers, and that waiting for the specs and
> implementations to be done simply deprives the community of
> valuable input. This tool is an opportunity, not a threat.
>
>
>> Holger
>>
>>
>> On 3/12/2015 10:07 PM, Peter F. Patel-Schneider wrote:
>>> The paper that Jose was talking about today is submitted to Semantic Web –
>>> Interoperability, Usability, Applicability and can be found on their "Under
>>> Review" page http://www.semantic-web-journal.net/underreview
>>>
>>> Look for Validating and Describing Linked Data Portals using Shapes
>>>
>>>
>>> peter
>>>
>>

Received on Wednesday, 16 December 2015 10:26:42 UTC