Re: paper that Jose was talking about today

* Holger Knublauch <holger@topquadrant.com> [2015-12-16 10:25+0100]
> I am disappointed that this paper includes performance comparisons
> including my experimental prototype. I made it very clear that the
> current implementation is not optimized at all, so what is the point
> of printing this? The academic value of this comparison is nil.

The paper explicitly states that it was testing the validator on
"preliminary implementations":

Abstract: [[
  We then performed some preliminary experiments comparing performance
  of two validation engines based on Shape Expressions and SHACL
  respectively against the proposed benchmark.
]]

Section 8: [[
  While these results are calculated using early betas of both SHACL and
  ShEx, it demonstrates how wiGen can be used for evaluation of
  validation tools and algorithms.
]]

It goes on to describe how the tool hilights negative performace
behavior of ShEx (not SHACL) [[
  The results show for instance that the ShEx implementation's
  calculation time grows considerably when validating many Datasets,
  perhaps because the shape is recursive (validating Datasets requires
  validating Observations which in turn validates other Datasets).
]]

and further describes how the tool will be useful [[
  The wiGen tool can be scripted to explore many relevent parameters:
  size of the validation graph, number of nodes to be validated,
  interrelations between nodes in recursive shapes.  This will permit
  principled desing choices in language developement and tool selection
  and ultimately contribute to improved quality in Linked Data.
]]

concluding with one more emphasis that these are preliminary results [[
  The take-home message from this very preliminary evaluation is that,
  while the performance figures still leave much to be desired,
  reasonable performance is definitely reachable using either the ShEx
  or SHACL approach.
]]

It seems very clear from this that a benchmarking tool is useful,
both to users and developers, and that waiting for the specs and
implementations to be done simply deprives the community of
valuable input. This tool is an opportunity, not a threat.


> Holger
> 
> 
> On 3/12/2015 10:07 PM, Peter F. Patel-Schneider wrote:
> >The paper that Jose was talking about today is submitted to Semantic Web –
> >Interoperability, Usability, Applicability and can be found on their "Under
> >Review" page http://www.semantic-web-journal.net/underreview
> >
> >Look for Validating and Describing Linked Data Portals using Shapes
> >
> >
> >peter
> >
> 
> 

-- 
-ericP

office: +1.617.599.3509
mobile: +33.6.80.80.35.59

(eric@w3.org)
Feel free to forward this message to any list for any purpose other than
email address distribution.

There are subtle nuances encoded in font variation and clever layout
which can only be seen by printing this message on high-clay paper.

Received on Wednesday, 16 December 2015 09:56:15 UTC