Re: paper that Jose was talking about today

On 16/12/2015 2:42 PM, Eric Prud'hommeaux wrote:
> * Holger Knublauch <holger@topquadrant.com> [2015-12-16 12:33+0100]
>> On 16/12/2015 12:25 PM, Eric Prud'hommeaux wrote:
>>> * Holger Knublauch <holger@topquadrant.com> [2015-12-16 11:26+0100]
>>>> The benchmarking process may be of interest yet then there is no
>>>> need to print tables of actual numbers, which (surprise!) show that
>>> It also shows that our performance gets much worse as we run out of
>>> memory. We didn't adjust our implementation to deal with that because
>>> it wouldn't be fair to optimize our code without you getting a chance
>>> to do the same. If you want, we can help you set up with the benchmark
>>> and spend a couple weeks optimizing. That would change the tone of the
>>> paper from "here's a tool for benchmarking" to "here's a performance
>>> comparison of ShEx vs. SHACL".
>> I am personally not yet interested in optimizations while the
>> language is unstable. It is obvious that all kinds of optimizations
>> will be possible (for the core language) in the future, but I don't
>> have the bandwidth to work on such things right now.
>>
>>>
>>>> your ShEx implementation is 20 times faster than my current SHACL
>>>> prototype. Of course I could make mine orders of magnitude faster by
>>>> hard-coding the core language instead of turning them into many
>>>> small SPARQL queries. The paper is comparing apples with oranges.
>>> How would you propose we demonstrate that the benchmark tool runs and
>>> returns useful results?
>>>    run it on ShEx alone?
>>>    run it on SHACL alone?
>>>    add more disclaimers?
>> As it is printed right now, a casual reader will skim through the
>> table and see the bare numbers. It is not very clear that the SHACL
>> implementation is a proof of concept only. I believe if the focus is
>> on your benchmarking approach then you could simply compare the
>> various ShEx implementations, and not make this appear a
>> SHACL-vs-ShEx bake-off?
> I'm happy to do that, but I'd like your permission to point to this
> email to include a line like "upon request from the author, we're not
> including results for the SHACL proof-of-concept implementation." Does
> that work for you?

That would be OK. Better might be something along the lines of: "At the 
time of writing the TopBraid SHACL implementation was merely a 
proof-of-concept that was not at all optimized for performance and 
therefore did not make sense to be included in this comparison."

Holger

Received on Wednesday, 16 December 2015 13:54:42 UTC