Re: Socio technical/Qualitative metrics for LD Benchmarks

> Example:
>
> A technical benchmark that isolates say, performance, such as load speed, is
> pointless, unless we can compute that the outcome of the query is actually
> 'accurate' (true).
>
> so a technical parameter such as 'speed of resolving the query' is only
> meaningful if related to 'accuracy of the outcome', yet accuracy is not a
> black/white thing.

Here my geek background helps, i am positive you're off. DB results
(SQL, SPARQL etc)  are always accurate "find me all entities that have
this or that property". The closest you get is when you have some
OPTIONAL clause .. "if possible this or that" and you might be able to
use that to rank

Information retrieval systems are much more lose, they stress
"ranking" and top results. Competitions like the Yahoo Semantic Search
are appropriate http://semsearch.yahoo.com/  (or were , given that
there was no 2012 not sure why..)

>
> This is how we ought to model a technical benchmark, making sure the
> technical parameters we measure are not purely hot air costing the public
> tons of good money.

Beware  getting yourseld into the hot air productionnow. If you're
saying we should test ranking quality of semantic information
retrieval system i am with you, but measuring ranking can be done in
technical terms with well known methods


> Everyone I have spoken with in the consortium agrees that the technical
> parameters need to be wrapped into broader common sense issue, in particular
> I had great conversations who people who showed support, agreement and would
> be interested to see these views incorporated in the project since what I
> suggested is perfectly in scope.
>

i am not sure about the views at this point. I thought you wanted to
benchmark usefullness/ROI or similar of using linked data
technologies. but if you're saying "results are correct" i dont get
it.

> I feel sorry about the lack of credibility of EU semantic web research, and
> I think since so many researchers benefit from its generosity, they have to
> shut up.

in general i see where you're coming from and i think you're
partially right.

There should be a peer review mechanism that downgrades the ability
for those that executed projects that really led to nothing to get
more funding. You'll get an entirely different attention to the "what
they hell we'll be doing, for real" as opposed to just passing
immediate short term reviews which allwant to see passing anyway.

in the specific case however it is a matter of articulating your
proposition in a understandable way and defending it rationally. Going
off the tangent and accusing "the system" rarely will get you
anywhere. The system is stronger than you.


>
> see above. a technical benchmark isolated by other factors is meaningless,
> therefore more  waste of public money
>

its not meaningless. you might say "it risks missing out on important
factors without which one could miss the core point which is adoption
and societal usefullness"
it is still meaningful in technical term.

In general, if you argument your point properly and make it public
e.g. on a blog or whatever and yur point makes sense i am sure that
the consortium will have to discuss this with their project officer
eventually

> That brings up another issue: how are consortium decisions made... and how
> are they documented, anyone who has worked with the EU knows that there is
> some abuse going on in the system.... and no way of proving this is taking
> place

People play by the rules that are given. The rules should be changed
if we want to use better public money and obtain real benefits. Nobody
getting this money in large quantities due to well oiled mechanisms
will want to change the rules, really. If you speak with very smart
people at project officer level they silently nod when you say things
like the king is naked but its clear.. they really cant do much
themselves to change the rules. And in general its such a large
machine anyway.
Long stories, long path, but you wont even succeede in a few steps if
you dont take it easy and stick to factual, clear to understand well
motivated points (e.g. about your complaints above).


Gio

Received on Wednesday, 21 November 2012 21:23:33 UTC