Re: Socio technical/Qualitative metrics for LD Benchmarks

Thank you Adam

Very interesting points and as you note,  lots and lots of big big issues
there

Scoping is a powerful tool for research management - also a political
research instrument

I appreciate that keeping a tight grip on 'scope' in academic efforts, is a
way to ensure that a project is feasible and technically defensible, and to
avoid that a potentially interesting candid suggestion results in an
explosive  open ended cans of worms that nobody (especially a narrowly
focused computer scientist) would know how to handle.
The discussion about - listhub.w3c.org "rdf semantics and timelessly true"
is fundamental, fascinating and challenging, but difficult to tackle. that
does not mean we can avoid it, although i can understand why one would try
to avoid it.

I also agree that there could scope for one, or ten thousands more funded
projects to deal with these issues in relation to linked data and semantic
web etc, so maybe another consortium could be formed, and this is not
something people like to discuss on public mailing lists.

But as you note, the discussion about the socio-technical aspects is too
 big and too relevant to technical project to be simply dismissed as 'out
of scope' purely  as a defensive measure to ensure that people who have the
project in hand remain capable of managing it.

It would be a mistake  to declare such considerations out of scope, when we
already have the situation in place (the project the people the funding
etc) because one particular member of a consortium refuses to take an
interest in them.

Since this particular project has just been announced, and the community of
practice /vendors/researchers in the field have been invited to present use
cases and provide input to the project , this is the time to pitch in

Any later, or any lack of due consideration could result in another eu
funded semantic web research project missing the mark entirely and result
in possibly misleading outcome.

PDM


On Fri, Nov 23, 2012 at 1:18 AM, adasal <adam.saltiel@gmail.com> wrote:

> Reading an email from newsletter@skillsmatter.com and following a link to Light
> Table <http://www.kickstarter.com/projects/ibdknox/light-table> , a very
> interesting project, I found the inspiration from Bret Victor (
> worrydream.com) who gives a wonderful talk found here
> http://vimeo.com/36579366 (cusec 2012)
> I strongly urge everyone to view this video. Scan the comments to see the
> enthusiasm with which it is received.
> Hi s v. brief bio is: Bret Victor invents tools that enable people to
> understand and create. He has designed experimental UI concepts at Apple,
> interactive data graphics for Al Gore, and musical instruments at Alesis.
>
> Bret Victor gives a well thought out, very reasonable, moving talk about a
> subject which is, for him, entirely within the technical sphere.
> He divides his ideas up into Principle, Identity and Insight.
> Bret's Principle is:-
> Creators need an immediate connection to what they create.
>
> The fascinating thing for me is the way in which he elaborates his theme.
> While, on the one hand, he is talking about something like the way an
> artist relates to the easel through the paint brush or the use of
> scaffolding in the creation of the Sistine Chapel, he is also talking about
> moral guiding principles. Human principles.
>
> We do not get into why there are moral dilemas in life - although I would
> think that is something he has thought about as he has developed. Nor is he
> anything but quite reasonable and modest about his suggestions.
>
> Nevertheless, he touches on the issue of the social context and the
> computing environment quite a few times and it is clear that this is a
> theme that deserves a lot of attention.
>
> It is the same area that I have been thinking about off and on, for the
> last year and a half.
> I wont use this email to go too far into this, but I will attempt to make
> a few salient points.
>
>  Society only exists in its tensions and those often manifest as moral
> dilemas.
> A road cuts through beautiful country side but we still wish to use the
> road for all sorts of reasons.
> Where we consider roads and country side such issues may arouse a spectrum
> of reactions from little to great passion.
> But when it comes to computing we do not know to what the analogy refers,
> if anything at all.
>
> I suggest that this is the first hurdle, the first moral dilema. It maybe
> the greatest one, in fact, to acknowledge that there are issues at stake.
> Equivalent to the women's suffrage fought for by Elizabeth Cady Stanton in
> the 19 c. US, a life example he gives.
>
> I had thought that Paola might be referring to something like UI's that
> make data ingestion more possible for people.
>
> Of course many vendors are working very hard exactly in this area.
>
> So we must be talking about something more essential than that.
> To cut to the point the analogy to the road (part of a road system) that I
> have in mind is the available data on the Internet.
> There is no need to push the analogy.
> What I want to say about it is that it has a potential for harm as well as
> for good and one of our first duties (yes, Bret Victor uses that word) is
> to look for and evaluate these potentials.
> In talking about potential good and harm of data available over the
> Internet there are many possible factors to consider.
> For instance there are issues of veracity and trust.
> There are also issues of ownership.
> Then there is machine processing and, finally, human consumption.
> These are incredibly difficult issues to begin to evaluate.
> Just briefly, take the one of veracity - this comes up in a separate
> thread and is mentioned by Henry Story and others.
> - listhub.w3c.org "rdf semantics and timelessly true"
> While one may want veracity, one does not want to mandate it, or else no
> stories, or mistakes, could ever be made available in data represented on
> the Internet.
>
> However, we would admit to some harm where people are made to believe what
> is not true to the advantage of those making that manipulation or in the
> pay of those so advantaged. Or would we? (Clearly there is a psychological
> and economic, capitalist, dimension to these issues too.)
>
> Now we come to the nitty gritty, and I think it instructive to turn again
> to the video I am mentioning.
> Bret Victor demo's immediate feedback from programs he is using. Take the
> one of the video game.
> Now imagine a couple of extra dimensions - represented by additional
> characters.
> Map those characters into graph traversals performing deep IR.
> Each traversal could represent a state of affairs that one may wish exists
> or over which one wishes to establish the possible degree of influence.
>
> In other words we now have a dynamic means of querying huge data sets (big
> data). Bret Victor's video allows us to envisage what this might be like.
> And the question remains:to what end is this being put?
> This is where principal, value and moral dilema comes in.
>
> And, I believe, that these are issues that will have to be confronted full
> on, no matter that at the moment there is a sort of dissociation from this
> reality by differing to 'business principles' and a valorisation of geekdom.
>
> Adam
>
>
>
> On 21 November 2012 21:22, Giovanni Tummarello <
> giovanni.tummarello@deri.org> wrote:
>
>> > Example:
>> >
>> > A technical benchmark that isolates say, performance, such as load
>> speed, is
>> > pointless, unless we can compute that the outcome of the query is
>> actually
>> > 'accurate' (true).
>> >
>> > so a technical parameter such as 'speed of resolving the query' is only
>> > meaningful if related to 'accuracy of the outcome', yet accuracy is not
>> a
>> > black/white thing.
>>
>> Here my geek background helps, i am positive you're off. DB results
>> (SQL, SPARQL etc)  are always accurate "find me all entities that have
>> this or that property". The closest you get is when you have some
>> OPTIONAL clause .. "if possible this or that" and you might be able to
>> use that to rank
>>
>> Information retrieval systems are much more lose, they stress
>> "ranking" and top results. Competitions like the Yahoo Semantic Search
>> are appropriate http://semsearch.yahoo.com/  (or were , given that
>> there was no 2012 not sure why..)
>>
>> >
>> > This is how we ought to model a technical benchmark, making sure the
>> > technical parameters we measure are not purely hot air costing the
>> public
>> > tons of good money.
>>
>> Beware  getting yourseld into the hot air productionnow. If you're
>> saying we should test ranking quality of semantic information
>> retrieval system i am with you, but measuring ranking can be done in
>> technical terms with well known methods
>>
>>
>> > Everyone I have spoken with in the consortium agrees that the technical
>> > parameters need to be wrapped into broader common sense issue, in
>> particular
>> > I had great conversations who people who showed support, agreement and
>> would
>> > be interested to see these views incorporated in the project since what
>> I
>> > suggested is perfectly in scope.
>> >
>>
>> i am not sure about the views at this point. I thought you wanted to
>> benchmark usefullness/ROI or similar of using linked data
>> technologies. but if you're saying "results are correct" i dont get
>> it.
>>
>> > I feel sorry about the lack of credibility of EU semantic web research,
>> and
>> > I think since so many researchers benefit from its generosity, they
>> have to
>> > shut up.
>>
>> in general i see where you're coming from and i think you're
>> partially right.
>>
>> There should be a peer review mechanism that downgrades the ability
>> for those that executed projects that really led to nothing to get
>> more funding. You'll get an entirely different attention to the "what
>> they hell we'll be doing, for real" as opposed to just passing
>> immediate short term reviews which allwant to see passing anyway.
>>
>> in the specific case however it is a matter of articulating your
>> proposition in a understandable way and defending it rationally. Going
>> off the tangent and accusing "the system" rarely will get you
>> anywhere. The system is stronger than you.
>>
>>
>> >
>> > see above. a technical benchmark isolated by other factors is
>> meaningless,
>> > therefore more  waste of public money
>> >
>>
>> its not meaningless. you might say "it risks missing out on important
>> factors without which one could miss the core point which is adoption
>> and societal usefullness"
>> it is still meaningful in technical term.
>>
>> In general, if you argument your point properly and make it public
>> e.g. on a blog or whatever and yur point makes sense i am sure that
>> the consortium will have to discuss this with their project officer
>> eventually
>>
>> > That brings up another issue: how are consortium decisions made... and
>> how
>> > are they documented, anyone who has worked with the EU knows that there
>> is
>> > some abuse going on in the system.... and no way of proving this is
>> taking
>> > place
>>
>> People play by the rules that are given. The rules should be changed
>> if we want to use better public money and obtain real benefits. Nobody
>> getting this money in large quantities due to well oiled mechanisms
>> will want to change the rules, really. If you speak with very smart
>> people at project officer level they silently nod when you say things
>> like the king is naked but its clear.. they really cant do much
>> themselves to change the rules. And in general its such a large
>> machine anyway.
>> Long stories, long path, but you wont even succeede in a few steps if
>> you dont take it easy and stick to factual, clear to understand well
>> motivated points (e.g. about your complaints above).
>>
>>
>> Gio
>>
>>
>