W3C home > Mailing lists > Public > semantic-web@w3.org > November 2012

Re: Socio technical/Qualitative metrics for LD Benchmarks

From: adasal <adam.saltiel@gmail.com>
Date: Fri, 23 Nov 2012 01:18:16 +0000
Message-ID: <CANJ1O4phm_9tDpHRm1SDJa0jHzvNv2OZNL8vpvv0T1spajiJeg@mail.gmail.com>
To: Giovanni Tummarello <giovanni.tummarello@deri.org>
Cc: Paola Di Maio <paoladimaio10@googlemail.com>, semantic-web at W3C <semantic-web@w3c.org>
Reading an email from newsletter@skillsmatter.com and following a link to Light
Table <http://www.kickstarter.com/projects/ibdknox/light-table> , a very
interesting project, I found the inspiration from Bret Victor (
worrydream.com) who gives a wonderful talk found here
http://vimeo.com/36579366 (cusec 2012)
I strongly urge everyone to view this video. Scan the comments to see the
enthusiasm with which it is received.
Hi s v. brief bio is: Bret Victor invents tools that enable people to
understand and create. He has designed experimental UI concepts at Apple,
interactive data graphics for Al Gore, and musical instruments at Alesis.

Bret Victor gives a well thought out, very reasonable, moving talk about a
subject which is, for him, entirely within the technical sphere.
He divides his ideas up into Principle, Identity and Insight.
Bret's Principle is:-
Creators need an immediate connection to what they create.

The fascinating thing for me is the way in which he elaborates his theme.
While, on the one hand, he is talking about something like the way an
artist relates to the easel through the paint brush or the use of
scaffolding in the creation of the Sistine Chapel, he is also talking about
moral guiding principles. Human principles.

We do not get into why there are moral dilemas in life - although I would
think that is something he has thought about as he has developed. Nor is he
anything but quite reasonable and modest about his suggestions.

Nevertheless, he touches on the issue of the social context and the
computing environment quite a few times and it is clear that this is a
theme that deserves a lot of attention.

It is the same area that I have been thinking about off and on, for the
last year and a half.
I wont use this email to go too far into this, but I will attempt to make a
few salient points.

Society only exists in its tensions and those often manifest as moral
dilemas.
A road cuts through beautiful country side but we still wish to use the
road for all sorts of reasons.
Where we consider roads and country side such issues may arouse a spectrum
of reactions from little to great passion.
But when it comes to computing we do not know to what the analogy refers,
if anything at all.

I suggest that this is the first hurdle, the first moral dilema. It maybe
the greatest one, in fact, to acknowledge that there are issues at stake.
Equivalent to the women's suffrage fought for by Elizabeth Cady Stanton in
the 19 c. US, a life example he gives.

I had thought that Paola might be referring to something like UI's that
make data ingestion more possible for people.

Of course many vendors are working very hard exactly in this area.

So we must be talking about something more essential than that.
To cut to the point the analogy to the road (part of a road system) that I
have in mind is the available data on the Internet.
There is no need to push the analogy.
What I want to say about it is that it has a potential for harm as well as
for good and one of our first duties (yes, Bret Victor uses that word) is
to look for and evaluate these potentials.
In talking about potential good and harm of data available over the
Internet there are many possible factors to consider.
For instance there are issues of veracity and trust.
There are also issues of ownership.
Then there is machine processing and, finally, human consumption.
These are incredibly difficult issues to begin to evaluate.
Just briefly, take the one of veracity - this comes up in a separate thread
and is mentioned by Henry Story and others.
- listhub.w3c.org "rdf semantics and timelessly true"
While one may want veracity, one does not want to mandate it, or else no
stories, or mistakes, could ever be made available in data represented on
the Internet.

However, we would admit to some harm where people are made to believe what
is not true to the advantage of those making that manipulation or in the
pay of those so advantaged. Or would we? (Clearly there is a psychological
and economic, capitalist, dimension to these issues too.)

Now we come to the nitty gritty, and I think it instructive to turn again
to the video I am mentioning.
Bret Victor demo's immediate feedback from programs he is using. Take the
one of the video game.
Now imagine a couple of extra dimensions - represented by additional
characters.
Map those characters into graph traversals performing deep IR.
Each traversal could represent a state of affairs that one may wish exists
or over which one wishes to establish the possible degree of influence.

In other words we now have a dynamic means of querying huge data sets (big
data). Bret Victor's video allows us to envisage what this might be like.
And the question remains:to what end is this being put?
This is where principal, value and moral dilema comes in.

And, I believe, that these are issues that will have to be confronted full
on, no matter that at the moment there is a sort of dissociation from this
reality by differing to 'business principles' and a valorisation of geekdom.

Adam



On 21 November 2012 21:22, Giovanni Tummarello <giovanni.tummarello@deri.org
> wrote:

> > Example:
> >
> > A technical benchmark that isolates say, performance, such as load
> speed, is
> > pointless, unless we can compute that the outcome of the query is
> actually
> > 'accurate' (true).
> >
> > so a technical parameter such as 'speed of resolving the query' is only
> > meaningful if related to 'accuracy of the outcome', yet accuracy is not a
> > black/white thing.
>
> Here my geek background helps, i am positive you're off. DB results
> (SQL, SPARQL etc)  are always accurate "find me all entities that have
> this or that property". The closest you get is when you have some
> OPTIONAL clause .. "if possible this or that" and you might be able to
> use that to rank
>
> Information retrieval systems are much more lose, they stress
> "ranking" and top results. Competitions like the Yahoo Semantic Search
> are appropriate http://semsearch.yahoo.com/  (or were , given that
> there was no 2012 not sure why..)
>
> >
> > This is how we ought to model a technical benchmark, making sure the
> > technical parameters we measure are not purely hot air costing the public
> > tons of good money.
>
> Beware  getting yourseld into the hot air productionnow. If you're
> saying we should test ranking quality of semantic information
> retrieval system i am with you, but measuring ranking can be done in
> technical terms with well known methods
>
>
> > Everyone I have spoken with in the consortium agrees that the technical
> > parameters need to be wrapped into broader common sense issue, in
> particular
> > I had great conversations who people who showed support, agreement and
> would
> > be interested to see these views incorporated in the project since what I
> > suggested is perfectly in scope.
> >
>
> i am not sure about the views at this point. I thought you wanted to
> benchmark usefullness/ROI or similar of using linked data
> technologies. but if you're saying "results are correct" i dont get
> it.
>
> > I feel sorry about the lack of credibility of EU semantic web research,
> and
> > I think since so many researchers benefit from its generosity, they have
> to
> > shut up.
>
> in general i see where you're coming from and i think you're
> partially right.
>
> There should be a peer review mechanism that downgrades the ability
> for those that executed projects that really led to nothing to get
> more funding. You'll get an entirely different attention to the "what
> they hell we'll be doing, for real" as opposed to just passing
> immediate short term reviews which allwant to see passing anyway.
>
> in the specific case however it is a matter of articulating your
> proposition in a understandable way and defending it rationally. Going
> off the tangent and accusing "the system" rarely will get you
> anywhere. The system is stronger than you.
>
>
> >
> > see above. a technical benchmark isolated by other factors is
> meaningless,
> > therefore more  waste of public money
> >
>
> its not meaningless. you might say "it risks missing out on important
> factors without which one could miss the core point which is adoption
> and societal usefullness"
> it is still meaningful in technical term.
>
> In general, if you argument your point properly and make it public
> e.g. on a blog or whatever and yur point makes sense i am sure that
> the consortium will have to discuss this with their project officer
> eventually
>
> > That brings up another issue: how are consortium decisions made... and
> how
> > are they documented, anyone who has worked with the EU knows that there
> is
> > some abuse going on in the system.... and no way of proving this is
> taking
> > place
>
> People play by the rules that are given. The rules should be changed
> if we want to use better public money and obtain real benefits. Nobody
> getting this money in large quantities due to well oiled mechanisms
> will want to change the rules, really. If you speak with very smart
> people at project officer level they silently nod when you say things
> like the king is naked but its clear.. they really cant do much
> themselves to change the rules. And in general its such a large
> machine anyway.
> Long stories, long path, but you wont even succeede in a few steps if
> you dont take it easy and stick to factual, clear to understand well
> motivated points (e.g. about your complaints above).
>
>
> Gio
>
>
Received on Friday, 23 November 2012 01:18:47 GMT

This archive was generated by hypermail 2.3.1 : Tuesday, 26 March 2013 21:45:52 GMT