Re: [Fwd: browsable test results]

On Fri, 26 Sep 2003, Sandro Hawke wrote:

>
>Charles McCathieNevile said:
>> Well, you've reinvented the axle it goes on...
>
>Ooops.  Sorry, I forgot all about EARL while doing this work.  Well,
>mostly it never sunk into my brain that you were positioning EARL for
>such broad use; I thought it was just about reporting issues with
>content.
>
>Of course my ontology is much simpler and probably somewhat easier to
>use.  I wonder when, if ever, it would pay off to for the OWL folk to
>use EARL extended as necessary....   I'm very, very happy with the
>adoption so far; I wish I knew if a more complex vocabulary would have
>hindered adoption.  I guess I can ask the current sources how they
>feel about such a change.

Well, there is some value in using the same things if we are trying to
promote interoperability. Of course there are several ways to get there.
(But it seems that saying "I've got a simpler ontology than Dublin Core, so
I'll use it" is a backward step. In fact I ahve argued as much in the
development of earl, and think there is more mileage to be made on using
common Classes or properties in the spec as it develops).

>Do you have an equivalent of my report generator which would make nice
>pages about test results if these folks had used EARL instead?  Or
>which would tell us which PROPOSED tests were passed by two systems,
>which APPROVED ones were passed by none, etc?

Not off hand, but if what we are talking about is simply two ontologies that
cover mostly the same ground it should be trivial to map any tool from one to
the other...

>> Your RDF models a lot less than EARL [1], and not much more (it does include
>> a specific property for the test duration, whereas EARL includes a general
>> comment property...)
>
>I don't see documentation for "cannotTell", so maybe it covers this,
>but one thing we need is to report when the system being tested failed
>to give any answer.  This is different from giving the right answer
>(Passing) or giving the wrong answer (Failing).  I called this
>"Incomplete" originally, but I'm now changing it to "Undecided".
>(That term is weak in that it suggests that the TESTER couldn't decide
>if the tested system passed or failed (which is probably what
>"cannotTell" means), but it matches many decades of decidability
>research.)  I'm not sure how the term could apply to content or even
>user agents.
>
>Why not just use rdfs:Comment instead of earl:message?

'cause the ER group currently doesn't have a charter son can't publish an
update to the spec... (I think it would be sensible to use rdfs:Comment in a
new version).

>> Other simple differences are not as many types of result,
>
>Yeah, I figured "notTested" would just be no data, and if the test was
>notApplicable, then why would it be tested?   That is, I don't think
>those actually are test results.

In some conformance assessment you need to state a result for each test, and
since some tests aren't applicable you can't always claim to have passed...
I suspect that it is (at least very often) a type of pass in fact - depends
on how well the test is designed. In the javascript thingy Danbri and I made
oh-so-long-ago we had a property called "OKFor" which covered both Pass and
NotApplicable.

>> the fact that there
>> is no RDF description of things being tested (earl defines webContent and I
>> think userAgent),
>
>That would be in an ontology of Tested-Things (which I think of as
>"Systems", but you need something broader to cover content), which I
>don't need yet.  Actually, I've worked on it a lot for OWL, but it's
>totally orthogonal.  If you include it in the test ontology, people
>may well think that's the only kind of thing the test ontology is for
>(like I did with EARL).

OK...

>> or the way that they were tested (automatically, manually,
>> or heurisitically - i.e. deriving a conformance result from other conformance
>> results).
>
>I'm not sure what those terms would mean for OWL.
>
>BTW, I think you're using the word "heuristic" when you mean
>(and say in the explanation) "derived", or perhaps "implied",
>"inferred", or "deduced".   For a test to be completed heuristically
>would, I think, mean that you guessed what the results would be and
>then verified that that's what they actually were.   That's probably
>not what you meant.

The property name is just a URI, ergo an opaque sequence of characters ;-)
Again, good editorial comment for the spec.

>> The important difference is that you have no notion of provenance in your
>> model. Provenance is built into the EARL model as a protection against
>> conflicting claims (which would cause your stuff to just have a
>> contradiction) or to enable choosing how to deal with conflicting claims
>> (trust management).
>
>Indeed.  Provenance is also orthogonal and does not belong in a
>test-results ontology, although I was tempted, too. [further explanation
snipped]

I agree that provenance is much more general than EARL, and ideally would
have been outside as a general feature. But the RDF Core group decided not to
do that, and we needed something simple and fast so we did one...

I don't see that it would be hard to make it interoperate with a newer,
better way of doing provenance. But EARL covers lots of times where you don't
necessarily trust all results, so it can't work without it. We know - we
tried...

cheers

Chaals

Received on Friday, 26 September 2003 22:39:00 UTC