RE: Test-Results and Systems Ontologies (was Re: AGENDA/LOGISTICS...)

On Thu, 28 Aug 2003, Charles White wrote:

>
> Hey All,
>
> The first question that comes to my mind is - do we want to report all
> results of tests or wimply those that passed? I believe that the exit
> criteria is that we report that all the tests have been passed by
> someone (or two someones), not which ones are passed, which ones timed
> out, which ones failed, etc.
>
> I am of the mind that we simply report who passed the tests, and
> possibly put that into severa; levels, such as, passedSyntaxCheck,
> passedTest, etc. Likewise for the length of time it took the run, etc. I
> amnot sure that is necessarily useful to get to exit.

My inclination would be to include more, rather than less, information
(but see a contradictory opinion below :-). Although it's not perhaps
critical in terms of the exit criteria, I think knowing the difference
between a test that failed because the system gave the *wrong* result and
one that failed because the reasoner ran out of time/space or was simply
unable to make a decision is a useful one. Timings may also be interesting
-- knowing that a reasoner passes all the tests but requires a
supercomputer to run for a week to do so would be useful information :-)

> > Sandro writes:
> > > Sean writes:
> > > > On the telecon, Dan mentioned the possibility of marking
> > up/publishing
> > > > results using RDF. This sounds like a sensible idea and
> > it would be
> > easy
> > > > for me to produce this. Would it be possible to extend the test
> > ontology
> > > > [2] to include descriptions of test results?
> > > >
> > > > [1] http://wonderweb.man.ac.uk/owl/first-order.shtml
> > > > [2] http://www.w3.org/2002/03owlt/testOntology
> > >
> > > I've been working on this, playing around with some
> > possibilities.  I
> > > don't have the ontology or presentation code done, so I've
> > just jotted
> > > down some notes, below.   I'm thinking of this as a more general
> > > test-results ontology, not really part of the OWL testOntology.
> > >
> > >
> > > class TestRun
> > > subclasses PassingRun, FailingRun, IncompleteRun
> > >
> > >       I'm thinking "incomplete" covers the "Unknown" result,
> > >       as well as various sorts of resource-limits, or other
> > >       reported errors.   None of these are as bad as an incorrect
> > >       result being reported as if it were correct, which would
> > >       be a FailingRun
> > >
> > > has properties:
> > >
> > >    test -- the specific test being run here
> > >    begins -- point in time the run of this test began
> > (xsd:dateTime [1])
> > >    ends -- point in time the run of this test ended
> > >    system -- the complete system being tested (see below)
> > >    output -- a foaf:Document [2] (aka web page) showing
> > more detailed
> > output
> >
> > That's an interesting proposal and I've tried to work
> > it out partially. For an example run (on my slow laptop
> > and fetching the files from a cache) see
> > http://lists.w3.org/Archives/Public/www-archive/2003Aug/att-00
> > 27/owl-tr.n3
> >
> > The important point however is to fix the namespace
> > and I took <http://www.w3.org/2003/08testresult#>
> > but it's really up to you/W3C to take one ;-)
> > The "system" and "output" are still to be improved
> > and I wonder where a proof argument could fit in ?!
> > (whatever that may be ;-))

Proof arguments is probably going into too much depth for these purposes.
Particularly as what constitutes a proof argument is very much system
dependent.

> > > For "system", I'm thinking it's important to distinguish between
> > > different releases, since of course the test results could be
> > > different, and to track what OS/Hardware it's running on -- or at
> > > least keep open the option of doing so.   I imagine this being in a
> > > different, "systems" ontology, which would be what the
> > implementation
> > > report would come from.   (This overlaps with what Charles
> > is working
> > > on; I'm not sure how much he's done, or how best to share the work
> > > here.)
> > >
> > > My thinking on systems is:
> > >
> > > a System consists of one or more components.  Usually one component
> > > would be the testFocusComponent -- the thing being tested -- while
> > > others would be incidental (but potentially important) bits like the
> > > OS and hardware.    The components are probably Releases, as in:
> > >
> > > class Release
> > > class Project
> > >
> > >    a Project has zero or more Releases
> > >
> > >    a Release may obsolete a previous one,
> > >    maybe be a StableRelease or DevelopmentRelease,
> > >    has a date, a version string, a label, ...
> > >
> > >    I think the Project has people and web pages
> > >    associated with it, as well as the main name
> > >    ("euler", "racer", ...).    I'm not sure if
> > >    programming language and OS information belongs
> > >    with the Project or the Release.
> > >
> > > Thoughts?

Sounds ok. Certainly tracking release/version of systems is important,
although as with all these things there is a danger that we try and do too
much detail :-). As a minimal information set that we need _now_ I would
suggest:

result: at least pass/fail, possibly more detail.
start date/time
end date/time
system (name/version/platform?)

	Sean

-- 
Sean Bechhofer
seanb@cs.man.ac.uk
http://www.cs.man.ac.uk/~seanb

Received on Friday, 29 August 2003 11:02:40 UTC