Re: Monday 05 Jan QAWG telecon agenda

> 3.) Presentation by Sandro Hawke of the work done by the OWL
> WG and the RDF WG for the development of their test materials
> and the gathering of the tests results. A few references: 
> - OWL Test repository: http://www.w3.org/2002/03owlt/ 
> - OWL Test Cases document: http://www.w3.org/TR/owl-test/ 
> - OWL Test Results: http://www.w3.org/2003/08/owl-systems/test-results-out
> - RDF Test Cases: http://www.w3.org/TR/rdf-testcases/ 
> - RDF Test Results: http://www.w3.org/2003/11/results/rdf-core-tests
> 
> [hopefully Sandro will send other references to complete those]

Sorry, just getting back from vacation now.    Here are my notes for
the session.  

        -- sandro

===================================================================


1  Introduction      

    This is Mostly a Story about a Web Page

    Prolog: A Test-Driven Implementation of OWL ("Surnia")

    Effects the page has had

    What's next?

    About Me
        [ At home today (a bit of a cold, you may hear kids) ]
        - W3C Team, Semantic Web Activity, DARPA/Research Funding 
        - Joined WebOnt (OWL) in June
        
2  Surnia 

    - Immediate motivate: demonstrating implementability of OWL Full
    - I Didn't Read The Spec!  I just implemented based on my rough idea,
      then modified to pass more and more tests.
    - Results were decent 
    - Also for RDF Core entailment & non-entailment tests

3  The Test Results Page

    3.1 Test Results Ontology (for reporting results in RDF)
        - TestRun (PassingRun, FailingRun,  IncompleteRun / UndecidedRun)
           + which test
           + where is the output (details of the run)
           + which system was tested
             (NOT tracking Project/Release/Platform/etc)
           + time test began
           + test duration

    3.2 Test Results Page -- First Version
        SEE: http://www.w3.org/2003/08/owl-systems/res20030904
        - big table of tests/systems
        - regen at bottom
        - self-explanitory

    3.3 Feeds 
        - give me the URL of your test data
        - results page let people try it themselves, to do some debugging
        - some people advertised their results telling people to do a custom
          version of the page!
        - various bugs in their systems and a few in mine

    3.4 Test Results Refinements
        NOW:  http://www.w3.org/2003/08/owl-systems/test-results-out
        - group tests
        - leave out systems with no results for a group
        - summary table at top
        - group by number-passed/failed

4  Effects?

    4.1 Working Group
        - helped us decide which test to look at more closely, approve,
          move to extra-credit
        - let us push implementors on key tests

    4.2 Implementors
        - could see who was doing what, get some publicity
          (not enough system data per test, though, IMHO -- output link)

    4.3 W3C Members and Director
        - for advancement to PR -- Director liked it; I hope the members do

5  What's next?

    5.1 Test Suite stable now, as we're at PR
        - non-normative tests from WG?
        - tests from other submitters?

    5.2 Page usability
        - smaller bits!   searchable!
        - better info/links about implementations

    5.3 More info on page
        - benchmarking
        - change-over-time
        - nuances of test results (use of "output" link?)
        - facilitating *discussion* of tests

Received on Monday, 5 January 2004 10:29:59 UTC