Re: [RIF-Test] RIF Test Cases

Thanks, Dave.  I agree with  your points, except just have a question on 
one of them:

  -- on the metadata:  are you saying  we don't need separate manifest
      files for each test case, but rather put all that information as 
metadata
      in the (or one of the) rule documents of the test?


Before we start going through the list of questions, we wanted to spend 
some time today discussing the overall purpose/mission of the test suite
and document.

Based on the charter statement  ("A set of Test Cases which reflect issue 
resolution
and which aid in conformance evaluation" ) and on past discussions, our 
main
purpose might be:

        To illustrate  the language and its semantics, including 
subtleties and
        corner cases,  and to be a  very good aid  (i.e. wide, although 
not complete,
        coverage)  in evaluating conformance of RIF processors.

        Also,  Adrian suggests we can provide a RIF test case format that 
allows users 
        to describe  their application specific test cases and test 
suites. These test cases
        can be interchanged  together with the rule programs in RIF and 
can be used to
        validate the interchanged  rule programs in the execution 
environments. That is,
        a kind of RIF test case dialect.

Stella




Dave Reynolds <der@hplb.hpl.hp.com> 
Sent by: public-rif-wg-request@w3.org
08/12/2008 04:21 AM

To
Adrian Paschke <Adrian.Paschke@gmx.de>
cc
public-rif-wg@w3.org
Subject
Re: [RIF-Test] RIF Test Cases







Lots of good questions in here which require thought but just wanted to 
react to a couple.

> 2. Normative or Not; Conformance suite or informative?
> 
> - Are test cases normative and if yes which categories / types are 
normative
> which not?
> - What does it mean to be conformant to the "normative" RIF tests?
> 
> The RIF charter requires us to deliver test cases which reflect issue 
resolution and  which aid  in conformance evaluation see
> 
http://www.w3.org/2005/rules/wiki/TCS#Conformance_suite_or_informative.3F

To me the test cases are normative but *not* a conformance suite. There 
should be no suggestion that the case cases are complete, nor that 
passing all the test cases constitutes conformance. They are just there 
to illustrate the corner cases and help developers gain confidence.

> 3. What does it mean to say that a RIF test is passed?
> 
> - Do we say it passes if  (a) we can express this premise, and (b) the
> semantics entails that all models that satisfy the premise satisfy the 
conclusion
>     ---- in BLD?
>     ---- in all dialects of RIF?
>     ---  in all languages that we expect can be translated into RIF or 
dialects of RIF?

Each test defines what it means to pass it. Some of the examples 
generated before were not full model checks they simply checked that a 
particular entailment was found or not found.

Tests are specific to RIF dialects. But presumably any extension of 
dialect D will pass all the tests for D (and if we produce a Core then 
all Core tests would be relevant for every dialect).

> 4. Presentation and representation of RIF test cases and test suites
> 
> -  Formal representation
> -  Concrete XML-based RIF syntax
> -  Human-oriented presentation syntax
> 
> see http://www.w3.org/2005/rules/wiki/TCS#RIF_test_case_structure
> and the RIF Test Case Format
> http://www.w3.org/2005/rules/wiki/Test_Case_Format for the existing test
> case examples on the test case category page
> http://www.w3.org/2005/rules/wiki/Category:Test_Case

Agreed with the suggestion in there.

> 5. Process of collection and releasing test cases in the RIF working 
group
> 
> - Shall we solicit test cases from the community or only the RIF working
> group?

The working group validates and curates the tests. If we can get any 
tests from the community that would be great but those should be checked 
and only included in the test suite at the WG's discretion. The suite 
needs to be deliberately designed by the working group to probe the 
corner cases.

> - Setup a repository for RIF test cases, like:
> http://www.w3.org/2002/03owlt/ or (re-)use the WIKI
> http://www.w3.org/2005/rules/wiki/Category:Test_Case

My preference would be for a simple file repository. Auto-generating 
wiki pages from the files would be a nice extra.  All the metadata about 
status etc should be part of the rule metadata.

Dave
-- 
Hewlett-Packard Limited
Registered Office: Cain Road, Bracknell, Berks RG12 1HN
Registered No: 690597 England