AW: [RIF-Test] RIF Test Cases

>  -- on the metadata:  are you saying  we don't need separate manifest 
>      files for each test case, but rather put all that information as
metadata 
>      in the (or one of the) rule documents of the test?

 

Since we have an expressive metadata mechanism in RIF, I would propose to
put all the info as metadata into a RIF rule document which defines a test
case. This will make it easier to describing test cases in RIF and
interchanging test cases together with the RIF rule sets.

 

However, as I explained today, to represent test cases directly in the
concrete XML syntax of a RIF dialect, we will probably need some extensions
or meta data annotations. For instance, to define the intended result of a
test case such as the variable-value-binding pairs (e.g. X=1, X=2, X=3), the
intended answer value (yes, no, unkown), the semantics which should be used
for the test case, the test assertions (e.g. test facts) which are used to
test a rule program, etc. 

 

- Adrian

 

 

 

  _____  

Von: public-rif-wg-request@w3.org [mailto:public-rif-wg-request@w3.org] Im
Auftrag von Stella Mitchell
Gesendet: Dienstag, 12. August 2008 14:50
An: Dave Reynolds
Cc: Adrian Paschke; public-rif-wg@w3.org; public-rif-wg-request@w3.org
Betreff: Re: [RIF-Test] RIF Test Cases

 


Thanks, Dave.  I agree with  your points, except just have a question on 
one of them: 

  -- on the metadata:  are you saying  we don't need separate manifest 
      files for each test case, but rather put all that information as
metadata 
      in the (or one of the) rule documents of the test? 


Before we start going through the list of questions, we wanted to spend 
some time today discussing the overall purpose/mission of the test suite 
and document. 

Based on the charter statement  ("A set of Test Cases which reflect issue
resolution 
and which aid in conformance evaluation" ) and on past discussions, our main

purpose might be: 

        To illustrate  the language and its semantics, including subtleties
and 
        corner cases,  and to be a  very good aid  (i.e. wide, although not
complete, 
        coverage)  in evaluating conformance of RIF processors. 

        Also,  Adrian suggests we can provide a RIF test case format that
allows users 
        to describe  their application specific test cases and test suites.
These test cases 
        can be interchanged  together with the rule programs in RIF and can
be used to 
        validate the interchanged  rule programs in the execution
environments. That is, 
        a kind of RIF test case dialect.

Stella 





Dave Reynolds <der@hplb.hpl.hp.com> 
Sent by: public-rif-wg-request@w3.org 

08/12/2008 04:21 AM 


To

Adrian Paschke <Adrian.Paschke@gmx.de> 


cc

public-rif-wg@w3.org 


Subject

Re: [RIF-Test] RIF Test Cases

 


 

 





Lots of good questions in here which require thought but just wanted to 
react to a couple.

> 2. Normative or Not; Conformance suite or informative?
> 
> - Are test cases normative and if yes which categories / types are
normative
> which not?
> - What does it mean to be conformant to the "normative" RIF tests?
> 
> The RIF charter requires us to deliver test cases which reflect issue
resolution and  which aid  in conformance evaluation see
> http://www.w3.org/2005/rules/wiki/TCS#Conformance_suite_or_informative.3F

To me the test cases are normative but *not* a conformance suite. There 
should be no suggestion that the case cases are complete, nor that 
passing all the test cases constitutes conformance. They are just there 
to illustrate the corner cases and help developers gain confidence.

> 3. What does it mean to say that a RIF test is passed?
> 
> - Do we say it passes if  (a) we can express this premise, and (b) the
> semantics entails that all models that satisfy the premise satisfy the
conclusion
>     ---- in BLD?
>     ---- in all dialects of RIF?
>     ---  in all languages that we expect can be translated into RIF or
dialects of RIF?

Each test defines what it means to pass it. Some of the examples 
generated before were not full model checks they simply checked that a 
particular entailment was found or not found.

Tests are specific to RIF dialects. But presumably any extension of 
dialect D will pass all the tests for D (and if we produce a Core then 
all Core tests would be relevant for every dialect).

> 4. Presentation and representation of RIF test cases and test suites
> 
> -  Formal representation
> -  Concrete XML-based RIF syntax
> -  Human-oriented presentation syntax
> 
> see http://www.w3.org/2005/rules/wiki/TCS#RIF_test_case_structure
> and the RIF Test Case Format
> http://www.w3.org/2005/rules/wiki/Test_Case_Format for the existing test
> case examples on the test case category page
> http://www.w3.org/2005/rules/wiki/Category:Test_Case

Agreed with the suggestion in there.

> 5. Process of collection and releasing test cases in the RIF working group
> 
> - Shall we solicit test cases from the community or only the RIF working
> group?

The working group validates and curates the tests. If we can get any 
tests from the community that would be great but those should be checked 
and only included in the test suite at the WG's discretion. The suite 
needs to be deliberately designed by the working group to probe the 
corner cases.

> - Setup a repository for RIF test cases, like:
> http://www.w3.org/2002/03owlt/ or (re-)use the WIKI
> http://www.w3.org/2005/rules/wiki/Category:Test_Case

My preference would be for a simple file repository. Auto-generating 
wiki pages from the files would be a nice extra.  All the metadata about 
status etc should be part of the rule metadata.

Dave
-- 
Hewlett-Packard Limited
Registered Office: Cain Road, Bracknell, Berks RG12 1HN
Registered No: 690597 England