Re: AW: [RIF-Test] RIF Test Cases

Adrian Paschke wrote:
>>  -- on the metadata:  are you saying  we don't need separate manifest 
>>      files for each test case, but rather put all that information as
> metadata 
>>      in the (or one of the) rule documents of the test?
> 
>  
> 
> Since we have an expressive metadata mechanism in RIF, I would propose to
> put all the info as metadata into a RIF rule document which defines a test
> case. This will make it easier to describing test cases in RIF and
> interchanging test cases together with the RIF rule sets.

Agreed.

> However, as I explained today, to represent test cases directly in the
> concrete XML syntax of a RIF dialect, we will probably need some extensions
> or meta data annotations. For instance, to define the intended result of a
> test case such as the variable-value-binding pairs (e.g. X=1, X=2, X=3), the
> intended answer value (yes, no, unkown), the semantics which should be used
> for the test case, the test assertions (e.g. test facts) which are used to
> test a rule program, etc. 

I see what you mean now.  On the telecon I thought you were talking about 
designing a test-cases dialect for a rule language that is more expressive than 
BLD. Really you mean, I think, a set of standard meta-data "properties" to hold 
the test-case manifest.

I completely agree with that.

-Chris


> 
>  
> 
> - Adrian
> 
>  
> 
>  
> 
>  
> 
>   _____  
> 
> Von: public-rif-wg-request@w3.org [mailto:public-rif-wg-request@w3.org] Im
> Auftrag von Stella Mitchell
> Gesendet: Dienstag, 12. August 2008 14:50
> An: Dave Reynolds
> Cc: Adrian Paschke; public-rif-wg@w3.org; public-rif-wg-request@w3.org
> Betreff: Re: [RIF-Test] RIF Test Cases
> 
>  
> 
> 
> Thanks, Dave.  I agree with  your points, except just have a question on 
> one of them: 
> 
>   -- on the metadata:  are you saying  we don't need separate manifest 
>       files for each test case, but rather put all that information as
> metadata 
>       in the (or one of the) rule documents of the test? 
> 
> 
> Before we start going through the list of questions, we wanted to spend 
> some time today discussing the overall purpose/mission of the test suite 
> and document. 
> 
> Based on the charter statement  ("A set of Test Cases which reflect issue
> resolution 
> and which aid in conformance evaluation" ) and on past discussions, our main
> 
> purpose might be: 
> 
>         To illustrate  the language and its semantics, including subtleties
> and 
>         corner cases,  and to be a  very good aid  (i.e. wide, although not
> complete, 
>         coverage)  in evaluating conformance of RIF processors. 
> 
>         Also,  Adrian suggests we can provide a RIF test case format that
> allows users 
>         to describe  their application specific test cases and test suites.
> These test cases 
>         can be interchanged  together with the rule programs in RIF and can
> be used to 
>         validate the interchanged  rule programs in the execution
> environments. That is, 
>         a kind of RIF test case dialect.
> 
> Stella 
> 
> 
> 
> 
> 
> Dave Reynolds <der@hplb.hpl.hp.com> 
> Sent by: public-rif-wg-request@w3.org 
> 
> 08/12/2008 04:21 AM 
> 
> 
> To
> 
> Adrian Paschke <Adrian.Paschke@gmx.de> 
> 
> 
> cc
> 
> public-rif-wg@w3.org 
> 
> 
> Subject
> 
> Re: [RIF-Test] RIF Test Cases
> 
>  
> 
> 
>  
> 
>  
> 
> 
> 
> 
> 
> Lots of good questions in here which require thought but just wanted to 
> react to a couple.
> 
>> 2. Normative or Not; Conformance suite or informative?
>>
>> - Are test cases normative and if yes which categories / types are
> normative
>> which not?
>> - What does it mean to be conformant to the "normative" RIF tests?
>>
>> The RIF charter requires us to deliver test cases which reflect issue
> resolution and  which aid  in conformance evaluation see
>> http://www.w3.org/2005/rules/wiki/TCS#Conformance_suite_or_informative.3F
> 
> To me the test cases are normative but *not* a conformance suite. There 
> should be no suggestion that the case cases are complete, nor that 
> passing all the test cases constitutes conformance. They are just there 
> to illustrate the corner cases and help developers gain confidence.
> 
>> 3. What does it mean to say that a RIF test is passed?
>>
>> - Do we say it passes if  (a) we can express this premise, and (b) the
>> semantics entails that all models that satisfy the premise satisfy the
> conclusion
>>     ---- in BLD?
>>     ---- in all dialects of RIF?
>>     ---  in all languages that we expect can be translated into RIF or
> dialects of RIF?
> 
> Each test defines what it means to pass it. Some of the examples 
> generated before were not full model checks they simply checked that a 
> particular entailment was found or not found.
> 
> Tests are specific to RIF dialects. But presumably any extension of 
> dialect D will pass all the tests for D (and if we produce a Core then 
> all Core tests would be relevant for every dialect).
> 
>> 4. Presentation and representation of RIF test cases and test suites
>>
>> -  Formal representation
>> -  Concrete XML-based RIF syntax
>> -  Human-oriented presentation syntax
>>
>> see http://www.w3.org/2005/rules/wiki/TCS#RIF_test_case_structure
>> and the RIF Test Case Format
>> http://www.w3.org/2005/rules/wiki/Test_Case_Format for the existing test
>> case examples on the test case category page
>> http://www.w3.org/2005/rules/wiki/Category:Test_Case
> 
> Agreed with the suggestion in there.
> 
>> 5. Process of collection and releasing test cases in the RIF working group
>>
>> - Shall we solicit test cases from the community or only the RIF working
>> group?
> 
> The working group validates and curates the tests. If we can get any 
> tests from the community that would be great but those should be checked 
> and only included in the test suite at the WG's discretion. The suite 
> needs to be deliberately designed by the working group to probe the 
> corner cases.
> 
>> - Setup a repository for RIF test cases, like:
>> http://www.w3.org/2002/03owlt/ or (re-)use the WIKI
>> http://www.w3.org/2005/rules/wiki/Category:Test_Case
> 
> My preference would be for a simple file repository. Auto-generating 
> wiki pages from the files would be a nice extra.  All the metadata about 
> status etc should be part of the rule metadata.
> 
> Dave

-- 
Dr. Christopher A. Welty                    IBM Watson Research Center
+1.914.784.7055                             19 Skyline Dr.
cawelty@gmail.com                           Hawthorne, NY 10532
http://www.research.ibm.com/people/w/welty

Received on Thursday, 14 August 2008 13:53:19 UTC