Re: Test suite for services approach

Hello Thierry,

thank you for your mail. I see now that my proposal needs more
explanatations. I am aware of the difference between a test suite and an
implementation report. However I think in the case of our working group, we
have a rather simple scenario for test cases:

1) we have input data, that is files with multimedia objects
2) we have MA properties to be found in that input data, e.g. the identifier
of a mutimedia object
3) we have an ideal output of 3)
4) we have output produced by the actual implementation

The page I produced is a collection of currently dead links to test cases in
this structure. I think that the assertions we make in the API document are
so simple that we don't a need more complex structure in the test suite, and
that hence it can be closely related to the implementation report. Btw.,
this design is based on a test suite / implementation report at
http://www.w3.org/International/its/tests/
which was used for the Internationalization Tag Set, see
http://www.w3.org/TR/its/ , and it moved to PR with that. The test cases are
now also used for furture implementation testing.

Best,

Felix

2010/2/19 Thierry MICHEL <tmichel@w3.org>

>
> Felix,
>
> Thanks for your proposal for a test suite design.
>
> I guess what you are providing is the implementation report (based on the
> Test suite).
>
> - The test suite is a collection a testcases.
> - The implementation is a table that shows which testcases of the test
> suite are passed (or failed) by each implementation.
>
> The distinction is important because
> - the purpose of the implementation reports is to show to the Director that
> the WG has fulfill the CR exit criteria, and therefore move to PR.
> - The test suite may allow further implementations (for example after PR or
> REC, to test their product.
>
> Here are a few examples I have experienced in different WGs
>
> * the SYMM WG
>
> - testsuite
>
> http://www.w3.org/2005/SMIL21/testsuite/New-SMIL21/NewS21functionalities.html#Layout
> - impementation report
> http://www.w3.org/2005/SMIL21/SMIL21-implementation-result.html
>
> * the webCGM WG
>
> - testsuite
> http://www.w3.org/Graphics/WebCGM/2009/WebCGM21/testsuite21.html
> - impementation report
> http://www.w3.org/Graphics/WebCGM/2009/WebCGM21/new21-matrix.html
>
>
>
>
>
>
>
>
>
>
>
>
>  Sasaki wrote:
>
>> Hi all,
>>
>> at http://fabday.fh-potsdam.de/~sasaki/mawg/mawg-testsuite-proposal.html<http://fabday.fh-potsdam.de/%7Esasaki/mawg/mawg-testsuite-proposal.html>there is a proposal for a test suite design, see ACTION-213 <
>> http://www.w3.org/2008/WebVideo/Annotations/track/actions/213>. It
>> contains only a pattern for a test for one property, but the pattern should
>> be applicable to all properties:
>>
>> - give each property a row in the table
>> - have a column with the source. If we have several formats to be tested,
>> we would need several rows for the property
>> - have a column with the expected results.
>> - have further columns for other results.
>> Please note the proposed directory structure which you can see by hovering
>> other the links, with directories for input files, expected output, and
>> actual output for each implementation.
>>
>> Best,
>>
>> Felix
>>
>