Suggestion to re-consider exit criterion for CT guidelines

Dear BPWG participants,

Some news about the transition request for the Guidelines for Web 
Content Transformation Proxies to Candidate Recommendation.

On internal review of the transition request there was push back on the 
absence of a test case repository. Accordingly, I want to re-raise the 
proposal to bind exit criteria to passing tests of a yet-to-exist test 
suite.

More specifically, below is a list of suggestions that I think the group 
should consider, and adopt if possible. They would all affect the Status 
Of This Document (SOTD) section of the guidelines. Latest version of 
this SOTD, that I had prepared for the transition in accordance with the 
group's resolutions, is available at:
 
http://www.w3.org/2005/MWI/BPWG/Group/TaskForces/CT/editors-drafts/Guidelines/CR-ct-guidelines-20100427/#status


1. Link to a test case repository
-----
I had actually prepared a test case repository on W3C CVS servers along 
with a document that references the existing test cases:
  http://dev.w3.org/2010/ct-guidelines/test-suite/

First idea would be to say that this test case repository exists and 
link to that document from the SOTD section.



2. Nature of the test suite
-----
Not all the guidelines can be tested, and creating a complete test suite 
for those who can be is not easy either. We cannot just come up with a 
test suite that ensures that a proxy follows the HTTP RFC!

Thus, second idea would be to precise that the test suite is not 
intended to cover each and every aspect of each and every guideline, but 
is more to be viewed as a collection of challenge test cases on 
assertions for which such challenges may be created.

For instance, it is easy to write challenges to test assertions such as 
"[Proxies] MUST NOT delete header fields" (ta-13875). On the contrary, 
it is basically impossible to write test cases for "Proxies MUST assess 
whether a user's expressed preference for a restructured representation 
is still valid ..." (ta-38061).

Clarifying the fact that the test suite cannot be complete and is mostly 
to contain challenge test cases would make it possible:
- to ask for contributions from the community
- to note guidelines that cannot be tested and whose implementation can 
but rely on good faith assertions from implementers
- to actually finish the test suite!

Note that working on the test suite does not mean that the group needs 
to do the testing itself.



3. Commitment to work on the test suite
-----
The repository should at least contain one test case before the 
transition occurs.
The group should also commit to best effort to add tests to the 
repository during the Candidate Recommendation phase, but I think we 
already agreed that this latter point was a priority once the transition 
occur.



4. Bind exit criteria to test suite
-----
Once The test case repository has been identified and clarified in terms 
of content, final idea would be to amend the exit criterion and add the 
fact that a claim of conformance involves passing the challenge tests of 
the test repository, completed by good faith assertions that each of the 
mandatory requirements has been implemented. It is a combination 
approach between the group's current position and a position based on a 
complete test suite which we know cannot be achieved.


Dan, Jo, could you add this to next call's agenda?
(also, next call is on May 11th, should we rather hold a call next week 
on May 4th?)


Thanks,
Francois.

Received on Friday, 30 April 2010 14:01:14 UTC