Test case review

(Splitting the threads)

On 05/10/2011 12:55 AM, Linss, Peter wrote:
>
> On May 9, 2011, at 2:58 PM, James Graham wrote:
>
>> On Mon, 9 May 2011, Linss, Peter wrote:
>>
>>> Shepherd is designed to be a web interface tightly integrated
>>> with our test suite repository. It'll facilitate reviewing,
>>> approving, and bug tracking of the test files as well as adding a
>>> query and editing system for the test case metadata. There are
>>> plans to also allow some degree of direct creation and editing of
>>> the tests in the web ui. It will also manage the layout of the
>>> test source files within the repository and integrate with our
>>> build system.
>>
>> That sounds interesting. Are there details anywhere? How tied is it
>> to CSS-specific assumptions (one tets per file, metadata embedded
>> in the test, etc.)
>
> There are some notes on our wiki at:
> http://wiki.csswg.org/test/review-system

Interesting. I have been thinking about similar issues and was 
considering a slightly different design. It seems that your approach is 
highly specialized for reviewing *tests*. That seems too specific to me 
because there are a great number of other things that one may have in a 
test repository that are not really tests in themselves. For example in 
the HTML5 test repository we have all of:

* Test harness files such as testharness.js
* Data files and scripts to generate test cases from the data files e.g. 
the html5lib tests
* Substantial pieces of javascript code that run tests from a small set 
of (also javascript) inputs e.g. the meta reflection tests
* Substantial pieces of javascript that provide a 
reference-implementation of some API (e.g. the atob tests)

All of these things need review. Therefore I would like a review system 
that is rather more like ordinary code review. In particular the model I 
had in mind was:

* Contributor makes a number of commits
* They create a review request for those commits
* Any number of reviewers can create responses to the review request 
where they provide comments on a specific set of lines in a specific 
revision of the changed files
* Contributor makes any necessary changes, makes a new commit, and adds 
those commits to the review
* Once the reviewers are happy with the changes, the review is marked as 
approved, which causes the commits to be considered approved

Received on Tuesday, 10 May 2011 09:15:00 UTC