Re: Test Management Task Force

On Feb 22, 2013, at 2:52 PM, Tobie Langel wrote:

> Hi all,
> 
> I've set up a home for the Test Management Task Force[1].
> 
> This task force's role is to look at test management and see how we can improve and simplify through lighter process and better tooling.
> 
> Like all other testing task forces we'll be using the public-test-infra@w3.org (mailto:public-test-infra@w3.org) mailing list. If you haven't already, please sign-up for it[2].
> 
> As you're probably well aware by now, I'm a strong proponent of moving to a GitHub-based workflow[3] and building lightweight tooling on top of it[4].
> 
> This raises a number of issues:
> 
> 1. Moving to GitHub represents a number of different risks. These risks need to be assessed and a mitigation strategy defined and its cost estimated.
> 
> 2. Moving to GitHub also represents an upfront migration cost. For certain group, this is trivial and doesn't even need to be budgeted. For other, however, this migration might be costly and complex (I'm thinking mainly about the CSS WG with its Shepherd tool, here). First of all we must enquire to see if these groups are interested in migrating. If so we must define what feature set such groups need in a new system and estimate their cost. We must also define a migration strategy and estimate its costs.
> 
> I'll be preparing a document for the first point shortly and will enquire on the risk mitigation costs directly with W3C sys team.
> 
> I'd be very interested to hear your thoughts on the second point, especially Peter's who is directly concerned.

While we don't have a formal group resolution yet, I don't think the CSSWG will resist using GitHub as a front end for the test repository. The current plan is to modify the Shepherd workflow (and existing review data) to more align with the structure of GitHub review and issue data and add the GitHub integration directly into Shepherd. This way we can retain our existing repository and Shepherd front-end while having both the repository and issue/review data synced with GitHub's front end. Users can then use either to interact with the repository. 

Shepherd already has a bunch of the features that we to layer on top of GitHub, like automatically filing issues for metadata errors/omissions, as well as the test to specification anchor mapping. I feel that it will be easier to extend Shepherd than re-write this functionality from scratch. Shepherd also has features like being able to search by metadata that will be difficult, if not impossible to add to GitHub's front end (it does expose those features via an API so again, it may be able to serve back end roles for GitHub front end extensions).

While Shepherd does has a few of the CSSWG's particular requirements they're generally behind abstractions in the code and it should be adaptable to other group's needs. As an example, while the CSSWG has had a rule about unique filenames across the repository (so that tests can be mixed and matched between suites without need for name collision management), Shepherd has only required unique asset names (test assets being an abstract concept), where the mapping between file names and asset names is configurable on a per-repository basis (as is the directory structure of the repository).  Furthermore I'm about 2/3rds of the way through adding asset branch support. This alleviates the unique file name requirement, even in the CSSWG repository. When multiple files get mapped to the same asset, but they have different contents, Shepherd will automatically track those as different logical branches of the asset (in effect independent entities) (in addition it also tracks multiple physical branches in the repository as independent entities).

> 
> Lastly, I've started digging two other tangentially related subjects, upstreaming and repo syncing with browser vendors, and handling test suites which have build steps (i18n, accessibility, WebIDL auto-generated tests, etc). Would you be happy to have this conversation within this task force or do you feel like we should create a dedicated task force for it?
> 
> Thanks for your input.
> 
> Best,
> 
> --tobie
> ---
> [1]: http://www.w3.org/wiki/Testing/Test_Management_TF
> [2]: Just send an email to mailto:public-test-infra-request@w3.org
> 
> 
> 
> [3]: http://www.w3.org/wiki/Webapps/Submitting_tests
> 
> 
> [4]: http://w3c.github.com/testing-task-forces/test-management-tooling.html
> 
> 
>

Received on Monday, 25 February 2013 20:17:02 UTC