Test Review Tools

Hi folks, 

Yesterday, I received an email from GitHub advising me that a PR on web-platform-org had been pulled in.

I followed the link[1], only to find no information in the PR issue itself, but a link to Critic.

I had a knee-jerk reaction on the comment thread itself, which James Graham rightfully called me out for. Sorry, James.

That said, I believe my point still holds.

There are a number of reasons we're centralizing our testing effort on GitHub, notably to:

- accept external contributions more easily,
- share process and repository across WGs, and
- be more visible.

This is jeopardized if the review tools are going to be specific to WGs or worse, individual reviewers. Review tools aren't like text editor. Unless there's a seamless bi-directional syncing of comments between GH and the review tool (that's *hard* and tends to defeat the purpose of using an external tool), the review tools is being forced onto the contributor (and the rest of the community that might that might want to follow along) by the reviewer. As I contributor, if I have to learn a completely different review tool whether I'm writing a test for the WebPerf WG (e.g. GH itself), the CSS WG (e.g. shepherd) or WebApps (e.g. Critic), I'm just going to give up.

While I understand the limitations of the GitHub review tools, they work fairly well for a number of projects with a lot more going on than ours. I feel like the reason this works is those projects adopt the "commit early, commit often" Git mindset, which guarantees small atomic commits and PRs focused on a single topic. Until we enforce that in the way we work with Git, it'll be difficult to assess faithfully whether the limitation of GH review tools are intrinsic or of our own making.

If we do end-up finding out better tooling is necessary, we have to carefully weigh the tradeoffs we're making by switching to a custom solution. At which point we'll also have to decide whether we keep the GitHub Issue system up (in which case we'll need to fully synchronize the comments on both systems) or close GitHub issues and switch the whole repository to using something different (which we'll need to have agreement on and also have to archive).

Until then, I feel like we should embrace the Git and GitHub ways to at least give them a try. Maybe we'll find that they're not so bad after all.

Best,

--tobie

---
[1]: https://github.com/w3c/web-platform-tests/pull/38

Received on Tuesday, 19 March 2013 10:23:14 UTC