Re: [testing] Common way to "manage" test bugs?

(not sure the best way to reply to this list and the infra list)

On Thu, Dec 19, 2013 at 9:54 AM, Arthur Barstow <>wrote:

> * Bugzilla - WebApps has a single Testing component in Bugzilla for all of
> the specs and it has been used to report a few test case bugs [Bugs]. Using
> Bugzilla addresses #1, however, a single component makes #2 a bit tricky
> although that could be addressed by prefixing titles with the spec name
> (e.g. [workers] ...).

We should definitely pick one or the other, but there doesn't seem to be a
clear favorite right now based on number of open issues. GitHub has 135
open issues, with 124 PRs, so there are 11 actual issues. Bugzilla has 6
open issues. If Bugzilla is chosen, then GitHub issues should be disabled
for the repo.

* GitHub PRs - some bugs are noted in comments to a PR (and thus making #2
> a bit difficult).

If the bug isn't contained inside the PR itself, then this seems like it's
just bad behavior. Any existing bugs that are discovered as a result of
reviewing a PR should go through the normal bug submission process. Any
bugs within the PR should be comments on the PR.

* GitHub Issues - some bugs are reported as a Github Issue but GH's Issue
> granularity is for all of [WPT] and not per spec (thus making #2 a bit
> tricky, although title prefixing could help).

For GitHub, we should use labels instead of title prefixing. An issue can
have multiple labels, so that addresses the multi-component issue as well.

* The WPT root [WPT] is silent on how to file bugs (although some
> sub-resource could address that) and that could be appropriate if we expect
> every test suite to be able to customize its bug reporting policy.

We should update and to document whatever
decision is made.

If we agree Bugzilla should be used to report all test case bugs: 1) should
> we have an agreed way for a test suite in WPT to point to Bugzilla
> (although Bugzilla has bug reports for websockets and workers tests, that
> link is missing from the test suites); 2) should we continue to lump all of
> the tests in a single component or create per test suite components (e.g.
> tests-workers, workers-tests, ...).

Can Bugzilla listen to GitHub events and auto-comment?

I think we should definitely have organization by components as we already
know this is a pain point in other areas, such as email notifications.

Received on Thursday, 19 December 2013 15:23:51 UTC