Re: [testing] Common way to "manage" test bugs?

(not sure the best way to reply to this list and the infra list)

On Thu, Dec 19, 2013 at 9:54 AM, Arthur Barstow <art.barstow@nokia.com>wrote:

> * Bugzilla - WebApps has a single Testing component in Bugzilla for all of
> the specs and it has been used to report a few test case bugs [Bugs]. Using
> Bugzilla addresses #1, however, a single component makes #2 a bit tricky
> although that could be addressed by prefixing titles with the spec name
> (e.g. [workers] ...).
>

We should definitely pick one or the other, but there doesn't seem to be a
clear favorite right now based on number of open issues. GitHub has 135
open issues, with 124 PRs, so there are 11 actual issues. Bugzilla has 6
open issues. If Bugzilla is chosen, then GitHub issues should be disabled
for the repo.

* GitHub PRs - some bugs are noted in comments to a PR (and thus making #2
> a bit difficult).
>

If the bug isn't contained inside the PR itself, then this seems like it's
just bad behavior. Any existing bugs that are discovered as a result of
reviewing a PR should go through the normal bug submission process. Any
bugs within the PR should be comments on the PR.

* GitHub Issues - some bugs are reported as a Github Issue but GH's Issue
> granularity is for all of [WPT] and not per spec (thus making #2 a bit
> tricky, although title prefixing could help).
>

For GitHub, we should use labels instead of title prefixing. An issue can
have multiple labels, so that addresses the multi-component issue as well.

* The WPT root [WPT] is silent on how to file bugs (although some
> sub-resource could address that) and that could be appropriate if we expect
> every test suite to be able to customize its bug reporting policy.
>

We should update README.md and CONTRIBUTING.md to document whatever
decision is made.

If we agree Bugzilla should be used to report all test case bugs: 1) should
> we have an agreed way for a test suite in WPT to point to Bugzilla
> (although Bugzilla has bug reports for websockets and workers tests, that
> link is missing from the test suites); 2) should we continue to lump all of
> the tests in a single component or create per test suite components (e.g.
> tests-workers, workers-tests, ...).
>

Can Bugzilla listen to GitHub events and auto-comment?

I think we should definitely have organization by components as we already
know this is a pain point in other areas, such as email notifications.