Re: [testing] Common way to "manage" test bugs?

Messages from public-webapps:

----------------------------------------
>From Scott González:

On Thu, Dec 19, 2013 at 9:54 AM, Arthur Barstow <art.barstow@nokia.com>
 wrote:

> * Bugzilla - WebApps has a single Testing component in Bugzilla for all of
> the specs and it has been used to report a few test case bugs [Bugs]. Using
> Bugzilla addresses #1, however, a single component makes #2 a bit tricky
> although that could be addressed by prefixing titles with the spec name
> (e.g. [workers] ...).
>

We should definitely pick one or the other, but there doesn't seem to be a
clear favorite right now based on number of open issues. GitHub has 135
open issues, with 124 PRs, so there are 11 actual issues. Bugzilla has 6
open issues. If Bugzilla is chosen, then GitHub issues should be disabled
for the repo.

* GitHub PRs - some bugs are noted in comments to a PR (and thus making #2
> a bit difficult).
>

If the bug isn't contained inside the PR itself, then this seems like it's
just bad behavior. Any existing bugs that are discovered as a result of
reviewing a PR should go through the normal bug submission process. Any
bugs within the PR should be comments on the PR.

* GitHub Issues - some bugs are reported as a Github Issue but GH's Issue
> granularity is for all of [WPT] and not per spec (thus making #2 a bit
> tricky, although title prefixing could help).
>

For GitHub, we should use labels instead of title prefixing. An issue can
have multiple labels, so that addresses the multi-component issue as well.

* The WPT root [WPT] is silent on how to file bugs (although some
> sub-resource could address that) and that could be appropriate if we expect
> every test suite to be able to customize its bug reporting policy.
>

We should update README.md and CONTRIBUTING.md to document whatever
decision is made.

If we agree Bugzilla should be used to report all test case bugs: 1) should
> we have an agreed way for a test suite in WPT to point to Bugzilla
> (although Bugzilla has bug reports for websockets and workers tests, that
> link is missing from the test suites); 2) should we continue to lump all of
> the tests in a single component or create per test suite components (e.g.
> tests-workers, workers-tests, ...).
>

Can Bugzilla listen to GitHub events and auto-comment?

I think we should definitely have organization by components as we already
know this is a pain point in other areas, such as email notifications.

----------------------------------------
>From Domenic Denicola:

I would encourage use of GitHub for greater developer involvement. I think
elsewhere in the thread it's been covered how to adapt GitHub issues to
solve the potential problems you mention, so hopefully it's sufficient
technically for your needs. Just wanted to give a voice to that
perspective, if it comes down to a matter of preference.


On Thu, Dec 19, 2013 at 10:16 AM, Ms2ger <ms2ger@gmail.com> wrote:

> [ Bcc: public-webapps, public-webapps-testsuite ]
>
> Hi Art,
>
>
> On 12/19/2013 03:54 PM, Arthur Barstow wrote:
>
>> I think it would be helpful if WebApps had a common way to "manage" test
>> case bugs,  in particular:
>>
>> 1. A common way to report a test bug
>>
>> 2. An easy way to determine if a specific test suite has any open bugs.
>> (For example, if someone goes to the websockets test suite on WPT, it
>> should be easy to determine if any of the test cases have open bugs.)
>>
> >
>
>> Comments, proposals, etc. are welcome.
>>
>
> I strongly believe we should decide on a single way of managing bugs for
> all test suites in w-p-t (so moving the discussion to public-test-infra).
> It doesn't really matter to me whether that's done through github or
> bugzilla. Two things to consider, though:
>
> * Some issues might span multiple test suites.
> * It seems possible that we'd want to cross-reference test suite issues
> and specification issues.
> * Cross-referencing commits/PRs with issues might also be useful.
>
> HTH
> Ms2ger
>
>