Re: [testing] Common way to "manage" test bugs?

On Thursday 19 December 2013 at 17:34, Scott González wrote:
> On Thu, Dec 19, 2013 at 9:54 AM, Arthur Barstow <art.barstow@nokia.com (mailto:art.barstow@nokia.com)> wrote:
> > * Bugzilla - WebApps has a single Testing component in Bugzilla for all of the specs and it has been used to report a few test case bugs [Bugs]. Using Bugzilla addresses #1, however, a single component makes #2 a bit tricky although that could be addressed by prefixing titles with the spec name (e.g. [workers] ...).
>  
> We should definitely pick one or the other, but there doesn't seem to be a clear favorite right now based on number of open issues. GitHub has 135 open issues, with 124 PRs, so there are 11 actual issues. Bugzilla has 6 open issues. If Bugzilla is chosen, then GitHub issues should be disabled for the repo.

Disabling Github issues isn't possible as the repo is shared among a number of working groups. It would also prevent filtering PR per labels which can only be done through
>  
> > * GitHub PRs - some bugs are noted in comments to a PR (and thus making #2 a bit difficult).
>  
> If the bug isn't contained inside the PR itself, then this seems like it's just bad behavior. Any existing bugs that are discovered as a result of reviewing a PR should go through the normal bug submission process. Any bugs within the PR should be comments on the PR.
> > * GitHub Issues - some bugs are reported as a Github Issue but GH's Issue granularity is for all of [WPT] and not per spec (thus making #2 a bit tricky, although title prefixing could help).
>  
> For GitHub, we should use labels instead of title prefixing. An issue can have multiple labels, so that addresses the multi-component issue as well.

Using labels is the way to go and already automated for pull requests.  
> > * The WPT root [WPT] is silent on how to file bugs (although some sub-resource could address that) and that could be appropriate if we expect every test suite to be able to customize its bug reporting policy.
>  

The whole point of moving to a single repository is to share process and tools across working groups so customizing bug report policy completely defeats the purpose.
> We should update README.md (http://README.md) and CONTRIBUTING.md (http://CONTRIBUTING.md) to document whatever decision is made.

Documentation belongs on testthewebforward.org. The README file should be updated to point to it.

CONTRIBUTING.md helps us maintain a lightweight and reasonable flow while complying to legal requirements, it's best if it stays untouched. New content will need to go below the legal requirements posted at the top of it.
> > If we agree Bugzilla should be used to report all test case bugs: 1) should we have an agreed way for a test suite in WPT to point to Bugzilla (although Bugzilla has bug reports for websockets and workers tests, that link is missing from the test suites); 2) should we continue to lump all of the tests in a single component or create per test suite components (e.g. tests-workers, workers-tests, ...).
>  

This would be very detrimental to vendors running these test suites. So I strongly object to it: these tests are only really valuable for the Web at large if they're ran by vendors.  
> Can Bugzilla listen to GitHub events and auto-comment?
>  
> I think we should definitely have organization by components as we already know this is a pain point in other areas, such as email notifications.
That's fixable.

--tobie

Received on Thursday, 19 December 2013 20:43:02 UTC