Re: [testing] Common way to "manage" test bugs?

Hi All,

Given the responses to this thread (as well as replies only on 
public-test-infra), the consensus is to use Github Issues for test bugs 
and to use Github labels to identify the related spec ("component" in 


As such, I updated WebApps' test submission document [1] accordingly, 
and filed [Issue-474] as a reminder WPT documentation should be clear GH 
Issues (with labels) is used to manage test bugs.

I will create a new GH Issue for all of WebApps' open test bugs in 
Bugzilla (see [Bugz]) and then close those Bugz bugs as DUPs (and 
include a reference to the associated Issue).

-Thanks, ArtB

[1] <>
[Issue-474] <>
[Bugz] < >

On 12/19/13 9:54 AM, ext Arthur Barstow wrote:
> [ Bcc: public-webapps-testsuite ]
> Hi All,
> I think it would be helpful if WebApps had a common way to "manage" 
> test case bugs,  in particular:
> 1. A common way to report a test bug
> 2. An easy way to determine if a specific test suite has any open 
> bugs. (For example, if someone goes to the websockets test suite on 
> WPT, it should be easy to determine if any of the test cases have open 
> bugs.)
> Currently, tests bugs are reported in a few different ways:
> * Bugzilla - WebApps has a single Testing component in Bugzilla for 
> all of the specs and it has been used to report a few test case bugs 
> [Bugs]. Using Bugzilla addresses #1, however, a single component makes 
> #2 a bit tricky although that could be addressed by prefixing titles 
> with the spec name (e.g. [workers] ...).
> * GitHub PRs - some bugs are noted in comments to a PR (and thus 
> making #2 a bit difficult).
> * GitHub Issues - some bugs are reported as a Github Issue but GH's 
> Issue granularity is for all of [WPT] and not per spec (thus making #2 
> a bit tricky, although title prefixing could help).
> * The WPT root [WPT] is silent on how to file bugs (although some 
> sub-resource could address that) and that could be appropriate if we 
> expect every test suite to be able to customize its bug reporting policy.
> If we agree Bugzilla should be used to report all test case bugs: 1) 
> should we have an agreed way for a test suite in WPT to point to 
> Bugzilla (although Bugzilla has bug reports for websockets and workers 
> tests, that link is missing from the test suites); 2) should we 
> continue to lump all of the tests in a single component or create per 
> test suite components (e.g. tests-workers, workers-tests, ...).
> Comments, proposals, etc. are welcome.
> -Thanks, ArtB
> [WPT] <>
> [Bugs] 
> <>

Received on Friday, 3 January 2014 14:26:49 UTC