Re: [testing] Common way to "manage" test bugs?

Hi All,

Given the responses to this thread (as well as replies only on 
public-test-infra), the consensus is to use Github Issues for test bugs 
and to use Github labels to identify the related spec ("component" in 
Bugzilla):

    <https://github.com/w3c/web-platform-tests/issues>

As such, I updated WebApps' test submission document [1] accordingly, 
and filed [Issue-474] as a reminder WPT documentation should be clear GH 
Issues (with labels) is used to manage test bugs.

I will create a new GH Issue for all of WebApps' open test bugs in 
Bugzilla (see [Bugz]) and then close those Bugz bugs as DUPs (and 
include a reference to the associated Issue).

-Thanks, ArtB

[1] <http://www.w3.org/wiki/Webapps/Submitting_tests>
[Issue-474] <https://github.com/w3c/web-platform-tests/issues/474>
[Bugz] <http://tinyurl.com/WebApps-testing-Bugz >

On 12/19/13 9:54 AM, ext Arthur Barstow wrote:
> [ Bcc: public-webapps-testsuite ]
>
> Hi All,
>
> I think it would be helpful if WebApps had a common way to "manage" 
> test case bugs,  in particular:
>
> 1. A common way to report a test bug
>
> 2. An easy way to determine if a specific test suite has any open 
> bugs. (For example, if someone goes to the websockets test suite on 
> WPT, it should be easy to determine if any of the test cases have open 
> bugs.)
>
> Currently, tests bugs are reported in a few different ways:
>
> * Bugzilla - WebApps has a single Testing component in Bugzilla for 
> all of the specs and it has been used to report a few test case bugs 
> [Bugs]. Using Bugzilla addresses #1, however, a single component makes 
> #2 a bit tricky although that could be addressed by prefixing titles 
> with the spec name (e.g. [workers] ...).
>
> * GitHub PRs - some bugs are noted in comments to a PR (and thus 
> making #2 a bit difficult).
>
> * GitHub Issues - some bugs are reported as a Github Issue but GH's 
> Issue granularity is for all of [WPT] and not per spec (thus making #2 
> a bit tricky, although title prefixing could help).
>
> * The WPT root [WPT] is silent on how to file bugs (although some 
> sub-resource could address that) and that could be appropriate if we 
> expect every test suite to be able to customize its bug reporting policy.
>
> If we agree Bugzilla should be used to report all test case bugs: 1) 
> should we have an agreed way for a test suite in WPT to point to 
> Bugzilla (although Bugzilla has bug reports for websockets and workers 
> tests, that link is missing from the test suites); 2) should we 
> continue to lump all of the tests in a single component or create per 
> test suite components (e.g. tests-workers, workers-tests, ...).
>
> Comments, proposals, etc. are welcome.
>
> -Thanks, ArtB
>
> [WPT] <https://github.com/w3c/web-platform-tests>
> [Bugs] 
> <https://www.w3.org/Bugs/Public/buglist.cgi?component=Test%20suite&list_id=30503&product=WebAppsWG&resolution=--->

Received on Friday, 3 January 2014 14:26:50 UTC