[testing] Common way to "manage" test bugs?

[ Bcc: public-webapps-testsuite ]

Hi All,

I think it would be helpful if WebApps had a common way to "manage" test 
case bugs,  in particular:

1. A common way to report a test bug

2. An easy way to determine if a specific test suite has any open bugs. 
(For example, if someone goes to the websockets test suite on WPT, it 
should be easy to determine if any of the test cases have open bugs.)

Currently, tests bugs are reported in a few different ways:

* Bugzilla - WebApps has a single Testing component in Bugzilla for all 
of the specs and it has been used to report a few test case bugs [Bugs]. 
Using Bugzilla addresses #1, however, a single component makes #2 a bit 
tricky although that could be addressed by prefixing titles with the 
spec name (e.g. [workers] ...).

* GitHub PRs - some bugs are noted in comments to a PR (and thus making 
#2 a bit difficult).

* GitHub Issues - some bugs are reported as a Github Issue but GH's 
Issue granularity is for all of [WPT] and not per spec (thus making #2 a 
bit tricky, although title prefixing could help).

* The WPT root [WPT] is silent on how to file bugs (although some 
sub-resource could address that) and that could be appropriate if we 
expect every test suite to be able to customize its bug reporting policy.

If we agree Bugzilla should be used to report all test case bugs: 1) 
should we have an agreed way for a test suite in WPT to point to 
Bugzilla (although Bugzilla has bug reports for websockets and workers 
tests, that link is missing from the test suites); 2) should we continue 
to lump all of the tests in a single component or create per test suite 
components (e.g. tests-workers, workers-tests, ...).

Comments, proposals, etc. are welcome.

-Thanks, ArtB

[WPT] <https://github.com/w3c/web-platform-tests>
[Bugs] 
<https://www.w3.org/Bugs/Public/buglist.cgi?component=Test%20suite&list_id=30503&product=WebAppsWG&resolution=--->

Received on Thursday, 19 December 2013 15:05:16 UTC