Re: bugs filed on Firefox

Ray Kiddy wrote:
> 
> Just FYI, I am starting to file bugs in http://bugzilla.mozilla.org 
> based on the tests in http://www.w3.org/Style/CSS/Test/CSS2.1/.
> 
> I was surprised that when I checked for bugs that have 
> "http://www.w3.org/Style/CSS/Test/CSS2.1" in the URL field, I found only 
> a few. I realized, though, that the URL field is not always used in this 
> way and that bugs may be filed against the test suite, or because of the 
> test suite, but the description of the bug could mention only the 
> functionality.
> 
> It makes sense to me, though, to track bugs found with this test suite 
> by listing the test case URL in the URL field.
> 
> I have only filed 3 so far, but there are more than a dozen I am going 
> to triage. If anyone has thoughts, ideas or suggestions, I would be 
> interested. Or if you want to comment on the bugs directly, please do. I 
> am by no means a CSS expert, so any informed opinions recorded on the 
> bugs may help. I am sure I am going to hear from the Mozilla guys on 
> some of these, since I have already seen cases where they disagree on 
> the meanings of some small things in some other tests.

Ok, this is a bit off-topic for this group but..

It's great that you're going through the test suite and noting which
tests fail. This is useful information. However, the reports you're writing
are not something a layout developer is going to tackle, because it's not
clear what the actual problem is. The summary and description of the bug
report should, ideally, explain what the bug is, not how it manifests itself
in the outcome of a CSS2.1 test case. The test cases are in many cases
somewhat convoluted to make it very easy to distinguish a passing test from
a failing one, but it is not so clear from the test case what, exactly, is
failing. Looking at your bug reports, I can't tell what the problem is, not
even roughly. The reports you're filing are at the level of a user reporting
"This page doesn't work because the navbar is too wide." It requires a lot
more QA work to get to the point where we have a summary saying "percentage
widths don't work on absolutely positioned elements with auto margins",
and that's where a) we can tell whether the bug report is a duplicate or
not and b) a developer doesn't need to do QA detective work to figure out
what the impact of the problem is and therefore when and how to fix it.

So, ultimately what I'm trying to say is, if your goal is to file useful
bug reports for Gecko, you have to work a bit at understanding what exactly
in the test is failing, and see if it's already reported. If it's already
reported, add the test case URL to the URL field or in a comment; and if
it's not, *then* file a bug--using your understanding of the underlying
problem and not the test case's symptoms as the summary.

If you're just going through the test suite and want to contribute the
pass/fail information you're finding to the Mozilla project, don't start
filing bugs right away. Start compiling a pass/fail list, and write a post
to mozilla.dev.quality <http://www.mozilla.org/community/developer-forums.html#dev-quality>
to say what you're doing and ask how it can become useful input to the
QA process. Perhaps other people can analyze the failures into bug reports,
or you could do that yourself later when you're prepared to dive a little
deeper into the individual test cases.

(BTW, note that the test suite itself has many bugs, and some of the failures
may be due to that.)

> By the way, I do not intend to write more e-mails like this. I just 
> wanted to give people a place to look for these if they are interested. 
> And now I am not just lurking.

~fantasai

Received on Tuesday, 3 July 2007 03:03:43 UTC