Firefox 4.0b6 (Linux) implementation report for CSS 2.1

A CSS 2.1 implementation report for Firefox 4.0b6 (on Linux) is
available here:
http://lists.w3.org/Archives/Public/www-archive/2010Oct/att-0020/implement-report-Firefox4.0b6-Linux.data

The summary is:
  18188 (93.83%) pass
    691 ( 3.56%) failure due to bug or missing feature ("fail")
    494 ( 2.55%) failure due to invalid test ("invalid")
     84 ( 0.43%) unknown ("?")

Of the 84 tests marked unknown, 10 were not run for the reasons
described in
http://lists.w3.org/Archives/Public/public-css-testsuite/2010Sep/0230.html
http://lists.w3.org/Archives/Public/public-css-testsuite/2010Sep/0232.html
and the remaining 74 were tests listed in the implementation report
but not actually in the test suite (23 HTMLonly, 51 nonHTML).

The percentages above are computed relative to the actual number of
tests in the test suite (19383), not the number listed in the
implementation report (19457).

Of the 691 tests marked failing, I still haven't completed analyzing
18 of the tests for whether the failure is due to a bug in the test
or a bug in our implementation.  These are the 10 mentioned in
http://lists.w3.org/Archives/Public/public-css-testsuite/2010Oct/0178.html ,
the 2 mentioned in
http://lists.w3.org/Archives/Public/public-css-testsuite/2010Oct/0175.html ,
the 2 margin-collapse-157, the 2 margin-collapse-clear-005, and the
2 margin-collapse-clear-011.  But given that I haven't analyzed them
yet, I gave the test suite the benefit of the doubt and marked them
"fail" rather than "invalid".


A list (not quite complete) of reasons for the failures (i.e., links
to bug reports in either Mozilla or the test suite) is available
here:
http://lists.w3.org/Archives/Public/www-archive/2010Oct/att-0020/fail-analysis.csv

I hope to write a slightly more detailed analysis of the Mozilla
bugs shown by the test suite in the future.  However, I'd note the
following highlights:

  166 failures were caused by lack of support for display:run-in
      (Mozilla bug 2056)

  100 failures were caused by incomplete support for page-break-*
      (Mozilla bug 132035) and lack of support for @page (Mozilla
      bug 115199), and widows and orphans (Mozilla bug 137367).
      (I'm listing these together since they were sometimes hard to
      separate.)

   74 failures were caused by the extension of :first-letter across
      punctuation (e.g., to make the :first-letter hold both a
      quotation mark and a letter) not crossing changes in character
      directionality (Mozilla bug 604720)

   54 (or perhaps as many as 66) failures were caused by a bug,
      probably not present on Windows or Mac, where the max extents
      of the Ahem font come out one pixel larger than the glyphs
      (Mozilla bug 604836)


Since I ran the tests relatively quickly, it's likely that I missed
a few failures, although I hope the number is relatively small.


The scripts I used for generating the implementation report and
notes on how I made it are available in this Mercurial repository:
http://hg.mozilla.org/users/dbaron_mozilla.com/css21-implementation-report/
and any future revisions will likely go there.

-David

-- 
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/

Received on Tuesday, 19 October 2010 01:17:54 UTC