Test harness comments

Hello.

There was an article[1] published recently in Russian local internet
news agency (Lenta.ru), with a link to your test report page[2].
The article states that IE has the best test results of all
browsers examined.

While there's a comment clearly stating that the results are
preliminary and should not be cited, there are several problems with
the test which, have them been fixed, would prevent this situation.

First, summary of the results shows only percentages, and not absolute
counts of passed/failed tests. This is absolutely wrong, as 
calculating
a sum of percentages for all the browsers leads to grossly incorrect
conclustion that IE has the best overall score. Moreover, I've found
no way to see counts of tests on the report page (other than manually
counting them), and so I've written a script[3] to do that.

The script parses XML files with test results (located at [4] by 
manually
examining the source of page [2]) which should be downloaded to the
current directory, and counts the tests, with percentages to the
total number of tests (212 at 6 Nov), and to the number of tests
for which results are present in XML file. Its output can be seen as 
[5],
and a relevant part is cited below.

IE9_PPB4.xml
Done: 187/212 (88%)
Fail:  17/212 ( 8%), of done:  17/187 ( 9%)
Pass: 170/212 (80%), of done: 170/187 (90%)

chrome_5.0.xml
Done: 212/212 (100%)
Fail:  17/212 ( 8%), of done:  17/212 ( 8%)
Pass: 195/212 (91%), of done: 195/212 (91%)

(The "done" numbers refer to count of <result> tags in XML.)

The results are rather interesting. IE actually passes 11% less tests 
than
Chrome 5, and also the results marked as "No Result" are not present 
in
the file (e.g. canvas(2d.transformation.scale.basic.html )), while 
tests
with "No Result" status for other browsers _are_ present in XML, and 
have
an actual result (e.g. canvas(2d.transformation.order.html ), which
is passed for IE and has no result for others)!

Second, the test video(video_008.htm )[7] is incorrect, as it depends 
on result
of test video(video_003.htm )[6]. If the 
type='application/octet-stream' attribute
is removed from <source> elements, as it is not required by the 
spec[8],
Chrome (particularily, Chromium 8) passes the test, as it should.

 1: http://www.lenta.ru/news/2010/11/02/iehtml/
 2: http://test.w3.org/html/tests/reporting/report.htm
 3: failpass.rb, attached
 4: http://test.w3.org/html/tests/reporting/
 5: correct_results.txt, attached
 6: http://test.w3.org/html/tests/approved/video/video_003.htm
 7: http://test.w3.org/html/tests/approved/video/video_008.htm
 8: http://www.w3.org/TR/html5/video.html#the-source-element

--
  WBR, Peter Zotov.