W3C home > Mailing lists > Public > public-html-testsuite@w3.org > November 2010

Re: Test results and report table

From: Maciej Stachowiak <mjs@apple.com>
Date: Tue, 02 Nov 2010 14:26:32 +0100
Cc: Jonathan Griffin <jgriffin@mozilla.com>, Alexander Mertens <alexander.mertens@stud.fh-dortmund.de>, miss.verstaendnis@nurfuerspam.de, Mikkel Toudal Kristiansen <mikkel.kristiansen@gmail.com>, Kris Krueger <krisk@microsoft.com>, Philippe Le Hegaret <plh@w3.org>, "'public-html-testsuite@w3.org'" <public-html-testsuite@w3.org>
Message-id: <B6FEC8D7-DEF4-4BCB-94C4-05671170A512@apple.com>
To: Anne van Kesteren <annevk@opera.com>

On Nov 2, 2010, at 2:14 PM, Anne van Kesteren wrote:

> On Tue, 02 Nov 2010 11:28:09 +0100, Philippe Le Hegaret <plh@w3.org> wrote:
>> Thank you all for sending test results. I updated the table:
>> http://test.w3.org/html/tests/reporting/report.htm
>> 
>> We will need to improve the report table on our side however because
>> sometimes "0% means no results", or 100% means "one pass, two no
>> results"... If anyone has time to take a stab at it, don't hesitate!
>> 
>> So, the table at it stands is misleading currently.
> 
> Reportedly the results for Opera are at least partly bogus.
> 
> This test suite is vastly incomplete. Publishing unverified results of a vastly incomplete test suite without a big fat warning is extremely silly. Why was this done?

It's also strange that the results include alpha/beta/preview versions of most browsers, but the stable version of Safari.

Wouldn't be a big deal other than the fact that this rather buggy test results page was labeled as "Official" and then picked up in the press as authoritative:
http://www.readwriteweb.com/archives/ie9_outperforms_other_browsers_for_html5_complianc.php

We should probably be cautious about the chance of creating PR events based on incorrect information.

Regards,
Maciej
Received on Tuesday, 2 November 2010 13:27:09 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 2 November 2010 13:27:09 GMT