W3C home > Mailing lists > Public > public-test-infra@w3.org > October to December 2011

[Bug 14839] Separate WebKit into Safari and Chrome

From: <bugzilla@jessica.w3.org>
Date: Wed, 23 Nov 2011 19:50:27 +0000
To: public-test-infra@w3.org
Message-Id: <E1RTIpv-0003e4-Lz@jessica.w3.org>
http://www.w3.org/Bugs/Public/show_bug.cgi?id=14839

--- Comment #3 from Peter Linss <peter.linss@hp.com> 2011-11-23 19:50:27 UTC ---
(In reply to comment #2)
> (In reply to comment #1)

> > In other words, if your CR exit criteria are two independent implementations,
> > and the only passes you have are WebKit (Safari) and WebKit (Chrome), does that
> > meet your exit criteria?
> 
> No. You are right, I didn't think of it that way. However, the converse is
> equally concerning. If we had run the test more often in Safari than Chrome, it
> would appear that we didn't have a conforming engine at all.

The relative number of results are not relevant. Currently, if there's one pass
for an engine, any version, any browser, that engine is considered passed,
regardless of how many other results there may be (so long as none of the
results are indications that the test itself is invalid). Results where there
are passes, but also other results are reported in a lighter shade of green.

At some point we may add a better filtering algorithm, but what we have now has
worked well for the CSS WG. Either way, it'll never be "more passes than
fails", it would compare versions of engines, browsers, etc. and try to
eliminate false pass results.

> 
> > Also, the intent of the current report is to show the status of meeting CR exit
> > criteria. As all results are tracked by UA string, and engines are tracked
> > independently of browsers, it would be easy to make a different report that
> > shows results per browser and leave the current report alone.
> 
> I'd be in favor of a second report that separates out browsers. If for no other
> reason than to feed bloggers and the like who enjoy quoting numbers like
> "Browser X passes Y% of conformance tests."
> 
> However, that doesn't necessarily fully address the concern above.

I suggest closing this issue, since this report is really only about CR exit
criteria, and filing a feature request for a per-browser result report. There
are also filters available internally (you can currently get them by
hand-editing the report URI) to generate reports for specific browsers,
platforms and versions of engine, browser and platform, but no UI yet to select
them, that UI needs to be added as well.

-- 
Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
Received on Wednesday, 23 November 2011 19:50:34 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 23 November 2011 19:50:35 GMT