- From: Arthur Barstow <art.barstow@gmail.com>
- Date: Thu, 26 Mar 2015 16:05:46 -0400
- To: Boris Zbarsky <bzbarsky@mit.edu>, public-webapps@w3.org, Robin Berjon <robin@w3.org>
On 3/26/15 1:02 PM, Boris Zbarsky wrote: > On 3/26/15 10:51 AM, Arthur Barstow wrote: >> * All results <http://w3c.github.io/test-results/websockets/all.html> >> >> * <2 passes >> <http://w3c.github.io/test-results/websockets/less-than-2.html> >> >> Overall these results are pretty good: 97% of the 495 tests have two or >> more passes. > > Arthur, > > It looks like the tests that are failed with an "Error" as opposed to > a "Fail" are not being counted in the "<2 passes" list? For example, > http://www.w3c-test.org/websockets/Create-Secure-valid-url-array-protocols.htm > is not in that list even though everyone fails it. > > Is that an accident, or a purposeful decision? I'm glad you reported this Boris! Indeed the less-than-2 table is missing several tests that have 4 Errors + 1 Timeout. I don't know why they aren't included. Such tests are also missing from the other automatically generated file "complete-fails" [cf] (which is a proper subset of the <2 table). I'm not sure of the reason(s) to omit such errors from the less-than-2 table or the complete-fail table. It could be a bug in the tool [wptreport] and perhaps there is a switch that would include the missing errors. I also vaguely recall a discussion about pervasive errors being a relatively strong indicator of a test case bug. Perhaps Robin can help here; Robin? Regardless, we will also need to investigate all of these other failures missing from <2. -Thanks, AB [cf] <http://w3c.github.io/test-results/websockets/complete-fails.html> [wptreport] <https://github.com/darobin/wptreport>
Received on Thursday, 26 March 2015 20:06:15 UTC