W3C home > Mailing lists > Public > public-webapps@w3.org > January to March 2015

Re: [websockets] Test results available

From: Arthur Barstow <art.barstow@gmail.com>
Date: Thu, 26 Mar 2015 16:05:46 -0400
Message-ID: <5514669A.4090602@gmail.com>
To: Boris Zbarsky <bzbarsky@mit.edu>, public-webapps@w3.org, Robin Berjon <robin@w3.org>
On 3/26/15 1:02 PM, Boris Zbarsky wrote:
> On 3/26/15 10:51 AM, Arthur Barstow wrote:
>> * All results <http://w3c.github.io/test-results/websockets/all.html>
>>
>> * <2 passes 
>> <http://w3c.github.io/test-results/websockets/less-than-2.html>
>>
>> Overall these results are pretty good: 97% of the 495 tests have two or
>> more passes.
>
> Arthur,
>
> It looks like the tests that are failed with an "Error" as opposed to 
> a "Fail" are not being counted in the "<2 passes" list?  For example, 
> http://www.w3c-test.org/websockets/Create-Secure-valid-url-array-protocols.htm 
> is not in that list even though everyone fails it.
>
> Is that an accident, or a purposeful decision?

I'm glad you reported this Boris! Indeed the less-than-2 table is 
missing several tests that have 4 Errors + 1 Timeout. I don't know why 
they aren't included. Such tests are also missing from the other 
automatically generated file "complete-fails" [cf] (which is a proper 
subset of the <2 table).

I'm not sure of the reason(s) to omit such errors from the less-than-2 
table or the complete-fail table. It could be a bug in the tool 
[wptreport] and perhaps there is a switch that would include the missing 
errors. I also vaguely recall a discussion about pervasive errors being 
a relatively strong indicator of a test case bug. Perhaps Robin can help 
here; Robin?

Regardless, we will also need to investigate all of these other failures 
missing from <2.

-Thanks, AB

[cf] <http://w3c.github.io/test-results/websockets/complete-fails.html>
[wptreport] <https://github.com/darobin/wptreport>
Received on Thursday, 26 March 2015 20:06:15 UTC

This archive was generated by hypermail 2.3.1 : Friday, 27 October 2017 07:27:26 UTC