Re: [websockets] Test results available

On 3/26/15 1:02 PM, Boris Zbarsky wrote:
> On 3/26/15 10:51 AM, Arthur Barstow wrote:
>> * All results <http://w3c.github.io/test-results/websockets/all.html>
>>
>> * <2 passes 
>> <http://w3c.github.io/test-results/websockets/less-than-2.html>
>>
>> Overall these results are pretty good: 97% of the 495 tests have two or
>> more passes.
>
> Arthur,
>
> It looks like the tests that are failed with an "Error" as opposed to 
> a "Fail" are not being counted in the "<2 passes" list?  For example, 
> http://www.w3c-test.org/websockets/Create-Secure-valid-url-array-protocols.htm 
> is not in that list even though everyone fails it.
>
> Is that an accident, or a purposeful decision?

I'm glad you reported this Boris! Indeed the less-than-2 table is 
missing several tests that have 4 Errors + 1 Timeout. I don't know why 
they aren't included. Such tests are also missing from the other 
automatically generated file "complete-fails" [cf] (which is a proper 
subset of the <2 table).

I'm not sure of the reason(s) to omit such errors from the less-than-2 
table or the complete-fail table. It could be a bug in the tool 
[wptreport] and perhaps there is a switch that would include the missing 
errors. I also vaguely recall a discussion about pervasive errors being 
a relatively strong indicator of a test case bug. Perhaps Robin can help 
here; Robin?

Regardless, we will also need to investigate all of these other failures 
missing from <2.

-Thanks, AB

[cf] <http://w3c.github.io/test-results/websockets/complete-fails.html>
[wptreport] <https://github.com/darobin/wptreport>

Received on Thursday, 26 March 2015 20:06:15 UTC