RE: wptreport issue

This seems like it obscures useful information on groups trying to get two passing implementations.  We need to identify tests that have a single pass, and it's useful in determining that the test itself can be passed without indeterminate results.

Jerry

-----Original Message-----
From: Francois Daoust [mailto:fd@w3.org] 
Sent: Monday, August 22, 2016 1:13 AM
To: Michael [tm] Smith <mike@w3.org>; Jerry Smith (WPT) <jdsmith@microsoft.com>
Cc: Robin Berjon <robin@berjon.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; public-html-media@w3.org; Paul Cotton <Paul.Cotton@microsoft.com>
Subject: Re: wptreport issue

Jerry, Mike, Robin,

On 20/08/2016 07:39, Michael[tm] Smith wrote:
> +Robin Berjon
>
> "Jerry Smith (WPT)" <jdsmith@microsoft.com>, 2016-08-19 22:30 +0000:
>> Archived-At: 
>> <http://www.w3.org/mid/BY2PR03MB04156852535B860F021498EA4160@BY2PR03M

>> B041.namprd03.prod.outlook.com>
>>
>> I'm trying to track down an issue I'm hitting using wptreport.  EME 
>> has JSONs that show results for specific tests where the report pages 
>> just list the test itself with no results.  Any ideas on how to fix this?
>>
>> An example is the 
>> /encrypted-media/Google/encrypted-media-keystatus.html
>> test:
>> https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/all.html#test-file-9.

>> Current JSONs include some timeout and passing results for it.
>
> I looked at the JSON files today and tried to troubleshoot it but 
> still have no clue either.

In wptreport, the line that skips the subtests is in index.js, L187:
https://github.com/w3c/wptreport/blob/master/index.js#L187


Subtest results only appear provided the JSON files contain at least two results that are different from TIMEOUT and NOTRUN. In the provided example, only CH54.json contains a PASS result for this substest, the other JSON files have TIMEOUT results.

Changing the incriminated line to "if (haveResults < 1) continue;" would fix the problem. However, that line comes with a comment "skip tests with all but one undefined" and this check was first introduced in May 2014 in a commit entitled "filter out broken results":
https://github.com/w3c/wptreport/commit/6c9c20545aa3230bff6f165c1e129f8b89dd53df


In other words, this behavior seems intended, although I don't understand the rationale. Robin, do you remember why you had to do that?

Thanks,
Francois.

Received on Monday, 22 August 2016 19:30:58 UTC