- From: Jerry Smith (WPT) <jdsmith@microsoft.com>
- Date: Mon, 22 Aug 2016 19:30:23 +0000
- To: Francois Daoust <fd@w3.org>, "Michael [tm] Smith" <mike@w3.org>
- CC: Robin Berjon <robin@berjon.com>, "Philippe Le Hegaret (plh@w3.org)" <plh@w3.org>, "public-html-media@w3.org" <public-html-media@w3.org>, "Paul Cotton" <Paul.Cotton@microsoft.com>
This seems like it obscures useful information on groups trying to get two passing implementations. We need to identify tests that have a single pass, and it's useful in determining that the test itself can be passed without indeterminate results. Jerry -----Original Message----- From: Francois Daoust [mailto:fd@w3.org] Sent: Monday, August 22, 2016 1:13 AM To: Michael [tm] Smith <mike@w3.org>; Jerry Smith (WPT) <jdsmith@microsoft.com> Cc: Robin Berjon <robin@berjon.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; public-html-media@w3.org; Paul Cotton <Paul.Cotton@microsoft.com> Subject: Re: wptreport issue Jerry, Mike, Robin, On 20/08/2016 07:39, Michael[tm] Smith wrote: > +Robin Berjon > > "Jerry Smith (WPT)" <jdsmith@microsoft.com>, 2016-08-19 22:30 +0000: >> Archived-At: >> <http://www.w3.org/mid/BY2PR03MB04156852535B860F021498EA4160@BY2PR03M >> B041.namprd03.prod.outlook.com> >> >> I'm trying to track down an issue I'm hitting using wptreport. EME >> has JSONs that show results for specific tests where the report pages >> just list the test itself with no results. Any ideas on how to fix this? >> >> An example is the >> /encrypted-media/Google/encrypted-media-keystatus.html >> test: >> https://rawgit.com/w3c/test-results/gh-pages/encrypted-media/all.html#test-file-9. >> Current JSONs include some timeout and passing results for it. > > I looked at the JSON files today and tried to troubleshoot it but > still have no clue either. In wptreport, the line that skips the subtests is in index.js, L187: https://github.com/w3c/wptreport/blob/master/index.js#L187 Subtest results only appear provided the JSON files contain at least two results that are different from TIMEOUT and NOTRUN. In the provided example, only CH54.json contains a PASS result for this substest, the other JSON files have TIMEOUT results. Changing the incriminated line to "if (haveResults < 1) continue;" would fix the problem. However, that line comes with a comment "skip tests with all but one undefined" and this check was first introduced in May 2014 in a commit entitled "filter out broken results": https://github.com/w3c/wptreport/commit/6c9c20545aa3230bff6f165c1e129f8b89dd53df In other words, this behavior seems intended, although I don't understand the rationale. Robin, do you remember why you had to do that? Thanks, Francois.
Received on Monday, 22 August 2016 19:30:58 UTC