RE: Update on MSE and EME Testing

Also, we should likely turn off the tests in the Google sub-folder, and start focusing on the modified versions in the main one.  That can be done by commenting out the testharness js script lines.  Are we ready to do that?

Jerry

From: Jerry Smith (WPT)
Sent: Friday, August 19, 2016 2:10 PM
To: 'David Dorwin' <ddorwin@google.com>; Mark Watson <watsonm@netflix.com>
Cc: Henri Sivonen <hsivonen@hsivonen.fi>; Chris Pearce <cpearce@mozilla.com>; public-html-media@w3.org; jyavenard@mozilla.com; Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com>
Subject: RE: Update on MSE and EME Testing

I intend to update these test reports soon.  Of issues raised below, I believe:

  *   There is a bug in util/utils.js in getSupportedKeysystem() which returns com.widevine.alpha for Edge, meaning that playback tests on Edge all fail. I presume you fixed this before generating the results.
     *   Jerry:  This has been resolved
  *   The "less than 2" script should not consider two versions of Chrome or two versions of Firefox as 2 versions passing
     *   Jerry:  This will resolve by having just one version for each browser
  *   The test names for the drm tests include the keysystem. So, when you have it working with appropriate keysystems on each browser, they will not match up correctly on the test-results page. We'll need to remove the keysystem name from the test name and just put "drm"
     *   Jerry:  This has been resolved
  *   The test names for the polyfill tests should probably start "Polyfill" for readability
     *   Jerry:  This hasn’t yet been changed – Mark, can the polyfill make script be modified to do this directly?
I looked also at this issue raised by David:

·       I don't understand some of the output. For example, http://www.w3c-test.org/encrypted-media/Google/encrypted-media-keystatuses.html has no result listed for several browsers. It passes in the latest Chrome but there is no entry. Also, TIMEOUT seems to appear inconsistently in some header rows. The persistent-usage-record tests don't even have results. Does this mean they are new / not run?
The JSONs contain test results, but they aren’t showing up in the test reports.  I reduced the JSONs down to CH54, ED14 and FF51 and created a report based just on those; where ED14 and FF51 have timeouts and CH54 passes on the encrypted-media-keystatuses test.  The resulting report still shows a blank result.  There is no obvious difference between this test, where results aren’t showing up correctly, and other tests where the report is complete.  I’m still looking for an explanation.
Jerry
From: David Dorwin [mailto:ddorwin@google.com]
Sent: Monday, August 8, 2016 4:54 PM
To: Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>>
Cc: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; Henri Sivonen <hsivonen@hsivonen.fi<mailto:hsivonen@hsivonen.fi>>; Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; public-html-media@w3.org<mailto:public-html-media@w3.org>; jyavenard@mozilla.com<mailto:jyavenard@mozilla.com>; Matthew Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> (wolenetz@google.com<mailto:wolenetz@google.com>) <wolenetz@google.com<mailto:wolenetz@google.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>
Subject: Re: Update on MSE and EME Testing


  *   There is a bug in util/utils.js in getSupportedKeysystem() which returns com.widevine.alpha for Edge, meaning that playback tests on Edge all fail. I presume you fixed this before generating the results.
Mark has a fix for this in https://github.com/w3c/web-platform-tests/pull/3424/files#diff-c5c73a9a657afac9f2d88b36c6348243 if someone wants to land that separately.

I plan to pull down the older versions of Chrome and Firefox soon, but wanted to be sure the deltas between the two make sense.  Comments are welcome.
Jerry, the two failures in CH52 where CH54 passes are fine:

  *   /encrypted-media/Google/encrypted-media-async-setcert-with-gc.html fails in CH52 because we changed the behavior and test to be spec compliant.
  *   /encrypted-media/clearkey-mp4-playback-temporary-events.html fails because of a bug that we fixed when it was identified by this test.

I don't understand some of the output. For example, http://www.w3c-test.org/encrypted-media/Google/encrypted-media-keystatuses.html has no result listed for several browsers. It passes in the latest Chrome but there is no entry. Also, TIMEOUT seems to appear inconsistently in some header rows. The persistent-usage-record tests don't even have results. Does this mean they are new / not run?

On Fri, Aug 5, 2016 at 2:14 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
Jerry,

There are a few things which need to be fixed up before we can really read anything into the test results:

  *   There is a bug in util/utils.js in getSupportedKeysystem() which returns com.widevine.alpha for Edge, meaning that playback tests on Edge all fail. I presume you fixed this before generating the results.
  *   The "less than 2" script should not consider two versions of Chrome or two versions of Firefox as 2 versions passing
  *   The test names for the drm tests include the keysystem. So, when you have it working with appropriate keysystems on each browser, they will not match up correctly on the test-results page. We'll need to remove the keysystem name from the test name and just put "drm"
  *   The test names for the polyfill tests should probably start "Polyfill" for readability
...Mark

On Fri, Aug 5, 2016 at 1:44 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:

I’ve made an initial posting of test results on a few browsers (Edge, two versions of Chrome and two also of Firefox).  It’s a bit messy, because it includes Google subfolder tests for Clear-Key (dups) and also tests with and without some polyfills Mark has been developing.



You can see current EME results using these links:



•                     http://w3c.github.io/test-results/encrypted-media/all.html


•                     http://w3c.github.io/test-results/encrypted-media/complete-fails.html


•                     http://w3c.github.io/test-results/encrypted-media/less-than-2.html




I plan to pull down the older versions of Chrome and Firefox soon, but wanted to be sure the deltas between the two make sense.  Comments are welcome.



For convenience, here are the MSE links:



•                     http://w3c.github.io/test-results/media-source/all.html


•                     http://w3c.github.io/test-results/media-source/complete-fails.html


•                     http://w3c.github.io/test-results/media-source/less-than-2.html



Jerry



From: Mark Watson [mailto:watsonm@netflix.com<mailto:watsonm@netflix.com>]
Sent: Friday, August 5, 2016 12:44 PM
To: Henri Sivonen <hsivonen@hsivonen.fi<mailto:hsivonen@hsivonen.fi>>
Cc: Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; public-html-media@w3.org<mailto:public-html-media@w3.org>; jyavenard@mozilla.com<mailto:jyavenard@mozilla.com>; Matthew Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> (wolenetz@google.com<mailto:wolenetz@google.com>) <wolenetz@google.com<mailto:wolenetz@google.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>
Subject: Re: Update on MSE and EME Testing



We have been working on additional tests for the Encrypted Media Extensions specification.



Unfortunately, it has been very difficult to get the tests we have been working to work, due to lack of the necessary server support and uncertainty as to what is and is not implemented in  browsers. At this stage, therefore, we do not yet have DRM tests for persistent-license or persistent-usage-record session types.



There are two outstanding pull requests:

- https://github.com/w3c/web-platform-tests/pull/3424 adds support for persistent-license (clearkey)

- https://github.com/w3c/web-platform-tests/pull/3417 adds a waitingforkey test



I expect we will make further progress over the coming week (I will be out, but Sukmal will have some time to work on this).



...Mark





On Fri, Aug 5, 2016 at 7:50 AM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:





On Fri, Aug 5, 2016 at 1:20 AM, Henri Sivonen <hsivonen@hsivonen.fi<mailto:hsivonen@hsivonen.fi>> wrote:

On Thu, Aug 4, 2016 at 7:26 AM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
> I believe that persistent-usage-record functionality works with Firefox /
> Primetime, with some rough edges on the Primetime side. However, we do not
> have a Primetime server for our testing, so we cannot verify this.

While some initial steps (that I've referred to previously on this
list) were taken towards persistent-usage-record -like functionality
on the Firefox side, those steps never reached the point of supporting
the persistent-usage-record API. As Chris said, we have no plans to
implement "persistent-usage-record" sessions.

> The Edge and CastTV implementations are close to compliance, I believe, and we are working on tests that will demonstrate that.

Do those implementations write to plain disk-like storage on CDM
shutdown or do they use some form of tamper-resistant storage during
playback? Which Key System does CastTV use? Was the text in the spec
alone sufficiently detailed to reach interop?



​I don't know the implementation details. They are both PlayReady. They don't reach interop. When I say "close to compliance", I mean that with a bit of JS help you can patch up the non-compliances. At least, that is my hope at present: we don't have everything working yet.



...Mark



--
Henri Sivonen
hsivonen@hsivonen.fi<mailto:hsivonen@hsivonen.fi>
https://hsivonen.fi/