RE: Update on MSE and EME Testing

> I expect we will make further progress over the coming week (I will be out, but Sukmal will have some time to work on this).

With Mark away on vacation do you need any assistance to create new Web Platform Tests for EME?   Are you expecting to make any EME testing progress this week?

I note that the following two PRs are still open:
https://github.com/w3c/web-platform-tests/labels/encrypted-media


/paulc

From: Mark Watson [mailto:watsonm@netflix.com]
Sent: Friday, August 5, 2016 5:15 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com>
Cc: Henri Sivonen <hsivonen@hsivonen.fi>; Chris Pearce <cpearce@mozilla.com>; public-html-media@w3.org; jyavenard@mozilla.com; Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com>
Subject: Re: Update on MSE and EME Testing

Jerry,

There are a few things which need to be fixed up before we can really read anything into the test results:

  *   There is a bug in util/utils.js in getSupportedKeysystem() which returns com.widevine.alpha for Edge, meaning that playback tests on Edge all fail. I presume you fixed this before generating the results.
  *   The "less than 2" script should not consider two versions of Chrome or two versions of Firefox as 2 versions passing
  *   The test names for the drm tests include the keysystem. So, when you have it working with appropriate keysystems on each browser, they will not match up correctly on the test-results page. We'll need to remove the keysystem name from the test name and just put "drm"
  *   The test names for the polyfill tests should probably start "Polyfill" for readability
...Mark

On Fri, Aug 5, 2016 at 1:44 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
I’ve made an initial posting of test results on a few browsers (Edge, two versions of Chrome and two also of Firefox).  It’s a bit messy, because it includes Google subfolder tests for Clear-Key (dups) and also tests with and without some polyfills Mark has been developing.

You can see current EME results using these links:


•                     http://w3c.github.io/test-results/encrypted-media/all.html


•                     http://w3c.github.io/test-results/encrypted-media/complete-fails.html


•                     http://w3c.github.io/test-results/encrypted-media/less-than-2.html


I plan to pull down the older versions of Chrome and Firefox soon, but wanted to be sure the deltas between the two make sense.  Comments are welcome.

For convenience, here are the MSE links:


•                     http://w3c.github.io/test-results/media-source/all.html


•                     http://w3c.github.io/test-results/media-source/complete-fails.html


•                     http://w3c.github.io/test-results/media-source/less-than-2.html


Jerry

From: Mark Watson [mailto:watsonm@netflix.com<mailto:watsonm@netflix.com>]
Sent: Friday, August 5, 2016 12:44 PM
To: Henri Sivonen <hsivonen@hsivonen.fi<mailto:hsivonen@hsivonen.fi>>
Cc: Chris Pearce <cpearce@mozilla.com<mailto:cpearce@mozilla.com>>; Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; public-html-media@w3.org<mailto:public-html-media@w3.org>; jyavenard@mozilla.com<mailto:jyavenard@mozilla.com>; Matthew Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> (wolenetz@google.com<mailto:wolenetz@google.com>) <wolenetz@google.com<mailto:wolenetz@google.com>>; Anthony Jones <ajones@mozilla.com<mailto:ajones@mozilla.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>
Subject: Re: Update on MSE and EME Testing

We have been working on additional tests for the Encrypted Media Extensions specification.

Unfortunately, it has been very difficult to get the tests we have been working to work, due to lack of the necessary server support and uncertainty as to what is and is not implemented in  browsers. At this stage, therefore, we do not yet have DRM tests for persistent-license or persistent-usage-record session types.

There are two outstanding pull requests:
- https://github.com/w3c/web-platform-tests/pull/3424 adds support for persistent-license (clearkey)
- https://github.com/w3c/web-platform-tests/pull/3417 adds a waitingforkey test

I expect we will make further progress over the coming week (I will be out, but Sukmal will have some time to work on this).

...Mark


On Fri, Aug 5, 2016 at 7:50 AM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:


On Fri, Aug 5, 2016 at 1:20 AM, Henri Sivonen <hsivonen@hsivonen.fi<mailto:hsivonen@hsivonen.fi>> wrote:
On Thu, Aug 4, 2016 at 7:26 AM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
> I believe that persistent-usage-record functionality works with Firefox /
> Primetime, with some rough edges on the Primetime side. However, we do not
> have a Primetime server for our testing, so we cannot verify this.

While some initial steps (that I've referred to previously on this
list) were taken towards persistent-usage-record -like functionality
on the Firefox side, those steps never reached the point of supporting
the persistent-usage-record API. As Chris said, we have no plans to
implement "persistent-usage-record" sessions.

> The Edge and CastTV implementations are close to compliance, I believe, and we are working on tests that will demonstrate that.

Do those implementations write to plain disk-like storage on CDM
shutdown or do they use some form of tamper-resistant storage during
playback? Which Key System does CastTV use? Was the text in the spec
alone sufficiently detailed to reach interop?

​I don't know the implementation details. They are both PlayReady. They don't reach interop. When I say "close to compliance", I mean that with a bit of JS help you can patch up the non-compliances. At least, that is my hope at present: we don't have everything working yet.

...Mark


--
Henri Sivonen
hsivonen@hsivonen.fi<mailto:hsivonen@hsivonen.fi>
https://hsivonen.fi/