Re: Update on MSE and EME Testing

On Fri, Aug 19, 2016 at 3:09 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> Also, we should likely turn off the tests in the Google sub-folder, and
> start focusing on the modified versions in the main one.  That can be done
> by commenting out the testharness js script lines.  Are we ready to do that?
>
>
>
> Jerry
>
>
>
> *From:* Jerry Smith (WPT)
> *Sent:* Friday, August 19, 2016 2:10 PM
> *To:* 'David Dorwin' <ddorwin@google.com>; Mark Watson <
> watsonm@netflix.com>
> *Cc:* Henri Sivonen <hsivonen@hsivonen.fi>; Chris Pearce <
> cpearce@mozilla.com>; public-html-media@w3.org; jyavenard@mozilla.com;
> Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <
> wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le
> Hegaret (plh@w3.org) <plh@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com>
> *Subject:* RE: Update on MSE and EME Testing
>
>
>
> I intend to update these test reports soon.  Of issues raised below, I
> believe:
>
>    - There is a bug in util/utils.js in getSupportedKeysystem() which
>    returns com.widevine.alpha for Edge, meaning that playback tests on Edge
>    all fail. I presume you fixed this before generating the results.
>       - Jerry:  This has been resolved
>    - The "less than 2" script should not consider two versions of Chrome
>    or two versions of Firefox as 2 versions passing
>       - Jerry:  This will resolve by having just one version for each
>       browser
>    - The test names for the drm tests include the keysystem. So, when you
>    have it working with appropriate keysystems on each browser, they will not
>    match up correctly on the test-results page. We'll need to remove the
>    keysystem name from the test name and just put "drm"
>       - Jerry:  This has been resolved
>    - The test names for the polyfill tests should probably start
>    "Polyfill" for readability
>       - Jerry:  This hasn’t yet been changed – Mark, can the polyfill
>       make script be modified to do this directly?
>
> ​Yes, I can do that. I ought to change it to a Python script too, since
WPT apparently doesn't want people to use Makefiles.

>
>    -
>       -
>
> I looked also at this issue raised by David:
>
> ·       I don't understand some of the output. For example,
> http://www.w3c-test.org/encrypted-media/Google/
> encrypted-media-keystatuses.html has no result listed for several
> browsers. It passes in the latest Chrome but there is no entry. Also,
> TIMEOUT seems to appear inconsistently in some header rows. The
> persistent-usage-record tests don't even have results. Does this mean they
> are new / not run?
>
> The JSONs contain test results, but they aren’t showing up in the test
> reports.  I reduced the JSONs down to CH54, ED14 and FF51 and created a
> report based just on those; where ED14 and FF51 have timeouts and CH54
> passes on the encrypted-media-keystatuses test.  The resulting report still
> shows a blank result.  There is no obvious difference between this test,
> where results aren’t showing up correctly, and other tests where the report
> is complete.  I’m still looking for an explanation.
>
> Jerry
>
> *From:* David Dorwin [mailto:ddorwin@google.com <ddorwin@google.com>]
> *Sent:* Monday, August 8, 2016 4:54 PM
> *To:* Mark Watson <watsonm@netflix.com>
> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Henri Sivonen <
> hsivonen@hsivonen.fi>; Chris Pearce <cpearce@mozilla.com>;
> public-html-media@w3.org; jyavenard@mozilla.com; Matthew Wolenetz <
> wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Anthony
> Jones <ajones@mozilla.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>;
> Paul Cotton <Paul.Cotton@microsoft.com>
>
> *Subject:* Re: Update on MSE and EME Testing
>
>
>
>
>    - There is a bug in util/utils.js in getSupportedKeysystem() which
>    returns com.widevine.alpha for Edge, meaning that playback tests on Edge
>    all fail. I presume you fixed this before generating the results.
>
> Mark has a fix for this in https://github.com/w3c/web-
> platform-tests/pull/3424/files#diff-c5c73a9a657afac9f2d88b36c6348243 if
> someone wants to land that separately.
>
>
>
> I plan to pull down the older versions of Chrome and Firefox soon, but
> wanted to be sure the deltas between the two make sense.  Comments are
> welcome.
>
> Jerry, the two failures in CH52 where CH54 passes are fine:
>
>    - /encrypted-media/Google/encrypted-media-async-setcert-with-gc.html
>    fails in CH52 because we changed the behavior and test to be spec compliant.
>    - /encrypted-media/clearkey-mp4-playback-temporary-events.html fails
>    because of a bug that we fixed when it was identified by this test.
>
>
>
> I don't understand some of the output. For example, http://www.w3c-test.
> org/encrypted-media/Google/encrypted-media-keystatuses.html has no result
> listed for several browsers. It passes in the latest Chrome but there is no
> entry. Also, TIMEOUT seems to appear inconsistently in some header rows.
> The persistent-usage-record tests don't even have results. Does this mean
> they are new / not run?
>
>
>
> On Fri, Aug 5, 2016 at 2:14 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> Jerry,
>
>
>
> There are a few things which need to be fixed up before we can really read
> anything into the test results:
>
>    - There is a bug in util/utils.js in getSupportedKeysystem() which
>    returns com.widevine.alpha for Edge, meaning that playback tests on Edge
>    all fail. I presume you fixed this before generating the results.
>    - The "less than 2" script should not consider two versions of Chrome
>    or two versions of Firefox as 2 versions passing
>    - The test names for the drm tests include the keysystem. So, when you
>    have it working with appropriate keysystems on each browser, they will not
>    match up correctly on the test-results page. We'll need to remove the
>    keysystem name from the test name and just put "drm"
>    - The test names for the polyfill tests should probably start
>    "Polyfill" for readability
>
> ...Mark
>
>
>
> On Fri, Aug 5, 2016 at 1:44 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
> I’ve made an initial posting of test results on a few browsers (Edge, two
> versions of Chrome and two also of Firefox).  It’s a bit messy, because it
> includes Google subfolder tests for Clear-Key (dups) and also tests with
> and without some polyfills Mark has been developing.
>
>
>
> You can see current EME results using these links:
>
>
>
> •                     http://w3c.github.io/test-
> results/encrypted-media/all.html
>
> •                     http://w3c.github.io/test-results/encrypted-media/
> complete-fails.html
>
> •                     http://w3c.github.io/test-
> results/encrypted-media/less-than-2.html
>
>
>
> I plan to pull down the older versions of Chrome and Firefox soon, but
> wanted to be sure the deltas between the two make sense.  Comments are
> welcome.
>
>
>
> For convenience, here are the MSE links:
>
>
>
> •                     http://w3c.github.io/test-
> results/media-source/all.html
>
> •                     http://w3c.github.io/test-
> results/media-source/complete-fails.html
>
> •                     http://w3c.github.io/test-results/media-source/less-
> than-2.html
>
>
>
> Jerry
>
>
>
> *From:* Mark Watson [mailto:watsonm@netflix.com]
> *Sent:* Friday, August 5, 2016 12:44 PM
> *To:* Henri Sivonen <hsivonen@hsivonen.fi>
> *Cc:* Chris Pearce <cpearce@mozilla.com>; Jerry Smith (WPT) <
> jdsmith@microsoft.com>; public-html-media@w3.org; jyavenard@mozilla.com;
> Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <
> wolenetz@google.com>; Anthony Jones <ajones@mozilla.com>; Philippe Le
> Hegaret (plh@w3.org) <plh@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com>
> *Subject:* Re: Update on MSE and EME Testing
>
>
>
> We have been working on additional tests for the Encrypted Media
> Extensions specification.
>
>
>
> Unfortunately, it has been very difficult to get the tests we have been
> working to work, due to lack of the necessary server support and
> uncertainty as to what is and is not implemented in  browsers. At this
> stage, therefore, we do not yet have DRM tests for persistent-license or
> persistent-usage-record session types.
>
>
>
> There are two outstanding pull requests:
>
> - https://github.com/w3c/web-platform-tests/pull/3424 adds support for
> persistent-license (clearkey)
>
> - https://github.com/w3c/web-platform-tests/pull/3417 adds a
> waitingforkey test
>
>
>
> I expect we will make further progress over the coming week (I will be
> out, but Sukmal will have some time to work on this).
>
>
>
> ...Mark
>
>
>
>
>
> On Fri, Aug 5, 2016 at 7:50 AM, Mark Watson <watsonm@netflix.com> wrote:
>
>
>
>
>
> On Fri, Aug 5, 2016 at 1:20 AM, Henri Sivonen <hsivonen@hsivonen.fi>
> wrote:
>
> On Thu, Aug 4, 2016 at 7:26 AM, Mark Watson <watsonm@netflix.com> wrote:
> > I believe that persistent-usage-record functionality works with Firefox /
> > Primetime, with some rough edges on the Primetime side. However, we do
> not
> > have a Primetime server for our testing, so we cannot verify this.
>
> While some initial steps (that I've referred to previously on this
> list) were taken towards persistent-usage-record -like functionality
> on the Firefox side, those steps never reached the point of supporting
> the persistent-usage-record API. As Chris said, we have no plans to
> implement "persistent-usage-record" sessions.
>
> > The Edge and CastTV implementations are close to compliance, I believe,
> and we are working on tests that will demonstrate that.
>
> Do those implementations write to plain disk-like storage on CDM
> shutdown or do they use some form of tamper-resistant storage during
> playback? Which Key System does CastTV use? Was the text in the spec
> alone sufficiently detailed to reach interop?
>
>
>
> ​I don't know the implementation details. They are both PlayReady. They
> don't reach interop. When I say "close to compliance", I mean that with a
> bit of JS help you can patch up the non-compliances. At least, that is my
> hope at present: we don't have everything working yet.
>
>
>
> ...Mark
>
>
>
>
> --
> Henri Sivonen
> hsivonen@hsivonen.fi
> https://hsivonen.fi/
>
>
>
>
>
>
>
>
>