RE: Urgent need for EME test case review

Mark,

I will personally review your PRs tomorrow and am working on additional test dev review.

I thought I’d already endorsed your approach, but I’ll give that one more look tomorrow as well.

Jerry

From: Matt Wolenetz [mailto:wolenetz@google.com]
Sent: Thursday, July 28, 2016 2:49 PM
To: Mark Watson <watsonm@netflix.com>
Cc: Paul Cotton <Paul.Cotton@microsoft.com>; public-html-media@w3.org; Jerry Smith (WPT) <jdsmith@microsoft.com>; Greg Rutz <G.Rutz@cablelabs.com>; David Dorwin <ddorwin@google.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; Francois Daoust <fd@w3.org>; Iraj Sodagar <irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
Subject: Re: Urgent need for EME test case review

In case you all are not aware, @ddorwin has low availability this week. I understand that he's expecting to be back in the office on Monday. I believe he mentioned this during previous editors' sync, but I don't see such in the minutes text.

On Thu, Jul 28, 2016 at 10:38 AM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
All,

I have broken down the test cases I have created into 7 Pull Requests:

https://github.com/w3c/web-platform-tests/pull/3355

https://github.com/w3c/web-platform-tests/pull/3359

https://github.com/w3c/web-platform-tests/pull/3360

https://github.com/w3c/web-platform-tests/pull/3361 (this has a lint problem which I will fix)
https://github.com/w3c/web-platform-tests/pull/3362

https://github.com/w3c/web-platform-tests/pull/3363

https://github.com/w3c/web-platform-tests/pull/3364


You can see them all here<https://github.com/w3c/web-platform-tests/labels/encrypted-media>.

I will close the huge one without merging.

Remaining items on the list for Sukhmal and I are:

  *   Migrate further Google tests (at least those which do no rely in Blink-specific window.internals)
  *   Create WebM versions (though I would prefer someone else to pick this up)
  *   Persistent license tests
  *   Update the content for multi-key / multi-session test to remove discontinuity
  *   Sequential multi-key / single session test
  *   Review Firefox results and check if any failures are due to test bugs / polyfillable browser bugs
  *   Run on Edge and review results to check if any failures are due to test bugs / polyfillable browser bugs
  *   Run on CastTV and review results to check if any failures are due to test bugs / polyfillable browser bugs
We will not be able to run on Safari / IE unless we polyfill the entire API on top of their prefixed versions. This is possible, and not all that much work, but I am not sure how informative it would be.

FWIW, the test results for Chrome (Canary) (runner-results.json) and Firefox 47 (runner-results(1).json) (both on Mac) are attached. As noted above, I have not reviewed the Firefox ones yet.

Btw, I see the test results JSON lists the test file and the test name. Presently, the only difference between the "real" tests and the "polyfill" ones is the test file path - the test names are the same. If we need to also reflect that in the test name, I can do that. LMK.

...Mark




On Wed, Jul 27, 2016 at 1:28 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
All,

The previous pull request has become very large. I will break it up into a sequence of smaller ones. To facilitate this, I need to merge one PR that establishes the latest framework. The other PRs can then be reviewed in parallel.

So, the first of this sequence is https://github.com/w3c/web-platform-tests/pull/3355/files


I URGENTLY need review of this PR. If we are to meet our deadline of August 2, I presume we need the tests running this week and I will have limited time to work on this tomorrow and Friday. I don't think there is much point in attempting to meet the August 2 deadline if noone is able to review the test PRs.

...Mark



On Mon, Jul 25, 2016 at 5:29 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
All,

Update for today: My Pull Request (https://github.com/w3c/web-platform-tests/pull/3324) now has updated mp4 content and consolidated content metadata. Some re-arrangement is still needed to support the AV multi-key case (separate audio and video keys). I expect to get content tomorrow with multiple keys within one video stream. I'll continue to add tests tomorrow.

The space of combinations of content type, key system and init data type is quite large and not fully covered (or all that well organized in the code). Some tests just need to discover a supported combination, others are constrained by what the DRM server supports as well as what the client component supports. I'll also look at how we can organize this better.

As before, review of the PR is welcome!

...Mark



On Mon, Jul 25, 2016 at 11:21 AM, Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>> wrote:
> I will be working on the other tests on and off early next week. If anyone else is able to help, please coordinate with me.

I am moving this discussion [1] to public-html-media@w3.org<mailto:public-html-media@w3.org> so that we touch as many HME WG members as possible.

Mark and others:  Please use this thread for any future updates on the status of EME testing.

/paulc

[1] https://lists.w3.org/Archives/Public/public-hme-editors/2016Jul/0056.html


From: Mark Watson [mailto:watsonm@netflix.com<mailto:watsonm@netflix.com>]
Sent: Friday, July 22, 2016 5:25 PM
To: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>
Cc: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; Greg Rutz <G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>>; David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>>; Matthew Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> (wolenetz@google.com<mailto:wolenetz@google.com>) <wolenetz@google.com<mailto:wolenetz@google.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; Francois Daoust <fd@w3.org<mailto:fd@w3.org>>; public-hme-editors@w3.org<mailto:public-hme-editors@w3.org>; Iraj Sodagar <irajs@microsoft.com<mailto:irajs@microsoft.com>>; John Simmons <johnsim@microsoft.com<mailto:johnsim@microsoft.com>>; Sukhmal Kommidi <skommidi@netflix.com<mailto:skommidi@netflix.com>>
Subject: Re: DRM Today-based test case for EME

All,

I migrated a handful of the Google tests: https://github.com/w3c/web-platform-tests/pull/3324


Currently, the "drm-events" test fails because the DRM server is unaware of the key id that is used (and I am unaware of the key id that it does know). I'm working with Greg on this.

I noticed the following whilst doing this work:

  *   We probably need to organize the content metadata (key ids, keys, MIME types etc.) into one place - presently they are scattered around in a confusing way
  *   The idea of auto-generating the HTML stubs needs some more thought - they are presently a bit different for each test
  *   Some of the Google tests look (to me at least) like they might be Chrome-specific. Specifically, the lifetime tests use window.internals and I am not sure if this is cross-platform. Likewise, I do not know if the garbage collection stuff is cross-platform or not.
  *   We need some additional mp4 content files (I am in touch with Greg about this):

     *   Encrypted audio
     *   Video with multiple keys
I will be working on the other tests on and off early next week. If anyone else is able to help, please coordinate with me.

Sukhmal is working on persistent-usage-record and persistent-license tests, which should be ready next week.

...Mark