Re: Urgent need for EME test case review

Thanks, Jerry, for the reviews!

I have merged all the PRs. Please note that regarding the polyfills, you
don't get any of the polyfill tests unless you go into the polyfill
directory and run make. If you don't do that, you'll get the normal tests
and the polyfill files will have no effect. If you run make, then a new
copy of each test will be made in the polyfill directory, so you will get
tests with and without the polyfills.

The polyfills fulfil different purposes: the "clear key" polyfill adds
support for persistent sessions to a browser that supports only temporary
for clear key. It's purpose is to allow the tests for those features to run
and validate the API design. Obviously, it does not demonstrate native
browser interop in any way. The others pretty much patch up minor browser
differences.

...Mark

On Thu, Jul 28, 2016 at 4:00 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> Mark,
>
>
>
> I will personally review your PRs tomorrow and am working on additional
> test dev review.
>
>
>
> I thought I’d already endorsed your approach, but I’ll give that one more
> look tomorrow as well.
>
>
>
> Jerry
>
>
>
> *From:* Matt Wolenetz [mailto:wolenetz@google.com]
> *Sent:* Thursday, July 28, 2016 2:49 PM
> *To:* Mark Watson <watsonm@netflix.com>
> *Cc:* Paul Cotton <Paul.Cotton@microsoft.com>; public-html-media@w3.org;
> Jerry Smith (WPT) <jdsmith@microsoft.com>; Greg Rutz <G.Rutz@cablelabs.com>;
> David Dorwin <ddorwin@google.com>; Philippe Le Hegaret (plh@w3.org) <
> plh@w3.org>; Francois Daoust <fd@w3.org>; Iraj Sodagar <
> irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Sukhmal
> Kommidi <skommidi@netflix.com>
> *Subject:* Re: Urgent need for EME test case review
>
>
>
> In case you all are not aware, @ddorwin has low availability this week. I
> understand that he's expecting to be back in the office on Monday. I
> believe he mentioned this during previous editors' sync, but I don't see
> such in the minutes text.
>
>
>
> On Thu, Jul 28, 2016 at 10:38 AM, Mark Watson <watsonm@netflix.com> wrote:
>
> All,
>
>
>
> I have broken down the test cases I have created into 7 Pull Requests:
>
>
>
> https://github.com/w3c/web-platform-tests/pull/3355
>
> https://github.com/w3c/web-platform-tests/pull/3359
>
> https://github.com/w3c/web-platform-tests/pull/3360
>
> https://github.com/w3c/web-platform-tests/pull/3361 (this has a lint
> problem which I will fix)
>
> https://github.com/w3c/web-platform-tests/pull/3362
>
> https://github.com/w3c/web-platform-tests/pull/3363
>
> https://github.com/w3c/web-platform-tests/pull/3364
>
>
>
> You can see them all here
> <https://github.com/w3c/web-platform-tests/labels/encrypted-media>.
>
>
>
> I will close the huge one without merging.
>
>
>
> Remaining items on the list for Sukhmal and I are:
>
>    - Migrate further Google tests (at least those which do no rely in
>    Blink-specific window.internals)
>    - Create WebM versions (though I would prefer someone else to pick
>    this up)
>    - Persistent license tests
>    - Update the content for multi-key / multi-session test to remove
>    discontinuity
>    - Sequential multi-key / single session test
>    - Review Firefox results and check if any failures are due to test
>    bugs / polyfillable browser bugs
>    - Run on Edge and review results to check if any failures are due to
>    test bugs / polyfillable browser bugs
>    - Run on CastTV and review results to check if any failures are due to
>    test bugs / polyfillable browser bugs
>
> We will not be able to run on Safari / IE unless we polyfill the entire
> API on top of their prefixed versions. This is possible, and not all that
> much work, but I am not sure how informative it would be.
>
>
>
> FWIW, the test results for Chrome (Canary) (runner-results.json) and
> Firefox 47 (runner-results(1).json) (both on Mac) are attached. As noted
> above, I have not reviewed the Firefox ones yet.
>
>
>
> Btw, I see the test results JSON lists the test file and the test name.
> Presently, the only difference between the "real" tests and the "polyfill"
> ones is the test file path - the test names are the same. If we need to
> also reflect that in the test name, I can do that. LMK.
>
>
>
> ...Mark
>
>
>
>
>
>
>
>
>
> On Wed, Jul 27, 2016 at 1:28 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> All,
>
>
>
> The previous pull request has become very large. I will break it up into a
> sequence of smaller ones. To facilitate this, I need to merge one PR that
> establishes the latest framework. The other PRs can then be reviewed in
> parallel.
>
>
>
> So, the first of this sequence is
> https://github.com/w3c/web-platform-tests/pull/3355/files
>
>
>
> I URGENTLY need review of this PR. If we are to meet our deadline of
> August 2, I presume we need the tests running this week and I will have
> limited time to work on this tomorrow and Friday. I don't think there is
> much point in attempting to meet the August 2 deadline if noone is able to
> review the test PRs.
>
>
>
> ...Mark
>
>
>
>
>
>
>
> On Mon, Jul 25, 2016 at 5:29 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> All,
>
>
>
> Update for today: My Pull Request (
> https://github.com/w3c/web-platform-tests/pull/3324) now has updated mp4
> content and consolidated content metadata. Some re-arrangement is still
> needed to support the AV multi-key case (separate audio and video keys). I
> expect to get content tomorrow with multiple keys within one video stream.
> I'll continue to add tests tomorrow.
>
>
>
> The space of combinations of content type, key system and init data type
> is quite large and not fully covered (or all that well organized in the
> code). Some tests just need to discover a supported combination, others are
> constrained by what the DRM server supports as well as what the client
> component supports. I'll also look at how we can organize this better.
>
>
>
> As before, review of the PR is welcome!
>
>
>
> ...Mark
>
>
>
>
>
>
>
> On Mon, Jul 25, 2016 at 11:21 AM, Paul Cotton <Paul.Cotton@microsoft.com>
> wrote:
>
> > I will be working on the other tests on and off early next week. If
> anyone else is able to help, please coordinate with me.
>
>
>
> I am moving this discussion [1] to public-html-media@w3.org so that we
> touch as many HME WG members as possible.
>
>
>
> Mark and others:  Please use this thread for any future updates on the
> status of EME testing.
>
>
>
> /paulc
>
>
>
> [1]
> https://lists.w3.org/Archives/Public/public-hme-editors/2016Jul/0056.html
>
>
>
> *From:* Mark Watson [mailto:watsonm@netflix.com]
> *Sent:* Friday, July 22, 2016 5:25 PM
> *To:* Paul Cotton <Paul.Cotton@microsoft.com>
> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Greg Rutz <
> G.Rutz@cablelabs.com>; David Dorwin <ddorwin@google.com>; Matthew
> Wolenetz <wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>;
> Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; Francois Daoust <fd@w3.org>;
> public-hme-editors@w3.org; Iraj Sodagar <irajs@microsoft.com>; John
> Simmons <johnsim@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
> *Subject:* Re: DRM Today-based test case for EME
>
>
>
> All,
>
>
>
> I migrated a handful of the Google tests:
> https://github.com/w3c/web-platform-tests/pull/3324
>
>
>
> Currently, the "drm-events" test fails because the DRM server is unaware
> of the key id that is used (and I am unaware of the key id that it does
> know). I'm working with Greg on this.
>
>
>
> I noticed the following whilst doing this work:
>
>    - We probably need to organize the content metadata (key ids, keys,
>    MIME types etc.) into one place - presently they are scattered around in a
>    confusing way
>    - The idea of auto-generating the HTML stubs needs some more thought -
>    they are presently a bit different for each test
>    - Some of the Google tests look (to me at least) like they might be
>    Chrome-specific. Specifically, the lifetime tests use window.internals and
>    I am not sure if this is cross-platform. Likewise, I do not know if the
>    garbage collection stuff is cross-platform or not.
>    - We need some additional mp4 content files (I am in touch with Greg
>    about this):
>
>
>    - Encrypted audio
>       - Video with multiple keys
>
> I will be working on the other tests on and off early next week. If anyone
> else is able to help, please coordinate with me.
>
>
>
> Sukhmal is working on persistent-usage-record and persistent-license
> tests, which should be ready next week.
>
>
>
> ...Mark
>
>
>
>
>
>
>
>
>
>
>
>
>