Re: Urgent need for EME test case review

All,

I have broken down the test cases I have created into 7 Pull Requests:

https://github.com/w3c/web-platform-tests/pull/3355
https://github.com/w3c/web-platform-tests/pull/3359
https://github.com/w3c/web-platform-tests/pull/3360
https://github.com/w3c/web-platform-tests/pull/3361 (this has a lint
problem which I will fix)
https://github.com/w3c/web-platform-tests/pull/3362
https://github.com/w3c/web-platform-tests/pull/3363
https://github.com/w3c/web-platform-tests/pull/3364

You can see them all here
<https://github.com/w3c/web-platform-tests/labels/encrypted-media>.

I will close the huge one without merging.

Remaining items on the list for Sukhmal and I are:

   - Migrate further Google tests (at least those which do no rely in
   Blink-specific window.internals)
   - Create WebM versions (though I would prefer someone else to pick this
   up)
   - Persistent license tests
   - Update the content for multi-key / multi-session test to remove
   discontinuity
   - Sequential multi-key / single session test
   - Review Firefox results and check if any failures are due to test bugs
   / polyfillable browser bugs
   - Run on Edge and review results to check if any failures are due to
   test bugs / polyfillable browser bugs
   - Run on CastTV and review results to check if any failures are due to
   test bugs / polyfillable browser bugs

We will not be able to run on Safari / IE unless we polyfill the entire API
on top of their prefixed versions. This is possible, and not all that much
work, but I am not sure how informative it would be.

FWIW, the test results for Chrome (Canary) (runner-results.json) and
Firefox 47 (runner-results(1).json) (both on Mac) are attached. As noted
above, I have not reviewed the Firefox ones yet.

Btw, I see the test results JSON lists the test file and the test name.
Presently, the only difference between the "real" tests and the "polyfill"
ones is the test file path - the test names are the same. If we need to
also reflect that in the test name, I can do that. LMK.

...Mark




On Wed, Jul 27, 2016 at 1:28 PM, Mark Watson <watsonm@netflix.com> wrote:

> All,
>
> The previous pull request has become very large. I will break it up into a
> sequence of smaller ones. To facilitate this, I need to merge one PR that
> establishes the latest framework. The other PRs can then be reviewed in
> parallel.
>
> So, the first of this sequence is
> https://github.com/w3c/web-platform-tests/pull/3355/files
>
> I URGENTLY need review of this PR. If we are to meet our deadline of
> August 2, I presume we need the tests running this week and I will have
> limited time to work on this tomorrow and Friday. I don't think there is
> much point in attempting to meet the August 2 deadline if noone is able to
> review the test PRs.
>
> ...Mark
>
>
>
> On Mon, Jul 25, 2016 at 5:29 PM, Mark Watson <watsonm@netflix.com> wrote:
>
>> All,
>>
>> Update for today: My Pull Request (
>> https://github.com/w3c/web-platform-tests/pull/3324) now has updated mp4
>> content and consolidated content metadata. Some re-arrangement is still
>> needed to support the AV multi-key case (separate audio and video keys). I
>> expect to get content tomorrow with multiple keys within one video stream.
>> I'll continue to add tests tomorrow.
>>
>> The space of combinations of content type, key system and init data type
>> is quite large and not fully covered (or all that well organized in the
>> code). Some tests just need to discover a supported combination, others are
>> constrained by what the DRM server supports as well as what the client
>> component supports. I'll also look at how we can organize this better.
>>
>> As before, review of the PR is welcome!
>>
>> ...Mark
>>
>>
>>
>> On Mon, Jul 25, 2016 at 11:21 AM, Paul Cotton <Paul.Cotton@microsoft.com>
>> wrote:
>>
>>> > I will be working on the other tests on and off early next week. If
>>> anyone else is able to help, please coordinate with me.
>>>
>>>
>>>
>>> I am moving this discussion [1] to public-html-media@w3.org so that we
>>> touch as many HME WG members as possible.
>>>
>>>
>>>
>>> Mark and others:  Please use this thread for any future updates on the
>>> status of EME testing.
>>>
>>>
>>>
>>> /paulc
>>>
>>>
>>>
>>> [1]
>>> https://lists.w3.org/Archives/Public/public-hme-editors/2016Jul/0056.html
>>>
>>>
>>>
>>> *From:* Mark Watson [mailto:watsonm@netflix.com]
>>> *Sent:* Friday, July 22, 2016 5:25 PM
>>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>
>>> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Greg Rutz <
>>> G.Rutz@cablelabs.com>; David Dorwin <ddorwin@google.com>; Matthew
>>> Wolenetz <wolenetz@google.com> (wolenetz@google.com) <
>>> wolenetz@google.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>;
>>> Francois Daoust <fd@w3.org>; public-hme-editors@w3.org; Iraj Sodagar <
>>> irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Sukhmal
>>> Kommidi <skommidi@netflix.com>
>>> *Subject:* Re: DRM Today-based test case for EME
>>>
>>>
>>>
>>> All,
>>>
>>>
>>>
>>> I migrated a handful of the Google tests:
>>> https://github.com/w3c/web-platform-tests/pull/3324
>>>
>>>
>>>
>>> Currently, the "drm-events" test fails because the DRM server is unaware
>>> of the key id that is used (and I am unaware of the key id that it does
>>> know). I'm working with Greg on this.
>>>
>>>
>>>
>>> I noticed the following whilst doing this work:
>>>
>>>    - We probably need to organize the content metadata (key ids, keys,
>>>    MIME types etc.) into one place - presently they are scattered around in a
>>>    confusing way
>>>    - The idea of auto-generating the HTML stubs needs some more thought
>>>    - they are presently a bit different for each test
>>>    - Some of the Google tests look (to me at least) like they might be
>>>    Chrome-specific. Specifically, the lifetime tests use window.internals and
>>>    I am not sure if this is cross-platform. Likewise, I do not know if the
>>>    garbage collection stuff is cross-platform or not.
>>>    - We need some additional mp4 content files (I am in touch with Greg
>>>    about this):
>>>
>>>
>>>    - Encrypted audio
>>>       - Video with multiple keys
>>>
>>> I will be working on the other tests on and off early next week. If
>>> anyone else is able to help, please coordinate with me.
>>>
>>>
>>>
>>> Sukhmal is working on persistent-usage-record and persistent-license
>>> tests, which should be ready next week.
>>>
>>>
>>>
>>> ...Mark
>>>
>>>
>>>
>>>
>>>
>>
>>
>