Re: Urgent need for EME test case review

In case you all are not aware, @ddorwin has low availability this week. I
understand that he's expecting to be back in the office on Monday. I
believe he mentioned this during previous editors' sync, but I don't see
such in the minutes text.

On Thu, Jul 28, 2016 at 10:38 AM, Mark Watson <watsonm@netflix.com> wrote:

> All,
>
> I have broken down the test cases I have created into 7 Pull Requests:
>
> https://github.com/w3c/web-platform-tests/pull/3355
> https://github.com/w3c/web-platform-tests/pull/3359
> https://github.com/w3c/web-platform-tests/pull/3360
> https://github.com/w3c/web-platform-tests/pull/3361 (this has a lint
> problem which I will fix)
> https://github.com/w3c/web-platform-tests/pull/3362
> https://github.com/w3c/web-platform-tests/pull/3363
> https://github.com/w3c/web-platform-tests/pull/3364
>
> You can see them all here
> <https://github.com/w3c/web-platform-tests/labels/encrypted-media>.
>
> I will close the huge one without merging.
>
> Remaining items on the list for Sukhmal and I are:
>
>    - Migrate further Google tests (at least those which do no rely in
>    Blink-specific window.internals)
>    - Create WebM versions (though I would prefer someone else to pick
>    this up)
>    - Persistent license tests
>    - Update the content for multi-key / multi-session test to remove
>    discontinuity
>    - Sequential multi-key / single session test
>    - Review Firefox results and check if any failures are due to test
>    bugs / polyfillable browser bugs
>    - Run on Edge and review results to check if any failures are due to
>    test bugs / polyfillable browser bugs
>    - Run on CastTV and review results to check if any failures are due to
>    test bugs / polyfillable browser bugs
>
> We will not be able to run on Safari / IE unless we polyfill the entire
> API on top of their prefixed versions. This is possible, and not all that
> much work, but I am not sure how informative it would be.
>
> FWIW, the test results for Chrome (Canary) (runner-results.json) and
> Firefox 47 (runner-results(1).json) (both on Mac) are attached. As noted
> above, I have not reviewed the Firefox ones yet.
>
> Btw, I see the test results JSON lists the test file and the test name.
> Presently, the only difference between the "real" tests and the "polyfill"
> ones is the test file path - the test names are the same. If we need to
> also reflect that in the test name, I can do that. LMK.
>
> ...Mark
>
>
>
>
> On Wed, Jul 27, 2016 at 1:28 PM, Mark Watson <watsonm@netflix.com> wrote:
>
>> All,
>>
>> The previous pull request has become very large. I will break it up into
>> a sequence of smaller ones. To facilitate this, I need to merge one PR that
>> establishes the latest framework. The other PRs can then be reviewed in
>> parallel.
>>
>> So, the first of this sequence is
>> https://github.com/w3c/web-platform-tests/pull/3355/files
>>
>> I URGENTLY need review of this PR. If we are to meet our deadline of
>> August 2, I presume we need the tests running this week and I will have
>> limited time to work on this tomorrow and Friday. I don't think there is
>> much point in attempting to meet the August 2 deadline if noone is able to
>> review the test PRs.
>>
>> ...Mark
>>
>>
>>
>> On Mon, Jul 25, 2016 at 5:29 PM, Mark Watson <watsonm@netflix.com> wrote:
>>
>>> All,
>>>
>>> Update for today: My Pull Request (
>>> https://github.com/w3c/web-platform-tests/pull/3324) now has updated
>>> mp4 content and consolidated content metadata. Some re-arrangement is still
>>> needed to support the AV multi-key case (separate audio and video keys). I
>>> expect to get content tomorrow with multiple keys within one video stream.
>>> I'll continue to add tests tomorrow.
>>>
>>> The space of combinations of content type, key system and init data type
>>> is quite large and not fully covered (or all that well organized in the
>>> code). Some tests just need to discover a supported combination, others are
>>> constrained by what the DRM server supports as well as what the client
>>> component supports. I'll also look at how we can organize this better.
>>>
>>> As before, review of the PR is welcome!
>>>
>>> ...Mark
>>>
>>>
>>>
>>> On Mon, Jul 25, 2016 at 11:21 AM, Paul Cotton <Paul.Cotton@microsoft.com
>>> > wrote:
>>>
>>>> > I will be working on the other tests on and off early next week. If
>>>> anyone else is able to help, please coordinate with me.
>>>>
>>>>
>>>>
>>>> I am moving this discussion [1] to public-html-media@w3.org so that we
>>>> touch as many HME WG members as possible.
>>>>
>>>>
>>>>
>>>> Mark and others:  Please use this thread for any future updates on the
>>>> status of EME testing.
>>>>
>>>>
>>>>
>>>> /paulc
>>>>
>>>>
>>>>
>>>> [1]
>>>> https://lists.w3.org/Archives/Public/public-hme-editors/2016Jul/0056.html
>>>>
>>>>
>>>>
>>>> *From:* Mark Watson [mailto:watsonm@netflix.com]
>>>> *Sent:* Friday, July 22, 2016 5:25 PM
>>>> *To:* Paul Cotton <Paul.Cotton@microsoft.com>
>>>> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Greg Rutz <
>>>> G.Rutz@cablelabs.com>; David Dorwin <ddorwin@google.com>; Matthew
>>>> Wolenetz <wolenetz@google.com> (wolenetz@google.com) <
>>>> wolenetz@google.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>;
>>>> Francois Daoust <fd@w3.org>; public-hme-editors@w3.org; Iraj Sodagar <
>>>> irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Sukhmal
>>>> Kommidi <skommidi@netflix.com>
>>>> *Subject:* Re: DRM Today-based test case for EME
>>>>
>>>>
>>>>
>>>> All,
>>>>
>>>>
>>>>
>>>> I migrated a handful of the Google tests:
>>>> https://github.com/w3c/web-platform-tests/pull/3324
>>>>
>>>>
>>>>
>>>> Currently, the "drm-events" test fails because the DRM server is
>>>> unaware of the key id that is used (and I am unaware of the key id that it
>>>> does know). I'm working with Greg on this.
>>>>
>>>>
>>>>
>>>> I noticed the following whilst doing this work:
>>>>
>>>>    - We probably need to organize the content metadata (key ids, keys,
>>>>    MIME types etc.) into one place - presently they are scattered around in a
>>>>    confusing way
>>>>    - The idea of auto-generating the HTML stubs needs some more
>>>>    thought - they are presently a bit different for each test
>>>>    - Some of the Google tests look (to me at least) like they might be
>>>>    Chrome-specific. Specifically, the lifetime tests use window.internals and
>>>>    I am not sure if this is cross-platform. Likewise, I do not know if the
>>>>    garbage collection stuff is cross-platform or not.
>>>>    - We need some additional mp4 content files (I am in touch with
>>>>    Greg about this):
>>>>
>>>>
>>>>    - Encrypted audio
>>>>       - Video with multiple keys
>>>>
>>>> I will be working on the other tests on and off early next week. If
>>>> anyone else is able to help, please coordinate with me.
>>>>
>>>>
>>>>
>>>> Sukhmal is working on persistent-usage-record and persistent-license
>>>> tests, which should be ready next week.
>>>>
>>>>
>>>>
>>>> ...Mark
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>