Re: DRM Today-based test case for EME

Jerry, that sounds promising. For the multidrm tests, maybe we can have
Clear Key, System1, ... SystemN results, which would be useful for some
things, along with a single result for any non-Clear Key system. For the
implementation report, that final result and maybe the Clear Key result
would be most interesting along with clearkey. It would be nice if the
infrastructure we create also supported also testing multiple media the
types, but I don't think we need that in the report for PR.

The directories look fine except that we might want to move the test files
under a single top-level directory (e.g. data/ or content/)

Mark wrote:

> We would then have mp4 versions of all the tests and we may want to
> (re)create some WebM ones. I don't expect we need to do every test with
> both WebM and mp4.
>

If the  MP4s are encrypted with the same keys as the WebM files, it should
be straightforward to support both from the beginning. If you need any new
WebM files, let Matt and me know.

https://github.com/w3c/web-platform-tests/pull/3317 should make supporting
multiple types easier since the tests now try to find any supported type.
We still only have one set of media files for the playback tests, but the
more use of types is more abstracted. Please review and merge this PR. This
is the last pending change we have from Chromium.

David

On Wed, Jul 20, 2016 at 3:37 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> A RegExpr can tell the runner to repeat each found test (under some path)
> to re-run for a list of keySystems?  That sounds pretty good.
>
>
>
> Does this work better if scripts are in a sub-folder?  If so, then maybe
> these folders under encrypted-media make sense:
>
>
>
> -          clearkey
>
> -          multidrm
>
> -          mp4
>
> -          webm
>
> -          util
>
>
>
> Jerry
>
>
>
> *From:* Mark Watson [mailto:watsonm@netflix.com]
> *Sent:* Wednesday, July 20, 2016 3:29 PM
> *To:* David Dorwin <ddorwin@google.com>
> *Cc:* Greg Rutz <G.Rutz@cablelabs.com>; Matthew Wolenetz <
> wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Jerry
> Smith (WPT) <jdsmith@microsoft.com>; Philippe Le Hegaret (plh@w3.org) <
> plh@w3.org>; Francois Daoust <fd@w3.org>; public-hme-editors@w3.org; Iraj
> Sodagar <irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Paul
> Cotton <Paul.Cotton@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
> *Subject:* Re: DRM Today-based test case for EME
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 3:11 PM, David Dorwin <ddorwin@google.com> wrote:
>
> The abstraction Greg describes makes sense, at least to my rough
> understanding. Greg, would we vary the test configurations or are all
> configurations always present and just a way of isolating the logic for
> each key system?
>
>
>
> In case there is any uncertainty, I want to emphasize that most of the
> "Google clearkey tests" are really just EME API tests that happen to use
> Clear Key. (The reason they use Clear Key (and WebM) has is related to the
> fact that they are Blink layout tests that run inside a subset of the code,
> pass in Chromium, and not depend on external servers.) Most interact with
> at least a portion of the Clear Key CDM implementation, meaning the
> behavior and results depend in part on the Clear Key implementation. This
> is similar to how most media tests are also testing a specific
> pipeline/decoder. There are some tests that explicitly test Clear Key
> behavior defined in https://w3c.github.io/encrypted-media/#clear-key, and
> we should ensure these are labeled "clearkey" in the path. Everything else
> should probably be converted to general tests.
>
>
>
> ​Ok, so IIUC, the process we should follow for each test currently in the
> Google directory (and any others we want to add) is:
>
> (i) migrate this test to the framework / utilities we have just proposed,
> including the drmtoday infractructure, to create a test using a real DRM
>
> (ii) make a copy of that test that just uses the Clear Key options in that
> same framework / utilities
>
>
>
> (It may not make sense to do both for every test)
>
>
>
> After we have migrated all the tests, we can remove the Google directory.
>
>
>
> We would then have mp4 versions of all the tests and we may want to
> (re)create some WebM ones. I don't expect we need to do every test with
> both WebM and mp4.
>
>
>
> The only way I can see to selectively run tests is to specify a path or
> RegExp in the test runner, so ​we should agree on a naming convention
> and/or folder heirarchy to organize the tests.
>
>
>
>
>
> Mark, my concern is that using Clear Key, which is almost certainly
> simpler than any other system, could paper over API design, etc. issues for
> other systems. In practice, I don't think this should be an issue since
> Edge doesn't implement Clear Key. (Thus, I also think we should err on the
> side of excluding Clear Key for now.)
>
>
>
> ​It's a valid concern, but so is the problem that we have a hard deadline,
> so I think we should err on the side of gathering as much evidence as we
> can and providing it with appropriate caveats.
>
>
>
> ...Mark​
>
>
>
>
>
>
>
> For full coverage, all supported combinations would be executed (something
> I discussed
> <https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0100.html>
> earlier
> <https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0104.html>). It
> would be nice if we could get results for the general tests run on each key
> system (and type), but we'd need to create some infrastructure.
>
>
>
> David
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 1:17 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> Greg - this makes sense and it would be easy to take the drmtoday test we
> have written and make a new clearkey version of that by enhancing the utils
> and the config as you describe.
>
>
>
> However, we already have a clearkey version of that test in the Google
> directory (which uses its own utils). So, doing what you say would increase
> the commonality / consistency between the tests, but it wouldn't get us
> more tests.
>
>
>
> David - the clearkey results are useful information for the implementation
> report. Again, as with tests based on polyfills, they validate the API
> design, implementability and specification. These are factors in the
> decision as well as the current state of commercially useful features in
> commercial browsers. We are in the unusual situation of not being able to
> just wait until implementations have matured, so this is going to be an
> unusual decision.
>
>
>
> ...Mark
>
>
>
> On Wed, Jul 20, 2016 at 1:05 PM, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
> For (B), I wasn’t suggesting that there be two different tests in one
> file, I was suggesting that we put operations like license requests into
> utils files that would perform either DRMToday or ClearKey license
> requests.  For DRMToday, the implementation in these utils files would make
> the request to the actual DRMToday license server.  For ClearKey, the
> implementation would likely return a response message that is placed into
> the test configuration JSON (drmconfig.json in the example test created by
> Sukhmal).  The JSON config file can help configure both the key system and
> the desired license response message that we need in order to properly
> execute the test.
>
>
>
> G
>
>
>
> On 7/20/16, 1:30 PM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
> So, what we have right now is:
>
> (1) A large number of ClearKey-only tests in a "Google" folder, and
>
> (2) One of those tests (basic playback) migrated to DRM Today, in the root
> folder
>
>
>
> There are two approaches:
>
> (A) Keep ClearKey and DRM tests separate: move the "Google" tests into the
> root or a "clearkey" folder, continue making new DRMToday versions of each
> of those ClearKey tests
>
> (B) Make the DRMToday test also support ClearKey, continue making new
> ClearKey+DRMToday versions of each of the Google tests and, eventually,
> drop the Google folder
>
>
>
> For (B), we need to run two tests in one file, which requires some care
> with async tests and there's been comments that we should not have multiple
> tests in one file.
>
>
>
> Opinions ?
>
>
>
> ...Mark
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 11:27 AM, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
> I think the test utilities should be designed to be as DRM-independent as
> possible.  This would allow us to run any of the test cases that apply to
> ClearKey simply by providing a DRMConfig and test content that indicates
> use of ClearKey.  I apologize that I have not been following the EME spec
> progression that much over the last 12-18 months, but I recall there not
> being a ton of differences between ClearKey support and other DRMs as I
> implemented it in dash.js.
>
>
>
> For test cases that are valid for ClearKey, the test case would simply
> execute multiple times on the UA under test — once with ClearKey content
> and one or more additional times for the “real” DRMs that are to be tested
> on that UA.  No sense in maintaining separate test code if we don’t have to.
>
>
>
> G
>
>
>
> On 7/20/16, 10:34 AM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
> Question: should we expand this test case to cover ClearKey ? Or will we
> rely on the tests in the Google folder for ClearKey ?
>
>
>
> If the latter, should we move those tests into the main directory (I see
> they are now working) ? Or, if others would like to add ClearKey tests,
> should they add them to the Google folder ?
>
>
>
> ...Mark
>
>
>
> On Tue, Jul 19, 2016 at 7:18 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> All,
>
>
>
> Sukhmal has created a Pull Request for a temporary session test case using
> DRM Today. We have tested this on Chrome with Widevine and it should work
> on Edge with PlayReady as well:
>
>
>
> https://github.com/w3c/web-platform-tests/pull/3313
>
>
>
> Please review this and comment on whether it is a good template / model
> for us to work from. We can quickly migrate more of the Google clearkey
> tests to drmtoday as well as implementing tests for other session types
> based on this model.
>
>
>
> ...Mark
>
>
>
>
>
>
>
>
>
>
>