Re: DRM Today-based test case for EME

On Thu, Jul 21, 2016 at 12:45 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> Mark:  At one point, you mentioned that it should be sufficient for “drm”
> tests to just test the single supported CDM.  This may be okay for our V1
> test suite, but I’m not sure it’s an assumption that will hold up over
> time.  If a browser supports two CDMs, then we need a way for it to control
> testing of each.  We specifically will want to avoid the test logic going
> through a list of testable DRMs and testing the first one it finds as
> supported.
>
>
>
> We may elect to use the single CDM approach for now, but it would be good
> to give some thought into what it would mean to support multiple.  The
> brute force, but not very scalable solution, would be to clone the tests
> per tested DRM.
>

​I think we have settled on the main test code being JS files which each
run one test for a provided (keysystem, media)​ pair.

Then we will hand construct, and eventually auto-generate, HTML files for
the combinations we want.

There are various ways the auto-generated HTML files could run tests for
multiple DRMs on a browser that supported multiple, but I think we can work
that out later: the main test logic will not need to change.

...Mark



>
>
> Jerry
>
>
>
> *From:* Greg Rutz [mailto:G.Rutz@cablelabs.com]
> *Sent:* Thursday, July 21, 2016 12:36 PM
> *To:* Mark Watson <watsonm@netflix.com>; David Dorwin <ddorwin@google.com>
> *Cc:* Jerry Smith (WPT) <jdsmith@microsoft.com>; Matthew Wolenetz <
> wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>;
> Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; Francois Daoust <fd@w3.org>;
> public-hme-editors@w3.org; Iraj Sodagar <irajs@microsoft.com>; John
> Simmons <johnsim@microsoft.com>; Paul Cotton <Paul.Cotton@microsoft.com>;
> Sukhmal Kommidi <skommidi@netflix.com>
>
> *Subject:* Re: DRM Today-based test case for EME
>
>
>
> This is excellent.  I’m glad we came up with a way to minimize code
> duplication while still working within the W3C framework.
>
>
>
> On 7/21/16, 1:13 PM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
>
>
>
>
> On Thu, Jul 21, 2016 at 11:49 AM, David Dorwin <ddorwin@google.com> wrote:
>
> I assumed the mp4 and webm directories were just the content, which is
> currently the case. Other than some targeted tests, such as testing
> specific initDataTypes, "encrypted" event generation for various formats,
> or testing playback of specific types, most tests should be
> media-independent. See how
> https://github.com/w3c/web-platform-tests/pull/3317 finds any supported
> type and uses it. (The tests that use media files need some additional
> work.)
>
>
>
> Thus, I think (drm|clearkey)-xxxx.html should be sufficient. It would be
> nice if we didn't need to maintain wrappers, but this will work for now.
> Writing the tests in .js files also makes it easier to add more tests later
> if we or implementers wish. We should design the JS files with such
> extensibility in mind. For example:
>
> function runTest(keySystem = null, mediaType/Config = null) {
>
> if (!keySystem) selectSupportedNonClearKeyKeySystem();
>
> if (!mediaType) getSupportedConfigAndMediaFiles();
>
> // Do test.
>
> }
>
>
>
> While not required now, it would be nice if we could automatically
> generate the .html files with a script. For example, for each file in the
> test-scripts/ directory, generate an HTML file that calls it for each of
> "drm" and "clearkey. Again, implementers and others could update this
> script to test multiple commercial DRM systems and/or types (or even modify
> it to run the tests in their own infrastructure without necessarily
> generating the HTML files.)
>
>
>
> Please review and merge the PR above before migrating the existing tests.
>
>
>
> ​Ok, done.
>
>
>
> Sukhmal is working on a configurable test. Likely it will accept a
> "config" object and then it would indeed be a good idea for it to fill in
> any missing fields with default values. The configurable things to begin
> with will be the DRM type and the media files / types.
>
>
>
> It should then be possible to auto-generate the HTML files, but perhaps
> we'll create a few by hand to begin with and see how we go.
>
>
>
> ...Mark
>
>
>
> ​
>
>
>
>
>
>
> On Thursday, July 21, 2016, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
> OK — Given the limitations of the test framework, Mark’s approach seems
> acceptable to me.
>
>
>
> On 7/21/16, 8:08 AM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
> Hi Greg,
>
>
>
> You cannot pass arguments to the tests, or configure the test runner to
> run multiple times with different arguments.
>
>
>
> You can run multiple tests from one HTML file (WebCrypto has files with
> tens of thousands of tests), which is what I originally proposed on June
> 21st. But there were comments saying we should have one test per HTML file.
> Additionally, they tend to time out, so for our tests involving playback
> you cannot do too many. At this point we should pick an approach. We only
> have a week left.
>
>
>
> I was not proposing duplicating all the test code in every HTML file. I
> was proposing a JS file which could run any of four versions of the test
> (drm|clearkey)x(webm|mp4) and then four HTML files which each basically set
> the configuration and call the JS. So, the actual test code would be common
> between DRM and ClearKey as you suggest.
>
>
>
> What is missing in my proposal is the possibility to test multiple DRMs on
> one browser. But we have no browsers that support multiple DRMs, so I
> suggest we leave that for another day.
>
>
>
> Could I get comments on the Pull Request asap, please. I'd like to devote
> some time today to creating more tests following that pattern.
>
>
>
> ...Mark
>
>
>
>
>
>
>
> On Thu, Jul 21, 2016 at 4:00 AM, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
> (apologies for my late response — I’m in Europe this week)
>
>
>
> I am unfortunately not familiar with the W3C test harness.  Is it at all
> possible to pass “arguments” when you select a test to run?  It seems that
> by extending the JSON configuration that is currently used for the
> multi-DRM (drmconfig.json), you could also pass the media mime types for
> particular test configuration.  So, instead of having separate HTML test
> files for each media type, it could simply be passed in as part of the test
> configuration.
>
>
>
> Also, do we really need separate files for ClearKey?  I understand that
> not all tests would be valid for a ClearKey configuration, but isn’t
> ClearKey just another key system in the eyes of the EME spec?  Sure, the
> specs provides some normative language to describe what key messages look
> like, but other than that, you still create key sessions, retrieve a
> license (in some fashion), and pass that license to update().
>
>
>
> I know we are trying to get this done soon and this might be proposing too
> much of a complex architecture into the tests, but EME seems like a pretty
> new paradigm within the W3C that has so many optional features that it
> would make sense to minimize the amount of “cut-and-paste” test code just
> to support additional key systems and media types.
>
>
>
> G
>
>
>
> On 7/20/16, 7:06 PM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
> All,
>
>
>
> I have some time tomorrow to work on this and would like us to start
> making progress on the drm tests, so that we can have a substantial number
> ready this week. Our deadline is, after all, basically the end of next week.
>
>
>
> Has anyone had a chance to review the Pull Request I sent this-morning ?
> Is that a good template ? I would prefer not to invest time migrating lots
> of tests to that pattern only to have people ask for significant changes to
> be applied to many files.
>
>
>
> Can we agree to the model of four HTML files for each test (clearkey-mp4,
> clearkey-webm, drm-mp4, drm-webm) calling a common JS test file ?
>
>
>
> Finally, one possibility for also getting results for tests using
> polyfills would be to create a script which can take all the tests and add
> polyfill <script> elements to create new scripts in a subdirectory. You
> would then have a complete copy of all tests, with an easy way to
> regenerate (the polyfilled versions may or may not be checked in).
>
>
>
> ...Mark
>
>
>
> On Wed, Jul 20, 2016 at 4:58 PM, Mark Watson <watsonm@netflix.com> wrote:
>
>
>
>
>
> On Wed, Jul 20, 2016 at 4:42 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
> Would these actually be specific DRMs?
>
>
>
> drm-mp4-temporary-cenc.html
>
> drm-webm-temporary-cenc.html
>
>
>
> i.e., separate files for each drm supported in test.  That would group
> Widevine and PlayReady files together, so they would likely execute as in
> sequence (and as a group).
>
>
>
> Or does “drm” stand for “multi-drm”?
>
>
>
> ​It just means using a DRM rather than using ClearKey. Which DRM to use
> would depend on the browser (I'm assuming each browser only supports one
> and the test auto-detects which one to use).
>
>
>
> ...Mark​
>
>
>
>
>
>
>
> *From:* Mark Watson [mailto:watsonm@netflix.com <watsonm@netflix.com>]
> *Sent:* Wednesday, July 20, 2016 4:18 PM
> *To:* Jerry Smith (WPT) <jdsmith@microsoft.com>
> *Cc:* David Dorwin <ddorwin@google.com>; Greg Rutz <G.Rutz@cablelabs.com>;
> Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <
> wolenetz@google.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>;
> Francois Daoust <fd@w3.org>; public-hme-editors@w3.org; Iraj Sodagar <
> irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Paul Cotton <
> Paul.Cotton@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
>
>
> *Subject:* Re: DRM Today-based test case for EME
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 3:37 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
> wrote:
>
> A RegExpr can tell the runner to repeat each found test (under some path)
> to re-run for a list of keySystems?  That sounds pretty good.
>
>
>
> ​No, it can just select a subset of the html files to run.​
>
>
>
>
>
> Does this work better if scripts are in a sub-folder?  If so, then maybe
> these folders under encrypted-media make sense:
>
>
>
> -          clearkey
>
> -          multidrm
>
> -          mp4
>
> -          webm
>
> -          util
>
>
>
> ​Well, there are permutations and combinations:
>
> - any clearkey test that involves media could be run with either mp4 or
> webm, but it is not clear that it is necessary to do so.
>
> - the drm tests on some browsers will only work with mp4/cenc
>
>
>
> ​Here's a suggestion for a naming convention:
>
>
>
> (drm|clearkey)-(mp4|webm)-xxxx.html
>
>
>
> We could then have a file, generic-xxxx.js, which could contain most of
> the test code which could be called from the (at most) 4 html files names
> as above.
>
>
>
> We could convert the proposed drmtoday-temporary-cenc.html into
> generic-temporary-cenc.js and
>
>
>
> drm-mp4-temporary-cenc.html
>
> drm-webm-temporary-cenc.html
>
> clearkey-mp4-temporary-cenc.html
>
> clearkey-webm-temporary-cenc.html
>
>
>
> WDYAT ?
>
>
>
> ...Mark
>
>
>
>
>
>
>
> Jerry
>
>
>
> *From:* Mark Watson [mailto:watsonm@netflix.com <watsonm@netflix.com>]
> *Sent:* Wednesday, July 20, 2016 3:29 PM
> *To:* David Dorwin <ddorwin@google.com>
> *Cc:* Greg Rutz <G.Rutz@cablelabs.com>; Matthew Wolenetz <
> wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Jerry
> Smith (WPT) <jdsmith@microsoft.com>; Philippe Le Hegaret (plh@w3.org) <
> plh@w3.org>; Francois Daoust <fd@w3.org>; public-hme-editors@w3.org; Iraj
> Sodagar <irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Paul
> Cotton <Paul.Cotton@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
> *Subject:* Re: DRM Today-based test case for EME
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 3:11 PM, David Dorwin <ddorwin@google.com> wrote:
>
> The abstraction Greg describes makes sense, at least to my rough
> understanding. Greg, would we vary the test configurations or are all
> configurations always present and just a way of isolating the logic for
> each key system?
>
>
>
> In case there is any uncertainty, I want to emphasize that most of the
> "Google clearkey tests" are really just EME API tests that happen to use
> Clear Key. (The reason they use Clear Key (and WebM) has is related to the
> fact that they are Blink layout tests that run inside a subset of the code,
> pass in Chromium, and not depend on external servers.) Most interact with
> at least a portion of the Clear Key CDM implementation, meaning the
> behavior and results depend in part on the Clear Key implementation. This
> is similar to how most media tests are also testing a specific
> pipeline/decoder. There are some tests that explicitly test Clear Key
> behavior defined in https://w3c.github.io/encrypted-media/#clear-key, and
> we should ensure these are labeled "clearkey" in the path. Everything else
> should probably be converted to general tests.
>
>
>
> ​Ok, so IIUC, the process we should follow for each test currently in the
> Google directory (and any others we want to add) is:
>
> (i) migrate this test to the framework / utilities we have just proposed,
> including the drmtoday infractructure, to create a test using a real DRM
>
> (ii) make a copy of that test that just uses the Clear Key options in that
> same framework / utilities
>
>
>
> (It may not make sense to do both for every test)
>
>
>
> After we have migrated all the tests, we can remove the Google directory.
>
>
>
> We would then have mp4 versions of all the tests and we may want to
> (re)create some WebM ones. I don't expect we need to do every test with
> both WebM and mp4.
>
>
>
> The only way I can see to selectively run tests is to specify a path or
> RegExp in the test runner, so ​we should agree on a naming convention
> and/or folder heirarchy to organize the tests.
>
>
>
>
>
> Mark, my concern is that using Clear Key, which is almost certainly
> simpler than any other system, could paper over API design, etc. issues for
> other systems. In practice, I don't think this should be an issue since
> Edge doesn't implement Clear Key. (Thus, I also think we should err on the
> side of excluding Clear Key for now.)
>
>
>
> ​It's a valid concern, but so is the problem that we have a hard deadline,
> so I think we should err on the side of gathering as much evidence as we
> can and providing it with appropriate caveats.
>
>
>
> ...Mark​
>
>
>
>
>
>
>
> For full coverage, all supported combinations would be executed (something
> I discussed
> <https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0100.html>
> earlier
> <https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0104.html>). It
> would be nice if we could get results for the general tests run on each key
> system (and type), but we'd need to create some infrastructure.
>
>
>
> David
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 1:17 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> Greg - this makes sense and it would be easy to take the drmtoday test we
> have written and make a new clearkey version of that by enhancing the utils
> and the config as you describe.
>
>
>
> However, we already have a clearkey version of that test in the Google
> directory (which uses its own utils). So, doing what you say would increase
> the commonality / consistency between the tests, but it wouldn't get us
> more tests.
>
>
>
> David - the clearkey results are useful information for the implementation
> report. Again, as with tests based on polyfills, they validate the API
> design, implementability and specification. These are factors in the
> decision as well as the current state of commercially useful features in
> commercial browsers. We are in the unusual situation of not being able to
> just wait until implementations have matured, so this is going to be an
> unusual decision.
>
>
>
> ...Mark
>
>
>
> On Wed, Jul 20, 2016 at 1:05 PM, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
> For (B), I wasn’t suggesting that there be two different tests in one
> file, I was suggesting that we put operations like license requests into
> utils files that would perform either DRMToday or ClearKey license
> requests.  For DRMToday, the implementation in these utils files would make
> the request to the actual DRMToday license server.  For ClearKey, the
> implementation would likely return a response message that is placed into
> the test configuration JSON (drmconfig.json in the example test created by
> Sukhmal).  The JSON config file can help configure both the key system and
> the desired license response message that we need in order to properly
> execute the test.
>
>
>
> G
>
>
>
> On 7/20/16, 1:30 PM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
> So, what we have right now is:
>
> (1) A large number of ClearKey-only tests in a "Google" folder, and
>
> (2) One of those tests (basic playback) migrated to DRM Today, in the root
> folder
>
>
>
> There are two approaches:
>
> (A) Keep ClearKey and DRM tests separate: move the "Google" tests into the
> root or a "clearkey" folder, continue making new DRMToday versions of each
> of those ClearKey tests
>
> (B) Make the DRMToday test also support ClearKey, continue making new
> ClearKey+DRMToday versions of each of the Google tests and, eventually,
> drop the Google folder
>
>
>
> For (B), we need to run two tests in one file, which requires some care
> with async tests and there's been comments that we should not have multiple
> tests in one file.
>
>
>
> Opinions ?
>
>
>
> ...Mark
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 11:27 AM, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
> I think the test utilities should be designed to be as DRM-independent as
> possible.  This would allow us to run any of the test cases that apply to
> ClearKey simply by providing a DRMConfig and test content that indicates
> use of ClearKey.  I apologize that I have not been following the EME spec
> progression that much over the last 12-18 months, but I recall there not
> being a ton of differences between ClearKey support and other DRMs as I
> implemented it in dash.js.
>
>
>
> For test cases that are valid for ClearKey, the test case would simply
> execute multiple times on the UA under test — once with ClearKey content
> and one or more additional times for the “real” DRMs that are to be tested
> on that UA.  No sense in maintaining separate test code if we don’t have to.
>
>
>
> G
>
>
>
> On 7/20/16, 10:34 AM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
> Question: should we expand this test case to cover ClearKey ? Or will we
> rely on the tests in the Google folder for ClearKey ?
>
>
>
> If the latter, should we move those tests into the main directory (I see
> they are now working) ? Or, if others would like to add ClearKey tests,
> should they add them to the Google folder ?
>
>
>
> ...Mark
>
>
>
> On Tue, Jul 19, 2016 at 7:18 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> All,
>
>
>
> Sukhmal has created a Pull Request for a temporary session test case using
> DRM Today. We have tested this on Chrome with Widevine and it should work
> on Edge with PlayReady as well:
>
>
>
> https://github.com/w3c/web-platform-tests/pull/3313
>
>
>
> Please review this and comment on whether it is a good template / model
> for us to work from. We can quickly migrate more of the Google clearkey
> tests to drmtoday as well as implementing tests for other session types
> based on this model.
>
>
>
> ...Mark
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>