RE: DRM Today-based test case for EME

Go for it Mark!!  Glad to see us making progress on a structure for EME tests.

/paulc

From: Mark Watson [mailto:watsonm@netflix.com]
Sent: Thursday, July 21, 2016 8:06 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com>
Cc: Greg Rutz <G.Rutz@cablelabs.com>; David Dorwin <ddorwin@google.com>; Matthew Wolenetz <wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Philippe Le Hegaret (plh@w3.org) <plh@w3.org>; Francois Daoust <fd@w3.org>; public-hme-editors@w3.org; Iraj Sodagar <irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Paul Cotton <Paul.Cotton@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
Subject: Re: DRM Today-based test case for EME

Ok, so I have updated the Pull Request per our discussion.

https://github.com/w3c/web-platform-tests/pull/3313


I would like to get moving with porting over additional tests, so if there are no objections, and I can get the CI checks to pass, I will merge this shortly.

Note that we presently have different content for the clearkey and drm tests. I am just waiting on finding out the key from Greg and then I'll drop the Chimera content.

...Mark

On Thu, Jul 21, 2016 at 5:01 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:


On Thu, Jul 21, 2016 at 4:46 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
I don’t understand the role of the HTML files.  My understanding was that the harness would run any js files in the directory, if they were properly annotated as tests.  It sounds like this can be done in an HTML file that lists out the underlying tests files?

The harness runs all HTML files it finds in the specified directory / pattern. (It might also do what you say, but I'm not aware of that).

Each HTML file could contain multiple tests, but we're following the approach where each file contains only one test, as discussed.



Please retain a recognizable name snippet from the current test files in new “drm” copies to assist in updating coverage.

From: Mark Watson [mailto:watsonm@netflix.com<mailto:watsonm@netflix.com>]
Sent: Thursday, July 21, 2016 12:53 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>
Cc: Greg Rutz <G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>>; David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>>; Matthew Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> (wolenetz@google.com<mailto:wolenetz@google.com>) <wolenetz@google.com<mailto:wolenetz@google.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; Francois Daoust <fd@w3.org<mailto:fd@w3.org>>; public-hme-editors@w3.org<mailto:public-hme-editors@w3.org>; Iraj Sodagar <irajs@microsoft.com<mailto:irajs@microsoft.com>>; John Simmons <johnsim@microsoft.com<mailto:johnsim@microsoft.com>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Sukhmal Kommidi <skommidi@netflix.com<mailto:skommidi@netflix.com>>

Subject: Re: DRM Today-based test case for EME



On Thu, Jul 21, 2016 at 12:45 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
Mark:  At one point, you mentioned that it should be sufficient for “drm” tests to just test the single supported CDM.  This may be okay for our V1 test suite, but I’m not sure it’s an assumption that will hold up over time.  If a browser supports two CDMs, then we need a way for it to control testing of each.  We specifically will want to avoid the test logic going through a list of testable DRMs and testing the first one it finds as supported.

We may elect to use the single CDM approach for now, but it would be good to give some thought into what it would mean to support multiple.  The brute force, but not very scalable solution, would be to clone the tests per tested DRM.

​I think we have settled on the main test code being JS files which each run one test for a provided (keysystem, media)​ pair.

Then we will hand construct, and eventually auto-generate, HTML files for the combinations we want.

There are various ways the auto-generated HTML files could run tests for multiple DRMs on a browser that supported multiple, but I think we can work that out later: the main test logic will not need to change.

...Mark



Jerry

From: Greg Rutz [mailto:G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>]
Sent: Thursday, July 21, 2016 12:36 PM
To: Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>>; David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>>
Cc: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; Matthew Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> (wolenetz@google.com<mailto:wolenetz@google.com>) <wolenetz@google.com<mailto:wolenetz@google.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; Francois Daoust <fd@w3.org<mailto:fd@w3.org>>; public-hme-editors@w3.org<mailto:public-hme-editors@w3.org>; Iraj Sodagar <irajs@microsoft.com<mailto:irajs@microsoft.com>>; John Simmons <johnsim@microsoft.com<mailto:johnsim@microsoft.com>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Sukhmal Kommidi <skommidi@netflix.com<mailto:skommidi@netflix.com>>

Subject: Re: DRM Today-based test case for EME

This is excellent.  I’m glad we came up with a way to minimize code duplication while still working within the W3C framework.

On 7/21/16, 1:13 PM, "Mark Watson" <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:



On Thu, Jul 21, 2016 at 11:49 AM, David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>> wrote:
I assumed the mp4 and webm directories were just the content, which is currently the case. Other than some targeted tests, such as testing specific initDataTypes, "encrypted" event generation for various formats, or testing playback of specific types, most tests should be media-independent. See how https://github.com/w3c/web-platform-tests/pull/3317 finds any supported type and uses it. (The tests that use media files need some additional work.)

Thus, I think (drm|clearkey)-xxxx.html should be sufficient. It would be nice if we didn't need to maintain wrappers, but this will work for now. Writing the tests in .js files also makes it easier to add more tests later if we or implementers wish. We should design the JS files with such extensibility in mind. For example:
function runTest(keySystem = null, mediaType/Config = null) {
if (!keySystem) selectSupportedNonClearKeyKeySystem();
if (!mediaType) getSupportedConfigAndMediaFiles();
// Do test.
}

While not required now, it would be nice if we could automatically generate the .html files with a script. For example, for each file in the test-scripts/ directory, generate an HTML file that calls it for each of "drm" and "clearkey. Again, implementers and others could update this script to test multiple commercial DRM systems and/or types (or even modify it to run the tests in their own infrastructure without necessarily generating the HTML files.)

Please review and merge the PR above before migrating the existing tests.

​Ok, done.

Sukhmal is working on a configurable test. Likely it will accept a "config" object and then it would indeed be a good idea for it to fill in any missing fields with default values. The configurable things to begin with will be the DRM type and the media files / types.

It should then be possible to auto-generate the HTML files, but perhaps we'll create a few by hand to begin with and see how we go.

...Mark

​



On Thursday, July 21, 2016, Greg Rutz <G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>> wrote:
OK — Given the limitations of the test framework, Mark’s approach seems acceptable to me.

On 7/21/16, 8:08 AM, "Mark Watson" <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:

Hi Greg,

You cannot pass arguments to the tests, or configure the test runner to run multiple times with different arguments.

You can run multiple tests from one HTML file (WebCrypto has files with tens of thousands of tests), which is what I originally proposed on June 21st. But there were comments saying we should have one test per HTML file. Additionally, they tend to time out, so for our tests involving playback you cannot do too many. At this point we should pick an approach. We only have a week left.

I was not proposing duplicating all the test code in every HTML file. I was proposing a JS file which could run any of four versions of the test (drm|clearkey)x(webm|mp4) and then four HTML files which each basically set the configuration and call the JS. So, the actual test code would be common between DRM and ClearKey as you suggest.

What is missing in my proposal is the possibility to test multiple DRMs on one browser. But we have no browsers that support multiple DRMs, so I suggest we leave that for another day.

Could I get comments on the Pull Request asap, please. I'd like to devote some time today to creating more tests following that pattern.

...Mark



On Thu, Jul 21, 2016 at 4:00 AM, Greg Rutz <G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>> wrote:
(apologies for my late response — I’m in Europe this week)

I am unfortunately not familiar with the W3C test harness.  Is it at all possible to pass “arguments” when you select a test to run?  It seems that by extending the JSON configuration that is currently used for the multi-DRM (drmconfig.json), you could also pass the media mime types for particular test configuration.  So, instead of having separate HTML test files for each media type, it could simply be passed in as part of the test configuration.

Also, do we really need separate files for ClearKey?  I understand that not all tests would be valid for a ClearKey configuration, but isn’t ClearKey just another key system in the eyes of the EME spec?  Sure, the specs provides some normative language to describe what key messages look like, but other than that, you still create key sessions, retrieve a license (in some fashion), and pass that license to update().

I know we are trying to get this done soon and this might be proposing too much of a complex architecture into the tests, but EME seems like a pretty new paradigm within the W3C that has so many optional features that it would make sense to minimize the amount of “cut-and-paste” test code just to support additional key systems and media types.

G

On 7/20/16, 7:06 PM, "Mark Watson" <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:

All,

I have some time tomorrow to work on this and would like us to start making progress on the drm tests, so that we can have a substantial number ready this week. Our deadline is, after all, basically the end of next week.

Has anyone had a chance to review the Pull Request I sent this-morning ? Is that a good template ? I would prefer not to invest time migrating lots of tests to that pattern only to have people ask for significant changes to be applied to many files.

Can we agree to the model of four HTML files for each test (clearkey-mp4, clearkey-webm, drm-mp4, drm-webm) calling a common JS test file ?

Finally, one possibility for also getting results for tests using polyfills would be to create a script which can take all the tests and add polyfill <script> elements to create new scripts in a subdirectory. You would then have a complete copy of all tests, with an easy way to regenerate (the polyfilled versions may or may not be checked in).

...Mark

On Wed, Jul 20, 2016 at 4:58 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:


On Wed, Jul 20, 2016 at 4:42 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
Would these actually be specific DRMs?

drm-mp4-temporary-cenc.html
drm-webm-temporary-cenc.html

i.e., separate files for each drm supported in test.  That would group Widevine and PlayReady files together, so they would likely execute as in sequence (and as a group).

Or does “drm” stand for “multi-drm”?

​It just means using a DRM rather than using ClearKey. Which DRM to use would depend on the browser (I'm assuming each browser only supports one and the test auto-detects which one to use).

...Mark​



From: Mark Watson [mailto:watsonm@netflix.com]
Sent: Wednesday, July 20, 2016 4:18 PM
To: Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>
Cc: David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>>; Greg Rutz <G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>>; Matthew Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> (wolenetz@google.com<mailto:wolenetz@google.com>) <wolenetz@google.com<mailto:wolenetz@google.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; Francois Daoust <fd@w3.org<mailto:fd@w3.org>>; public-hme-editors@w3.org<mailto:public-hme-editors@w3.org>; Iraj Sodagar <irajs@microsoft.com<mailto:irajs@microsoft.com>>; John Simmons <johnsim@microsoft.com<mailto:johnsim@microsoft.com>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Sukhmal Kommidi <skommidi@netflix.com<mailto:skommidi@netflix.com>>

Subject: Re: DRM Today-based test case for EME



On Wed, Jul 20, 2016 at 3:37 PM, Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>> wrote:
A RegExpr can tell the runner to repeat each found test (under some path) to re-run for a list of keySystems?  That sounds pretty good.

​No, it can just select a subset of the html files to run.​


Does this work better if scripts are in a sub-folder?  If so, then maybe these folders under encrypted-media make sense:


-          clearkey

-          multidrm

-          mp4

-          webm

-          util

​Well, there are permutations and combinations:
- any clearkey test that involves media could be run with either mp4 or webm, but it is not clear that it is necessary to do so.
- the drm tests on some browsers will only work with mp4/cenc

​Here's a suggestion for a naming convention:

(drm|clearkey)-(mp4|webm)-xxxx.html

We could then have a file, generic-xxxx.js, which could contain most of the test code which could be called from the (at most) 4 html files names as above.

We could convert the proposed drmtoday-temporary-cenc.html into generic-temporary-cenc.js and

drm-mp4-temporary-cenc.html
drm-webm-temporary-cenc.html
clearkey-mp4-temporary-cenc.html
clearkey-webm-temporary-cenc.html

WDYAT ?

...Mark



Jerry

From: Mark Watson [mailto:watsonm@netflix.com]
Sent: Wednesday, July 20, 2016 3:29 PM
To: David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>>
Cc: Greg Rutz <G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>>; Matthew Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> (wolenetz@google.com<mailto:wolenetz@google.com>) <wolenetz@google.com<mailto:wolenetz@google.com>>; Jerry Smith (WPT) <jdsmith@microsoft.com<mailto:jdsmith@microsoft.com>>; Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org<mailto:plh@w3.org>>; Francois Daoust <fd@w3.org<mailto:fd@w3.org>>; public-hme-editors@w3.org<mailto:public-hme-editors@w3.org>; Iraj Sodagar <irajs@microsoft.com<mailto:irajs@microsoft.com>>; John Simmons <johnsim@microsoft.com<mailto:johnsim@microsoft.com>>; Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>; Sukhmal Kommidi <skommidi@netflix.com<mailto:skommidi@netflix.com>>
Subject: Re: DRM Today-based test case for EME



On Wed, Jul 20, 2016 at 3:11 PM, David Dorwin <ddorwin@google.com<mailto:ddorwin@google.com>> wrote:
The abstraction Greg describes makes sense, at least to my rough understanding. Greg, would we vary the test configurations or are all configurations always present and just a way of isolating the logic for each key system?

In case there is any uncertainty, I want to emphasize that most of the "Google clearkey tests" are really just EME API tests that happen to use Clear Key. (The reason they use Clear Key (and WebM) has is related to the fact that they are Blink layout tests that run inside a subset of the code, pass in Chromium, and not depend on external servers.) Most interact with at least a portion of the Clear Key CDM implementation, meaning the behavior and results depend in part on the Clear Key implementation. This is similar to how most media tests are also testing a specific pipeline/decoder. There are some tests that explicitly test Clear Key behavior defined in https://w3c.github.io/encrypted-media/#clear-key, and we should ensure these are labeled "clearkey" in the path. Everything else should probably be converted to general tests.

​Ok, so IIUC, the process we should follow for each test currently in the Google directory (and any others we want to add) is:
(i) migrate this test to the framework / utilities we have just proposed, including the drmtoday infractructure, to create a test using a real DRM
(ii) make a copy of that test that just uses the Clear Key options in that same framework / utilities

(It may not make sense to do both for every test)

After we have migrated all the tests, we can remove the Google directory.

We would then have mp4 versions of all the tests and we may want to (re)create some WebM ones. I don't expect we need to do every test with both WebM and mp4.

The only way I can see to selectively run tests is to specify a path or RegExp in the test runner, so ​we should agree on a naming convention and/or folder heirarchy to organize the tests.


Mark, my concern is that using Clear Key, which is almost certainly simpler than any other system, could paper over API design, etc. issues for other systems. In practice, I don't think this should be an issue since Edge doesn't implement Clear Key. (Thus, I also think we should err on the side of excluding Clear Key for now.)

​It's a valid concern, but so is the problem that we have a hard deadline, so I think we should err on the side of gathering as much evidence as we can and providing it with appropriate caveats.

...Mark​



For full coverage, all supported combinations would be executed (something I discussed<https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0100.html> earlier<https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0104.html>). It would be nice if we could get results for the general tests run on each key system (and type), but we'd need to create some infrastructure.

David



On Wed, Jul 20, 2016 at 1:17 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
Greg - this makes sense and it would be easy to take the drmtoday test we have written and make a new clearkey version of that by enhancing the utils and the config as you describe.

However, we already have a clearkey version of that test in the Google directory (which uses its own utils). So, doing what you say would increase the commonality / consistency between the tests, but it wouldn't get us more tests.

David - the clearkey results are useful information for the implementation report. Again, as with tests based on polyfills, they validate the API design, implementability and specification. These are factors in the decision as well as the current state of commercially useful features in commercial browsers. We are in the unusual situation of not being able to just wait until implementations have matured, so this is going to be an unusual decision.

...Mark

On Wed, Jul 20, 2016 at 1:05 PM, Greg Rutz <G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>> wrote:
For (B), I wasn’t suggesting that there be two different tests in one file, I was suggesting that we put operations like license requests into utils files that would perform either DRMToday or ClearKey license requests.  For DRMToday, the implementation in these utils files would make the request to the actual DRMToday license server.  For ClearKey, the implementation would likely return a response message that is placed into the test configuration JSON (drmconfig.json in the example test created by Sukhmal).  The JSON config file can help configure both the key system and the desired license response message that we need in order to properly execute the test.

G

On 7/20/16, 1:30 PM, "Mark Watson" <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:

So, what we have right now is:
(1) A large number of ClearKey-only tests in a "Google" folder, and
(2) One of those tests (basic playback) migrated to DRM Today, in the root folder

There are two approaches:
(A) Keep ClearKey and DRM tests separate: move the "Google" tests into the root or a "clearkey" folder, continue making new DRMToday versions of each of those ClearKey tests
(B) Make the DRMToday test also support ClearKey, continue making new ClearKey+DRMToday versions of each of the Google tests and, eventually, drop the Google folder

For (B), we need to run two tests in one file, which requires some care with async tests and there's been comments that we should not have multiple tests in one file.

Opinions ?

...Mark





On Wed, Jul 20, 2016 at 11:27 AM, Greg Rutz <G.Rutz@cablelabs.com<mailto:G.Rutz@cablelabs.com>> wrote:
I think the test utilities should be designed to be as DRM-independent as possible.  This would allow us to run any of the test cases that apply to ClearKey simply by providing a DRMConfig and test content that indicates use of ClearKey.  I apologize that I have not been following the EME spec progression that much over the last 12-18 months, but I recall there not being a ton of differences between ClearKey support and other DRMs as I implemented it in dash.js.

For test cases that are valid for ClearKey, the test case would simply execute multiple times on the UA under test — once with ClearKey content and one or more additional times for the “real” DRMs that are to be tested on that UA.  No sense in maintaining separate test code if we don’t have to.

G

On 7/20/16, 10:34 AM, "Mark Watson" <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:

Question: should we expand this test case to cover ClearKey ? Or will we rely on the tests in the Google folder for ClearKey ?

If the latter, should we move those tests into the main directory (I see they are now working) ? Or, if others would like to add ClearKey tests, should they add them to the Google folder ?

...Mark

On Tue, Jul 19, 2016 at 7:18 PM, Mark Watson <watsonm@netflix.com<mailto:watsonm@netflix.com>> wrote:
All,

Sukhmal has created a Pull Request for a temporary session test case using DRM Today. We have tested this on Chrome with Widevine and it should work on Edge with PlayReady as well:

https://github.com/w3c/web-platform-tests/pull/3313


Please review this and comment on whether it is a good template / model for us to work from. We can quickly migrate more of the Google clearkey tests to drmtoday as well as implementing tests for other session types based on this model.

...Mark