Re: DRM Today-based test case for EME

On Thu, Jul 21, 2016 at 11:49 AM, David Dorwin <ddorwin@google.com> wrote:

> I assumed the mp4 and webm directories were just the content, which is
> currently the case. Other than some targeted tests, such as testing
> specific initDataTypes, "encrypted" event generation for various formats,
> or testing playback of specific types, most tests should be
> media-independent. See how
> https://github.com/w3c/web-platform-tests/pull/3317 finds any supported
> type and uses it. (The tests that use media files need some additional
> work.)
>
> Thus, I think (drm|clearkey)-xxxx.html should be sufficient. It would be
> nice if we didn't need to maintain wrappers, but this will work for now.
> Writing the tests in .js files also makes it easier to add more tests later
> if we or implementers wish. We should design the JS files with such
> extensibility in mind. For example:
>
> function runTest(keySystem = null, mediaType/Config = null) {
>
> if (!keySystem) selectSupportedNonClearKeyKeySystem();
>
> if (!mediaType) getSupportedConfigAndMediaFiles();
>
> // Do test.
>
> }
>
>
> While not required now, it would be nice if we could automatically
> generate the .html files with a script. For example, for each file in the
> test-scripts/ directory, generate an HTML file that calls it for each of
> "drm" and "clearkey. Again, implementers and others could update this
> script to test multiple commercial DRM systems and/or types (or even modify
> it to run the tests in their own infrastructure without necessarily
> generating the HTML files.)
>
> Please review and merge the PR above before migrating the existing tests.
>

​Ok, done.

Sukhmal is working on a configurable test. Likely it will accept a "config"
object and then it would indeed be a good idea for it to fill in any
missing fields with default values. The configurable things to begin with
will be the DRM type and the media files / types.

It should then be possible to auto-generate the HTML files, but perhaps
we'll create a few by hand to begin with and see how we go.

...Mark

​



>
> On Thursday, July 21, 2016, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
>> OK — Given the limitations of the test framework, Mark’s approach seems
>> acceptable to me.
>>
>> On 7/21/16, 8:08 AM, "Mark Watson" <watsonm@netflix.com> wrote:
>>
>> Hi Greg,
>>
>> You cannot pass arguments to the tests, or configure the test runner to
>> run multiple times with different arguments.
>>
>> You can run multiple tests from one HTML file (WebCrypto has files with
>> tens of thousands of tests), which is what I originally proposed on June
>> 21st. But there were comments saying we should have one test per HTML file.
>> Additionally, they tend to time out, so for our tests involving playback
>> you cannot do too many. At this point we should pick an approach. We only
>> have a week left.
>>
>> I was not proposing duplicating all the test code in every HTML file. I
>> was proposing a JS file which could run any of four versions of the test
>> (drm|clearkey)x(webm|mp4) and then four HTML files which each basically set
>> the configuration and call the JS. So, the actual test code would be common
>> between DRM and ClearKey as you suggest.
>>
>> What is missing in my proposal is the possibility to test multiple DRMs
>> on one browser. But we have no browsers that support multiple DRMs, so I
>> suggest we leave that for another day.
>>
>> Could I get comments on the Pull Request asap, please. I'd like to devote
>> some time today to creating more tests following that pattern.
>>
>> ...Mark
>>
>>
>>
>> On Thu, Jul 21, 2016 at 4:00 AM, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>>
>>> (apologies for my late response — I’m in Europe this week)
>>>
>>> I am unfortunately not familiar with the W3C test harness.  Is it at all
>>> possible to pass “arguments” when you select a test to run?  It seems that
>>> by extending the JSON configuration that is currently used for the
>>> multi-DRM (drmconfig.json), you could also pass the media mime types for
>>> particular test configuration.  So, instead of having separate HTML test
>>> files for each media type, it could simply be passed in as part of the test
>>> configuration.
>>>
>>> Also, do we really need separate files for ClearKey?  I understand that
>>> not all tests would be valid for a ClearKey configuration, but isn’t
>>> ClearKey just another key system in the eyes of the EME spec?  Sure, the
>>> specs provides some normative language to describe what key messages look
>>> like, but other than that, you still create key sessions, retrieve a
>>> license (in some fashion), and pass that license to update().
>>>
>>> I know we are trying to get this done soon and this might be proposing
>>> too much of a complex architecture into the tests, but EME seems like a
>>> pretty new paradigm within the W3C that has so many optional features that
>>> it would make sense to minimize the amount of “cut-and-paste” test code
>>> just to support additional key systems and media types.
>>>
>>> G
>>>
>>> On 7/20/16, 7:06 PM, "Mark Watson" <watsonm@netflix.com> wrote:
>>>
>>> All,
>>>
>>> I have some time tomorrow to work on this and would like us to start
>>> making progress on the drm tests, so that we can have a substantial number
>>> ready this week. Our deadline is, after all, basically the end of next week.
>>>
>>> Has anyone had a chance to review the Pull Request I sent this-morning ?
>>> Is that a good template ? I would prefer not to invest time migrating lots
>>> of tests to that pattern only to have people ask for significant changes to
>>> be applied to many files.
>>>
>>> Can we agree to the model of four HTML files for each test
>>> (clearkey-mp4, clearkey-webm, drm-mp4, drm-webm) calling a common JS test
>>> file ?
>>>
>>> Finally, one possibility for also getting results for tests using
>>> polyfills would be to create a script which can take all the tests and add
>>> polyfill <script> elements to create new scripts in a subdirectory. You
>>> would then have a complete copy of all tests, with an easy way to
>>> regenerate (the polyfilled versions may or may not be checked in).
>>>
>>> ...Mark
>>>
>>> On Wed, Jul 20, 2016 at 4:58 PM, Mark Watson <watsonm@netflix.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Jul 20, 2016 at 4:42 PM, Jerry Smith (WPT) <
>>>> jdsmith@microsoft.com> wrote:
>>>>
>>>>> Would these actually be specific DRMs?
>>>>>
>>>>>
>>>>>
>>>>> drm-mp4-temporary-cenc.html
>>>>>
>>>>> drm-webm-temporary-cenc.html
>>>>>
>>>>>
>>>>>
>>>>> i.e., separate files for each drm supported in test.  That would group
>>>>> Widevine and PlayReady files together, so they would likely execute as in
>>>>> sequence (and as a group).
>>>>>
>>>>>
>>>>>
>>>>> Or does “drm” stand for “multi-drm”?
>>>>>
>>>>
>>>> ​It just means using a DRM rather than using ClearKey. Which DRM to use
>>>> would depend on the browser (I'm assuming each browser only supports one
>>>> and the test auto-detects which one to use).
>>>>
>>>> ...Mark​
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>> *From:* Mark Watson [mailto:watsonm@netflix.com]
>>>>> *Sent:* Wednesday, July 20, 2016 4:18 PM
>>>>> *To:* Jerry Smith (WPT) <jdsmith@microsoft.com>
>>>>> *Cc:* David Dorwin <ddorwin@google.com>; Greg Rutz <
>>>>> G.Rutz@cablelabs.com>; Matthew Wolenetz <wolenetz@google.com> (
>>>>> wolenetz@google.com) <wolenetz@google.com>; Philippe Le Hegaret (
>>>>> plh@w3.org) <plh@w3.org>; Francois Daoust <fd@w3.org>;
>>>>> public-hme-editors@w3.org; Iraj Sodagar <irajs@microsoft.com>; John
>>>>> Simmons <johnsim@microsoft.com>; Paul Cotton <
>>>>> Paul.Cotton@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
>>>>>
>>>>> *Subject:* Re: DRM Today-based test case for EME
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 20, 2016 at 3:37 PM, Jerry Smith (WPT) <
>>>>> jdsmith@microsoft.com> wrote:
>>>>>
>>>>> A RegExpr can tell the runner to repeat each found test (under some
>>>>> path) to re-run for a list of keySystems?  That sounds pretty good.
>>>>>
>>>>>
>>>>>
>>>>> ​No, it can just select a subset of the html files to run.​
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Does this work better if scripts are in a sub-folder?  If so, then
>>>>> maybe these folders under encrypted-media make sense:
>>>>>
>>>>>
>>>>>
>>>>> -          clearkey
>>>>>
>>>>> -          multidrm
>>>>>
>>>>> -          mp4
>>>>>
>>>>> -          webm
>>>>>
>>>>> -          util
>>>>>
>>>>>
>>>>>
>>>>> ​Well, there are permutations and combinations:
>>>>>
>>>>> - any clearkey test that involves media could be run with either mp4
>>>>> or webm, but it is not clear that it is necessary to do so.
>>>>>
>>>>> - the drm tests on some browsers will only work with mp4/cenc
>>>>>
>>>>>
>>>>>
>>>>> ​Here's a suggestion for a naming convention:
>>>>>
>>>>>
>>>>>
>>>>> (drm|clearkey)-(mp4|webm)-xxxx.html
>>>>>
>>>>>
>>>>>
>>>>> We could then have a file, generic-xxxx.js, which could contain most
>>>>> of the test code which could be called from the (at most) 4 html files
>>>>> names as above.
>>>>>
>>>>>
>>>>>
>>>>> We could convert the proposed drmtoday-temporary-cenc.html into
>>>>> generic-temporary-cenc.js and
>>>>>
>>>>>
>>>>>
>>>>> drm-mp4-temporary-cenc.html
>>>>>
>>>>> drm-webm-temporary-cenc.html
>>>>>
>>>>> clearkey-mp4-temporary-cenc.html
>>>>>
>>>>> clearkey-webm-temporary-cenc.html
>>>>>
>>>>>
>>>>>
>>>>> WDYAT ?
>>>>>
>>>>>
>>>>>
>>>>> ...Mark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Jerry
>>>>>
>>>>>
>>>>>
>>>>> *From:* Mark Watson [mailto:watsonm@netflix.com]
>>>>> *Sent:* Wednesday, July 20, 2016 3:29 PM
>>>>> *To:* David Dorwin <ddorwin@google.com>
>>>>> *Cc:* Greg Rutz <G.Rutz@cablelabs.com>; Matthew Wolenetz <
>>>>> wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>;
>>>>> Jerry Smith (WPT) <jdsmith@microsoft.com>; Philippe Le Hegaret (
>>>>> plh@w3.org) <plh@w3.org>; Francois Daoust <fd@w3.org>;
>>>>> public-hme-editors@w3.org; Iraj Sodagar <irajs@microsoft.com>; John
>>>>> Simmons <johnsim@microsoft.com>; Paul Cotton <
>>>>> Paul.Cotton@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
>>>>> *Subject:* Re: DRM Today-based test case for EME
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 20, 2016 at 3:11 PM, David Dorwin <ddorwin@google.com>
>>>>> wrote:
>>>>>
>>>>> The abstraction Greg describes makes sense, at least to my rough
>>>>> understanding. Greg, would we vary the test configurations or are all
>>>>> configurations always present and just a way of isolating the logic for
>>>>> each key system?
>>>>>
>>>>>
>>>>>
>>>>> In case there is any uncertainty, I want to emphasize that most of the
>>>>> "Google clearkey tests" are really just EME API tests that happen to use
>>>>> Clear Key. (The reason they use Clear Key (and WebM) has is related to the
>>>>> fact that they are Blink layout tests that run inside a subset of the code,
>>>>> pass in Chromium, and not depend on external servers.) Most interact with
>>>>> at least a portion of the Clear Key CDM implementation, meaning the
>>>>> behavior and results depend in part on the Clear Key implementation. This
>>>>> is similar to how most media tests are also testing a specific
>>>>> pipeline/decoder. There are some tests that explicitly test Clear Key
>>>>> behavior defined in https://w3c.github.io/encrypted-media/#clear-key,
>>>>> and we should ensure these are labeled "clearkey" in the path. Everything
>>>>> else should probably be converted to general tests.
>>>>>
>>>>>
>>>>>
>>>>> ​Ok, so IIUC, the process we should follow for each test currently in
>>>>> the Google directory (and any others we want to add) is:
>>>>>
>>>>> (i) migrate this test to the framework / utilities we have just
>>>>> proposed, including the drmtoday infractructure, to create a test using a
>>>>> real DRM
>>>>>
>>>>> (ii) make a copy of that test that just uses the Clear Key options in
>>>>> that same framework / utilities
>>>>>
>>>>>
>>>>>
>>>>> (It may not make sense to do both for every test)
>>>>>
>>>>>
>>>>>
>>>>> After we have migrated all the tests, we can remove the Google
>>>>> directory.
>>>>>
>>>>>
>>>>>
>>>>> We would then have mp4 versions of all the tests and we may want to
>>>>> (re)create some WebM ones. I don't expect we need to do every test with
>>>>> both WebM and mp4.
>>>>>
>>>>>
>>>>>
>>>>> The only way I can see to selectively run tests is to specify a path
>>>>> or RegExp in the test runner, so ​we should agree on a naming convention
>>>>> and/or folder heirarchy to organize the tests.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Mark, my concern is that using Clear Key, which is almost certainly
>>>>> simpler than any other system, could paper over API design, etc. issues for
>>>>> other systems. In practice, I don't think this should be an issue since
>>>>> Edge doesn't implement Clear Key. (Thus, I also think we should err on the
>>>>> side of excluding Clear Key for now.)
>>>>>
>>>>>
>>>>>
>>>>> ​It's a valid concern, but so is the problem that we have a hard
>>>>> deadline, so I think we should err on the side of gathering as much
>>>>> evidence as we can and providing it with appropriate caveats.
>>>>>
>>>>>
>>>>>
>>>>> ...Mark​
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> For full coverage, all supported combinations would be executed
>>>>> (something I discussed
>>>>> <https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0100.html>
>>>>> earlier
>>>>> <https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0104.html>). It
>>>>> would be nice if we could get results for the general tests run on each key
>>>>> system (and type), but we'd need to create some infrastructure.
>>>>>
>>>>>
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 20, 2016 at 1:17 PM, Mark Watson <watsonm@netflix.com>
>>>>> wrote:
>>>>>
>>>>> Greg - this makes sense and it would be easy to take the drmtoday test
>>>>> we have written and make a new clearkey version of that by enhancing the
>>>>> utils and the config as you describe.
>>>>>
>>>>>
>>>>>
>>>>> However, we already have a clearkey version of that test in the Google
>>>>> directory (which uses its own utils). So, doing what you say would increase
>>>>> the commonality / consistency between the tests, but it wouldn't get us
>>>>> more tests.
>>>>>
>>>>>
>>>>>
>>>>> David - the clearkey results are useful information for the
>>>>> implementation report. Again, as with tests based on polyfills, they
>>>>> validate the API design, implementability and specification. These are
>>>>> factors in the decision as well as the current state of commercially useful
>>>>> features in commercial browsers. We are in the unusual situation of not
>>>>> being able to just wait until implementations have matured, so this is
>>>>> going to be an unusual decision.
>>>>>
>>>>>
>>>>>
>>>>> ...Mark
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 20, 2016 at 1:05 PM, Greg Rutz <G.Rutz@cablelabs.com>
>>>>> wrote:
>>>>>
>>>>> For (B), I wasn’t suggesting that there be two different tests in one
>>>>> file, I was suggesting that we put operations like license requests into
>>>>> utils files that would perform either DRMToday or ClearKey license
>>>>> requests.  For DRMToday, the implementation in these utils files would make
>>>>> the request to the actual DRMToday license server.  For ClearKey, the
>>>>> implementation would likely return a response message that is placed into
>>>>> the test configuration JSON (drmconfig.json in the example test created by
>>>>> Sukhmal).  The JSON config file can help configure both the key system and
>>>>> the desired license response message that we need in order to properly
>>>>> execute the test.
>>>>>
>>>>>
>>>>>
>>>>> G
>>>>>
>>>>>
>>>>>
>>>>> On 7/20/16, 1:30 PM, "Mark Watson" <watsonm@netflix.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> So, what we have right now is:
>>>>>
>>>>> (1) A large number of ClearKey-only tests in a "Google" folder, and
>>>>>
>>>>> (2) One of those tests (basic playback) migrated to DRM Today, in the
>>>>> root folder
>>>>>
>>>>>
>>>>>
>>>>> There are two approaches:
>>>>>
>>>>> (A) Keep ClearKey and DRM tests separate: move the "Google" tests into
>>>>> the root or a "clearkey" folder, continue making new DRMToday versions of
>>>>> each of those ClearKey tests
>>>>>
>>>>> (B) Make the DRMToday test also support ClearKey, continue making new
>>>>> ClearKey+DRMToday versions of each of the Google tests and, eventually,
>>>>> drop the Google folder
>>>>>
>>>>>
>>>>>
>>>>> For (B), we need to run two tests in one file, which requires some
>>>>> care with async tests and there's been comments that we should not have
>>>>> multiple tests in one file.
>>>>>
>>>>>
>>>>>
>>>>> Opinions ?
>>>>>
>>>>>
>>>>>
>>>>> ...Mark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 20, 2016 at 11:27 AM, Greg Rutz <G.Rutz@cablelabs.com>
>>>>> wrote:
>>>>>
>>>>> I think the test utilities should be designed to be as DRM-independent
>>>>> as possible.  This would allow us to run any of the test cases that apply
>>>>> to ClearKey simply by providing a DRMConfig and test content that indicates
>>>>> use of ClearKey.  I apologize that I have not been following the EME spec
>>>>> progression that much over the last 12-18 months, but I recall there not
>>>>> being a ton of differences between ClearKey support and other DRMs as I
>>>>> implemented it in dash.js.
>>>>>
>>>>>
>>>>>
>>>>> For test cases that are valid for ClearKey, the test case would simply
>>>>> execute multiple times on the UA under test — once with ClearKey content
>>>>> and one or more additional times for the “real” DRMs that are to be tested
>>>>> on that UA.  No sense in maintaining separate test code if we don’t have to.
>>>>>
>>>>>
>>>>>
>>>>> G
>>>>>
>>>>>
>>>>>
>>>>> On 7/20/16, 10:34 AM, "Mark Watson" <watsonm@netflix.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Question: should we expand this test case to cover ClearKey ? Or will
>>>>> we rely on the tests in the Google folder for ClearKey ?
>>>>>
>>>>>
>>>>>
>>>>> If the latter, should we move those tests into the main directory (I
>>>>> see they are now working) ? Or, if others would like to add ClearKey tests,
>>>>> should they add them to the Google folder ?
>>>>>
>>>>>
>>>>>
>>>>> ...Mark
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jul 19, 2016 at 7:18 PM, Mark Watson <watsonm@netflix.com>
>>>>> wrote:
>>>>>
>>>>> All,
>>>>>
>>>>>
>>>>>
>>>>> Sukhmal has created a Pull Request for a temporary session test case
>>>>> using DRM Today. We have tested this on Chrome with Widevine and it should
>>>>> work on Edge with PlayReady as well:
>>>>>
>>>>>
>>>>>
>>>>> https://github.com/w3c/web-platform-tests/pull/3313
>>>>>
>>>>>
>>>>>
>>>>> Please review this and comment on whether it is a good template /
>>>>> model for us to work from. We can quickly migrate more of the Google
>>>>> clearkey tests to drmtoday as well as implementing tests for other session
>>>>> types based on this model.
>>>>>
>>>>>
>>>>>
>>>>> ...Mark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>