Re: DRM Today-based test case for EME

On Wed, Jul 20, 2016 at 3:37 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

> A RegExpr can tell the runner to repeat each found test (under some path)
> to re-run for a list of keySystems?  That sounds pretty good.
>

​No, it can just select a subset of the html files to run.​


>
>
> Does this work better if scripts are in a sub-folder?  If so, then maybe
> these folders under encrypted-media make sense:
>
>
>
> -          clearkey
>
> -          multidrm
>
> -          mp4
>
> -          webm
>
> -          util
>

​Well, there are permutations and combinations:
- any clearkey test that involves media could be run with either mp4 or
webm, but it is not clear that it is necessary to do so.
- the drm tests on some browsers will only work with mp4/cenc

​Here's a suggestion for a naming convention:

(drm|clearkey)-(mp4|webm)-xxxx.html

We could then have a file, generic-xxxx.js, which could contain most of the
test code which could be called from the (at most) 4 html files names as
above.

We could convert the proposed drmtoday-temporary-cenc.html into
generic-temporary-cenc.js and

drm-mp4-temporary-cenc.html
drm-webm-temporary-cenc.html
clearkey-mp4-temporary-cenc.html
clearkey-webm-temporary-cenc.html

WDYAT ?

...Mark



>
>
> Jerry
>
>
>
> *From:* Mark Watson [mailto:watsonm@netflix.com]
> *Sent:* Wednesday, July 20, 2016 3:29 PM
> *To:* David Dorwin <ddorwin@google.com>
> *Cc:* Greg Rutz <G.Rutz@cablelabs.com>; Matthew Wolenetz <
> wolenetz@google.com> (wolenetz@google.com) <wolenetz@google.com>; Jerry
> Smith (WPT) <jdsmith@microsoft.com>; Philippe Le Hegaret (plh@w3.org) <
> plh@w3.org>; Francois Daoust <fd@w3.org>; public-hme-editors@w3.org; Iraj
> Sodagar <irajs@microsoft.com>; John Simmons <johnsim@microsoft.com>; Paul
> Cotton <Paul.Cotton@microsoft.com>; Sukhmal Kommidi <skommidi@netflix.com>
> *Subject:* Re: DRM Today-based test case for EME
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 3:11 PM, David Dorwin <ddorwin@google.com> wrote:
>
> The abstraction Greg describes makes sense, at least to my rough
> understanding. Greg, would we vary the test configurations or are all
> configurations always present and just a way of isolating the logic for
> each key system?
>
>
>
> In case there is any uncertainty, I want to emphasize that most of the
> "Google clearkey tests" are really just EME API tests that happen to use
> Clear Key. (The reason they use Clear Key (and WebM) has is related to the
> fact that they are Blink layout tests that run inside a subset of the code,
> pass in Chromium, and not depend on external servers.) Most interact with
> at least a portion of the Clear Key CDM implementation, meaning the
> behavior and results depend in part on the Clear Key implementation. This
> is similar to how most media tests are also testing a specific
> pipeline/decoder. There are some tests that explicitly test Clear Key
> behavior defined in https://w3c.github.io/encrypted-media/#clear-key, and
> we should ensure these are labeled "clearkey" in the path. Everything else
> should probably be converted to general tests.
>
>
>
> ​Ok, so IIUC, the process we should follow for each test currently in the
> Google directory (and any others we want to add) is:
>
> (i) migrate this test to the framework / utilities we have just proposed,
> including the drmtoday infractructure, to create a test using a real DRM
>
> (ii) make a copy of that test that just uses the Clear Key options in that
> same framework / utilities
>
>
>
> (It may not make sense to do both for every test)
>
>
>
> After we have migrated all the tests, we can remove the Google directory.
>
>
>
> We would then have mp4 versions of all the tests and we may want to
> (re)create some WebM ones. I don't expect we need to do every test with
> both WebM and mp4.
>
>
>
> The only way I can see to selectively run tests is to specify a path or
> RegExp in the test runner, so ​we should agree on a naming convention
> and/or folder heirarchy to organize the tests.
>
>
>
>
>
> Mark, my concern is that using Clear Key, which is almost certainly
> simpler than any other system, could paper over API design, etc. issues for
> other systems. In practice, I don't think this should be an issue since
> Edge doesn't implement Clear Key. (Thus, I also think we should err on the
> side of excluding Clear Key for now.)
>
>
>
> ​It's a valid concern, but so is the problem that we have a hard deadline,
> so I think we should err on the side of gathering as much evidence as we
> can and providing it with appropriate caveats.
>
>
>
> ...Mark​
>
>
>
>
>
>
>
> For full coverage, all supported combinations would be executed (something
> I discussed
> <https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0100.html>
> earlier
> <https://lists.w3.org/Archives/Public/public-hme-editors/2016Jun/0104.html>). It
> would be nice if we could get results for the general tests run on each key
> system (and type), but we'd need to create some infrastructure.
>
>
>
> David
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 1:17 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> Greg - this makes sense and it would be easy to take the drmtoday test we
> have written and make a new clearkey version of that by enhancing the utils
> and the config as you describe.
>
>
>
> However, we already have a clearkey version of that test in the Google
> directory (which uses its own utils). So, doing what you say would increase
> the commonality / consistency between the tests, but it wouldn't get us
> more tests.
>
>
>
> David - the clearkey results are useful information for the implementation
> report. Again, as with tests based on polyfills, they validate the API
> design, implementability and specification. These are factors in the
> decision as well as the current state of commercially useful features in
> commercial browsers. We are in the unusual situation of not being able to
> just wait until implementations have matured, so this is going to be an
> unusual decision.
>
>
>
> ...Mark
>
>
>
> On Wed, Jul 20, 2016 at 1:05 PM, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
> For (B), I wasn’t suggesting that there be two different tests in one
> file, I was suggesting that we put operations like license requests into
> utils files that would perform either DRMToday or ClearKey license
> requests.  For DRMToday, the implementation in these utils files would make
> the request to the actual DRMToday license server.  For ClearKey, the
> implementation would likely return a response message that is placed into
> the test configuration JSON (drmconfig.json in the example test created by
> Sukhmal).  The JSON config file can help configure both the key system and
> the desired license response message that we need in order to properly
> execute the test.
>
>
>
> G
>
>
>
> On 7/20/16, 1:30 PM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
> So, what we have right now is:
>
> (1) A large number of ClearKey-only tests in a "Google" folder, and
>
> (2) One of those tests (basic playback) migrated to DRM Today, in the root
> folder
>
>
>
> There are two approaches:
>
> (A) Keep ClearKey and DRM tests separate: move the "Google" tests into the
> root or a "clearkey" folder, continue making new DRMToday versions of each
> of those ClearKey tests
>
> (B) Make the DRMToday test also support ClearKey, continue making new
> ClearKey+DRMToday versions of each of the Google tests and, eventually,
> drop the Google folder
>
>
>
> For (B), we need to run two tests in one file, which requires some care
> with async tests and there's been comments that we should not have multiple
> tests in one file.
>
>
>
> Opinions ?
>
>
>
> ...Mark
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 11:27 AM, Greg Rutz <G.Rutz@cablelabs.com> wrote:
>
> I think the test utilities should be designed to be as DRM-independent as
> possible.  This would allow us to run any of the test cases that apply to
> ClearKey simply by providing a DRMConfig and test content that indicates
> use of ClearKey.  I apologize that I have not been following the EME spec
> progression that much over the last 12-18 months, but I recall there not
> being a ton of differences between ClearKey support and other DRMs as I
> implemented it in dash.js.
>
>
>
> For test cases that are valid for ClearKey, the test case would simply
> execute multiple times on the UA under test — once with ClearKey content
> and one or more additional times for the “real” DRMs that are to be tested
> on that UA.  No sense in maintaining separate test code if we don’t have to.
>
>
>
> G
>
>
>
> On 7/20/16, 10:34 AM, "Mark Watson" <watsonm@netflix.com> wrote:
>
>
>
> Question: should we expand this test case to cover ClearKey ? Or will we
> rely on the tests in the Google folder for ClearKey ?
>
>
>
> If the latter, should we move those tests into the main directory (I see
> they are now working) ? Or, if others would like to add ClearKey tests,
> should they add them to the Google folder ?
>
>
>
> ...Mark
>
>
>
> On Tue, Jul 19, 2016 at 7:18 PM, Mark Watson <watsonm@netflix.com> wrote:
>
> All,
>
>
>
> Sukhmal has created a Pull Request for a temporary session test case using
> DRM Today. We have tested this on Chrome with Widevine and it should work
> on Edge with PlayReady as well:
>
>
>
> https://github.com/w3c/web-platform-tests/pull/3313
>
>
>
> Please review this and comment on whether it is a good template / model
> for us to work from. We can quickly migrate more of the Google clearkey
> tests to drmtoday as well as implementing tests for other session types
> based on this model.
>
>
>
> ...Mark
>
>
>
>
>
>
>
>
>
>
>