Re: EME test reports

I think we should replace the key system name in the test name with either
'drm' or 'clearKey'. Since each browser supports only a single DRM (that we
can test), then we know the keysystem that is being tested in each case.

...Mark


On Aug 29, 2016, at 12:36 PM, Jerry Smith (WPT) <jdsmith@microsoft.com>
wrote:

One thing we should probably resolve:  Including key system strings in the
subtest name is creating misleading test gaps vs. our 2 passing
implementation requirement.  The wptreport tool treats these as separate
tests.  All com.microsoft.playready subtests are flagged currently as not
having two passing implementations.



As a simple experiment, I edited the JSONs to pull the key system names
from the drm subtest names.  That results in:



-          *Test files*: 105; *Total subtests*: 257

-          *Test files without 2 passes*: 29; *Subtests without 2 passes: *49;
*Failure level*: 49/257 (19.07%)

-          *Completely failed files*: 29; *Completely failed
subtests*: 8; *Failure
level*: 8/257 (3.11%)



Vs. online (with key system names in subtest names);



-          *Test files*: 105; *Total subtests*: 299

-          *Test files without 2 passes*: 50; *Subtests without 2 passes: *91;
*Failure level*: 91/299 (30.43%)

-          *Completely failed files*: 50; *Completely failed
subtests*: 10; *Failure
level*: 10/299 (3.34%)



That means:



-          Listing the key system in the subtest names results in 42
additional subtests.

-          Most of these have more than one passing implementation
presently.

-          2 subtests (drm-keystatuses-multiple-sessions) report as
complete fail with key system in the subtest name, but have one passing
implementation when key system is not.



Options to handle this are:



-          Remove the keysystem names from the subtests, but lose
visibility into keysystem specific failures.

-          Leave them in and post those results to the website, but prepare
manual tallies when we submit PR for review.



Jerry