RE: Updated EME Test Results

Some breakdown:


-          Wrong error type:                        24 failures

-          Wrong promise handling:            14 failures

That’s close to half of the total failures.

There are about 19 test timeouts, which is also a large bucket.

As previously, there are 14 “failures” that aren’t real.  They have one passing implementation because they are PlayReady keySystem specific.

Jerry

From: Jerry Smith (WPT)
Sent: Monday, October 10, 2016 1:44 PM
To: 'Mark Watson' <watsonm@netflix.com>
Cc: David Dorwin <ddorwin@google.com>; Paul Cotton <Paul.Cotton@microsoft.com>; Matt Wolenetz <wolenetz@google.com>; Philippe Le Hégaret <plh@w3.org>; public-html-media@w3.org
Subject: RE: Updated EME Test Results

I’ve posted updated test results, but haven’t had a chance to go through them in detail.  I intend to do that by end of today.

This is the first test pass that has all Google tests migrated fully, and should represent the majority of our planned test cases.


-          http://w3c.github.io/test-results/encrypted-media/all.html


o   Test files: 97; Total subtests: 320



-          http://w3c.github.io/test-results/encrypted-media/less-than-2.html


o   Test files without 2 passes: 42; Subtests without 2 passes: 83; Failure level: 83/320 (25.94%)



-          http://w3c.github.io/test-results/encrypted-media/complete-fails.html


o   Completely failed files: undefined; Completely failed subtests: 45; Failure level: 45/320 (14.06%)

Jerry