- From: Chris Pearce <cpearce@mozilla.com>
- Date: Fri, 23 Sep 2016 08:36:25 +1200
- To: David Dorwin <ddorwin@google.com>
- Cc: "Jerry Smith (WPT)" <jdsmith@microsoft.com>, Paul Cotton <Paul.Cotton@microsoft.com>, Mark Watson <watsonm@netflix.com>, Matt Wolenetz <wolenetz@google.com>, "<public-html-media@w3.org>" <public-html-media@w3.org>
- Message-ID: <CAFWdrwYy7nnF_di26jJRTxyzRPQ=Q36g_E9WrZ2wKCUAUyWaVg@mail.gmail.com>
On Wed, Sep 21, 2016 at 1:50 PM, Chris Pearce <cpearce@mozilla.com> wrote: > Hi, > > Apologies for my tardy response, I've been on leave. > > On Wed, Sep 14, 2016 at 4:55 PM, David Dorwin <ddorwin@google.com> wrote: > >> [WAS: Updated EME Test Status] >> >> The following is an analysis of the less-than-2 results >> <http://w3c.github.io/test-results/encrypted-media/less-than-2.html> Jerry >> published earlier today. There are specific requests for *Jerry* and >> *Chris* below, but we could use help from everyone. >> >> If you can help with the DRMtoday server issues or migrate the tests to >> use the new test account >> <https://github.com/w3c/web-platform-tests/issues/3624>, please let us >> know ASAP as this is blocking all drm-* tests. >> >> Until the DRMtoday test server issue is resolved, we'll have to mostly >> ignore the drm-* tests since we do not have good results. However, I've >> commented on some that were failing in the test results before the today's >> update. >> >> In addition to the failures below, we may also see additional test >> failures as the Google/ tests are migrated and start running on Edge and >> Firefox as well exercising commercial Key Systems. >> >> *Google/** >> All of the test in Google/ are expected to potentially fail on other >> browsers until the tests are migrated. We need help migrating the remaining >> tests: https://github.com/w3c/web-platform-tests/issues/3583 >> #issuecomment-243577488. >> >> (I merged an update to Google/encrypted-media-syntax.html that fixes the >> new failure reported in Chrome 55.) >> >> *idlharness.html* >> >> - Three "interface: attribute" tests for the new event handler >> attributes: >> - Chrome passes these. >> - We need another browser to implement the three new event handler >> attributes. This is trivial, so while it should not block PR, it should be >> easy to fix. >> - *Jerry* and *Chris*, do you have plans to implement these? >> >> > Yes, it should be easy for us to fix. > I have implemented the three new event handler attributes. They should appear in Firefox Nightly in the next day or so. Chris. > > >> >> - Nine "interface: operation" tests: >> - Firefox passes these. >> - The failures in Chrome are known >> <https://bugs.chromium.org/p/chromium/issues/detail?id=635688> and >> caused by a Blink issue >> <https://bugs.chromium.org/p/chromium/issues/detail?id=627309> unrelated >> to EME. >> - Edge also fails these tests. *Jerry*, do you know why? >> - I think we can just note this in the test report. >> >> (While not an issue for PR since Edge and Firefox pass, for future >> reference, the six "existence and properties of interface prototype object" >> test failures in Chrome are also known >> <https://bugs.chromium.org/p/chromium/issues/detail?id=635694> and >> unrelated to EME.) >> >> *clearkey-*persistent** >> These eight tests (all complete-fails) are all related to persistent-* >> session types. No Clear Key implementation supports anything other than >> temporary sessions, which makes sense, and I don't expect this to change. >> Thus, these tests will not pass in the v1 timeframe. >> >> *clearkey-keystatuses.html* >> Chrome passes. Since Edge does not implement Clear Key, we need *Firefox* >> to pass this test. It currently fails with: >> >>> assert_equals: keystatus value for invalid key should be undefined (1) >>> expected (undefined) undefined but got (string) "internal-error" >>> >> >> *Chris*, please take a look. >> > > > This now passes in Firefox 51 (in our Developer Edition/Aurora channel > tomorrow I believe). > > >> >> *drm-keystatuses.html* >> Mark wrote >> <https://github.com/w3c/web-platform-tests/issues/3618#issuecomment-243835790> >> : >> >>> ... it appears that all three browsers are non-compliant: >>> >>> - Chrome does not generate a keystatuseschange event after close() is >>> called (for the DRM case, it does for the ClearKey case) >>> >>> >>> - Firefox and Edge both have incorrect values in the keystatuses map >>> (in different ways). >>> >>> >> The Chrome issue is likely the same as this known >> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> issue. >> *Jerry* and *Chris*, please take a look. >> > > This is caused by Firefox dispatching one keystatuseschange event for > every key that changes status, rather than aggregating keys changing status > at the same time into a single keystatsueschange event. The spec says we > should be aggregating the status changes into a single event, so we'll > update to match the spec. We're working on this in bug 1303922 > <https://bugzilla.mozilla.org/show_bug.cgi?id=1303922>. > > > >> >> *drm-mp4-playback-temporary-events.html* >> Last time the tests were run, this was passing on Firefox, failing on >> Chrome, and timing out on Edge. >> >> This Chrome issue is known >> <https://bugs.chromium.org/p/chromium/issues/detail?id=622956> and under >> investigation. >> *Jerry*, do you know why Edge was timing out? >> >> *drm-mp4-playback-temporary-waitingforkey.html* >> Last time the tests were run, this was timing out on all three browsers, >> but Mark landed a fix. >> > > Firefox Nightly passes this test. > > >> >> *drm-mp4-playback-*persistent-license** >> "persistent-license" sessions are only supported on Chrome on Chrome OS, >> and I'm not aware of any plans to support them in other implementations >> before PR. >> >> At least one of these were failing the last step on Chrome OS due to a >> DRMtoday server issue. >> >> *mp4-playback-*persistent-usage-record** >> "persistent-usage-record" sessions are only supported in Edge. >> >> Last time the tests were run, none of the three tests were passing on >> Edge. >> >> >> On Tue, Sep 13, 2016 at 11:24 AM, Jerry Smith (WPT) < >> jdsmith@microsoft.com> wrote: >> >>> I’ve posted updated results for EME tests: >>> >>> >>> >>> - http://w3c.github.io/test-results/encrypted-media/all.html >>> >>> o Test files: 70; Total subtests: 220 >>> >>> - http://w3c.github.io/test-results/encrypted-media/less-than- >>> 2.html >>> >>> o Test files without 2 passes: 44; Subtests without 2 passes: 64; >>> Failure level: 64/220 (29.09%) >>> >>> - http://w3c.github.io/test-results/encrypted-media/complete-f >>> ails.html >>> >>> o Completely failed files: 44; Completely failed subtests: 29; >>> Failure level: 29/220 (13.18%) >>> >>> >>> >>> Some comments about these results: >>> >>> >>> >>> 1. They don’t filter for single valid test outcomes (either pass >>> or fail), and timeouts as failures. Most of the complete-fails are test >>> timeouts. >>> >>> 2. Test cases have only been partially (~25%) migrated to full >>> drm from Clear Key versions contributed by Google. The original Google >>> tests are run if not migrated, and can be distinguished by “Google” in the >>> test file path. >>> >>> >>> >>> Jerry >>> >> >> > Chris Pearce. > >
Received on Thursday, 22 September 2016 20:36:55 UTC