- From: Jerry Smith (WPT) <jdsmith@microsoft.com>
- Date: Mon, 19 Sep 2016 18:44:23 +0000
- To: Paul Cotton <Paul.Cotton@microsoft.com>, Matt Wolenetz <wolenetz@google.com>
- CC: Philippe Le Hégaret <plh@w3.org>, "public-hme-editors@w3.org" <public-hme-editors@w3.org>
- Message-ID: <SN2PR03MB04703833FF34A7E882201C0A4F40@SN2PR03MB047.namprd03.prod.outlook.com>
10 of the 14 failures are for “Unrecognized type EventHandler”. These appear to be test bugs, since EventHandler is the correct type in all cases.
I believe Philippe is aware of these.
From: Paul Cotton [mailto:Paul.Cotton@microsoft.com]
Sent: Monday, September 19, 2016 6:11 AM
To: Matt Wolenetz <wolenetz@google.com>
Cc: Philippe Le Hégaret <plh@w3.org>; public-hme-editors@w3.org
Subject: RE: MSE IDL tests
> PFA updated Chrome JSONs. I'll continue analysis on them tomorrow.
Can you confirm that there no work left to do on the MSE WebIDL tests?
Nearly half of the sub-test failures in the current MSE test suite are for tests on media-source/interfaces.html:
https://w3c.github.io/test-results/media-source/less-than-2.html
What is our explanation for these failures?
/paulc
From: Matt Wolenetz [mailto:wolenetz@google.com]
Sent: Thursday, September 8, 2016 10:16 PM
To: Philippe Le Hégaret <plh@w3.org<mailto:plh@w3.org>>
Cc: Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>
Subject: Re: MSE IDL tests
Note, those JSONs are produced with the "--enable-experimental-web-platform-features" cmdline flag to Chrome.
On Thu, Sep 8, 2016 at 7:14 PM, Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> wrote:
#3662 is fixed (though FF and others may fail, shouldn't block PR though).
PFA updated Chrome JSONs. I'll continue analysis on them tomorrow.
These are from tip-of-tree Chrome local builds against w3c-test.org<http://w3c-test.org>'s runner. "no-isobmff" is a version for which I've forced my local build of Chrome to say "false" to any audio/mp4 or video/mp4 isTypeSupported() queries, so that the webm alternative (if any) is tested instead. Any tests that fail for mp4 but pass without mp4 support are likely due to mp4 PTS/DTS conflation in Chrome. IIUC, Edge also has such an issue, but FF has supposedly done this correctly.
On Thu, Sep 8, 2016 at 10:59 AM, Matt Wolenetz <wolenetz@google.com<mailto:wolenetz@google.com>> wrote:
Thank you, Philippe. https://codereview.chromium.org/2315113002/ landed last night, but is not yet in Canary. If I get some time today, I'll generate local Chrome with and without isobmff JSONs with builds including that fix, and I'll include links to the compliance (or test suite) bugs discovered as a result.
First though, I'll try to alleviate the problems FF is having with the current appendbuffer-quotaexceeded test: https://github.com/w3c/web-platform-tests/issues/3662#issuecomment-245683923
On Thu, Sep 8, 2016 at 10:41 AM, Philippe Le Hégaret <plh@w3.org<mailto:plh@w3.org>> wrote:
I regenerated the test results.
https://w3c.github.io/test-results/media-source/all.html
https://w3c.github.io/test-results/media-source/less-than-2.html
I think we're good to go on that front.
Philippe
On 9/8/2016 10:09 AM, Paul Cotton wrote:
Can you merge this PR today so we can refresh the MSE results?
/paulc
From: Matt Wolenetz [mailto:wolenetz@google.com]
Sent: Thursday, September 8, 2016 9:48 AM
To: Francois Daoust <fd@w3.org><mailto:fd@w3.org>
Cc: Philippe Le Hegaret (plh@w3.org<mailto:plh@w3.org>) <plh@w3.org><mailto:plh@w3.org>; <public-html-media@w3.org><mailto:public-html-media@w3.org> <public-html-media@w3.org><mailto:public-html-media@w3.org>; Paul Cotton <Paul.Cotton@microsoft.com><mailto:Paul.Cotton@microsoft.com>
Subject: Re: MSE IDL tests (was: Re: HME WG meeting minutes, Tue Aug 30)
That makes sense. Thank you for finding and fixing the test problem.
On Sep 8, 2016 6:09 AM, "Francois Daoust" <fd@w3.org<mailto:fd@w3.org>> wrote:
Matt,
In essence, I think that the test harness, called by the IDL harness, expects to get the list of tests before the "load" event fires on window. In other words, it expects all calls to "test" and "async_test" to have been made by then.
In the MSE case, we need to wait for the MediaSource object to be ready before we may call the IDL harness, and the "load" event will have already fired at that point. The IDL harness tells the test harness to run the first test, and since the "load" event already fired, the test harness believes it is the only test to run, and switches to an "I'm done!" mode, getting ready to report the results. Problem is this is when it receives a number of additional tests to perform. This makes the test harness go berserk.
The solution is relatively simple: we just need to be explicit as to when the list of tests is complete, which can be done with an initial call to "setup({ explicit_done: true });" followed by a call to "done()" after having run the IDL harness.
Done in the following PR:
https://github.com/w3c/web-platform-tests/pull/3664
Francois.
Le 07/09/2016 à 23:22, Matt Wolenetz a écrit :
Thanks Philippe. Help with the IDL failures would be appreciated -- it
looks like it's not specific to just Chrome. As mentioned in narrower
thread, I've updated and landed #3655, and will rerun Chrome's JSON (for
builds with, and without, ISO-BMFF support claimed by isTypeSupported())
once I land https://codereview.chromium.org/2315113002/ (which should
fix a bunch of the TypeError-related Chrome failures). Landing that
Chrome patch may not come before EOD today, though.
On Wed, Sep 7, 2016 at 12:03 PM, Philippe Le Hégaret <plh@w3.org<mailto:plh@w3.org>
<mailto:plh@w3.org<mailto:plh@w3.org>>> wrote:
On 9/2/2016 7:24 PM, Matt Wolenetz wrote:
I've made some progress, not as much as I had hoped. 2 of my PRs
are currently pending co-editor review. 5 pending PRs are still
awaiting my full review.
Thank you Matt for making progress!
I merged one of your PRs, still looking at #3655.
I regenerated test results and added a filter to eliminate
VideoPlaybackQuality and TrackDefault related:
http://w3c.github.io/test-results/media-source/less-than-2.html
<http://w3c.github.io/test-results/media-source/less-than-2.html>
We get a lot of errors on the idlharness tests which I'll look at as
well.
Philippe
Detail:
* Prepped PR to cover 4 gaps from #823
: https://github.com/w3c/web-platform-tests/pull/3635
<https://github.com/w3c/web-platform-tests/pull/3635>
o *Awaiting @jdsmith3000's review to land it*
o crbug 623781 <http://crbug/623781> tracks fixing the
Chrome compliance issue behind at least the failure of 2
subtests (TypeError) in mediasource-addsourcebuffer.ht<http://mediasource-addsourcebuffer.ht>
<http://mediasource-addsourcebuffer.ht>ml
o Filed crbug 643788 <http://crbug/643788> to fix debug
build assertion in w-p-t
mediasource-removesourcebuffers.html and landed my fix for
it <https://codereview.chromium.org/2305023002>
* Reviewed and closed
ancient https://github.com/w3c/web-platform-tests/pull/939
<https://github.com/w3c/web-platform-tests/pull/939>
o Found only 1 trivial test coverage gap, prepped and merged
trivial PR to close that
gap: https://github.com/w3c/web-platform-tests/pull/3636
<https://github.com/w3c/web-platform-tests/pull/3636>
* Deferred to VNext ancient appendStream coverage test
PR https://github.com/w3c/web-platform-tests/pull/954
<https://github.com/w3c/web-platform-tests/pull/954>
* Deferred to VNext more strict isTypeSupported() coverage test
PR https://github.com/w3c/web-platform-tests/pull/1405
<https://github.com/w3c/web-platform-tests/pull/1405>
*
Commented on mechanical
PR https://github.com/w3c/web-platform-tests/pull/1816
<https://github.com/w3c/web-platform-tests/pull/1816> (doesn't
block MSE v1 PR, it's a cross-cutting infra PR)
*
In-progress:
Fixing https://github.com/w3c/web-platform-tests/pull/3082
<https://github.com/w3c/web-platform-tests/pull/3082>
o Filed, and in-progress fixing (short-term Chrome MSE piece
of) crbug 643846 <http://crbug/643846>
*
TODO 5 more pending PRs...
On Tue, Aug 30, 2016 at 6:42 PM, Matt Wolenetz
<wolenetz@google.com<mailto:wolenetz@google.com> <mailto:wolenetz@google.com<mailto:wolenetz@google.com>>> wrote:
On Tue, Aug 30, 2016 at 11:20 AM, Paul Cotton
<Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com> <mailto:Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>>
wrote:
*MSE test suite results*
<*/plh/*> Paul: still some outstanding pull requests
https://lists.w3.org/Archives/Public/public-html-media/2016Aug/0102.html
<https://lists.w3.org/Archives/Public/public-html-media/2016Aug/0102.html>
is Matt's most recent update
<*/plh/*> Matt: going through them...
Outstanding PRs:
https://github.com/w3c/web-platform-tests/pulls?q=is%3Apr+is%3Aopen+label%3Amedia-source
<https://github.com/w3c/web-platform-tests/pulls?q=is%3Apr+is%3Aopen+label%3Amedia-source>
<*/plh/*> ... resolving the merge conflict
<*/plh/*> ... I'll send out an update later today
I've worked through about 5 of the 15 pending PRs today, and
will continue tomorrow. Current status:
*
* Merged
@tidoust's https://github.com/w3c/web-platform-tests/pull/3239
<https://github.com/w3c/web-platform-tests/pull/3239>
* Prepped and landed
my https://github.com/w3c/web-platform-tests/pull/3612
<https://github.com/w3c/web-platform-tests/pull/3612>
* Needed Chromium+W3C license stamp, so this
replaces/extends
@tidoust's https://github.com/w3c/web-platform-tests/pull/3264
<https://github.com/w3c/web-platform-tests/pull/3264>
* Updated and landed
my https://github.com/w3c/web-platform-tests/pull/3296
<https://github.com/w3c/web-platform-tests/pull/3296>
* Landed
@tidoust's https://github.com/w3c/web-platform-tests/pull/3233
<https://github.com/w3c/web-platform-tests/pull/3233>
* Prepped PR to fix the test introduced in #3233 to comply
with
w3c/media-source#154: https://github.com/w3c/web-platform-tests/pull/3613
<https://github.com/w3c/web-platform-tests/pull/3613>
o *Awaiting @jdsmith3000's review to land it*
o crbug 639144 <http://crbug/639144> tracks fixing the
Chrome compliance issue behind failure of subtest 1/3
in this PR
o crbug 373039 <http://crbug/373039> is one of a group
of bugs tracking fixing the Chrome compliance issue
behind failure of subtest 3/3 in this PR
* Reviewed
ancient https://github.com/w3c/web-platform-tests/pull/823
<https://github.com/w3c/web-platform-tests/pull/823>
o Found 4 gaps, but otherwise closed this PR since most
of the coverage it includes is already covered
elsewhere in our current suite
*
In-progress: put together PR to cover 4 gaps from #823
o crbug 623781 <http://crbug/623781> tracks fixing the
Chrome compliance issue behind at least the failure of
2 subtests (TypeError) in
mediasource-addsourcebuffer.ht<http://mediasource-addsourcebuffer.ht>
<http://mediasource-addsourcebuffer.ht>ml
o *In-progress: Investigating Chromium issue with PR I'm
building to cover the 4 test gaps.*
* TODO 10 more pending PRs, followed by call to regenerate
reports and lists of known user agent bugs tracking test
failures.
Received on Monday, 19 September 2016 18:44:55 UTC