Re: My testing-related TPAC notes

I missed one important thing from Tuesday because it wasn't its own heading
in minutes:

In wpt.fyi metrics <https://www.w3.org/2017/11/07-testing-minutes.html>
discussion (no heading, search for "dknox"), we discussed the PRD
<https://docs.google.com/document/d/1g22NuZ82RqPUYEOUzihc3quCuWqZ2NcearOGOIYlF20/edit?usp=sharing>
and options for presentation. Converged on columns for 4/3/2/1/0 passing
with absolute numbers of tests, no percentages, and no aggregate score. How
to switch between that and the current per-browser view not entirely clear.

+Mark Dittmer <markdittmer@chromium.org> will be working on this, and
trying to actually build something that works will no doubt require
reworking/refining this idea.

GitHub issue is https://github.com/w3c/wptdashboard/issues/83.

On Mon, Nov 13, 2017 at 2:28 PM Philip Jägenstedt <foolip@chromium.org>
wrote:

> Hi all,
>
> Writing my weekly snippets today, it's almost entirely about TPAC and
> testing, so I thought I'd share here. I'd love to see notes from other
> people, especially for sessions I didn't attend.
>
> My first TPAC, talking about web-platform-tests for 3 days.
>
> Joined various WG meetings on Monday:
>
>    - WebRTC: Test as You Commit Policy
>    <https://www.w3.org/2017/11/06-webrtc-minutes.html#item03>. Concerns
>    about decreased spec change velocity, argued that rate of interoperably
>    implemented changes ought to still increase. Document "test as you
>    commit" policy in CONTRIBUTING.md
>    <https://github.com/w3c/webrtc-pc/pull/1657> was eventually merged.
>    - Second Screen: Testing API
>    <https://www.w3.org/2017/11/06-webscreens-minutes.html#x13>. Showed
>    the new testdriver
>    <http://web-platform-tests.org/writing-tests/testdriver.html>
>    mechanism in wpt. Looked at the Presentation API Testing API
>    <https://github.com/mfoltzgoogle/presentation-test-api/blob/master/presentation_test_api.md>
>    and discussing how feasible it would be to represent as WebDriver. Looks
>    fairly doable, mfoltz@ to try it and see. Ask for tests for normative
>    changes in CONTRIBUTING.md
>    <https://github.com/w3c/presentation-api/pull/445> was merged.
>    - CSS: Feedback on testing policy
>    <https://www.w3.org/2017/11/06-css-minutes.html#item19>. Wording that
>    was adopted <https://github.com/w3c/csswg-drafts/pull/1767> hasn't
>    quite been working. Great discussion, core issue is how to do this well
>    without it simply becoming the job of the editors to write all tests. My
>    claim that it shouldn't be the editors and that implementers should write
>    most of the tests is just a claim, needs to be proven in reality.
>    Ultimately moved on without resolution.
>    - WebAppSec: Requiring tests for spec changes
>    <https://www.w3.org/2017/11/06-webappsec-minutes.html#item16>.
>    Discussed some things that are hard to test, HSTS preload like, and
>    anything not eTLD+1
>    <https://github.com/w3c/web-platform-tests/issues/2669>. Resolved to
>    try it anyway, plh sent Added testing policy and fixed a few links
>    <https://github.com/w3c/webappsec-csp/pull/230>.
>
>
> Testing talk all of Tuesday:
>
>    - Minutes <https://www.w3.org/2017/11/07-testing-minutes.html>
>    - In roadblocks
>    <https://www.w3.org/2017/11/07-testing-minutes.html#item02>, most
>    interesting was what it'll take to automatically upstream changes from
>    Edge. The can be pushed here without any further review by supplying a
>    link to the upstream review
>    <https://github.com/w3c/web-platform-tests/#test-review> policy
>    doesn't work if the original review is not public. We'll change the policy
>    to whatever is needed to get Edge contributions.
>    - In triage <https://www.w3.org/2017/11/07-testing-minutes.html#item06>,
>    I volunteered to create an experimental cross-vendor triage process and see
>    if enough people sign up for it.
>    - In Automating manual tests
>    <https://www.w3.org/2017/11/07-testing-minutes.html#item07> I
>    summarized the current advice as "given we have gsnedders's thing, if you
>    can use WD, do it in your own spec and wrap it in testdriver.js ... if it's
>    deeper than that, maybe go the way of webusb". WebVR came up as a harder
>    case, also discussed in a breakout session on Wednesday.
>    - In Test discoverability and coverage
>    <https://www.w3.org/2017/11/07-testing-minutes.html#item09>, we talked
>    about the upcoming Bikeshed feature Inline WPT test information into
>    specs <https://github.com/tabatkins/bikeshed/issues/1116>. Maybe
>    that'll work better than the other way around.
>    - Had dinner with Tab, Elika, Florian and Simon to continue CSS
>    testing discussion. We're going to focus some efforts on flexbox, which is
>    quite stable, to see if we can make it work.
>
>
> Wednesday:
>
>    - Did my Improving interop with web-platform-tests
>    <https://bit.ly/tpac-wpt> talk in the morning before the breakout
>    sessions. Apart initial technical challenge it went well. Didn't expect
>    questions on stage but got some, including a good one from Jeff Jaffe about
>    including more browsers. We will if people partner up with us to do the
>    running.
>    - Then breakout sessions.
>    - web-platform-tests breakout session
>    <https://www.w3.org/2017/11/08-testing-minutes.html>:
>    - Lots of people showed up, a much wider group than in Tuesday's
>    session.
>    - flackr@ (and others) want the most recent builds with all features
>    enabled, so increased priority of Have two groups of browsers: stable
>    and latest with experimental features enabled
>    <https://github.com/w3c/wptdashboard/issues/146>
>    - domenic@ reported issues running locally on Windows. No clear
>    outcome, but Edge folks aware.
>    - mobile testing
>    <https://www.w3.org/2017/11/08-testing-minutes.html#item02> discussion
>    initiated by Baidu led to Support mobile browsers
>    <https://github.com/w3c/wptdashboard/issues/219> issue.
>    - Testing WebBluetooth, WebUSB, WebVR, etc.
>    <https://www.w3.org/2017/11/08-testing-minutes.html#item03> Still
>    finding out what works here. Maybe we should use the testdriver API even
>    when not backed by WebDriver, but no clear consensus on that point.
>    - I attended Dr. Alex's presentation on KITE, can't find minutes.
>    - Mike Smith's #stories session
>    <https://www.w3.org/2017/11/08-stories-minutes.html>. I liked this
>    idea, exciting if we could fit 2018 OKRs into an even bigger picture of
>    where we're all going. Marcos and JakeA and had some very nice things to
>    say about testing: "then became a bit more strict about testing, which
>    really improved the quality of the spec" + "starting from tests make it way
>    faster to implement things"
>
>