My testing-related TPAC notes

Hi all,

Writing my weekly snippets today, it's almost entirely about TPAC and
testing, so I thought I'd share here. I'd love to see notes from other
people, especially for sessions I didn't attend.

My first TPAC, talking about web-platform-tests for 3 days.

Joined various WG meetings on Monday:

   - WebRTC: Test as You Commit Policy
   <https://www.w3.org/2017/11/06-webrtc-minutes.html#item03>. Concerns
   about decreased spec change velocity, argued that rate of interoperably
   implemented changes ought to still increase. Document "test as you
   commit" policy in CONTRIBUTING.md
   <https://github.com/w3c/webrtc-pc/pull/1657> was eventually merged.
   - Second Screen: Testing API
   <https://www.w3.org/2017/11/06-webscreens-minutes.html#x13>. Showed the
   new testdriver
   <http://web-platform-tests.org/writing-tests/testdriver.html> mechanism
   in wpt. Looked at the Presentation API Testing API
   <https://github.com/mfoltzgoogle/presentation-test-api/blob/master/presentation_test_api.md>
   and discussing how feasible it would be to represent as WebDriver. Looks
   fairly doable, mfoltz@ to try it and see. Ask for tests for normative
   changes in CONTRIBUTING.md
   <https://github.com/w3c/presentation-api/pull/445> was merged.
   - CSS: Feedback on testing policy
   <https://www.w3.org/2017/11/06-css-minutes.html#item19>. Wording that
   was adopted <https://github.com/w3c/csswg-drafts/pull/1767> hasn't quite
   been working. Great discussion, core issue is how to do this well without
   it simply becoming the job of the editors to write all tests. My claim that
   it shouldn't be the editors and that implementers should write most of the
   tests is just a claim, needs to be proven in reality. Ultimately moved on
   without resolution.
   - WebAppSec: Requiring tests for spec changes
   <https://www.w3.org/2017/11/06-webappsec-minutes.html#item16>. Discussed
   some things that are hard to test, HSTS preload like, and anything not
   eTLD+1 <https://github.com/w3c/web-platform-tests/issues/2669>. Resolved
   to try it anyway, plh sent Added testing policy and fixed a few links
   <https://github.com/w3c/webappsec-csp/pull/230>.


Testing talk all of Tuesday:

   - Minutes <https://www.w3.org/2017/11/07-testing-minutes.html>
   - In roadblocks
   <https://www.w3.org/2017/11/07-testing-minutes.html#item02>, most
   interesting was what it'll take to automatically upstream changes from
   Edge. The can be pushed here without any further review by supplying a
   link to the upstream review
   <https://github.com/w3c/web-platform-tests/#test-review> policy doesn't
   work if the original review is not public. We'll change the policy to
   whatever is needed to get Edge contributions.
   - In triage <https://www.w3.org/2017/11/07-testing-minutes.html#item06>,
   I volunteered to create an experimental cross-vendor triage process and see
   if enough people sign up for it.
   - In Automating manual tests
   <https://www.w3.org/2017/11/07-testing-minutes.html#item07> I summarized
   the current advice as "given we have gsnedders's thing, if you can use WD,
   do it in your own spec and wrap it in testdriver.js ... if it's deeper than
   that, maybe go the way of webusb". WebVR came up as a harder case, also
   discussed in a breakout session on Wednesday.
   - In Test discoverability and coverage
   <https://www.w3.org/2017/11/07-testing-minutes.html#item09>, we talked
   about the upcoming Bikeshed feature Inline WPT test information into
   specs <https://github.com/tabatkins/bikeshed/issues/1116>. Maybe that'll
   work better than the other way around.
   - Had dinner with Tab, Elika, Florian and Simon to continue CSS testing
   discussion. We're going to focus some efforts on flexbox, which is quite
   stable, to see if we can make it work.


Wednesday:

   - Did my Improving interop with web-platform-tests
   <https://bit.ly/tpac-wpt> talk in the morning before the breakout
   sessions. Apart initial technical challenge it went well. Didn't expect
   questions on stage but got some, including a good one from Jeff Jaffe about
   including more browsers. We will if people partner up with us to do the
   running.
   - Then breakout sessions.
   - web-platform-tests breakout session
   <https://www.w3.org/2017/11/08-testing-minutes.html>:
   - Lots of people showed up, a much wider group than in Tuesday's session.
   - flackr@ (and others) want the most recent builds with all features
   enabled, so increased priority of Have two groups of browsers: stable
   and latest with experimental features enabled
   <https://github.com/w3c/wptdashboard/issues/146>
   - domenic@ reported issues running locally on Windows. No clear outcome,
   but Edge folks aware.
   - mobile testing
   <https://www.w3.org/2017/11/08-testing-minutes.html#item02> discussion
   initiated by Baidu led to Support mobile browsers
   <https://github.com/w3c/wptdashboard/issues/219> issue.
   - Testing WebBluetooth, WebUSB, WebVR, etc.
   <https://www.w3.org/2017/11/08-testing-minutes.html#item03> Still
   finding out what works here. Maybe we should use the testdriver API even
   when not backed by WebDriver, but no clear consensus on that point.
   - I attended Dr. Alex's presentation on KITE, can't find minutes.
   - Mike Smith's #stories session
   <https://www.w3.org/2017/11/08-stories-minutes.html>. I liked this idea,
   exciting if we could fit 2018 OKRs into an even bigger picture of where
   we're all going. Marcos and JakeA and had some very nice things to say
   about testing: "then became a bit more strict about testing, which really
   improved the quality of the spec" + "starting from tests make it way faster
   to implement things"