Re: My testing-related TPAC notes

> Testing talk all of Tuesday:
> 
>   * Minutes <https://www.w3.org/2017/11/07-testing-minutes.html>
>   * In roadblocks
>     <https://www.w3.org/2017/11/07-testing-minutes.html#item02>, most
>     interesting was what it'll take to automatically upstream changes
>     from Edge. The can be pushed here without any further review by
>     supplying a link to the upstream review
>     <https://github.com/w3c/web-platform-tests/#test-review> policy
>     doesn't work if the original review is not public. We'll change the
>     policy to whatever is needed to get Edge contributions.
>   * In triage
>     <https://www.w3.org/2017/11/07-testing-minutes.html#item06>, I
>     volunteered to create an experimental cross-vendor triage process
>     and see if enough people sign up for it.
>   * In Automating manual tests
>     <https://www.w3.org/2017/11/07-testing-minutes.html#item07> I
>     summarized the current advice as "given we have gsnedders's thing,
>     if you can use WD, do it in your own spec and wrap it in
>     testdriver.js ... if it's deeper than that, maybe go the way of
>     webusb". WebVR came up as a harder case, also discussed in a
>     breakout session on Wednesday.

WebVR actually turns out to not be a very hard case; as far as I can 
tell so far. I spent a bit of time talking to Nell from the WebVR group 
and it seems like their requirements map quite neatly on to WebDriver. I 
am going to add some extra hooks for them in the spec and they are going 
to draft a proposal.

FWIW I also heard out-of-band feedback that one of the most common 
things that developers on native platforms want to test is "how does my 
app repsond to bluetooth going up/down/etc." so it seems possible that 
some of the things that we have said should be done in test-specific 
APIs for bluetooth, usb, etc. are actually missed opportunities to make 
the Web Platform easier to test compared to native in complex device 
scenarios.


>     <https://github.com/w3c/wptdashboard/issues/146>
>   * domenic@ reported issues running locally on Windows. No clear
>     outcome, but Edge folks aware.

To expand on this a little, the problems that I know of on windows are:
1) No Python/git by default
2) No virtualenv by default
3) Windows requires slightly different command line syntax than other 
platforms
4) We don't know how to download the right Edgedriver automatically.

The first three issues are non-problems if you use Windows Subsystem for 
Linux; I would recommend that environment for people using wpt where 
possible. In the absence of that, the first is likely not going to be 
solved in a meaningful way except via docs (unless someone puts together 
a wpt installer and keeps it up to date, or adds some bootstrap batch 
files to help install deps), the second seems solvable within the wpt 
tool itself, the third isn't solvable without using WSL and I have heard 
from a Microsoft developer that there's prior art for solving the fourth 
by running some tool to determine the Windows version and reverse 
engineering the URL scheme.

So from my point of view the main missing piece is someone who actually 
uses Windows regularly contributing to the project; I can solve some of 
these things in a one-off way but it's disproportionately difficult to 
get the environment set up and fixes are likely to regress.

Received on Monday, 13 November 2017 14:34:05 UTC