Re: Help with testing

On June 20, 2017 at 12:21:06 AM, Telford-Reed, Nick
(nick.telford-reed@worldpay.com) wrote:
> On behalf of the chairs, can I ask the group to consider whether their organisation would
> be able to provide additional help or resources to aid our testing efforts? Your support
> is much appreciated in advance. We will return to this in our next teleconferences.

Mozilla is committed to the testing effort and has provided a
significant number of tests. Thanks also to our friends at Google, who
have also provided a great set of tests.

However, we could use more help! If people want to help, but don't
know what needs to be done or how to get started, let me know and I'd
be happy to guide you.

Start here: http://web-platform-tests.org/

We really need people to review, and help come up with more tests for
edge-cases. If your organization would prefer to throw money at the
problem, that's also an option that is very welcomed and we can try to
find an intern to work on testing.

We are maybe 20% into the testing effort and we've already:

 * Crashed 3 browsers, across multiple platforms.
 * Found security bugs in multiple browsers.
 * Found mis-match in currency handling (!) across browsers.
 * Exposed large interop issues, even with simple things like the
PaymentRequest constructor.
 * Found race conditions - whereby payment sheets can end up being shown blank.
 * Found significant flaws with regards to PMI data handling - leading
to spec changes.
 * Found significant flaws with payment method modifiers (related to
the previous point) - leading to spec changes.

Thankfully, most of the issues above have already been fixed - showing
the value of finding these bugs while browser vendors are still
actively implementing. If we miss the boat with testing, then browser
vendors will move onto other things and it will take much longer to
get things fixed.

Additionally, without the test suite we cannot verify the quality and
interoperability of the spec, and thus it cannot become a Web
Standard. Issues which we don't catch with testing will manifest
themselves in the wild (at the detriment of end users, merchants, and
those processing payments). Thus, I'd kindly ask that more
organizations get involved with the testing now, otherwise it's gonna
be really expensive once the PR API becomes widely available
(hindering adoption, and benefiting non-standard payment systems,
which are better QA'd right now).

Kind regards,
Marcos

Received on Tuesday, 20 June 2017 04:36:57 UTC