Testing progress

I'll be on the call today. Here's a quick update on work I've been
doing on testing.

My work in progress is at my fork of the Test the Web Forward
repository: https://github.com/engelke/web-platform-tests/tree/master/WebCryptoAPI.

I'm focusing on a single method at first: generateKey. I'm first
testing that every possible combination of proper parameters creates a
valid key with the correct properties, then that lots of possible
combinations of invalid parameters throws an error, AND that it throws
the correct error per the spec.

Issues:

- No test coverage of RSA-OAEP, ECDSA, or ECDH yet. I don't see any
particular problems adding them, I'm just not that far along yet.

- Can't automate testing of PBKDF2 because it requires user input
outside control of the browser.

- The tests are slow, and timeouts have been a problem. I think I've
handed that, but the test pages take many minutes to run. I think the
root problem is just that generating RSA keys is slow, and that there
are a lot of combination to check (there are currently 1800 "success"
tests and almost 30,000 "failure" tests).

I've only run these tests for Chrome and Firefox (and tried an
earlier, less complete version for Edge) so far. The browsers do very
well on the tests for successes, but have a fair number of issues on
tests for failures. They fail where they should, but not necessarily
by throwing the correct error. Even checking the correct error is
problematic, because the browsers don't necessarily have the necessary
DOMException property names defined in each case.

Chrome's non-successes are for 192 AES keys, which it doesn't support.
Firefox's non-successes are for not supporting RSA-PSS.

Charlie

Received on Monday, 2 May 2016 17:56:34 UTC