Re: Testing progress

On 05/02/2016 01:56 PM, Charles Engelke wrote:
> I'll be on the call today. Here's a quick update on work I've been
> doing on testing.
>
> My work in progress is at my fork of the Test the Web Forward
> repository: https://github.com/engelke/web-platform-tests/tree/master/WebCryptoAPI.

Thanks so much for the update! This is great work, exactly what the
Working Group needs.
>
> I'm focusing on a single method at first: generateKey. I'm first
> testing that every possible combination of proper parameters creates a
> valid key with the correct properties, then that lots of possible
> combinations of invalid parameters throws an error, AND that it throws
> the correct error per the spec.
>
> Issues:
>
> - No test coverage of RSA-OAEP, ECDSA, or ECDH yet. I don't see any
> particular problems adding them, I'm just not that far along yet.
>
> - Can't automate testing of PBKDF2 because it requires user input
> outside control of the browser.

We could test some collection of strings I imagine.
>
> - The tests are slow, and timeouts have been a problem. I think I've
> handed that, but the test pages take many minutes to run. I think the
> root problem is just that generating RSA keys is slow, and that there
> are a lot of combination to check (there are currently 1800 "success"
> tests and almost 30,000 "failure" tests).

Could we just reuse the same key?
>
> I've only run these tests for Chrome and Firefox (and tried an
> earlier, less complete version for Edge) so far. The browsers do very
> well on the tests for successes, but have a fair number of issues on
> tests for failures. They fail where they should, but not necessarily
> by throwing the correct error. Even checking the correct error is
> problematic, because the browsers don't necessarily have the necessary
> DOMException property names defined in each case.
>
> Chrome's non-successes are for 192 AES keys, which it doesn't support.
> Firefox's non-successes are for not supporting RSA-PSS.
As long as two browsers support 192 keys, we can keep it in the spec.

I think they said the  added RSA-PSS into nightly build.


>
> Charlie
>

Received on Monday, 2 May 2016 18:00:46 UTC