Re: Testing progress

On Mon, May 2, 2016 at 11:10 AM, Charles Engelke <w3c@engelke.com> wrote:

> Re: key reuse. We're testing that we can generate a key, so it's going
> through that slow, expensive, process every time the parameters are
> good enough to get that far in the algorithm.
>
> Now that I'm fairly far along, I want to see if there's use case
> trimming that can cut that way down without reducing the actual scope
> of test coverage.
>
> Charlie
>
> On Mon, May 2, 2016 at 2:00 PM, Harry Halpin <hhalpin@w3.org> wrote:
> >
> >
> > On 05/02/2016 01:56 PM, Charles Engelke wrote:
> >> I'll be on the call today. Here's a quick update on work I've been
> >> doing on testing.
> >>
> >> My work in progress is at my fork of the Test the Web Forward
> >> repository:
> https://github.com/engelke/web-platform-tests/tree/master/WebCryptoAPI.
> >
> > Thanks so much for the update! This is great work, exactly what the
> > Working Group needs.
> >>
> >> I'm focusing on a single method at first: generateKey. I'm first
> >> testing that every possible combination of proper parameters creates a
> >> valid key with the correct properties, then that lots of possible
> >> combinations of invalid parameters throws an error, AND that it throws
> >> the correct error per the spec.
> >>
> >> Issues:
> >>
> >> - No test coverage of RSA-OAEP, ECDSA, or ECDH yet. I don't see any
> >> particular problems adding them, I'm just not that far along yet.
> >>
> >> - Can't automate testing of PBKDF2 because it requires user input
> >> outside control of the browser.
>

RE: generateKey() for PBKDF2:

Aside from the problem with automation, there is a bigger issue -- does any
User Agent actually implement generateKey() for PBKF2?

Chrome certainly doesn't.
And as far as I can tell neither does Firefox
(I can only speak authoritatively for Chromium's implementation).

I raised my concerns with PBKDF2's specification in this bug:
https://www.w3.org/Bugs/Public/show_bug.cgi?id=27689

I don't believe as specified generateKey() is implementable -- or at the
very least would lead to a *good* implementation.

I am happy to re-visit Chrome's implementation if we generate more
discussion around this issue.

Barring that if no UA implements it and the WG isn't able to respond to
feedback on it, I say we remove it from the spec.

>
> > We could test some collection of strings I imagine.
> >>
> >> - The tests are slow, and timeouts have been a problem. I think I've
> >> handed that, but the test pages take many minutes to run. I think the
> >> root problem is just that generating RSA keys is slow, and that there
> >> are a lot of combination to check (there are currently 1800 "success"
> >> tests and almost 30,000 "failure" tests).
>

Unfortunately this is going to be slow, and we can't really fix that :(

RSA key generation is extremely slow.

Your test surely wants to try a variety of key sizes, so sticking to small
key sizes (which would be on way to improve the performance) is not really
an option.

>
> > Could we just reuse the same key?
> >>
> >> I've only run these tests for Chrome and Firefox (and tried an
> >> earlier, less complete version for Edge) so far. The browsers do very
> >> well on the tests for successes, but have a fair number of issues on
> >> tests for failures. They fail where they should, but not necessarily
> >> by throwing the correct error. Even checking the correct error is
> >> problematic, because the browsers don't necessarily have the necessary
> >> DOMException property names defined in each case.
>

What property names are missing?


> >>
> >> Chrome's non-successes are for 192 AES keys, which it doesn't support.
> >> Firefox's non-successes are for not supporting RSA-PSS.
> > As long as two browsers support 192 keys, we can keep it in the spec.
>

Other things to be testing for generateKey() for RSA:

  * Supported key size
  * Supported public exponent.

We will see differences here -- Chrome has a minimum and maximum allowed
key size for RSA, and also restricts the public exponent to being 3 or
65537 (more details at https://www.chromium.org/blink/webcrypto).

Side comment: not sure if we would want this under the conformance test
suite, or leave it to implementation testing.. but when testing our own
implementation we had an early bug where generated keys were unextractable
(exportKey() would fail on them even when marked extractable). To catch
these sorts of problems you would ideally run all the operation tests on
each successfully generated key, and verify that all key usages are
respected, and that operations work as expected on the key.

>
> > I think they said the  added RSA-PSS into nightly build.
> >
> >
> >>
> >> Charlie
> >>
> >
> >
>
>