Re: Testing progress

Eric Roman wrote:
> 
> 
> On Mon, May 2, 2016 at 11:10 AM, Charles Engelke <w3c@engelke.com
> <mailto:w3c@engelke.com>> wrote:
> 
>     Re: key reuse. We're testing that we can generate a key, so it's going
>     through that slow, expensive, process every time the parameters are
>     good enough to get that far in the algorithm.
> 
>     Now that I'm fairly far along, I want to see if there's use case
>     trimming that can cut that way down without reducing the actual scope
>     of test coverage.
> 
>     Charlie
> 
>     On Mon, May 2, 2016 at 2:00 PM, Harry Halpin <hhalpin@w3.org
>     <mailto:hhalpin@w3.org>> wrote:
>     >
>     >
>     > On 05/02/2016 01:56 PM, Charles Engelke wrote:
>     >> I'll be on the call today. Here's a quick update on work I've been
>     >> doing on testing.
>     >>
>     >> My work in progress is at my fork of the Test the Web Forward
>     >> repository:
>     https://github.com/engelke/web-platform-tests/tree/master/WebCryptoAPI.
>     >
>     > Thanks so much for the update! This is great work, exactly what the
>     > Working Group needs.
>     >>
>     >> I'm focusing on a single method at first: generateKey. I'm first
>     >> testing that every possible combination of proper parameters
>     creates a
>     >> valid key with the correct properties, then that lots of possible
>     >> combinations of invalid parameters throws an error, AND that it
>     throws
>     >> the correct error per the spec.
>     >>
>     >> Issues:
>     >>
>     >> - No test coverage of RSA-OAEP, ECDSA, or ECDH yet. I don't see any
>     >> particular problems adding them, I'm just not that far along yet.
>     >>
>     >> - Can't automate testing of PBKDF2 because it requires user input
>     >> outside control of the browser.
> 
> 
> RE: generateKey() for PBKDF2:
> 
> Aside from the problem with automation, there is a bigger issue -- does
> any User Agent actually implement generateKey() for PBKF2?
> 
> Chrome certainly doesn't. 
> And as far as I can tell neither does Firefox
> (I can only speak authoritatively for Chromium's implementation).
> 
> I raised my concerns with PBKDF2's specification in this bug:
> https://www.w3.org/Bugs/Public/show_bug.cgi?id=27689
> 
> I don't believe as specified generateKey() is implementable -- or at the
> very least would lead to a *good* implementation.
> 
> I am happy to re-visit Chrome's implementation if we generate more
> discussion around this issue.
> 
> Barring that if no UA implements it and the WG isn't able to respond to
> feedback on it, I say we remove it from the spec.

+1. We have no current plans to implement it. It's heavily
underspecified, and can't possibly work for (Service)Workers.

- Tim


>     > We could test some collection of strings I imagine.
>     >>
>     >> - The tests are slow, and timeouts have been a problem. I think I've
>     >> handed that, but the test pages take many minutes to run. I think the
>     >> root problem is just that generating RSA keys is slow, and that there
>     >> are a lot of combination to check (there are currently 1800 "success"
>     >> tests and almost 30,000 "failure" tests).
> 
> 
> Unfortunately this is going to be slow, and we can't really fix that :(
> 
> RSA key generation is extremely slow.
> 
> Your test surely wants to try a variety of key sizes, so sticking to
> small key sizes (which would be on way to improve the performance) is
> not really an option.
> 
>     >
>     > Could we just reuse the same key?
>     >>
>     >> I've only run these tests for Chrome and Firefox (and tried an
>     >> earlier, less complete version for Edge) so far. The browsers do very
>     >> well on the tests for successes, but have a fair number of issues on
>     >> tests for failures. They fail where they should, but not necessarily
>     >> by throwing the correct error. Even checking the correct error is
>     >> problematic, because the browsers don't necessarily have the
>     necessary
>     >> DOMException property names defined in each case.
> 
> 
> What property names are missing?
>  
> 
>     >>
>     >> Chrome's non-successes are for 192 AES keys, which it doesn't
>     support.
>     >> Firefox's non-successes are for not supporting RSA-PSS.
>     > As long as two browsers support 192 keys, we can keep it in the spec.
> 
> 
> Other things to be testing for generateKey() for RSA:
> 
>   * Supported key size
>   * Supported public exponent.
> 
> We will see differences here -- Chrome has a minimum and maximum allowed
> key size for RSA, and also restricts the public exponent to being 3 or
> 65537 (more details at https://www.chromium.org/blink/webcrypto).
> 
> Side comment: not sure if we would want this under the conformance test
> suite, or leave it to implementation testing.. but when testing our own
> implementation we had an early bug where generated keys were
> unextractable (exportKey() would fail on them even when marked
> extractable). To catch these sorts of problems you would ideally run all
> the operation tests on each successfully generated key, and verify that
> all key usages are respected, and that operations work as expected on
> the key.
> 
>     >
>     > I think they said the  added RSA-PSS into nightly build.
>     >
>     >
>     >>
>     >> Charlie
>     >>
>     >
>     >
> 
>

Received on Monday, 2 May 2016 19:10:19 UTC