Re: [W3C Web Crypto WG] about tests

On 12/18/2014 08:22 PM, Eric Roman wrote:
> On Thu, Dec 18, 2014 at 9:20 AM, Boris Zbarsky <bzbarsky@mit.edu> wrote:
>>
>> On 12/18/14, 4:10 AM, Harry Halpin wrote:
>>
>>> What in your opinion is a *good* test suite?
>>>
>>
>> One which has the property that an implementation passing it is very
>> likely to be compliant with the specification.
> 
> 
> An important consideration for the test structure will be how unsupported
> operations are handled
> 
> For instance how would an implementation respond to the tests when:
>   * It doesn't support a particular algorithm (Ex: Chromium's algorithm
> support on Linux depends on the system's NSS library version)
>   * Doesn't support a certain key size (Ex: Chromium doesn't support 192
> bit AES keys)
>   * Doesn't support a particular key format (Ex: WebKit doesn't support
> "spki" and "pkcs8" key formats)
> 
> 
>>  There's all sorts of levels of complexity one can do with testing, and I
>>> think how much testing we do depends on how much the WG can do and if
>>> W3C can be successful in more resources.
>>>
>>
>> The right way to do testing is probably to set up a way for people to
>> contribute tests and then get them to do that.  That's what the web
>> platform tests effort is about, right?
>>
>> In particular, getting implementors to contribute their existing test
>> suites seems like a good start.
> 
> 
> Agreed.
> 
> I think good next steps are:
>   (1) Set up a repository for the test collaboration that is easy for
> others to contribute to (github)
>   (2) Copy over the W3C testing framework
>   (3) Write 2-3 sample tests (needn't be perfect, but give some guidelines
> on the format)
>   (4) Implementors and volunteers start converting their test suite by
> mimicking the sample tests

This all sounds like a great plan, and I'll work on setting this up over
at Github. Most people will be going on some sort of winter vacation
next week, but I'd like to begin 2015 with a flurry of testing to make
sure interop on WebCrypto is rock-solid.

> 
> As far as which tests to include in the test suite, I suggest prioritizing
> as:
> 
> (1) Correct usages of the API when called using supported algorithms (with
> common parameters)
>        Examples:
>            - Does SHA-1 digest yield the expected result for known inputs
>            - Does RSASSA-PKCS1-v1_5 verify correct signatures, and fail
> verifying incorrect signatures
>            - Can you successfully generate an RSA key with a public
> exponent of 3
>            - AES-CBC decryption throws an OperationError when the padding
> is incorrect.
> 
> (2) Incorrect usages of the API which pose a security risk
>        Examples:
>           - Cannot export an un-extractable key
>           - Cannot wrap an un-extractable key
>           - When unwrapping a key from JWK, the "alg", "use", "key_ops",
> "ext" are respected
>           - Cannot use the same CryptoKey between algorithms
>           - Cannot use a CryptoKey that lacks the required KeyUsages
> 
> (3) Correct usages of the API with more exotic parameters:
>        Examples:
>           - Different cased algorithm names
>           - Using string-based AlgorithmIdentifiers instead of an object
>           - Different combinations for optional parameters
>           - Different combinations of BufferSource's type used in verify()
>           - Including unrecognized "key_ops" in a JWK is disregarded
> 
> (4) The myriad of other error cases
>        Examples:
>           - Import an empty HMAC key
>           - Import an HMAC key with a shorter optional length than provided
> key data
>           - Import an RSA key in JWK format using base64 encoding rather
> than base64url encoding
>           - Passed null in place of an "unsigned long" for a required
> dictionary field
>           - Specified an invalid key usage during creation
>           - Generated an asymmetric key pair without specifying any private
> key usages
>           - Generated an asymmetric key pair containing key usages that
> were not applicable to the algorithm
>           - Case sensitivity for JWK's "key_ops" and "alg" members
>           - AES-GCM called with a tagLength of 13
>           - AES-CTR called with an IV of length 3
>           - Duplicate key_ops values in a JWK fails import (including
> unrecognized ones)
>           - When importing an EC key from JWK, throws an error if "x", "y",
> or "d" are not zero-padded to the right length.
>           - When importing an RSA from JWK, throws an error if any of the
> parameters DO contain a leading zero (yay consistency?)
>           ... and many more such things
>

Received on Thursday, 18 December 2014 19:28:31 UTC