Re: [W3C Web Crypto WG] about tests

On Thu, Dec 18, 2014 at 9:20 AM, Boris Zbarsky <bzbarsky@mit.edu> wrote:
>
> On 12/18/14, 4:10 AM, Harry Halpin wrote:
>
>> What in your opinion is a *good* test suite?
>>
>
> One which has the property that an implementation passing it is very
> likely to be compliant with the specification.


An important consideration for the test structure will be how unsupported
operations are handled

For instance how would an implementation respond to the tests when:
  * It doesn't support a particular algorithm (Ex: Chromium's algorithm
support on Linux depends on the system's NSS library version)
  * Doesn't support a certain key size (Ex: Chromium doesn't support 192
bit AES keys)
  * Doesn't support a particular key format (Ex: WebKit doesn't support
"spki" and "pkcs8" key formats)


>  There's all sorts of levels of complexity one can do with testing, and I
>> think how much testing we do depends on how much the WG can do and if
>> W3C can be successful in more resources.
>>
>
> The right way to do testing is probably to set up a way for people to
> contribute tests and then get them to do that.  That's what the web
> platform tests effort is about, right?
>
> In particular, getting implementors to contribute their existing test
> suites seems like a good start.


Agreed.

I think good next steps are:
  (1) Set up a repository for the test collaboration that is easy for
others to contribute to (github)
  (2) Copy over the W3C testing framework
  (3) Write 2-3 sample tests (needn't be perfect, but give some guidelines
on the format)
  (4) Implementors and volunteers start converting their test suite by
mimicking the sample tests

As far as which tests to include in the test suite, I suggest prioritizing
as:

(1) Correct usages of the API when called using supported algorithms (with
common parameters)
       Examples:
           - Does SHA-1 digest yield the expected result for known inputs
           - Does RSASSA-PKCS1-v1_5 verify correct signatures, and fail
verifying incorrect signatures
           - Can you successfully generate an RSA key with a public
exponent of 3
           - AES-CBC decryption throws an OperationError when the padding
is incorrect.

(2) Incorrect usages of the API which pose a security risk
       Examples:
          - Cannot export an un-extractable key
          - Cannot wrap an un-extractable key
          - When unwrapping a key from JWK, the "alg", "use", "key_ops",
"ext" are respected
          - Cannot use the same CryptoKey between algorithms
          - Cannot use a CryptoKey that lacks the required KeyUsages

(3) Correct usages of the API with more exotic parameters:
       Examples:
          - Different cased algorithm names
          - Using string-based AlgorithmIdentifiers instead of an object
          - Different combinations for optional parameters
          - Different combinations of BufferSource's type used in verify()
          - Including unrecognized "key_ops" in a JWK is disregarded

(4) The myriad of other error cases
       Examples:
          - Import an empty HMAC key
          - Import an HMAC key with a shorter optional length than provided
key data
          - Import an RSA key in JWK format using base64 encoding rather
than base64url encoding
          - Passed null in place of an "unsigned long" for a required
dictionary field
          - Specified an invalid key usage during creation
          - Generated an asymmetric key pair without specifying any private
key usages
          - Generated an asymmetric key pair containing key usages that
were not applicable to the algorithm
          - Case sensitivity for JWK's "key_ops" and "alg" members
          - AES-GCM called with a tagLength of 13
          - AES-CTR called with an IV of length 3
          - Duplicate key_ops values in a JWK fails import (including
unrecognized ones)
          - When importing an EC key from JWK, throws an error if "x", "y",
or "d" are not zero-padded to the right length.
          - When importing an RSA from JWK, throws an error if any of the
parameters DO contain a leading zero (yay consistency?)
          ... and many more such things