Re: [W3C Web Crypto WG] about tests

On 01/13/2015 04:00 PM, GALINDO Virginie wrote:
> Guys,
> 2015 is here !
> How do we progress on test ?
> Who's volunteering to help for migrating test on github with Harry ?

I'm working with Robin to set up the integrated testing of WebCrypto
with the rest of the Open Web Platform. I'll have an update in a few days.

Nonetheless, what we really need socially from the WG is someone to
volunteer to take leadership on the testing document.

On another note, we'd also love to have some funding support if any W3C
members have any ideas there in order to Testthewebforward.org events or
finish updating the previously externally-funded HTML5 testing framework
to handle WebCrypto.

   cheers,
       harry

> Regards,
> Virginie
> 
> -----Original Message-----
> From: Harry Halpin [mailto:hhalpin@w3.org]
> Sent: jeudi 18 décembre 2014 20:28
> To: public-webcrypto@w3.org
> Subject: Re: [W3C Web Crypto WG] about tests
> 
> On 12/18/2014 08:22 PM, Eric Roman wrote:
>> On Thu, Dec 18, 2014 at 9:20 AM, Boris Zbarsky <bzbarsky@mit.edu> wrote:
>>>
>>> On 12/18/14, 4:10 AM, Harry Halpin wrote:
>>>
>>>> What in your opinion is a *good* test suite?
>>>>
>>>
>>> One which has the property that an implementation passing it is very
>>> likely to be compliant with the specification.
>>
>>
>> An important consideration for the test structure will be how
>> unsupported operations are handled
>>
>> For instance how would an implementation respond to the tests when:
>>   * It doesn't support a particular algorithm (Ex: Chromium's
>> algorithm support on Linux depends on the system's NSS library version)
>>   * Doesn't support a certain key size (Ex: Chromium doesn't support
>> 192 bit AES keys)
>>   * Doesn't support a particular key format (Ex: WebKit doesn't
>> support "spki" and "pkcs8" key formats)
>>
>>
>>>  There's all sorts of levels of complexity one can do with testing,
>>> and I
>>>> think how much testing we do depends on how much the WG can do and
>>>> if W3C can be successful in more resources.
>>>>
>>>
>>> The right way to do testing is probably to set up a way for people to
>>> contribute tests and then get them to do that.  That's what the web
>>> platform tests effort is about, right?
>>>
>>> In particular, getting implementors to contribute their existing test
>>> suites seems like a good start.
>>
>>
>> Agreed.
>>
>> I think good next steps are:
>>   (1) Set up a repository for the test collaboration that is easy for
>> others to contribute to (github)
>>   (2) Copy over the W3C testing framework
>>   (3) Write 2-3 sample tests (needn't be perfect, but give some
>> guidelines on the format)
>>   (4) Implementors and volunteers start converting their test suite by
>> mimicking the sample tests
> 
> This all sounds like a great plan, and I'll work on setting this up over at Github. Most people will be going on some sort of winter vacation next week, but I'd like to begin 2015 with a flurry of testing to make sure interop on WebCrypto is rock-solid.
> 
>>
>> As far as which tests to include in the test suite, I suggest
>> prioritizing
>> as:
>>
>> (1) Correct usages of the API when called using supported algorithms
>> (with common parameters)
>>        Examples:
>>            - Does SHA-1 digest yield the expected result for known inputs
>>            - Does RSASSA-PKCS1-v1_5 verify correct signatures, and
>> fail verifying incorrect signatures
>>            - Can you successfully generate an RSA key with a public
>> exponent of 3
>>            - AES-CBC decryption throws an OperationError when the
>> padding is incorrect.
>>
>> (2) Incorrect usages of the API which pose a security risk
>>        Examples:
>>           - Cannot export an un-extractable key
>>           - Cannot wrap an un-extractable key
>>           - When unwrapping a key from JWK, the "alg", "use",
>> "key_ops", "ext" are respected
>>           - Cannot use the same CryptoKey between algorithms
>>           - Cannot use a CryptoKey that lacks the required KeyUsages
>>
>> (3) Correct usages of the API with more exotic parameters:
>>        Examples:
>>           - Different cased algorithm names
>>           - Using string-based AlgorithmIdentifiers instead of an object
>>           - Different combinations for optional parameters
>>           - Different combinations of BufferSource's type used in verify()
>>           - Including unrecognized "key_ops" in a JWK is disregarded
>>
>> (4) The myriad of other error cases
>>        Examples:
>>           - Import an empty HMAC key
>>           - Import an HMAC key with a shorter optional length than
>> provided key data
>>           - Import an RSA key in JWK format using base64 encoding
>> rather than base64url encoding
>>           - Passed null in place of an "unsigned long" for a required
>> dictionary field
>>           - Specified an invalid key usage during creation
>>           - Generated an asymmetric key pair without specifying any
>> private key usages
>>           - Generated an asymmetric key pair containing key usages
>> that were not applicable to the algorithm
>>           - Case sensitivity for JWK's "key_ops" and "alg" members
>>           - AES-GCM called with a tagLength of 13
>>           - AES-CTR called with an IV of length 3
>>           - Duplicate key_ops values in a JWK fails import (including
>> unrecognized ones)
>>           - When importing an EC key from JWK, throws an error if "x",
>> "y", or "d" are not zero-padded to the right length.
>>           - When importing an RSA from JWK, throws an error if any of
>> the parameters DO contain a leading zero (yay consistency?)
>>           ... and many more such things
>>
> 
> ________________________________
>  This message and any attachments are intended solely for the addressees and may contain confidential information. Any unauthorized use or disclosure, either whole or partial, is prohibited.
> E-mails are susceptible to alteration. Our company shall not be liable for the message if altered, changed or falsified. If you are not the intended recipient of this message, please delete it and notify the sender.
> Although all reasonable efforts have been made to keep this transmission free from viruses, the sender will not be liable for damages caused by a transmitted virus.
>

Received on Tuesday, 13 January 2015 19:32:36 UTC