Next Steps on Testing

Basically, every part of the spec needs to be tested to get out of
Candidate Recommendation. This means that all parts of the spec that are
normative should have the same amount of coverage,and we have an
additional constraint (rather sensible) imposed that each algorithm
needs to be tested, even though they are non-normative, due to the issue
of the 'browser profile'.

Tests are contributed via pull requests on Github. We have a relatively
lonely WebCryptoAPI section:

https://github.com/w3c/web-platform-tests/commits/master/WebCryptoAPI

With the mysterious ms2ger starting to test.

The test approval process generally requires a few reviewers to make
sure the test fits. See here for details:

http://testthewebforward.org/docs/review-process.html

This is done using the Critic software ran by James Graham that
integrates well with Github. For an example, see here:

https://critic.hoppipolla.co.uk/r/3900

Unless someone steps up, I'll work on auto-converting every single
WebIDL into a test to populate the initial WebCryptoAPI section next
week. Once I do that, I recommend that we have an tutorial telecon later
this month to overview the minimum requirements for passing CR and how
to convert the tests over.

 Mozilla's tests are the closest to the W3C test format for
testharness.js [1].

That should at least give us an imperfect skeleton to start testing on.
 I'll also take charge of making a final HTML document that demonstrates
the interoperability across browsers so we can exit Candidate
Recommendation so people don't have to worry about W3C Process but can
just get on with making/converting some tests.

There is extensive documentation on TestTheWebForward
(http://testthewebforward.org) and on
Github.(https://github.com/w3c/web-platform-tests/)

We are still looking for a test leader for WebCrypto!

If you know of anyone that would donate funding for a professional test
manager, we have a former staff member who is in need of funding with
extensive experience in the HTML test-suite that would be happy to lead
our test-suite.

Generally, if we have a full-time testing staff ASAP, we estimate we can
exit with a world-class test-suite within a month or two, close to
estimated exit date given in CR.

If we do not, the process will of course take longer than our estimated
deadline, which we'll have to take into account in our re-chartering.
It's hard to estimate now, but if we start testing  by end of this
month, we'll get a good guess by looking at our progress by the time we
need to recharter.

   cheers,
         harry

[1]
http://darobin.github.io/test-harness-tutorial/docs/using-testharness.html

Received on Wednesday, 4 February 2015 01:28:58 UTC