Follow up on Test Jam

My apologies that this has sat on my to-do list for a week - thanks to Gopal for the summary, and let's keep the momentum going.


====begin message from gopal.raghavan@nokia.com<mailto:gopal.raghavan@nokia.com>============

**************************************************************
Summary of testJam on May 2 and 3, 2010.
------------------------------------------------------------

First of all thanks to Brad Hill , Odin Horthe Omdal, Tanvi Vyas, Adam Barth, Peleus Uhley and kris , for committing test cases during testJam.

As part of transition from CR to proposed recommendation, we need test cases that cover most of the features of the specification.

Brad provided VM during testJam on DVD, which has all the environment to get started with writing test cases.
You can also download it from http://dl.dropbox.com/u/76057758/WebAppSecTestsuite.vdi.bz2

The username/pass is webappsec/webappsec

Here is the test repository https://dvcs.w3.org/hg/webappsec.
After you clone the repository, checkout testJam branch. (hg update testJam)

For CORS, we should have positive and negative tests cases for every claim under specification section 5, 6 and 7.
Similarly, CSP needs positive and negative tests for all directives specified in section 3.1 and 4.

Fortunately, we have lot of test cases for CORS under cors/submitted/webkit and cors/submitted/opera.
The goal is to consolidate all the tests under cors/submitted/cors1.0 and csp/submitted/csp1.0.

We have also installed testRunner under /var/www.  If you don't have it in your VM, please let me know I can help you set it up.
If you add your test case to cors/submitted/cors1.0/MANIFEST, it will get picked up by testRunner.

You can run auto-tests by navigating to http://www.w3c-test.org/testRunner/index.html inside your VM.

Currently, we have two buttons "Run CORS 1.0 tests" and "Run CORS Opera tests"
When you click on one of these buttons, it will kick off automated tests.

I just ran the tests under Firefox 12.0

CORS 1.0 test results:
Score: 100.00%
Pass: 5
Fail: 0

CORS Opera test results:
Score: 9.28%
Pass: 18
Fail: 176
(I am sure Odin can get much better score under Opera browser)

As you can see there is lot of work to be done.

I would strongly encourage everyone to start contributing to the test suite.

How you can contribute:
1.            Port webkit cgi scripts to php. W3C recommends only php scripts.
2.            Modify webkit tests to make it executable. (See examples under cors/submitted/cors1.0 )
3.            Contribute new tests to cors or csp
4.            Help someone setup the test VM and test framework
5.            Periodically run testRunner and keep track of coverage
6.            Maintain the test repository clean and help out with merge issues
Best Regards,
--
Gopal

Meeting minutes captures some info about the testJam
[1] http://lists.w3.org/Archives/Public/www-archive/2012May/att-0011/minutes-2012-05-02.html
[4] http://lists.w3.org/Archives/Public/www-archive/2012May/att-0011/minutes-2012-05-03.html