RE: [websocket-test] What's the plan for reviewing and approving Opera's WebSocket tests?

> We tasked a capable and experienced engineering team with designing 
> and reviewing our test approach and then authoring the resulting test 
> cases. I imagine this is not dissimilar to how Mozilla and Opera 
> approached their contributions.

Sure. But I wouldn't consider our submission to be complete on this basis, and I certainly wouldn't expect you to without evidence to back up that assertion.

[K>O>R>] okay. I think I understand where you're coming from. Besides the competent engineers making quality decisions aspect our tests did find a few bugs which were fixed and the tests were actually passing (more on that below). Upon these results we concluded that our tests were in fact assessing correct implementation.

In this case, it is possible to find submitted tests that fail in IE 10 but pass in other browsers; [1] is such a test. On this basis I think it is unlikely that this is adequately checked in the "approved" testsuite. 
However it was difficult for me to verify this as the approved directory was missing a critical file (websocket.js). Even with that added, most of the tests seem to fail with "Not Run" errors. I suppose the jetty server is down and no one has noticed.

 [K>O>R>] well that is quite disappointing. Websocket.JS used to exist and the tests used to pass. I suspect you are correct that some aspect of the infrastructure has failed.

On the basis of this, it is clear that we shouldn't consider these tests sufficient for testing websockets. More worryingly, we seem to be taking our eyes off the goal here; our objective should not be to rubberstamp the spec with the minimum amount of fuss but to improve interoperability of implementations. If the tests are mostly broken - strongly suggesting they are not being run - and large numbers of submitted tests are being cast aside for want of some slight syntactic changes to work with testharness.js, we are clearly failing at this goal.

[K>O>R>]  the objective certainly is not to rubberstamp the spec. Months ago we had an approved, working set of tests. Now, after having spent several months in limbo hoping Opera would complete their test case submission it seems some aspect of the infrastructure has degraded and we now have to spend additional cycles fixing that. If anything, patiently waiting for the Opera submission has retarded our progress towards establishing interoperability and continuing to make progress on ratifying the spec.

For my part I have restarted work on fixing up the Opera submissions. I can't promise a timescale for completing the work because it isn't part of my day job. With all the talk about getting organisations to pay for testsuites, it would be nice if this kind of relatively trivial work - it doesn't require deep understanding of the spec - was one of the first things to be funded. It has a much lower cost/benefit ratio than trying to write greenfield test cases.

[K>O>R>] that's great, but I really think Opera needs to commit to a timeline to have their submission ready for the group to consider including in the approved set of tests. I suspect we all are very busy with the day job but we really cannot continue to use that as an excuse to leave WebSockets in an indeterminate state.

[1]
http://w3c-test.org/webapps/WebSockets/tests/submissions/Opera/constructor/004.html
* I would have verified this, but the testsuite seems to be broken; all tests I tried returned "Not Run". Also we were missing a file from the approved directory.

Received on Wednesday, 27 February 2013 22:06:19 UTC