Re: [websocket-test] What's the plan for reviewing and approving Opera's WebSocket tests?

On 02/25/2013 06:37 PM, Ken Reed wrote:
> Hi James,
>
> We tasked a capable and experienced engineering team with designing
> and reviewing our test approach and then authoring the resulting test
> cases. I imagine this is not dissimilar to how Mozilla and Opera
> approached their contributions.

Sure. But I wouldn't consider our submission to be complete on this 
basis, and I certainly wouldn't expect you to without evidence to back 
up that assertion.

In this case, it is possible to find submitted tests that fail in IE 10 
but pass in other browsers; [1] is such a test. On this basis I think it 
is unlikely that this is adequately checked in the "approved" testsuite. 
However it was difficult for me to verify this as the approved directory 
was missing a critical file (websocket.js). Even with that added, most 
of the tests seem to fail with "Not Run" errors. I suppose the jetty 
server is down and no one has noticed.

On the basis of this, it is clear that we shouldn't consider these tests 
sufficient for testing websockets. More worryingly, we seem to be taking 
our eyes off the goal here; our objective should not be to rubberstamp 
the spec with the minimum amount of fuss but to improve interoperability 
of implementations. If the tests are mostly broken — strongly suggesting 
they are not being run — and large numbers of submitted tests are being 
cast aside for want of some slight syntactic changes to work with 
testharness.js, we are clearly failing at this goal.

For my part I have restarted work on fixing up the Opera submissions. I 
can't promise a timescale for completing the work because it isn't part 
of my day job. With all the talk about getting organisations to pay for 
testsuites, it would be nice if this kind of relatively trivial work — 
it doesn't require deep understanding of the spec — was one of the first 
things to be funded. It has a much lower cost/benefit ratio than trying 
to write greenfield test cases.

[1] 
http://w3c-test.org/webapps/WebSockets/tests/submissions/Opera/constructor/004.html
* I would have verified this, but the testsuite seems to be broken; all 
tests I tried returned "Not Run". Also we were missing a file from the 
approved directory.

Received on Wednesday, 27 February 2013 08:53:40 UTC