W3C home > Mailing lists > Public > www-qa-wg@w3.org > December 2003

Test Suites and Standards Process

From: Karl Dubost <karl@w3.org>
Date: Sat, 13 Dec 2003 17:39:44 -0500
Message-Id: <3FC42E29-2DBD-11D8-AED1-000A95718F82@w3.org>
To: www-qa-wg@w3.org

On Matt May's ***personal*** weblog

The last two paragraphs of the entry is interesting.

Farouk addresses the second question by submitting the Java Community 
Process, which requires conformance testing in order for specs to move 
forward. Philippe agrees, citing the W3C Candidate Recommendation 
process, where two implementations of each part of the specification 
must be found. Additionally, CR requires a test suite to ensure interop 
and conformance. Jamie says that OASIS requires groups to certify that 
they conform to a standard, but they don't require evidence of that 
conformance. "Frankly, it extends the process quite a bit." Michael 
interjects: "Conformance tests cannot prove conformance, because tests 
cannot prove correctness." You can only scientifically disprove 
something. (This is why I don't get into arguments with logicians.)

Eve Maler of Sun: "Once a specification starts to show some traction," 
it's important to have test suites, etc., but before then, it's a lot 
of work to impose on a group. Conformance clauses and testable 
assertions are sufficient before then. Mark Palmer adds that two 
interoperable implementations is the minimum, and OASIS's approach of 
implementability is insufficient. Jamie says that at the final stage, 
they actually require three implementations, but still don't require 
interop. WS-Security had "several big" interoperability tests, but it 
isn't documented in the process. Janet Daly, head of communications for 
W3C, notes that a publicly-viewable implementation report is required 
to exit Candidate Recommendation. Also, comments on the spec must have 
some disposition in public.

Karl Dubost - http://www.w3.org/People/karl/
W3C Conformance Manager
*** Be Strict To Be Cool ***
Received on Saturday, 13 December 2003 17:49:59 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:14:31 UTC