- From: Mark Skall <mark.skall@nist.gov>
- Date: Thu, 23 May 2002 22:00:46 -0400
- To: Alex Rousskov <rousskov@measurement-factory.com>, Karl Dubost <karl@w3.org>
- Cc: www-qa@w3.org
At 07:13 PM 05/23/2002 -0600, Alex Rousskov wrote: >Conformance >Conformance Testing > > 1. The definitions do not answer a fundamental > question: What exactly is sufficient to claim > (or certify) Conformance based on a test result? > For example, if my HTTP test suite has test > cases for all HTTP MUSTs, and the device under test > passes all those cases, is the device conformant? > In other words, does the absence of known violations > mean conformance?? > > It is a difficult but very important question to > answer. On one hand, if we cannot find any violations, > then we have no objective evidence that the device > violates the specs. On the other hand, it is impossible > to test all combinations of inputs so we may be missing > a test case that detects violations. Thus, we either > end up certifying non-conformant devices or refuse > to certify any device! > > While the above problem is mostly obvious for protocols, > I believe it does exist for specifications like XML and > for Conforming Documents: While the "input" in this case > is fixed (it is the document itself), the very program that > checks for conformance may be buggy. So, again, > we cannot say for sure that the document is conformant. > > I do not know how to solve this "uncertainty" problem, > but it needs a solution if we want to talk about > conformance tools and especially certification. As a > conformance test suite author, I need to know how my > tool can issue a "Conforms to the [W3C] Standard" > certificate using W3C definition of Conformance. > > I hope that the "uncertainty" problem has been solved > elsewhere, and we just need to pick the best solution. > Does anybody know of any good solutions? I can only think > of "Designed for MS Windows" and UL "safety" certification > that boils down to "if company X says you conform, then > you conform". Not very applicable to the W3C situation > where W3C develops a set of objective rules rather than > issues subjective conformance certificates. > > It is easy for the specification to say that a device > is conformant if it meets A, B, and C. We need to define > how one can test/prove/testify that the device meets A, B, > and C. Do we need a "[beyond] reasonable doubt" clause? > Ugh. > > Alex, You're confusing the definition of "conformance" with the concept of "proving conformance". The definition "The ability of a product to meet all the requirements claimed to be supported" is what conformance is. Proving that, in fact, this definition is met is a completely different issue. Falsification testing can never prove conformance, only non-conformance (if an error is found). In fact, when we advise bodies giving out certificates of conformance, we tell them to state that passing all the tests does not guarantee conformance. In fact, we say something like this in our QA guidelines. Of course, the more test cases you have the better the chances are that passing all the tests results in conformance. You can never solve this uncertainty problem. At NIST, we've actually looked at a way of using statistics to come up with a probability, that, in fact, you do conform if you've passed all the tests. Again, this is just a probability. It can never be a certainty. So, in summary, just because we can't prove conformance does not make its definition any less precise. I haven't looked closely at the rest of your comments. I will when I get a chance. Mark **************************************************************** Mark Skall Chief, Software Diagnostics and Conformance Testing Division Information Technology Laboratory National Institute of Standards and Technology (NIST) 100 Bureau Drive, Stop 8970 Gaithersburg, MD 20899-8970 Voice: 301-975-3262 Fax: 301-590-9174 Email: skall@nist.gov ****************************************************************
Received on Thursday, 23 May 2002 22:02:01 UTC