W3C home > Mailing lists > Public > www-qa@w3.org > May 2002

Re: QA Glossary - New version

From: Mark Skall <mark.skall@nist.gov>
Date: Thu, 23 May 2002 22:00:46 -0400
Message-Id: <5.1.0.14.2.20020523214728.00a48750@mailserver.nist.gov>
To: Alex Rousskov <rousskov@measurement-factory.com>, Karl Dubost <karl@w3.org>
Cc: www-qa@w3.org
At 07:13 PM 05/23/2002 -0600, Alex Rousskov wrote:

>Conformance
>Conformance Testing
>
>         1. The definitions do not answer a fundamental
>            question: What exactly is sufficient to claim
>            (or certify) Conformance based on a test result?
>            For example, if my HTTP test suite has test
>            cases for all HTTP MUSTs, and the device under test
>            passes all those cases, is the device conformant?
>            In other words, does the absence of known violations
>            mean conformance??
>
>            It is a difficult but very important question to
>            answer. On one hand, if we cannot find any violations,
>            then we have no objective evidence that the device
>            violates the specs. On the other hand, it is impossible
>            to test all combinations of inputs so we may be missing
>            a test case that detects violations. Thus, we either
>            end up certifying non-conformant devices or refuse
>            to certify any device!
>
>            While the above problem is mostly obvious for protocols,
>            I believe it does exist for specifications like XML and
>            for Conforming Documents: While the "input" in this case
>            is fixed (it is the document itself), the very program that
>            checks for conformance may be buggy. So, again,
>            we cannot say for sure that the document is conformant.
>
>            I do not know how to solve this "uncertainty" problem,
>            but it needs a solution if we want to talk about
>            conformance tools and especially certification. As a
>            conformance test suite author, I need to know how my
>            tool can issue a "Conforms to the [W3C] Standard"
>            certificate using W3C definition of Conformance.
>
>            I hope that the "uncertainty" problem has been solved
>            elsewhere, and we just need to pick the best solution.
>            Does anybody know of any good solutions? I can only think
>            of "Designed for MS Windows" and UL "safety" certification
>            that boils down to "if company X says you conform, then
>            you conform". Not very applicable to the W3C situation
>            where W3C develops a set of objective rules rather than
>            issues subjective conformance certificates.
>
>            It is easy for the specification to say that a device
>            is conformant if it meets A, B, and C. We need to define
>            how one can test/prove/testify that the device meets A, B,
>            and C. Do we need a "[beyond] reasonable doubt" clause?
>            Ugh.
>
>


Alex,

You're confusing the definition of "conformance" with the concept of 
"proving conformance".  The definition "The ability of a product to meet 
all the requirements claimed to be supported" is what conformance 
is.  Proving that, in fact, this definition is met is a completely 
different issue.  Falsification testing can never prove conformance, only 
non-conformance (if an error is found).  In fact, when we advise bodies 
giving out certificates of conformance, we tell them to state that passing 
all the tests does not guarantee conformance. In fact, we say something 
like this in our QA guidelines.  Of course, the more test cases you have 
the better the chances are that passing all the tests results in conformance.

You can never solve this uncertainty problem. At NIST, we've actually 
looked at a way of using statistics to come up with a probability, that, in 
fact, you do conform if you've passed all the tests.  Again, this is just a 
probability.  It can never be a certainty.

So, in summary, just because we can't prove conformance does not make its 
definition any less precise.

I haven't looked closely at the rest of your comments.  I will when I get a 
chance.

Mark



****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
****************************************************************
Received on Thursday, 23 May 2002 22:02:01 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:40:29 UTC