- From: Mark Skall <mark.skall@nist.gov>
- Date: Fri, 09 May 2003 16:15:13 -0400
- To: Alex Rousskov <rousskov@measurement-factory.com>
- Cc: Lofton Henderson <lofton@rockynet.com>, www-qa@w3.org
At 01:27 PM 5/9/2003 -0600, Alex Rousskov wrote: >Now, let's see if you can give an example of a UNtestable requirement. >I bet I will be able to use your own logic to show that that >requirement is testable. Actually, I like your example. The requirement that all requirements MUST be testable is untestable. One cannot write a test suite without knowing the exact requirement. > > So vendors will say they conform but we can't determine if they're > > telling the truth? > >Yes. That's the informal world we leave in. It is impractical to >change that, even in the virtual world, unless all specs and all >implementations are based on some formal verifiable model. > >If you disagree, you have to prove, among many other things, that your >test tool is _always_ correct. What if your test tool is lying? Why >should I trust the vendor less than you? Because the vendor is biased (This is not necessarily a bad thing - it's their business to promote their product). The ones who designed the test are neutral (or at least they should be). >Are we going to talk about >vendor incentives to lie versus test lab incentives to lie? Is it >above motive? Yes. > > Again, it may be possible to implement correctly, but it still is > > impossible to know if it has been. > >True, but we have to live with that uncertainty. It is not possible to >eliminate it unless we formalize the entire virtual environment. See >above. > > > Again, we can usually come up with some test, no matter how > > inefficient. > >The "no matter how inefficient" part makes all requirements testable. >Does it not? No. See my example above. However, it makes the very large majority testable. That's why it's so appalling when a requirement is untestable. With a little bit of work, I believe, it could be made testable. We've just agreed how easy that is. > > Since compliance is the ultimate goal, every requirement needs to be > > testable to ensure compliance. > >Compliance is the ultimate goal. Being able to ensure (test for) >compliance is not the ultimate goal; it is only a highly desirable >feature (i.e., a SHOULD). > >Thanks, > >Alex. **************************************************************** Mark Skall Chief, Software Diagnostics and Conformance Testing Division Information Technology Laboratory National Institute of Standards and Technology (NIST) 100 Bureau Drive, Stop 8970 Gaithersburg, MD 20899-8970 Voice: 301-975-3262 Fax: 301-590-9174 Email: skall@nist.gov ****************************************************************
Received on Friday, 9 May 2003 17:45:33 UTC