- From: Alex Rousskov <rousskov@measurement-factory.com>
- Date: Fri, 6 Jun 2003 09:27:00 -0600 (MDT)
- To: Lynne Rosenthal <lynne.rosenthal@nist.gov>
- cc: www-qa@w3.org
On Fri, 6 Jun 2003, Lynne Rosenthal wrote: > Should specification developers be required to include Test > Assertions (TA) in their specifications? No. The goal of a specification is to document and explain format and/or behavior, not to test a format or behavior of an implementation. > some Pros: > 1. ensures testable specifications It does not. It decreases the number of untestable requirements due to poor wording and such. That side-effect can be achieved directly by other means (see last paragraph of this e-mail). > 2. consensus in what is required Not at all. Consensus on how each testable requirement can be tested at a minimal level. This usually boils down to a simple example illustrating a requirement. > 3. facilitates test generation True. However, it may facilitate generation of both inferior implementations and inferior tests because: - Spec authors usually do not care about test cases much so they are not going to spend much time developing decent TAs and they will have just the minimum set of cases to cover all testable statements once. - Implementors with little resources may focus on satisfying (coding for) documented TAs instead of satisfying the specification itself. A very common pitfall, IMO. - Testers will have to implement documented TAs and may stop there, instead of developing a more complete and independent test suite. By forcing spec authors to do testing we assign an important task to the wrong people. Look, for example, at test cases for SOAP at http://www.w3.org/TR/soap12-testcollection/ I am not a SOAP expert, but these look a lot more like examples then quality test cases. A decent test suite would do much better than that. > IMO specifications should (must?) identify the testable statements > or test requirements. Specifications MUST define conformance criteria. Identifying testable part of that criteria would be useful, but since "testable" is not easy or even possible to define in general, we may not be able to require such identification. > Lets think about the objective - identifying the testable statements > and facilitate the building of test suites - without going overboard > and burdening the specification developer. So, what would be > approparite? Authors MUST identify conformance-affecting statements (e.g., by using MUST/SHOULD keywords). Authors MUST consider testability of each identified conformance-affecting statement. The last requirement is not testable. HTH, Alex. -- | HTTP performance - Web Polygraph benchmark www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite | all of the above - PolyBox appliance
Received on Friday, 6 June 2003 11:27:10 UTC