W3C home > Mailing lists > Public > www-qa@w3.org > April 2006

Re: Testability and normative requirements

From: Karl Dubost <karl@w3.org>
Date: Tue, 11 Apr 2006 15:51:52 +0900
Message-Id: <773C4373-2E45-4F4F-A436-79B5D45F0330@w3.org>
Cc: "Mark W. Skall" <mark.skall@nist.gov>
To: www-qa@w3.org

(Mark, a question for you in the text)

Le 06-03-08 à 23:40, Dominique Hazael-Massieux a écrit :
> As discussed during the IG F2F [1], I started a Wiki page on this  
> topic:
> http://esw.w3.org/topic/NormativeButNotTestable

And the text is for now
When writing a specificiation, it is sometimes tempting to use non- 
testable text for normative requirements; indeed, making sure a text  
is testable requires much more work than using simple text without  
caring about testability (see also TestableOrNot).

But using non testable text as normative requirements has many  

     1. mandating something that cannot be tested is a no-op; how can  
you check whether something was indeed implemented if you cannot test  

     2. non-testable requirements means that implementations are  
likely to differ on the said requirement, meaning that  
interoperability will be loose at best

     3. leaving a requirement in a fuzzy non testable state means  
leaving the disambiguation work to the implementors, making it much  
more costly and much more likely to generate confusion for the end users
-- Normative But Not Testable - ESW Wiki
Tue, 11 Apr 2006 06:02:39 GMT

Dom has requested for example of non testable requirements.

Provided by Ian Hickson
  [...] user agents must make a best attempt to render all characters,
  regardless of the value specified by lang.
  -- HTML4 section 8.1.

  Those browsers that interpret soft hyphens must observe the following
  semantics: If a line is broken at a soft hyphen, a hyphen character
  must be displayed at the end of the first line. If a line is not
  broken at a soft hyphen, the user agent must not display a hyphen
  -- HTML4 section 9.3.3.

  User agents must know where to render the header and footer.
  -- HTML4 section 11.2.1.

-- Re: Testability and normative requirements from Ian Hickson on  
2005-12-21 (www-qa@w3.org from December 2005)
Thu, 22 Dec 2005 20:40:48 GMT

I think there's a missing part in the wiki text for now to give a bit  
more context:

What is the meaning of "normative"? Why do I ask something which  
seems obvious?

    * A normative requirement is a requirement defined by one of the  
RFC2119 keywords.
    * A normative requirement is a requirement defined by MUST keywords.

(Here I have chosen RFC 2119 for simplicity but it could be any kind  
of requirements rules defined in the Conformance section of a  

RFC 2119 uses:
	- absolute requirement (MUST, REQUIRED, SHALL)
	- absolute prohibition (MUST NOT, SHALL NOT)
	- particular item      (SHOULD, RECOMMENDED)
         - particular behaviour (SHOULD NOT, NOT RECOMMENDED)
         - item                 (MAY)

which shows btw that RFC 2119 is not really consistent. I remember  
that at the WWW2002 Conference, Mark Skall had presented a paper  
about the problems of RFC 2119.

Mark, do you still have this paper and could you send the text in a  
mail on this list?

Ok going (slowly) to my point. IMHO, a non testable requirement  
doesn't have the same implications if it's a
	- MUST-non_testable-requirement
	- SHOULD-non_testable-requirement
except if we consider that a SHOULD is not a requirement, but I don't  
believe so. It has other implications.

So a thumb rule which comes to my mind is

	* If your requirement is not testable, make it optional.

 From the Mailing List Front
"Non testable":   10 occurences

"Not testable":  217 occurences

I think it is related to another issue:
	Meaning versus Behavior

Karl Dubost - http://www.w3.org/People/karl/
W3C Conformance Manager, QA Activity Lead
   QA Weblog - http://www.w3.org/QA/
      *** Be Strict To Be Cool ***
Received on Tuesday, 11 April 2006 06:51:59 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:40:37 UTC