Re: Testable assertion tagging for W3C specifications

On Mon, 6 May 2002, Alex Rousskov wrote:

  There are many ways to "enable our specifications to be tested". And
  we should not forget that ability to test is not the only goal.

Absolutely. But not being able to test whether a specification is being met
means that it is less a specification than a general description of an idea.

  An ideal specifications would be both readable by humans and testable
  by machines. Most current specifications, especially complex and
  interesting ones, do not satisfy both of these properties. There can
  be two general directions for improving the situation:
  	- making humans write machine-testable text (code)
  	- making machines understand human-oriented text

  Some steps can be done in both directions. For example, making every
  bit of a spec addressable is not much extra work for humans and helps
  machines a lot.

EARL takes this approach - it simply requires that there is an identifiable
test - i.e. it can be addressed, and the test itself is assumed to be
specified in human-readable language. For some categories of test it is
possible to compare a reference example to a test example (e.g. visually),
and for others it is useful to provide the relevant part of the specification
(which is assumed to explain what the test is) along with the example being
tested (e.g. the CSS test suite, or WAI checkpoints).

There is another class of cases where it is possible to do the testing - e.g.
validation of various kinds, but where it is difficult to build general tools
that can use the results of testing because there is little machine-readable
information to identify which requirement was failed.

I agree that a useful step would be to make each requirement addressable, and
to work with that before we go further into this realtively uncharted
territory. (Well, there are organisations who have spent years on this. Maybe
some of them are represented on this list).

cheers

Chaals

  Interestingly, having smart enough machines is sufficient for humans
  to write testable specs because a smart machine would tell the author
  whether what s/he just wrote is testable. Compare this with writing
  poems using pre-declared Java identifiers to avoid spelling mistakes
  and having a spelling checker that highlights misspelled words as you
  type your poem.

  Personally, I do not like the idea of making humans write more code
  than they already have to. I think it is a counter-productive approach
  in the long term, especially since not all real specifications can
  have 100% test coverage (some requirements cannot be pragmatically
  tested at all).

  Making spec bits addressable is as far as I would go [today and within
  W3C scope]. My understanding is that we already have the tools to make
  such addressing. And, practically speaking, it is not too much work to
  apply any given addressing scheme to address every interesting piece
  of a given specification. Do we really need a one-for-all standard
  that editors would be mandated to use?

  Thanks,

  Alex.


-- 
Charles McCathieNevile    http://www.w3.org/People/Charles  phone: +61 409 134 136
W3C Web Accessibility Initiative     http://www.w3.org/WAI  fax: +33 4 92 38 78 22
Location: 21 Mitchell street FOOTSCRAY Vic 3011, Australia
(or W3C INRIA, Route des Lucioles, BP 93, 06902 Sophia Antipolis Cedex, France)

Received on Monday, 6 May 2002 17:01:42 UTC