Markup Validator Test Suite

Hi,

  Some quick notes regarding our Markup Validator testing efforts. I
think what we should develop is a Test::Builder based Test module that
provides numerous functions that can be used from standard Perl test
scripts. This is specifically for the functionality in `check`, I
expect our dependencies like SGML::Parser::OpenSP to ship with their
own `make test`able test suite. This would have a script that can be
called via

  perl script http://localhost/ [.../validator.conf]

where the second parameter defaults to W3C_VALIDATOR or whatever the
env var is valled. http://localhost/ would be the base uri that has
`check`, etc. This Test::Builder module would have functions such as

  * has_outline
  * is_http200
  * is_wellformed
  * is_valid
  * is_texthtml
  * has_utf8_charset
  * result_doctype_is
  * is_appendix_c
  * ...

or something along these lines. A script that uses this module would
then be able to perform operations such as

  my $doc = ...->("/");
  is_http200($doc);
  is_wellformed($doc);
  is_valid($doc);
  ...

(these could be abbreviated by something like is_proper_doc(...) but
generally you would like to know what is wrong with the document if
anything). This can specifically be used to perform operations on
check's output e.g. like

  my $doc2 = ...->("check?uri=...;outline");
  ...
  has_outline($doc2);
  ...

or a simplified interface that does not require to %hh encode special
characters, e.g.

  my $doc3 = ...->check(uri => ..., outline => 1);
  ...

Some of these scripts (or all of them) can and probably should be
generated from other data sources, for example we might just have a list
of all the documents relevant to the Markup Validator like

  * /feedback.html
  * /about.html
  * ...

all of which must exist and be well-formed, valid, Appendix C
compatible, ... XHTML 1.0 Strict documents. For many things a POST
interface to the Validator can be used to avoid having too many files
around somewhere, but for some things (that attempt to check the GET
interface) we would need additional means. This would suggest that we
have a .htaccess in /dev/tests/ and maintain test case documents there.

We would further establish a requirement for changes in the Markup
Validator (especially bug fixes) that these have proper test cases
somewhere so that the bug is not re-introduced. Which suggests that we
should give creating at least a framework for the test suite so that we
can start adding tests as soon as possible. Writing the Test module
should be straightforward using LWP and XML::LibXML which can validate,
check for well-formedness, supports SGML catalogs, and supports XPath
which means that the has_outline sub would take just a few lines
checking id('outlineresult').

regards.

Received on Sunday, 12 September 2004 16:54:58 UTC