Re: Markup Validator Test Suite

* olivier Thereaux wrote:
>6) [[Must evolve with the development of the validator, and be able to 
>test old versions as well as the "newest, latest, greatest]] is more of 
>a problem for a T::B-based test suite, because it would not be able to 
>test versions prior to the switch to the appropriate architecture.

I am not sure which "switch" you mean here and this is either not the
case here or not relevant to a Test::Builder based test module. I am
not sure which exactly as I do not really understand why it needs to
be able to test outdated versions and how this would be implemented.
It seems that this might make some limited sense in some rare cases,
but considering the effort required to make this useful it is nothing
we should worry much about.

>So, no automatic tying of bugzilla with the test cases. But we could 
>still do something a la SVG WG, setting a rule that no bug is accepted 
>if a test case is not attached to it (thus allowing us to add it to our 
>repository of test cases).

I am opposed to such a rule. It would make more sense to require
("should") that the relevant test suite includes one or more tests
for the specific item before the issue can be closed as fixed.

>Finally 1) [[  Must be applicable to automated tests .. and 
>human-driven tests .. and a gray area in between ]] is, I think, the 
>weak point of a test suite solely based on T::B. Bjoern was the first 
>one to point out that a fully automated test framework was not the way 
>to go with the validator.

Human-testing only makes sense for things that would be difficult or
impossible to test using machine-based testing, this would be pretty
much limited to "does this work in all relevant browsers". Here I am
only concerned about machine-based testing.

>>   sub has_errors
>>   {
>>       my $self = shift;
>>       my $mess = shift || 'Results contain errors';
>>       $Test->like($self->{res}->content, qr/<ol id="errors">/, $mess);
>>   }
>
>Well, the example above is not so bad if the semantics of the test is 
>"did we output the ordered list delimiter", not so much if it's 
>supposed to test if there were errors.

Whether the code fragment above is present in the output gives you a
fairly good indication for that. We can, of course, spend some more
years on trying to create the perfect test suite before we can use it...

>I'm also wondering about the fact that, in theory at least, 
>Test::Builder *could* be used to create human-centered testing, ranging 
>from a crude commandline interfacing telling the user "go to that page, 
>do you see a red square? [Y/n]" to perhaps something more 
>sophisticated?

If you can do it in Perl you can do it using Test::Builder.
However, this does not really make sense for the Validator.

Received on Friday, 15 October 2004 03:23:24 UTC