W3C home > Mailing lists > Public > public-qa-dev@w3.org > September 2003

Re: Markup Validator's test suite - using EARL?

From: Olivier Thereaux <ot@w3.org>
Date: Wed, 1 Oct 2003 11:23:19 +0900
Cc: Karl Dubost <karl@w3.org>
To: public-qa-dev@w3.org
Message-Id: <38A87E6E-F3B6-11D7-B421-000393A63FC8@w3.org>


On Wednesday, Oct 1, 2003, at 03:28 Asia/Tokyo, karl wrote:
> do you mean EARL for the report of passing the Test Suite?
> 	or to describe the test?

I meant to describe the test.
Using as a basis for a Test Description Language, though obviously not 
its primary purpose, has some appeal, because most if not all of the 
needed concepts are there already.


> We will have to consider something where the success of passing the 
> test suite is not the fact of being valid.
>
> Test A -> Has to be always invalid -> Success if invalid
> 								   -> Failed  if valid
> Test B -> Has to be always valid   -> Success if invalid
> 								   -> Failed  if invalid

There is actually one more dimension, which is the "regular" result 
given by the current validator. I suppose we could keep those apart as 
a list of assertions with the assertor being 
<http://validator.w3.org/check> a earl:Tool.


> So I guess something like
>
> ===================
[snip] the example looks quite good.
> =============================


> :validatorResult points to an URI
> 	valid
> 	invalid
> 	nocharset

nocharset is included in invalid ;)
I'd rather have the result be either valid or invalid, and add a result 
property stating why it wasinvalid (charset, doctype, etc.)

> Are the information about the test itself should be inside the test 
> itself? At first we could say yes thinking in terms of HTML, but it 
> will mean we will have to define of describing metadata for each kind 
> of markup. It might be easier to put this information in the n3 file.

I think so too. Besides for some tests we may want to have no test or 
comment whatsoever inside, to test a specific case.

> reference could be one or more reference to understand the context of 
> the test itself. People who have submitted, references in the specs, 
> discussions about the tests.

Then I would distinguish reference as in "Reference specification(s)" 
(may be multiple in case of mixed namespaces) and reference as in "see 
also".

> There is quite a huge number of tests in the HTML 4.01 Test Suite as 
> well.
> http://www.w3.org/MarkUp/Test/HTML401/

Right. We're definitely not lacking test documents, but we're seriously 
lacking some organization for them :)

Thanks a lot, your examples and suggestions help me a lot.

-- 
olivier
Received on Tuesday, 30 September 2003 22:23:21 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Thursday, 19 August 2010 18:12:43 GMT