Re: W3C's XHTML QA plan

Having just finalized the first release of the DOM Level 1 Core Test 
Suite, here are my two (euro) cents:

The DOM TS was jointly launched by the W3C and NIST, and invited people 
from the community to help out in order to make it publically accessible 
and developed not behind the scenes. The issue with the quality of tests 
was to be dealt with using some kind of moderator mechanism, where I 
(appointed coordinator between the DOM WG and the DOM TS group as 
invited expert), a NIST representative and the main developer formed the 
moderator group of the DOM TS.

The only drawback was time; as there was no economical incentive such as 
early access, things tended to take longer than they needed. I believe 
the quality of the frameowrk to be acceptible, given that the TS was 
developed by what we call external resources (the DOM WG did not 
particiapte as much as they could, with notable exceptions).

As far as the general outline is concerned I cannot but agree with you: 
in order to make the TS (of any specification) quickly developed and of 
good quality, the W3C would need to "put their money where their mouth 
is", which also goes for the member companies of the WG in question. You 
note that ApTest is a small company and cannot carry the cost of 
developing the TS for free; in my case, Ontologicon is just my freelance 
name, so you can imagine what coordinating the TS production meant 
economically (at least for me). Having received no funds whatsoever for 
coordination (except for forthcoming travel funds for the Cannes Tech 
Plenary), I now wonder how an equally good test suite for other levels 
of the DOM and other W3C specificaiton can be created if the TS is just 
to be considered as a "WG deliverable" without any stronger commitment 
on the part of the member companies in the WG (any WG that produces a 
test suite). The market value of producing a test suite is of course 
what companies are willing to pay for the competence and expertise 
involved in creating such a thing, but it certainly does not come for 
free.

So, brifely, my views are the following (and they should be worth 
something as the DOM TS process document is mentioned in every second 
test suite related document coming from the W3C or related bodies):

1. The W3C needs to raise awareness among its member companies on the 
relevance of producing a TS and why it is worth pursuing as high a 
quality as possible (which of course means paying for it)
2. The member companies need to allocate resources for this, either by 
seeing to that they deliver the job themselves, or by seeing to that 
someone else does it
3. In the charter or not, the W3C should strive for having sound Test 
Suites for any technology it specifies
4. The old question of conformance and rating should perhaps be brought 
up again, as I'm sure it would make people more aware of the costs 
involved in creating proper test suites (this is partly being adressed 
by the QA WG) and quite possibly more keen to pay for a product (a test 
suite) which it is meaningful to claim conformance to.

In short: there needs to be a carrot.

Again, my two cents. Please visit http://www.w3.org/DOM/Test and tell me 
how things could be improved.

Kind regards,

/Dimitris Dimitriadis, W3C DOM & QA WG

On Wednesday, January 30, 2002, at 05:52 , Shane P. McCarron wrote:

> There is a huge difference between a test suite and a good test 
> suite.  I
> think what the HTML Working Group wants is a good test suite, not just 
> a bag
> of HTML documents that got collected from here and there.  We are 
> taking a
> thorough, analytical approach to the specifications, and producing
> assertion-level tests that fully exercise the specification.  This type 
> of
> work requires significant investment and expertise.  ApTest does this 
> all
> the time, but we are a small company and cannot afford to just eat the 
> cost
> of this work. That is why we have asked the interested parties to help 
> share
> the burden of producing the quality tests that are needed.
>
> Karl Dubost wrote:
>
>> Shane,
>>
>> At 18:33 -0600 2002-01-29, Shane P. McCarron wrote:
>>> Just to be clear... While this project involves people from the HTML
>>> Working Group, and while we will give the results to the W3C 
>>> membership
>>> when we are done, it is not a W3C project.  We needed to fund the
>>> project, and we found a way to do it.  Early access is the carrot that
>>> is getting people to pony up some funds to get the development work
>>> done.  Once it is done, we will make it available to all W3C members.
>>>
>>> This is farily common practice in the standards industry.  If the W3C
>>> had chosen to fund this project, we wouldn't need to do it this way.
>>> They didn't.  So we do.  I don't like it, but I don't really have a
>>> choice.
>>
>> Most of the working group have done their test suites without funding
>> of the W3C. It has happened that someone has been hired to coordinate
>> the work, but no funding to create the test cases themselves. As
>> members participate to the working groups, and give resources to
>> elaborate the standards, they commit resources also to create such a
>> test suite.
>>
>> For the newly created WG charters, I'm trying to make it a part of
>> the charter. It means, that the test suite is a deliverable of the
>> WG. I never understood why the HTML WG has always been so reluctant
>> to create such a test suite. I perfectly understand that it was not
>> on the charter. But HTML being the most used language on the web, it
>> would have been not difficult to have a broad participation from the
>> community.
>>
>> --
>> Karl Dubost / W3C - Conformance Manager
>>            http://www.w3.org/QA/
>>
>>       --- Be Strict To Be Cool! ---
>
> --
> Shane P. McCarron                     Phone: +1 763 786-8160 x120
> Managing Director                       Fax: +1 763 786-8180
> ApTest Minnesota                       Inet: shane@aptest.com
>
>

Received on Thursday, 21 February 2002 11:22:01 UTC