Re: Getting vendors using W3C tests

On Tue, 31 May 2011, Aryeh Gregor wrote:

> On Tue, May 31, 2011 at 6:52 AM, James Graham <jgraham@opera.com> wrote:
>> * Getting the results out of the test into our systems. In theory this is
>> solved for testharness.js tests by including a second <script>;
>> testharnessreport.js that is blank on the W3C site and which vendors can use
>> to convert the testharness.js results to whatever is needed
>> result-collection system they use. However not all tests have consistently
>> included this extra script.
>
> I never got the point of that.  Why don't reusers just hack
> testharness.js itself?  Or if we want it to be a separate file for
> convenience, why doesn't testharness.js add the <script> to include
> that file automatically?

The idea is to have a single, well defined, place for vendor-specific code 
thus reducing the chance of merge conflicts when updating the testsuite 
and helping future maintainers of the local import identify "our code" and 
"their code". Experience from importing other external testsuites suggests 
that such a clean seperation is a big win.

I suppose testharness.js could include the testharnessreport.js script 
itself, although it seems a bit like unnecessary magic which is therefore 
best avoided. It is also not-quite-trivial to work out what the right path 
to the script should be. Nothing insurmountable, but I would prefer to 
avoid unless people feel strongly that including two scripts rather than 
one is a unreasonable burden.

>> The second problem needs a bit of thought. Possibly the testsuite
>> could have a build step and servers could be variables in the original
>> files. That has some disadvantages though since there is non-negligible
>> overhead to making everyone build the testsuite before they can run it.
>
> If we make sure that the set of hosts is fixed and their roles are
> well-documented, would it be unreasonable to ask implementers to just
> edit the host file on their test machine to point the domain to
> someplace they control?  That seems like it would be a lot simpler
> than any of the alternatives I can think of.

It seems likely we will do something like that.

Received on Wednesday, 1 June 2011 07:45:37 UTC