test suite licensing digest Jan 2008 (fantasai's comments only)

It seems that the W3C Document License is not very appropriate for test suites,
or at least for the CSS test suite. It makes requirements that are not appropriate
and doesn't allow re-use in ways that are useful for test suites. The MIT license
was suggested as an alternative but I think it gives a little /too/ much leeway
in making changes without requiring that the result not be misrepresented as a
W3C test.

-----

The discussion in the CSSWG was about including the CSS tests in a
test framework that may require changes to the tests. As an example,
Dominique's test harness makes changes to the tests:
   http://www.w3.org/2007/03/mth/harness
These changes are not allowed by the W3C Document License.

-----

Another problem is that the W3C Document License doesn't allow modification.
So a developer is not allowed to tweak the testcase to make it more useful
in fixing a particular bug.

-----

That's fine for the MWI test harness or any other harness that we choose
to use here at W3C, but the problem here is that other groups will need
to incorporate the tests into their own harnesses as part of their
development process, and they shouldn't need to request permission from
W3C to use the tests in such a manner. That's kinda what they're for.

-----

We *want* developers to be able to modify the tests without resubmitting
them to the CSSWG. This is a necessary part of using them in the layout
engine development process. The only thing we don't want is for the modified
tests to be confused with the official tests. If we can use a combination of
the MIT license and trademark law for that, that would be ideal. Because that's
really the issue here. If we can't then maybe we can write a new test suite
license.

-----

The longer we wait on this, the harder it is going to be to relicense
existing tests. I want this issue resolved *soon*.

Here are the allowances we need to have:

   a) Format-related modification of all tests to hook into a test
      harness.
   b) Unlimited modification of individual tests in order to simplify
      them or to create variations.

Neither should require asking permission. They are both things that are
done in the process of developing a layout engine, and layout engine
developers shouldn't need to register with the W3C.

The only requirement should be that the resulting tests not be easily
confused with official tests from W3C. In case A, a simple disclaimer
that the harness might affect the tests' correctness would probably
suffice, provided that the harness attempts to make no substantial
changes to the tests. We do this already for our own harnesses. In
case B, it should be clear from the modified test itself that it is not
part of the test suite, i.e. anything that identifies the test as part
of the W3C's test suite should be changed or deleted.

I believe it would be easiest for everyone involved if W3C picked a
very liberal software license and enforced the above restrictions via
trademark law, as the Mozilla Project is doing with their own code.

-----

I don't expect very many test suite harnesses, but I do expect many test
cases to be derived from the ones in the conformance test suite, some of
which will make their way back to the official conformance test suite and
most of which will be attachments to bug reports or added to regression
test frameworks.

-----

I think so, as long as we define a branding and restrict its use to
W3C alone. E.g. modified tests may not be labeled "CSS2.1 Conformance
Test".

-----

I don't think anyone wants tests modified outside the W3C to be used for
defining compliance or conformance to the spec. But it is very often
useful and convenient to take tests from the test suite and use them as
part of a regression test suite or to use parts of them to create new
tests that then form part of some other test suite. We want to allow that,
and to make it easy and painless.

-----

Rigo, no one has any intention of changing the licensing on the CSS specs.
It's just the test suite that's under discussion right now.

I understand that there are other industry consortia that could interfere
with our work here, but from my (limited) experience with the printing
standards consortia I think CSS is actually more threatened by accidentally
incorrect tests and incomplete test suites than by maliciously incorrect
tests.

Under the W3C Document License a different consortium can't take our tests
as the basis of an incorrect test suite, but there is nothing preventing
them from creating their own incorrect tests and labelling them as CSS
Conformance Tests. So restrictive licensing can't solve this problem
effectively anyway. (And as hixie notes, the license on W3C test suites often
isn't followed anyway because it is so inconvenient.) We're better off with
a strong trademark policy, because that's capable of addressing both these
cases--and if we have that, then we should be able to relax the licensing to
make the tests more usable for layout engine developers and open source
projects.

-----

Full discussion should be available somewhere on team-legal's archives.
See also
   http://lists.w3.org/Archives/Public/public-css-testsuite/2008Jan/0021.html
thread at
   http://lists.w3.org/Archives/Public/public-css-testsuite/2007Dec/0012.html
(which continues on  http://lists.w3.org/Archives/Public/public-css-testsuite/2008Jan/thread.html )
and
   http://lists.w3.org/Archives/Public/public-css-testsuite/2008Feb/0003.html

~fantasai

Received on Wednesday, 27 February 2008 22:43:52 UTC