Re: Ringmark, Core Mobile, and the Financial Times

On Mon, Jun 11, 2012 at 4:38 AM, Robin Berjon <robin@berjon.com> wrote:

> On Jun 11, 2012, at 10:39 , James Graham wrote:
> > The best solution for bugginess is to have high quality public
> testsuites for various features. The various W3C working groups already
> curate tests for their various specs and have guidelines / frameworks for
> writing tests. The main problem is sourcing the tests themselves; so far
> the dominant contributors by far are browser vendors, and even they do not
> yet contribute as much as you might hope due to a history of writing tests
> in ways that that are specific to their particular browser or testing
> setup. This has been improving recently, however.
>
> That's certainly part of the problem, but not the whole story either. For
> most of the issues that Robert brings up the features are actually
> supported and would be reported as such in traditional conformance test
> suites. The fact that they're there but simply unusable for developers
> would not be noted in the implementation report.
>

I brought this up at the recent WebApps F2F, namely, that the traditional
purpose for test suites in the W3C has been to demonstrate implementability
of a spec, i.e., as a tool to help persuade the director that a spec can
move from CR to PR. Other purposes for testing, e.g., interoperability and
quality, have not been an explicit goal in the W3C test process, at least
not the way it has been practiced in general. Certainly, some groups and
some specs have had test suites created with this in mind, but the W3C
process does not (presently) encode this as a goal in test creation or use.

In addition, because the W3C does not perform any type of compliance
testing (and it is unlikely to do so any time soon if ever), these other
goals are typically given low priority or dropped entirely.

I would suggest that we need to find ways to encode the goals of
interoperability and quality (of the technical products, not the process
itself) in the W3C Process in a manner that translates to specific actions
in the way W3C test suites are developed and used.