What *is* a spec test? (musings on feature interop tests)

Hello everybody,

I'm writing this email to get some feedback from this wider forum, based on some ideas that I've already bounced by some of the people here (Rebecca H., Francois R., thank you!).

Allow me to start with a little story first: when we first started testing CSS Regions in WebKit and Blink, we decided it would be good to have as many tests as possible as part of the W3C tests suite for the spec. Rebecca also created a script that would allow importing and running a W3C test suite inside WebKit/Blink. And armed with this ideal, we started creating W3C tests for CSS Regions.

However, it proved that the most "interesting" tests were not those that tested only CSS Regions but CSS Regions *and* X feature (already standardized or still in work but widely accepted and implemented). But in writing these tests, I realized they stretch quite a bit the current definition of what should be a spec tests in a CSSWG spec test suite - as some people have already pointed out. These tests are more "feature interoperability" tests than "spec tests".

Now we're face with the following dilemma, and here's where your input is most welcome. I strongly believe there's value in having these "feature interop" tests developed and shared through the W3C. As more complex specs are nearing maturity, I expect these kinds of tests to pop up more and more, as browsers inherently share some implementation similarities.

The challenge is thus threefold:

*         How to identify/advertise these tests (and the specs they refer to)?

*         Where to store them?

*         How to "advertise" them and make browser implementers and the community at large aware of their existence (to use them and contribute new tests)?

My proposal for a rather simple (and a little incomplete to begin with) solution would be to just store these "feature interop" test in the test suite corresponding to the feature they are most clearly linked to and introduce a new flag[1] or something similar to easily track and filter them. Does it make any sense?

Eagerly waiting to hear your take on this,
Mihai

[1] http://wiki.csswg.org/test/format#requirement-flags

Mihai Balan | Quality Engineer @ Web Engine team |  mibalan@adobe.com<mailto:mibalan@adobe.com> | +4-031.413.3653 / x83653 | Adobe Systems Romania