W3C home > Mailing lists > Public > public-css-testsuite@w3.org > November 2013

Re: What *is* a spec test? (musings on feature interop tests)

From: Mihai Balan <mibalan@adobe.com>
Date: Tue, 26 Nov 2013 14:15:32 +0000
To: Tobie Langel <tobie@w3.org>, Alan Gresley <alan@css-class.com>, Ms2ger <ms2ger@gmail.com>, "Public CSS Test suite mailing list (public-css-testsuite@w3.org)" <public-css-testsuite@w3.org>
Message-ID: <CEBA29FC.23D30%mibalan@adobe.com>
Ms2ger, Alan, Tobie, thanks a lot for chiming in. I'll reply point by
point to your comments:

@ Ms2ger
Using `<link rel=help>` solves only half the problem, namely identifying
all the spec (sections) a test is related to.

The other problem - and the one that I find more difficult to solve - is
where should these tests reside (i.e. in which test suite). In some
instances this is obvious (e.g. Testing how a flex container fragments is
a flexbox and not a fragmentation test case, since the flex spec is the
one that actually specifies this behaviour, or at least the one that
introduces this new case). However, there are instances where the
distinction is more difficult to make (e.g. Testing a particularly tricky
situation involving auto-sized regions and fragmentation might be a
regions test as well as a fragmentation test, (also) because the two specs
reference each other).

Or does the build step for test suites includes all the tests that
reference a particular spec (section)? In which case, I guess the problem
of the inclusion of a test in one suite or another is moot since tests
will be automatically duplicated across test suite without the need to
sync them "manually".

@ Alan
Having read the section on Interoperability in the Overview it seems there
are two meanings for "interoperability" - which can lead to some confusion.

The interoperability that the Overview mentions refers to the ability of
different CSS implementation to produce the same output given the same
input. The interoperability I was talking about in my initial email refers
to something slightly different: the property of any 2+ CSS/HTML/JS
modules/features that, when used together, produce a result that makes
sense (it's not surprising to the end-user) and is non-equivocal and
unique (e.g. two people reading the specs for the features involved will
not expect two different results).

To my knowledge, there's little talk of the latter, even though there are
numerous ways in which different CSS modules can interact which are not
always neither obvious nor well specified.

This was the aspect that I was trying to get some feedback on: how to
efficiently test for this kind of interoperability?

(Regarding the CSS3-Regions test suite, I'm one of the main contributors
to it and this was the very thing that prompted me to start this thread :)

@ Tobie
A couple of examples that come to mind would be the regions & multicolumn
tests[1], regions and HTML <dialog> tests[2] or the regions and CSS
counters tests[3]. They don't test as many of the assertions of the spec
as they test the interaction between regions and other pieces of
CSS/HTML5, interactions that are not (yet) clearly specified.

As I mentioned above, I'm not asking to specify every possible
interaction: this is not feasible and in some instances, "it should just
work"(TM) it's a very reasonable expectation. What I'm asking is how to
better advertise, manage and get interest in contributing these kind of
tests and this kind of feedback on the spec(s).

Let me know if things are still unclear :)


Mihai Balan | Quality Engineer @ Web Engine team |  mibalan@adobe.com |
+4-031.413.3653 / x83653 | Adobe Systems Romania

On 11/4/13 3:08 PM, "Tobie Langel" <tobie@w3.org> wrote:

>On Sunday, November 3, 2013 at 8:23 PM, Mihai Balan wrote:
>> However, it proved that the most ≥interesting≤ tests were not those
>>that tested only CSS Regions but CSS Regions *and* X feature (already
>>standardized or still in work but widely accepted and implemented). But
>>in writing these tests, I realized they stretch quite a bit the current
>>definition of what should be a spec tests in a CSSWG spec test suite ≠
>>as some people have already pointed out. These tests are more ≥feature
>>interoperability≤ tests than ≥spec tests≤.
>Would you be able to point to specific examples? IT's hard to discuss
>this in the abstract.
Received on Tuesday, 26 November 2013 14:17:44 UTC

This archive was generated by hypermail 2.4.0 : Friday, 20 January 2023 19:58:20 UTC