Re: What *is* a spec test? (musings on feature interop tests)

Hello,
Ms2ger, Alan, Tobie, thanks a lot for chiming in. I'll reply point by
point to your comments:

@ Ms2ger
--------
Using `<link rel=help>` solves only half the problem, namely identifying
all the spec (sections) a test is related to.

The other problem - and the one that I find more difficult to solve - is
where should these tests reside (i.e. in which test suite). In some
instances this is obvious (e.g. Testing how a flex container fragments is
a flexbox and not a fragmentation test case, since the flex spec is the
one that actually specifies this behaviour, or at least the one that
introduces this new case). However, there are instances where the
distinction is more difficult to make (e.g. Testing a particularly tricky
situation involving auto-sized regions and fragmentation might be a
regions test as well as a fragmentation test, (also) because the two specs
reference each other).

Or does the build step for test suites includes all the tests that
reference a particular spec (section)? In which case, I guess the problem
of the inclusion of a test in one suite or another is moot since tests
will be automatically duplicated across test suite without the need to
sync them "manually".

@ Alan
------
Having read the section on Interoperability in the Overview it seems there
are two meanings for "interoperability" - which can lead to some confusion.

The interoperability that the Overview mentions refers to the ability of
different CSS implementation to produce the same output given the same
input. The interoperability I was talking about in my initial email refers
to something slightly different: the property of any 2+ CSS/HTML/JS
modules/features that, when used together, produce a result that makes
sense (it's not surprising to the end-user) and is non-equivocal and
unique (e.g. two people reading the specs for the features involved will
not expect two different results).

To my knowledge, there's little talk of the latter, even though there are
numerous ways in which different CSS modules can interact which are not
always neither obvious nor well specified.

This was the aspect that I was trying to get some feedback on: how to
efficiently test for this kind of interoperability?

(Regarding the CSS3-Regions test suite, I'm one of the main contributors
to it and this was the very thing that prompted me to start this thread :)
)

@ Tobie
-------
A couple of examples that come to mind would be the regions & multicolumn
tests[1], regions and HTML <dialog> tests[2] or the regions and CSS
counters tests[3]. They don't test as many of the assertions of the spec
as they test the interaction between regions and other pieces of
CSS/HTML5, interactions that are not (yet) clearly specified.

As I mentioned above, I'm not asking to specify every possible
interaction: this is not feasible and in some instances, "it should just
work"(TM) it's a very reasonable expectation. What I'm asking is how to
better advertise, manage and get interest in contributing these kind of
tests and this kind of feedback on the spec(s).

Let me know if things are still unclear :)
Mihai

[1] 
https://github.com/w3c/csswg-test/tree/master/contributors/adobe/submitted/
regions/multicolumn
[2] 
https://github.com/w3c/csswg-test/commit/9d668789c3bc89fb28df4d4a94a07cc651
495aed
[3] 
https://github.com/w3c/csswg-test/tree/master/contributors/adobe/submitted/
regions/counters




Mihai Balan | Quality Engineer @ Web Engine team |  mibalan@adobe.com |
+4-031.413.3653 / x83653 | Adobe Systems Romania





On 11/4/13 3:08 PM, "Tobie Langel" <tobie@w3.org> wrote:

>On Sunday, November 3, 2013 at 8:23 PM, Mihai Balan wrote:
>> However, it proved that the most ³interesting² tests were not those
>>that tested only CSS Regions but CSS Regions *and* X feature (already
>>standardized or still in work but widely accepted and implemented). But
>>in writing these tests, I realized they stretch quite a bit the current
>>definition of what should be a spec tests in a CSSWG spec test suite ­
>>as some people have already pointed out. These tests are more ³feature
>>interoperability² tests than ³spec tests².
>Would you be able to point to specific examples? IT's hard to discuss
>this in the abstract.
>
>Thanks,
>
>--tobie  
>
>

Received on Tuesday, 26 November 2013 14:17:44 UTC