Re: dependencies in tests

Le 2015-06-18 06:26, Florian Rivoal a écrit :
>> On 17 Jun 2015, at 22:24, Gérard Talbot <css21testsuite@gtalbot.org> 
>> wrote:
>> 
>> Le 2015-06-17 02:41, Florian Rivoal a écrit :
>>>> If the user agent does not support vertical writing-mode, then the 
>>>> test must fail.
>>> Why? I am writing a test for the css3-ui test suite. css3-ui has a
>>> provision for what to do if vertical text is supported (which is what
>>> I am trying to test), but it does not require vertical text to be
>>> supported.
>> 
>> css3-ui does not require vertical text to be supported: that's 
>> correct.
>> 
>> But then...
>> 
>> For 'cursor: text', "User agents may automatically display a 
>> horizontal I-beam/cursor (e.g. same as the vertical-text keyword) for 
>> vertical text"
>> 
>> http://www.w3.org/TR/2015/WD-css3-ui-20150519/#cursor
>> 
>> How do you define 'vertical text' in that sentence?
>> 
>> How is it possible to have vertical text while, at the same time, not 
>> being able to support vertical 'writing-mode'?
>> 
>> Isn't 'vertical text' in that sentence implicitly assuming that a 
>> vertical writing-mode has to be supported?
> 
> My understanding is that vertical text doesn't refer to any particular
> mechanism. The one we have is writing modes, but css3-ui is being
> neutral about that, and merely says:
> 
> Regardless of how you do it, if you support vertical text, here's how
> the 'text' cursor behaves when mousing over vertical text.

This is a matter of interpretation of the specification sentence. When 
reading the sentence, I did not understand it as "regardless of how you 
do vertical text" or as "regardless of mechanism used to create vertical 
text". I understood it as "when you create text thanks to vertical 
'writing-mode'".


>> That's why there is a "may" in that sentence. Failing that test will 
>> not make the UA fail the required tests for conformance to css3-ui. My 
>> understanding.
> 
> 
> There's two things interacting here.
> 
> 1) There a "may", so failing the test is not the end of the world,

A "may" or a "should" makes a test not required to pass in order to 
claim conformance with the spec: that's all.

> but
> the "may" does not affect what the conditions are for pass of fail.

We agree: the "may" does not affect what the conditions are for pass of 
fail.

> 2) If you go with standard logic, when testing the statement "if A
> then B", then having A and not B is a failure, having A and B is a
> success, and not having A is a success, regardless of B.

A is a boolean condition of some sort; if A does not exist, then we can 
not check for B.

Not having A makes the test result undefined, unknown or makes the test 
not applicable.

Eg. Prince version 10.2r1 is a conversion HTML-to-PDF-with-CSS web-aware 
application. So, tests with flags "animated" and "interact" should be 
avoided and do not apply to such UA and the related test results should 
be ignored.

> However, saying that a browser passes if it didn't implement anything
> is not terribly helpful.

I did not say such thing... just want to make that clear.

A wide majority of tests in, say, writing-modes test suite have no 
boolean condition or have no test prerequisite or no test precondition 
and are in the form of: "Test passes if there is a filled green square 
and *no red*." Therefore, I expect buggy browsers and old browsers to 
fail such tests.

> But saying that it fail when it didn't need
> to do anything (since vertical text isn't supported) isn't great
> either, which is why I am looking for a "if not A, skip this test"
> answer.
> 
> But from your comments, I guess this isn't the way we do things. In
> this case, thanks to the "may", it turns out that it's not a major
> problem, but we have other similar cases where there is no "may" to
> save the day, and it's more annoying. For example, we have some tests
> that check for the interaction of multicol and regions.

CSS Regions Module is a Working Draft
http://www.w3.org/TR/css3-regions/

CSS Multi-column Layout Module is a Candidate Recommendation
http://www.w3.org/TR/css3-multicol/

Yes, some tests check interaction of specifications. In general, there 
is not a lot of tests testing interaction of specifications.

> If you support
> both and fail the test, then it's a clearly a failure, but I think it
> is a problem that an implementation that is only trying to implement
> one of the 2 would fail the test.

It would fail that test (singular). And failure of such test does not 
mean that an UA can not (or will not) pass all the other tests in the 
test suite of the specification that it is trying to implement.

How and why a test failed is up to manufacturer of the UA: it is up to 
the manufacturer of such user agent or rendering engine to establish or 
to untangle the reason of the test failure by their user agent or 
rendering engine.

> If such tests were not included in
> any particular spec's conformance test suite, I wouldn't mind

Well, I probably would mind .. real-world webpages resort to CSS 
properties from several specs.

> but they
> get pulled into both, and that means that are effectively discouraging
> writing cross-spec tests, since they create (or appear to create)
> dependencies that weren't required by the spec itself.
> 
> 
>  - Florian

I've read your last sentence and did not quite understand it: why are 
you saying "that means that are effectively discouraging writing 
cross-spec tests"?

Some tests testing interaction of specs are *needed* at some point: when 
all basic tests have been created (and reviewed!) and when specs 
involved are stable (at least CR).

Gérard
-- 
Test Format Guidelines
http://testthewebforward.org/docs/test-format-guidelines.html

Test Style Guidelines
http://testthewebforward.org/docs/test-style-guidelines.html

Test Templates
http://testthewebforward.org/docs/test-templates.html

CSS Naming Guidelines
http://testthewebforward.org/docs/css-naming.html

Test Review Checklist
http://testthewebforward.org/docs/review-checklist.html

CSS Metadata
http://testthewebforward.org/docs/css-metadata.html

Received on Thursday, 18 June 2015 20:03:38 UTC