W3C home > Mailing lists > Public > public-css-testsuite@w3.org > July 2013

Re: [css3-ui] Tests submitted for directional focus navigation properties

From: Gérard Talbot <css21testsuite@gtalbot.org>
Date: Tue, 16 Jul 2013 22:45:24 -0400
Message-ID: <7a494599d504bd95b396b2ecadd9e8a1.squirrel@ed-sh-cp3.entirelydigital.com>
To: "Rebecca Hauck" <rhauck@adobe.com>
Cc: "Leif Arne Storset" <lstorset@opera.com>, "public-css-testsuite@w3.org" <public-css-testsuite@w3.org>, "Tantek Çelik" <tantek@cs.stanford.edu>, "Jorrit Vermeiren" <jorritv@opera.com>, "Giuseppe Pascale" <giuseppep@opera.com>

Le Mar 16 juillet 2013 21:04, Rebecca Hauck a écrit :
> Hi Gérard,
>
>
> On 7/16/13 4:42 PM, ""Gérard Talbot"" <css21testsuite@gtalbot.org>
> wrote:
>
>>
>>Le Mar 16 juillet 2013 16:24, Rebecca Hauck a écrit :
>>> Hi Leif,
>>>
>>> I took a look and added comments on the commit in Github:
>>>
>>>
>>>https://github.com/w3c/csswg-test/commit/f394a9f65071d53b4f5b74d5dd8b14d9
>>>d0
>>> 96f2e6
>>>
>>>
>>> Other than the few minor changes I suggested, they look good.
>>>
>>> Cheers,
>>> -Rebecca
>>
>>Rebecca,
>>
>>"
>>This test passes for me on a UA that does not support this property.
>> You
>>can add an intermediate step to first verify the property is supported.
>>"
>>contributors/opera/submitted/css3-ui/nav-dir-target-006.html
>>
>>I have in the past also criticized some tests for not being designed to
>>fail when the test is testing an invalid syntax (or invalid
>> declaration)
>>and when an UA does not support a property. I think this issue should
>> be
>>clarified.
>
> What exactly should be clarified?


I think it should be stated in the Design Requirements section of Test
Format
http://wiki.csswg.org/test/format#design-requirements
that if the [property or declaration or feature] being tested isn't
supported, then the test should always fail.

Right now, there is only "The test will not pass inadvertently." in
CSS Test Review Checklist, Test Design
http://wiki.csswg.org/test/css2.1/review-checklist#test-design
which is a not-so-well-known wiki page.

> I think that for any spec test, if
> the
> thing you're testing isn't supported, the test should always fail.

In the past, Florian and I had an email exchange on this precise issue

http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0014.html

http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0017.html

http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0020.html

and I think the Test Format Guidelines

http://wiki.csswg.org/test/format

should explicitly cover such test design situation. Because there is a
minority of tests that will "pass" when the UA does not support the
[property or declaration or feature] being tested.

Eg. UAs that do not support border-radius will pass these 8 tests

http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-009.htm

http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-010.htm

http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-012.htm

http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-013.htm

http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-014.htm

http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-015.htm

http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-016.htm

http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-top-left-radius-values-003.htm

If you look more, you can easily find more like these.


> if
> the
> thing you're testing isn't supported, the test should always fail.
> Running
> such as suite (particularly in an automated engine) would give the
> impression of partial support for that feature when there is none.


Exactly.


> For
> these type of tests that test invalid or incomplete syntax, the
> reference
> file should not be designed to render whatever the default behavior is.
> For this case, however, I (maybe mistakenly) used Tab and Shift+Tab to
> test directional navigation, which causes them to pass on Chrome, which
> does not support this, for example.
>
> We're in the process of a W3C testing documentation project [1], which
> will be rolled out sometime soon. When I edit the Review Process doc
> [2],
> I can articulate this point there. I will surely solicit your feedback
> when it's ready.
>
>
>>
>>Eg.
>>http://test.csswg.org/harness/results/CSS3-CONDITIONAL_DEV/
>>shows that IE10 scores 21% but, in reality, the only tests IE10 passes
>>are those testing invalid syntax causing parsing errors.
>>
>>Some CSS2.1 and CSS3 tests will actually be passed by buggy or old
>>browsers (say, Netscape 4 or IE4). I think this is wrong.
>
> We are in agreement. Sounds like there might be some poorly designed
> tests
> here.
>
>
>>
>>-----------
>>
>>Opera has now 2 accounts:
>>
>>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/opera_
>>software/
>>
>>will return "No Assets Found." while
>>
>>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/opera/
>>will return "15 Test Cases Found".
>>
>>I think there should be only 1 Opera account.
>>
>>------------
>
> There are many tests under both accounts [3][4],


Indeed.


> so I'm not sure what we
> can do about that.
> It looks like there was one account created through
> the Shepherd UI under the name "Opera Software ASA" and tests submitted
> with that author title map to that account. Other tests were submitted
> under just "Opera Software", which did not match anything so Shepherd
> created a new account with the underscoring method. You're right, this
> could be confusing, so I'll check with Peter how to handle this.
>
>
>>
>>http://test.csswg.org/source/contributors/opera/submitted/css3-ui/nav-dir-
>>001.html
>>
>>"
>>First, use directional navigation to navigate the focus to the "START"
>>link below.
>>
>>Test passes if navigating left once moves the focus to the "FINISH"
>> link.
>>"
>>
>>I have no idea if this is possible... but is it possible to make the
>>wording of pass-fail conditions sentences in tests like those to be
>> less
>>technical? I am asking because I really do not know this.
>
> This was my first thought as well, but then I looked at the spec, which
> explicitly says: "This specification does not define which keys of a
> device are directional navigational keys,"

Okay.

> so this is really the only
> way
> to express the steps here. The steps aren't so much technical as they
> are
> vague because so is the spec - intentionally.  One must assume the
> implementor running these test knows which keys you can use to navigate
> (though there is a note in the spec stating that arrow keys are a likely
> choice as well as shift+tab for reverse direction).
>
>
>>
>>I am using Opera 12.16 and I have no idea what exactly I must do in
>> such
>>test.
>
> Opera's implementation and how to use it is described here [5].


"
(...)
Structural elements within a document are also navigable. The previous
and next header on the page can be found using W and S. The keys E and D
do the same for text elements.

The keys Q and A allow you to jump to the previous or next link in the
document. (Ctrl+Down and Ctrl+Up do the same.)
"
Use Opera without a mouse: Document Navigation
http://www.opera.com/help/tutorials/nomouse/#nav

I am still somewhat unsure on how exactly I should be "doing" these
tests as a tester using Opera 12.16. I have found and checked the box
marked "Enable single-key shortcuts"... And so Q and A keys navigate, E
and D keys navigate and Ctrl+Down and Ctrl+Up navigate.


> I don't
> think this specific instruction should be in the test itself though
> because as I mentioned above the spec makes no keyboard requirements.
> It
> might have been nice if this info were provided with the review was
> solicited, but it was easy enough to find :)


As test creator, I would provide such kind of info (the keyboard
combination for various browsers) inside <!-- comments --> in the source
code.


>
>
>
>>
>>------------
>>
>>Rebecca,
>>
>>your comments in
>>
>>github.com/w3c/csswg-test/commit
>>
>>are not "echoed" into Shepherd system. I believe comments on tests
>>should not be scattered around.
>>
>
> Yes, I know. When these tests are resubmitted with reviewer links, I
> will
> then add a link to the review in Github when I set them to Approved. I
> figured there would be a fairly quick turnaround with Leif and his team,
> so I was waiting for their resubmission in the next few days.
>
> Keep in mind that Peter is doing work to integrate Shepherd with Github
> so
> this will happen automatically when that work is done.  Right now, we're
> favoring Github's review tools to Shepherd's for many reasons (more
> widely
> used and understood, support for inline comments, email notifications,
> subscriptions, etc.). I'd like to move away from having all of the
> review
> comments done on this mailing list, where the comments are so far away
> from the code.
>
> In this interim period, you can continue to do things as you have been.
> When we're fully integrated with Github (and with the rest of the w3c
> test
> suites), I'll propose this change in process and make sure it's
> documented
> in the 'Reviewing Tests' doc.
>

Oh well... I like using Shepherd ... but anyway...

Gérard

>
> [1] https://github.com/w3c/testtwf-website/issues?milestone=1&state=open
> (note that the closed issues are in the process
> [2] https://github.com/w3c/testtwf-website/issues/13
> [3] http://test.csswg.org/shepherd/search/author/opera_software/
> [4] http://test.csswg.org/shepherd/search/author/opera/
> [5] http://www.opera.com/help/tutorials/nomouse/

-- 
Contributions to the CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/

CSS 2.1 Test suite RC6, March 23rd 2011:
http://test.csswg.org/suites/css2.1/20110323/html4/toc.html

CSS 2.1 test suite harness:
http://test.csswg.org/harness/

Contributing to to CSS 2.1 test suite:
http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contributions-css21-testsuite.html
Received on Wednesday, 17 July 2013 02:46:02 UTC

This archive was generated by hypermail 2.4.0 : Friday, 20 January 2023 19:58:19 UTC