Re: [css3-ui] Tests submitted for directional focus navigation properties

On 7/16/13 7:45 PM, ""Gérard Talbot"" <css21testsuite@gtalbot.org> wrote:

>
>Le Mar 16 juillet 2013 21:04, Rebecca Hauck a écrit :
>> Hi Gérard,
>>
>>
>> On 7/16/13 4:42 PM, ""Gérard Talbot"" <css21testsuite@gtalbot.org>
>> wrote:
>>
>>>
>>>Le Mar 16 juillet 2013 16:24, Rebecca Hauck a écrit :
>>>> Hi Leif,
>>>>
>>>> I took a look and added comments on the commit in Github:
>>>>
>>>>
>>>>https://github.com/w3c/csswg-test/commit/f394a9f65071d53b4f5b74d5dd8b14
>>>>d9
>>>>d0
>>>> 96f2e6
>>>>
>>>>
>>>> Other than the few minor changes I suggested, they look good.
>>>>
>>>> Cheers,
>>>> -Rebecca
>>>
>>>Rebecca,
>>>
>>>"
>>>This test passes for me on a UA that does not support this property.
>>> You
>>>can add an intermediate step to first verify the property is supported.
>>>"
>>>contributors/opera/submitted/css3-ui/nav-dir-target-006.html
>>>
>>>I have in the past also criticized some tests for not being designed to
>>>fail when the test is testing an invalid syntax (or invalid
>>> declaration)
>>>and when an UA does not support a property. I think this issue should
>>> be
>>>clarified.
>>
>> What exactly should be clarified?
>
>
>I think it should be stated in the Design Requirements section of Test
>Format
>http://wiki.csswg.org/test/format#design-requirements
>that if the [property or declaration or feature] being tested isn't
>supported, then the test should always fail.
>
>Right now, there is only "The test will not pass inadvertently." in
>CSS Test Review Checklist, Test Design
>http://wiki.csswg.org/test/css2.1/review-checklist#test-design
>which is a not-so-well-known wiki page.
>
>> I think that for any spec test, if
>> the
>> thing you're testing isn't supported, the test should always fail.
>
>In the past, Florian and I had an email exchange on this precise issue
>
>http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0014.html
>
>http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0017.html
>
>http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0020.html
>
>and I think the Test Format Guidelines
>
>http://wiki.csswg.org/test/format
>
>should explicitly cover such test design situation. Because there is a
>minority of tests that will "pass" when the UA does not support the
>[property or declaration or feature] being tested.
>
>Eg. UAs that do not support border-radius will pass these 8 tests
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-009.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-010.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-012.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-013.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-014.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-015.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-016.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-top-left-radius-values-003.htm
>
>If you look more, you can easily find more like these.
>
>
>> if
>> the
>> thing you're testing isn't supported, the test should always fail.
>> Running
>> such as suite (particularly in an automated engine) would give the
>> impression of partial support for that feature when there is none.
>
>
>Exactly.
>
>
>> For
>> these type of tests that test invalid or incomplete syntax, the
>> reference
>> file should not be designed to render whatever the default behavior is.
>> For this case, however, I (maybe mistakenly) used Tab and Shift+Tab to
>> test directional navigation, which causes them to pass on Chrome, which
>> does not support this, for example.
>>
>> We're in the process of a W3C testing documentation project [1], which
>> will be rolled out sometime soon. When I edit the Review Process doc
>> [2],
>> I can articulate this point there. I will surely solicit your feedback
>> when it's ready.
>>
>>
>>>
>>>Eg.
>>>http://test.csswg.org/harness/results/CSS3-CONDITIONAL_DEV/
>>>shows that IE10 scores 21% but, in reality, the only tests IE10 passes
>>>are those testing invalid syntax causing parsing errors.
>>>
>>>Some CSS2.1 and CSS3 tests will actually be passed by buggy or old
>>>browsers (say, Netscape 4 or IE4). I think this is wrong.
>>
>> We are in agreement. Sounds like there might be some poorly designed
>> tests
>> here.
>>
>>
>>>
>>>-----------
>>>
>>>Opera has now 2 accounts:
>>>
>>>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/oper
>>>a_
>>>software/
>>>
>>>will return "No Assets Found." while
>>>
>>>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/oper
>>>a/
>>>will return "15 Test Cases Found".
>>>
>>>I think there should be only 1 Opera account.
>>>
>>>------------
>>
>> There are many tests under both accounts [3][4],
>
>
>Indeed.
>
>
>> so I'm not sure what we
>> can do about that.
>> It looks like there was one account created through
>> the Shepherd UI under the name "Opera Software ASA" and tests submitted
>> with that author title map to that account. Other tests were submitted
>> under just "Opera Software", which did not match anything so Shepherd
>> created a new account with the underscoring method. You're right, this
>> could be confusing, so I'll check with Peter how to handle this.
>>
>>
>>>
>>>http://test.csswg.org/source/contributors/opera/submitted/css3-ui/nav-di
>>>r-
>>>001.html
>>>
>>>"
>>>First, use directional navigation to navigate the focus to the "START"
>>>link below.
>>>
>>>Test passes if navigating left once moves the focus to the "FINISH"
>>> link.
>>>"
>>>
>>>I have no idea if this is possible... but is it possible to make the
>>>wording of pass-fail conditions sentences in tests like those to be
>>> less
>>>technical? I am asking because I really do not know this.
>>
>> This was my first thought as well, but then I looked at the spec, which
>> explicitly says: "This specification does not define which keys of a
>> device are directional navigational keys,"
>
>Okay.
>
>> so this is really the only
>> way
>> to express the steps here. The steps aren't so much technical as they
>> are
>> vague because so is the spec - intentionally.  One must assume the
>> implementor running these test knows which keys you can use to navigate
>> (though there is a note in the spec stating that arrow keys are a likely
>> choice as well as shift+tab for reverse direction).
>>
>>
>>>
>>>I am using Opera 12.16 and I have no idea what exactly I must do in
>>> such
>>>test.
>>
>> Opera's implementation and how to use it is described here [5].
>
>
>"
>(...)
>Structural elements within a document are also navigable. The previous
>and next header on the page can be found using W and S. The keys E and D
>do the same for text elements.
>
>The keys Q and A allow you to jump to the previous or next link in the
>document. (Ctrl+Down and Ctrl+Up do the same.)
>"
>Use Opera without a mouse: Document Navigation
>http://www.opera.com/help/tutorials/nomouse/#nav
>
>I am still somewhat unsure on how exactly I should be "doing" these
>tests as a tester using Opera 12.16. I have found and checked the box
>marked "Enable single-key shortcuts"... And so Q and A keys navigate, E
>and D keys navigate and Ctrl+Down and Ctrl+Up navigate.
>

Sorry, I realized I didn't respond to this part of your message.  For
navigational direction, use Shift+ arrow keys.

See:

"Spatial navigation allows you to move in any direction to the next link
or form input. To use spatial navigation, press Shift+Down, Shift+Up,
Shift+Left, and Shift+Right, and Opera will select the most appropriate
item in that direction."




>
>> I don't
>> think this specific instruction should be in the test itself though
>> because as I mentioned above the spec makes no keyboard requirements.
>> It
>> might have been nice if this info were provided with the review was
>> solicited, but it was easy enough to find :)
>
>
>As test creator, I would provide such kind of info (the keyboard
>combination for various browsers) inside <!-- comments --> in the source
>code.
>
>
>>
>>
>>
>>>
>>>------------
>>>
>>>Rebecca,
>>>
>>>your comments in
>>>
>>>github.com/w3c/csswg-test/commit
>>>
>>>are not "echoed" into Shepherd system. I believe comments on tests
>>>should not be scattered around.
>>>
>>
>> Yes, I know. When these tests are resubmitted with reviewer links, I
>> will
>> then add a link to the review in Github when I set them to Approved. I
>> figured there would be a fairly quick turnaround with Leif and his team,
>> so I was waiting for their resubmission in the next few days.
>>
>> Keep in mind that Peter is doing work to integrate Shepherd with Github
>> so
>> this will happen automatically when that work is done.  Right now, we're
>> favoring Github's review tools to Shepherd's for many reasons (more
>> widely
>> used and understood, support for inline comments, email notifications,
>> subscriptions, etc.). I'd like to move away from having all of the
>> review
>> comments done on this mailing list, where the comments are so far away
>> from the code.
>>
>> In this interim period, you can continue to do things as you have been.
>> When we're fully integrated with Github (and with the rest of the w3c
>> test
>> suites), I'll propose this change in process and make sure it's
>> documented
>> in the 'Reviewing Tests' doc.
>>
>
>Oh well... I like using Shepherd ... but anyway...
>
>Gérard
>
>>
>> [1] https://github.com/w3c/testtwf-website/issues?milestone=1&state=open
>> (note that the closed issues are in the process
>> [2] https://github.com/w3c/testtwf-website/issues/13
>> [3] http://test.csswg.org/shepherd/search/author/opera_software/
>> [4] http://test.csswg.org/shepherd/search/author/opera/
>> [5] http://www.opera.com/help/tutorials/nomouse/
>
>-- 
>Contributions to the CSS 2.1 test suite:
>http://www.gtalbot.org/BrowserBugsSection/css21testsuite/
>
>CSS 2.1 Test suite RC6, March 23rd 2011:
>http://test.csswg.org/suites/css2.1/20110323/html4/toc.html
>
>CSS 2.1 test suite harness:
>http://test.csswg.org/harness/
>
>Contributing to to CSS 2.1 test suite:
>http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contr
>ibutions-css21-testsuite.html
>

Received on Wednesday, 17 July 2013 18:35:29 UTC