Re: [css3-ui] Tests submitted for directional focus navigation properties

On 7/16/13 7:45 PM, ""Gérard Talbot"" <css21testsuite@gtalbot.org> wrote:

>
>Le Mar 16 juillet 2013 21:04, Rebecca Hauck a écrit :
>> Hi Gérard,
>>
>>
>> On 7/16/13 4:42 PM, ""Gérard Talbot"" <css21testsuite@gtalbot.org>
>> wrote:
>>
>>>
>>>Le Mar 16 juillet 2013 16:24, Rebecca Hauck a écrit :
>>>> Hi Leif,
>>>>
>>>> I took a look and added comments on the commit in Github:
>>>>
>>>>
>>>>https://github.com/w3c/csswg-test/commit/f394a9f65071d53b4f5b74d5dd8b14
>>>>d9
>>>>d0
>>>> 96f2e6
>>>>
>>>>
>>>> Other than the few minor changes I suggested, they look good.
>>>>
>>>> Cheers,
>>>> -Rebecca
>>>
>>>Rebecca,
>>>
>>>"
>>>This test passes for me on a UA that does not support this property.
>>> You
>>>can add an intermediate step to first verify the property is supported.
>>>"
>>>contributors/opera/submitted/css3-ui/nav-dir-target-006.html
>>>
>>>I have in the past also criticized some tests for not being designed to
>>>fail when the test is testing an invalid syntax (or invalid
>>> declaration)
>>>and when an UA does not support a property. I think this issue should
>>> be
>>>clarified.
>>
>> What exactly should be clarified?
>
>
>I think it should be stated in the Design Requirements section of Test
>Format
>http://wiki.csswg.org/test/format#design-requirements
>that if the [property or declaration or feature] being tested isn't
>supported, then the test should always fail.
>
>Right now, there is only "The test will not pass inadvertently." in
>CSS Test Review Checklist, Test Design
>http://wiki.csswg.org/test/css2.1/review-checklist#test-design
>which is a not-so-well-known wiki page.
>
>> I think that for any spec test, if
>> the
>> thing you're testing isn't supported, the test should always fail.
>
>In the past, Florian and I had an email exchange on this precise issue
>
>http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0014.html
>
>http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0017.html
>
>http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0020.html
>
>and I think the Test Format Guidelines
>
>http://wiki.csswg.org/test/format
>
>should explicitly cover such test design situation. Because there is a
>minority of tests that will "pass" when the UA does not support the
>[property or declaration or feature] being tested.
>
>Eg. UAs that do not support border-radius will pass these 8 tests
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-009.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-010.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-012.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-013.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-014.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-015.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-radius-applies-to-016.htm
>
>http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/backgro
>und/border-top-left-radius-values-003.htm
>
>If you look more, you can easily find more like these.
>
>
>> if
>> the
>> thing you're testing isn't supported, the test should always fail.
>> Running
>> such as suite (particularly in an automated engine) would give the
>> impression of partial support for that feature when there is none.
>
>
>Exactly.
>
>
>> For
>> these type of tests that test invalid or incomplete syntax, the
>> reference
>> file should not be designed to render whatever the default behavior is.
>> For this case, however, I (maybe mistakenly) used Tab and Shift+Tab to
>> test directional navigation, which causes them to pass on Chrome, which
>> does not support this, for example.
>>
>> We're in the process of a W3C testing documentation project [1], which
>> will be rolled out sometime soon. When I edit the Review Process doc
>> [2],
>> I can articulate this point there. I will surely solicit your feedback
>> when it's ready.
>>
>>
>>>
>>>Eg.
>>>http://test.csswg.org/harness/results/CSS3-CONDITIONAL_DEV/
>>>shows that IE10 scores 21% but, in reality, the only tests IE10 passes
>>>are those testing invalid syntax causing parsing errors.
>>>
>>>Some CSS2.1 and CSS3 tests will actually be passed by buggy or old
>>>browsers (say, Netscape 4 or IE4). I think this is wrong.
>>
>> We are in agreement. Sounds like there might be some poorly designed
>> tests
>> here.
>>
>>
>>>
>>>-----------
>>>
>>>Opera has now 2 accounts:
>>>
>>>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/oper
>>>a_
>>>software/
>>>
>>>will return "No Assets Found." while
>>>
>>>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/oper
>>>a/
>>>will return "15 Test Cases Found".
>>>
>>>I think there should be only 1 Opera account.
>>>
>>>------------
>>
>> There are many tests under both accounts [3][4],
>
>
>Indeed.
>
>
>> so I'm not sure what we
>> can do about that.
>> It looks like there was one account created through
>> the Shepherd UI under the name "Opera Software ASA" and tests submitted
>> with that author title map to that account. Other tests were submitted
>> under just "Opera Software", which did not match anything so Shepherd
>> created a new account with the underscoring method. You're right, this
>> could be confusing, so I'll check with Peter how to handle this.
>>
>>
>>>
>>>http://test.csswg.org/source/contributors/opera/submitted/css3-ui/nav-di
>>>r-
>>>001.html
>>>
>>>"
>>>First, use directional navigation to navigate the focus to the "START"
>>>link below.
>>>
>>>Test passes if navigating left once moves the focus to the "FINISH"
>>> link.
>>>"
>>>
>>>I have no idea if this is possible... but is it possible to make the
>>>wording of pass-fail conditions sentences in tests like those to be
>>> less
>>>technical? I am asking because I really do not know this.
>>
>> This was my first thought as well, but then I looked at the spec, which
>> explicitly says: "This specification does not define which keys of a
>> device are directional navigational keys,"
>
>Okay.
>
>> so this is really the only
>> way
>> to express the steps here. The steps aren't so much technical as they
>> are
>> vague because so is the spec - intentionally.  One must assume the
>> implementor running these test knows which keys you can use to navigate
>> (though there is a note in the spec stating that arrow keys are a likely
>> choice as well as shift+tab for reverse direction).
>>
>>
>>>
>>>I am using Opera 12.16 and I have no idea what exactly I must do in
>>> such
>>>test.
>>
>> Opera's implementation and how to use it is described here [5].
>
>
>"
>(...)
>Structural elements within a document are also navigable. The previous
>and next header on the page can be found using W and S. The keys E and D
>do the same for text elements.
>
>The keys Q and A allow you to jump to the previous or next link in the
>document. (Ctrl+Down and Ctrl+Up do the same.)
>"
>Use Opera without a mouse: Document Navigation
>http://www.opera.com/help/tutorials/nomouse/#nav
>
>I am still somewhat unsure on how exactly I should be "doing" these
>tests as a tester using Opera 12.16. I have found and checked the box
>marked "Enable single-key shortcuts"... And so Q and A keys navigate, E
>and D keys navigate and Ctrl+Down and Ctrl+Up navigate.
>
>
>> I don't
>> think this specific instruction should be in the test itself though
>> because as I mentioned above the spec makes no keyboard requirements.
>> It
>> might have been nice if this info were provided with the review was
>> solicited, but it was easy enough to find :)
>
>
>As test creator, I would provide such kind of info (the keyboard
>combination for various browsers) inside <!-- comments --> in the source
>code.
>
>
>>
>>
>>
>>>
>>>------------
>>>
>>>Rebecca,
>>>
>>>your comments in
>>>
>>>github.com/w3c/csswg-test/commit
>>>
>>>are not "echoed" into Shepherd system. I believe comments on tests
>>>should not be scattered around.
>>>
>>
>> Yes, I know. When these tests are resubmitted with reviewer links, I
>> will
>> then add a link to the review in Github when I set them to Approved. I
>> figured there would be a fairly quick turnaround with Leif and his team,
>> so I was waiting for their resubmission in the next few days.
>>
>> Keep in mind that Peter is doing work to integrate Shepherd with Github
>> so
>> this will happen automatically when that work is done.  Right now, we're
>> favoring Github's review tools to Shepherd's for many reasons (more
>> widely
>> used and understood, support for inline comments, email notifications,
>> subscriptions, etc.). I'd like to move away from having all of the
>> review
>> comments done on this mailing list, where the comments are so far away
>> from the code.
>>
>> In this interim period, you can continue to do things as you have been.
>> When we're fully integrated with Github (and with the rest of the w3c
>> test
>> suites), I'll propose this change in process and make sure it's
>> documented
>> in the 'Reviewing Tests' doc.
>>
>
>Oh well... I like using Shepherd ... but anyway...

We'll continue to use Shepherd (and I like it, too!), but we're trying to
make contribution--both authoring and reviewing-- easier and more
consistent across the W3C. The other WGs have already fully adopted the
github process and as we've seen at the past TestTWF events, it's working
well.  Shepherd give us a nice additional layer that will only get better
when it becomes integrated with github.

Received on Wednesday, 17 July 2013 17:07:56 UTC