Re: [css3-ui] Tests submitted for directional focus navigation properties

Hi Gérard, 


On 7/16/13 4:42 PM, ""Gérard Talbot"" <css21testsuite@gtalbot.org> wrote:

>
>Le Mar 16 juillet 2013 16:24, Rebecca Hauck a écrit :
>> Hi Leif,
>>
>> I took a look and added comments on the commit in Github:
>>
>> 
>>https://github.com/w3c/csswg-test/commit/f394a9f65071d53b4f5b74d5dd8b14d9
>>d0
>> 96f2e6
>>
>>
>> Other than the few minor changes I suggested, they look good.
>>
>> Cheers,
>> -Rebecca
>
>Rebecca,
>
>"
>This test passes for me on a UA that does not support this property. You
>can add an intermediate step to first verify the property is supported.
>"
>contributors/opera/submitted/css3-ui/nav-dir-target-006.html
>
>I have in the past also criticized some tests for not being designed to
>fail when the test is testing an invalid syntax (or invalid declaration)
>and when an UA does not support a property. I think this issue should be
>clarified.

What exactly should be clarified?  I think that for any spec test, if the
thing you're testing isn't supported, the test should always fail. Running
such as suite (particularly in an automated engine) would give the
impression of partial support for that feature when there is none. For
these type of tests that test invalid or incomplete syntax, the reference
file should not be designed to render whatever the default behavior is.
For this case, however, I (maybe mistakenly) used Tab and Shift+Tab to
test directional navigation, which causes them to pass on Chrome, which
does not support this, for example.

We're in the process of a W3C testing documentation project [1], which
will be rolled out sometime soon. When I edit the Review Process doc [2],
I can articulate this point there. I will surely solicit your feedback
when it's ready.


>
>Eg.
>http://test.csswg.org/harness/results/CSS3-CONDITIONAL_DEV/
>shows that IE10 scores 21% but, in reality, the only tests IE10 passes
>are those testing invalid syntax causing parsing errors.
>
>Some CSS2.1 and CSS3 tests will actually be passed by buggy or old
>browsers (say, Netscape 4 or IE4). I think this is wrong.

We are in agreement. Sounds like there might be some poorly designed tests
here.


>
>-----------
>
>Opera has now 2 accounts:
>
>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/opera_
>software/
>
>will return "No Assets Found." while
>
>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/opera/
>will return "15 Test Cases Found".
>
>I think there should be only 1 Opera account.
>
>------------

There are many tests under both accounts [3][4], so I'm not sure what we
can do about that.  It looks like there was one account created through
the Shepherd UI under the name "Opera Software ASA" and tests submitted
with that author title map to that account. Other tests were submitted
under just "Opera Software", which did not match anything so Shepherd
created a new account with the underscoring method. You're right, this
could be confusing, so I'll check with Peter how to handle this.


>
>http://test.csswg.org/source/contributors/opera/submitted/css3-ui/nav-dir-
>001.html
>
>"
>First, use directional navigation to navigate the focus to the "START"
>link below.
>
>Test passes if navigating left once moves the focus to the "FINISH" link.
>"
>
>I have no idea if this is possible... but is it possible to make the
>wording of pass-fail conditions sentences in tests like those to be less
>technical? I am asking because I really do not know this.

This was my first thought as well, but then I looked at the spec, which
explicitly says: "This specification does not define which keys of a
device are directional navigational keys," so this is really the only way
to express the steps here. The steps aren't so much technical as they are
vague because so is the spec - intentionally.  One must assume the
implementor running these test knows which keys you can use to navigate
(though there is a note in the spec stating that arrow keys are a likely
choice as well as shift+tab for reverse direction).


>
>I am using Opera 12.16 and I have no idea what exactly I must do in such
>test.

Opera's implementation and how to use it is described here [5]. I don't
think this specific instruction should be in the test itself though
because as I mentioned above the spec makes no keyboard requirements.  It
might have been nice if this info were provided with the review was
solicited, but it was easy enough to find :)



>
>------------
>
>Rebecca,
>
>your comments in
>
>github.com/w3c/csswg-test/commit
>
>are not "echoed" into Shepherd system. I believe comments on tests
>should not be scattered around.
>

Yes, I know. When these tests are resubmitted with reviewer links, I will
then add a link to the review in Github when I set them to Approved. I
figured there would be a fairly quick turnaround with Leif and his team,
so I was waiting for their resubmission in the next few days.

Keep in mind that Peter is doing work to integrate Shepherd with Github so
this will happen automatically when that work is done.  Right now, we're
favoring Github's review tools to Shepherd's for many reasons (more widely
used and understood, support for inline comments, email notifications,
subscriptions, etc.). I'd like to move away from having all of the review
comments done on this mailing list, where the comments are so far away
from the code.  

In this interim period, you can continue to do things as you have been.
When we're fully integrated with Github (and with the rest of the w3c test
suites), I'll propose this change in process and make sure it's documented
in the 'Reviewing Tests' doc.


[1] https://github.com/w3c/testtwf-website/issues?milestone=1&state=open
(note that the closed issues are in the process
[2] https://github.com/w3c/testtwf-website/issues/13
[3] http://test.csswg.org/shepherd/search/author/opera_software/
[4] http://test.csswg.org/shepherd/search/author/opera/
[5] http://www.opera.com/help/tutorials/nomouse/

Received on Wednesday, 17 July 2013 01:04:10 UTC