- From: Gérard Talbot <css21testsuite@gtalbot.org>
- Date: Tue, 16 Jul 2013 22:45:24 -0400
- To: "Rebecca Hauck" <rhauck@adobe.com>
- Cc: "Leif Arne Storset" <lstorset@opera.com>, "public-css-testsuite@w3.org" <public-css-testsuite@w3.org>, "Tantek Çelik" <tantek@cs.stanford.edu>, "Jorrit Vermeiren" <jorritv@opera.com>, "Giuseppe Pascale" <giuseppep@opera.com>
Le Mar 16 juillet 2013 21:04, Rebecca Hauck a écrit : > Hi Gérard, > > > On 7/16/13 4:42 PM, ""Gérard Talbot"" <css21testsuite@gtalbot.org> > wrote: > >> >>Le Mar 16 juillet 2013 16:24, Rebecca Hauck a écrit : >>> Hi Leif, >>> >>> I took a look and added comments on the commit in Github: >>> >>> >>>https://github.com/w3c/csswg-test/commit/f394a9f65071d53b4f5b74d5dd8b14d9 >>>d0 >>> 96f2e6 >>> >>> >>> Other than the few minor changes I suggested, they look good. >>> >>> Cheers, >>> -Rebecca >> >>Rebecca, >> >>" >>This test passes for me on a UA that does not support this property. >> You >>can add an intermediate step to first verify the property is supported. >>" >>contributors/opera/submitted/css3-ui/nav-dir-target-006.html >> >>I have in the past also criticized some tests for not being designed to >>fail when the test is testing an invalid syntax (or invalid >> declaration) >>and when an UA does not support a property. I think this issue should >> be >>clarified. > > What exactly should be clarified? I think it should be stated in the Design Requirements section of Test Format http://wiki.csswg.org/test/format#design-requirements that if the [property or declaration or feature] being tested isn't supported, then the test should always fail. Right now, there is only "The test will not pass inadvertently." in CSS Test Review Checklist, Test Design http://wiki.csswg.org/test/css2.1/review-checklist#test-design which is a not-so-well-known wiki page. > I think that for any spec test, if > the > thing you're testing isn't supported, the test should always fail. In the past, Florian and I had an email exchange on this precise issue http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0014.html http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0017.html http://lists.w3.org/Archives/Public/public-css-testsuite/2012Aug/0020.html and I think the Test Format Guidelines http://wiki.csswg.org/test/format should explicitly cover such test design situation. Because there is a minority of tests that will "pass" when the UA does not support the [property or declaration or feature] being tested. Eg. UAs that do not support border-radius will pass these 8 tests http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-009.htm http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-010.htm http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-012.htm http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-013.htm http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-014.htm http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-015.htm http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-radius-applies-to-016.htm http://test.csswg.org/source/contributors/microsoft/submitted/CSS3/background/border-top-left-radius-values-003.htm If you look more, you can easily find more like these. > if > the > thing you're testing isn't supported, the test should always fail. > Running > such as suite (particularly in an automated engine) would give the > impression of partial support for that feature when there is none. Exactly. > For > these type of tests that test invalid or incomplete syntax, the > reference > file should not be designed to render whatever the default behavior is. > For this case, however, I (maybe mistakenly) used Tab and Shift+Tab to > test directional navigation, which causes them to pass on Chrome, which > does not support this, for example. > > We're in the process of a W3C testing documentation project [1], which > will be rolled out sometime soon. When I edit the Review Process doc > [2], > I can articulate this point there. I will surely solicit your feedback > when it's ready. > > >> >>Eg. >>http://test.csswg.org/harness/results/CSS3-CONDITIONAL_DEV/ >>shows that IE10 scores 21% but, in reality, the only tests IE10 passes >>are those testing invalid syntax causing parsing errors. >> >>Some CSS2.1 and CSS3 tests will actually be passed by buggy or old >>browsers (say, Netscape 4 or IE4). I think this is wrong. > > We are in agreement. Sounds like there might be some poorly designed > tests > here. > > >> >>----------- >> >>Opera has now 2 accounts: >> >>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/opera_ >>software/ >> >>will return "No Assets Found." while >> >>http://test.csswg.org/shepherd/search/testcase/spec/css-ui-3/author/opera/ >>will return "15 Test Cases Found". >> >>I think there should be only 1 Opera account. >> >>------------ > > There are many tests under both accounts [3][4], Indeed. > so I'm not sure what we > can do about that. > It looks like there was one account created through > the Shepherd UI under the name "Opera Software ASA" and tests submitted > with that author title map to that account. Other tests were submitted > under just "Opera Software", which did not match anything so Shepherd > created a new account with the underscoring method. You're right, this > could be confusing, so I'll check with Peter how to handle this. > > >> >>http://test.csswg.org/source/contributors/opera/submitted/css3-ui/nav-dir- >>001.html >> >>" >>First, use directional navigation to navigate the focus to the "START" >>link below. >> >>Test passes if navigating left once moves the focus to the "FINISH" >> link. >>" >> >>I have no idea if this is possible... but is it possible to make the >>wording of pass-fail conditions sentences in tests like those to be >> less >>technical? I am asking because I really do not know this. > > This was my first thought as well, but then I looked at the spec, which > explicitly says: "This specification does not define which keys of a > device are directional navigational keys," Okay. > so this is really the only > way > to express the steps here. The steps aren't so much technical as they > are > vague because so is the spec - intentionally. One must assume the > implementor running these test knows which keys you can use to navigate > (though there is a note in the spec stating that arrow keys are a likely > choice as well as shift+tab for reverse direction). > > >> >>I am using Opera 12.16 and I have no idea what exactly I must do in >> such >>test. > > Opera's implementation and how to use it is described here [5]. " (...) Structural elements within a document are also navigable. The previous and next header on the page can be found using W and S. The keys E and D do the same for text elements. The keys Q and A allow you to jump to the previous or next link in the document. (Ctrl+Down and Ctrl+Up do the same.) " Use Opera without a mouse: Document Navigation http://www.opera.com/help/tutorials/nomouse/#nav I am still somewhat unsure on how exactly I should be "doing" these tests as a tester using Opera 12.16. I have found and checked the box marked "Enable single-key shortcuts"... And so Q and A keys navigate, E and D keys navigate and Ctrl+Down and Ctrl+Up navigate. > I don't > think this specific instruction should be in the test itself though > because as I mentioned above the spec makes no keyboard requirements. > It > might have been nice if this info were provided with the review was > solicited, but it was easy enough to find :) As test creator, I would provide such kind of info (the keyboard combination for various browsers) inside <!-- comments --> in the source code. > > > >> >>------------ >> >>Rebecca, >> >>your comments in >> >>github.com/w3c/csswg-test/commit >> >>are not "echoed" into Shepherd system. I believe comments on tests >>should not be scattered around. >> > > Yes, I know. When these tests are resubmitted with reviewer links, I > will > then add a link to the review in Github when I set them to Approved. I > figured there would be a fairly quick turnaround with Leif and his team, > so I was waiting for their resubmission in the next few days. > > Keep in mind that Peter is doing work to integrate Shepherd with Github > so > this will happen automatically when that work is done. Right now, we're > favoring Github's review tools to Shepherd's for many reasons (more > widely > used and understood, support for inline comments, email notifications, > subscriptions, etc.). I'd like to move away from having all of the > review > comments done on this mailing list, where the comments are so far away > from the code. > > In this interim period, you can continue to do things as you have been. > When we're fully integrated with Github (and with the rest of the w3c > test > suites), I'll propose this change in process and make sure it's > documented > in the 'Reviewing Tests' doc. > Oh well... I like using Shepherd ... but anyway... Gérard > > [1] https://github.com/w3c/testtwf-website/issues?milestone=1&state=open > (note that the closed issues are in the process > [2] https://github.com/w3c/testtwf-website/issues/13 > [3] http://test.csswg.org/shepherd/search/author/opera_software/ > [4] http://test.csswg.org/shepherd/search/author/opera/ > [5] http://www.opera.com/help/tutorials/nomouse/ -- Contributions to the CSS 2.1 test suite: http://www.gtalbot.org/BrowserBugsSection/css21testsuite/ CSS 2.1 Test suite RC6, March 23rd 2011: http://test.csswg.org/suites/css2.1/20110323/html4/toc.html CSS 2.1 test suite harness: http://test.csswg.org/harness/ Contributing to to CSS 2.1 test suite: http://www.gtalbot.org/BrowserBugsSection/css21testsuite/web-authors-contributions-css21-testsuite.html
Received on Wednesday, 17 July 2013 02:46:02 UTC