- From: Christopher Loiselle <loiselles@me.com>
- Date: Mon, 26 Jun 2017 17:15:31 +0000 (GMT)
- To: Shadi Abou-Zahra <shadi@w3.org>
- Cc: Accessibility Conformance Testing <public-wcag-act@w3.org>
- Message-id: <f78da1d2-45c1-4254-9bd5-41e69506e846@me.com>
Hi Shadi, I've answered in line. I'll think more on the outstanding questions as well. Thank you, Chris Loiselle On Jun 26, 2017, at 01:04 PM, Shadi Abou-Zahra <shadi@w3.org> wrote: Hi Chris, On 26/06/2017 17:58, Christopher Loiselle wrote: Hi ACT, I know there was mention on the call today that we might put the rules / test cases out for review for 2 weeks and if someone doesn't raise the flag on an issue, that it would be accepted. (At least I think that is what I heard, apologies if I'm misinterpreting). My question is whether or not a rule/ test case would have to have a certain number of test cases in place in order for it to be raised as a possible rule / test case. Sorry if I'm missing that somewhere. I think both ideas were raised. They could also be combined - for example, when a test rule has X number of test cases it announced for review for Z days. What would be your suggestion? How many test cases? I like the idea of having both. I don't think we should limit the amount of test cases, but the number of days should be set , as not to exceed that number so we can move to the next test rule in the queue. I do think that having a minimum of one positive and one negative test would help though. In a related question : Should the requirements section for ACT Review Process state that the test cases should show one passable (passed) test and one negative (failed) test for a particular rule / test case? I reviewed the ACT rules format outline section 5.2 Test Cases and it mentions : "test case result indicating whether test case passed or failed." I didn't see anything on whether the test case / rule should have an example of a pass and a fail within that rule set though. Would it be good for the auditor (or whoever is using this rule set / test case) to have a positive and a negative example? This would help catch false positives and false negatives. But it would also add burden on test rule developers. What would you suggest? I will think more on this and add my thoughts as they come :) Separate question: At what point would we stop submission of a set rule / test case ? Is there a cut off date for submitting rule? I know that per the "acceptance" that the ACT test rules will be updated periodically with the techniques and understanding documents (currently every 6 months). If a rule was not accepted, but was in queue, will we continue reviewing that potential rule / test case with that queue moving forward on the next iteration in terms of acceptance? I understand the goal is to build a never ending repository of tests as technology changes. Good question. Currently the Techniques are published every six months. This may be a good trigger for also updating the Test Rules. We will need to discuss that with the AG WG if they can offer other options. Just a couple of thoughts, which are probably already documented somewhere in our WCAG / ACT documentation... Excellent thoughts! They are not yet documented, as we are working out these exact questions. This is also why I'm very rudely answering your questions with questions back... ;-) Not rude at all! Positive feedback and questions make the process better. Thanks for the reply. Best, Shadi Thank you, Chris Loiselle On Jun 26, 2017, at 09:40 AM, Wilco Fiers <wilco.fiers@deque.com> wrote: Hey ACT'ers In advance of our meeting in a little while, here is some of the feedback I got from Deque: 1) We should break each submission into a single request - mixing too many rules into a single request makes the feedback and decision making difficult - even when tools like GitHub are used. 2) What about best practice rules? E.g. you don't have to have an H1 but you really should try. 3) X should be 2 under implementations. Also - there should be a condition of two implementations being shown to work before a rule makes it into a CR 4) What about false positives? How do we ensure that rules do not make it into the fully automated portion of this spec that generate false positives? 5) contributions from Deque staff that could impact axe-core would require an internal review before posting. Wilco On Thu, Jun 22, 2017 at 4:47 PM, Detlev Fischer <detlev.fischer@testkreis.de> wrote: I think these test cases may become very useful when related test rules are contributed. Do you plan to make this fully public at a later time? Yes. The COMPARE repository will be read-only for the general public and read/write for experts/testers (I repeat my invitation to get signed up right now!) -- Detlev Fischer testkreis c/o feld.wald.wiese Thedestr. 2, 22767 Hamburg Mobil +49 (0)157 57 57 57 45 Fax +49 (0)40 439 10 68-5 http://www.testkreis.de Beratung, Tests und Schulungen für barrierefreie Websites Shadi Abou-Zahra schrieb am 22.06.2017 14:40: Hi Detlev, On 20/06/2017 11:00, Detlev Fischer wrote: Hi Shadi, ACT TF I had a look at the review Process document. The basic problem for me is to understand how the process (submission of a rule backed by supporting test cases) would work in practice, so I would think it is worthwhile taking one or a few non-trivial examples of real web content and looking what the rule(s) might look like that would be supportive when deciding about conformance. THis exercise would show the uninitiated how it's going to play out. Agree. I believe sample rules that comply with the current Rules Format specification are being developed. We can also use them for trying out and refining this review process. A complex and at the same time very frequent example might be something like drop-down navigation menus (take for example a recent discussion between Matt King and Mallory on the Webaim list - the tail end is here http://webaim.org/discussion/mail_message?id=34968 ) What would rules look like that help me establish whether some menu conforms to 1.3.1, 2.1.1, 4.1.2, 2.4.3 etc? How can the rule be isolated from content aspects that may co-determine whether we think of some solution as acceptable or not (take the length of the submenus in cases where they are opened automaticlly when focused)? When does the aria menu pattern apply, and what deviations of the pattern are OK (conform) in what contexts? We all know the difficulty of attributing an issue to the right SC - when an element does not get tab focus but you CAN activate it when arrowing there, does it violate 2.1.1? Or only 2.4.3? If a main menu item opens the submenu and a second activation does not close it but goes to a section page, is that a usability issue or necessarily a fail of some SC? Etc, etc... So I believe working through a few practical real world implementations and showing how the ACT framework would support developers / testers in assessing real-world implementations would really help making the ACT activity a lot more tangible (it often feels quite abstract to me). Agree. Though this is slightly orthogonal to the review process itself. Finally, an invitation: ACT TF members wanting a test login to our COMPARE repository ( http://www.funka.com/en/projekt/compare/ ) are welcome - just give me a shout. The repository is early days, not yet in its final shape, and not yet public but it already has a few real world cases with accessiblility ratings. Should you want to add your rating, the comment field would give scope to outline the rules according to which someone has arrived at a PASS or FAIL conclusion. As a contributor of ratings you will be picking the SC (or multiple SCs) that you think should fail (or pass with comment). I think these test cases may become very useful when related test rules are contributed. Do you plan to make this fully public at a later time? Best, Shadi Best, Detlev -- Detlev Fischer testkreis c/o feld.wald.wiese Thedestr. 2, 22767 Hamburg Mobil +49 (0)157 57 57 57 45 Fax +49 (0)40 439 10 68-5 http://www.testkreis.de Beratung, Tests und Schulungen für barrierefreie Websites Shadi Abou-Zahra schrieb am 19.06.2017 19:54: Dear ACT TF, Ref: https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/ACT_Review_Process As discussed during the call today, please review the outline for the proposed ACT Review Process. Feel free to add your feedback to the wiki discussion tab or by email. Regards, Shadi -- Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Accessibility Strategy and Technology Specialist Web Accessibility Initiative (WAI) World Wide Web Consortium (W3C) -- Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Accessibility Strategy and Technology Specialist Web Accessibility Initiative (WAI) World Wide Web Consortium (W3C) -- Wilco Fiers Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG -- Shadi Abou-Zahra - http://www.w3.org/People/shadi/ Accessibility Strategy and Technology Specialist Web Accessibility Initiative (WAI) World Wide Web Consortium (W3C)
Received on Monday, 26 June 2017 17:16:21 UTC