Re: [for review] ACT Review Process

Hi ACT,


I know there was mention on the call today that we might put the rules / test cases out for review for 2 weeks and if someone doesn't raise the flag on an issue, that it would be accepted. (At least I think that is what I heard, apologies if I'm misinterpreting). My question is whether or not a rule/ test case would have to have a certain number of test cases in place in order for it to be raised as a possible rule / test case. Sorry if I'm missing that somewhere.



In a related question : Should the requirements section for ACT Review Process state that the test cases should show one passable (passed) test and one negative (failed) test for a particular rule / test case? I reviewed the ACT rules format outline section 5.2 Test Cases and it mentions : "test case result indicating whether test case passed or failed." I didn't see anything on whether the test case / rule should have an example of a pass and a fail within that rule set though. Would it be good for the auditor (or whoever is using this rule set / test case) to have a positive and a negative example?



Separate question:


At what point would we stop submission of a set rule / test case ? Is there a cut off date for submitting rule? I know that per the "acceptance" that the ACT test rules will be updated periodically with the techniques and understanding documents (currently every 6 months). If a rule was not accepted, but was in queue, will we continue reviewing that potential rule / test case with that queue moving forward on the next iteration in terms of acceptance? I understand the goal is to build a never ending repository of tests as technology changes.



Just a couple of thoughts, which are probably already documented somewhere in our WCAG / ACT documentation...

Thank you,
Chris Loiselle

On Jun 26, 2017, at 09:40 AM, Wilco Fiers <wilco.fiers@deque.com> wrote:


Hey ACT'ers
In advance of our meeting in a little while, here is some of the feedback I got from Deque:


1) We should break each submission into a single request - mixing too many rules into a single request makes the feedback and decision making difficult - even when tools like GitHub are used.
2) What about best practice rules? E.g. you don't have to have an H1 but you really should try.
3) X should be 2 under implementations. Also - there should be a condition of two implementations being shown to work before a rule makes it into a CR
4) What about false positives? How do we ensure that rules do not make it into the fully automated portion of this spec that generate false positives?
5) contributions from Deque staff that could impact axe-core would require an internal review before posting.



Wilco




On Thu, Jun 22, 2017 at 4:47 PM, Detlev Fischer <detlev.fischer@testkreis.de> wrote:

> I think these test cases may become very useful when related test rules
> are contributed. Do you plan to make this fully public at a later time?

Yes. The COMPARE repository will be read-only for the general public and
read/write for experts/testers (I repeat my invitation to get signed up right
now!)

--
Detlev Fischer
testkreis c/o feld.wald.wiese
Thedestr. 2, 22767 Hamburg

Mobil +49 (0)157 57 57 57 45
Fax +49 (0)40 439 10 68-5

http://www.testkreis.de
Beratung, Tests und Schulungen für barrierefreie Websites


Shadi Abou-Zahra schrieb am 22.06.2017 14:40:

> Hi Detlev,
>
> On 20/06/2017 11:00, Detlev Fischer wrote:
>> Hi Shadi, ACT TF
>> I had a look at the review Process document. The basic problem for me is to
>> understand how the process (submission of a rule backed by supporting test
>> cases) would work in practice, so I would think it is worthwhile taking one or
>> a few non-trivial examples of real web content and looking what the rule(s)
>> might look like that would be supportive when deciding about conformance. THis
>> exercise would show the uninitiated how it's going to play out.
>
> Agree. I believe sample rules that comply with the current Rules Format
> specification are being developed. We can also use them for trying out
> and refining this review process.
>
>
>> A complex and at the same time very frequent example might be something like
>> drop-down navigation menus (take for example a recent discussion between Matt
>> King and Mallory on the Webaim list - the tail end is here
>> http://webaim.org/discussion/mail_message?id=34968 )
>>
>> What would rules look like that help me establish whether some menu conforms
>> to 1.3.1, 2.1.1, 4.1.2, 2.4.3 etc? How can the rule be isolated from content
>> aspects that may co-determine whether we think of some solution as acceptable
>> or not (take the length of the submenus in cases where they are opened
>> automaticlly when focused)? When does the aria menu pattern apply, and what
>> deviations of the pattern are OK (conform) in what contexts?
>>
>> We all know the difficulty of attributing an issue to the right SC - when an
>> element does not get tab focus but you CAN activate it when arrowing there,
>> does it violate 2.1.1? Or only 2.4.3? If a main menu item opens the submenu
>> and a second activation does not close it but goes to a section page, is that
>> a usability issue or necessarily a fail of some SC? Etc, etc...
>>
>> So I believe working through a few practical real world implementations and
>> showing how the ACT framework would support developers / testers in assessing
>> real-world implementations would really help making the ACT activity a lot
>> more tangible (it often feels quite abstract to me).
>
> Agree. Though this is slightly orthogonal to the review process itself.
>
>
>> Finally, an invitation: ACT TF members wanting a test login to our COMPARE
>> repository ( http://www.funka.com/en/projekt/compare/ ) are welcome - just
>> give me a shout. The repository is early days, not yet in its final shape, and
>> not yet public but it already has a few real world cases with accessiblility
>> ratings. Should you want to add your rating, the comment field would give
>> scope to outline the rules according to which someone has arrived at a PASS or
>> FAIL conclusion. As a contributor of ratings you will be picking the SC (or
>> multiple SCs) that you think should fail (or pass with comment).
>
> I think these test cases may become very useful when related test rules
> are contributed. Do you plan to make this fully public at a later time?
>
> Best,
>   Shadi
>
>
>> Best,
>> Detlev
>>
>>
>>
>> --
>> Detlev Fischer
>> testkreis c/o feld.wald.wiese
>> Thedestr. 2, 22767 Hamburg
>>
>> Mobil +49 (0)157 57 57 57 45
>> Fax +49 (0)40 439 10 68-5
>>
>> http://www.testkreis.de
>> Beratung, Tests und Schulungen für barrierefreie Websites
>>
>> Shadi Abou-Zahra schrieb am 19.06.2017 19:54:
>>
>>> Dear ACT TF,
>>>
>>> Ref:
>>> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/ACT_Review_Process
>>>
>>> As discussed during the call today, please review the outline for the
>>> proposed ACT Review Process. Feel free to add your feedback to the wiki
>>> discussion tab or by email.
>>>
>>> Regards,
>>>    Shadi
>>>
>>> --
>>> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
>>> Accessibility Strategy and Technology Specialist
>>> Web Accessibility Initiative (WAI)
>>> World Wide Web Consortium (W3C)
>>>
>>
>>
>
> --
> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
> Accessibility Strategy and Technology Specialist
> Web Accessibility Initiative (WAI)
> World Wide Web Consortium (W3C)
>








--

Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG