Re: [for review] ACT Review Process

Hi Shadi,



I've answered in line. I'll think more on the outstanding questions as well.

Thank you,
Chris Loiselle

On Jun 26, 2017, at 01:04 PM, Shadi Abou-Zahra <shadi@w3.org> wrote:


Hi Chris,

On 26/06/2017 17:58, Christopher Loiselle wrote:

Hi ACT,




I know there was mention on the call today that we might put the rules /
test cases out for review for 2 weeks and if someone doesn't raise the
flag on an issue, that it would be accepted. (At least I think that is
what I heard, apologies if I'm misinterpreting). My question is whether
or not a rule/ test case would have to have a certain number of test
cases in place in order for it to be raised as a possible rule / test
case. Sorry if I'm missing that somewhere.

I think both ideas were raised. They could also be combined - for
example, when a test rule has X number of test cases it announced for
review for Z days. What would be your suggestion? How many test cases?



I like the idea of having both. I don't think we should limit the amount of test cases, but the number of days should be set , as not to exceed that number so we can move to the next test rule in the queue.  I do think that having a minimum of one positive and one negative test would help though.




In a related question : Should the requirements section for ACT Review
Process state that the test cases should show one passable (passed) test
and one negative (failed) test for a particular rule / test case? I
reviewed the ACT rules format outline section 5.2 Test Cases and it
mentions : "test case result indicating whether test case passed or
failed." I didn't see anything on whether the test case / rule should
have an example of a pass and a fail within that rule set though. Would
it be good for the auditor (or whoever is using this rule set / test
case) to have a positive and a negative example?

This would help catch false positives and false negatives. But it would
also add burden on test rule developers. What would you suggest?


I will think more on this and add my thoughts as they come :) 




Separate question:




At what point would we stop submission of a set rule / test case ? Is
there a cut off date for submitting rule? I know that per the
"acceptance" that the ACT test rules will be updated periodically with
the techniques and understanding documents (currently every 6 months).
If a rule was not accepted, but was in queue, will we continue reviewing
that potential rule / test case with that queue moving forward on the
next iteration in terms of acceptance? I understand the goal is to build
a never ending repository of tests as technology changes.

Good question. Currently the Techniques are published every six months.
This may be a good trigger for also updating the Test Rules. We will
need to discuss that with the AG WG if they can offer other options.



Just a couple of thoughts, which are probably already documented
somewhere in our WCAG / ACT documentation...

Excellent thoughts! They are not yet documented, as we are working out
these exact questions. This is also why I'm very rudely answering your
questions with questions back... ;-)

Not rude at all! Positive feedback and questions make the process better. Thanks for the reply.



Best,
Shadi



Thank you,
Chris Loiselle


On Jun 26, 2017, at 09:40 AM, Wilco Fiers <wilco.fiers@deque.com> wrote:




Hey ACT'ers
In advance of our meeting in a little while, here is some of the
feedback I got from Deque:




1) We should break each submission into a single request - mixing too
many rules into a single request makes the feedback and decision making
difficult - even when tools like GitHub are used.
2) What about best practice rules? E.g. you don't have to have an H1 but
you really should try.
3) X should be 2 under implementations. Also - there should be a
condition of two implementations being shown to work before a rule makes
it into a CR
4) What about false positives? How do we ensure that rules do not make
it into the fully automated portion of this spec that generate false
positives?
5) contributions from Deque staff that could impact axe-core would
require an internal review before posting.






Wilco








On Thu, Jun 22, 2017 at 4:47 PM, Detlev Fischer
<detlev.fischer@testkreis.de> wrote:


I think these test cases may become very useful when related test rules
are contributed. Do you plan to make this fully public at a later time?


Yes. The COMPARE repository will be read-only for the general public and
read/write for experts/testers (I repeat my invitation to get signed up
right
now!)


--
Detlev Fischer
testkreis c/o feld.wald.wiese
Thedestr. 2, 22767 Hamburg


Mobil +49 (0)157 57 57 57 45
Fax +49 (0)40 439 10 68-5


http://www.testkreis.de

Beratung, Tests und Schulungen für barrierefreie Websites




Shadi Abou-Zahra schrieb am 22.06.2017 14:40:


Hi Detlev,


On 20/06/2017 11:00, Detlev Fischer wrote:
Hi Shadi, ACT TF
I had a look at the review Process document. The basic problem for me
is to
understand how the process (submission of a rule backed by supporting
test
cases) would work in practice, so I would think it is worthwhile
taking one or
a few non-trivial examples of real web content and looking what the
rule(s)
might look like that would be supportive when deciding about
conformance. THis
exercise would show the uninitiated how it's going to play out.


Agree. I believe sample rules that comply with the current Rules Format
specification are being developed. We can also use them for trying out
and refining this review process.




A complex and at the same time very frequent example might be
something like
drop-down navigation menus (take for example a recent discussion
between Matt
King and Mallory on the Webaim list - the tail end is here
http://webaim.org/discussion/mail_message?id=34968 )


What would rules look like that help me establish whether some menu
conforms
to 1.3.1, 2.1.1, 4.1.2, 2.4.3 etc? How can the rule be isolated from
content
aspects that may co-determine whether we think of some solution as
acceptable
or not (take the length of the submenus in cases where they are opened
automaticlly when focused)? When does the aria menu pattern apply,
and what
deviations of the pattern are OK (conform) in what contexts?


We all know the difficulty of attributing an issue to the right SC -
when an
element does not get tab focus but you CAN activate it when arrowing
there,
does it violate 2.1.1? Or only 2.4.3? If a main menu item opens the
submenu
and a second activation does not close it but goes to a section page,
is that
a usability issue or necessarily a fail of some SC? Etc, etc...


So I believe working through a few practical real world
implementations and
showing how the ACT framework would support developers / testers in
assessing
real-world implementations would really help making the ACT activity
a lot
more tangible (it often feels quite abstract to me).


Agree. Though this is slightly orthogonal to the review process itself.




Finally, an invitation: ACT TF members wanting a test login to our
COMPARE
repository ( http://www.funka.com/en/projekt/compare/ ) are welcome -
just
give me a shout. The repository is early days, not yet in its final
shape, and
not yet public but it already has a few real world cases with
accessiblility
ratings. Should you want to add your rating, the comment field would
give
scope to outline the rules according to which someone has arrived at
a PASS or
FAIL conclusion. As a contributor of ratings you will be picking the
SC (or
multiple SCs) that you think should fail (or pass with comment).


I think these test cases may become very useful when related test rules
are contributed. Do you plan to make this fully public at a later time?


Best,
Shadi




Best,
Detlev






--
Detlev Fischer
testkreis c/o feld.wald.wiese
Thedestr. 2, 22767 Hamburg


Mobil +49 (0)157 57 57 57 45
Fax +49 (0)40 439 10 68-5


http://www.testkreis.de

Beratung, Tests und Schulungen für barrierefreie Websites


Shadi Abou-Zahra schrieb am 19.06.2017 19:54:


Dear ACT TF,


Ref:
https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/ACT_Review_Process





As discussed during the call today, please review the outline for the
proposed ACT Review Process. Feel free to add your feedback to the wiki
discussion tab or by email.


Regards,
Shadi


--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/

Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)








--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/

Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)


















--


Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG

--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)