Re: Auto-WCAG - Expert system approach?

Hi Jon,
Yes, interoperability of rules and test results is one of the goals of the
ACT Framework. Exactly how that will work is still up for discussion.

Wilco

On Sun, Nov 20, 2016 at 8:49 PM, Gunderson, Jon R <jongund@illinois.edu>
wrote:

> Wilco,
>
> Does the ACT define a testing framework for developing and executing test
> cases for insuring interoperability of rules?
>
> Jon
>
> Jon  Gunderson
> Sent from my iPhone
>
> On Nov 20, 2016, at 6:44 AM, Wilco Fiers <wilco.fiers@deque.com> wrote:
>
> Hi all,
>
> A late response from me, to a very valuable discussion. I think Alistair
> is making good points. As discussed during the Auto-WCAG call last week, it
> is worth us revisiting our existing rules, and update them to our current
> way of thinking. I think this makes sense and will help us keep them
> closely aligned to the approach we expect to come out of ACT.
>
> Having said that, I don't want to get too experimental at this point. With
> the ACT TF working actively to develop the first draft of the framework, I
> think it makes more sense for us to await this. We'll need to clean up some
> of the older work, so that we can be ready to start working with the first
> draft. Our feedback about using the ACT Framework draft will be valuable
> input for ACT TF. But we shouldn't try to get ahead of that. For one
> because it doesn't make much sense, and secondly because ACT has a much
> greater velocity than Auto-WCAG does.
>
> Regards,
>
> W
>
> On Wed, Oct 26, 2016 at 2:32 PM, Alistair Garrison <alistair.garrison@
> ssbbartgroup.com> wrote:
>
>> Dear All,
>>
>>
>>
>> The diagrams on https://www.w3.org/community/a
>> uto-wcag/wiki/SC1-1-1-text-alternative#Activity_diagram_of_
>> transitions_between_steps_1_to_18
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_community_auto-2Dwcag_wiki_SC1-2D1-2D1-2Dtext-2Dalternative-23Activity-5Fdiagram-5Fof-5Ftransitions-5Fbetween-5Fsteps-5F1-5Fto-5F18&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=1fJ9E8uhxCr2xr-vWUoCZ_c-KTC9Ak6lVb03bWC0BdY&e=>
>> indicate how truly complex this way of creating rules has in fact become –
>> interweaving many steps together into giant “rules”. If one step changes;
>> or if new steps need to be added, it may break many things.
>>
>>
>>
>> Personally, I’d be happiest if Auto-WCAG’s first-base was to look to
>> create all those separate atomic tests that:
>>
>>
>>
>> 1)       Are quick to define; and
>>
>> 2)       Are quick to get group agreement on.
>>
>>
>>
>> There must be a reasonable number, for example, checking if “one or more
>> iframe nodes does not have a title attribute”. [Not yet discussing the
>> quality of the title text if provided, just if it is present.]
>>
>>
>>
>> To my mind, this would allow a quick win for Auto-WCAG, as we’d almost
>> certainly be able to step quickly towards at least a low-level of testing
>> parity between tools – whilst providing fully automatic tests for
>> monitoring like those in http://checkers.eiii.eu/en/tests/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__checkers.eiii.eu_en_tests_&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=VbMUBFYov39ddF0Oz5dkFYsYk-aO4IEU54GwFxEPZGg&e=>
>> .
>>
>>
>>
>> Then we could move on to more complicated (but still atomic tests) like
>> checking if “One or more img nodes (excluding those that have an alt
>> attribute set to a null value, or a role="presentation" attribute),
>> available in the DOM, does not have a mechanism that allows an accessible
>> name value to be calculated”.    The “available in the DOM” aspect means -
>> not impacted by a computed CSS “display:none” style.
>>
>>
>>
>> These tests are where you could inject guesses about what constitutes a
>> decorative image (e.g. height <= 5px or width <= 3px) – which we could all
>> discuss.  Or, what constitutes suspicious text in an iframe title. Etc…
>>
>>
>>
>> The quality of the text could be asked about in separate tests that would
>> involve user judgement - really ones that simply collect content for the
>> user to look at.
>>
>>
>>
>> This more atomic implementation would, again to my mind, allow easier
>> adoption of Auto-WCAG tests into more main-stream products – as some could
>> be adopted into automatic testing engines; and others into semi-automatic
>> content collection / visual testing tools.
>>
>>
>>
>> All the best
>>
>>
>>
>> Alistair
>>
>>
>>
>> Alistair Garrison
>>
>> Senior Accessibility Engineer
>>
>> SSB Bart Group
>>
>>
>>
>>
>>
>>
>>
>> *From: *Frank Berker <fb@ftb-esv.de>
>> *Date: *Wednesday, 26 October 2016 at 12:00
>> *To: *"public-auto-wcag@w3.org" <public-auto-wcag@w3.org>
>> *Subject: *Re: Auto-WCAG - Expert system approach?
>> *Resent-From: *<public-auto-wcag@w3.org>
>> *Resent-Date: *Wednesday, 26 October 2016 at 12:01
>>
>>
>>
>> Hi Alistair, John and all,
>>
>> we faced the issue of the need of human input for valid conformance
>> testing quite early:
>> RE: Human input in the Auto-WCAG test-cases
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.w3.org_Archives_Public_public-2Dauto-2Dwcag_2014Oct_0028.html&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=GZoklHD42BJMUnx_InMMECrLPzjvJK3PhVAJfNgJeWM&e=>
>>
>> And we came to the decision that we should try to provide the full test
>> chain avoiding "earl:cantTell"-outcomes. This also led to the development
>> of the UTT-bookmarklet.
>>
>> It was planned to keep the choice at the developers, if they implement
>> just the automatic parts or the full specification. We also specified the
>> knowledge, which an assertor must have to provide a reliable assessment:
>> Assertor requirements (optional)
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_community_auto-2Dwcag_wiki_Introduction-5Fto-5Fauto-2Dwcag-5Ftest-5Fdesign-23Assertor-5Frequirements-5F.28optional.29&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=4FuRRkYMNXdYYk0IXfTJzOL99HC9z716kKa96pjtnNQ&e=>
>>
>> Besides, human input is also included in the test procedure of some
>> sufficient techniques.
>>
>>
>>
>>
>> *"Procedure Examine each img element in the content Check that each img
>> element which conveys meaning contains an alt attribute. If the image
>> contains words that are important to understanding the content, the words
>> are included in the text alternative." *H37: Using alt attributes on img
>> elements
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_WAI_GL_2016_WD-2DWCAG20-2DTECHS-2D20160105_H37&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=9X23KIri3hi6wnnLEJT97FrA-PkGx0-jIKc51xl2Z-Y&e=>
>>
>> Even if its clear, to underline, take the 1.1.1 example: If all images on
>> a page contain alt-attributes with the value "Lorem ipsum", this must fail.
>> Without human input, a non-empty textual alternative can only be
>> "earl:cantTell". And lots of tools provide lots of such outcomes, which
>> force an evaluator to have a look on each image. This could and should be
>> supported by a tool.
>>
>> According to the Activity diagram of transitions between steps 1 to 18
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_community_auto-2Dwcag_wiki_SC1-2D1-2D1-2Dtext-2Dalternative-23Activity-5Fdiagram-5Fof-5Ftransitions-5Fbetween-5Fsteps-5F1-5Fto-5F18&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=1fJ9E8uhxCr2xr-vWUoCZ_c-KTC9Ak6lVb03bWC0BdY&e=>,
>> fail1, pass3/fail4, pass4, fail6 and pass6/fail7 can be assessed fully
>> automatically.
>>
>> But as far as I read, how user testing and automated testing come
>> together in ACT Rules will be a topic in the spec:
>> ACT Framework Spec
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_WAI_GL_task-2Dforces_conformance-2Dtesting_wiki_ACT-5FDeliverables-231.-5FACT-5FFramework-5FSpec&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=e3D8quLM5TEnCL9-FMsBDE0rkiRLntnRDwh0wL9VaM4&e=>
>>
>> So it definitely will have to be discussed.
>>
>> Best regards
>> Frank
>>
>> Am 26.10.2016 um 10:50 schrieb John Hicks:
>>
>> Hello
>>
>> Apologies if I misconstrued the question!
>>
>>
>>
>> On a related issue, in the github we often see "Ruletype : automatic" but
>> then in the steps "Get user input"
>>
>> For me an automatic rule (as well as an "expert system") is always
>> without user input (once it's running, of course).
>>
>>
>>
>> Maybe some definitions woudl be useful ; as far as I understand it a
>> user-input question/answer system is not an "Expert System" in the
>> traditional AI sense.
>>
>>
>>
>> 2nd the call for clarification from the others
>>
>>
>>
>> John
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 26 October 2016 at 10:16, Alistair Garrison <
>> alistair.garrison@ssbbartgroup.com> wrote:
>>
>> Hi John, All,
>>
>>
>>
>> The fundamental shift over to expert-system type “rules” also came with
>> the development of https://www.w3.org/community/a
>> uto-wcag/wiki/Template:UserInput
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_community_auto-2Dwcag_wiki_Template-3AUserInput&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=Q8BT-2AaC0n3DMrR779wdkX7p-Lg3NN5oEzfecFHTpE&e=>.
>> This shift would not, almost certainly, have been made based on a mentioned
>> reference to a tool by a participant.  Certainly, for a shift to take place
>> group consensus must have been sort – about moving to expert-system type
>> rules, over the more atomic rules people had been previously working on
>> (for example, in the EIII project’s http://checkers.eiii.eu/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__checkers.eiii.eu_&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=laO7Ilsj245sz9XeacUWUFri4aOuzB9Dud7qM4LuNjU&e=>
>> ).
>>
>>
>>
>> Mikael mentioned the “EIII project’s User Testing Tool”, which could well
>> have been the expert-system being referenced - but this is very much in its
>> infancy, and seemingly no longer in development.
>>
>>
>>
>> So then, if no one in the group is currently developing an expert-system
>> (is anyone?); and we have several in the group developing tools that run
>> more atomic tests (like the EIII tests) - my question to the group is still
>> “are we developing Auto-WCAG rules for an expert system tool”? and, if yes
>> – why exactly?
>>
>>
>>
>> Especially when all my personal experience to date, and seemingly John’s
>> personal experience (with regard to the first bullet), indicates that such
>> expert-system interview-based tools:
>>
>>
>>
>> -          become so very tedious so quickly to use;
>>
>> -          take ages to develop (as “rules” are so inter-dependent);
>>
>> -          will almost certainly not fit the broad-scale auto-monitoring
>> usecase, as they require user interaction at many interim stages in tests;
>> and
>>
>> -          are only as accurate as the user’s judgement – which was
>> surely what we’re trying to avoid by having a base set of agreed fully
>> automatic tests we could compare results from.
>>
>>
>>
>> Again, very interested to hear from Wilco / group members.
>>
>>
>>
>> All the best
>>
>>
>>
>> Alistair
>>
>>
>>
>> Alistair Garrison
>>
>> Senior Accessibility Engineer
>>
>> SSB Bart Group
>>
>>
>>
>> *From: *John Hicks <jwjhix@gmail.com>
>> *Date: *Tuesday, 25 October 2016 at 13:24
>> *To: *Alistair Garrison <alistair.garrison@ssbbartgroup.com>
>> *Cc: *"public-auto-wcag@w3.org" <public-auto-wcag@w3.org>
>> *Subject: *Re: Auto-WCAG - Expert system approach?
>>
>>
>>
>> Dear Alistair,
>>
>> I am not sure exactly which meeting it was or if it referred to something
>> I might have said :
>>
>> Urbilog.fr has developped 3 automatic testing tools based on Expert
>> System in the AI sense.
>>
>> The "Expert" part was not user input but the "clips" expert system as
>> created by nasa :  https://en.wikipedia.org/wiki/CLIPS
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_CLIPS&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=Np9qTrb51fTx6byPtRq5CoqmeC1jbbnEFe9EYi4Bm8w&e=>
>>
>> The essence of the idea is to transform each HTML element into a
>> statement in a declarative language and then test the truth of these
>> statements w.r.t. the rule-set in question (508 and Wcag 1 in the
>> beginning, RGAA later).
>>
>> One of these tools also had the question and answer part, which, as you
>> say becomes so very tedious so quickly.
>>
>> I am still hoping to get one of these applications, designed for IBM
>> initially, but to which the IP rights belong to the developper, into open
>> source.
>>
>> John
>>
>>
>>
>>
>>
>>
>>
>> On 25 October 2016 at 14:07, Alistair Garrison <
>> alistair.garrison@ssbbartgroup.com> wrote:
>>
>> Hi Wilco, All,
>>
>>
>>
>> In the July 2015 Auto-WCAG blog - https://www.w3.org/community/a
>> uto-wcag/2015/07/24/introducing-the-auto-wcag-user-input-template/
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_community_auto-2Dwcag_2015_07_24_introducing-2Dthe-2Dauto-2Dwcag-2Duser-2Dinput-2Dtemplate_&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=hPslazWzsl7a_k0MIH79ffDKO3BL0SdKHQiJJ3tlpHA&e=>,
>> under Next steps I was reading that:
>>
>>
>>
>> “Some participants of the auto-wcag community group are currently
>> implementing the prototype of a User Testing Tool based on the questions
>> developed in the structured approach described in this post. The tool runs
>> in the user’s web browser and connects to a database storing the user
>> input.”
>>
>>
>>
>> Out of interest, could I ask which participants are working on this
>> “expert-system” tool? And, if work is still under way?
>>
>> I too developed an interview based expert-system ages ago – for testing
>> the accessibility of a web page (thankfully they were more static back
>> then).
>>
>>
>>
>> With all such systems you call your tests “rules”, and you follow a very
>> similar grammar to the one proposed in Auto-WCAG.  I used Jess formatting
>> initially (http://herzberg.ca.sandia.gov/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__herzberg.ca.sandia.gov_&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=un9FGnQkkmtPoCLac1RQNOvzU6wslq5UMnN7Iw-H8Pw&e=>),
>> then developed my own system…
>>
>> I finalised my expert system some years ago – it looked at WCAG 1.0 AA.
>> I demoed it to several organisations, and got some good reviews!
>>
>> The issue was that although an interesting way to proceed – only when you
>> actually used it for commercial audits did you realize how slow such as
>> process is.  The same questions have to be asked again and again of the
>> user – for example, for each img node – which is overkill if you are only
>> looking to find enough faults to show something is an issue.
>>
>>
>>
>> For example, http://wilcofiers.github.io/auto-wcag/rules/SC1-1-1-text-alt
>> ernative.html
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__wilcofiers.github.io_auto-2Dwcag_rules_SC1-2D1-2D1-2Dtext-2Dalternative.html&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=mhELHaB54TLShflHWMBrG7SnbuSYcDMNzvQhmodwQBk&e=>
>> - Contains questions you need to ask the user about each image – “Is this
>> element solely for decorative purposes”?
>>
>>
>>
>> With actual implementation knowledge, it is certainly not an approach I
>> would suggest for large-scale monitoring purposes, as it simply takes too
>> long to assess each page looked at; and requires human judgement which can
>> be wildly different.   Auto-WCAG tests, being formatted in a very specific
>> way, also will not slip easily into other testing platforms.
>>
>> My understanding was that we were concentrating on developing fully
>> automatic tests – which could be plugged into any testing platform – the
>> output from which could easily be compared.
>>
>>
>>
>> With manual steps in a number of the current tests, which also include
>> design constraints such as “Presented item - Web page (with title either
>> highlighted or in a seperate textbox)”, I think we are making it hard for
>> ourselves to achieve the comparability goal; or even create tests that
>> achieve AUTO-WCAG’s desired aims.
>>
>>
>>
>> It would only take a short amount of time to re-assemble the current
>> “rules” into sets of atomic fully-automated tests – by leaving the manual
>> testing steps aside; and I wonder if this isn’t the direction we should be
>> moving in instead – and may prove significantly quicker.  Which, I also
>> should mention seems to have been the approach of the EIII project from
>> which Auto-WCAG was initially born (http://checkers.eiii.eu/en/tests/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__checkers.eiii.eu_en_tests_&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=VbMUBFYov39ddF0Oz5dkFYsYk-aO4IEU54GwFxEPZGg&e=>
>> ).
>>
>>
>>
>> My question to the group is “are we developing Auto-WCAG rules for an
>> expert system tool”? and, if yes – why exactly?
>>
>>
>>
>> I’d be very interested to discuss the above, and hear comments from the
>> whole group.
>>
>> All the best
>>
>> Alistair
>>
>> ---
>>
>> Alistair Garrison
>>
>> Senior Accessibility Engineer
>>
>> SSB Bart Group
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Frank Berker
>>
>> fb@ftb-esv.de                      http://ftb-esv.de <https://urldefense.proofpoint.com/v2/url?u=http-3A__ftb-2Desv.de&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=6gLrGKa1gd2F-dB45cKvztWK-RDdLjl2jWbLGehCH0I&s=4Wivbo9cjvSPJLmeEXEhPC0KQrwW5ZnXGQi7SA81dg0&e=>
>>
>> FTB - Forschungsinstitut Technologie und Behinderung
>>
>> Grundschötteler Strasse 40,             58300 Wetter
>>
>> Telefon: 02335/9681-34        Telefax: 02335/9681-19
>>
>>
>
>
> --
> *Wilco Fiers* - Senior Accessibility Engineer
>
> <deque_logo_180p.gif>
>
>


-- 
*Wilco Fiers*
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG