Re: Considering existing AI/Machine-Learning metrics for assessing value quality under WCAG 3.0...

Alistair,

Thank you for sending this to the group! Apologies for the delayed
response, just returning from vacation.

I thoroughly agree about the potential impact of ML in this space! I've
also seen many projects starting to sprout up, like Chrome's ability
to generate
image descriptions <https://support.google.com/chrome/answer/9311597>
and Facebook's
as well <https://www.facebook.com/help/216219865403298>, and this kind of
technology can absolutely help in identifying problematic alternative text
just as that SPICE project outlines, as well as helping designers and
authors to write more helpful alternative text.

Similarly to ACT, we'll aim to make the outcomes and overall tests for WCAG
3 clear and understandable, regardless of the mechanisms used for the
actual testing (manual, automated, tooling-supported, etc.). We can also
certainly note opportunities for how different technologies can help make
this more streamlined or better supported, through test tooling, authoring
tools, and the platform itself. With many of us working in that space, I
think spotting those will likely even come naturally.

Looking forward to exploring this with you as we move forward from here!

Much appreciated,

Shawn

On Tue, Sep 28, 2021 at 11:54 AM Alistair Garrison <
alistair.garrison@accesseo.co.uk> wrote:

> Hi,
>
> [Forwarding for FYI]
>
> In the field of AI/Machine-Learning one of the most important things to
> set in a project is a measurable success metric.
>
> Over the past few years more and more work has being done to define
> measurable success metrics In AI/ML, some of which might be very useful to
> consider when looking at better defining WCAG 3.0 Outcomes.
>
> I will use Alternative Text as an example.  Simply stating that
> alternative text provided by a system be “descriptive” is not sufficient
> for AI/Machine-Learning projects - which want to clearly indicate their
> improvements. So over the past few years some seemingly good work has been
> done on defining metrics that measure the quality of generated captions by
> analyzing their semantic content.  See SPICE - Semantic Propositional Image
> Caption Evaluation (https://panderson.me/images/SPICE.pdf) as an
> example.  SPICE would be capable of testing human created alternative text
> for an image; even though it has been designed to test dynamically
> generated text alternatives.
>
> With such an example in mind, might the AGWG be open to considering the
> use of such metrics, when available and appropriate of course, in their
> future development of testable WCAG 3.0 Outcomes?
>
> Raised as a point of interest really, as I am seeming such metrics through
> my research work.
>
> Very best regards
>
> Alistair
>
> Alistair Garrison
> CEO / Founder Accesseo Limited
> ((Newly) Invited Expert - AGWG)
>