Re: [Protocols] Minutes for March 4th, 2022

Thank you for posting the minutes of today's call. As I read them, I noted
the following:

<Rachael> ...Jeanne spoke 3-4 years ago with lawyers there was the
possibility to add more subjectiveness. I'm not sure that's true or not. Its
about measuring the effort.

While I can respect that perspective, I must also strongly disagree.
Protocols are to address needs/requirements that *cannot be measured*, so
respectfully it's not about measuring the effort (and how would you do
that?), it is about *soliciting the commitment*, and having that
commitment *publicly
documented* (the protocol and the 'promise').

   - "...the point of protocols is to achieve outcomes that we don't know
   how to measure."
   - "...protocols are about procedures you use to meet an outcome that is
   not easily measurable"

Again, using the example of Plain Language (as outlined by
- the 'effort' to ensure plain language will also be directly correlated to
the content itself: some content will be harder to express in plain
language than other content, so the effort will be *mostly* based on the

In that scenario, things that 'could be measured' that I can think of
include the number of hours it took, impact on staffing needs (it took 3
people 2 days, versus it took 1 person half-an-hour - but that does not
account for experience or expertise, both factors in determining "how
long"), contrasting time to author/remediate against volume of content
(paragraphs, pages, chapters...) - none of those metrics relate to the
final output however, which is still intended to be arriving at *content
written in plain language* (audience reading level, comfort in reading the
content, etc.). Measuring the effort does not correlate to the final
outcome in any meaningful way that I can think of.

I also noted the discussion on what makes a protocol and who decides.

In my initial proposal, I had suggested two types of protocols, with
differing levels of "reward" (points toward a final score that leads to

*Vetted and Approved:*
I had suggested that the first 'class' (type) of protocol would be a
protocol specifically called out (vetted) by the Working Group. In that
case, subject matter experts within our group can determine both the
suitability and impact that specific protocols would bring to the overall
outcomes, and we could assign those specific protocols 'points' based on
our judgement. This kind of protocol could come directly from the W3C/WAI
itself, or our own experts could 'validate' other protocols authored by
governments, companies, and other entities. (In my proposal I had offered
as a straw-man suggestion Adobe's User & Usability Testing Questions -

For example, "Making Content Usable for...COGA
<>"(*) is another document that I've
frequently suggested meets *my* mental model of a Protocol: like the US
Plain Language content currently being used for evaluation, "Making
Content..." has 8 specific 'Outcomes' that cannot be measured granularly,
however because of how that document is written, it clearly provides both
content creators and content evaluators with enough data to make informed
decisions (explanations, user stories, etc.) - both in the authoring phase
as well as in the 'evaluation' phase. The document establishes parameters
and expected outcomes to "look for", but falls short of stating, "you must
do this, or you must do that". In this case, because it is a document
(protocol) that comes from within, I would suggest we assign it a 'higher'
value when contributing to a final score.

(* My proposal for "Protocols" was very much driven by me originally and
directly asking myself "How can we include "Making Content..." into WCAG
3?" I believe it is both an excellent resource that addresses a number of
real gaps and problems today, and it clearly - at least to me - defines the
non-measurable requirements that the COGA group determined were required
'at scale'. Given that both and "Making Content..." have
been authored with a similar structure, they may also be the basis of a
'protocol template' we could look at)

*Internal or Company-Authored:*
This may be a bit trickier (controversial?), but I believe it will be an
important part of the larger effort. If our WG clearly defines the
structure of what we expect to see in a protocol (minimum requirements), we
could then potentially let content creators author their own protocols -
especially when it comes to specialized content (strawman: META publishes a
protocol for XR environments that addresses accessibility needs. Allowing
custom protocols encourages new technologies to also document accessibility
considerations at a scale faster/larger than we can handle here within the
W3C - and so should be actively encouraged in my mind).

Of course, custom protocols would still need to be directly addressing the
needs of users with disabilities (and we could potentially even mandate
proof of direct consultation with impacted user-groups as part of the
requirements for such custom protocols).

In that scenario, to be considered a 'valid' protocol, at a minimum I would
expect that the document/protocol itself be *Publicly available* (no
paywalls or intranet-only documents) so that any evaluator could easily
reference that 'promise' to determine if it has been applied to the
content. In my previous email I had proposed one method of how to provide
that reference (i.e. it's a public URL) into an assertion statement.

In terms of 'scoring' however, those protocols would be worth "less",
primarily because they have not been directly vetted by the W3C WG. (I
could also envision a mechanism where 3rd parties could propose a protocol
to our group for consideration - if we agreed it was hitting the mark, we
could then elevate that protocol to the "vetted list" - allowing for
extensibility. In my initial presentation, I had referenced *Moodle
Usability testing protocol* [] as a potential
example of this; the group seems to have also landed on BBC Gel, which
would be a similar document that today is unvetted, but potentially could
'graduate' to our list of vetted protocols - ditto the Adobe doc referenced



On Fri, Mar 4, 2022 at 10:09 AM Bradley-Montgomery, Rachael <> wrote:

> The minutes from the protocols subgroup
> <> are
> available.
> Summary:
>    - We will be going through an exercise for  the next few weeks to
>    evaluate:
>       - How to evaluate whether the protocol was done
>       - How well the protocol was followed
>       - How to evaluate the quality of the results (if possible)
>    - We will be using the following (possible) protocols as examples to
>    help with discussion:
>       - Plain Language, Visma UX, (BBC Gel A11y section if 3rd is needed)
>    - Possible ways to evaluate whether a protocol was done (Discussion
>    still ongoing)
>       - Require the organization to publicly state:
>          - What protocol/part of protocol was done
>          - How the protocol was embedded in content or organization?
>          - How can the public see that the protocol was embedded?
>          - Date statement was made
>       - Key questions that need to be addressed later:
>       - Definition of a protocol?
>       - Is a protocol a document or part of a document?
>       - How will we handle overlap with WCAG? The overlap will shift
> *From: *Chuck Adams <>
> *Date: *Wednesday, March 2, 2022 at 1:42 PM
> *To: *"" <>, "" <
> *Subject: *[Protocols] Agenda for March 4th, 2022
> *Resent-From: *<>
> *Resent-Date: *Wednesday, March 2, 2022 at 1:41 PM
> Hi All,
> The Protocols Subgroup will meet again this Friday, March 4th at 9:00 AM
> Boston Time (1400 UTC).
> The Zoom teleconference data is provided at this link:
> <;!!ACWV5N9M2RV99hQ!ZvVx1wh89EAXhBiorHpgvdpQRlEtQPxaEsJbJ7_Q3MrxtnQGs5lwbIC34yacGIQO4g$>
> We will be on IRC using the W3C server at
> <;!!ACWV5N9M2RV99hQ!ZvVx1wh89EAXhBiorHpgvdpQRlEtQPxaEsJbJ7_Q3MrxtnQGs5lwbIC34ybOl3ZsYw$>,
> in channel *#wcag3-protocols*
> These and additional details of our work, including minutes, current,
> and archived draft documents are available on our subgroup wiki page here:
> <;!!ACWV5N9M2RV99hQ!ZvVx1wh89EAXhBiorHpgvdpQRlEtQPxaEsJbJ7_Q3MrxtnQGs5lwbIC34ya-s3KL6w$>
> *** Agenda ***
> agenda+ Develop a way for a lay-person to assess whether a protocol was
> followed
>    1. Pick 2-3 things that are likely protocols (, BBC
>       style guidelines, ?)
>       2. Propose a way to evaluate (pass/fail):
>                                                                i.      Whether
> the protocol was done
>                                                              ii.      How
> well the protocol was followed
>                                                            iii.      The
> quality of the results
> Regards,
> Charles Adams

*John Foliot* |
Senior Industry Specialist, Digital Accessibility |
W3C Accessibility Standards Contributor |

"I made this so long because I did not have time to make it shorter." -
Pascal "links go places, buttons do things"

Received on Friday, 4 March 2022 21:22:19 UTC