Re: Costs of testing with Silver

After re-reading the majority of this thread, it seems to be (mostly) contingent upon and in response to the idea of levels of conformance – particularly the model with Bronze, etc.

This is simply an idea being prototyped – one of many. Testing that prototype would likely reveal a concern of or considerations for levels of effort, cost and timing. Those responses would then be taken into account to make any relevant and appropriate revisions.

It has never been stated that Silver would {n} number of new or additional success criteria – only that it would simplify the language and format of the existing ones and establish a governance and contribution model that makes it easier to edit them, expire them and yes, add them.

If there is cost incurred in some organizations simply because the language changed, I see this as no different than any other specification or guideline – for example HTML 5 to 5.3. I still contend that attempting to control or mitigate cost is both: not the role, purpose or responsibility of the guideline; and not even possible due to the extreme variability in operations, processes, scale, market, proficiency, and thousands of other business variables.

Yes, I agree that cost will be a factor in how and how fast Silver is adopted. But we cannot lose site of the fact that what matters most is people. Silver exists so that people can access the web (and various related technologies). Better guidelines have the potential of enabling more access to more people. This can be viewed as a parallel to any other business effort to increase reach – like marketing. It is a given that there is cost associated. It is also proven that there are higher returns.

I was unaware that ‘reasonable cost’ was a major factor in WCAG 2.x development. If that is the case, then there must be a model for determining cost as well as a definition of ‘reasonable’. If there is a majority opinion that practice should be continued, then perhaps that existing wording can be added to the prototype? My opinion is simply that cost is the topic that makes individuals and organizations question making things accessible and delay and defer and avoid any effort to do so. That should never be a question.



Charles Hall // UX Architect, Technology

charles.hall@mrm-mccann.com<mailto:charles.hall@mrm-mccann.com?subject=Note%20From%20Signature>
w 248.203.8723
m 248.225.8179
360 W Maple Ave, Birmingham MI 48009
mrm-mccann.com<https://www.mrm-mccann.com/>

[MRM//McCann]
Relationship Is Our Middle Name

Ad Age B-to-B Agency of the Year, 2018
Ad Age Agency A-List 2016, 2017
Ad Age Creativity Innovators 2016, 2017
North American Agency of the Year, Cannes 2016
Leader in Gartner Magic Quadrant, 2017, 2018



From: Alastair Campbell <acampbell@nomensa.com>
Date: Thursday, August 30, 2018 at 7:34 AM
To: Silver Task Force <public-silver@w3.org>
Subject: Re: [EXTERNAL] Costs of testing with Silver
Resent-From: Silver Task Force <public-silver@w3.org>
Resent-Date: Thursday, August 30, 2018 at 7:34 AM

Hi Charles & everyone,

Sorry for the long email, the TL;DR is:

I think we need to be cognisant of cost, not necessarily capping it, but providing options for requirements that could incur a lot of time/cost. Not doing so would prevent the standard being taken up.

Unless there are changes to the education system and how people become designers/developers, accessibility will be new to most people, and an audit is often their first introduction to accessibility.
Cost matters.

If Silver is structured around slightly higher-level user-requirements, each with general and/or tech-specific criteria to fulfil, we could include guidance on the more costly/subjective requirements without making them criteria for silver/bronze levels.

I’m advocating that we do not disguise the overall requirements for people, but structure the levels & testing needed for each level to keep it ‘reasonable’. (See the example at the bottom of this email.)


The longer version:

CH wrote:
> Accessibility guidelines are intended to be a set of instructions on how to create content and documents correctly in the first place

They also need to support the testing and compliance scenarios, otherwise they do not get used in policy and legal contexts. If that doesn’t happen, they do not get general take up.


> Organizations should not have to run automated or manual tests to check if a caption exists or if headings are in the right order. These should be bare minimum expectations placed upon creators.

That takes a very optimistic view of how organisations (especially large ones) operate. The headings on a page often come from 3 different teams, and more than 3 people. Anyone could introduce errors, and with staff turn-over it is impossible to ensure everyone has been training before they start real work.


> If we make the guidelines easier, it is reasonable to assume that testing is subsequently easier. If validation methods are (still) included in the guidelines, then half of each test case is already written.

I don’t follow the logic here, I think there are too many meanings behind ‘easier’. A stringent new guidelines could be easy to understand, but that does not make it easy to implement. Running a marathon is conceptually simple, but not easy.

There are aspects to accessibility, especially when accounting for COGA issues that are easy to understand but difficult to do, or know when they have been done. For example, using plain English / popular words in the navigation. Every page update could undermine that.

I still advocate that some requirements will need to be met by following a process rather than trying to apply a subjective test (e.g. tick that you’ve tested the navigation with a UCD-method). That does incur cost, but rather than assigning it to ‘accessibility’ it’s being assigned to design process.


> I don’t believe the scope should include considerations for cost.

If there is a huge cost (or even just perceive cost)  for organisations meeting the standard it will not be taken up. It would probably not make it through the W3C process, but that would just be a pre-cursor to not making it into policies.

As Wilco outlined, for an organisation new to accessibility procuring an audit, the higher the price the less likely they are to start their accessibility journey. You can argue they should be building it in before that, but until there are legal reforms to the education system, most designers and developers in industry will not know accessibility basics.

As most designers / devs / testers do not know accessibility basics, there has to be a starting point, and as a consultancy & training provider that starting point is often an audit to assess their current state.

Going from WCAG 2.0 to 2.1 increased the time needed (and therefore cost) to audit 10 pages by about 10% for us. If some of the draft COGA requirements had been included as they were, I could see that might have increased it by 100-300%, with usability testing on top in some scenarios.

That time cost is similar for internal QA testers (who we often train). It is partly mitigate by making sure accessibility is built in from the start, but there would be a proportional increase that makes the business case more difficult.


> No previous accessibility guideline considers or tracks the cost of how people use it.

We don’t track globally, but making the costs ‘reasonable’ is built into the laws (in UK/EU at least), and was a major factor in WCAG 2.x development.

I’m not suggesting that we worry about remediation costs as such, but the core time requirements for doing something (e.g. audio description) and testing that it has been done are important considerations.

There has to be a reasonable ‘zen’ path, where doing it right the first time, and knowing you’ve got it right does not add more than 5-10% to the overall project costs.

DavidS wrote:
> I’m concerned at the risk that this conversation might lead to the perception that certain requirements that would benefit specific disability groups are more likely to be rejected on cost grounds.

That is a concern, and we’ll need to walk the line between including those requirements and making unreasonable demands on organisations.

My suggestion for that is to include every user-requirement, regardless of whether there are reasonable solutions or tests.

However, conformance levels should account for there being reasonable solutions / tests. There will be gaps, and those gaps are acknowledgement that more work needs to be done.

For example, if there were a guideline / user-requirement for:
“Content is presented in management blocks” (I’m adapting from here [1])

Description:
“Chunking content, whether it is visual or auditory, supports those with working memory deficits, such as those with learning disabilities and brain injury. The breaking down of content into small sections, whether it is developed as audio or video output; mathematical symbols; or a paragraph of text; improves levels of comprehension.”

Bronze level criteria: None.
Silver level criteria (General): “Statements which instruct a user to make a choice or take an action have only one instruction per sentence…”
Gold level criteria: “Usability testing of the journey is conducted with at least 6 participants who have working memory deficits or learning disabilities.”

There could be multiple criteria at each level, but this is just one example.

The fact there is no bronze level criteria takes it out of the testing process at that level, but acknowledges there could be issues for some users. For Governments/businesses with a policy for taking the Silver level, they will need to figure that into their content creation and testing.

Cheers,

-Alastair

https://github.com/w3c/wcag21/issues/24

This message contains information which may be confidential and privileged. Unless you are the intended recipient (or authorized to receive this message for the intended recipient), you may not use, copy, disseminate or disclose to anyone the message or any information contained in the message.  If you have received the message in error, please advise the sender by reply e-mail, and delete the message.  Thank you very much.