Re: Costs of testing with Silver

Hi Charles & everyone,

Sorry for the long email, the TL;DR is:

I think we need to be cognisant of cost, not necessarily capping it, but providing options for requirements that could incur a lot of time/cost. Not doing so would prevent the standard being taken up.

Unless there are changes to the education system and how people become designers/developers, accessibility will be new to most people, and an audit is often their first introduction to accessibility.
Cost matters.

If Silver is structured around slightly higher-level user-requirements, each with general and/or tech-specific criteria to fulfil, we could include guidance on the more costly/subjective requirements without making them criteria for silver/bronze levels.

I’m advocating that we do not disguise the overall requirements for people, but structure the levels & testing needed for each level to keep it ‘reasonable’. (See the example at the bottom of this email.)


The longer version:

CH wrote:
> Accessibility guidelines are intended to be a set of instructions on how to create content and documents correctly in the first place

They also need to support the testing and compliance scenarios, otherwise they do not get used in policy and legal contexts. If that doesn’t happen, they do not get general take up.


> Organizations should not have to run automated or manual tests to check if a caption exists or if headings are in the right order. These should be bare minimum expectations placed upon creators.

That takes a very optimistic view of how organisations (especially large ones) operate. The headings on a page often come from 3 different teams, and more than 3 people. Anyone could introduce errors, and with staff turn-over it is impossible to ensure everyone has been training before they start real work.


> If we make the guidelines easier, it is reasonable to assume that testing is subsequently easier. If validation methods are (still) included in the guidelines, then half of each test case is already written.

I don’t follow the logic here, I think there are too many meanings behind ‘easier’. A stringent new guidelines could be easy to understand, but that does not make it easy to implement. Running a marathon is conceptually simple, but not easy.

There are aspects to accessibility, especially when accounting for COGA issues that are easy to understand but difficult to do, or know when they have been done. For example, using plain English / popular words in the navigation. Every page update could undermine that.

I still advocate that some requirements will need to be met by following a process rather than trying to apply a subjective test (e.g. tick that you’ve tested the navigation with a UCD-method). That does incur cost, but rather than assigning it to ‘accessibility’ it’s being assigned to design process.


> I don’t believe the scope should include considerations for cost.

If there is a huge cost (or even just perceive cost)  for organisations meeting the standard it will not be taken up. It would probably not make it through the W3C process, but that would just be a pre-cursor to not making it into policies.

As Wilco outlined, for an organisation new to accessibility procuring an audit, the higher the price the less likely they are to start their accessibility journey. You can argue they should be building it in before that, but until there are legal reforms to the education system, most designers and developers in industry will not know accessibility basics.

As most designers / devs / testers do not know accessibility basics, there has to be a starting point, and as a consultancy & training provider that starting point is often an audit to assess their current state.

Going from WCAG 2.0 to 2.1 increased the time needed (and therefore cost) to audit 10 pages by about 10% for us. If some of the draft COGA requirements had been included as they were, I could see that might have increased it by 100-300%, with usability testing on top in some scenarios.

That time cost is similar for internal QA testers (who we often train). It is partly mitigate by making sure accessibility is built in from the start, but there would be a proportional increase that makes the business case more difficult.


> No previous accessibility guideline considers or tracks the cost of how people use it.

We don’t track globally, but making the costs ‘reasonable’ is built into the laws (in UK/EU at least), and was a major factor in WCAG 2.x development.

I’m not suggesting that we worry about remediation costs as such, but the core time requirements for doing something (e.g. audio description) and testing that it has been done are important considerations.

There has to be a reasonable ‘zen’ path, where doing it right the first time, and knowing you’ve got it right does not add more than 5-10% to the overall project costs.

DavidS wrote:
> I’m concerned at the risk that this conversation might lead to the perception that certain requirements that would benefit specific disability groups are more likely to be rejected on cost grounds.

That is a concern, and we’ll need to walk the line between including those requirements and making unreasonable demands on organisations.

My suggestion for that is to include every user-requirement, regardless of whether there are reasonable solutions or tests.

However, conformance levels should account for there being reasonable solutions / tests. There will be gaps, and those gaps are acknowledgement that more work needs to be done.

For example, if there were a guideline / user-requirement for:
“Content is presented in management blocks” (I’m adapting from here [1])

Description:
“Chunking content, whether it is visual or auditory, supports those with working memory deficits, such as those with learning disabilities and brain injury. The breaking down of content into small sections, whether it is developed as audio or video output; mathematical symbols; or a paragraph of text; improves levels of comprehension.”

Bronze level criteria: None.
Silver level criteria (General): “Statements which instruct a user to make a choice or take an action have only one instruction per sentence…”
Gold level criteria: “Usability testing of the journey is conducted with at least 6 participants who have working memory deficits or learning disabilities.”

There could be multiple criteria at each level, but this is just one example.

The fact there is no bronze level criteria takes it out of the testing process at that level, but acknowledges there could be issues for some users. For Governments/businesses with a policy for taking the Silver level, they will need to figure that into their content creation and testing.

Cheers,

-Alastair

https://github.com/w3c/wcag21/issues/24