W3C home > Mailing lists > Public > public-silver@w3.org > August 2018

Re: Costs of testing with Silver

From: Hall, Charles (DET-MRM) <Charles.Hall@mrm-mccann.com>
Date: Wed, 29 Aug 2018 16:46:23 +0000
To: Markus Mielke <mmielke@microsoft.com>, Wilco Fiers <wilco.fiers@deque.com>
CC: "w3c@garcialo.com" <w3c@garcialo.com>, Jeanne Spellman <jspellman@spellmanconsulting.com>, "public-silver@w3.org" <public-silver@w3.org>
Message-ID: <FF75E309-2E2A-4FB7-91F8-0FC7EC3D2AD9@mrm-mccann.com>
But doesn’t it assume that cost is fixed to be able to say that “Silver has reworded some criteria, added some criteria and revised the conformance model in a manner that adds no additional cost”?

The cost for Microsoft plus the vendor is very different than any other structure.

Encouraging good development behavior is not directly related to Silver. Silver simply purports to make that endeavor easier. Development is also only a single discipline among many. Sometimes testing is manual and remediation is the task of a copywriter – for issues like “…look for the red button below”. Learning how to write is as important as learning how to code and each are expectations placed on people. When the instructions are easier, it is reasonable to conclude that any training cost is lower. “An ounce of prevention…”

If Silver or even simply the conformance model of Silver is to consider cost, it seems like we would need an entire framework for how that cost can be calculated – that includes an entire spectrum of scenarios, like: we use a free browser extension to audit, then post issues on GitHub for free community remediation; to we have expensive enterprise licensure for 100 quality assurance analysts to run monthly reports and create a prioritized backlog of issues for our off-shore team of 200 developers to include in weekly sprints and releases. It seems like an endeavor that will create more arguments than the entire process of writing success criteria.

Charles Hall // UX Architect, Technology

w 248.203.8723
m 248.225.8179
360 W Maple Ave, Birmingham MI 48009

Relationship Is Our Middle Name

Ad Age B-to-B Agency of the Year, 2018
Ad Age Agency A-List 2016, 2017
Ad Age Creativity Innovators 2016, 2017
North American Agency of the Year, Cannes 2016
Leader in Gartner Magic Quadrant, 2017, 2018

From: Markus Mielke <mmielke@microsoft.com>
Date: Wednesday, August 29, 2018 at 10:23 AM
To: Wilco Fiers <wilco.fiers@deque.com>, "Hall, Charles (DET-MRM)" <Charles.Hall@mrm-mccann.com>
Cc: "w3c@garcialo.com" <w3c@garcialo.com>, Jeanne Spellman <jspellman@spellmanconsulting.com>, Silver Task Force <public-silver@w3.org>
Subject: RE: [EXTERNAL] Costs of testing with Silver

H, my name is Markus Mielke and I just recently joined Siler as part of Microsoft. My team, within Microsoft, is building accessibility tools for our developers to help find and fix accessibility issues (standalone and CI/CD) in automation and a set of manual assessments to help identify all WCAG issues.

We have identified 37 (+12) assessments based on WCAG 2.0 (and 2.1). Each assessment means a good chunk of a developers time. And as you can see, we have a 25% increase going from 2.0 to 2.1 alone. I agree with Charles that some of these assessments will be educational (the dev/content creator/team will run it once, learn from mistakes and then hopefully never make it again) but in an organization like Microsoft the verification of accessibility is often split between developers/team and vendors who run test passes. Here cost also incurs, since the assessment has to be performed independent of the learnings.

I second Wilco’s ask to take testing cost into consideration since it lowers the barrier to entry, instills trust and good development behavior at a smaller scale and lower cost.

-- Markus

From: Wilco Fiers <wilco.fiers@deque.com>
Sent: Wednesday, August 29, 2018 4:01 AM
To: Hall, Charles (DET-MRM) <Charles.Hall@mrm-mccann.com>
Cc: w3c@garcialo.com; Jeanne Spellman <jspellman@spellmanconsulting.com>; public-silver@w3.org
Subject: Re: Costs of testing with Silver

Hey Charles,
Let me verify that I understand your position. So you are suggesting that costs of accessibility testing should not factor into the considerations for Silver at all? So in a hypothetical situation where accessibility testing from Silver ends up costing 5 times as much as an existing WCAG 2.0 Level AA audit, that would still be acceptable for you? What about if it ends up as 20 times the cost of a WCAG 2.0 Level AA audit? I would wager a guess that there is an upper limit to what you think is acceptable costs for accessibility audits. For most people there would be, so lets talk about it and see what we think it is.

I see your point of down stream benefits, but I don't think that is going to change from WCAG 2.0 to Silver. If testing with Silver costs 5 times what it does to test WCAG 2.0, but the down stream benefits are the same, Silver is going to be a difficult sell. I want to distinguish here between two arguments. My first one is that testing costs matter, and we should set ourselves some goals. To Luis point, I don't think there are current estimates, but we can most certainly come up with them if we ask different organisations to give us some numbers. Man-hours per page for example?

My second argument, that we should seriously consider lowering the cost of testing Silver at Bronze level, from testing WCAG 2.0 Level AA, is this. For most organisations, especially for smaller organisations, an accessibility audit is the first thing they do once they've decided they want to invest in making their digital content more accessible. It is their first opportunity to learn about accessibility, to develop some basic knowledge about how web accessibility applies to their digital assets. We've all seen cases where people thought they were doing just fine until they saw that first audit report come back. Accessibility audits are a key tool in our arsenal for raising awareness and developing basic accessibility skills. For most smaller organisations it is often the only thing they'll ever invest in when it comes to accessibility.

If Silver is going to make accessibility testing more expansive, than it is not unreasonable to expect that fewer organisations will make that initial investment in having an audit done. I think that lowering the costs of accessibility audits lowers the barrier to entry for Silver. That is already a goal the group set itself. When I moved from testing WCAG 1.0 to WCAG 2.0, I saw a lot of smaller organisations stop working on accessibility, because they couldn't afford it anymore. Testing is an absolutely essential part of doing accessibility. If you don't test, you don't know how accessible something is. Audits are often the only tool small businesses have to verify the quality of websites that were developed by a third party. If those are not affordable, if accessibility audits cost 10% or more of the entire budget for developing a new website, accessibility testing just doesn't happen. That kind of money doesn't make business sense unless there's a legal risk to it. We're already at that level with WCAG 2.0. I've seen those kinds of number.


On Wed, Aug 29, 2018 at 2:53 AM Hall, Charles (DET-MRM) <Charles.Hall@mrm-mccann.com<mailto:Charles.Hall@mrm-mccann.com>> wrote:
I have a pretty strong opinion on this topic.

First, the cost of testing and cost of remediation represent different tasks – each with a large degree of variability. They both imply that web content and documents on a domain already exist. This omits all the cases where it doesn’t.

Accessibility guidelines are intended to be a set of instructions on how to create content and documents correctly in the first place, so that the variability of testing is easily controlled and remediation should be minimal to zero.

It is the role of advocacy groups and the cohort of accessibility professionals to continue to champion this message of “follow the guidelines, and do it right in the first place”. The guidelines should make it easy to do so.

Organizations should not have to run automated or manual tests to check if a caption exists or if headings are in the right order. These should be bare minimum expectations placed upon creators. At the very least, tests like that should be (and sometimes are) the standard linting of a unit test long before content and documents are public. This cost is not something organizations calculate (that I am aware of). It is simply a workflow.

If someone incurs an additional cost for testing and/or remediation, it will not be the fault of the guidelines. It will be the fault of not following web standards and accessibility guidelines. If we make the guidelines easier, it is reasonable to assume that testing is subsequently easier. If validation methods are (still) included in the guidelines, then half of each test case is already written.

I can see there being an added cost incurred by organizations that provide accessibility testing and remediation services, to update language and documentation and reports and refactor a handful of code and process. But I also see this is a fair and reasonable cost of doing business in a technology market.

The scope of Silver should include guidelines that are able to test with a method to test and a validation of that test. They may even include a score or degree or percentage of that validation. I don’t believe the scope should include considerations for cost. It has always been discussed that Silver should consider how it is used by various stakeholders. But “how” intentionally includes a little ambiguity. If I hire 10 people to train 1000 people on Silver for 30 minutes, that has everything to do with a business decision and nothing to do with the actual guidelines. If I write 11 new test cases that take 3 minutes each per page of a site, I find a way to make up that time elsewhere and improve upstream decisions that allow me run them less and find zero defects, and I get a better product.

The economic impact of accessibility is an argument we all use in advocacy. That should be enough to continue to win cost arguments.


Charles Hall // UX Architect, Technology

w 248.203.8723
m 248.225.8179
360 W Maple Ave, Birmingham MI 48009

Relationship Is Our Middle Name

Ad Age B-to-B Agency of the Year, 2018
Ad Age Agency A-List 2016, 2017
Ad Age Creativity Innovators 2016, 2017
North American Agency of the Year, Cannes 2016
Leader in Gartner Magic Quadrant, 2017, 2018

From: Luis Garcia <w3c@garcialo.com<mailto:w3c@garcialo.com>>
Date: Tuesday, August 28, 2018 at 2:30 PM
To: Jeanne Spellman <jspellman@spellmanconsulting.com<mailto:jspellman@spellmanconsulting.com>>
Cc: Silver Task Force <public-silver@w3.org<mailto:public-silver@w3.org>>
Subject: Re: [EXTERNAL] Costs of testing with Silver
Resent-From: Silver Task Force <public-silver@w3.org<mailto:public-silver@w3.org>>
Resent-Date: Tuesday, August 28, 2018 at 2:29 PM

I'd love to make "reduce the cost of accessibility testing" a goal. That said, is there already a way that "cost of accessibility testing" has been calculated for WCAG 1.0, 2.0, and 2.1?

If it's not something that already exists, I don't know that we'd want to have it be a primary goal. As it is, there will already be a cost associated with adjusting to Silver.

I think we might be able to lower the overall cost of accessibility by making it more a part of everyone's job. And I think we can do that by making the guidelines more accessible.


On Tue, Aug 28, 2018 at 7:55 AM Jeanne Spellman <jspellman@spellmanconsulting.com<mailto:jspellman@spellmanconsulting.com>> wrote:

This is a very interesting idea that I do think we need to discuss.  When I was first thinking about Silver two years ago, I had a thought that we could do automated testing for a basic level of accessibility.  As we went through all the research and started forming the ideas and proposals, I had forgotten about it.  I am open to looking at this in more detail.

My concern would be the amount of disability needs that could be included in reduced-cost testing, particularly the non-physical needs, like cognitive disabilities.  I know there is new research and testing in the last couple years that could be a viable solution.

I would like to schedule this discussion so we could have a number of people involved who care about this issue, and devote an entire meeting (or more) to it.

On 8/28/2018 6:19 AM, Wilco Fiers wrote:
Hey all,

Firstly, all hats off. Sharing a personal view here. I wanted to reach out about a thing that I've been concerned with regarding Silver. With WCAG 2.1 I saw some discussions about the increased cost of testing compared to WCAG 2.0. Thinking about the adoption of WCAG 2.0 from WCAG 1.0, there too I saw that the amount of work it took to do accessibility testing had increased quite significantly.

I'm concerned that (as far as I can tell) there hasn't yet been a discussion about costs of testing with Silver. I know its still early days, but I think that we should have that discussion, and decide what kind of target we'd like to hit for Silver. There are all these fantastic ideas floating around, about score cards, usability testing, expanding to include non-web technologies. Lots of great stuff, but we have to be aware that all of these things are going to have a price tag.

I would very much like for the Silver group to decide how much they think the cost of doing accessibility testing is allowed to increase. Is it okay for the costs of testing to double between WCAG 2.0 and Silver like they did from WCAG 1.0 to 2.0? Is it allowed to increase at all? Should Silver be designed to decrease costs instead?

Making Silver easier to use, lowering the barrier to entry, those are fantastic goals. But those things really don't matter if someone can't get the budget to do accessibility testing. And without testing, you can't maintain an accessible site. I don't think it's unreasonable to think that if Silver decreases the cost of accessibility testing, it could get wider adoption than WCAG 2 did. Where is if the opposite happens, if testing for Silver is far more expansive than it is for WCAG 2, that organisations might just stick with WCAG 2 for a long time to come.

Personally, I'm of the opinion that WCAG 2 is already too expansive. When I was still testing WCAG 1, I'd regularly test websites of smaller organisations. Those organisations stopped coming when the costs went up for WCAG 2. I think a good target for Silver would be that at the bronze level, costs for testing are about half what they are for WCAG 2.0 Level AA. I believe that that would make it affordable again for small businesses, which I think should be a goal for Silver.

Either way, Silver needs to be designed with an eye on testing costs, and it would help if we had some goals defined for it.

Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
Error! Filename not specified.

This message contains information which may be confidential and privileged. Unless you are the intended recipient (or authorized to receive this message for the intended recipient), you may not use, copy, disseminate or disclose to anyone the message or any information contained in the message.  If you have received the message in error, please advise the sender by reply e-mail, and delete the message.  Thank you very much.

Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG

(image/jpeg attachment: image001.jpg)

Received on Wednesday, 29 August 2018 16:47:23 UTC

This archive was generated by hypermail 2.4.0 : Thursday, 24 March 2022 20:31:43 UTC