Re: Costs of testing with Silver

Hi, Wilco,
Thank you for bringing this discussion up. I believe it is both relevant and important.
I think we need to agree on what we are discussing. As I see it there are different purposes for assessing the accessibility of websites, and different methods and tools are useful and effective for different purposes. I typically see these three main categories:

  1.  Conformance testing/validation/verification – where the goal is to assess whether a digital solutions meets the requirements set out in e.g. WCAG or Silver. The motivation may be following legislation, company policies or validating the quality of a product in a procurement process.
  2.  Identifying possible accessibility improvements – where the goal is to identify and prioritize potential improvements to the accessibility (and normally also usability) of a digital solution. In this type of assessment guidelines, heuristics or more detailed lists of requirements may be useful.
  3.  Accessibility progress monitoring – where the goal is to assess/benchmark the level of accessibility in a digital solution. Many organizations will want to do this to identify trends, verify they are making sufficient progress etc., or this may be an activity carried out by monitoring bodies.

These different purposes call for different methods and tools. If your goal is to improve the accessibility of your website – existing or under development – you need clear guidelines that tell you how to do things right, and how to identify situations where you need to improve.
For the first and last scenarios what you need are not guidelines to tell you how to make your website accessible, but clear and concise requirements that are testable,  - either by qualitative or quantitative evaluation methods. For cost reduction purposes, I personally believe that using technology for automated or semi-automated, as well as tool assisted, evaluations should be a priority.

In the hypothetical scenario where the guidelines were so clear and understandable that most website designers, developers, content writers etc. could use them effectively and efficiently, cost is less of an issue. In most other scenarios cost is a significant factor, imho, that needs to be taken into account as what we are really developing – also with Silver – is not only guidance on making accessible web content, but at the same time tangible and testable accessibility requirements.

Best,
Stein Erik




Stein Erik Skotkjerra
Lead Accessibility Strategist, EMEA & APAC

[cid:image002.png@01D37335.32C5A170]<http://siteimprove.com/da/>

Sankt Annæ Plads 28  |  DK-1250 København K
Mobile +45 26 34 34 55  |  ses@siteimprove.com<mailto:ses@siteimprove.com>

Facebook<https://www.facebook.com/Siteimprove>   Twitter<https://twitter.com/Siteimprove>   LinkedIn<https://www.linkedin.com/company/siteimprove>

Unsubscribe<http://go.siteimprove.com/hs/manage-preferences/unsubscribe-simple>Stein Erik Skotkjerra

From: Wilco Fiers <wilco.fiers@deque.com>
Date: Wednesday, 29 August 2018 at 13.02
To: "Hall, Charles (DET-MRM)" <Charles.Hall@mrm-mccann.com>
Cc: "w3c@garcialo.com" <w3c@garcialo.com>, Jeanne Spellman <jspellman@spellmanconsulting.com>, "public-silver@w3.org" <public-silver@w3.org>
Subject: Re: Costs of testing with Silver
Resent-From: <public-silver@w3.org>
Resent-Date: Wednesday, 29 August 2018 at 13.01

Hey Charles,
Let me verify that I understand your position. So you are suggesting that costs of accessibility testing should not factor into the considerations for Silver at all? So in a hypothetical situation where accessibility testing from Silver ends up costing 5 times as much as an existing WCAG 2.0 Level AA audit, that would still be acceptable for you? What about if it ends up as 20 times the cost of a WCAG 2.0 Level AA audit? I would wager a guess that there is an upper limit to what you think is acceptable costs for accessibility audits. For most people there would be, so lets talk about it and see what we think it is.

I see your point of down stream benefits, but I don't think that is going to change from WCAG 2.0 to Silver. If testing with Silver costs 5 times what it does to test WCAG 2.0, but the down stream benefits are the same, Silver is going to be a difficult sell. I want to distinguish here between two arguments. My first one is that testing costs matter, and we should set ourselves some goals. To Luis point, I don't think there are current estimates, but we can most certainly come up with them if we ask different organisations to give us some numbers. Man-hours per page for example?

My second argument, that we should seriously consider lowering the cost of testing Silver at Bronze level, from testing WCAG 2.0 Level AA, is this. For most organisations, especially for smaller organisations, an accessibility audit is the first thing they do once they've decided they want to invest in making their digital content more accessible. It is their first opportunity to learn about accessibility, to develop some basic knowledge about how web accessibility applies to their digital assets. We've all seen cases where people thought they were doing just fine until they saw that first audit report come back. Accessibility audits are a key tool in our arsenal for raising awareness and developing basic accessibility skills. For most smaller organisations it is often the only thing they'll ever invest in when it comes to accessibility.

If Silver is going to make accessibility testing more expansive, than it is not unreasonable to expect that fewer organisations will make that initial investment in having an audit done. I think that lowering the costs of accessibility audits lowers the barrier to entry for Silver. That is already a goal the group set itself. When I moved from testing WCAG 1.0 to WCAG 2.0, I saw a lot of smaller organisations stop working on accessibility, because they couldn't afford it anymore. Testing is an absolutely essential part of doing accessibility. If you don't test, you don't know how accessible something is. Audits are often the only tool small businesses have to verify the quality of websites that were developed by a third party. If those are not affordable, if accessibility audits cost 10% or more of the entire budget for developing a new website, accessibility testing just doesn't happen. That kind of money doesn't make business sense unless there's a legal risk to it. We're already at that level with WCAG 2.0. I've seen those kinds of number.

Wilco

On Wed, Aug 29, 2018 at 2:53 AM Hall, Charles (DET-MRM) <Charles.Hall@mrm-mccann.com<mailto:Charles.Hall@mrm-mccann.com>> wrote:
I have a pretty strong opinion on this topic.

First, the cost of testing and cost of remediation represent different tasks – each with a large degree of variability. They both imply that web content and documents on a domain already exist. This omits all the cases where it doesn’t.

Accessibility guidelines are intended to be a set of instructions on how to create content and documents correctly in the first place, so that the variability of testing is easily controlled and remediation should be minimal to zero.

It is the role of advocacy groups and the cohort of accessibility professionals to continue to champion this message of “follow the guidelines, and do it right in the first place”. The guidelines should make it easy to do so.

Organizations should not have to run automated or manual tests to check if a caption exists or if headings are in the right order. These should be bare minimum expectations placed upon creators. At the very least, tests like that should be (and sometimes are) the standard linting of a unit test long before content and documents are public. This cost is not something organizations calculate (that I am aware of). It is simply a workflow.

If someone incurs an additional cost for testing and/or remediation, it will not be the fault of the guidelines. It will be the fault of not following web standards and accessibility guidelines. If we make the guidelines easier, it is reasonable to assume that testing is subsequently easier. If validation methods are (still) included in the guidelines, then half of each test case is already written.

I can see there being an added cost incurred by organizations that provide accessibility testing and remediation services, to update language and documentation and reports and refactor a handful of code and process. But I also see this is a fair and reasonable cost of doing business in a technology market.

The scope of Silver should include guidelines that are able to test with a method to test and a validation of that test. They may even include a score or degree or percentage of that validation. I don’t believe the scope should include considerations for cost. It has always been discussed that Silver should consider how it is used by various stakeholders. But “how” intentionally includes a little ambiguity. If I hire 10 people to train 1000 people on Silver for 30 minutes, that has everything to do with a business decision and nothing to do with the actual guidelines. If I write 11 new test cases that take 3 minutes each per page of a site, I find a way to make up that time elsewhere and improve upstream decisions that allow me run them less and find zero defects, and I get a better product.

The economic impact of accessibility is an argument we all use in advocacy. That should be enough to continue to win cost arguments.

Cheers,


Charles Hall // UX Architect, Technology

charles.hall@mrm-mccann.com<mailto:charles.hall@mrm-mccann.com?subject=Note%20From%20Signature>
w 248.203.8723
m 248.225.8179
360 W Maple Ave, Birmingham MI 48009
mrm-mccann.com<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mrm-mccann.com%2F&data=02%7C01%7Cses%40siteimprove.com%7C61b1af3ec6714aec2dfb08d60d9ee03e%7Cad30e5bc301d40dba10a0e8d40abe0f9%7C0%7C0%7C636711373362081379&sdata=pp5Ew8T4mLPacJRgtN2sLgozJDvBKfXo17CUdOkyfhY%3D&reserved=0>

[MRM//McCann]
Relationship Is Our Middle Name

Ad Age B-to-B Agency of the Year, 2018
Ad Age Agency A-List 2016, 2017
Ad Age Creativity Innovators 2016, 2017
North American Agency of the Year, Cannes 2016
Leader in Gartner Magic Quadrant, 2017, 2018



From: Luis Garcia <w3c@garcialo.com<mailto:w3c@garcialo.com>>
Date: Tuesday, August 28, 2018 at 2:30 PM
To: Jeanne Spellman <jspellman@spellmanconsulting.com<mailto:jspellman@spellmanconsulting.com>>
Cc: Silver Task Force <public-silver@w3.org<mailto:public-silver@w3.org>>
Subject: Re: [EXTERNAL] Costs of testing with Silver
Resent-From: Silver Task Force <public-silver@w3.org<mailto:public-silver@w3.org>>
Resent-Date: Tuesday, August 28, 2018 at 2:29 PM

I'd love to make "reduce the cost of accessibility testing" a goal. That said, is there already a way that "cost of accessibility testing" has been calculated for WCAG 1.0, 2.0, and 2.1?

If it's not something that already exists, I don't know that we'd want to have it be a primary goal. As it is, there will already be a cost associated with adjusting to Silver.

I think we might be able to lower the overall cost of accessibility by making it more a part of everyone's job. And I think we can do that by making the guidelines more accessible.

luis

On Tue, Aug 28, 2018 at 7:55 AM Jeanne Spellman <jspellman@spellmanconsulting.com<mailto:jspellman@spellmanconsulting.com>> wrote:

This is a very interesting idea that I do think we need to discuss.  When I was first thinking about Silver two years ago, I had a thought that we could do automated testing for a basic level of accessibility.  As we went through all the research and started forming the ideas and proposals, I had forgotten about it.  I am open to looking at this in more detail.

My concern would be the amount of disability needs that could be included in reduced-cost testing, particularly the non-physical needs, like cognitive disabilities.  I know there is new research and testing in the last couple years that could be a viable solution.

I would like to schedule this discussion so we could have a number of people involved who care about this issue, and devote an entire meeting (or more) to it.

On 8/28/2018 6:19 AM, Wilco Fiers wrote:
Hey all,

Firstly, all hats off. Sharing a personal view here. I wanted to reach out about a thing that I've been concerned with regarding Silver. With WCAG 2.1 I saw some discussions about the increased cost of testing compared to WCAG 2.0. Thinking about the adoption of WCAG 2.0 from WCAG 1.0, there too I saw that the amount of work it took to do accessibility testing had increased quite significantly.

I'm concerned that (as far as I can tell) there hasn't yet been a discussion about costs of testing with Silver. I know its still early days, but I think that we should have that discussion, and decide what kind of target we'd like to hit for Silver. There are all these fantastic ideas floating around, about score cards, usability testing, expanding to include non-web technologies. Lots of great stuff, but we have to be aware that all of these things are going to have a price tag.

I would very much like for the Silver group to decide how much they think the cost of doing accessibility testing is allowed to increase. Is it okay for the costs of testing to double between WCAG 2.0 and Silver like they did from WCAG 1.0 to 2.0? Is it allowed to increase at all? Should Silver be designed to decrease costs instead?

Making Silver easier to use, lowering the barrier to entry, those are fantastic goals. But those things really don't matter if someone can't get the budget to do accessibility testing. And without testing, you can't maintain an accessible site. I don't think it's unreasonable to think that if Silver decreases the cost of accessibility testing, it could get wider adoption than WCAG 2 did. Where is if the opposite happens, if testing for Silver is far more expansive than it is for WCAG 2, that organisations might just stick with WCAG 2 for a long time to come.

Personally, I'm of the opinion that WCAG 2 is already too expansive. When I was still testing WCAG 1, I'd regularly test websites of smaller organisations. Those organisations stopped coming when the costs went up for WCAG 2. I think a good target for Silver would be that at the bronze level, costs for testing are about half what they are for WCAG 2.0 Level AA. I believe that that would make it affordable again for small businesses, which I think should be a goal for Silver.

Either way, Silver needs to be designed with an eye on testing costs, and it would help if we had some goals defined for it.

--
Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
Error! Filename not specified.


This message contains information which may be confidential and privileged. Unless you are the intended recipient (or authorized to receive this message for the intended recipient), you may not use, copy, disseminate or disclose to anyone the message or any information contained in the message.  If you have received the message in error, please advise the sender by reply e-mail, and delete the message.  Thank you very much.


--
Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
[cid:BCBD7D4B-677E-4B95-AE3F-60005DBD9EE4]