RE: Accessibility testing for onrr.gov

I don’t have any problem with VPATs or other conformance statement formats because they don’t constrain you at all. You choose what to test, how to test it and how to report it. That’s how it should be.

My objection to most certifications (in all fields, not just accessibility) is that they demonstrably do not deliver what they claim to. Clients will expect a Trusted Tester Program to deliver the most accurate possible results, but it doesn’t – it delivers the most repeatable results at the cost of accuracy.

All of us have off days, which is why we review each other’s work. As well as improving accuracy, it’s a learning opportunity for both parties.

Steve


From: Peter Shikli <pshikli@bizware.com>
Sent: 23 November 2022 15:28
To: w3c-wai-ig@w3.org
Subject: Re: Accessibility testing for onrr.gov

Steve,

This sounds a bit like the resistance to following a VPAT of any kind, "I'm too smart to follow someone else's directions."  And yet we have found that even the most experienced analyst will have an off day when they may forget something.  This is why the most experienced airline pilot will include a checklist in his pre-flight, and as a passenger I'm glad.

Had you actually completed the Trusted Tester Program, you would know that the process was designed to allow for creative improvements over the reliable baseline. Talking to someone who has started is not the same as scoring at least 90% on their rather challenging final exam yourself.

Cheers,
Peter

Steve Green wrote on 11/23/2022 4:38 AM:
That’s an interesting observation. However, being good at your job isn’t a binary good / useless distinction. There is a wide range of “good”, ranging from “tolerable” to “genius”. People at the “tolerable” end of that scale will make some mistakes that are picked up on review and they might make sub-optimal recommendations for fixes. Unless you are at the “genius” end of that scale you should be able to learn something useful from a certification.

Most of my team have 18 to 20 years’ accessibility experience and no one has less than 10 years. Nevertheless, we spend 2 to 3 hours on a conference call every week, discussing new testing techniques, new code patterns we’ve seen, new assistive technology behaviours and drilling deep into what the various WCAG success criteria really mean.

The fact that some of the team didn’t know those things previously doesn’t make them bad testers – it just illustrates how there is an unending amount to learn. That’s why I am opposed to a testing methodology that crystallises the use of tools and techniques that are 5 or more years old, as if we knew everything back then and nothing has changed.

Steve


From: bryan rasmussen <rasmussen.bryan@gmail.com><mailto:rasmussen.bryan@gmail.com>
Sent: 23 November 2022 09:57
To: Steve Green <steve.green@testpartners.co.uk><mailto:steve.green@testpartners.co.uk>
Cc: w3c-wai-ig@w3.org<mailto:w3c-wai-ig@w3.org>
Subject: Re: Accessibility testing for onrr.gov

> but even he says he hasn’t learned anything he can use to improve the quality of his work.

maybe being naive here but I would expect that if you are good at your job doing a certification meant to provide a baseline of quality for people doing that job wouldn't be expected to give you anything you could use to improve the quality of your work?


On Wed, Nov 23, 2022 at 3:28 AM Steve Green <steve.green@testpartners.co.uk<mailto:steve.green@testpartners.co.uk>> wrote:
One of my team is doing the Trusted Tester Program, so I do have some insight into it. He is doing it because he is one of those people who likes to collect certifications, but even he says he hasn’t learned anything he can use to improve the quality of his work.

I understand that some people “want a defensible position if they are to give a vendor the bad news that they are not compliant”, but I regard that behaviour as toxic. They are essentially saying that the Program can’t be wrong because it gives the same answer no matter who does the test. In my view it’s far better for a skilled tester to do the best job they can. If anyone wants to challenge their findings, the tester should be able to justify them rather than hiding behind the Program.

I have changed my mind when a client has presented a compelling argument against my findings, and I have changed my team’s findings when reviewing their work. I see that as a positive because it increases our understanding and the accuracy of our work. The repeatability of the Trusted tester Program means that if the mandated testing process gives the wrong result, it will also give the wrong result when someone else reviews the test report. How can that be good?

Steve

From: Peter Shikli <pshikli@access2online.com<mailto:pshikli@access2online.com>>
Sent: 22 November 2022 21:28
To: Dan Horning <dsoundmn@gmail.com<mailto:dsoundmn@gmail.com>>
Cc: Steve Green <steve.green@testpartners.co.uk<mailto:steve.green@testpartners.co.uk>>; w3c-wai-ig@w3.org<mailto:w3c-wai-ig@w3.org>
Subject: Re: Accessibility testing for onrr.gov<http://onrr.gov>

Steve and Dan,

Whereas I do agree that the Trusted Tester certificate program does prioritize repeatability, I would disagree about how it trivialises WCAG testing.  Its structured and objective repeatability ensures a reliable level of quality that doesn't depend on the individual analyst's feelings in the matter. I realize that we each feel we're good at our job, but federal Program Managers need an assurance beyond, "trust me, I'm talented".

In cases where a lot of money is on the line, federal Procurement Agents want to check their Section 508 compliance box confident that they are getting accessibility even when the Procurement Agent doesn't have the job skills to verify. And they want a defensible position if they are to give a vendor the bad news that they are not compliant. This is why the federal Office of Accessible Systems and Technology was tasked with launching and maintaining the Trusted Tester Program.

As a Trusted Tester myself, I do agree that such a certificate is not the end of accessibility learning. We're given a baseline upon which we are expected to build, and the Trusted Tester VPAT provides a way to report beyond their baseline, particularly when it comes to remediation suggestions which they barely touch on. I have found Deque and IAAP's certification programs provide much of the overall accessibility training that Trusted Testers could use, but to say the Trusted Tester Program harms our profession, I suggest you first complete the Trusted Tester Program yourself and then weigh in with your verdict.

Cheers,
Peter Shikli
Access2online Inc.
www.access2online.com<http://www.access2online.com>
Prison inmates helping the internet become accessible

On Tue, Nov 22, 2022 at 12:31 PM Dan Horning <dsoundmn@gmail.com<mailto:dsoundmn@gmail.com>> wrote:
Thank you, Steve,

This is one of the best ways I've seen someone address this. I completely agree.

Daniel Horning
518-333-5100


On Tue, Nov 22, 2022 at 1:24 AM Steve Green <steve.green@testpartners.co.uk<mailto:steve.green@testpartners.co.uk>> wrote:
While I would give the team credit for doing more than many, perhaps most, teams, further testing would be required to meet the stated objective of "make sure this website was as accessible as possible for all users", which is a pretty high bar.

As Peter rightly says, a manual WCAG audit is essential. It will find most of the issues that user testing finds, but it will do so more easily and cheaply. You get the best results from user testing if you do it after the website is fully WCAG conformant.

However, I do not recommend the Trusted Tester certification. It trivialises the WCAG testing by reducing it to a checklist-based procedure that prioritises repeatability over accuracy. It does not take into account the need for investigation, which is essential given the complexity of modern code. It does not allow testers to use their skill or even develop their skill beyond what the course teaches. As such, I believe it harms our profession.

I recommend doing assistive technology testing before the user testing. They are not the same thing at all. In an assistive technology test, a skilled consultant tests every part of each page. User testing is usually task or scenario based to assess whether end to end tasks are accessible, so the user only views part of each page. The sessions should be moderated and involve members of the general public. The two types of testing are complementary and neither is sufficient on its own.

User testing ought to include other screen readers as well as screen magnifiers and voice recognition software.

Steve Green
Managing Director
Test Partners Ltd


-----Original Message-----
From: Peter Shikli <pshikli@access2online.com<mailto:pshikli@access2online.com>>
Sent: 21 November 2022 19:21
To: w3c-wai-ig@w3.org<mailto:w3c-wai-ig@w3.org>
Subject: Re: Accessibility testing for onrr.gov<http://onrr.gov>

Mr. Bailey / Ms. Thomas,

Glad to hear of accessibility progress. Did you use Trusted Testers certified by the federal Office of Accessible Systems and Technology to audit and validate the new onrr.gov<http://onrr.gov> website's accessibility per Section 508, perhaps by signing a Letter of Reasonable Accessibility to post on the website?

Cheers,
Peter Shikli
Access2online Inc.
www.access2online.com<http://www.access2online.com>
Prison inmates helping the internet become accessible


Bruce Bailey wrote on 11/21/2022 11:09 AM:
> From:  https://blog-nrrd.doi.gov/four-part-hybrid-testing/

>> Before launching the newly redesigned onrr.gov<http://onrr.gov>, we wanted to make sure this website was as accessible as possible for all users.  We previously chronicled our efforts to make all documents on onrr.gov<http://onrr.gov> accessible, and developed next steps for 508 compliance within ONRR.  Part of these next steps was to extend accessibility beyond linked documents to the entire website...
> Website under discussion is:  https://www.onrr.gov/ U.S. Department of
> the Interior (DOI) Office of Natural Resources Revenue (ONRR)
>> Open data, design, & development at the U.S. Department of the
>> Interior
> Please do not reply to me directly, as this great example and interesting best practice story is not anything I was involved with.  I found it worth sharing, and I pass it along with the author's permission.
> For more information via email, please use:  nrrd@onrr.gov<mailto:nrrd@onrr.gov>
>
>
> From: Agency IT Accessibility Coordinators (Sec508ITAC)
> <SEC508ITAC@LISTSERV.GSA.GOV<mailto:SEC508ITAC@LISTSERV.GSA.GOV>> On Behalf Of Thomas, Christine L
> Sent: Monday, November 21, 2022 11:09 AM
> To: SEC508ITAC@LISTSERV.GSA.GOV<mailto:SEC508ITAC@LISTSERV.GSA.GOV>
> Subject: [508] New accessibility blog post!
>
> Hello,
>
> The newly redesigned onrr.gov<http://onrr.gov> is now live!
>
> We published a new blog post outlining our accessibility testing on
> the new onrr.gov<http://onrr.gov>:  https://blog-nrrd.doi.gov/four-part-hybrid-testing/

>
> The blog post focuses on testing the new website prior to launch. We incorporated WCAG and Section 508 requirements "from the ground up" during the design process.
>
> Ensuring that our websites are accessible is an ongoing and iterative process for us, and we're happy to share these first steps.
>
> Thanks,
>
> Christine Thomas
> Program Analyst
> Open Data, Design, & Development
> Office of Natural Resources Revenue
>
>