RE: Accessibility testing for onrr.gov

Hello all,

Weighing in here because I'm a Trusted Tester (original and refresh). Also WAS certified (unfortunately recently expired).

I started my accessibility journey by learning HTML from a handful of printed pages of HTML tags with brief explanations of what they were for, and Notepad for my text editor. I was lucky enough to be on a small group of inquisitive people who were learning the same way, so we were able to discuss and collaborate. While we were learning HTML we also were learning WCAG 2.0, primarily from the specifications and WebAIM materials.

That is to say, our group started very basic, without a lot of formalized training. After a number of months, starting from zero relevant skills and information, our team was what I would call low-intermediate level accessibility testers.

We faced a few challenges - primary among them was agreeing on whether or not a particular element or pattern constituted a failure. Some things are quite clear, others are not. The normative language of WCAG is brief, and the Understanding and Techniques documents offer a lot of non-normative guidance, but both the normative and non-normative documentation can be hard to understand and quite open to interpretation.

Our group started studying for the original Trusted Tester exam and it was an extremely frustrating experience. On their face the testing procedures should have made things more clear and repeatable, but the reality was that the team had to forget a lot of what we had learned independently on our own and learn how to catalog issues based on Trusted Tester criteria. However, some of the testing processes were new to us and improved our overall testing methodology. More valuable than the Trusted Tester process itself were the heated discussions among the team as we tried to figure out what on earth the Trusted Tester process was asking us to look for, and how to record what we found. In the end, we were all certified and felt extremely accomplished. It was hard. 

We integrated some of the testing processes from the original Trusted Tester into our own methodology and allowed it to inform our interpretation of some results, but we didn't rely on the testing process unless we had a federal client who specifically requested it.

Fast forward a few years. The shop is more mature, our processes are more nuanced, and we now have both senior and junior members on the team. WCAG 2.1 has been released and added to our testing process, and we continue to refine our testing process based on new tools, new patterns, new specifications. Trusted Tester Refresh is released and as the Technical Lead I was the first to go through the program. 

Again, I had to forget much of the nuance I learned over the previous few years. I immediately realized how narrow the testing scope was, and how particular the process was about how/where an issue is logged. This is, of course, to produce the repeatability that Peter values, and that many federal agencies use as their benchmark. I regard the scope of Trusted Tester as so narrow and rigid as to not be a good fit for providing clients with what I would consider to be a valid, robust audit - unless, of course, that is what they are requesting.

Now, another few years later in my accessibility career, I only very occasionally use ANDI, the primary tool for the Trusted Tester process, and almost always cross-check the results in the accessibility tree. I don't ever refer to Trusted Tester testing processes for our team, and if there is a reason for me to need to look at the process I still find it confusing, narrowly scoped, and too rigid.

In my position now as the lead for a small, close knit team I have quite a bit of input on the hiring of new employees, at least as it relates to their technical qualifications for the position. When evaluating junior folks who have the Trusted Tester certification I do not consider the candidate to be "ready to jump in and test" - certainly not at the level we require for client deliverables. I know that if we hire someone who is a certified Trusted Tester there will be a week or more of working closely with the new hire, training them on our testing and documentation process, and probably a few months of increased support and oversight as the person learns how to dig beyond what Trusted Tester requires. I also know that the new hire should have a solid understanding of the success criteria, basic testing methodology, and the concepts and language to be a solid junior tester.

From where I stand, the benefit of Trusted Tester is that orientation to WCAG and accessibility, a baseline of testing process, and learning how to map issues to the correct success criteria. It is not that it produces a robust audit, addresses emerging technologies and patterns, or that it should be where a person's testing training ends.

For full transparency, the shop I was originally trained in was Access2Online, Peter's shop.

I hope everyone has a lovely holiday seasons, and thanks for the opportunity to reflect on what the real value of Trusted Tester is.

Best regards,
Juliette
On 11/23/2022 4:45:30 AM, Steve Green <steve.green@testpartners.co.uk> wrote:
That’s an interesting observation. However, being good at your job isn’t a binary good / useless distinction. There is a wide range of “good”, ranging from “tolerable” to “genius”. People at the “tolerable” end of that scale will make some mistakes that are picked up on review and they might make sub-optimal recommendations for fixes. Unless you are at the “genius” end of that scale you should be able to learn something useful from a certification.
 
Most of my team have 18 to 20 years’ accessibility experience and no one has less than 10 years. Nevertheless, we spend 2 to 3 hours on a conference call every week, discussing new testing techniques, new code patterns we’ve seen, new assistive technology behaviours and drilling deep into what the various WCAG success criteria really mean.
 
The fact that some of the team didn’t know those things previously doesn’t make them bad testers – it just illustrates how there is an unending amount to learn. That’s why I am opposed to a testing methodology that crystallises the use of tools and techniques that are 5 or more years old, as if we knew everything back then and nothing has changed.
 
Steve
 
 
From: bryan rasmussen <rasmussen.bryan@gmail.com>
Sent: 23 November 2022 09:57
To: Steve Green <steve.green@testpartners.co.uk>
Cc: w3c-wai-ig@w3.org
Subject: Re: Accessibility testing for onrr.gov
 
> but even he says he hasn’t learned anything he can use to improve the quality of his work.
 
maybe being naive here but I would expect that if you are good at your job doing a certification meant to provide a baseline of quality for people doing that job wouldn't be expected to give you anything you could use to improve the quality of your work?
 
 
On Wed, Nov 23, 2022 at 3:28 AM Steve Green <steve.green@testpartners.co.uk [mailto:steve.green@testpartners.co.uk]> wrote:
One of my team is doing the Trusted Tester Program, so I do have some insight into it. He is doing it because he is one of those people who likes to collect certifications, but even he says he hasn’t learned anything he can use to improve the quality of his work.
 
I understand that some people “want a defensible position if they are to give a vendor the bad news that they are not compliant”, but I regard that behaviour as toxic. They are essentially saying that the Program can’t be wrong because it gives the same answer no matter who does the test. In my view it’s far better for a skilled tester to do the best job they can. If anyone wants to challenge their findings, the tester should be able to justify them rather than hiding behind the Program.
 
I have changed my mind when a client has presented a compelling argument against my findings, and I have changed my team’s findings when reviewing their work. I see that as a positive because it increases our understanding and the accuracy of our work. The repeatability of the Trusted tester Program means that if the mandated testing process gives the wrong result, it will also give the wrong result when someone else reviews the test report. How can that be good?
 
Steve
 
From: Peter Shikli <pshikli@access2online.com [mailto:pshikli@access2online.com]>
Sent: 22 November 2022 21:28
To: Dan Horning <dsoundmn@gmail.com [mailto:dsoundmn@gmail.com]>
Cc: Steve Green <steve.green@testpartners.co.uk [mailto:steve.green@testpartners.co.uk]>; w3c-wai-ig@w3.org [mailto:w3c-wai-ig@w3.org]
Subject: Re: Accessibility testing for onrr.gov [http://onrr.gov]
 
Steve and Dan,
 
Whereas I do agree that the Trusted Tester certificate program does prioritize repeatability, I would disagree about how it trivialises WCAG testing.  Its structured and objective repeatability ensures a reliable level of quality that doesn't depend on the individual analyst's feelings in the matter. I realize that we each feel we're good at our job, but federal Program Managers need an assurance beyond, "trust me, I'm talented".
 
In cases where a lot of money is on the line, federal Procurement Agents want to check their Section 508 compliance box confident that they are getting accessibility even when the Procurement Agent doesn't have the job skills to verify. And they want a defensible position if they are to give a vendor the bad news that they are not compliant. This is why the federal Office of Accessible Systems and Technology was tasked with launching and maintaining the Trusted Tester Program.
 
As a Trusted Tester myself, I do agree that such a certificate is not the end of accessibility learning. We're given a baseline upon which we are expected to build, and the Trusted Tester VPAT provides a way to report beyond their baseline, particularly when it comes to remediation suggestions which they barely touch on. I have found Deque and IAAP's certification programs provide much of the overall accessibility training that Trusted Testers could use, but to say the Trusted Tester Program harms our profession, I suggest you first complete the Trusted Tester Program yourself and then weigh in with your verdict.
 
Cheers,
Peter Shikli
Access2online Inc.
www.access2online.com [http://www.access2online.com]
Prison inmates helping the internet become accessible
 
On Tue, Nov 22, 2022 at 12:31 PM Dan Horning <dsoundmn@gmail.com [mailto:dsoundmn@gmail.com]> wrote:
Thank you, Steve,
 
This is one of the best ways I've seen someone address this. I completely agree. 

 
Daniel Horning
518-333-5100
 
 
On Tue, Nov 22, 2022 at 1:24 AM Steve Green <steve.green@testpartners.co.uk [mailto:steve.green@testpartners.co.uk]> wrote:
While I would give the team credit for doing more than many, perhaps most, teams, further testing would be required to meet the stated objective of "make sure this website was as accessible as possible for all users", which is a pretty high bar.

As Peter rightly says, a manual WCAG audit is essential. It will find most of the issues that user testing finds, but it will do so more easily and cheaply. You get the best results from user testing if you do it after the website is fully WCAG conformant.

However, I do not recommend the Trusted Tester certification. It trivialises the WCAG testing by reducing it to a checklist-based procedure that prioritises repeatability over accuracy. It does not take into account the need for investigation, which is essential given the complexity of modern code. It does not allow testers to use their skill or even develop their skill beyond what the course teaches. As such, I believe it harms our profession.

I recommend doing assistive technology testing before the user testing. They are not the same thing at all. In an assistive technology test, a skilled consultant tests every part of each page. User testing is usually task or scenario based to assess whether end to end tasks are accessible, so the user only views part of each page. The sessions should be moderated and involve members of the general public. The two types of testing are complementary and neither is sufficient on its own.

User testing ought to include other screen readers as well as screen magnifiers and voice recognition software.

Steve Green
Managing Director
Test Partners Ltd


-----Original Message-----
From: Peter Shikli <pshikli@access2online.com [mailto:pshikli@access2online.com]>
Sent: 21 November 2022 19:21
To: w3c-wai-ig@w3.org [mailto:w3c-wai-ig@w3.org]
Subject: Re: Accessibility testing for onrr.gov [http://onrr.gov]

Mr. Bailey / Ms. Thomas,

Glad to hear of accessibility progress. Did you use Trusted Testers certified by the federal Office of Accessible Systems and Technology to audit and validate the new onrr.gov [http://onrr.gov] website's accessibility per Section 508, perhaps by signing a Letter of Reasonable Accessibility to post on the website?

Cheers,
Peter Shikli
Access2online Inc.
www.access2online.com [http://www.access2online.com]
Prison inmates helping the internet become accessible


Bruce Bailey wrote on 11/21/2022 11:09 AM:
> From:  https://blog-nrrd.doi.gov/four-part-hybrid-testing/ [https://blog-nrrd.doi.gov/four-part-hybrid-testing/]
>> Before launching the newly redesigned onrr.gov [http://onrr.gov], we wanted to make sure this website was as accessible as possible for all users.  We previously chronicled our efforts to make all documents on onrr.gov [http://onrr.gov] accessible, and developed next steps for 508 compliance within ONRR.  Part of these next steps was to extend accessibility beyond linked documents to the entire website...
> Website under discussion is:  https://www.onrr.gov/ [https://www.onrr.gov/] U.S. Department of
> the Interior (DOI) Office of Natural Resources Revenue (ONRR)
>> Open data, design, & development at the U.S. Department of the
>> Interior
> Please do not reply to me directly, as this great example and interesting best practice story is not anything I was involved with.  I found it worth sharing, and I pass it along with the author's permission.
> For more information via email, please use:  nrrd@onrr.gov [mailto:nrrd@onrr.gov]
>
>
> From: Agency IT Accessibility Coordinators (Sec508ITAC)
> <SEC508ITAC@LISTSERV.GSA.GOV [mailto:SEC508ITAC@LISTSERV.GSA.GOV]> On Behalf Of Thomas, Christine L
> Sent: Monday, November 21, 2022 11:09 AM
> To: SEC508ITAC@LISTSERV.GSA.GOV [mailto:SEC508ITAC@LISTSERV.GSA.GOV]
> Subject: [508] New accessibility blog post!
>
> Hello,
>
> The newly redesigned onrr.gov [http://onrr.gov] is now live!
>
> We published a new blog post outlining our accessibility testing on
> the new onrr.gov [http://onrr.gov]:  https://blog-nrrd.doi.gov/four-part-hybrid-testing/ [https://blog-nrrd.doi.gov/four-part-hybrid-testing/]
>
> The blog post focuses on testing the new website prior to launch. We incorporated WCAG and Section 508 requirements "from the ground up" during the design process.
>
> Ensuring that our websites are accessible is an ongoing and iterative process for us, and we're happy to share these first steps.
>
> Thanks,
>
> Christine Thomas
> Program Analyst
> Open Data, Design, & Development
> Office of Natural Resources Revenue
>   
>