severity & criticality metrics

Thanks Juliette. 

 

I was aiming to keep this off-list because it is off the topic of recent changes to the understanding documents, so I have changed the subject. 

 

I have always been a little uncomfortable with a functional testing-style severity overlay on levels of conformance like the below.

 

The definitions of levels of conformance already take into account the impact of a failure of a given success criterion on the people using a product. 

 

In Australia, most government agencies don’t bother resolving what the below schema calls moderate or minor defects in anything that would be considered a timely manner if at all regardless of the impact on conformance or user experience. And most won’t tolerate a ‘sev1’ or ‘critical’ in the definitions below because it still carries the meaning assigned to It by functional testing schemes (and they don’t think that accessibility is important).

 

So I am wondering what the purpose of using these descriptors really is (apart from that urge we all have about trying to get accessibility through to developers)?

 

The difficulties I have with these kinds of schemas are:

1. The assignment of severity metrics is largely subjective and may be inconsistent depending on the tester
2. Level AA defects can be ‘more severe’ than level A defects and therefore resolved first 
3. this type of schema doesn’t account for defect penetration or frequency in its measuring of impact on people using a product 
4. that there is some deeper and more profound interpretation and application of WCAG as demonstrated by the definition of minor defects below that isn’t relevant to S1 – S3

 

I am interested to know how the below schema fits with the concept of conformance and conformance levels in WCAG 2.x and why levels of conformance are not used on their own as a way of determining in which order defects are resolved? Is it historical accident? Orthodoxy? Client expectation? 

 

Cheers,

Adam 

 

 

From: Juliette Alexandria <mcshanejuliette@gmail.com> 
Sent: Saturday, August 3, 2024 10:39 AM
To: Adam Cooper <cooperad@bigpond.com>
Cc: bryan rasmussen <rasmussen.bryan@gmail.com>; w3c-wai-ig@w3.org
Subject: Re: Recent changes to the WCAG 2.2 SC 2.1.1 Understanding page

 

Hi Adam,

 

Here are our explanations of severity levels:

 

Critical Defects

Critical defects represent the most severe level of accessibility issues on a website. These defects are characterized by their completely obstructive nature, which prevents users from accessing vital information, engaging with key components, or successfully completing essential processes on the website. Such defects pose a substantial barrier, making it impossible for users, especially those with disabilities, to utilize the website as intended. This category demands immediate attention and prompt remediation to ensure the website meets the WCAG 2.2 Level AA standards for accessibility.

 

Serious Defects

Serious defects are significant accessibility issues that create substantial barriers for users, particularly those with disabilities. While not entirely blocking like critical defects, serious defects severely hinder the user's ability to access information, interact with website components, or complete processes. These defects can lead to a frustrating and challenging user experience, necessitating considerable effort or alternative strategies to navigate the website. Addressing these defects is crucial for enhancing the website's usability and aligning with the WCAG 2.2 Level AA compliance standards.

 

Moderate Defects

Moderate defects are accessibility issues that, while not completely obstructive, still pose challenges for users. These defects often require additional navigation efforts, increased cognitive energy, and may complicate the user's understanding of the content. However, they do not entirely prevent access to information or completion of processes. Moderate defects can create an inefficient and tiresome experience for users, particularly those relying on assistive technologies. Rectifying these issues is important for improving the overall user experience and ensuring compliance with WCAG 2.2 Level AA guidelines.

 

Minor Defects

Minor defects are the least severe category of accessibility issues, typically involving technical non-compliance with specific WCAG specifications. These defects have minimal, if any, impact on the user experience. They might present challenges only in rare use cases, cause slight navigation inconveniences, or result in a bit of increased verbosity when using assistive technologies. While these defects are not critical for basic website functionality, addressing them contributes to a more polished and fully accessible website, adhering to the finer details of the WCAG 2.2 Level AA standards.

 

We recognize that not all defects will affect each user to the same level, but If a user will be affected, this is how we evaluate the severity.

 

We advise our clients to combine this with page traffic metrics, or essential flows and processes within their site.

 

Combining this typically results in the most impactful issues being resolved in the most critical areas first. 

 

Of course, we do make it clear that they cannot claim full conformance unless they address all issues of all severity levels. 

 

We are also very careful to differentiate between strict WCAG defects and usability issues. Both could have any severity, while the first is specifically mapped to a specification which has legal ramifications in many locales while the latter does not.

 

Best,

Juliette 

 

On Fri, Aug 2, 2024 at 5:20 PM Adam Cooper <cooperad@bigpond.com <mailto:cooperad@bigpond.com> > wrote:

Many CRM platforms use proprietary and unconventional keystrokes for performing operations that would otherwise be ‘standard’ like pressing F4 to expand a dropdown in SAP … 

 

the highly questionable assumption being that either people will trawl through their help pages or undergo intensive training and thereby come to know these keystrokes by osmosis.

 

But then, all CRM user interfaces are notoriously badly designed and poorly coded garbage so being unusable is par for the course, I guess.

 

 

From: bryan rasmussen <rasmussen.bryan@gmail.com <mailto:rasmussen.bryan@gmail.com> > 
Sent: Saturday, August 3, 2024 2:22 AM
To: Steve Green <steve.green@testpartners.co.uk <mailto:steve.green@testpartners.co.uk> >
Cc: Patrick H. Lauke <redux@splintered.co.uk <mailto:redux@splintered.co.uk> >; w3c-wai-ig@w3.org <mailto:w3c-wai-ig@w3.org> 
Subject: Re: Recent changes to the WCAG 2.2 SC 2.1.1 Understanding page

 

I actually find it very weird, the concept of undiscoverable keyboard interactions - why would this ever exist? Is iit sort of like Easter Eggs, the devs put n for them and nobody else? 

I would expect that if someone puts in a keyboard interaction they want it to be discoverable and usable, and if it isn't that is actually a bug in their program that they would like you to point out whether or not it is an accessibility issue. 

 

 

On Fri, Aug 2, 2024 at 2:23 PM Steve Green <steve.green@testpartners.co.uk <mailto:steve.green@testpartners.co.uk> > wrote:

Thanks to everyone for all the responses. They raise a couple of questions, though:

 

1. If data entry requires the use of an undiscoverable keyboard interaction (and we do encounter them), can we report a non-conformance of SC 3.3.2 (Labels or Instructions)? The normative text and Understanding page don't mention this at all - they focus entirely on the labelling of controls and data validation rules.

 

2. If undiscoverable keyboard interactions relate to functionality other than data entry, it appears that they don't violate any success criterion. Surely that can't be right.

 

After spending an hour trawling through GitHub, I have some understanding of it. It's pretty daunting for someone who doesn't use GitHub in their work. It's safe to say I would never have found that Commit page if I didn't know it existed. And the distinction between Issues and Discussions is far from clear.

 

I have subscribed to notifications and will participate as best I can. Sadly, membership is unaffordable for me.

 

I had a look at keyboard.html commits <https://github.com/w3c/wcag/commits/main/understanding/20/keyboard.html> , but it’s full of all kinds of stuff. What I really want is a changelog for each Understanding page (and perhaps other pages such as techniques). I have no idea how easy that would be, but I will raise an issue anyway.

 

Steve

 

-----Original Message-----
From: Patrick H. Lauke <redux@splintered.co.uk <mailto:redux@splintered.co.uk> > 
Sent: Friday, August 2, 2024 10:37 AM
To: w3c-wai-ig@w3.org <mailto:w3c-wai-ig@w3.org> 
Subject: Re: Recent changes to the WCAG 2.2 SC 2.1.1 Understanding page

 

Thanks Bryan, these are all useful and good observations.

 

To the original point, these are all things that are not normatively required by the SC, and never have been. Many auditors have added these in the own interpretation or what 2.1.1 should say, and that these factors are all involved in deciding whether or not content passes or fails 2.1.1, even though this was not in the spec per se. Hence the recent additions to the understanding in 2.2 tried to clarify this, as it historically led to inconsistent audit results.

 

P

--

Patrick H. Lauke

 

*  <https://www.splintered.co.uk/> https://www.splintered.co.uk/

*  <https://github.com/patrickhlauke> https://github.com/patrickhlauke

*  <https://flickr.com/photos/redux/> https://flickr.com/photos/redux/

*  <https://mastodon.social/@patrick_h_lauke> https://mastodon.social/@patrick_h_lauke