Re: Heuristic testing

Hi Alastair,

I am the author of that document and a participant in several conversations around it.
I agree with nearly everything you stated. We had already decided that this method would not be solely or specifically recommended. We also tossed around the idea of recommending a Cognitive Walkthrough instead, since this method is task-based. There has been no conclusion on specific testing and evaluation methods. However, I feel it is still incredibly important to identify and recommend some specific human evaluation methods that can account for functional needs.

While I also concede that there may be incremental cost in evaluation for some authors for some types of content, I don’t believe it would result in any undue burden. The general idea would simply be to encourage practices that go beyond the minimum, but not require them.

As mentioned in other threads on the topic of the proposed conformance model or measurability, simply conducting an evaluation (regardless of method) would not be sufficient. The implementation of change based on the result of that evaluation is what is important. An author should not achieve a greater conformance level by simply conducting an evaluation. The dots would have to be connected to a result. We don’t yet have a great example of how this could work in a way that considers all of the functional needs. For example, if the author achieves the lowest score and level of conformance, and then conducts additional evaluation methods, discovers issues, resolves those issues, and theoretically achieves a higher score, that score should not achieve a higher level of conformance if the evaluation had only considered the functional need of “cannot see the content” and neglects all other functional needs.

Having the guidelines cite some specific evaluation methods should also not be specific to conformance. Again, the idea is to encourage practices and not just encourage higher scores. Considering and testing for and with different functional needs should not be limited to the end of a design process.

I think we all agree that this is complicated and it should not come at the cost of any users.

I would love to hear your opinion on a Cognitive Walkthrough or other methods.


Charles Hall // Senior UX Architect

charles.hall@mrm-mccann.com<mailto:charles.hall@mrm-mccann.com?subject=Note%20From%20Signature>
w 248.203.8723
m 248.225.8179
360 W Maple Ave, Birmingham MI 48009
mrm-mccann.com<https://www.mrm-mccann.com/>

[MRM//McCann]
Relationship Is Our Middle Name

Ad Age Agency A-List 2016, 2017, 2019
Ad Age Creativity Innovators 2016, 2017
Ad Age B-to-B Agency of the Year 2018
North American Agency of the Year, Cannes 2016
Leader in Gartner Magic Quadrant 2017, 2018, 2019
Most Creatively Effective Agency Network in the World, Effie 2018, 2019



From: Alastair Campbell <acampbell@nomensa.com>
Date: Tuesday, April 23, 2019 at 7:42 PM
To: Silver Task Force <public-silver@w3.org>
Subject: [EXTERNAL] Heuristic testing
Resent-From: Silver Task Force <public-silver@w3.org>
Resent-Date: Tuesday, April 23, 2019 at 7:41 PM

Hi everyone,

Jeanne mentioned a doc on the AG call today: Heuristics for Silver Conformance
https://docs.google.com/document/d/1lEkht-bhkaPMzOojWpDjZhbnGeckiYLh-J4-ze7cGS0/edit#<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1lEkht-2DbhkaPMzOojWpDjZhbnGeckiYLh-2DJ4-2Dze7cGS0_edit-23&d=DwMFAg&c=Ftw_YSVcGmqQBvrGwAZugGylNRkk-uER0-5bY94tjsc&r=FbsK8fvOGBHiAasJukQr6i2dv-WpJzmR-w48cl75l3c&m=7FNM7kdo2S6pSRSZM3lKxbgrXDj7rseIhUZzWGGF5QY&s=LtRzC9JM1Lt70U7cE70sYnQupbxFUmCQqolNUGXMTNM&e=>

The document isn’t open for comments, so a few comments here. For context: My background was originally Human Computer Interaction, I started off my professional life doing many heuristic “usability” evaluations for an agency.

The intent for heuristic evaluations was originally to come up with issues & recommendations based on the experience of the tester(s). Getting from that to a score is very arbitrary, and very prone to individual (tester) differences.

The history section notes the 80% consistency rate for WCAG, but I’d put money on that number being far lower for a heuristic approach due to the higher-level nature of the heuristics.

(Does a quick search), “when professional evaluators conducted heuristic evaluations, the most likely outcome was that about half of the problems identified would be true problems and half would  be  false positives.” [1]
My bolding of ‘professional’ as that’s a best case!

In short, I don’t see how the use of heuristic evaluation could meet the requirement to “have tests or procedures so that the results can be verified”, at least in any faintly feasible way. That isn’t what the method was created for.

I could see it being part of the gold/platinum level of conformance, or perhaps as a process check (“Have you done usability testing/heuristic evaluation and acted on the results?”)

Also, from the document: “Since a heuristic evaluation should be conducted by two or more expert evaluators” and later “Evaluators should have domain knowledge – not only in the product area, but in the heuristic area.”

This is would massively increasing costs in general, and make it something that small organisations cannot do themselves.

I’m all for new ways of measuring things, e.g. getting away from instant fails for minor-issues. However, the guidelines do need to spell out the way it is measured. It can’t rely on having a group of experts for each evaluation agreeing between themselves.

Sorry to trample in, I was just intrigued by Jeanne’s comment and wanted to find out more, then it hit a nerve of past experiences!

Kind regards,

-Alastair

1] https://www.researchgate.net/profile/Michael_Muller/publication/221515225_Usability_in_practice_Alternatives_to_formative_evaluations_-_Evolution_and_revolution/links/54dac1b70cf2ba88a68dd7de/Usability-in-practice-Alternatives-to-formative-evaluations-Evolution-and-revolution.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.researchgate.net_profile_Michael-5FMuller_publication_221515225-5FUsability-5Fin-5Fpractice-5FAlternatives-5Fto-5Fformative-5Fevaluations-5F-2D-5FEvolution-5Fand-5Frevolution_links_54dac1b70cf2ba88a68dd7de_Usability-2Din-2Dpractice-2DAlternatives-2Dto-2Dformative-2Devaluations-2DEvolution-2Dand-2Drevolution.pdf&d=DwMFAg&c=Ftw_YSVcGmqQBvrGwAZugGylNRkk-uER0-5bY94tjsc&r=FbsK8fvOGBHiAasJukQr6i2dv-WpJzmR-w48cl75l3c&m=7FNM7kdo2S6pSRSZM3lKxbgrXDj7rseIhUZzWGGF5QY&s=2Wppyv_QOdxoD428zDktN2xX78Vc8cKLiDyq6Mx3Zq8&e=> Page 886.

--

www.nomensa.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nomensa.com_&d=DwMFAg&c=Ftw_YSVcGmqQBvrGwAZugGylNRkk-uER0-5bY94tjsc&r=FbsK8fvOGBHiAasJukQr6i2dv-WpJzmR-w48cl75l3c&m=7FNM7kdo2S6pSRSZM3lKxbgrXDj7rseIhUZzWGGF5QY&s=Bo_5yVU1LuG6Ev9cTQz9mZKbEEoCuCm00d4DFSdV5Ys&e=>
tel: +44 (0)117 929 7333 / 07970 879 653
follow us: @we_are_nomensa or me: @alastc
Nomensa Ltd. King William House, 13 Queen Square, Bristol BS1 4NT

Company number: 4214477 | UK VAT registration: GB 771727411


This message contains information which may be confidential and privileged. Unless you are the intended recipient (or authorized to receive this message for the intended recipient), you may not use, copy, disseminate or disclose to anyone the message or any information contained in the message.  If you have received the message in error, please advise the sender by reply e-mail, and delete the message.  Thank you very much.