there is no such thing as user error

There is no such thing as User Error

PRESENTER:

Gavin LewManaging Director

User Centric, Inc.

Pamela Stoffregen-GaySr. Marketing Mgr.

User Centric, Inc.

MODERATOR: #uxlunch

@UserCentricInc

Korey JohnsonAssociate Director

User Centric, Inc.

Join the discussion!

2

Company Overview: Snapshot Profile

2

• Founded in 1999; privately held firm with over 50 full time UX research and design consultants

• Most have graduate degrees in psychology, human factors / ergonomics, ethnography, design and human computer interaction

• In 2007, 2008, 2009, 2010, 2011 User Centric was included on the Inc. 5000 List of the fastest growing companies in the U.S.

• UC is arguably the strongest user experience firm in the US

• Healthcare (medical devices and HIT: EHRs)

© 2011 – Proprietary and Confidential

Introduction: In Search of Use Errors

3© April 10, 2023 – Proprietary and Confidential

No One Wants to Make a Device that Causes Errors


Introduction

Usual Suspects When Use Errors Are Committed

Designers cannot ever think that being “highly trained” solves a medical device that is poorly designed


Introduction

Increased Vulnerability

While it is true that HCPs are highly educated, often the tasks performed puts them into an “automatic mode” which makes

them more vulnerable to use errors*


Introduction

* Guenter, et al (2008). Enteral Feeding Misconnections: A Consortium Position Statement, The Joint Commission Journal on Quality and Patient Safety, 34(5): 282-292.

Don’t Forget Other Users…

Originally, pharmacists were not considered to be a relevant user group for use errors, but Flynn, et al. (2003) estimated 51.5 million errors occur per year (4 per day per pharmacy)

Moreover, a human factors-related recall with pharmacists as the user as recent as September 23, 2011


Introduction

What About Patients?

Especially when a medical device used by HCPs is “repurposed” for use with patients…!


Introduction

Patients…


Introduction

Are incredibly diverse – Literacy, education, experience

Receive a wide range of training– Skewed towards no training

Have different perspective and context

Live in varied environments – i.e., uncontrolled environments

– Consider: dogs, younger children

Do research long enough and patients resemble…

A Truly Diverse User Group


Introduction

Must Users Really Shoulder the Blame?


Introduction

Definition of a Use Error

Use errors are made by humans and can often be attributed to the design itself

“Use” as opposed to “human error” or “user error” removes blame from the user as the cause of the failure

Targeting design deficiencies of the design itself links remedy to design changes (earlier the better)

Often, we focus on the device itself, but – Interaction with other devices, connections (things) and

– Instructions also contribute to use errors


Introduction

Focus Towards Improved Risk Management

In an effort to improve patient safety…– The healthcare industry

has focused more attention on risk management tools

– Awareness of the important role that human factors plays in the design process

– By driving more human factors into the design process sooner, rather than later, results in more positive change


Introduction

What is FMEA?


FMEA: Failure Modes and Effects Analysis

Answers the simple question: “If a system component fails, what is the effect on system

performance or safety?”

Usage dates back to the 1940’s to evaluate military M&P to identify risks using a bottom-up assessment

As technology advanced into areas, such as space exploration and nuclear power, the need to identify, prioritize and manage risk amplified

At that time, assessment of risk in the design of systems tended to involve engineering and material failure.


What is FMEA?

Engineers: Two types of FMEA Risk Analyses

1. DESIGN FMEA. A design FMEA is used to examine the components of a product to identify potential failures. – For example, in the automotive industry, a design FMEA is

conducted on all components and subsystems of a new car during the design and manufacturing phases.

– The FMEA tool is used to evaluate the correctness of the materials, accuracy of specifications, and all other elements of design required to make a safe automobile.

2. PROCESS FMEA. A process FMEA is used to analyze the processes used to make a product.


What is FMEA?

Wait — Where is the User in the FMEA?


What is FMEA?

Need to Focus on the User in the FMEA


How to…

“If a system component fails, what is the effect on system performance or safety?”

Severity of Harm (S)

Probability (P)

(S x P)

“If a user commits an error, what is the effect

on system performance or safety?”

How to Conduct an FMEA


Towards a More “Use–Error Focused-–FMEA”

Expanding upon a chapter entitled “Risk management in medical products” by Ed Israelski and Bill Muto In Pascale Carayon’s recently published textbook, Handbook of

Human Factors and Ergonomics in Health Care and Patient Safety, Second Edition

Ed and Bill describe a step-by-step process to identify hazards and integrate it into the medical product development process

They advocate one major addition to the FMEA procedure which adds the user into the process

– Perform a Task Analysis as this can predict user behaviors that relate to design interaction


How to…

But First, Here is an Example FMEA

FMEA appears in many different styles Ultimately, it is really a worksheet (table) Used by the team to document and work through issues Often has color to pull attention to specific rows


How to…

12-Step Process (Israelski & Muto, 2012)

1. Form a Team

2. Perform a Task Analysis

3. Start a Worksheet

4. Brainstorm Potential Use Errors (failure modes)

5. List potential Effects of each Failure Mode / operator error

6. Assign Severity ratings (S)

7. Estimate the Probability of occurrence ratings (P)

8. Derive a Risk Index (S x P)

9. Prioritize Risks by risk index values

10. Take Actions to eliminate or reduce the high priority failure modes

11. Estimate Effectiveness of action ratings

12. Revise the Risk Priorities


How to…

Israelski, E., & Muto. W. (2012). Risk management in medical products. In P. Carayon (Ed.), Handbook of Human Factors and Ergonomics in Health Care and Patient Safety, Second Edition. CRC Press.

Step 1: Form a Team

Team composition includes representatives from:– Product team

– Regulatory

– Human Factors

– Engineering (Development)

– Medical Affairs

– Quality Assurance

– Training

– User Group representatives

Agree upon goals and procedures to conduct the FMEA


How to…

Step 2: Conduct a Task Analysis

Conduct a formal task analysis

Don’t get lured into “we’ll do the task analysis AND brainstorm”

Provide team with tasks, user profiles, use environment, and examples of use errors


How to…

Back toEntryPoint

User may be first-time or repeat visitorUser is just browsing for information on upcoming events "I want to see what's coming soon"

Just Browsingfor Info by:- Act- Venue- Date- Genre- Geographic Area- Upcoming Events

Back toEntryPoint

Just-browsing...Seeing what's available...

Browse by Name(Event / Venue)

Browsing by Genre Browsing by Date Browse by Deals

Search by Name

Show GenreShow me what's coming up

this weekendShow me latest deals

Show search resultsList of Acts: MetroList of Venues: Metroe.g., Amazon.com

TixAvail?

TM.com should offer help- Email new dates- Avail tickets- Next Concert- etc.

Anythinginteresting?

TM.com should help- Try another venue?- List headliners

Go toBuy

"What'splaying at theMetro?"

"What dealsare available?"

"What isavailable thisweekend?"

"What wouldbe fun to dowith thefamily?"

Show List of Events

Select Act orVenue

SelectPerformance

Yes

No

ExitTicketmaster.com

No

Yes

Depression

Try to limit browseto 3-4 steps

Select Genre

Select sub-categories

Show sub-category lisitng

Search by full listing?AlphaNumeric Jum

Jump to Date

Sort by VenueSort by State

Sort by State

Closest to Zip

Search by Price

Show List of Acts

Search by full listing?

AlphaNumeric Jump

ShowEvents

ShowEvents

ShowEvents

Search by DatePromo on Home

Link to Promos

Show Genre

Select Promo

ShowEvents

Save “brainstorming” for identifying use errors

Step 3: Start a Worksheet

Settle on the format and columns

Consider using large sheets of paper (11” x 17” or larger) with gridlines

Documenting electronically via MS Excel and projecting screen to a wall can facilitate group interactions and discussions


How to…

Step 4: Brainstorm Potential Use Errors (Failure Modes)

Follow typical brainstorming rules (i.e., facilitate ideation over debate over merit discussions)

Recognize efficiencies of breaking into sub-groups to reduce “tangent takeover”

Actually helps to have the device (sounds silly…) Use data from customer complaints or MDRs


How to…

Step 5: List Potential Effects of Each Failure Mode

For each use error, describe the potential harm resulting from the Potential Effect

When multiple harms can occur– List harms and elevate worst-case scenarios to the top, so

they can be addressed first


How to…

Step 6: Assign Severity Ratings (S)

Assign a quantitative level of the Severity of the Harm Various levels of (S) are often used


How to…

Example of a Five-level Severity Scale (S)

Three-level Severity Scale– “Severe, Moderate, Negligible”


Methodology

Step 7: Estimate the Probability of Occurrence Ratings (P)

Assign a quantitative level of the Probability of the use error occurring

Various levels of (P) are often used


How to…


(P) levels can be 3 or 10 point scales derived from:– Historical data, prediction of probabilities using analytical

or simulation techniques, experimental data, and even expert judgment.


Methodology


Consider that this (P) encompass situations and circumstances, as well as, an entire sequence of events from the occurrence of the initiating cause through to the occurrence of harm– Does the hazardous situation occur in the absence of a

failure?

– Does the hazardous situation occur in a fault condition?

– Does the hazardous situation occur only in a multiple-fault condition?

– How likely is it that a hazardous situation will lead to harm?

– The likelihood that a hazardous situation will lead to harm is influenced by the life-cycle of the medical device and estimated number of devices in the market


Methodology

Step 8: Derive a Risk Index (S x P)

Risk Level = Severity x Probability Forms a Risk Priority Number (RPN) Compare to a Risk Evaluation Matrix


How to…

Risk Evaluation Matrix


Methodology

Regulators require special attention be paid to all catastrophic harm severities regardless of (P) – Historically, too many manufacturers have low balled (P)

Step 9: Prioritize the Risks by Risk Index Values

Sort by Risk Index Level Consider Risk Control Measures

– Failure modes that can be addressed immediately

– Failure modes that require

no action


How to…

Step 10: Take Action to Eliminate or Reduce Risk

Talking point:


How to…

Really Cool Opening Scene in a Nuclear Missile Silo


Introduction

West Wing star was more skeptical when a message came through and was hesitant to turn his key…

Forcing the Reservoir Dogs Star to Suggest He Comply


Introduction

“Turn your key, sir!”

Potentially Dangerous Actions Should Require More Effort

This is where design can play a role to mitigate use errors and improve patient safety

Many medical device manufacturers mitigate risk based on the ALARP (As Low As Reasonably Practicable) – Accepts that there is a practical limit to the time, effort, and

money that can be spent trying to drive a risk to zero Still, manufacturers must justify

– Burden is to show that cost required to reduce the tolerable risk further would be grossly disproportionate to the benefit gained and risks are those that are improbable and marginally severe thereby requiring no further mitigation

Make it hard for users to do something wrong

and easier for them to do it right


Introduction

Step 11: Estimate the Effectiveness of Actions

Did the Risk Control Measure make the device safer by:– Eliminating the particular hazard?

– Reducing the Severity (S) and/or Probability (P)?

– Adding protective measures, such as safeties or alarms?

– Providing warning labels and instructions?

Step 12: Revise Risk Priority (Risk Acceptability)


How to…

ISO 14971:2007 Annex Updates

Taking FMEA a Step Further…


Addendum to the 12 Steps…

This paper seeks to evolve the process and take it to the next level

The critical outcome of the FMEA with a Task Analysis is a list of potential use errors However, these potential use errors are limited to

what can be ideated in the brainstorming activity All too often, these brainstorming activities occur in a

closed office environment or even worse, sometimes conducted without the device!


A Step Further…

Where Does Brainstorming Occur?


A Step Further…

Review FMEA in a Simulated Environment


A Step Further…

Break Out When Necessary


A Step Further…

Should a set of use errors be uncovered…– Break off sub-team to explore the new line of errors while

the main team will continue down the list

Takeaways

An FMEA process combined with Task Analysis improves risk management

Addition of simulated environments can increase the number of identified use errors and precision

The goal is not to box the answer correctly the first time, but to get smarter all through the research process– FMEA should continue to evolve from exploratory to

validation testing


A Step Further…

References

ISO 14971:2007 (with special emphasis on Annex D and E) Flynn, E.A., Barker, K.N., & Carnahan, N.J. (2003). National Observational

Study of Prescription Dispensing: Methods, J Am Pharm Assoc. 43(2): 191-200.

Guenter, P., Hicks, R.W., Simmons, D., Crowley, J., Joseph, S., Croteau, R., Gosnell, C., Pratt, N.G., & Vanderveen, T.W. (2008). Enteral Feeding Misconnections: A Consortium Position Statement, The Joint Commission Journal on Quality and Patient Safety, 34(5): 282-292.

Israelski, E., & Muto. W. (2012). Risk management in medical products. In P. Carayon (Ed.), Handbook of Human Factors and Ergonomics in Health Care and Patient Safety, Second Edition. CRC Press.

Kaye, R,, & Crowley, J. (2000). Medical device use-safety: Incorporating human factors engineering into risk management. Food and Drug Administration

Logan, M.K. (2010). A roundtable discussion: Increased focus on human factors drives device safety improvements. Human Factors Horizons. 8-15


49

PRESENTER:

Gavin LewManaging Director

User Centric, Inc.

[email protected]

#uxlunch

@UserCentricInc

Korey JohnsonAssociate Director

User Centric, Inc.

[email protected]

Join the discussion!

Up next:Conducting Longitudinal Mobile StudiesTuesday, Aug 21, 12:00 PM CDTwww.usercentric.com/webinars

Questions?Questions?

there is no such thing as user error

Health & Medicine

design fmea

design process

process fmea

design consultants

design of systems

otherelements of design

fmea tool

use error use errors