applying usability and ucd methodologies to achieve meaningful use

Applying Usability and User-Centered Design Methodologies to Achieve Meaningful Use

June 9, 2010, HIMSS Virtual Conference

Lisa Battle, Jasmin Phua & Duane Degler Design for Context

Copyright © 2010 Lisa Battle, Jasmin Phua & Duane Degler Slide 2

Conflict of Interest Disclosure

Lisa Battle User-Centered Design Lead

Jasmin Phua User Experience Researcher & Designer HIMSS EHR Usability Taskforce member

Duane Degler User-Centered Design Strategist & Semantic Web Expert

Consult for and have no real or apparent conflicts of interest to report.


Agenda

•  Meaningful Use Overview

•  Role of Usability in EHR Meaningful Use

•  Build-in Usability through User-Centered Processes

•  What is user-centered design

•  Evaluating product usability

•  Designing for usability


Objectives

This education session aims to help you:

•  Understand the role of usability in helping achieve meaningful use

•  Identify methods of evaluating usability

•  Apply user-centered design methodologies to incorporate end-user feedback


“It’s not about technology,

it’s about transforming healthcare.”

Joshua Seidman, Acting Director, Meaningful Use Office of Provider Adoption Support, ONC


What is all this Meaningful Use buzz?

Through the provisions of the American Recovery and Reinvestment Act (ARRA; Recovery Act) of 2009, the Centers for Medicare & Medicaid Services (CMS) will provide incentive payments for the meaningful use of certified electronic health record (EHR) technology.

Source: Medicare and Medicaid Programs; Electronic Health Record Incentive Program; Proposed Rule. January 13, 2010


Who qualifies?

Eligible professionals (EPs), eligible hospitals, critical access hospitals (CAHs) for:

•  Efforts to adopt, implement, or upgrade certified EHR technology,

or

•  Meaningful use in first year of participation and for demonstrating meaningful use during each of 5 subsequent years.



How is Meaningful Use measured?

•  Stage 1 has been defined; meaningful use objectives and certification criteria provided by CMS and ONC.

•  Stages 2 & 3 have not been define yet but policy priorities have been painted in broad strokes.

•  Different measures for eligible professionals (EPs) vs. hospitals.

Sources: Medicare and Medicaid Programs; Electronic Health Record Incentive Program; Proposed Rule. January 13, 2010. Health Information Technology: Initial Set of Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology; Interim final Rule. January 12, 2010.

Stage 1 Stage 2 Stage 3

2011 2016 incremental process


Meaningful Use: Stage One

Stage One (2011-2012) focuses on: •  Electronically capturing health information in a coded format, •  Using that information to track key clinical conditions, •  Communicating that information for care coordination purposes, •  Implementing clinical decision support tools to facilitate disease and

medication management, •  Reporting clinical quality measures and public health information.

Defined measures for: •  Eligible professionals (EPs): 25 measures •  Eligible hospitals: 23 measures



Why Meaningful Use?

•  Improvements in quality, safety, efficiency, and reductions in health disparities,

•  Engagement of patients and families in their health care,

•  Improvements in care coordination, •  Improvements in population and public health, •  Adequate privacy and security protections for

personal health information.

Anticipated health policy outcomes for meaningful use of EHR technology are:


Example: Meaningful Use Criteria

Health outcomes policy priority

Care goals

Objectives

Measures

Improving quality, safety, efficiency, and reducing health disparities.

Use evidence-based order sets and CPOE. Apply clinical decision support at the point of care.

Implement drug-drug, drug-allergy, drug-formulary checks. (same for eligible professionals and hospitals)

Eligible professional/hospital has enabled this functionality.

EHR meaningful use


Role of Usability in Meaningful Use

Achieving meaningful use requires successful implementation of certified EHR technology.

People get things done quickly and productively

They get the info they need, complete work accurately and achieve their goals

They feel confident and pleased

The technology does not get in the way

They don’t make mistakes

They are not frustrated

Efficiency

Effectiveness

Satisfaction

Quality attribute defined in ISO 9241, Part 11 Usability


Why does usability matter?

Good usability can help...

reduce increase Patient and staff safety Productivity and accuracy Staff morale Customer loyalty Competitive advantage

Training time Error rates Staff turnover Product liability Customer support


14

Why does usability matter?

Risks of poor usability •  Endangers patients •  Increases adverse events •  Information overload leading to erroneous

decisions •  High costs of training, errors, rework •  Increased product & practice liability and last but not least...

Barrier to EHR implementation and adoption


Poor Usability = Adverse Events

Over a 2 year period, voluntary reporting of adverse events resulting from health IT malfunctions to FDA found:

260 reports with potential for patient harm 44 reports of injuries 6 deaths

Usability-related adverse events:

How many are unreported or caught before they become serious problems?

Source: Jeffrey Shuren, Director of CDRH, FDA. Testimony to ONC Health IT Policy Committee, February 25, 2010.

The user documented activities in the task list for one patient and used the “previous” or “next” arrows to select another patient chart, the patient’s task list displayed for second patient.

A sleep lab’s workstation software had confusing user interface, which led to the overwriting and replacement of one patient’s data with another patient’s study.

< prev next >

Task List

1. _______

2. _______

3. _______

BEST EMR SYSTEM

John Saint

John Smith

Janet Smith

L. Smite

Paul Smote


Poor Usability=EHR Adoption Barrier

Administrators, clinicians, CIOs, CMOs, and policymakers listed Top 10 barriers to EHR implementation:

Source: John Halamka, CIO, Beth Israel Deaconess Medical Center during “Leadership Strategies for Information Technology in Health Care” class at Harvard. February 1, 2010. http://geekdoctor.blogspot.com/2010/02/top-10-barriers-to-ehr-implementation.html

# 10 Usability Hard to use and not well engineered for clinician workflow.

# 9 Politics/naysayers Every organization has a powerful clinician or administrator who is convinced that EHRs will cause harm, disruption, and budget disasters.

# 8 Fear of lost productivity Concerned they will lose 25% productivity for 3 months after implementation.

# 7 Computer illiteracy/training Many clinicians are not comfortable with technology; often reluctant to attend training sessions.

# 6 Interoperability Applications do not seamlessly exchange data for coordination of care, performance reporting, and public health.

Solvable with good usability


Poor Usability=EHR Adoption Barrier

Administrators, clinicians, CIOs, CMOs, and policymakers listed Top 10 barriers to EHR implementation:

Source: John Halamka, CIO, Beth Israel Deaconess Medical Center during “Leadership Strategies for Information Technology in Health Care” class at Harvard. February 1, 2010. http://geekdoctor.blogspot.com/2010/02/top-10-barriers-to-ehr-implementation.html

# 5 Privacy Significant local variation in privacy policy and consent management strategies.

# 4 Infrastructure/IT reliability Many IT departments cannot provide reliable computing and storage support, leading to EHR downtime.

# 3 Vendor product selection/ suitability Hard to know what product to choose, particularly for specialists who have unique workflow needs.

# 2 Cost the stimulus money does not flow until meaningful use is achieved. Who will pay in the meantime?

#1 People Hard to get sponsorship from senior leaders, find clinician champions, and hire the trained workers to get the EHR rollout done.

Barriers mitigated with user-centered processes


Biggest EHR Usability Problem

HIMSS EHR Usability Pain Points Workgroup conducted a survey on the specific factors that resulted in poor usability (catch their presentation on Thursday, June 10).

Workflow is the overwhelming problem in almost all facets

“Must view many areas to capture the entire patient’s story”

“Too much info in too many different places, getting lost and overwhelming”

“Doesn't match clinician thought process”


“We failed to focus on the most important part of the decision—the human/computer interface....we didn’t listen to our guts on the design of the computer screen that we would have to look at for hours on end. It is about functionality and workflow.”

-Joseph G. Cramer, MD “Bought Wrong EMR” in Medical Economics Magazine

February 5, 2010, pp 28-30


Good Usability = a Good “Fit”

•  “Fit” between the object and … •  Its purpose •  Human mental and physical capabilities •  Target audience •  Environment in which it will be used •  Tasks it will be used for


Usability by Design

•  Usability does not happen by chance •  It can’t be “painted on” at the end •  It can’t be achieved through testing alone...

but usability testing gives great insights as to improvements needed!


How to build-in usability?

Usability Test

Release Date

Timeline

Fantastic! And we don’t even

need any training

Usability Test

Release Date

Observe the work

Interview users

Timeline

Brainstorm design ideas

Get user feedback

Create designs

Usability test

Revise designs

Timeline

This isn’t what I

expected…

Common problem


What is User-Centered Design?

Industry best practice for creating usable products from the beginning

Risk mitigation strategy

Perspective

Toolkit of methods & guidelines

Discipline based on human factors

ISO 13407

task analysis

walkthroughs

usability testing

evidence-based design

ethnographic studies

usage tracking

affordances

predictability

working memory

feedback

mental models

Fitts’ law


How does UCD help with Meaningful Use?

When done right... •  Meaningful use requirements are met in a way that fits

with how all staff members truly work. •  Your EHR has features that your staff actually need,

rather than unnecessary “cool ideas”. •  Finding clinician champions and gaining buy-in from

key stakeholders is easier because it is an inclusive process. (helps solve the “people” barrier!)

Participatory process that involves true end-users and other key stakeholders.


How does UCD help with Meaningful Use?

When done right... •  Realistic, productive workflows for meaningful use

requirements are identified. •  Scenarios where meaningful use criteria are applicable

and can be met are comprehensively identified. •  The impact of “meaningful use” implementation on

staff duties and responsibilities is anticipated.

Practice predicated on getting into users’ heads to understand specific:

•  Characteristics •  Tasks and goals •  Context in which they do work

EHR


User-Centered Design Process

Understanding the needs

Designing a solution that works

Who are the users?

What are their tasks and goals?

What situations bring them here?

What are their expectations?

Best practices for usable design

Progressive refinement

Multidisciplinary collaboration

ITERATION

Brainstorm

Test with users

Design


Always start with goals

•  Business goals

•  Stakeholder goals

•  User goals

•  Usability goals

  Business goals include care goals and meaningful use objectives as defined in the proposed rules.


UCD: Observe & Analyze

Observe &

Analyze

Envision &

Design

Evaluate &

Refine


Conduct user studies

A variety of activities that gather information about •  Users •  Tasks •  Context of use

Users are the actual people who use the product. Users are not:

•  The CEO •  Their organizations and managers •  You and the development team •  Your public affairs or marketing department

Observe & Analyze Envision & Design Evaluate & Refine


User research methods


•  Interviewing

•  Site visits/contextual inquiry

•  Usability testing

•  Surveys

•  Focus groups

•  Analysis of emails, requests, or issues

•  Conferences, training, user group meetings

•  Usage logs

•  Search logs

Not all at once—choose the techniques that fit best with your project


Different user groups have different needs

Emergency room unit

General practice physician

Patient Insurance and billing


When researching user needs, gather requirements from: •  Direct users •  Indirect users, e.g. billing •  Others who have contact with users, e.g. caregivers


UCD in Meaningful Use: User Needs

Meaningful use objective: Implement drug-drug, drug-allergy, drug-formulary checks.

Certification criteria example: Automatically and electronically generate and indicate in real-time, alerts at the point of care for drug-drug and drug-allergy contraindications based on medication list, medication allergy list, age, and CPOE.

Who will use these drug-drug and drug-allergy checks? Do these user groups have the same needs? Which user group can override alerts? Should they?


Document user needs in Personas

Personas are: •  Personal, composite sketches

of real users •  Discovered through user

research •  Representative of typical

users, not edge cases

Used to: •  Debunk false assumptions •  Help envision users and

design what is best for them •  Keep a focus on the user

throughout the project



Analyze user tasks


Tasks are specific activities that people need to accomplish. •  Examples:

•  Complete all prescription refill requests •  Discuss the MRI results with the neurologist •  Call Ms. Nelson to discuss test results

•  Many tasks include both information and action

Read over patient record, concentrating on problem list. Consider diagnosis possibilities.

Schedule discussion time with neurologist

Information Action

Discuss MRI results with the neurologist

Task


How to analyze tasks

•  List them

•  Match them to user groups

•  Prioritize them based on

•  Frequency

•  Criticality of failure

•  Break them down into

•  their component parts, and/or

•  the sequence of steps involved

•  Write stories that illustrate them (scenarios)

Freq

uenc

y Criticality

Low Medium High

High

Medium

Low



Documenting tasks in scenario form

Scenarios are realistic narrative descriptions of activities that users engage in, detailed enough that design implications can be inferred

—Based on J.M. Carroll, 1995

How it works: •  Write the story of the work from

the user’s perspective •  Share the story with team to help

them visualize how tasks occur in the actual work environment

•  Conduct walkthroughs of the proposed designs using these scenarios



Example: UCD in Physician Environment

Source: What’s Keeping Us So Busy in Primary Care? A Snapshot from One Practice by Richard J. Baron, M.D., New England Journal of Medicine, 362; 17, April 29, 2010S

Type of Service

Total No.

No. per Physician (day)

Telephone call 21, 796 23.7

Laboratory report 17,794 19.5

Visit 16,640 18.1

E-mail message 15,499 16.8

Consultation report 12,822 13.9

Prescription refill 11,145 12.1

Imaging report 10,229 11.1

4 FTE physicians, each working 50-60 hrs/wk, 230 days/yr.

Frequent (e.g. daily, weekly)

Infrequent (e.g. quarterly, annually)

One thing at a time (single)

Several at a time (multiple)


Would want to understand: •  Typical amount of time spent

during each service interaction •  Most beneficial patient

interaction •  Other categories of services e.g.

health plan correspondence, FMLA forms

•  Types of interruptions and frequency

•  Tasks that are queued up to be addressed all at once vs. piecemeal


Example: UCD in ER or ICU

Adapted from Representing Reality: The Human Factors of Health Care Information, C P. Nemeth,M. O'Connor, M. Nunnally, and R I. Cook Chapter 28, Handbook of Human Factors and Ergonomics in Health Care and Patient Safety

Clinician is developing individual and shared mental model of patient

How would you automate an ER

whiteboard?

Patient care at individual clinical and unit levels

Pt 1 needs CT scan, had blood

work-up this morning

Pt 1 scheduled for CTC at 1500. Needs

line changed beforehand


Pt 2 doesn’t look good. May need to be intubated.


UCD in Meaningful Use: Task & Context



In what situations will these alerts come up? •  Acute episodes / hospitalizations •  Chronic conditions •  Primary care encounters •  Preventative care

In what context will these alerts come up? Emergency room, general practice, pharmacy. ICU System

Drug alert! Do not use vancomycin

X


UCD: Envision & Design

Observe &

Analyze

Envision &

Design

Evaluate &

Refine


Bridging the gap from analysis to design

Humans are good at some things…

…machines are good at other things.

To optimize the system, let each focus on what they are good at.

calculate BMI

lookup drug allergies

remind me about

drug interactions



“It is all about design, which we see every day, but mostly ignore....Design of the computer screen and the underlying program is how our brains see the whole picture of the patient.”

-Joseph G. Cramer, MD “Bought Wrong EMR” in Medical Economics Magazine

February 5, 2010, pp 28-30

EHR


Design is Hard

•  You don’t get it right the first time •  There are always trade-offs •  Keys to success:

•  Start with a deep knowledge of your users and their tasks

•  Generate multiple ideas up front (e.g. through parallel design and brainstorming)

•  Iteration – walkthroughs and user feedback •  Progressive refinement •  Following guidelines and patterns for usable design



“Computers are strong medicine. Done well, they are wonderful: done poorly they can kill people”

- Justin Starren MD Marshfield Clinic, Wisconsin

Source: As Doctors Shift to Electronic Health Systems, Signs of Harm Emerge, by Fred Schulte and Emma Schwartz, April 20, 2010, Huffington Post


UCD in Meaningful Use: Design



I talked to my users and stakeholders, now magic happens!

It can still can go very wrong. For example, CPOE systems often flood doctors with warning alerts, leading physicians to ignore them, which is a human factor principle known as “alert fatigue” or “pop-up fatigue”.

Source: Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors, Ross Koppel; Joshua P. Metlay; Abigail Cohen; et al. JAMA. 2005;293(10):1197-1203


Magic happens or not...

Dangerous Drug Interaction: Warfarin x Sulfa

MyCPOE System Patient: Jane Smith

Patient Summary

Clinical Notes

Problem List

Medication

-----------------------

Consults

Lab Results

Warfarin x Sulfa interaction: Potential for bleeding

Pt currently on: Coumadin (warfarin)

Dangerous Drug Interaction

Don’t fill

Patient currently on: Coumadin, since 03/09/2001

Consider alternatives:

Warfarin x Sulfa Potential for bleeding details

Cephalexin Nitrofurantoin

Change order

Fill order


Visioning

What are we trying to build?

What would be best for our users?

What will help them accomplish their tasks?

How can we meet their expectations?

What is the best way to meet our goals?

Example: Designing a new house



Visioning

•  In user-centered design, this involves:

•  “Blue sky” brainstorming sessions

•  Fast, informal generation of ideas

•  Flip charts

•  Whiteboard drawings



Parallel Design

Generates a wide range of design possibilities quickly The full group discusses pros and cons of each

Sketch #B “What I really like about this

idea is…?”

Serious alerts are easy to spot Like being able to see the rest of the patient’s record

Alert is helpful and lets me change the order if I need to. I don’t need to click through 5 screens again to do that.

It lets me get to more details so I don’t need to go back out to look it up


Sketch #C Sketch #A


How should it be organized?

What are its main sections?

What will people do in each area?

What will people expect each area to be called?

How can we streamline the path from one section to another?

Structural Level: Organizing




Structural Level: Organizing

Abstract prototype


•  Abstract prototypes

•  Sitemaps

•  Flow charts

•  Card sorts

Sitemap



Abstract Prototype

Used in discussion with stakeholders to clarify content and organization



Which things should be near each other because they go together?

Is this the right style of interaction?

Does this layout support the flow of the task?

Representing and Refining




Representing and Refining


•  Sketches

•  Low-fidelity prototypes or mockups

•  Wireframes

Paper prototypes

Wireframes



Types of Prototypes

Good for testing:

- Concepts

- Organization

- Screen flow

- Main ideas

Good for testing:

-  Terminology

-  Headings

-  Navigation

- User tasks

Good for testing:

-  Visual appeal

-  Interactions

-  Accessibility


Low Fidelity High Fidelity


Iteration

•  Design in repeated cycles •  Results of each cycle feed into the next cycle

•  Get input and feedback early and often •  Prototypes don’t need to be working yet •  Less “finished” looking, more options, elicit more feedback

ITERATION

Brainstorm Design

Test with users

(or walkthrough with specialists)



UCD: Evaluate & Refine

Observe &

Analyze

Envision &

Design

Evaluate &

Refine


Many Ways to Get User Feedback

•  In addition to usability testing, you can use: •  Informal, scenario-based walkthroughs •  Surveys •  Web metrics and usage tracking •  Management information on transactions •  Help desk log •  Emailed feedback



User Feedback Sessions

•  Set expectations •  Ask the user to “think aloud” and interpret what they see •  Ask the user about realistic tasks •  Ask the user to compare alternatives


Refer to handout for details


Testing for Usability

Get real users

Ask them to perform realistic tasks using your system, prototype, or web site

Observe, take notes, and see what works and what doesn’t work

If they have problems, fix them before the release!



Types of Usability Tests

•  Formal vs informal •  Formal: In a lab with two-way mirrors, logging

software, video cameras, observers •  Informal: In a cafeteria, senior center, at home, or

wherever, with nothing but your prototype

•  Formative vs summative •  Formative: Identify problems and opportunities for

improvement •  Summative: Determine whether performance

measures were met, or to set a baseline •  In person vs remote




Typical Measures for Usability

•  Efficiency •  Time to complete task •  Number of clicks •  Number of days/hours training reduced

•  Effectiveness •  Success rate (or completion rate) •  Number or % of errors •  Number of attempts before successful completion •  Cost savings from reduced errors

•  Satisfaction •  Number of positive and negative statements or feedback

messages received from users •  Satisfaction scores on a survey (e.g. SUS, QUIS, ACSI) •  Number of users who rate the system as good or excellent •  Actual usage (number of people, % increase)



Design Walkthroughs

Walkthrough Abstract prototype

Low-Fi Prototype

High-Fi Prototype

1) Scenario-Based Walkthrough 2) Persona-Based Walkthrough 3) Requirements Walkthrough 4) Subject-Matter Expert Walkthrough 5) Database Walkthrough 6) Information Needs Walkthrough 7) Usable Design Principles Walkthrough 8) Accessibility Principles Walkthrough




Is there a magic number?

We have often been asked... “Is there a magic number to figure out if my product passes/fails usability?”

Why? Usability is measured by:

and is about balancing user needs. It is not a threshold measure.

Is there a magic number for human clinical trials? Why?

Efficiency Effectiveness Satisfaction


No, there isn’t.


When user requirements go awry


Takeaway: Do it early

Planning Requirements Design Development Roll-out Validation

Cost of

modifications

Identify usability and accessibility needs as early as possible to reduce costs


Takeaway: Build it in from the beginning

Involve real users early and often

Observe their actual work

Work collaboratively with a multi-disciplinary team

Follow human factors & usable design guidelines

Design the user interface deliberately

Iterate the design with user feedback


Takeaway: Start now!

Start now! You have the means...

•  Look at your paper forms and their contents: •  What you are collecting and why? •  How do you intend to use it when it is digital?

•  Analyze patterns of work e.g. patient requests, repetitive fixed events, tasks everyone can do

•  Look at your current workflow. What’s optimal? •  Scrutinize problem logs. Make sure you don’t

automate the problem source!


Better design, better healthcare

errors fatigue repetitive work stress & frustration loss of time training needs

ease of use productivity success rate human comfort user acceptance satisfaction

Transform healthcare through better design

reduce increase

Questions?

Lisa Battle: [email protected] Jasmin Phua: [email protected]

Duane Degler: [email protected]

applying usability and ucd methodologies to achieve meaningful use

Health & Medicine