2C – Using Data to Improve

Advanced Measurement for

Improvement Seminar

March 20-21, 2017

The Data CycleMeasures identified

and defined

Data collection process defined,

tested

A P

DS

Data acquisition

Data Entry

Storage,

aggregation,

analytics

Reporting

Interpretation

and application

Data Acquisition

Operational IT systems gather granular data on

standard processes

Clinical: Nursing, EHR, Labs, Pharmacy, etc.

Administrative: Billing, scheduling, etc.

Supplemented by systems to gather clinical

process data

Institutional

Ad-hoc

PDSA data is real-time, front-line, manual.

Interpretation and Application

Who needs to know what?

What level of information

How often? How soon?

Will the audience interpret the measures

appropriately?

How will you train them?

How will you keep them consistent?

Will process owners know how to respond?

How will you coach them?

Scoville, R., Little, K. et al. (2014). Sustaining Improvement. IHI White Paper. Cambridge, Massachusetts: Institute for Healthcare Improvement; 2016. IHI Whitepaper. Cambridge, MA, Institute for Healthcare Improvement. Avaliable at IHI.ORG.

A Management System Architecture

Dat

a

Dat

a

Gu

idan

ce

Align

men

t

Lean Management System

Ideal management system to support value-based

production:

Leader standard work

Visual controls

Daily accountability and planning

Respect for people who do the work

Unity of purpose

Strategic intent, operational goals, and system

views must be vertically aligned!

Mann, D. (2010). Creating a Lean Culture: Tools to sustain lean conversions. Boca Raton, FL, CRC Press.

Systems Hierarchy

Macro-systems

e.g. trust, facility, region

Meso-systems

e.g. division, clinical dept,

pathology, IT

Microsystems

e.g. unit, clinic, surgical team

‘Catchball’ P8

Source: Virginia Mason Health System

“Catchball” process aligns levels

Reporting Improvement

Senior Leaders, Boards, Executive

Sponsors (Macro-system)

Percent of target sites engaged in key

improvement initiatives

Percent of target population exposed to interventions

Phase of intervention by site or project: Plan? Pilot?

Implementation? Spread?

Time-series family of key ‘current care’ and ‘population’

measures by site, with goals

Comparison to ‘best practice,’ national/regional datasets,

comparative benchmarks

Comparison to control sites

Source: Keith Mandel MD

Reporting Improvement

Improvement Initiative Leaders, Department

Heads, etc. (Meso-system)

Time-series dashboard of all

‘current care’

and ‘population’ measures by site, with goals.

Key current care measures segmented by unit, patient

sub-population, risk groups. Measures matched to

domain of improvement work.

Current QI capability of site leaders and teams, other

‘foundational’ requirements (e.g. registry, EMR)

Degree of involvement/effort of QI teams

Data quality

Source: Keith Mandel MD

Reporting Improvement

Front-Line (Micro-system) Teams

Time-series dashboard of all

‘current care’

and ‘population’ measures by site, with goals.

Key current care measures segmented by unit, patient

sub-population, risk groups. Measures matched to

domain of improvement work.

PDSA measures for current process change testing.

Data quality

Source: Keith Mandel MD

Exercise

For Your Own Project:

Identify the key data ‘customers’ and their

relationship to (or role in) the project?

What is their degree of involvement in the

project and familiarity with QI methods?

How can you leverage measurement to

maximize their engagement in the work?

What information are they receiving now? Is it

timely and accurate?

What are your ideas for improving data

feedback?

Individuals or Groups Role in Project

Degree of

Involvement

(1=never – 5=daily)

Comprehension of

Methods and Goals Ideas for Engagement

SENIOR LEADERS, BOARDS, SPONSORS

IMPROVEMENT INITIATIVE LEADERS

FRONT LINE IMPROVEMENT TEAMS

Key Data

Customers

Currently Receiving

Information?

Time Lag,

Data Quality Ideas for Improvement

Percent of target sites engaged in key

improvement initiatives

Percent of target population exposed to

interventions

Phase of intervention by site or project:

Plan? Pilot? Implementation? Spread?

Time-series family of key ‘current care’ and

‘population’ measures by site, with goals

Comparison to ‘best practice,’

national/regional datasets, comparative

benchmarksComparison to control sites

Time-series dashboard of all ‘current care’

and ‘population’ measures by site, with

goals.

Key current care measures segmented by

unit, patient sub-population, risk groups.

Measures matched to domain of

improvement work.

Current QI capability of site leaders and

teams, other ‘foundational’ requirements

(e.g. registry, EMR)

Degree of involvement/effort of QI teams

Data quality

Time-series dashboard of all ‘current care’

and ‘population’ measures by site, with

goals.

Key current care measures segmented by

unit, patient sub-population, risk groups.

Measures matched to domain of

improvement work.

PDSA measures for current process change

testing.

Data quality

SENIOR LEADERS, BOARDS, SPONSORS

IMPROVEMENT INITIATIVE LEADERS

FRONT LINE IMPROVEMENT TEAMS

Are They

Being

Served?

Aligning Measurement for

Quality Control and

Improvement

Alignment

The measure ‘cascade’

Strategic measure

deployment

Dynamic & Static Views of a Process

0

10

20

30

40

50

60

70

80

90

100

3/1/

2008

3/8/

2008

3/15

/200

8

3/22

/200

8

3/29

/200

8

4/5/

2008

4/12

/200

8

4/19

/200

8

4/26

/200

8

5/3/

2008

5/10

/200

8

5/17

/200

8

5/24

/200

8

5/31

/200

8

6/7/

2008

Control charts show

change over time

Histogram, radar charts,

etc. show cross-

sectional ‘snapshots’ at

a point in time 0

2

4

6

8

10

12

14

16

18

20

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Caldwell, C. (1995). Mentoring Strategic Change in Health Care: An Action Guide. Milwaukee, ASQC Quality Press.

Strategic Intent and Strategic Measures

Short Term – This year’s goals

Cash flow & cost reduction

Productivity, net revenue, receivable days

Meet current clinical targets

CHF readmits

Mid Term – Next year’s goals

Increase market share

Customer satisfaction, complaints

Longer Term – 3 year goals

Increase organization agility

# Improvement projects, improvement project cycle time

Caldwell, C. (1998). Results-driven management: Strategic quality deployment. The handbook for managing change in health care. C. Caldwell. Milwaukee, ASQ Quality Press: 37-87.

Strategic Intent and Strategic Measures

Dimensions of system performance

Rate of innovation and improvement

Reduce non-value-added costs

Improve cash flow

Increase customer satisfaction

Progressively integrate the organization as a

system (additional business units, standard

practice, IT)

Vertical

Horizontal

Source: Caldwell, C. (1998)

West Paces Ferry Quality Dimensions c.1992

Productivity Sales Development Customer Loyalty

Source: Caldwell, C. (1998)

West Paces Ferry Level 1 Measures c.1992

Productivity Sales Development Customer Loyalty

Cost per member per month1

Target doctor recruits Net revenue from new products

Days to resolve a complaint

Cash flow percent prior year (growth)

Corporate contracts QI projects completed Health status – quality of life

Cost of poor quality Public awareness of brand

Employee satisfaction – open communication

Patient brag

Income percent prior year

Market share QI project percent complete

Operating expense percent prior year

Readmit percent2

1WPF was an integrated delivery system 2Quality target for corporate strategy

Source: Caldwell, C. (1998)

Kano – Customer Judgment as a Basis for

Performance Appraisal

Kano, N. (1984). "Attractive Quality and Must-Be Quality." Journal of the Japanese Society for Quality Control 14(2): 39-48.

I

II

III

III. Delightful. Unexpected and

exciting

II. Normal. A satisfactory

experience

I. Expected. Below this level repels

customers

Radar Chart: Quality Dimensions

I

II

III

Productivity

Productivity

Development

Sales

Patient BragCost of poor

quality

Source: Caldwell, C. (1998)

Suboptimized Systems

Source: Caldwell, C. (1998)

Level 1 Radar Chart in Action

Source: Caldwell, C. (1998)

The Information Cascade

Macro-systems

e.g. system, trust, facility, region

Meso-systems

e.g. service line, division,

clinical dept, pathology, IT

Microsystems

e.g. unit, clinic, surgical team

Levels of Measurement

1 - Strategic measures

• Derived from strategic

dimensions (e.g. Balanced

Scorecard)

• Target current, mid, long term

goals

• Align with strategic plan

2 - Division measures

• Structural units comprising key

organizational functions

• Most L3 are operational

‘management indicators’

3 - Business process indicators

• Measures of high-level process

effectiveness and efficiency

• Components may have different

owners

4 - Core mainstay and support process

indicators

• Single process owner

• This is where QI work is focused

(1)

(2)

(3)

(4)

(Levels)

Data

flo

w

Man

agemen

t View

Macro

Micro

Meso

Micro

Admin errorsper 100 scripts

Wrong patientper 100 scripts

% errors intercepted

Non-path orders% cases

Allergy alertsper 100 scripts

Medicationerrors % dsch

Prescribing errorsper 100 scripts

Moving up:• Cause-effect theory (e.g. driver diagram, clinical evidence)• Observed correlation (e.g. regression models)• Aggregation

Data flow to more macro levels

Management ‘line of sight’

‘Line of Sight’ Measures

Source: Caldwell, C. (1998)

# Calls to rapid response team

Environment

Hand hygiene compliance

‘Line of Sight’ Measures

Percent inpatient mortality

Compliance with “bundles”

% Surgical bundle

% Pressure ulcer bundle

% CL bundle

% VAP bundle

Hospital Acquired Infection

rates

% Sepsis bundle

L1 L2 L3 L4

AggregationDriver Model

Observed

correlation,

clinical

evidence

Aggregation Methods

• Individual Patient Data to Population

Average, median, distribution of patients: Cost, Time, Scores,

etc.

Percent conforming: Protocol-driven care

Count of events: Falls, Mortality, ADEs, etc.

• Micro to Meso to Macro

Numerators and denominators summed across units

Overall averages, medians

Average unit performance

• Aggregating Across Different Measures

Staging systems

Build composite measures or indices

Staging System

Griffin, F. A. and D. C. Classen (2008). "Detection of adverse events in surgical patients using the Trigger Tool approach" Qual Saf Health Care 17(4): 253-258.

Discussion

Consider how the aim of your project fits into

your organization’s strategic goals:

Do the key measures that track the success of

your project fit into a measure cascade within

the organization? What would that look like?

Do you have recommendations for your client

regarding a strategy for operational

measurement?

Dashboards

Examples

Why not ‘Red-Yellow-

Green’?

An ideal alternative

Who Uses Hospital Dashboards?

“Shorter, more focused dashboards that are reviewed on a

frequent basis are associated with higher performance.

According to the results of this dashboard analysis,

hospitals that use dashboards with fewer measures are

more likely to be in the high-performance group, suggesting

that higher-performing hospitals have developed

dashboards that focus on areas they see as critical for

quality. Furthermore, performance data are more

actionable when such data are consistently reviewed by the

board on a relatively frequent basis.”

Kroch et al. (2006)

A Common Type of Dashboard

Source: Provost, Murray & Britto (IHI Forum 2010)

This ‘goal-driven’ view does not provide an actionable view of system dynamics

How is Time to 3rd Next Available Doing?

Source: Provost, Murray & Britto (IHI Forum 2010)

How is Perfect Care Doing?

Source: Provost, Murray & Britto (IHI Forum 2010)

How Is Error Rate Doing?

Source: Provost, Murray & Britto (IHI Forum 2010)

Alternative

A view where

Each measure is displayed on an appropriate control chart

All control charts are on same page to see the whole system

Advantages

More accurately assess meaning of system changes

Become aware of system interrelationships

Appreciate dynamic complexity

Base decisions for action on improvement signals

HOWEVER…

Requires the viewer to understand variation!

Source: Provost, Murray & Britto (IHI Forum 2010)

12. Physician Satisfaction

Control Chart Dashboard

Source: Provost, Murray & Britto (IHI Forum 2010)

SPN Dashboard Report Fall 2010

Small Multiples

One measure,

all sites

Source: Dentaquest Institute

Small Multiples: One site, all measures

Source: Dentaquest Institute

Richard Gareth

Follow-Up Call

April 11, 2017 – 12:00 – 1:00 PM ET