Performance Measurement: The nuts and bolts of

establishing effective measures

Mary Davis, DrPH, MSPH; NC Institute for Public Health

Kristin Adams, Ph.D., CHES; Indiana State Department of Health

Performance Measurement Definitions

Performance measurement is the “regular collection and reporting of data to track work produced and results achieved”

Performance measure is “the specific quantitative representation of capacity, process, or outcome deemed relevant to the assessment of performance”

Performance measurement is NOT punishment

Organizational Performance

Infrastructure Process Coordination Results

Leadership & Decision making

Boards of Health Policies &

Procedures Enforcement/

Investigation Protocols

Financial management

Information & reporting

Set of Core Indicators

Program performance goals/ objectives

Assessment processes

CD investigation case write-ups

EH enforcement action and case files

Education sessions

Work of community groups and coalitions

Communication mechanisms

Public and private work on access to services

Hand-offs between local health and state programs

Program evaluation results

Key indicator outcomes (CD/EH/PP)

Financial performance

Monitoring the results of performance review against PHAB Standards and Program Evaluation

Monitoring Indicators and Outcomes

Mary Davis, UNCDr. Mary Davis is a Senior Investigator at the NC Institute for Public Health and Adjunct Assistant Professor in the Department of Health Behavior, both part of the UNC, Gillings School of Global Public Health. Since Dr. Davis conducts public health systems and services research and program evaluations using participatory approaches; specialty areas are accreditation, quality improvement, and public health partnerships.

Her experiences in these areas have been shaped by participating in the Multi-State Learning Collaborative, evaluation of the NC Local Health Department Accreditation program, and working with the NNPHI and CDC to evaluation the National Public Health Improvement Initiative. Dr. Davis earned her doctorate from the Johns Hopkins University School of Public Health and her masters in science in public health from the University of Alabama at Birmingham.

Dr. Kristin Adams is the Director of the Office of Public Health Performance Management at the Indiana State Department of Health (ISDH).  She oversees agency performance management, quality improvement, and future public health accreditation efforts.  She was the Principal Investigator in Indiana for the Multi-State Learning Collaborative grant. She is currently the Principal Investigator for the National Public Health Improvement Initiative cooperative agreement from CDC. 

Prior to joining ISDH, Dr. Adams worked for a national non-profit agency focusing on program evaluation and training for health programs targeted toward girls and young women. Dr. Adams has taught numerous health education courses and has presented at national and international conferences.  Dr. Adams holds a BS and MA in School Health Education from Indiana State University, a Ph.D. in Health Education from Southern Illinois University Carbondale, and is a Certified Health Education Specialist.

Kristin Adams, ISDH

Performance Measurement: Nuts and Bolts

Basics

Mary V. Davis, DrPH, MSPHOpen Forum for Quality Improvement

November 20, 2013

Outline

• What is a performance measure?• Where are performance measures in a

performance management system?• What should be in place before establishing

measures?• What’s a system of work?• What are the characteristics of a useful process

measure?• How do you go about establishing useful

process measures?

What is a Performance Measure?

• Performance Measurement is regular collection and reporting of data to track work produced and results achieved.

• Performance Measure is quantitative representation of – Capacity– Process– Outcome

Where are Performance Measures in a Performance Management System?

YOU ARE HERE!

Standards and Measures: PHAB Example

• Standard 9.1: Use a performance management system to monitor achievement of organizational objectives.– Measure 9.1.2 A Implement a performance

management system.– Use a process to determine and report on

achievement of goals, objectives, and measures set by the performance management system.

What Should be in Place Before Measures?

Quality Improvement Projectsand

Process ImprovementOpportunities

Resu

lts and

Cu

stom

er, Stak

eho

lder, P

artner F

eedb

ack

Vis

ion

Mis

sion

and

Val

ues

Goa

ls a

nd O

bjec

tives

Str

ateg

ies

Str

ateg

ic P

lan

nin

g 

Perform

ance Measures

and Operational Indicators O

peratio

nal/

Bu

siness P

lans

 

Performance Monitoring and Analysis

 

 

Performance Monitoring and Analysis

Improvem

entO

pportunitiesIm

p ro vement

Op po rtun it i es

 

Washington State Department of Health Example:

Results help to inform the planning process

Line of Sight Framework

Restaurant inspections conducted and reports produced

restaurants achieving acceptable sanitation grades can post grades

Public uses restaurants with acceptable sanitation grades

Reduction in food-borne illnesses and disease outbreaks

Work Process Measure

Process Outcome

Short term Outcome

Long term outcome

What is a System of Work?

Factory: How?

Widget: What?

Customers: Who?

Outcomes: Why?

Focus on the Widgets

• What’s a Widget?

Widgets

• Are Things• Are Deliverables• Can be Counted• Are Specific

Common Public Health Agency Widgets

• Environmental Permits, Notices of Violations• Rules and regulations• Vaccinations• Epidemiology Reports• Infrastructure

– Filled Vacancies– Contracts

What Widgets Do You Produce?

• Take a minute: write down 5 widgets you produce

• What examples do you have to share?

Measures: General Considerations

• Measures reveal our values• Measures drive behavior• Measures can inspire us• Measures can help us learn

Useful Measures: Specifics

• Meaningful• Focused on customer

needs• Accurate• Valid• Reliable• Responsive• Functional

• Credible• Simple enough to be

understood• Available• Abuse-Proof• Cost effective to

collect and report• Comparable data

over time

SMART Measures

• Specific• Measurable• Aggressive• Results-Oriented• Time-bound

Developing Measures

• Talk with customers and those involved in the process

• Identify which process measures– Are of value to customers and producers– Will drive behavior– Will inspire– Will promote learning

• Determine how process measures fit into other measurement processes in the agency (performance standards, strategic plan)

What to Include in a Measure

• Statement of the Measure• Target (population, process)• Numerator and denominator• Target goal• Who will collect the information

What to Include in a Measure

• Form or tool to collect information• How often data will be collected• Who will conduct data analysis• How often will the data be reported

Data Description FormPerformance measure:

Target population:

Numerator:

Denominator:

Which are you using—a target or benchmark?

What is the target/benchmark?

SMART objective:

Source of data:

Who will collect the information?

How often will the data be analyzed?

How often , by what mechanism and who reports these data and analysis?

Baseline measurement data and date(s):

Definitions, such as accuracy and validity, and other comments:

MarMason Consulting

Resources

• “We Don’t Make Widgets” Ken Miller• Turning Point Resources: Guidebook for

Performance Measurement• Marni Mason tools and presentations• Washington State Department of Health QI

Plan

Thank You!Mary V. Davis, DrPH, MSPH

mary_davis@unc.edu

Getting to the 500 foot level of Performance

MeasuresKristin A. Adams, Ph.D., CHES

Ask a series of questions:◦ Who is your customer◦ What do they care about◦ What is the purpose of your

funding/mission/vision of the program area◦ Does this goal and objective tie to the strategic

plan Does your program have a role in the key priorities of

the agency◦ Can you measure it

Is it meaningful???

Setting Goals/Objectives

Several areas of resources to determine measures:◦ State law/code◦ Program requirements ◦ Healthy People 2020◦ Historical data of the program◦ Benchmarks by national groups (i.e., March of

Dimes)

What should we measure?

Develop the key outcome measure◦ Decrease Smoking Rates◦ Decrease Obesity Rates◦ Increase Immunization Rates◦ Decrease the late payment fees

Identify key activities to get you to making a change in the measure◦ Increase number of people receiving coaching from quitline◦ Increase the complete streets communities throughout the

state ◦ Reminder recalls--% of those who become compliant due to

the reminder notice◦ Decrease invoice wait time; increase invoice processing

time

How Do We Get There

Development of a Dashboard

Sometimes it is all about Process

Ensure your epidemiologists are involved Ensure the experts to the program are

involved If you are doing a “data pull” ensure the

IT/database manager is involved Involve Executive Leadership Do a “mass” education campaign

Lessons Learned