john billings: applying predictive risk approaches and models effectively

June, 2012

New York University Robert F. Wagner Graduate School of Public Service

APPLYING PREDICTIVE RISK APPROACHES

AND MODELS EFFECTIVELY

WHAT I’M GOING TO TALK ABOUT

• What not to do

• What to try to do

• An example of how we almost got it right, but in the end, not so much

WHAT NOT TO DO

• Don’t do it the way we do it in the U.S.

WHAT NOT TO DO

• Don’t do it the way we do it in the U.S. – Model development limitations

WHAT NOT TO DO

• Don’t do it the way we do it in the U.S. – Model development limitations – Intervention design flaws

WHAT NOT TO DO

• Don’t do it the way we do it in the U.S. – Model development limitations – Intervention design flaws – Intervention implementation flaws

WHAT NOT TO DO


• Don’t do it the way you do it in the U.K. [With noteable exceptions]

WHAT NOT TO DO


• Don’t do it the way you do it in the U.K. – Model development limitations

[With noteable exceptions]

WHAT NOT TO DO


• Don’t do it the way you do it in the U.K. – Model development limitations – Intervention design flaws


WHAT NOT TO DO


• Don’t do it the way you do it in the U.K. – Model development limitations – Intervention design flaws – Intervention implementation flaws


WHAT TO DO

WHAT TO DO

• Model development limitations – Predict risks of expensive things you think you do something about

WHAT TO DO

• Model development limitations – Predict risks of expensive things you think you do something about – Make sure your data base has most of the key risk factors

WHAT TO DO

• Model development limitations – Predict risks of expensive things you think you do something about – Make sure your data base has most of the key risk factors – Recognize the trade-offs between model accuracy and sensitivity

CASE FINDING ALGORITHM RESULTS FROM A MULTI-HOSPITAL SYSTEM

USING ITS OWN DATA

PredictedNon-Adm

PredictedAdm

PredictedTotal

Actual - Non Adm 105,495 1,860 107,355Actual - Adm 18,459 2,909 21,368Actual - Total 123,954 4,769 128,723

Specificity 0.983Sensitivity 0.136PPV 0.610

CASE FINDING ALGORITHM RESULTS FROM A MULTI-HOSPITAL SYSTEM

USING ITS OWN DATA

Within Risk Score Range Cummulative At Cut-Off Level

RiskScore

Number ofPatients

% WithAdmission

2010

% OfAdmittedPatients

Costs2010

Number ofPatients

% WithAdmission

2010

% OfAdmittedPatients

Costs2010

0-5 2,919 2.2% 0.3% $1,769 128,723 16.6% 100.0% $7,9325-10 40,467 6.1% 11.4% $3,379 125,804 17.0% 99.7% $8,07510-15 37,909 11.8% 20.9% $5,439 85,337 22.2% 88.3% $10,30215-20 18,675 19.5% 17.0% $8,126 47,428 30.4% 67.4% $14,18920-25 9,789 26.4% 12.0% $11,044 28,753 37.6% 50.4% $18,12825-30 5,504 31.9% 8.2% $13,313 18,964 43.4% 38.4% $21,78530-35 3,508 35.9% 5.9% $15,735 13,460 48.1% 30.2% $25,24835-40 2,308 42.0% 4.5% $19,796 9,952 52.3% 24.3% $28,60240-45 1,666 45.9% 3.6% $21,343 7,644 55.4% 19.8% $31,26145-50 1,209 46.7% 2.6% $24,032 5,978 58.1% 16.2% $34,02550-55 951 51.2% 2.3% $25,686 4,769 61.0% 13.6% $36,55855-60 738 53.5% 1.8% $27,180 3,818 63.4% 11.3% $39,26660-65 612 63.1% 1.8% $33,925 3,080 65.8% 9.5% $42,16165-70 474 59.7% 1.3% $36,876 2,468 66.5% 7.7% $44,20470-75 412 59.0% 1.1% $37,404 1,994 68.1% 6.3% $45,94675-80 360 65.0% 1.1% $41,580 1,582 70.5% 5.2% $49,36380-85 307 67.4% 1.0% $42,405 1,222 72.1% 4.1% $51,65585-90 286 69.9% 0.9% $51,779 915 73.7% 3.1% $54,75990-95 252 67.5% 0.8% $53,117 629 75.4% 2.2% $56,11495+ 377 80.6% 1.4% $60,686 377 80.6% 1.4% $60,686

All Patients 128,723 16.7% 100.0% $7,932

WHAT TO DO


WHAT TO DO


• Intervention design flaws – Design the intervention after the risk model has been developed

WHAT TO DO


• Intervention design flaws – Design the intervention after the risk model has been developed – Use data from model development to help design the intervention

WHAT TO DO


• Intervention design flaws – Design the intervention after the risk model has been developed – Use data from model development to help design the intervention – Recognize you are probably going to need more information

WHAT TO DO


• Intervention design flaws – Design the intervention after the risk model has been developed – Use data from model development to help design the intervention – Recognize you are probably going to need more information – Get the incentives right

WHAT TO DO



• Intervention implementation flaws – Roll it out in at least quasi-experimental mode

WHAT TO DO



• Intervention implementation flaws – Roll it out in at least quasi-experimental mode – Track “dosage” levels (who does what to whom and how)

WHAT TO DO



• Intervention implementation flaws – Roll it out in at least quasi-experimental mode – Track “dosage” levels (who does what to whom and how) – Avoid enrollment criteria “leakage”

WHAT TO DO



• Intervention implementation flaws – Roll it out in at least quasi-experimental mode – Track “dosage” levels (who does what to whom and how) – Avoid enrollment criteria “leakage” – Evaluate impact of the intervention as rigorously as possible

A SOMEWHAT IDEALIZED SUGGESTED APPROACH TO PREDICTIVE RISK MODELING AND EFFECTIVE IMPLEMENTATION


Step 1: See if you can develop a predictive model to identify patients with risks that you think you can do something about



Step 2: Learn as much as you can about these patients to help in designing the intervention(s)

- Use available administrative data




- Use available administrative data - Apply predictive model to real patients – interview a sample of

these patients (and their providers, families, caregivers, etc.)






THEN GET SOME SMART PEOPLE IN THE ROOM AND DESIGN THE INTERVENTION






Step 3: Implement/evaluate pilot projects based on this information

- Pilot with a quasi-experimental design (intervention/control) - Conduct “formative” evaluation during early phases of

implementation - Assess impact of intervention on outcomes/utilization









Step 4: Disseminate results/Scale it up if it works










Evid

ence

d-ba

sed

man

agem

ent/p

olic

y m

akin

g

HOW WE ALMOST GOT IT RIGHT, BUT THEN NOT SO MUCH


• NY Medicaid fee-for-service patients • Adult disabled • Not on Medicare (non-duals)

• Not in residential care

NEW YORK MEDICAID CHRONIC ILLNESS DEMONSTRATION PROJECT



• Used five years of historic paid claims records

• Predicted hospitalization in next 12 months

• Ran quarterly

BASIC APPROACH FOR NY MEDICAID CHRONIC ILLNESS

DEMONSTRATION PROJECT [CIDP]

Index Quarters

Year 4 Year 5 Year 3 Year 2 Year 1 Q1 Q2 Q3 Q4

• Prior hospital utilization by type – Number of admissions – Intervals/recentness

• Prior emergency department utilization • Prior outpatient utilization/claims

– By type of visit (primary care, specialty care, substance abuse, etc) – By service type (transportation, home care, personal care, etc)

• Diagnostic information from prior hospital and outpatient utilization – Chronic conditions (type/number) – Hierarchical grouping (Hierarchical Condition Categories - HCCs)

• Prior costs – Pharmacy – DME – Total

• Characteristics of the predominant hospital and primary care provider • Patient characteristics: Age, gender, race/ethnicity, eligibility category

BASIC APPROACH TYPES OF VARIABLES USED IN NY MEDICAID’S

CHRONIC ILLNESS DEMONSTRATION PROJECT [CIDP]



Index Quarters




Index Quarters


Year 6 Q1 Q2 Q3 Q4

Intervention Quarters

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

50,000

40 45 50 55 60 65 70 75 80 85 90 95

CASE FINDING ALGORITHM NUMBER OF PATIENTS IDENTIFIED CIDP ELIGIBLE - MODEL DEVELOPMENT RUN

TOTAL FLAGGED

CORRECTLY FLAGGED

Risk Score Threshold

Pa

tient

s Id

entif

ied

False Positives

33%

False Positives

15% False

Positives 7%




- Use available administrative data

CHARACTERISTICS OF PATIENTS FLAGGED BY CASE FINDING ALGORITHM CIDP ELIGIBLE - MODEL DEVELOPMENT RUN

Demographic Characteristics

Risk Score> 50

Risk Score> 75

Risk Score> 90

AllPatients

N 33,363 8,713 2,176 64,446

Age 45.1 44.8 44.3 47.6Female 43.9% 38.5% 34.7% 49.7%

NYC Fiscal County 72.2% 80.0% 84.4% 69.1%

White 28.2% 23.6% 22.9% 32.7%Black 40.7% 48.1% 49.4% 33.1%Hispanic 15.0% 14.2% 12.2% 14.6%Other/Unknown 16.1% 14.2% 15.4% 19.5%


Diagnoses Reported in Claims Records

Risk Score> 50

Risk Score> 75

Risk Score> 90

AllPatients

Cereb Vasc Dis 4.9% 6.3% 8.1% 4.7%AMI 6.2% 9.5% 12.9% 5.2%Ischemic Heart Dis 22.5% 28.8% 35.5% 20.3%Congestive Heart Failure 16.4% 22.6% 26.9% 12.2%Hypertension 50.1% 58.3% 64.1% 48.2%Asthma 34.8% 45.7% 50.5% 26.2%COPD 23.5% 33.8% 42.3% 17.4%Diabetes 28.8% 33.7% 38.3% 26.0%Renal Disease 6.1% 9.3% 10.3% 4.1%Sickle Cell Dis 2.6% 5.2% 9.4% 1.6%

Any Chronic Disease 75.9% 86.2% 91.4% 70.9%Multiple Chronic Disease 52.2% 64.3% 73.3% 46.1%

Cancer 14.0% 13.7% 14.7% 15.1%

HIV/AIDS 23.0% 28.0% 26.1% 16.4%

Alcohol/Substance Abuse 73.0% 86.7% 90.8% 52.1%

Any Mental Illness 68.6% 78.4% 84.8% 57.2%Schizophrenia 26.7% 32.7% 36.9% 19.5%Pyschosis 19.6% 28.1% 36.6% 13.7%BiPoloar Disorder 39.0% 48.6% 54.3% 30.2%

MH or Substance Abuse 87.9% 94.4% 97.0% 73.8%MH and Substance Abuse 53.7% 70.8% 78.6% 35.6%


Selected Ambulatory Care Use Prior 12 Months

Risk Score> 50

Risk Score> 75

Risk Score> 90

AllPatients

Any primary care visit 71.7% 72.9% 68.3% 64.8%Any speciatly care visit 39.2% 40.8% 39.9% 35.6% No primary care visit 28.3% 27.1% 31.7% 35.2% No PC/spec care visit 24.2% 22.6% 26.7% 31.3% No PC/spec/OBGYN visit 23.7% 22.1% 26.1% 30.7%

Any psych visit 35.3% 35.8% 36.9% 29.6%Any alcohol/drug visit 29.5% 38.8% 38.8% 19.5%

Any dental visit 37.3% 39.6% 37.5% 32.4%Any home care 12.8% 17.2% 18.6% 8.5%Any transportation 45.9% 61.1% 70.2% 32.2%Any pharmacy 88.0% 89.5% 85.6% 78.3%Any DME 18.7% 20.9% 20.5% 15.2%

Any comp case mgt 7.6% 10.8% 10.3% 5.2%Any community rehab 1.1% 1.3% 0.8% 0.8%

“MEDICAL HOME” OUTPATIENT CARE

[PRIMARY/SPECIALTY/OB]

• “Loyal” patients: 3+ visits with one provider having ≥ 50% of visits during the 2-year period

• “Shoppers”: 3+ visits with no provider having ≥ 50% of visits during the 2-year period

• “Occasional users”: Less than 3 visits during the 2-year period

• “No PC/Spec/OB” patients: No primary care, specialty care, or OB visits during the 2-year period

Looking back at two years of claims data, classify patients as:

“Medical Home” for Patients with Risk Score ≥50 Based on Prior 2-Years of Ambulatory Use


"Medical Home" Status AllNYS

Number ofPC/Spec/OB

ProvidersTouched

Loyal 48.9% 2.80 OPD/Satellite 25.1% 2.97 D&TC 15.0% 2.55 MD 8.8% 2.71Shopper 18.8% 5.39Occasional User 13.3% 1.18No PC/Spec/OB 19.0% 0.00

Total 100.0% 2.54




Number ofPC/Spec/OB

ProvidersTouched


Total 100.0% 2.54

51%




Number ofPC/Spec/OB

ProvidersTouched


Total 100.0% 2.54




Number ofPC/Spec/OB

ProvidersTouched


Total 100.0% 2.54

Number ofPC/Spec/OB

ProvidersTouched

% ofPatients

AllNYS

1 Provider 0.0%2 Providers 4.9%3 Providers 22.7%4-5 Providers 35.7%5-9 Providers 28.8%10+ Providers 8.0% Total 100.0%


Risk Score> 50

Risk Score> 75

Risk Score> 90

AllPatients

Costs Prior 12 MonthsInpatient 20,973 42,357 75,221 12,442Emergency Department 306 576 1,040 199Primary Care Visit 489 535 495 416Specialty Care Visit 80 83 75 71Psychiatric Care Visit 1,045 862 693 899Substance Abuse Visit 1,129 1,342 1,070 748Other Ambulatory 1,989 2,746 3,223 1,494Pharmacy 6,470 7,711 7,545 4,905Transportation 427 658 810 289Community Rehab 109 112 57 73Case Management 349 544 554 230Personal Care 853 914 755 754Home Care 875 1,201 1,357 601LTHHC 49 116 214 29All Other 2,388 3,500 3,738 1,738

Total Cost 37,530 63,259 96,848 24,885

Costs Next 12 MonthsInpatient 26,777 45,513 70,491 16,791Emergency Department 299 527 921 198Primary Care Visit 415 394 360 375Specialty Care Visit 52 44 34 55Psychiatric Care Visit 1,041 786 582 964Substance Abuse Visit 1,155 1,320 1,061 796Other Ambulatory 2,183 2,831 2,987 1,678Pharmacy 7,246 7,726 7,194 5,834Transportation 548 752 794 389Community Rehab 170 184 59 173Case Management 392 547 533 267Personal Care 1,017 1,023 795 918Home Care 1,229 1,327 1,392 986LTHHC 117 117 63 110All Other 3,895 5,071 5,409 3,089

Total Cost 46,537 68,162 92,674 32,622

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

$16,000

$18,000

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10%

11%

12%

13%

14%

15%

16%

17%

18%

19%

20%

21%

22%

23%

24%

25%

`

CASE FINDING ALGORITHM MAXIMUM EXPENDITURE/PATIENT FOR BREAK EVEN

% Reduction in Future Admissions

Inte

rven

tion

Cos

t/Pat

ient

Risk Score 50+

Risk Score 90+

Risk Score 75+

$2,799

$4,521

$6,630

$4,199

$6,783

$9,990

$5,599

$9,044

$13,320


Top 25 Principal Diagnosis of “Future Admissions”

ICD9Code ICD9 Decription # of

Adms%

TotalCumm

%

78039 CONVULSIONS NEC 538 0.7% 58.5%29534 PARAN SCHIZO-CHR/EXACERB 509 0.6% 59.1%5770 ACUTE PANCREATITIS 499 0.6% 59.7%V5811 ANTINEOPLASTIC CHEMO ENC 493 0.6% 60.3%30400 OPIOID DEPENDENCE-UNSPEC 438 0.5% 60.9%30420 COCAINE DEPEND-UNSPEC 412 0.5% 61.4%29680 BIPOLAR DISORDER NOS 402 0.5% 61.9%25002 DMII WO CMP UNCNTRLD 392 0.5% 62.3%30301 AC ALCOHOL INTOX-CONTIN 391 0.5% 62.8%29634 REC DEPR PSYCH-PSYCHOTIC 385 0.5% 63.3%40391 HYP KID NOS W CR KID V 362 0.4% 63.7%5849 ACUTE RENAL FAILURE NOS 361 0.4% 64.2%29690 EPISODIC MOOD DISORD NOS 358 0.4% 64.6%5990 URIN TRACT INFECTION NOS 353 0.4% 65.0%7802 SYNCOPE AND COLLAPSE 353 0.4% 65.5%4660 ACUTE BRONCHITIS 337 0.4% 65.9%30411 SED,HYP,ANXIOLYT DEP-CON 326 0.4% 66.3%5589 NONINF GASTROENTERIT NEC 323 0.4% 66.7%34590 EPILEP NOS W/O INTR EPIL 318 0.4% 67.0%30480 COMB DRUG DEP NEC-UNSPEC 312 0.4% 67.4%25013 DMI KETOACD UNCONTROLD 309 0.4% 67.8%29532 PARANOID SCHIZO-CHRONIC 299 0.4% 68.2%2910 DELIRIUM TREMENS 292 0.4% 68.5%29633 RECUR DEPR PSYCH-SEVERE 282 0.3% 68.9%25080 DMII OTH NT ST UNCNTRLD 281 0.3% 69.2%

CHARACTERISTICS OF INTERVIEWED BELLEVUE PATIENTS

Characteristic % ofTotal

Marrital statusMarried/living with partner 14%Separated 16%Divorced 10%Widowed 4%Never married 56%

Curently living alone 52%

No "close" frriends/relatives 16%Two or fewer "close" friends/relatives 48%

Low "Perceived Availablity of Support" 42%

Bellevue Hospital Center


Usual source of careNone 16%Emergency department 42%OPD/Clinic 20%Community based clinic 8%Private/Group MD/other 14%

58%




Current housing statusApartment/home rental 34%Public housing 2%Residential facility 2%Staying with family/friends 24%Shelter 8%Homeless 28%

Homeless anytime previous 2 years 50%

60%








THEN GET SOME SMART PEOPLE IN THE ROOM AND DESIGN THE INTERVENTION

SO WHAT’S IT GOING TO TAKE?


• Multi-disciplinary approach for individualized needs assessment and care planning for participating patients



• Integrated/organized/coordinated health care delivery system – Primary care – Specialty care – Substance abuse/mental health services – Inpatient care – Etc, etc



• Integrated/organized/coordinated health care delivery system and linkage to social service delivery system – Primary care – Specialty care – Substance abuse/mental health services – Inpatient care – Community based social support programs/resources – Supportive housing for many – Etc, etc, etc, etc



• Integrated/organized/coordinated health care delivery system and linkage to social service delivery system

• Some sort of care/service-coordinator/arranger – With a reasonable caseload size – With a clear mission (to improve health and to reduce costs)




• Some sort of care/service-coordinator/arranger

• Core IT and analytic support capacity to… – Track patient utilization in close to real time – Mine administrative data and target interventions/outreach – Provide analysis of utilization patterns

• Identify trends/problems to continuously re-design intervention strategies • Provide feed-back to providers on performance

– Hospital admission rates – ED visit rates – Adherence to evidence based practice standards

– Support effective use of electronic medical records where available





• Core IT and analytic support capacity

• Ability to provide real time support at critical junctures – ED visit - prevention of “social admissions” – Hospital discharge - effective community support/management planning – Patient initiated - help for an emerging crisis






• Ability to provide real time support at critical junctures

• Incentives/reimbursement policies to encourage and reward “effective and cost efficient care” – Hospitals must have a shared interest in avoiding admissions – Reimbursement rates for OP services need to be related to their costs – Costs of social support need to be recognized – [No new money – any new/augmented services offset by IP savings]






• Ability to provide real time support at critical junctures

• Incentives/reimbursement policies to encourage and reward “effective and cost efficient care”





• Core IT and analytic support capacity ???

• Ability to provide real time support at critical junctures ???

• Incentives/reimbursement policies to encourage and reward “effective and cost efficient care” ???


• NY State legislature authorized $20M demonstration • After a competitive procurement process that took 13

months to implement, awards for 7 pilots March, 2009 • NYC sites had a goal of 500 patients, non-NYC 250 • Program provides $250/month for care coordination and

a “shared savings pool” • July, 2009: One pilot dropped out • August, 2009: Enrollment began in 6 remaining pilots • October, 2009 – July, 2011: sites received quarterly

enrollment refreshments


• Federal government would not allow randomization of patients into the initiative – All patients in a geographic area must have access to same set of

services unless obtain a “waiver” (18 months and politically “fraught”)



services unless obtain a “waiver” (18 months and politically “fraught”) – Brilliant solution: randomize zip codes (and tell federal government that

being implement only some areas of the state)





• The $48 billion NY Medicaid agency (equivalent in revenue to #57 on Fortune 500 list of U.S. companies) had no funds authorized to conduct an evaluation

– Local philanthropy provided limited funding – But not enough to survey patients or contact control group





• The $48 billion NY Medicaid agency (equivalent in revenue to #57 on Fortune 500 list of U.S. companies) had no funds authorized to conduct an evaluation

– Local philanthropy provided limited funding – But not enough to survey patients or contact control group

• Sites had enormous difficulty locating patients for enrollment – Found and enrolled only 25% of eligible patients – State dropped risk score cut-off from 50 to 40 and finally to 30


• At least in part inspired by CIDP, US national health reform legislation (Affordable Care Act, aka “Obama Care”), authorized new “Medicaid Health Home Program”

• Federal government provides 90%-10% funding match for care coordination initiatives like CIDP for patients with multiple chronic conditions

• In New York, the 2,000+ CIDP program folded into the new Health Home initiative, with plans to ultimately expand to 700,000 patients

• Preliminary evaluation of CIDP expected sometime in next 12 months

WHAT NOT TO DO


WHAT TO DO



• Intervention implementation flaws – Roll it out in at least quasi-experimental mode – Track “dosage” levels (who does what to whom and how) – Avoid enrollment criteria “leakage” – Evaluate impact of the intervention as rigorously as possible

john billings: applying predictive risk approaches and models effectively

Health & Medicine