AHRQ Quality IndicatorsAHRQ Quality IndicatorsRecent validation effortsRecent validation efforts

Patrick S. Romano, MD MPHPatrick S. Romano, MD MPHUC Davis Center for Healthcare Policy and ResearchUC Davis Center for Healthcare Policy and Research

AHRQ QI Users MeetingAHRQ QI Users MeetingBethesda, MD; September 28, 2007Bethesda, MD; September 28, 2007

OverviewOverview

Topics (focus on PSIs and PDIs)Topics (focus on PSIs and PDIs)– Validation of prior toolsValidation of prior tools– Extending face/consensual validityExtending face/consensual validity– Construct/predictive validity based on patient Construct/predictive validity based on patient

outcomes and resource utilizationoutcomes and resource utilization– Criterion validity based on present-of-admission Criterion validity based on present-of-admission

(POA) data(POA) data– Criterion validity based on data linkagesCriterion validity based on data linkages– Criterion validity based on recoded/abstracted dataCriterion validity based on recoded/abstracted data– Criterion validity based on clinician case reviewCriterion validity based on clinician case review– Other approachesOther approaches

Questions and answersQuestions and answers

Validation of prior tools based on Validation of prior tools based on literature review (MEDLINE/EMBASE)literature review (MEDLINE/EMBASE)

Validation studies of Iezzoni et al.’s Complications Validation studies of Iezzoni et al.’s Complications Screening ProgramScreening Program– At least one of three validation studies (coders, nurses, At least one of three validation studies (coders, nurses,

or physicians) confirmed PPV or physicians) confirmed PPV ≥≥75% among flagged 75% among flagged casescases

– Nurse-identified process-of-care failures were more Nurse-identified process-of-care failures were more prevalent among flagged cases than among unflagged prevalent among flagged cases than among unflagged controlscontrols

Other studies of coding validityOther studies of coding validity– Very few in peer-reviewed journals, some in “gray Very few in peer-reviewed journals, some in “gray

literature”literature”

Validation (%) of Complications Screening ProgramValidation (%) of Complications Screening ProgramMed CareMed Care 2000;38:785-806,868-76; 2000;38:785-806,868-76; Int J Qual Health CareInt J Qual Health Care 1999;11:107-18 1999;11:107-18

CSP Indicator PSI Coder: % ComplicPresent

MD: %Complicpresent

RN: % Process problem

MD: %Quality

problem

Postprocedural hemorrhage/ hematoma

#9 narrower: requires proc code + dx

83 (surg)49 (med)

57 (surg)55 (med)

66 vs 4613 vs 5

37 vs 231 vs 2

Postoperative pulmonary compromise

#11 narrower:includes only resp failure

72 75 52 vs 46 20 vs 2

DVT/PE #12 surgical onlySlight changes

59 (surg)32 (med)

70 (surg)28 (med)

72 vs 4669 vs 5

50 vs 220 vs 2

In-hosp hip frx and falls

#8 surgical only, no E codes

57 (surg)11 (med)

71 (surg)11 (med)

76 vs 4654 vs 5

24 vs 25 vs 2

Percentage with process/quality problem among flagged cases vs. unflagged controls

Validation of prior tools:Validation of prior tools:Construct validity evidence in literatureConstruct validity evidence in literature

IndicatorIndicator Explicit Explicit processprocess

Implicit Implicit processprocess

StaffingStaffing

Complications of anesthesiaComplications of anesthesia

Death in low mortality DRGsDeath in low mortality DRGs ++

Decubitus ulcerDecubitus ulcer ±±

Failure to rescueFailure to rescue ++++

Foreign body left during procedureForeign body left during procedure

Iatrogenic pneumothoraxIatrogenic pneumothorax

Selected infections due to medical careSelected infections due to medical care

Postop hip fracturePostop hip fracture ++ ++

Postop hemorrhage or hematomaPostop hemorrhage or hematoma ±± ++

Postop physiologic/metabolic derangementsPostop physiologic/metabolic derangements ––

Postop respiratory failurePostop respiratory failure ±± ++ ±±

Postop thromboembolismPostop thromboembolism ++ ++ ±±

Postop sepsisPostop sepsis ––

Accidental puncture or lacerationAccidental puncture or laceration

Transfusion reactionTransfusion reaction

Postop abdominopelvic wound dehiscencePostop abdominopelvic wound dehiscence

Extending consensual/face validity:Extending consensual/face validity:OECD Health Care Quality Indicators ProjectOECD Health Care Quality Indicators Project

Includes 21 countries, WHO, European Commission, World Includes 21 countries, WHO, European Commission, World Bank, ISQua, etc.Bank, ISQua, etc.

Patient safety is one of five priority areasPatient safety is one of five priority areas Indicator selection criteria:Indicator selection criteria:

– ImportanceImportance Impact on healthImpact on health Policy importance (concern for policymakers and Policy importance (concern for policymakers and

consumers)consumers) Susceptible to influence by the health care systemSusceptible to influence by the health care system

– Scientific soundnessScientific soundness Face validity (clinical rationale and past usage)Face validity (clinical rationale and past usage) Content validityContent validity

– FeasibilityFeasibility Data availability and reporting burdenData availability and reporting burden

Extending consensual/face validity: Extending consensual/face validity: OECD Review ProcessOECD Review Process

Patient safety panel constituted with 5 members Patient safety panel constituted with 5 members (Dr. John Millar, Chair)(Dr. John Millar, Chair)

50 indicators from 7 sources submitted for review 50 indicators from 7 sources submitted for review (US, Canada, Australia)(US, Canada, Australia)

Modified RAND/UCLA Appropriateness MethodModified RAND/UCLA Appropriateness Method Panelists rated each indicator on importance and Panelists rated each indicator on importance and

scientific soundness (2 rounds with intervening scientific soundness (2 rounds with intervening discussion)discussion)

Retained indicators with median score >7 (1-9 Retained indicators with median score >7 (1-9 scale) on both domains; rejected indicators with scale) on both domains; rejected indicators with median score 5 or below on either domainmedian score 5 or below on either domain

International OECD panel ratings of PSIsInternational OECD panel ratings of PSIsMcLoughlin V, et al. McLoughlin V, et al. Int J Qual Health CareInt J Qual Health Care 2006 Sep;18 Suppl 1:14-20 2006 Sep;18 Suppl 1:14-20

PSIs recommended

PSIs not recommended

Experimental or rejected PSIs recommended

Selected infections due to medical care

Death in low mortality DRG Postop wound infection

Decubitus ulcer Postop hemorhage/ hematoma

In-hospital hip fracture or fall

Complications of anesthesia Iatrogenic pneumothorax Postop PE or DVT Postop abdominopelvic

wound dehiscence

Postop sepsis Failure to rescue Technical difficulty with procedure

Postop physiologic/ metabolic derangement

Transfusion reaction Postop respiratory failure Foreign body left in Postop hip fracture Birth trauma Obstetric trauma (all types)

AHRQ panel ratings of PSI “preventability”AHRQ panel ratings of PSI “preventability”very similar to OECD ratingsvery similar to OECD ratings

Acceptable Acceptable (-) Unclear Unclear (-) Decubitus ulcer Complications of

anesthesia Death in low mortality DRG

Failure to rescue

Foreign body left in Selected infections due to medical care

Postop hemorhage/ hematoma

Postop physioic/ metabolic derange

Iatrogenic pneumothoraxa

Postop PE or DVTb Postop respiratory failure

Postop hip fracturea Transfusion reaction Postop abdominopelvic wound dehiscence

Technical difficulty with procedure

Birth trauma Postop sepsis

Obstetric trauma (all delivery types)

a Panel ratings were based on definitions different than final definitions. For “Iatrogenic pneumothorax,” the rated denominator was restricted to patients receiving thoracentesis or central lines; the final definition expands the denominator to all patients (with same exclusions). For “In-hospital fracture” panelists rated the broader Experimental indicator, which was replaced in the Accepted set by “Postoperative hip fracture” due to operational concerns. b Vascular complications were rated as Unclear (-) by surgical panel; multispecialty panel rating is shown here.

Approaches to assessing Approaches to assessing construct validityconstruct validity

Is the outcome indicator associated with Is the outcome indicator associated with explicit explicit processes of careprocesses of care (e.g., appropriate use of (e.g., appropriate use of medications)? – YOUR STUDYmedications)? – YOUR STUDY

Is the outcome indicator associated with Is the outcome indicator associated with implicit implicit process of careprocess of care (e.g., global ratings of quality)? (e.g., global ratings of quality)?

Is the outcome indicator associated with nurse Is the outcome indicator associated with nurse staffing or skill mix, physician skill mix, or other staffing or skill mix, physician skill mix, or other quality-related aspects of quality-related aspects of hospital structurehospital structure??

Is the outcome indicator associated with other Is the outcome indicator associated with other meaningful outcomes of caremeaningful outcomes of care (predictive validity)? (predictive validity)?

Validation Using Implicit Processes of Care:Validation Using Implicit Processes of Care:Regression of PSIs on JCAHO Overall EvaluationRegression of PSIs on JCAHO Overall Evaluation

Miller MR, et al., Miller MR, et al., Am J Med QualAm J Med Qual 2005;20:239-252 2005;20:239-252

PSIPSI Regression Regression coefficientcoefficient

PSIPSI Regression Regression coefficientcoefficient

Complications of anesthesia 0.029 Postop sepsis -0.209

Death in low-mortality DRG 0.012 Postop wound dehiscence -0.098

Decubitus ulcer 0.004 Accidental puncture/laceration -0.212 (p<.01)

Failure to rescue 0.112 Birth trauma 0.045

Foreign body left in -0.102 Ob trauma, vaginal w/out instrum

-0.114

Iatrogenic pneumothorax 0.261 (p=.03) Ob trauma, vaginal w instrum 0.165 (p=.04)

Selected infection 2° to care -0.037 Ob trauma, cesarean -0.027

Postop hip fracture -0.112 PSI factor 1 -0.108 (p=.02)

Postop hemorrhage/hematoma

0.096 PSI factor 2 0.026

Postop respiratory failure -0.284 (p<.01) PSI factor 3 -0.010

Postop DVT/PE -0.210 (p=.06)

Validation Using Structural Measures:Validation Using Structural Measures:Regression of FTR on Skill Mix MeasuresRegression of FTR on Skill Mix Measures

Silber J, et al., Silber J, et al., Med CareMed Care 2007; 2007;45(10):918-92545(10):918-925

IndicatorIndicatorFTR-SilberFTR-Silber

FTR-FTR-NeedlemanNeedleman FTR-AHRQFTR-AHRQ

Teaching hospital (COTH member)Teaching hospital (COTH member) 0.890.89 0.860.86 0.850.85

Medium-large hospital (>200 beds)Medium-large hospital (>200 beds) 0.920.92 0.940.94 0.920.92

Bed-to-nurse (RN+LVN) ratioBed-to-nurse (RN+LVN) ratio 1.041.04 1.041.04 1.041.04

Nursing skill mixNursing skill mixRN/(RN+LPN) RN/(RN+LPN)

0.920.92 0.870.87 0.870.87

Odds ratios from multivariable logistic regression, adjusted for all patient characteristics and all other specified hospital characteristics, based on 1999-2000 Medicare inpatient claims.

Odds ratios further from 1 indicate larger, more clinically important effects.

Yellow: 0.0001<p<0.01

Red: p<0.0001Red: p<0.0001

Predictive validity: Impact of preventing each PSI Predictive validity: Impact of preventing each PSI event on mortality, LOS, charges (ROI)event on mortality, LOS, charges (ROI)

NIS 2000 analysis by Zhan & Miller, NIS 2000 analysis by Zhan & Miller, JAMAJAMA 2003;290:1868-74 2003;290:1868-74

IndicatorIndicator ΔΔ Mort (%) Mort (%) ΔΔ LOS (d) LOS (d) ΔΔ Charge Charge ($) ($)

Postoperative septicemiaPostoperative septicemia 21.921.9 10.910.9 $57,700$57,700

Selected infections due to medical careSelected infections due to medical care 4.34.3 9.69.6 38,70038,700

Postop abd/pelvic wound dehiscencePostop abd/pelvic wound dehiscence 9.69.6 9.49.4 40,30040,300

Postoperative respiratory failurePostoperative respiratory failure 21.821.8 9.19.1 53,50053,500

Postoperative physiologic or metabolic Postoperative physiologic or metabolic derangementderangement

19.819.8 8.98.9 54,80054,800

Postoperative thromboembolismPostoperative thromboembolism 6.66.6 5.45.4 21,70021,700

Postoperative hip fracturePostoperative hip fracture 4.54.5 5.25.2 13,40013,400

Iatrogenic pneumothoraxIatrogenic pneumothorax 7.07.0 4.44.4 17,30017,300

Decubitus ulcerDecubitus ulcer 7.27.2 4.04.0 10,80010,800

Postoperative hemorrhage/hematomaPostoperative hemorrhage/hematoma 3.03.0 3.93.9 21,40021,400

Accidental puncture or lacerationAccidental puncture or laceration 2.22.2 1.31.3 8,3008,300

Excess mortality, LOS, and charges computed from mean values for PSI cases and matched controls.

Predictive validity: Impact of preventing each PSI Predictive validity: Impact of preventing each PSI event on mortality, LOS, VA expenditures (ROI)event on mortality, LOS, VA expenditures (ROI)VA PTF 2001 analysis by Rivard et alVA PTF 2001 analysis by Rivard et al., Med Care Res Rev; in pressin press

IndicatorIndicator ΔΔ Mort (%) Mort (%) ΔΔ LOS (d) LOS (d) ΔΔ Cost Cost ($) ($)

Postoperative septicemiaPostoperative septicemia 30.230.2 18.818.8 $31,264$31,264

Selected infections due to medical careSelected infections due to medical care 2.72.7 9.59.5 13,81613,816

Postop abd/pelvic wound dehiscencePostop abd/pelvic wound dehiscence 11.711.7 11.711.7 18,90518,905

Postoperative respiratory failurePostoperative respiratory failure 24.224.2 8.68.6 39,74539,745

Postoperative physiologic or metabolic Postoperative physiologic or metabolic derangementderangement

Postoperative thromboembolismPostoperative thromboembolism 6.16.1 5.55.5 7,2057,205

Postoperative hip fracturePostoperative hip fracture

Iatrogenic pneumothoraxIatrogenic pneumothorax 2.72.7 3.93.9 5,6335,633

Decubitus ulcerDecubitus ulcer 6.86.8 5.25.2 6,7136,713

Postoperative hemorrhage/hematomaPostoperative hemorrhage/hematoma 5.15.1 3.93.9 7,8637,863

Accidental puncture or lacerationAccidental puncture or laceration 3.23.2 1.41.4 3,3593,359

Excess mortality, LOS, and charges computed from mean values for PSI cases and matched controls.

Predictive validity: Impact of preventing each PSI Predictive validity: Impact of preventing each PSI event on mortality, LOS, VA expenditures (ROI)event on mortality, LOS, VA expenditures (ROI)VA PTF 2001 analysis by Rivard et alVA PTF 2001 analysis by Rivard et al., Med Care Res Rev; in pressin press

IndicatorIndicator ΔΔ LOS (d) LOS (d) ΔΔ Cost Cost ($) ($)

Postoperative septicemiaPostoperative septicemia 5.75.7 $13,395$13,395

Selected infections due to medical careSelected infections due to medical care 4.54.5 7,2927,292

Postop abd/pelvic wound dehiscencePostop abd/pelvic wound dehiscence 8.38.3 17,28117,281

Postoperative respiratory failurePostoperative respiratory failure 4.54.5 9,6419,641

Postoperative physiologic or metabolic Postoperative physiologic or metabolic derangementderangement

Postoperative thromboembolismPostoperative thromboembolism 4.54.5 9,064 9,064

Postoperative hip fracturePostoperative hip fracture

Iatrogenic pneumothoraxIatrogenic pneumothorax 3.43.4 5,4765,476

Decubitus ulcerDecubitus ulcer 3.73.7 5,5525,552

Postoperative hemorrhage/hematomaPostoperative hemorrhage/hematoma 4.74.7 10,01210,012

Accidental puncture or lacerationAccidental puncture or laceration 3.13.1 6,8806,880

Excess mortality, LOS, and charges computed from GEE regression models (logged costs and LOS).

Predictive validity questionable based on NIS/VAPredictive validity questionable based on NIS/VA Zhan & Miller, Zhan & Miller, JAMAJAMA 2003;290:1868-74 2003;290:1868-74 Rosen et al., Rosen et al., Med CareMed Care 2005;43:873-84 2005;43:873-84

IndicatorIndicator ΔΔ Mort (%) Mort (%) ΔΔ LOS (d) LOS (d) ΔΔ Charge Charge ($) ($)

Birth traumaBirth trauma -0.1 (NS)-0.1 (NS) -0.1 (NS)-0.1 (NS) 300 (NS)300 (NS)

Obstetric trauma –cesareanObstetric trauma –cesarean -0.0 (NS)-0.0 (NS) 0.40.4 2,7002,700

Obstetric trauma - vaginal w/out Obstetric trauma - vaginal w/out instrumentationinstrumentation

0.0 (NS)0.0 (NS) 0.050.05 -100 (NS)-100 (NS)

Obstetric trauma - vaginal w Obstetric trauma - vaginal w instrumentationinstrumentation

0.0 (NS)0.0 (NS) 0.070.07 220220

Complications of anesthesia*Complications of anesthesia* 0.2 (NS)0.2 (NS) 0.2 (NS)0.2 (NS) 1,6001,600

Transfusion reaction*Transfusion reaction* -1.0 (NS)-1.0 (NS) 3.4 (NS)3.4 (NS) 18,900 (NS)18,900 (NS)

Foreign body left during procedureForeign body left during procedure†† 2.12.1 2.12.1 13,30013,300

* All differences NS for transfusion reaction and complications of anesthesia in VA/PTF.

† Mortality difference NS for foreign body in VA/PTF.

Criterion validity: POA coding in NY and CACriterion validity: POA coding in NY and CAHouchens, Elixhauser, Romano. Houchens, Elixhauser, Romano. Joint Comm J Qual Safety; in pressJoint Comm J Qual Safety; in press

PSI CA cases %not POA NY cases %not POA

PSI 1: Complications of Anesthesia 934 100 284 100

PSI 3: Decubitus Ulcer 17,789 11 16,425 14

PSI 5: Foreign Body Left During Procedure 258 64 169 76

PSI 6: Iatrogenic Pneumothorax 1,256 73 782 65

PSI 7: Infection Due To Medical Care 4,286 65 2,406 65

PSI 8: Postop Hip Fracture 106 21 69 26

PSI 9: Postop Hemorrhage or Hematoma 1,800 79 859 71

PSI 10: Postop Physiologic and Metabolic Derangement 686 77 228 64

PSI 11: Postop Respiratory Failure 2,374 94 1,312 93

PSI 12: Postop PE or DVT 6,715 46 5,318 43

PSI 13: Postop Sepsis 865 73 453 70

PSI 15: Accidental Puncture/Laceration 9,107 87 3,743 87

PSI 16: Transfusion Reaction 12 58 9 78

Criterion validity: POA coding at Mayo hospitalsCriterion validity: POA coding at Mayo hospitalsNaessens et al. Naessens et al. Med Care 2007;45:781-788 Med Care 2007;45:781-788 (ob/birth indicators excluded)(ob/birth indicators excluded)

PSI# cases %not POA

AHRQ denomin

Corrected denomin

PSI 1: Complications of Anesthesia 16 6 29,681 29,681

PSI 3: Decubitus Ulcer 285 18 18,698 18,772

PSI 5: Foreign Body Left 13 54 58,206 58,206

PSI 6: Iatrogenic Pneumothorax 63 78 47,809 49,605

PSI 7: Infection Due To Medical Care 137 60 40,019 40,288

PSI 8: Postop Hip Fracture 9 22 16,770 16,788

PSI 9: Postop Hemorrhage or Hematoma 143 87 28,998 28,998

PSI 10: Postop Physiologic and Metabolic Derangement 48 46 23,654 23,669

PSI 11: Postop Respiratory Failure 123 74 18,270 18,270

PSI 12: Postop PE or DVT 492 40 28,876 28,949

PSI 13: Postop Sepsis 63 76 6,349 6,467

PSI 15: Accidental Puncture/Laceration 891 85 55,840 55,840

PSI 14: Postop Wound Dehiscence 34 100 7,637 7,637

Impact of POA coding in a hospital Impact of POA coding in a hospital report card: postop hemorrhagereport card: postop hemorrhage

Impact of POA coding in a hospital Impact of POA coding in a hospital report card: decubitus ulcerreport card: decubitus ulcer

Criterion validity: NY data linkageCriterion validity: NY data linkageGallagher et al., Gallagher et al., AHRQ Advances in Patient SafetyAHRQ Advances in Patient Safety; Shufelt et al., ; Shufelt et al., Am J Med QualAm J Med Qual

2005;20:210-8; Weller et al., 2005;20:210-8; Weller et al., Joint Comm J Qual SafeJoint Comm J Qual Safe 2004;30:497-504 2004;30:497-504

Linking 30 day readmissions increased overall rate of PSIs:Linking 30 day readmissions increased overall rate of PSIs:– Selected Infections from 2.02 to 2.52 per 1,000 eligible discharges Selected Infections from 2.02 to 2.52 per 1,000 eligible discharges

(56% dialysis patients)(56% dialysis patients)– Postoperative DVT/PE from 9.3 to 11.3 per 1,000 (45% PE)Postoperative DVT/PE from 9.3 to 11.3 per 1,000 (45% PE)– Postoperative Hemorrhage/Hematoma from 1.86 to 2.05 per 1,000Postoperative Hemorrhage/Hematoma from 1.86 to 2.05 per 1,000

Relaxing the dx-procedure linking criterion increased the rate of Relaxing the dx-procedure linking criterion increased the rate of Postoperative Hemorrhage/Hematoma from 1.86 to 2.35 per 1,000Postoperative Hemorrhage/Hematoma from 1.86 to 2.35 per 1,000

Based on procedure codes for repair of iatrogenic injuries, the PSI for Based on procedure codes for repair of iatrogenic injuries, the PSI for Accidental Punctures and Lacerations missed:Accidental Punctures and Lacerations missed:– 27% of bladder injuries from hysterectomy27% of bladder injuries from hysterectomy– 21% of bowel injuries from cholecystectomy21% of bowel injuries from cholecystectomy– 47% of abdominal injuries from lysis of adhesions47% of abdominal injuries from lysis of adhesions– 54% of abdominal injuries from nephroureterectomy54% of abdominal injuries from nephroureterectomy– 20% of spinal injuries from lumbar surgery20% of spinal injuries from lumbar surgery

Criterion validity based on recoded data:Criterion validity based on recoded data: CA Obstetric Validation StudyCA Obstetric Validation Study

Organized to assess validity of various potential Organized to assess validity of various potential measures of adverse events after deliverymeasures of adverse events after delivery

Cases sampled from OHSPD Patient Discharge Cases sampled from OHSPD Patient Discharge Data Set (nonfederal acute care hospitals)Data Set (nonfederal acute care hospitals)

Linked delivery, antepartum, postpartum Linked delivery, antepartum, postpartum records using SSN and DOBrecords using SSN and DOB

Stratified random cluster sample of 1,662 Stratified random cluster sample of 1,662 records from 52 hospitals (30% primary records from 52 hospitals (30% primary cesarean, 19% repeat cesarean, 51% vaginal)cesarean, 19% repeat cesarean, 51% vaginal)

97.1% of records received and reviewed by 97.1% of records received and reviewed by “expert” coder and obstetric nurse abstractor“expert” coder and obstetric nurse abstractor

Criterion validity in CA hospital discharge dataCriterion validity in CA hospital discharge dataRomano PS, et al. Romano PS, et al. Obstet GynecolObstet Gynecol 2005;106(4):717-725 2005;106(4):717-725

IndicatorIndicator

SensitivitySensitivity Pos Pred ValuePos Pred Value

UnweightedUnweighted WeightedWeighted UnweightedUnweighted WeightedWeighted

FORMER AHRQ PSI: FORMER AHRQ PSI: Obstetric trauma, Obstetric trauma, CesareanCesarean

11%11% 5%5% 67%67% 94%94%

HealthGrades: HealthGrades: major comps, Vaginal major comps, Vaginal

67%67% 58%58% 91%91% 91%91%

HealthGrades: HealthGrades: major comps, Cesareanmajor comps, Cesarean

55%55% 47%47% 64%64% 79%79%

AHRQ/JCAHO: 3AHRQ/JCAHO: 3rdrd or 4 or 4thth degree lacerationdegree laceration

90%90% 93%93% 90%90% 73%73%Sensitivity = TP/(TP+FN) – are all the real cases captured?PPV = TP/(TP+FP) – are all the flagged cases real?Brubaker L, et al. Obstet Gynecol 2007;109(5):1141-5 reported sensitivity of 77%, specificity of 99.7%, based on a clinical research data set with 393 positive (3rd/4th degree tears) and 383 negative vaginal deliveries.

Criterion validity:Criterion validity:Linking VA PTF and NSQIP abstractionLinking VA PTF and NSQIP abstraction

NSQIP is a national project that collects and feeds back NSQIP is a national project that collects and feeds back data on surgical outcomes from 123 VA facilitiesdata on surgical outcomes from 123 VA facilities

Trained surgical clinical nurse reviewers collect Trained surgical clinical nurse reviewers collect preoperative, intraoperative, and postoperative data.preoperative, intraoperative, and postoperative data.

Patients are followed for 30 days after index procedurePatients are followed for 30 days after index procedure Sampling frame: veterans, FY 2001, mainland US acute Sampling frame: veterans, FY 2001, mainland US acute

care facilitiescare facilities VA’s Patient Treatment File (similar to HCUP) was linked VA’s Patient Treatment File (similar to HCUP) was linked

with NSQIP using SSN, dates of admission and discharge, with NSQIP using SSN, dates of admission and discharge, facility numberfacility number

Final data file included 55,752 hospitalizations, representing Final data file included 55,752 hospitalizations, representing 59,838 surgeries and 51,832 patients in 110 hospitals 59,838 surgeries and 51,832 patients in 110 hospitals

Criterion validity of PSIs linked to Criterion validity of PSIs linked to NSQIP in VA hospitals NSQIP in VA hospitals

Romano PS, et al. Romano PS, et al. HSRHSR forthcoming? forthcoming?

IndicatorSensitivity PPV Positive

likelihood ratioCurrent

InpatientBetter

InpatientCurrent

InpatientBetter

Inpatient

Postoperative sepsis 32% 37% 44% 45% 123 131Postoperative thromboembolism 56% 58% 22% 22% 65 64Postoperative respiratory failure 19% 67% 74% 66% 194 134Postop physiologic/ metabolic derangement 44% 48% 54% 63% 524 744Postop abdominopelvic wound dehiscence 29% 61% 72% 57% 160 79Sensitivity = TP/(TP+FN) – are all the real cases captured?PPV = TP/(TP+FP) – are all the flagged cases real?PLR = Sensitivity/(100-Specificity) – how many times more likely is the event?

NACHRI Pediatric Patient Safety NACHRI Pediatric Patient Safety Indicator (PSI) CollaborativeIndicator (PSI) Collaborative

Ran the AHRQ PSIs on NACHRI’s Case Mix database, containing Ran the AHRQ PSIs on NACHRI’s Case Mix database, containing 3 million discharges from approximately 70 children’s hospitals. 3 million discharges from approximately 70 children’s hospitals.

Developed the NACHRI Pediatric PSI Collaborative, a self-Developed the NACHRI Pediatric PSI Collaborative, a self-selected group of 20 hospitals interested in further studyselected group of 20 hospitals interested in further study

Developed and released a PSI Toolkit with sample press release, Developed and released a PSI Toolkit with sample press release, op ed, Q&A, and background documents for hospitals to educate op ed, Q&A, and background documents for hospitals to educate their communities on the relevance and utility of PSIs for their communities on the relevance and utility of PSIs for pediatrics.pediatrics.

Developed an online, secure chart review tool that allowed Developed an online, secure chart review tool that allowed Collaborative participants to review the preventability of patients Collaborative participants to review the preventability of patients flagged as having any of 11 selected PSI events.flagged as having any of 11 selected PSI events.

Fostered a relationship with AHRQ and Stanford/UC Davis to Fostered a relationship with AHRQ and Stanford/UC Davis to update each other on NACHRI’s findings and the PedQI update each other on NACHRI’s findings and the PedQI development work. development work.

NACHRI Pediatric Patient Safety NACHRI Pediatric Patient Safety Indicator (PSI) CollaborativeIndicator (PSI) Collaborative

Collaborative ParticipantsCollaborative Participants AL / Children’s Hospital of Alabama / Dr. Crayton Farguson*AL / Children’s Hospital of Alabama / Dr. Crayton Farguson* CA / Lucile Packard CH at Stanford / Dr. Paul Sharek*CA / Lucile Packard CH at Stanford / Dr. Paul Sharek* CA / UC-Davis / Dr. James Marcin**CA / UC-Davis / Dr. James Marcin** DC / Children’s National Medical Center / Dr. Tony Slonim*DC / Children’s National Medical Center / Dr. Tony Slonim* CA / Mattel Children’s at UCLA / Ms. Mary Kimball**CA / Mattel Children’s at UCLA / Ms. Mary Kimball** FL / All Children’s / Dr. Jack Hutto*FL / All Children’s / Dr. Jack Hutto* KY / Kosair Children’s Hospital / Dr. Ben Yandell*KY / Kosair Children’s Hospital / Dr. Ben Yandell* LA / Children’s Hospital New Orleans / Ms. Cindy Nuesslein*LA / Children’s Hospital New Orleans / Ms. Cindy Nuesslein* MD / Johns Hopkins Children’s Center / Dr. Marlene Miller*MD / Johns Hopkins Children’s Center / Dr. Marlene Miller* MA / Children’s Hospital Boston / Drs. Daniel Nigrin and Don GoldmannMA / Children’s Hospital Boston / Drs. Daniel Nigrin and Don Goldmann MI / C.S. Mott Children’s Hospital – U Mich / Dr. Aileen Sedman*MI / C.S. Mott Children’s Hospital – U Mich / Dr. Aileen Sedman* MO / Children’s Mercy Kansas City / Dr. Cathy Carroll*MO / Children’s Mercy Kansas City / Dr. Cathy Carroll* OH / The Children’s Medical Center Dayton / Dr. Thomas Murphy*OH / The Children’s Medical Center Dayton / Dr. Thomas Murphy* OH / Cincinnati Children’s Medical Center / Drs. Uma Kotagal, Joseph Luria*OH / Cincinnati Children’s Medical Center / Drs. Uma Kotagal, Joseph Luria* OH / Children’s Hospital Columbus / Dr. Thomas Hansen*OH / Children’s Hospital Columbus / Dr. Thomas Hansen* OH / Children’s Hospital MC of Akron / Dr. Michael BirdOH / Children’s Hospital MC of Akron / Dr. Michael Bird PA / Children’s Hospital of Philadelphia / Drs. James Stevens, Joel PortnoyPA / Children’s Hospital of Philadelphia / Drs. James Stevens, Joel Portnoy TX / Texas Children’s Hospital / Dr. Joan Shook*TX / Texas Children’s Hospital / Dr. Joan Shook* TX / Children’s Medical Center of Dallas / Dr. Fiona Levy, Ms. Kathy Lauwers*TX / Children’s Medical Center of Dallas / Dr. Fiona Levy, Ms. Kathy Lauwers* WI / Children’s Hospital of Wisconsin / Dr. Matthew Scanlon*WI / Children’s Hospital of Wisconsin / Dr. Matthew Scanlon*

Criterion validity based on clinician review:Criterion validity based on clinician review:AHRQ PSIs in Children’s HospitalsAHRQ PSIs in Children’s Hospitals

Sedman A, et al. Sedman A, et al. PediatricsPediatrics 2005;115(1):135-145 2005;115(1):135-145

PSIPSI No. reviewedNo. reviewed(total events)(total events)

PreventablePreventable(PPV %)(PPV %)

NonpreventableNonpreventable UnclearUnclear

Complications of anesthesia 74 (503) 11 (15%) 37 25

Death in low-mortality DRG 121 (1282) 16 (13%) 89 16

Decubitus ulcer 130 (2300) 71 (55%) 47 10

Failure to rescue 187 (5271) 15 (8%) 148 11

Foreign body left in 49 (235) 25 (51%) 14 10

Postop hemorrhage or hematoma 114 (1571) 40 (35%) 51 23

Iatrogenic pneumothorax 114 (1113) 51 (45%) 42 21

Selected infection 2° to med care 152 (7291) 63 (41%) 45 39

Postop DVT/PE 126 (1956) 36 (29%) 61 29

Postop wound dehiscence 41 (232) 19 (46%) 16 6

Accidental puncture or laceration 133 (4020) 86 (65%) 19 26

Key findings from NACHRI’s Key findings from NACHRI’s PSI physician case reviewsPSI physician case reviews

“…“…while 40% to 50% may seem low for positive predictive while 40% to 50% may seem low for positive predictive value, in terms of real patients, this means that 4 or 5 value, in terms of real patients, this means that 4 or 5 out of 10 children had a preventable event for this out of 10 children had a preventable event for this indicator. This is worth looking at and the things we indicator. This is worth looking at and the things we are finding in some instances, will allow for immediate are finding in some instances, will allow for immediate changes that may impact outcomes for future patients.” changes that may impact outcomes for future patients.” [Collaborative physician reviewer][Collaborative physician reviewer]

Examples from NACHRI’s PSI Examples from NACHRI’s PSI physician case reviewsphysician case reviews

During removal of non functioning port cath the end of the catheter During removal of non functioning port cath the end of the catheter was noted to be "irregular and not smoooth cut". It appeared the tip was noted to be "irregular and not smoooth cut". It appeared the tip had been embolized for an unknown duration… had been embolized for an unknown duration…

During replacement of pacemaker lead, a fragment of the lead broke During replacement of pacemaker lead, a fragment of the lead broke off, embolized and ended up lodged (puncture) in the anterolateral off, embolized and ended up lodged (puncture) in the anterolateral papillary muscle.papillary muscle.

No notation in original operative note or nursing record that No notation in original operative note or nursing record that sponge/needle counts were done and correct.sponge/needle counts were done and correct.

Count was reported as correct. Sponge discovered on xray due to Count was reported as correct. Sponge discovered on xray due to complaints of abdominal pain by patient.complaints of abdominal pain by patient.

Child with bone tumor who had mandible removed with subsequent Child with bone tumor who had mandible removed with subsequent bone graft and much packing in wound. This was supposedly bone graft and much packing in wound. This was supposedly removed before extubation, but at the time of extubation a remaining removed before extubation, but at the time of extubation a remaining pack blocked her airway causing reintubation with pack removal.pack blocked her airway causing reintubation with pack removal.

Romano’s ConclusionsRomano’s Conclusions

Several studies addressing PSI/PDI validity have been published, and Several studies addressing PSI/PDI validity have been published, and several more are on the way.several more are on the way.

Most PSIs have domestic and international consensual (face) validity.Most PSIs have domestic and international consensual (face) validity. Most PSIs have strong evidence of predictive (construct) validity in both VA Most PSIs have strong evidence of predictive (construct) validity in both VA

and non-VA data.and non-VA data. 3 PSIs have significant “POA problems”: postoperative DVT/PE, 3 PSIs have significant “POA problems”: postoperative DVT/PE,

postoperative hip fracture, decubitus ulcer.postoperative hip fracture, decubitus ulcer. Linked readmissions data may be helpful in ruling in/out early discharge as Linked readmissions data may be helpful in ruling in/out early discharge as

a cause of low PSI rates.a cause of low PSI rates. ““Complications of Anesthesia” may be problematic.Complications of Anesthesia” may be problematic. Coding validity looks strong for obstetric PSIs, and mixed for postoperative Coding validity looks strong for obstetric PSIs, and mixed for postoperative

PSIs, but very limited data.PSIs, but very limited data. Case review suggests 33-67% of most PSIs are potentially preventable (in Case review suggests 33-67% of most PSIs are potentially preventable (in

children), except lower for death-based PSIs, DVT/PE, and complications of children), except lower for death-based PSIs, DVT/PE, and complications of anesthesia.anesthesia.

Questions?Questions?