The Development of Quality Indicators for High Acuity Pediatric Conditions: Challenges in the Translation of

Knowledge into Performance Measurement

Antonia Stang MDCM MBA MSc

Assistant Professor

University of Calgary

Departments of Pediatrics and Community Health Sciences

Disclosure

• I do not have an affiliation (financial or otherwise) with any commercial organization that may have a direct or indirect connection to the content of my presentation.

“If we want healthy citizens-as opposed to citizens who have ready access to sickness care-we need a profound philosophical shift in what we should expect from medical professionals. We need to reward and incent quality, not quantity”

Andre Picard, Globe and Mail, March 20, 2012

Background

Quality of Care: “the degree to which health services for individuals increases the likelihood of desired health outcomes and are consistent with current professional knowledge”

(Institute of Medicine)

Background

• Quality Indicators: explicitly defined and measurable items pertaining to the structures, processes or outcomes of care– Structures: staff, equipment, physical layout of the

department, laboratory and diagnostic imaging resources

– Processes: interactions between professionals and patients

– Outcomes: mortality, morbidity, patient satisfaction, quality of life

Quality Indicator Uses

• Improve health care and outcomes• Benchmark performance• Set minimum standards of care• Improve efficiency• Accountability• Transparency• Research• Pay-for-Performance

What Makes a Good Measure?

• Impact, Opportunity, Evidence—Important to Measure and Report

• Reliability and Validity—Scientific Acceptability of Measure Properties

• Usability• Feasibility

National Quality Forum Measure Evaluation Criteria http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx

Objectives

• To review methods for involving stakeholders in the indicator development process.

• To discuss the application of GRADE (the Grading of Recommendations Assessment, Development and Evaluation) in indicator development and selection.

• To describe the challenges in developing and testing broadly applicable performance measures for high impact, relatively low frequency, conditions.

Study Objective

• to use a systematic process involving multiple stakeholders to develop evidence based quality of care indicators for pediatric conditions requiring high acuity ED care.

Research Team

• Antonia Stang MD MBA MSC, Principal Investigator• Astrid Guttmann MD MSc, Co-Investigator• David Johnson MD, Co-Investigator• Sharon Straus MD MSc, Co-Investigator• Lisa Hartling MSc PhD, Collaborator/methodological

expert• Francois Belanger MD, Collaborator/decision maker• Angelo Mikrogianakis MD, Collaborator/decision maker • Jen Crotts RN, Research Assistant• Janie Williamson RN, Pediatric Emergency Research

Team Coordinator

Funding

• Funded by an operating grant from the Canadian Institutes of Health Research (CIHR)

Rationale

• Lack of research on quality indicators specific to the pediatric population.

• Quality measures that are part of pediatric emergency medicine practice have not been systematically validated.

• Performance measures specific to pediatrics and pediatric emergency medicine have been identified as a research priority.

Phase 4: Data Collection

Phase 3: Expert Panel Process

17 new indicators 114 considered 62 selectedPhase 2: Systematic Review

47 existing indicators 51 guideline and evidence based

Phase 1: Condition Selection

6 conditions selected for indicator development

Stakeholder Involvement

• Goals:–to represent different stakeholder

perspectives

–to incorporate scientific evidence and expert opinion

Stakeholder Involvement

• Systematic methods to combine expert opinion and medical evidence– Consensus development conferences– Guideline based– Delphi technique– Nominal group technique– RAND/UCLA appropriateness method

Phase 1 Condition Selection

• 32 Member advisory panel

• Data on the main diagnosis for high acuity (resuscitation and emergent at triage) pediatric patients (age 0-19 yrs) seen in all EDs in Ontario and Alberta.

• Criteria for Condition Selection; – importance (morbidity or mortality) – impact (potential to address gap between current and

best practice)– validity (adequacy of scientific evidence linking

performance of care to patient outcome)

Phase 1: Condition Selection

Table 1: High Acuity (Resuscitation and Emergent ) Pediatric visits in 2006/2007 and 2007/2008 for all EDs in Ontario and Alberta for Selected Conditions

Condition Number of ED VisitsDiabetic ketoacidosis 1138Status asthmaticus 489Anaphylaxis 1334Status epilepticus 439Severe head injury 941Sepsis/septic shock 240

Phase 2: Indicator Development

• Systematic Review of the Literature for each condition– Existing Indicators– High quality national and international guidelines

(AGREE), Systematic Reviews (AMSTAR), Randomized Controlled Trials

• Criteria for Indicator Development– High quality evidence linking care structure or process

to patient outcome– Strongly recommended– Consistency across guidelines

Literature Review

Condition Search Results Full Text Reviewed

Articles Guidelines

Diabetic ketoacidosis 2480 374 3 3

Status asthmaticus 4564 172 8 5

Anaphylaxis 5889 276 4 3

Status epilepticus 870 28 3 5

Severe head injury 4789 95 5 5

Sepsis/septic shock 3866 78 3 2

GRADE

Grading of Recommendations Assessment, Development and Evaluation

1-Very low quality: Any estimate of effect is very uncertain 2-Low quality: Further research is very likely to have an

important impact on our confidence in the estimate of effect and is likely to change the estimate

3-Moderate Quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

4-High Quality: Further research is very unlikely to change our confidence in the estimate of effect

02468

1012141618

Num

ber o

f Ind

icat

ors

Condition

GRADE Rating by Condition

1

2

3

4

GRADE

GRADE Inter-rater Reliability

*Cohen’s Kappa unweighted

Condition Kappa* AgreementDiabetic Ketoacidosis 0.74 0.83Status Asthmaticus 0.73 0.81Anaphylaxis 1.00 1.00Status Epilepticus 0.15 0.50Severe Head Injury 0.66 0.85Severe Sepsis/Septic Shock 0.55 0.73General Measures 1.00 1.00Overall 0.68 0.78

Challenges using GRADE

• Subjective• Lack of concordance between guidelines• Time consuming• Requires substantial knowledge of clinical

condition and research methodology• Variable inter-rater reliability

Phase 3: Indicator Selection

Expert Panel Process: 2 rounds of a web-based survey and a face-to-face meeting

Indicators were selected based on two criteria rated on a scale of 1(strongly agree) to 9 (strongly disagree):– Relevance – Impact

Indicators rated ≥7 on both criteria by 70% of panelists were retained

Relevance

Impact

Disagree Disagree Disagree Disagree Neutral Agree Agree Agree Agreestrongly moderately somewhat somewhat moderately strongly

1 2 3 4 5 6 7 8 9

Indicator Type Source GRADE Numerator Denominator

% of patients with anaphylaxis with documentation of specialist referral

P 25, 26, 28-31

2 Number or patients with anaphylaxis with documentation of specialist referral including primary MD follow-up for referral, or documentation of existing specialist relationship

Total number of patients with anaphylaxis (based on ICD-10 codes)

Disagree Disagree Disagree Disagree Neutral Agree Agree Agree Agreestrongly moderately somewhat somewhat moderately strongly

1 2 3 4 5 6 7 8 9

0

10

20

30

40

50

60

Structure Process Outcome

Type of Indicator

Challenges in Indicator Development

• Lack of high quality evidence on the link between treatment/processes and outcomes, particularly in the pediatric setting

• Difficulty in identifying performance measures applicable to all settings

"The only man I know who behaves sensibly is my tailor; he takes my measurements anew each time he sees me. The rest go on with their old measurements and expect me to fit them."

George Bernard Shaw

Phase 4:Data Collection

*Based on ICD-10 code

Condition Number of ED Visits*Diabetic ketoacidosis 112Status asthmaticus/severe asthma 852Anaphylaxis 269Status epilepticus 133Severe head injury 108Sepsis/septic shock 79

Diabetic Ketoacidosis

*Includes only visits with fluids or insulin started in study ED

Indicator Results GRADE Kappa% of Emergency Departments (EDs) with guidelines N/A (single site) 1

% of ED visits with:Intravenous (IV) fluids within 60 minutes of ED arrival* 36 (17/47) 1 1.00Isotonic solution as initial IV fluid* 72 (34/47) 2 1.00IV insulin given 88 (57/65) 4 1.00Appropriate initial insulin dose and route* 88 (52/59) 4 1.00Potassium replacement 91 (59/65) 4 1.00Bicarbonate given 0 (0/61) 3 N/C

Time (median minutes with interquartile range) from:Triage to initiation of IV fluids* 78 (45,114) 1Arrival to insulin 115 (60,148) 1Arrival to expert consultation 161 (130,201) 1

Anaphylaxis

* for food and insect sting induced reactions

Indicator Results GRADE Kappa% of EDs with:

Clinical guidelines for the treatment of anaphylaxis N/A (single site) 2% of patient visits with:

Epinephrine given in ED (or in 3 hours prior to ED visit) 68 (144/211) 3 0.89Epinephrine given in ED by the appropriate route 94 (77/82) 3 0.68Documentation of epinephrine auto-injector at discharge 85 (180/211) 2 0.02Documentation of discharge instructions to avoid offending allergen* 17 (29/173) 2 0.41Documentation of instruction for epinephrine self-administration 38 (81/211) 2 0.39Documentation of specialist referral 56 (119/211) 2 1.00

Feasibility and Reliability

*Number of ED visits based on ICD-10 code

Condition ED Visits* Missing Chart/Visit Meets Criteria KappaDiabetic ketoacidosis 112 34 65/78 (83%) 1.00Anaphylaxis 269 19 211/250 (84%) 1.00Status epilepticus 133 6 85/127 (67%) 0.67

Challenges in Indicator Testing

• Feasibility and Reliability– Retrospective– Proper diagnosis is in itself a quality

issue– Accuracy of ICD -10 codes – Cost/effort of data collection

• Small and variable sample size

Lessons Learned

• Allow ample time for systematic review and evidence grading

• Composition of expert panel is key• Need an experienced moderator • Formal qualitative analysis of expert panel

meeting• Include patient/care-giver perspective

Next Step

• Multicentre data collection on select high acuity indicators combined with existing pediatric and emergency department performance measures– Reliability and feasibility testing– Process to outcome link

Campbell 2003

Quality Improvement and Indicator Development

Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academy Press. 2001: Washington, DC.

Campbell, S.M., et al., Research methods used in developing and applying quality indicators in primary care. BMJ, 2003. 326(7393): p. 816-9.

Chassin MR et al. Accountability Measures-Using Measurement to Promote Quality Improvement. N Eng J Med 363:7.

Donabedian A. The quality of care. How can it be assessed? JAMA. Sep 23-30 1988;260(12):1743-1748.

http://www.qualityforum.org/docs/measure_evaluation_criteria.aspxCenter for Health Policy/Center for Primary Care and Outcomes

Research & Battelle Memorial Institute. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement (Prepared by Battelle, under Contract No. 290-04-0020). Rockville, MD: Agency for Healthcare Research and Quality. May 2011.

Pediatric and Emergency Indicators

Alessandrini E, Gorelick MH, Shaw K, Kennebeck S. Using Performance Measures to Drive Improvement in Pediatric Emergency Care 2010; http://webcast.hrsa.gov/postevents/archivedWebcastDetail.asp?aeid=534

Bardach NS, Chien AT, Dudley A. Small Numbers Limit the Use of the Inpatient Pediatric Quality Indicators for Hospital Comparison. Academic Pediatrics, 2010. 10(4).

Guttmann A, Razzaq A, Lindsay P, Zagorski B, Anderson GM. Development of measures of the quality of emergency department care for children using a structured panel process. Pediatrics, 2006. 118(1): p. 114-23.

Schull MJ, Guttmann A, Leaver CA, Vermeulen M, Hatcher CM, Rowe BH, Zwarenstein M, Anderson GM. Prioritizing performance measurement for emergency department care: consensus on evidence based quality of care indicators. CJEM 2011. 13(3):300-309.

Rating the Quality of Evidence• http://www.gradeworkinggroup.org/index.htm (accessed February 23, 2012)• Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE

Working Group. Rating quality of evidence and strength of recommendations: What is "quality of evidence" and why is it important to clinicians? BMJ. 2008 May 3;336(7651):995-8.

• Guyatt GH, Oxman AD, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ, for the GRADE Working Group. Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-926.

• Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW Jr, Kunz R, Craig J, Montori VM, Bossuyt P, Guyatt GH; GRADE Working Group. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008 May 17;336(7653):1106-10.

• Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009AGREE Next Steps Consortium (2009).

• The AGREE II Instrument [Electronic version]. Retrieved March 23 2012, from http://www.agreetrust.org .