performance improvement projects technical assistance – pip 101 monday, june 18, 2007 1:30 p.m....

Performance Improvement Projects Technical Assistance –

PIP 101

Monday, June 18, 20071:30 p.m. – 3:00 p.m.

David Mabb, MS, CHCA

Sr. Director of Statistical Evaluation

Presentation Outline

• Balance Budget Act (BBA) 1997• Who is HSAG?• Overview of the PIP process• PIP Summary Form Review

– MCO demographics – CMS rationale– HSAG evaluation elements

• PIP Scoring Methodology• HSAG Contact Information• Questions and Answers

Balanced Budget Act (BBA) of 1997

The BBA requires states with Medicaid managed care programs to implement certain standards and business practices pertaining to:

– Enrollee Rights and Responsibilities– Quality Assessment and Performance

Improvement– Grievance and Appeals System

Who is HSAG?

• We are an External Quality Review Organization.

• To date, we have validated over 300 PIP studies.

• We validate Managed Care Organization and Behavioral Health Organization PIPs.

Overview of PIPs

What is the purpose of a PIP?• To assess and improve processes, and

subsequently, outcomes of care. It typically consists of a baseline, intervention period(s), and remeasurement (s).

What is a PIP?• It is a quality improvement project.

Primary Objective of a PIP

• Measurement of performance using objective quality indicators.

• Implementation of system interventions to achieve improvement in quality.

• Evaluation of the effectiveness of the interventions.

• Planning and initiation of activities for increasing or sustaining improvement.

PIP Validation

Ensures that:• PIPs are designed, implemented, and

reported in a methodologically sound manner.• Real improvement in the quality of care can

be achieved. • Documentation is compliant with CMS

protocols for conducting PIPs.• Stakeholders can have confidence in the

reported improvements.

Overview of PIPs (cont.)

The PIP process provides an opportunity to:

– Identify and measure a targeted area (clinical or nonclinical)

– Analyze the results– Implement

interventions for improvement

Overview of PIPs (cont.)

HSAG’s role:

• Validates PIPs using CMS’ protocol, Validating Performance Improvement Projects, A protocol for use in Conducting Medicaid External Quality Review Activities, Final Protocol, Version 1.0.

• PIP Validation is a desk audit evaluation• HSAG validates the study’s findings on the likely

validity and reliability of the results• Provides PIP Validation Reports to AHCA and the

MCOs• Identify best practices

PIP Review Process

• The PIP team includes clinicians and statisticians.

• PIP review team members assigned to the PIP study reads it in its entirety.

• Each PIP study has one clinician and one statistician assigned to reading the study. They read the study independently.

• If discrepancies, the PIP reviewers meets to reconcile any scoring differences.

PIP Review Process (cont.)

• Once scored, the PIP Validation Reports are sent to AHCA.

• The MCOs have an opportunity to comment on any miscalculations or errors noted in the PIP Validation Report.

• The final PIP Validation Reports are then released to the MCOs.

PIP Summary Form Review

• Health plan demographics (first page of the submission form)

• Discuss the 10 PIP Activities– CMS Rationale– HSAG evaluation elements

A. Activity One: Choose the Selected Study Topic

CMS Rationale• Impacts a significant portion of the members.

• Reflects Medicaid enrollment in terms of demographic characteristics, prevalence of disease, and the potential consequences (risks) of the disease.


CMS Rationale• Addresses the need for a specific service. • Goal should be to improve processes and

outcomes of health care. • The study topic may be specified by the State

Medicaid agency or on the basis of Medicaid enrollee input.


HSAG Evaluation Elements• Reflects high-volume or high-risk conditions

(or was selected by the State).• Is selected following collection and analysis

of data (or was selected by the State).• Addresses a broad spectrum of care and

services (or was selected by the State).


HSAG Evaluation Elements (cont.)• Includes all Medicaid eligible populations that

meet the study criteria.• Includes members with special health care

needs.• Has the potential to affect member health,

functional status, or satisfaction.


Example Study Topics: • Cervical Cancer Screening• HbA1c testing• Flu Vaccinations• Timeliness of Case Management• Discharge Planning• Readmission to Inpatient Psychiatric Care

within 30, 90, and 120 days

B. Activity Two: The Study Question

CMS Rationale • Stating the question(s) helps maintain the

focus of the PIP and sets the framework for data collection, analysis, and interpretation.

B. Activity Two: The Study Question

HSAG Evaluation Elements• States the problem to be studied in simple

terms.• Is answerable.

In general, the question should illustrate the point of: Does doing X result in Y?

Example: Will increased planning and attention to the importance of follow-up after inpatient discharge improve the rate of members receiving follow-up services?

C. Activity Three: Selected Study Indicators

CMS Rationale• Quantitative or qualitative characteristic.• Discrete event (member has or has not had

XX).• Appropriate for the study topic.• Objective, clearly and unambiguously

defined.


HSAG Evaluation Elements

The study indicator(s):• Is well defined, objective, and measurable.• Is based on practice guidelines, with sources

identified.


HSAG Evaluation Elements (cont.)The study indicator(s):• Allows for the study question to be answered.• Measures changes (outcomes) in health or

functional status, member satisfaction, or valid process alternatives.


HSAG Evaluation Elements (cont.)

The study indicator(s):• Has available data that can be collected on

each indicator.• Is a nationally recognized measure such as

HEDIS®, when appropriate.• Includes the basis on which each indicator

was adopted, if internally developed.

HEDIS® is a registered trademark of the National Committee for Quality Assurance (NCQA).

D. Activity Four: Identified Study Population

CMS Rationale• Represents the entire Medicaid eligible

enrolled population.• Allows systemwide measurement.• Implements improvement efforts to which the

study indicators apply.

HSAG Evaluation ElementsThe method for identifying the eligible population: • Is accurately and completely defined.• Includes requirements for the length of a member’s enrollment in the managed care plan.• Captures all members to whom the study question applies.


Example of Study Population:All Medicaid children with at least 11 months (12 months with one 30-day gap of enrollment) of continuous enrollment in the health plan, who were born on or between January 1, 2001, and December 31, 2003.


CMS Rationale• Sample size impacts the level of statistical confidence in the study.

-Statistical confidence is a numerical statement of the probable degree of certainty or accuracy of an estimate.

• Reflects improvement efforts to which the study indicators apply.• Reflects the entire population or a sample of that population.

E. Activity Five: Valid Sampling Techniques

HSAG Evaluation Elements• Consider and specify the true or estimated frequency of occurrence • Identify the sample size • Specify the confidence level to be used


HSAG Evaluation Elements (cont.)• Specify the acceptable margin of error. • Ensure a representative sample of the eligible population.• Ensure that the sampling techniques are in accordance with generally accepted principles of research design and statistical analysis.


CMS Rationale• Administrative data collection.

• Manual data collection.

• Inter-rater reliability.

• Frequency of collection and analysis cycle.

F. Activity Six: Data Collection Procedures, Data Collection Cycle, and Data Analysis

HSAG Evaluation ElementsThe data collection techniques:• Provide clearly defined data elements to be collected.• Clearly specify sources of data.• Provide for a clearly defined and systematic process for collecting data that includes how baseline and remeasurement data will be collected.



The data collection techniques• Provide for a timeline for the collection of baseline and remeasurement data.• Provide for qualified staff and personnel to collect manual data.



The manual data collection tool:• Ensures consistent and accurate collection of data according to indicator specifications.• Supports inter-rater reliability.• Has clear and concise written instructions for completion.




• An overview of the study in the written manual data collection tool instructions.• Administrative data collection algorithms that show steps in the production of indicators.• An estimated degree of automated data completeness (important if using the administrative method).

CMS Rationale• An intervention designed to change behavior at all levels of the care delivery system, including the members.• Changing performance, according to predefined quality indicators.• Appropriate interventions.• Likelihood of effecting measurable change.

G. Activity Seven: Improvement Strategies

HSAG Evaluation ElementsPlanned/implemented strategies for improvement are:• Related to causes/barriers identified through data analysis and Quality Improvement (QI) processes.• System changes that are likely to induce permanent change.• Revised if original interventions are not successful.• Standardized and monitored if interventions are successful.


HSAG Evaluation Elements (cont.)Planned/implemented strategies for improvement:• May be at the health plan, provider, or member level• Should be realistic, feasible, and clearly defined• Need a reasonable amount of time to be effective


CMS Rationale• Initiated using statistical analysis techniques.• Included an interpretation of the extent to which the study was successful.

H. Activity Eight: Data Analysis and Interpretation of Study Results

HSAG Evaluation ElementsThe data analysis:• Is conducted according to the data analysis plan in the study design.• Allows for generalization of the results to the study population if a sample was selected.• Identifies factors that threaten internal or external validity of findings.• Includes an interpretation of findings.


HSAG Evaluation Elements (cont.)The data analysis:• Is presented in a way that provides accurate, clear, and easily understood information.• Identifies initial measurement and remeasurement of study indicators.• Identifies statistical differences between initial measurement and remeasurement.• Identifies factors that affect the ability to compare initial measurement with remeasurement.• Includes the extent to which the study was successful.


CMS Rationale• Probability that improvement is true improvement.• Included an interpretation of the extent to which any changes in performance is statistically significant.

I. Activity Nine: Study Results and Summary Improvement

I. Activity Nine: Study Results and Summary Improvement

HSAG Evaluation Elements• The remeasurement methodology is the

same as the baseline methodology.• There is documented improvement in

processes or outcomes of care.• The improvement appears to be the result of

intervention(s).• There is statistical evidence that observed

improvement is true improvement.

J. Activity Ten: Sustained Improvement

CMS Rationale• Change results from modifications in the

processes of health care delivery.• If real change has occurred, the project

should be able to achieve sustained improvement.

HSAG Evaluation Elements• Repeated measurements over comparable time periods demonstrate sustained improvement, or that a decline in improvement is not statistically significant.

J. Activity Ten: Sustained Improvement

PIP Scoring Methodology

HSAG Evaluation Tool• 13 Critical Elements• 53 Evaluation Elements

(including the Critical Elements)


Overall PIP Score

Percentage Score for all Evaluation Elements: Calculated by dividing the total Met (includes critical elements) by the sum of the total Met, Partially Met,

and Not Met.Percentage Score for Critical Elements: Calculated by dividing the total critical elements Met by

the sum of the critical elements Met, Partially Met, and Not Met.

Validation Status: Met, Partially Met, Not Met


Met

(1) All critical elements were Met, and

(2) 80%–100% of all elements were Met

across all activities.


Partially Met(1) All critical elements were Met,

and 60% to 79% of all elements were Met across all activities;

or

(2) One or more critical element(s) were Partially Met.


Not Met

(1) All critical elements were Met and <60% of all elements were Met across all activities;

or

(1) One or more critical element(s) were Not Met.


Not Applicable (NA)

NA elements (including critical elements) were removed from all scoring.

Not Assessed

Not Assessed elements (including critical elements) were removed from all scoring.


Example 1 • Met = 43, Partially Met = 2, Not Met = 0, NA

= 8, and all critical elements were Met. • The MCO receives an overall Met status,

indicating the PIP is valid. • The score for the MCO is calculated as

43/45 = 95.6 percent.

PIP Tips

1. Complete the demographic page before submission.2. Notify HSAG when the PIP documents are uploaded to the secure ftp site and state the number of documents uploaded. 3. Label ALL attachments and reference them

in the body of the PIP study.4. HSAG does not require personal health

information to be submitted. Submit only aggregate results.

5. Document, document, and document!!6. Look for the CMS Protocols on myfloridaeqro.com. If you

have additional questions, contact HSAG.

HSAG Contacts

For questions contact:• Cheryl Neel

– [email protected]– 602.745.6201

• Denise Driscoll– [email protected]– 602.745.6260

mailto:[email protected]

mailto:[email protected]

Questions and Answers

performance improvement projects technical assistance – pip 101 monday, june 18, 2007 1:30 p.m....

Documents

pip study

pip studies

pip reviewers

final pip validation

quality of care

quality improvement

review process cont

real improvement