Health Policy Evaluation:Concept and Application

PITAKPOL BOONYALIK, MD, PHD, MBA, MSC

Health Policy Evaluation: Concept and Application

Program evaluation concept

Frameworks for program evaluation

Steps to do program evaluation

Case studies

2

Program evaluation

A systematic method for collecting, analyzing, and

using information to answer questions about projects,

policies and programs, particularly about their

effectiveness and efficiency. In both the public and

private sectors, stakeholders often want to know

whether the programs they are funding, implementing,

voting for, receiving or objecting to are producing the intended effect.

3

Source: Administration for Children and Families (2010) The Program Manager's Guide to Evaluation. Chapter 2: What is program evaluation? Retrieved from https://en.wikipedia.org/wiki/Program_evaluation#CIPP_Model_of_evaluation

Evaluation vs Research4

Evaluation : Systematic approach

Assessment of the need for the program

Assessment of program theory

Assessment of implementation

Assessment of the program's outcome or impact

Assessment of the program's cost and efficiency

5

Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.

ปจจยแหงความส าเรจของโปรแกรมสขภาพ

การประเมนความตองการ (Need assessment)การออกแบบโปรแกรมโดยทฤษฎหรอหลกฐานเชง

ประจกษ (Theory-driven program and evidence-based program planning)

การมสวนรวมของชมชน (Community participation)

6

Assessing needs

the processes or methods used by evaluators to describe and diagnose social needs

7

Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.

Assessing needs (cont.)

Define the problem

Assess the extent of the problem

Define and identify the target of interventions

8

Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.

9

คณสมบตของเครองมอประเมนทางสขภาพ

ความเทยง (Reliability) = ความสามารถของเครองมอในการใหผลลพธเหมอนเดมInternal consistency reliability

เชน Cronbach’s alpha, split half reliabilityTest-retest reliability

เชน Pearson’s correlationInter-rater reliability

เชน Pearson’s correlation

10

คณสมบตของเครองมอประเมนทางสขภาพ(ตอ)

ความตรง (Validity) = ความสามารถของเครองมอในการแสดงผลในสงทตองการวดFace validityContent validityCriterion validity

Concurrent validity, predictive validity, postdictive validityConstruct validity

11

ดชนชวดขนาดปญหาสขภาพ

อบตการณ (Incidence) = จ านวน case ใหม ตอประชากรกลมเสยง

ความชก (Prevalence) = จ านวน case เกา +ใหม ตอประชากรทงหมด

Assessment of program theory

The program theory, also called a logic

model or impact pathway, is an

assumption, implicit in the way the

program is designed, about how the

program's actions are supposed to achieve the outcomes it intends.

12

Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11).

Assessment of program theory (cont.)

Assessment in relation to social needs

Assessment of logic and plausibility

Assessment through comparison with research and practice

Assessment via preliminary observation

13

กรอบแนวคด PRECEDE PROCEED Model (Green & Kreuter 1999)

การวเคราะหนโยบายทางเลอกทางการจดการปญหาการใชความรนแรงในนกเรยนอาชวศกษาโดยใชกรอบแนวคด PRECEDE Sub-model

การวเคราะหนโยบายทางเลอกทางการจดการปญหาการใชความรนแรงในนกเรยนอาชวศกษาโดยใชกรอบแนวคด PROCEED Sub-model

Socio-ecological model

Theories for Health Promotion Programs

Individual health behavior Interpersonal health behavior) Organization and community health behavior

18

ทฤษฎระดบบคคล

Health Belief Model Theory of Reasoned Action Transtheoretical Model and Stages of Change Precaution Adoption Process Model etc.

19

ทฤษฎระดบระหวางบคคล

Social Cognitive Theory Social Network Theory Transactional Model of Stress Interdependence Theory etc.

20

ทฤษฎระดบองคกรและชมชน

Community Building Community Organizing Diffusion of Innovations Communication Theories etc.

21

Health Belief Model) 22

ตระหนกถงความเสยงตอโรค

(Perceived Susceptibility)

- อาย เพศ เชอชาต- บคลกภาพ- ฐานะทางเศรษฐกจสงคม - ความร

ตระหนกถงภยคกคาม

(Perceived Threat)

สงกระตนใหเกดการกระท า

(Cues to Action)

การตระหนกถงผลดและผลเสยของการเปลยนพฤตกรรม

(Perceived Benefits versus Barrier to Behavioral Change)

แนวโนมของการเปลยนพฤตกรรม

(Likelihood of Behavioral Change)

การรบรสวนบคคล(Individual Perception)

ปจจยเสรม(Modifying Factors)

แนวโนมของการกระท า(Likelihood of Action)

Rosenstock in Egger et al 1990 :24

23The Transtheoretical Model of Behavioral Change

24ตวอยางการแกปญหาทางสขภาพจตของวยรนแบบใชทฤษฎ/หลกฐานทางวทยาศาสตร

สมาธสนในชนประถม

การเรยนดอย

ขาดการยอมรบจากกลมเพอน

กาวราว

ขอายใชสารเสพตดในวยรน

ซมเศราฆาตวตาย

คนหา proximal targets ในเดก

ประถมและใหการแกไขเพอปองกน distal

outcomes ในเดกวยรนSources: e.g., Kellam et al. (1991) Am J Community Psychol.

Aug;19(4):563-84; Kellam et al. (1989) NIDA Res Monogr.;95:368-9.

พฤตกรรมตอตาน

กฎระเบยบในวยรน

The Logical Framework Matrix (Logframe)25

Source: BARRETO (2010) cited in Leonellha Barreto Dillon Logical Framework Approach retrived from http://www.sswm.info/content/logical-framework-approach

Logframe example26

Assessing implementation

This evaluation determines whether the components identified as critical to the success of the program are being implemented.

27

Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.

Questions of implementation evaluation

Finding out about service use

Finding out about service delivery

Finding out about the organization of the program

Looking at the program’s resources

Looking at participants’ experiences of the program §

28

Source: Purdon, S. et.al. (2001) RESEARCH METHODS FOR POLICY EVALUATION, the Department Work and Pensions,

Methods of implementation evaluation

Monitoring and Operational Research

Social research

Quantitative surveys

Qualitative research: depth interviews and group discussions

Case studies

29

Source: Purdon, S. et.al. (2001) RESEARCH METHODS FOR POLICY EVALUATION, the Department Work and Pensions,

Assessing the outcome/impact (effectiveness)

This involves trying to measure if the program has achieved its intended outcomesOutcomes are the factors that the program

is expected to affect

Impact Evaluation: to measure the impact a policy or program has on defined outcome measures. It usually involves measuring the counterfactual

30

Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.

Purdon, S. et.al. (2001) RESEARCH METHODS FOR POLICY EVALUATION, the Department Work and Pensions,

Program evaluation: level of outcomes 31

Efficacy

Effectiveness

Sustainability

Methods of outcome/impact evaluation

Randomized trials – an experimental design

Matched area comparison design

Before-after study

Matched comparison group design

32

Source: Purdon, S. et.al. (2001) RESEARCH METHODS FOR POLICY EVALUATION, the Department Work and Pensions,

Assessing efficiency

cost-benefit or cost-efficiency analysis assesses the efficiency of a program

33

Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.

CIPP Model:

Context: What needs to be done? Vs. Were important needs addressed?

Input: How should it be done? Vs. Was a defensible design employed?

Process: Is it being done? Vs. Was the design well executed?

Product: Is it succeeding? Vs. Did the effort succeed?

34

CDC’s Framework for program evaluation in public health

35

Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48 (No. RR-11)

Five-tiered approach

Tier 1: needs assessment (sometimes referred to as pre-implementation)[29]

Tier 2: monitoring and accountability

Tier 3: quality review and program clarification (sometimes referred to as understanding and refining)[30]

Tier 4: achieving outcomes

Tier 5: establishing impact

36

Case studies

Background and significance

Evaluation questions and objectives

Evaluation Framework

Methods

Results and Discussions

37