monitoring and evaluation: information sources and systems

Monitoring and Evaluation: Information Sources and Systems

Session Objectives

At the end of this session, participants will be able to:• Name the main information sources for PHN M&E• Describe the main strengths and weaknesses of different

data sources• Discuss the main data-quality issues that need to be

considered• Explain why complementary data sources are often

required to monitor and evaluate health systems• Identify potential data sources that might apply in a

specific program context.

Overview

• Types of information• Strengths and weaknesses of selected

data sources• Data quality• Linking data sources• Exercise

The Finagle’s Laws of Information

The information you have is not the information you want

The information you want is not the information you need

And the information you need is usually not available

Definitions (1)

• Data: the raw facts that are collected and form the basis for what we know.

• Information: the product of transforming the data by adding order, context, and purpose

• Knowledge: the product of adding meaning to information by making connections and comparisons and by exploring causes and consequences

Definitions (2)

• Health system “all resources, organizations and actors that are involved in

the regulation, financing, and provision of actions whose primary intent is to protect, promote or improve health.” (WHO, 2000)

• ProgramA set of procedures to conduct activities. The objective is

normally the solution to a problem

Neither a health system or program is a static phenomena. They experience a continuous process of changes due to pressure from both outside the system and from within the system.

Definitions (3)

• Health Information System (HIS):A health-information system (HIS), similar to a health

management information system (HMIS)“…a system that provides specific information support to the

decision-making process at each level of an organization” (Hurtubise, 1984)

• Data Systems “a way of talking about the whole set of M&E indicators in a

performance monitoring-and-evaluation plan, and all of the data and other information that need to be gathered and understood in an orderly fashion that makes sense and help in program management and implementation”

Types of Information

• Surveillance – Epidemiological– Behavioral

• Routine service reporting • Special program reporting systems• Administrative systems• Vital registration systems• Facility surveys• Household surveys• Censuses• Research and special studies

Frequency of Data Collection

• ROUTINE or continuous data collection

• NON-ROUTINE or periodic data collection

Classify the previous information types by frequency of data collection

• Routine • Non-Routine

Frequency of Data Collection

• ROUTINE or continuous data collection– Health facility-based (patient information and service

statistics)– Community-based (service-statistics)– Program-based (administrative)– Vital registration– Sentinel reporting/demographic surveillance

• NON-ROUTINE or periodic data collection– Household or facility-based surveys– Population census – Rapid-assessment procedures (RAP)– Special studies/research

Geographic System Levels

• National

• Sub-national (e.g. district)

• Program area

The Health Information System: Data for Planning, Monitoring and Evaluation in the PHN Sector

Routine Non-Routine

Facility/Client

Community

Client Records Financial RecordsSupply RecordsFacility logbooks/data recordsAggregated Community Data

Client Mgmt and Follow-UpHealth Unit ManagementWork Planning/Priority Setting

Population-based surveyse.g. DHS

RapidAssessment

Methods

DistrictLevel

Facility-based surveyse.g. Situation Analysis, SPA

Aggregated Service StatisticsAggregated Mgmt DataSentinel Sites Observation ChecklistSelf-Evaluation (e.g. COPE)

Planning (Access)Management (Quality/Efficiency)Supervision (Performance)Disease Surveillance

NationalLevel

Aggregated Service StatisticsAggregated Mgmt DataAggregated Surveillance DataFinancial DataVital Registration Systems

Policy-MakingStrategic PlanningProgram TrackingDisease SurveillanceTechnical & Logistical Support

Birth and Death RecordsSchool RecordsCBD logbooksDrug Revolving Fund records

Client Mgmt and Follow-upSupplies ManagementCommunity Awareness

TYPE USE

Special Studiese.g. EPI cluster surveys,KAP studies, etc.

Census

Data-Collection Levels

Policy or program Service environment Client Population Spatial/geographic

Data Sources at the Policy/Program Level

Official documents (legislative, administrative) National budgets or other accounts data Policy inquiries Reputational rankings Program effort scores

Trends in Family-Planning Effort Score: 1972-1999

0

10

20

30

40

50

60

70

80

1972 1982 1989 1994 1999

Pe

rce

nt

of

ma

xim

um

Total East Asia So/SE AsiaNo Afr/Mid East Angl Africa Franc AfricaLatin Am/Caribb

Data Sources: Service Environment Level

• Administrative records – Service statistics– Management information– Financial data

• Service-delivery point– Routine service statistics– Audits/inventories– Facility surveys

• Agent, staff or provider– Performance, competency– Training records

EXAMPLE: SERVICE ENVIRONMENT LEVEL DATA

HEALTH MANAGEMENT INFORMATION SYSTEMS (HMIS)

• Note: An important way of monitoring routine data over time is through a Health Management Information System. An HMIS is a system for ongoing (routine) collection and reporting of data about client-service delivery. In many countries, this system operates at the national level. Ideally, these routine data are collected from a comprehensive set of service delivery points, and should cover topics such as:– Costs– Stockouts– Births– Mortality– Morbidity– Numbers of clients seen, referred (inpatient;

outpatient)– Numbers of clients by types of service

Data Sources: Client

• Client-exit interviews• Case surveillance

– Epidemiology of disease• Provider-client observation

– Management of the sick child– Vendor-client interaction

• Contact or visit registers• Customer record

Data Sources: Population Census Vital registration system Sample household surveys Special population surveys

- Demographic (elderly, youth)- Risk groups (CSWs, MSMs, IDUs)- Occupational (farmer, skilled labor)- Area-based (catastrophe-affected)

Biomarkers

Spatial/Geographic Data Sources

Satellite imagery Aerial photography Digital line graphs Digital elevation models Cadastral maps Global Positioning System data PLACE (site-based surveys)

Satellite Imagery

Aerial Photography Digital Elevation Models

GPS Data

Differential GPS

Digital Line Graphs

Chris Betz 1757 Millbrook Ln 28226 Y 2Christian Carl 1761 Millbrook Ln 28226 Y 1Chris McAfee 1765 Millbrook Ln 28226 Y 2Dale Legere 1776 Millbrook Ln 28226 N 6Donna Black 1780 Millbrook Ln 28226 Y 2

Demographic Data

Integrated GIS Database

Cadastral Data

High Transmissionof HIV

Guguletu, Cape TownSouth Africa

Carolina Population CenterUniversity of North Carolina at Chapel Hill

Air Photo Showing Potential High TransmissionSites in Guguletu Individual Structures

Can be Identified

1 - 1314 - 2526 - 3738 - 4950 - 61No Data

Neighborhood Statistics ShowingHigh Transmission Sites within 500 Meters

Strengths and Weaknesses of Selected Data Sources

Focus M&E Data Sources

• Facility-based routine information systems

• Facility surveys

• Population-based surveys

• Program records/administrative data

Facility-Based RHIS: Types of Information Generated

• Service statistics

• Outcomes of health interventions if individual patient records kept

• Not coverage (but can be estimated in some cases with other data)

• Not incidence (except nosocomial infections)

• Not prevalence

What is Wrong with Existing RHIS?

• Irrelevance of information gathered• Poor data quality• Duplication and waste among parallel health

information systems• Lack of timely reporting and feedback• Poor use of information • Centralization of information management

without feedback to lower levels

Strengths of Routine Health Information Systems

• Continuously collected – suitable for frequent reporting

• Existing system – no new data collection; builds local capacity; sustainability

• Typically available at lowest administrative levels (e.g. district, facility)

• Integral part of health system – direct link to health system actions

Common Problems With Facility-Based RHIS

• Variation in quality and completeness of reporting

• Timeliness of reporting

• Difficulty of providing coverage estimates

• Indicators may not be exactly what you want in a particular context

• May only cover government facilities

• Double-counting

Facility Surveys: Types of Information

• Readiness to provide services (inventory)– Infrastructure, staffing, hours of operation, fees

• Health worker knowledge– Provider interviews

• Quality of Care– Client-provider observation

• Client satisfaction– Exit interviews

Strengths of Facility Surveys

• Can cover both public and private health facilities

• More detailed information than is typically available in routine systems

• Can be tailored to specific program needs• Timing can coincide with program

implementation• Can combine with population survey for

outcome monitoring and impact evaluation

• Quality control may be easier than in routine systems

Limitations of Facility Surveys• Survey sampling design and analysis may be complex• Expensive, time-consuming• Stand-alone – sustainability concerns; less connected

to ongoing program decision-making• Information rapidly outdated, unless repeated – not

available regularly• Coverage/sample size constraints

– National vs. sub-national– By type of facility

• Small client sample sizes for some services (e.g. FP, STIs)

Facility Survey Initiatives and Tools• Service Provision Assessment (SPA/HSPA)

– DHS

• Service Availability Mapping (SAM) – WHO

• Quick Investigation of Quality (QIQ) – M/Eval (FP only)

• Situation Analysis (SA) – Population Council (FP only)

• JICA facility surveys and mapping

Population-based surveys: Types of information collected

• Knowledge and attitudes

• Practices

• Coverage

Strengths of population-based surveys

• Representative of the general population – no selection bias

• Wide range of outcome-level indicators can be collected

• Program coverage

• Well-tested instruments; quality control

Limitations of population-based surveys

• Coverage; national versus sub-national – not suitable for district-level estimates

• Frequency; typically only conducted every 3-5 years.

• Cannot detect small changes or changes over short periods of time without large sample sizes (expensive)

• Not suitable for some types of information (e.g. retrospective attitudes – recall bias)

Household survey programs (national)

• Demographic and Health Surveys (DHS)

• CDC Reproductive Health Surveys (RHS)

• UNICEF Multiple Indicator Cluster Surveys (MICS)

• PVO Knowledge Practices and Coverage Survey (KPC) (not national)

• CDC Young Adult Reproductive Health Surveys (YARHS)

Class Activity (1)How to improve facility-based routine information systems in developing countries?

Class Activity (2)

What are the determinants of health information systems performance?

Data Quality

Overview

• Types of information• Strengths and weaknesses of selected

data sources• Data quality• Linking data sources• Exercise

Data Quality Issues

Coverage Completeness (census, sample) Accuracy – measurement error; biases;

double-counting Frequency of collection Reporting flow Reporting schedule Public accessibility Supervision

• Hierholzer (Am. J. Med 1991; 91; 21-26) has called data the Researcher’s (M&E expert) sand. A lens maker takes sand, refines it, melts it, and through a long process of grinding and smoothing, fashions a lens with which to see the world more clearly. Similarly, a M&E expert takes data, refines it and smoothes it until a clearer picture of nature is revealed. If the sand is dirty or impure, the lens will be cloudy and distorted. If data is unreliable or invalid, the M&E expert’s understanding of nature will be clouded and distorted.

• By paying close attention to the data collection process from the conception of the data collection document through the editing of the data after it is collected, the M&E expert help keep his “sand” pure so that, in the end, nature may be viewed with much clarity and possible

• No amount of sophisticated analysis can salvage either a poorly designed or a badly carried out study

Linking data

Linking data

• Data can be linked from different sources, across different levels, or over time

• Linking data appropriately requires planning, preferably prior to data collection

• Understanding linked data can provide depth and continuity to enrich otherwise discrete points of information

Linking DataWhy link?• Survey data sets (e.g., household and facility information)

can be linked to compare services available and health outcomes across geographical units

• Geographical and survey data can be linked to examine the effects of physical attributes on service utilization

• Time series and panel data can help build causal explanations of program or project effects

Why not link?• May not be necessary for a given program in a given

context• Improper methodology can confuse issues more than

explain them• Analyzing linked data more appropriate for • evaluation than monitoring

Linking Data

Examples• Population and facility data can be linked to ascertain

health outcomes correlated with service availability, training, or quality of care (e.g. % of live births in catchment area attended by a trained personnel or % of women exclusively breastfeeding until 6 months among women going to facilities where provider training took place.)

• Facility and client data can be linked to learn about program expenditures per new family planning acceptor

• Facility and staff data can be combined to provide information about the proportion of clients per provider or the proportion of doctors per facility

Population-based data highlight limitations in HIS

Percent of women using a modern method by source of method

39

9

31

813

34

7

35

1014

05

1015202530354045

Govn't NGO Private Pharm Other

1997

1999

Facility-based data provide additional

explanations

Percent of government health facilities with a stock-out in the previous month

1015 12

2318

58

0

10

20

30

40

50

60

70

Pill Injectable Condoms

1997

1999

Comparing HIS and population-based data: confirming trends...

0

10000

20000

30000

40000

50000

1995 1996 1997 1998 1999 2000

Year

Nu

mb

er o

f H

ealt

h C

ente

r D

eliv

erie

s

3035404550556065707580

Per

cen

t D

eliv

erin

g at

a

hea

lth

fac

ilit

y

Number (HMIS)Percent (survey)

…..and raising questions

0

10000

20000

30000

40000

1995 1996 1997 1998 1999

Year

CY

P

0

5

10

15

20

25

30

Per

cen

t u

sin

g a

mo

der

n

met

ho

d

CYP

CPR

What do you need for evaluating program impact?

• A specific question: Are program inputs X influencing behavior (outcome) Y?,

If yes, by how much?

• A conceptual framework

• Appropriate Data

• An empirical model and estimation procedure

Linking Inputs to Outcomes for Evaluating Program Impact ……………...…..

Individual/HouseholdCharacteristics

Health ServiceSupply and Community Infrastructure

ServiceUtilization

HealthyPractice Healthy

Outcome

USAID Program

Other donors, NGOs, and the Government

Linking Inputs to Outcomes for Evaluating Program Impact (sources of data)



ServiceUtilization


Outcome

USAID Program


Household Survey

Facility SurveysCommunity survey

Compilation of dataon relevant interventions

community

household

individual

Clinic Based

CommunityBased

HealthServiceSupply

OtherCommunityCharacteristics

Levels of factors influencing health behavior and outcomes

HouseholdSurvey

Facility Survey

Community Survey

Clinic Based

CommunityBased

HealthServiceSupply

OtherCommunityCharacteristics

Data Sources

Data System needed for Program Impact Evaluation

• Population - based measures of individual demographic and health behaviors and outcomes, and household characteristics

Household survey

• Objective measures of health service supply and program inputs

Facility Survey

• Objective measures of other community characteristics

Community Survey

• Key: Areal Linkage of Surveys

Unlinked Program Inputs to Outcomes



ServiceUtilization


Outcome

USAID Program


Facility Survey linked to Household Survey

Health facilities selected on basis of household survey clusters

Objectives: Describe women’s accessibility to health services Describe communities’ health service supply environment

Example accessibility Indicators: “% women who have a SDP with FP within 10 kms.” “% women who have a physician within 5 kms.”

Sampling strategies: Service Availability Module (SAM)Family Life SurveyLiving Standard Measurement SurveyConcentric clusters

Linked Surveys:Service Availability Module (SAM)

Identify facilities by interviewing key community respondents with Community Questionnaire, then visit the closest facility of each type if distance <= 30 km.

Key issue: Access to services

Pros: • Facility indicators valid for “Average” woman or household (access indicators)

• Easy and cheap to implement

• Linked to household survey, so impact evaluation possible

Cons: • It does not necessarily give estimates for “universe” of facilities (but, see Hermalin,A. et.al.,1996 paper)

• Partial description of communities’ health service supply environment, likely

Service Availability Module (SAM) contd.

X

30 km.

X

X DHS cluster

Health facility

Linked Facility Surveys:Family Life Surveys (FLS)

Identify facilities by asking household survey respondents on sources of services,

compile list, and then visit facilities most frequently mentioned (up o a quota and < 45

minutes driving)

Key Issue: Community characteristics, not only Health supply



• More complete picture of supply of services / objective measures?

Cons: • It does not necessarily give estimates for “universe” of facilities (but, see Rand IFLS documentation)

• Less easy to implement (close coordination with HS)

Linked Facility Surveys:Living Standard Measurement Survey (LSMS)

Identify facilities by interviewing key community respondents with Community Questionnaire, then visit the closest facility of each type

Key issue: Basic Community characteristics, access to services


• Easy to implement


Cons: - It does not necessarily give estimates for “universe” of facilities

- Partial description of communities’ health service supply environment, likely

Linked Facility Survey: Contiguous Clusters Approach

Steps:

1• Define one or two rings of clusters around the DHS cluster

2• Canvass the DHS cluster and the surrounding ring of clusters to compile list of facilities

3• Conduct interview in all facilities in that area

4• Collect measure of size of clusters (population size) to calculate sample weights


HouseholdSurvey cluster

Cluster boundary

Health facility X

XX

X

X


Reliability of Estimates: Provides unbiased estimates of facility characteristics, but less efficient

Analytical Utility: Provides proxy of health service supply environment, It makes evaluation of program impact possible, without limiting monitoring

Cost: It is a variation of Area Frame Sample (1st. stage areas are DHS clusters) so, cost should be similar to stand-alone FS with area sampling and, lower than FS with list sampling (of same sample size). Cost also reduced if coordination of DHS and FS operations

Practical Feasibility: Needs careful planning since survey preparationlocation of clusters and their population size information required for

sample selection and weights

Tanzania Reproductive and Child HealthFacility Survey, 1999

• Objective: 1. To collect information on availability and characteristics of reproductive and child health services

2. To provide data for evaluating program impact

• Sampling Design: linked to TRCH Household Survey, contiguous clusters approach

Clusters: 146 (mainland)

Government NGO/Private Total

Hospitals 77 11 88

Health Centers 40 22 62

Dispensaries 138 117 255

UMATI / MS /oth 5 35 40

Total 260 185 445

Pharmacies 306

MwanzaMwanza

Tanzania Reproductive and Child Facility Survey, 1999

ÑÑ

Ñ

Ñ

Ñ

Ñ

ÑÑ

Ñ

ÑÑ

Ñ

Ñ

Ñ

10 0 10 20 Kilometers

WRAP UP

Supplemental SlideRHIS: Issues

• Do we trust HIS data?

– Reliability and validity of routine data

– Timeliness and relevance of data

– Costs of data collection and sustainability

– Rational process and parsimonious selection of indicators

• Is there an “information culture” with clear purposes of data

collected for managerial and strategic decisions?

– Environment that facilitates the direct use of routine data

• Does HIS meet data need of decentralized health services?

• Does HIS monitor referral systems effectively?

• Does HIS monitor private sector activities that are of public

health interest (what are the incentives, if any?)

Group Work: Case study 1

• Instructions:

• You have been asked to advise the following two programs on their M&E plan. Identify the potential data sources that you might explore to provide information for the M&E plan. List the factors that you consider as you assess the suitability of different data sources for each program.

• Program A is an NGO-run RH/MCH program operating in 3 districts in a country. The program aims to improve use of MCH services such as immunization, ANC, and family planning use in the districts in which it works. It provides training to staff in MOH clinics in the districts to improve the quality of services provided. Private sector health services are limited in the program areas so most people use government sector services. The program also undertakes community mobilization through community health workers and local radio spots to promote use of services. The program wishes to use some of the M&E plan data for ongoing program management and will be required to report to its donor annually on its performance as well as at the end of the project on its overall results.

Program B is a national AIDS prevention program. The program includes a mass media campaign on the ABCs aimed at reducing risk behaviors in the general population, the initiation of a PMTCT program and the expansion of its VCT program. The PMTCT and VCT program activities include training of health workers to provide quality VCT and PMTCT services, strengthening logistics systems to provide reliable supplies of HIV test kits to PMTCT and VCT sites as well as ARVs to PMTCT sites, opening new sites to increase the physical accessibility of these services to the population, and community mobilization to use VCT and PMTCT services through local media and community-based activities in areas where sites are located. In addition, new data collection forms will be added to the RHIS for PMTCT and VCT sites to collect service statistics on the new services, and sites will receive regular supervisory visits during their first few years of operation. The program wishes to use the M&E plan data for ongoing program management and annual reporting, as well as to fulfil relevant UNGASS and donor reporting requirements.

Group Work: Case study 2

• Instructions:

•

• Identify for each indicator the appropriate source of data needed in its calculation. Present and discuss your work in a plenary session.

1. Proportion of households with at least one ITN2. Proportion of children under 5 years old who slept

under an ITN the previous night3. Proportion of children under 5 years old with fever

in last 2 weeks who received antimalarial treatment according to national policy within 24 hours from onset of fever

4. Proportion of pregnant women who slept under an ITN the previous night

5. Proportion of women who received intermittent preventive treatment for malaria during their last pregnancy

6. HIV seroprevalence among all TB patients

7. Number of health facilities involved in DOTS with sufficient drug and laboratory supplies

8. Number of health facilities and laboratories involved in DOTS with sufficient capacity for DOTS

9. Number of health facilities where TB and HIV services are both available

10.Number of project staff trained

11.% of project budget spent on health infrastructure

12.% of project beneficiaries (patients) who are accurately referred

13.Number of networks/partnerships involved in project

14.Number of project service deliverers trained in M&E

15.% of overall project budget spent on M&E

16.Number of project services deliverers trained in procurement and supply management

17.% of project service delivery points with sufficient drug supplies

18. Unit cost(s) of project drug(s) and commodities

19.Number of people reached by the services20.Number of service points supported by the

funding21.Number of providers trained in the service22.HIV-infected pregnant women receiving a

complete course of antiretroviral prophylaxis to reduce the risk of MTCT (number and percentage)

23.Districts with access to donor recruitment and blood transfusion

24.Transfused blood units screened for HIV25.People with advanced HIV infection receiving

antiretroviral combination therapy (number and percentage)

monitoring and evaluation: information sources and systems

Documents