Download - Meeting an Evaluation Challenge: Identifying and Overcoming Data and Measurement Difficulties
Meeting an Evaluation Challenge: Identifying and Overcoming Data
and Measurement DifficultiesAEA Evaluation 2006
“The Consequences of Evaluation”RTD TIG, Think Tank Session
Portland, OregonNovember 4, 2006
Rosalie T. RueggManaging DirectorTIA Consulting, [email protected]
Connie K.N. ChangResearch DirectorTechnology AdministrationU.S. Department of [email protected]
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
2
The 4th in a Series of Think Tanks on Barriers to Evaluation 2003: Identification of 6 Types of Barriers
to Evaluation
2004: Focus on Institutional and Cultural Barriers—Feedback Loops
2005: Focus on Methodological Barriers
2006: Focus on Data and Measurement Difficulties
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
3
OverviewSix Categories of Barriers Identified --
20031. Institutional/cultural -- 20042. Methodological -- 20053. Resources4. Communications5. Data/Measurement -- 20066. Conflicting stakeholder agendas
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
4
2003 Think Tank found …Striking commonality of evaluation barriers among programs and across countries
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
5
These barriers were said to impede … Demand for evaluation Planning and conducting evaluation Understanding of evaluation studies Acceptance and interpretation of
findings Use of results to inform
program management budgetary decisions public policy
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
6
2006 Think Tank focus – data and measurement difficultiesData difficulties Trail gone cold Missing data Data quality Other?Measurement difficulties Incommensurable effects Forecasts for prospective analysis Aggregation across studies Inadequate treatment of uncertainty and risk Accounting for additionality and defender technologies Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
7
Data difficulties Trail gone cold Missing data Data quality Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
8
Difficulty: Trail gone coldWith long time gaps between research, results, and
evaluation, evaluators find … Memory lapses “Over the transom” effect with trail broken Mixture of funding sources Distinctiveness lost with technology integrations Departure of key employees Acquisition, merger, death of companies Other? (Generated by Think Tank discussion)
Use of a financial instrument that does not have a legal requirement or dedicated budget to stimulate reporting/cooperation with evaluators
Reliance on a partner to report without regard to capability Mechanics of surveying may be a problem
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
9
Dealing with trail gone cold Your thoughts? (Generated by Think Tank
discussion) Build in requirement to report Be proactiveness in conducting surveys a few
years post project (e.g., Tekes – 3 yrs out; 67% response rate; no big changes between 3 yrs and 5 yrs out)
Incentives to high response rates are key because 5% of projects account for 95% of economic gains that random sampling would miss
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
10
Dealing with trail gone cold—Overview Third best: Conduct “archeological
digs”
Second best: Use clues to focus “detective work”
Best: Be proactive. Track and document data in real time
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
11
Difficulty: Missing Data Data collection is spotty Responses are incomplete Files are corrupted Not all paper records have been
converted to electronic files Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
12
Dealing with missing data Your thoughts? (Generated by Think Tank discussion)
Compose data from several sources to fill gaps. If questions are too difficult to respond to – not reasonable,
or confidential – you end up with missing data. So, do trial testing before survey launch.
If program is too young to have data, use proxy data, e.g. ATP used Japanese data to test a firm productivity model, and later ran it with ATP data and got similar results.
Explore statistical techniques to impute missing data. Use security to ensure staff are not taking or corrupting
data. Look for pattern of missing data. Look for biasing effects in data collection. Check for errors in transcribing data from paper to
electronic records (e.g., a check of one survey found tabulated 200 pregnant men!)
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
13
Dealing with missing data—Overview Prevention is the best approach: implement sound data
collection
Find the missing data using multiple strategies
Use proxy data
Use techniques for dealing with partial data
Use techniques to impute data
Forthcoming book: Missing Data by Patrick McKnight, and Katherine McKnight (George Mason U); Souraya Sidani (U of Toronto); and Aurelio Jose Figueredo (U of Arizona)
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
14
Difficulty: data quality issues Are the data valid for the intended use? Other? (Generated by Think Tank discussion)
Source is key to data quality (i.e., are we asking the right people; are they motivated to answer truthfully?)
Are data used for making comparisons comparable? Has a thorough definitional effort been conducted prior
to data collection to ensure the right data are collected?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
15
Dealing with data quality issues
Your thoughts? (Generated by Think Tank dicussion) Think through upfront, scope out what you want to
collect; how you go about collecting (e.g., NEDO thought through data collection; at the same time, flexibility is important – to avoid locking-in too early and to allow for adjusting/modifying/correcting – i.e., to support an iterative process in data design)
Avoid overly complex data collection instruments Check data entry to detect human error Provide means of calibrating answers and spotting
outliers
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
16
Dealing with data quality issues—Overview Understand what is required by detailed up-front
exploration Assess reliability of data processes Use data quality assurance tools
Monitor data quality over time “Clean” data Standardize data to conform to quality rules
Verify calculations[Possible Sources: International Association for Information and Data Quality
(IAIDQ); Data Management Association (DAMA)]
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
17
Measurement difficulties Incommensurable effects Forecast for prospective analysis Aggregation -- across studies, projects, levels, etc. Inadequate treatment of uncertainty and risk Accounting for additionality and defender technologies Other?
Instrumentation, who does the measurement? How does calibration occur?
Problem of attribution is profound, scientists argue over it (discovery, use), and problem increases exponentially closer to commercialization (lack of acknowledgement of significance of competitor’s work). Reluctance to give government credit (e.g., companies don’t want gov’t to recoup; want to deny gov’t support helped create success today).
Double counting Difficulties in measuring innovation
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
18
Difficulty: Incommensurable effectsPresenting effects measured in
different units in a single study Knowledge – # papers, patents Economic -- $ Environmental – level of emissions Safety -- # accidents Employment -- # jobs Energy security – barrels of oil imports …
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
19
Dealing with incommensurables—some ideas In some cases, you can express different
effects in a common measure (e.g., make them commersurable)
In other cases, decision makers may want effects to be expressed separately in their own units In which case, the decision maker must
make trade-offs subjectively Or, the evaluator weights and combines
different effects using index values that are easier to compare, e.g., ATP’s Composite Performance Rating System (CPRS)
Or, other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
20
Difficulty: Forecast for prospective analysis More uncertainties compared with
ex-post analysis technical uncertainties resource uncertainties market uncertainties (market size,
timing, speed of commercialization) other …
Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
21
Dealing with forecast for prospective analysis—some ideas Build uncertainty into the estimations Consider all important alternatives
different levels of technical success different market applications to consider
Revise forecast as additional info becomes available
Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
22
Difficulty: Aggregation across studies Different base years Different time periods Different methods Differences in underlying
assumptions, models, algorithms Different types of effects measured Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
23
Dealing with aggregation across studies—some ideas Best is to reduce incompatibility by
standardization, and require transparency and replicability
Where there is internal consistency, combine common measures across studies
In place of aggregation, summarize across studies in terms of a single measure (e.g., a table of IRRs), at your own risk!
Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
24
Difficulty: Inadequate treatment of uncertainty and risk, leading to -- Overstatement of results
Unrealistic expectations
Faulty decisions
Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
25
Dealing with inadequate treatment of uncertainty and risk Use a techniques such as the following: Sensitivity analysis Statistical test of variation (e.g., confidence
interval) Expected value analysis Decision trees Risk-adjusted discount rates Certainty equivalent technique Computer simulations using random draw across
range of values Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
26
Difficulty: Accounting for additionality and defender technology Incorrectly attributing all observed changes to a
program’s effect Partial identification or double counting of
additionality effects Problems in defining reference groups for control
studies Recognizing limitations of counterfactual
questions (i.e., non-experimental design) Ignoring or incorrectly modeling the defender
technology Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
27
Dealing with additionality and defender technology—some ideas Always consider effects with and without
the program (e.g., counterfactuals, before/after comparisons, control groups)
Breakout additionality effects into component parts
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
28
Dealing with additionality and defender technologies, cont’d Systematic comparison
Program--yes
Program--no
Success rate ASuccess rate BSuccess rate C
Success rate DDefender tech with improvement rate XDefender tech with improvement rate Y
prob=10%
prob=50%
prob=40%
“Dynamic modelling of defender tech”
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
29
2006 Think Tank focus – data and measurement difficultiesData difficulties Trail gone cold Missing data Data quality Other?Measurement difficulties Incommensurable effects Forecast for prospective analysis Aggregation across studies Inadequate treatment of uncertainty and risk Accounting for additionality and defender technologies Other?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
30
SummarySix Categories of Barriers Identified --
20031. Institutional/cultural -- 20042. Methodological -- 20053. Resources4. Communications5. Measurement/data -- 20066. Conflicting stakeholder agendasNov 2007 … what’s next?
2006
AEA
Eva
luat
ion
Conf
eren
ce T
hink
Tank
Rue
gg &
Cha
ng
31
Contact information Rosalie T. RueggManaging DirectorTIA Consulting, [email protected]
Connie K.N. ChangResearch DirectorTechnology AdministrationU.S. Department of