aea ji hyun park
DESCRIPTION
Ji Hyun ParkTRANSCRIPT
Developing Key Performance Indicators Developing Key Performance Indicators Developing Key Performance Indicators Developing Key Performance Indicators
for the R&D Programs using Logic Model for the R&D Programs using Logic Model for the R&D Programs using Logic Model for the R&D Programs using Logic Model
and Matrix Mapand Matrix Mapand Matrix Mapand Matrix Map
JiJi hyunhyun ParkPark, Researcher (, Researcher ([email protected]@kistep.re.kr))
Sang Sang kiki JeongJeong, Director (, Director ([email protected]@kistep.re.kr))
Dong Dong hoonhoon Oh, Research Fellow (Oh, Research Fellow ([email protected]@kistep.re.kr))
November 3 2006November 3 2006
R&D Industry & Infrastructure Evaluation Team of R&D Evaluation Center
Korea Institute of Science and Technology Evaluation and Planning (KISTEP)
Contents
II
IIII
IIIIII Challenges
Background for developing Standard Performance Indicators for the
evaluation of R&D programs
- Status of R&D program evaluation in Korea
- Necessity of performance indicators
Methodologies for developing Performance indicators and Key
Performance Indicators reflecting R&D characteristics
- Case studies for the development of performance
indicators using Logic Models
- Setting up Key Performance indicators on the Matrix
History of Research & Development program evaluation
1999Program evaluation was started by NSTC*
2006
2005
Performance based evaluation of R&D programs was
implemented with ‘Act on performance evaluation and
management of national R&D program’.
Program evaluation was output oriented.
It was a monitoring system to evaluate R&D programs
in a short term every year.
* NSTC : National Science and Technology Council
Status of national R&D evaluation in Korea
Ministries/
Research Councils
NSTC
(National Science and
Technology Council)
R&D
Centers
Funded
By Gov.
(R1)
Project
(R2)Project
R&D Program
Policy/Planning
Project
Technical Output
Program
Outcome & Impact
Project
(R3)
Social & economic
effect
Evaluation system of national R&D programs
Performance based evaluation of R&D programs
Meta EvaluationSpecific Evaluation Self Evaluation
Subject NSTC NSTC Ministries
Periods Feb ~ July Feb ~ AprMay
Evaluation for the achievement of
program objectives using
performance indicators
Summary In-depth evaluation issue
oriented and comparative
analysis
Evaluation for the results of self
evaluation
VisionVision
National policy level National policy level
goalsgoalsgoalsgoalsgoalsgoalsgoalsgoalsDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategies
Develop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic models
Program planningProgram planning
Ministrial program levelMinistrial program level
ProgrmsProgrms
AA BB . . .. . .
100%100% 50%50% %%
InitiativesInitiatives
Key programsKey programs
aaaaaaaa
bbbbbbbb
cccccccc........
MOSTMOSTMOSTMOST
....
....
Program PIProgram PI
Individual project Level Individual project Level
Main projectsMain projectsPrivate projectsPrivate projects
Performance indicators at the projects levelsPerformance indicators at the projects levelsPerformance indicators at the projects levelsPerformance indicators at the projects levels
. . . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . .
Future
image
Future
image Policy goalPolicy goal Generic LMGeneric LM Standard PI(SPI)
Standard PI(SPI)Strategic plansStrategic plans Key strategies
(National plans of each areas)
Key strategies(National plans of each areas)
Mutual
Agreement
Overview of the national R&D performance measure system
Standard national R&D program classifications
• Basic Knowledge
- Pure basic, Specific basic
• Industrial Technology
- Short term, Mid-Long term
• Public Services/Region/etc.- Public service, Bio·Agricul·Envir, Region
Result utilization
Program objectives
• Research & Development
• Development of Human Resources
• International Cooperation
• Utility/Equipment. Etc.
• Guideline for the self-evaluation provides Standard Performance Indicators with R&D program classification.
National master plans of S&T in Korea
2003~2007
2002~2006
2000~2006
2005~2007
2004~2008
2006~2010
Periods
Environment
Bio·Health
Agriculture & Forestry
Region
Industrial technology
Basic research
Area
Ministry of Science &Technology, Ministry of Commerce, Industry and Energy etc.
Improving regional science & technology
Ministry of Agriculture & Forestry, Ministry of Maritime affairs & fisheries, Forest government Information Agency etc.
improving Agriculture & Forestry
Ministry of Health & Welfare, Ministry of Science &Technology, etc.
Fostering Bioscience
Improving Basic research
Ministry of Environment, Ministry of Science &Technology etc.
Development of Environmental technology
Ministry of Commerce, Industry and Energy
Ministry of Science &Technology, Ministry of Education & Human Resources Development
Innovation of industrial technology
Involved ministriesPlans
Example : Developing standard logic model from the national plans
(Bio area)
Back
ground
Input
Activity
Output
Out
come
Increasing
concerns on
bioscience
Weak
competitiveness
in technology
Securing genetic
resources and promoting utilization
Increasing
interests in bio
safety
Weak industry-academic cooperation
and information network
Gov.
investment
Private
investment
Human resource
in Bioscience
Facilities and
information system
Related institution and acts Real time analysis
and monitoring on
invest portfolio
국가적 차원의 종합
조정
Expanding R&D areas and
concentrating on selected areas
Strengthen
ing HRD
in BT
Supplying
research
facilities
Global
networking
System
maintenance
Coordination
& allocation Establishing
policies
Expanding R&D
in New tech and
fusion tech area
Developing HR in BT area
Construction of research centers & systems
Constructi-on of
systematic infra
Construct-ion of
internatio-nal
linkages
Fostering
venture,
small co.
Promoting
industrializ
ation
External
factors
Strengthening
cooperation
between co.
Active
network
Increase of
government
investment
Concentrating on new tech and mixed tech development
Building R&D hubs for global
networking
Creating high values
and enforcing bio
resources
Upgrade of policy
analysis and
evaluation system
Fostering industries
strategically
Strengthen public and
R&D infrastructure
Concentrating on selected BT
areas
Intensively fostering Bio
industry as a strategic area
Policy
Issues
Government
Policies
Uncertainty
of R&D
Social
interests
-Biotechnology promotion law
-Biotechnology promotion plan-5,1620 billion -9392(’00yr)
Strong
specialty
-7,7455 billion
Overall Coordination
Arranging
program
system
Fostering
intensively BT
industry
R&D Industrialization Infrastructure
Establish-ment of
systematic infra
Policy for
industriali
zation
R&D priority
setting and
coordination
Example : Performance indicators from the logic model (Bio area)
·Social cognition
about importance
of Bio-tech
·International
competitiveness
·Promoting application of future genetic resources
·Public concerns about
national security and
bioethics etc. related
with Bio tech
·Status of industry-
academic cooperation and
information network
·Gov/private investment
·Portion of investment of BT
in a whole R&D investment
·Human resources involved
in BT research area
·Ratio of high level HR
·Research
facilities and
information
systems
· Related system and laws ·Ratio of R&D budget
depending on various
standards
·Development high, new
and fusion technologies
(papers, patent etc.)
·Development of HR and research institutions in BT area·Construction of research centers/bio-complex·Building R&D hubs for international cooperation and attracting international co.·Protection of system maintenance such as national safety, bio ethics and intellectual property rights·Application of bio resources
·Fostering ventures and
small co.
·Utilization and
commercialization of BT
·Revitalization of tech transactions
business to business
·Success rates of utilization and
commercialization
·Rates of government
investment
·Market share by creating
high values
·Analyzing in a view of
industrial economy
·Utilizing systems such as
evaluation system
·Extending the share of the market (Effect of industry development)·Improving productivity (Effect of reducing costs)
·Informing about Bio tech to people
·Establishing c-GMP and bioplants
·Strengthening international basis for clinical study
·Certifying safety of LMOs
·Internal bio tech degree comparing with
developed countries
·Advanced bio technology
·Ratio of BT industry among the main
strategic industries
·Important strategic tech in BT industry
·Minimizing redundancy and
duplication
·Establishing related policies
·Upgrade of R&D capacity
·Diffusion of research results
(Public relations etc.)
·Improvement of specialties in BT area·Utilization of network such as information & facilities·Construction of international cooperation system·Standardization and internationalization of BT·Preservation of bio resources for the national benefits
Back
ground
Input
Activity
Output
Out
come
External
factors
Infrastructure
Policy
Issues
Government
Policies
Uncertainty
of R&D
Social
interests
국가적 차원의 종합
조정
Expanding R&D areas and
concentrating on selected areas
Strengthen
ing HRD
in BT
Supplying
research
facilities
Global
networking
System
maintenance
Overall Coordination
Arranging
program
system
Fostering
intensively BT
industry
R&D Industrialization Infrastructure
R&D priority
setting and
coordination
-Biotechnology promotion law
-Biotechnology promotion plan
Positioning R&D programs in a
group of area on the Matrix
Setting up key performance indicators
of each area and utilizing other area’s
indicators as indirect performance
indicators
Setting up Key Performance Indicators on the Matrix
Need for analysis of specific characteristics in different areas
Area
-Too concentrated on public & basic research
-Benefits are invisible in spite of investment
-Long-term planning is necessary
Aerospace & astronomy
-Influenced on international restriction such as national energy safety & the Climatic Change Convention
-Too strong on public research Energy
-Development of industry is very fast
-Progressed rapidly fusion interlaboratory
-Nongorvernment lead area
Information & Communication
-Too strong on public goods
-Influenced on the law, system and restriction
-A technology intensive industry
Environment
-No key trends due to various area involved
-low level infrastructureAgriculture & Forestry
-Many beneficiaries should be satisfied in a short time
-Public area needs for research on aging society
-Ongoing information industrialization is necessary to open health care information
Health care
-Basic industrial technology
-A technology intensive industry in various area
-Outputs are produced in short term
Machinery & Materials
-Government lead mass programs
-Strongly influenced by government on decision making such as construction of new town and highway
-Too concentrated on centralization of capital
Construction & Transportation
Technological & Industrial Characteristics
Example : Key Performance Indicators
• Performance indicators of Bio·Health area
• Technology (know-how) transfer• Strengthening health care technology• Increasing competitiveness of health industries • Information industrialization Information industrialization Information industrialization Information industrialization • Improve customer satisfaction for public service Improve customer satisfaction for public service Improve customer satisfaction for public service Improve customer satisfaction for public service
• Papers• Patents• Effect on making social & economic profits • Advances in technology development• Public based technology • Reinvigorate local economy• Productivity increase• Development of human resources in bio·health area• International cooperative research • Construction and utilization of research facilities
Key Performance
Indicators
Indirect Performance
Indicators
Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & industrial industrial industrial industrial industrial industrial industrial industrial
characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D areaareaareaareaareaareaareaarea
Key Key Key Key Performance Performance Performance Performance Indicators Indicators Indicators Indicators
Key Key Key Key Key Key Key Key Performance Performance Performance Performance Performance Performance Performance Performance Indicators Indicators Indicators Indicators Indicators Indicators Indicators Indicators
Setting up
effective
performance
indicators of each
R&D program
Setting up Setting up
effective effective
performance performance
indicators of each indicators of each
R&D programR&D program
Process of developing R&D program performance indicators
Logic model of Logic model of Logic model of Logic model of Logic model of Logic model of Logic model of Logic model of individual programindividual programindividual programindividual programindividual programindividual programindividual programindividual program
Standard Standard Standard Standard Performance Performance Performance Performance IndicatorsIndicatorsIndicatorsIndicators
Standard Standard Standard Standard Standard Standard Standard Standard Performance Performance Performance Performance Performance Performance Performance Performance IndicatorsIndicatorsIndicatorsIndicatorsIndicatorsIndicatorsIndicatorsIndicators
Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Policy & National Policy & National Policy & National Policy & National Policy & National Policy & National Policy & National Policy & National
master planmaster planmaster planmaster planmaster planmaster planmaster planmaster plan
Introduction of ‘R&D commercialization for Genome’ program
• Objectives :
- Establish infrastructure for national resources and high tech medical technology
- Promote advanced clinical technology through disease research
- Realize genomic medicine by the predictive and tailored medicine
• Period : 2001∼2010
• Budget : 1,220 billion
• Flow of activities
1111st st st st (01~03) 2222nd nd nd nd (04~07) 3th 3th 3th 3th (08~10)Commercialization of genomic
research
4th(11~15)
Simulation of commercialized
techniques
Genetic analysis related life and
disease
• Program contents
- Planning to analyze Korean genome
- Genome based R&D commercialization
- Development of applied and predictive technology
- Analysis of genes which are related with life cycle and diseases
Establishment of infrastructure for genomic research
Background InputInputInputInput ActivityActivityActivityActivity OutputOutputOutputOutput OutcomeOutcomeOutcomeOutcome
Government invest 1,220billion
National resourcesProblems
related aging society
Increasing average span of human’s life
Accomplishment of Genome projects
Diseases
Development of resource genetic informationEstablishment of base tech system
Forming cooperation networks
Accomplishing genetic information delivery
Developing disease related genetic markersAnalyzing genes generating disease from animals
Systematization of disease knowledge
Utilization of predictive tech
Construction of cooperation for diagnosis and medical treatment
Development of public health policy/program
Realization of utilizing techProtection disease in advance
InfraInfraInfraInfra R&DR&DR&DR&D UtilizationUtilizationUtilizationUtilizationStabilizationStabilizationStabilizationStabilization
Preservation of bio resources
Construction of information
Research on gene function
Results of disease research
Development of High /Predictive tech
Healthcare/disease management program
Logic model of ‘R&D commercialization for Genome’ program
National long-term plans (Genetic research fostering plan)
▶ Selecting important indicators among the performance indicators generated from program logic model and
deciding final program KPI by mutual agreement between the NSTC and program managers.
Genomics
Disease research
Development of high tech medical technology
Genomic medicine (Predictive, tailored medicine)
Performance indicators of ‘Genome based R&D commercialization program’
ActivityActivityActivityActivity
OutputOutputOutputOutput
OutcomeOutcomeOutcomeOutcome
National resources
GenomicsDisease research
Development of high tech medical technology
Genomic medicine (Predictive, tailored medicine)
•No. of developing genomic characters•No. of constructing information DB
•Results of research on gene function-papers, patents
•Heath care program management
•Rates of guaranteeing bio resources
•Rates of genetic information DB
•Rates of developing human resources-Specialized HR -Teaching materials
•Construction of international cooperation network
•Analysis of Korea-specific and target diseases genetic characteristics
•Accomplishment of genetic information delivery map
•Development of gene markers related specific disease
•Mapping of specific diseases related genes
•Degree of establishing basis for genetic information system
•Utilization of high/predictive tech-Developing program for disease prediction-Covering information system for direct response
•Rates of cooperating for developing diagnosis and medical treatment
•Degree of advance in public health policy/program comparing with developed countries
•Degree of utilizing medical tech
•Rates of early diagnosis of diseases
•Results of research on diseases-papers, patents
•Results of developing high/predictive tech papers, patents
• Logic Model was used to develop performance indicators of R&D
programs.
• Performance indicators of R&D program should be developed through
mutual agreement between government and each ministry to decide
whether program’s performance indicators reflect the national master plan.
• R&D programs were evaluated output oriented but now on it would be
evaluated focusing on the performance such as outcome and impact. So,
further research would be developing methodologies how to measure the
impact of R&D programs.
• Peer review and survey are necessary to verify the key performance
indicators of specific area and to decide allocation of them.
Challenges
Thank you !
Presented by Presented by Presented by Presented by JiJiJiJi Hyun ParkHyun ParkHyun ParkHyun Park
Homepage| Homepage| Homepage| Homepage| www.kistep.re.krwww.kistep.re.krwww.kistep.re.krwww.kistep.re.kr
EEEE----Mail| Mail| Mail| Mail| [email protected]@[email protected]@kistep.re.kr
Tel| +82Tel| +82Tel| +82Tel| +82----00002222----589589589589----2275227522752275