evidence-based policy making practice based on the international evaluation of funding and...
TRANSCRIPT
Evidence-based policy making practice based on the international evaluation of funding and management of NSFC
Zheng YongheDeputy Director General, Bureau of Science Policy, NSFC
Kunming, May 23, 2014
Outline
NSFC’s funding policy development: Dimension discussion
The basis of policy making: Evidence-based evaluation
Essential work of evaluation: Evidence collection
Case study: International evaluation of funding and management of NSFC and the policy making
NSFC’s funding policy development: Dimension discussion
Principals: Problem-driven; reform-driven
The national performance evaluation system has not established in China, currently. NSFC’s international evaluation on funding and management performance provides a typical success case for comprehensive evaluation in China.
1. Evaluation as an research-based approach plays an important role in policy making.
2. Evidence-based evaluation insure the legality and credibility of policy in policy argument process.
3. Question-oriented evaluation and independent third-party evaluation are more feasible and convincing.
The basis of policy making: Evidence-based evaluation
The pathways and period of suggestion into policy
The basis of policy making: Evidence-based evaluation
Essential work of evaluation: Evidence collection
Evidence-based
Evidence-based
Evidence-based
1. Brief overview of international evaluation of funding and management of NSFC
The evaluation is commissioned by MOF and NSFC together. The State Council attached importance to this evaluation.
Dual Objectives of the Evaluation: – Accountability: Provide an independent assessment of the
overall performance of NSFC’s funding and management during the past 25 years.
– Lesson learning: Improve the NSFC’s funding and management performance; redefine NSFC’s strategic role within the NIS of China
Case Study: International evaluation of funding and management of NSFC and the policy making
Brief overview of international evaluation of funding and management of NSFC
Evaluation Approach and Principals– Modality: Domestic preparation followed by international evaluation
– Approach: Evidence-based
– Principals: Independent, Objective
Evaluation FrameworkBased on the dimension discussion
Functions of Actors in the evaluation
Brief overview of international evaluation of funding and management of NSFC
Typical cases: Provided by NSFC
Synthesis evidence report: Provided by independent evaluation organization
International evaluation report
Brief overview of international evaluation of funding and management of NSFC
Implementation of the Evaluation
第 12页(共 25页)International evaluation report
Synthesis evidence report
Thematic report
Cases report
Literature metrology
report
Survey report
Interviewreport
Special report
Domestic preparation
International evaluation
Designing evaluation
Thematic researchTypical cases Collecting evidences
scheme design
Individual Evaluating by expert
Focused evaluating by Expert Panel
Writing report byChair & rapporteur
2. Evidences and the evaluation procedure
factor: evidence DFG RCN JSPS FWF SGCRP NSF FP6 Interview
Director of NSFC
Literature ofevaluation
Backgroud ■ ■
Objectives ■ ■ ■ ■
Commissioning entity& Organizers ■ ■ ■
Framework ■ ■ ■ ■ ■ ■ ■ ■
Information source ■ ■ ■ ■ ■ ■ ■ ■ ■
Subject of assessment ■ ■ ■ ■ ■ ■ ■ ■
Procedure & Time ■ ■ ■ ■ ■
Performance report ■ ■ ■
Application of results ■ ■ ■ ■ ■
Budget ■ ■
A. Designing evaluation: Factor analysis in designing framework based on evidence
B. Domestic preparation: collecting evidences from questionnaire surveys, interviews and other activities
Six sets of questionnaire surveys: delivered 77799, returned 20221
B. Domestic preparation: collecting evidences from questionnaire survey, interview and other activities
Interviews: Focus group meeting & face-to-face interview 53times, involving 294 persons
C. International evaluation: From ‘Synthesis evidence report’ to ‘International evaluation report’
IECNCSTE
IEC presents the evaluation results in ‘International evaluation report’ based on ‘Synthesis evidence report by NCSTE individually.
3. Evidence-based policy making: After Evaluation
Case 1: Raising ‘Excellent Young Scientists Fund’ Case 2: One year ‘Time out’ for applicants who have
been unsuccessful in two consecutive years.
Case Study: International evaluation of funding and management of NSFC and the policy making
Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund
Evidence Naturally, having fostered young scientists in its talent programs
NSFC is now seeing them age. The effect is especially marked in the DYS found, where increasingly those just below the age limit win the grants.
Suggestion from ‘International evaluation report’:
We recommend that a new program should be established for younger researchers, no longer based on biological age but rather on ‘scientific age’……
and, recommend that a substantial ‘China Chair’ program (in the style of the Canada Research Chairs) be established by NSFC for junior investigators with sustained support over years……
Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund
New policy making of NSFC
Raising “Excellent Young Scientists Fund” in 2012
Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund
The ‘Excellent Young Scientists Fund’ plans to support with funding of 1 million RMB per project.
Type: Directly into policy Evidences source: Data statistics, Evaluation report
The challenge of NSFC in 2009
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
More management pressure on NSFC staff: The growth in applications has been much faster than the growth in staff in recent years. The annual number of applications per permanent member of staff has risen from 142 in 2001 to 505 in 2009.
More pressure on Reviewers comparing with NSF or DFG.
Proposal Staff Reviewer
NSF 45000 1400 42000
NSFC 119000 200 40000
DFG 35000 750 9000
More pressure on Reviewers
The challenge of NSFC in 2009 Rapid increasing proposals of GP: As the main instrument of NSFC, the
General Program application has been increasing rapidly since 2002.
?the future?
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Alternatives of Policy
1
2
3
How to make the decision? Evidence-based evaluation Policy making
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Step 1: Analysis the reason of rapid increasing NSFC grants are increasingly attractive to actual and potential beneficiaries. Universities and other organizations are using receipt of NSFC funding as an esteem
indicator for promotions and in the assessment of large projects, further increasing application pressure.
Evidence: Questionnaire survey of supporting institutions • 75% of respondents use NSFC grants as an important basis for various merit-based
assessments of research staff.
• 58% of respondents regard NSFC grants as a condition for job promotion or assignment.
• 56% of respondents allocate additional research funds to NSFC projects.
• 51% of respondents pay cash bonuses or other benefits to NSFC grantees.
• Only 8% of respondents have no special policies for NSFC awards or awardees.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Step 2: Evaluation based on the evidence of stakeholder surveys and data analysisStakeholder 1: Staff of NSFC a) When interviewed, the staff from the Science departments complain
that “The current workload is too heavy, and the only time we can have for ourselves is the time for sleep.”
b) Most of the staff who attended the group meeting also complained about the overloading. One of the participants pointed out that a program director had only eight minutes to review a proposal in the given timeframe.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Stakeholder 2: Institutes and universities a) Most institutes and universities oppose to limit the number of
applications, because more conflicts will emerge between applicants and institutions, and more and more applicants will get the opportunity by “old-friend network”.
b) 43.5% of 841 institution respondents considered the current success rate of application for the GP was appropriate, while 46.0% considered it moderately low or low.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Stakeholder 3: PIs of General Program The results from the questionnaire survey of PIs of the GP show that,
53.8% of 10288 respondents considered the grant size of GP was appropriate, while 43.5% considered it was less than the actual research costs. Only 0.6% of the respondents thought it was more than actual research costs. The rest 2.1% of respondents’ answers to the question is “I’m not sure”.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Stakeholder 4: Reviewers 45.5% of reviewer respondents thought the current success rate of the GP
was appropriate, while 49.8% thought that it was moderately low or low.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Stakeholder 5: Rejected Applicants 44.1% of rejected applicant respondents agree that applicants who were
consecutively denied for two years are not permitted to submit proposal in the third year, but can submit proposal in the fourth year.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
• Evaluation suggestion:At least one year time our for applicant who has been unsuccessful in two consecutive years.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
资助率:21.96%
23.88% 23.38%24.17%
20%
22%
24%
26%
28%
30%
N=2 N=3 N=4
2.85%
-3.35%
-1.92%
General Program(GP)
Young Scientists fund(YSF)
Fund for Less Developed Region(LDR)
资助率:23.45%
26.80%27.92%
28.88%
20%
22%
24%
26%
28%
30%
N=2 N=3 N=4
资助率:19.24%
16.39%15.43%
14.82%
12%
14%
16%
18%
20%
22%
N=2 N=3 N=4
The success rates of GP after 2-4 consecutive rejected years are lower than the average rate.
But the success rates of YSF and LDR after 2-4 rejected consecutive years are higher than the average rate.
The trend will be more obvious following the increasing of consecutive rejected years.
Step 3: Further research for policy making after evaluation
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Step 3: Further research for policy making after evaluation
6071 8272 8014 883813159
1667219214
250428646
12144
15108 14380
18036
2207624660
31369
3725370.22% 68.12%
53.04%
61.46%
72.96%75.52%
77.92% 79.83%
0%
20%
40%
60%
80%
100%
0
10000
20000
30000
40000
50000
2005 2006 2007 2008 2009 2010 2011 2012 2013
申请减少量 未申请部分 N次失败后申请率
The proportion of applicants who have been unsuccessful in two or above consecutive years maintains 70%- 80% in recent years.
The prediction said that it would decrease about 30000 proposals if NSFC applies a ‘time-out’ policy to forbid the proposals of applicants who were rejected in both 2011 and 2012. (The potential applicants proportion was proposed 80%)
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Step 4: Policy making & performance of the new policyPolicy making:
a) Enforcing one year ‘time out’ for applications who have been unsuccessful in two consecutive years since 2014. NSFC had informed the message to all potential applicants in 2013.
b) Adjusting the grant size, and the duration of General Program from 3 years to 4 years.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
result of the new policy: The number of the whole proposals is obviously decreasing in 2013 and 2014 because of the ‘time-out’ policy.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
result of the new policy: In contrast to the growth trend, the number of proposal for the GP began decreasing dramatically in 2013.
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
result of the new policy: The grant size and success rate of GP began increasing obviously since the implementing of the new policy.
The grant size and success rate of General Program in 2009-2013
Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.
Thanks!