guideline: research proposals evaluation · office of the vice-president for research and community...
TRANSCRIPT
Mekelle University
Office of the Vice-President for Research and Community Services
Guideline:
Research Proposals Evaluation
April 2016
Mekelle
1
IntroductionMekelle University researchers have been undertaking researches at different scales using funds
sourced from University's research recurrent budget and other external fund sources. Selection of
proposals has through times varied according to specific conditions and requirements of funds.
However, since varying criteria for evaluation has been a reason for repeated complaints and
grievances among different colleges and institutes, the office of VPRCS has decided to develop
permanent criteria (with provisions for specific timely modifications). The evaluation criteria are
developed based on concepts and principles stipulated in the Mekelle University Research and
Development Policy document. These principles include objectivity, enhanced impact,
ethicality, servitude to the public etc.
Generally, research projects in Mekelle University fall within three different categories namely:
large scale projects, Medium Scale Projects, Small Scale Projects (which include Young
and Female Researcher Funds). These categories are developed based on currently existing
research needs and staff involvement. Therefore, in the future, if a new level of categorization
arises, new criteria, based on the principles of the research and policy guidelines might be
developed. Separate guidelines are developed for each of the three categories. The need for
having separate guidelines for large-scale, young and female proposals is because the different
categories have different objectives and require different these have different objectives (stated
in the document), we need to have different guidelines that accommodate the differences in
objectives. For instance while affirmative action for involvement of women researchers can and
should be a measurable criteria for large scale, we cannot make it a criteria for the female-
researcher fund.
Moreover, any given guideline is made to have enough provisions for modifying according to
newly emerging issues. For instance, while we can use the GTP-II issues to serve as a guide for
next five years, we cannot have a constant thematic area for all calls in the future, . Therefore,
the thematic area criteria is constant, but specifics will depend on the thematic area of
importance in the time. Thematic areas might also vary with different funders, objectives of calls
etc. Therefore, guidelines, while constant interns of the principles of the criteria, will be open for
modification.
2
Generally, the criteria are developed based on the following concepts or principles:
1. Team of experts based evaluation: Evaluation is made by a team of multi-disciplinary
professions selected and agreed by colleges and institutes who participate in proposal
selection.
2. Automatic exclusion criteria : Proposals that do not fulfill certain criteria are
automatically excluded. These exclusion criteria, which are different for the different
categories are clearly listed
3. Public defense: Shortlisted proposals (number depends on availability of fund) will be
invited to public defense
4. Provision of feedback of comments to applicants Evaluation committee will provide
generic comments to all proposals
Objectives
These set of guidelines is prepared for the following purposes
1. To minimize bias or subjectivity in the evaluation of research proposals, enhancing
effective competition among applicants
2. Ensuring that high quality research proposals that promise the highest possible quality
and quantity of outputs are funded
3. Ensuring that proposals relevant to thematic area and priorities of a certain call are
selected
Working definition of Project types
1. Large-scale projects
Large scale research projects are those multi-disciplinary and inter-disciplinary research projects
with a budget requirement minimum 250,000 Birr and that will take at least three years to
complete. The objectives of large-scale projects is to bring usable outputs or solutions for
specified real life problems, while enhancing the research experience of junior and senior
3
researchers. Thematic areas for large-scale projects shall be defined by the fund source. Large
scale projects are evaluated and administered at the University (center level)
2. Medium scale research projects
Medium scale research projects are those multi-disciplinary and inter-disciplinary research
projects with a budget minimum of 150,000- Ethiopian Birr and that will take up to two years to
complete. The objectives of medium scale projects is to developed the research experience of
junior researchers in solving identified problems that require urgent solutions. Moreover, this
research category is aimed at developing the experience of junior researchers in publishing in
high impact factor journals. Evaluation procedure for medium scale projects will be in the same
way as to the large-scale projects.
3. Small-Scale research Projects
Small-scale research projects are research projects that require minimum of 60,00 Ethiopian Birr
and that will take maximum of one year to complete. The objective of small-scale research
projects is to provide young researchers (academic rank below assistant professor) with
opportunities to practice and develop their research skills while working on a real life problem.
Small-scale research fund are aimed young researchers to start writing papers, make scientific
presentations and generally join the research community. It shall consist of a minimum of three
members. Teams are encouraged to fulfill the senior-junior composition for sharing and learning
experiences. Applications for small scale project fund are evaluated at college level (both for
technical evaluation and presentations), with the supervision of members from the Directorate
office for Research and Graduate Studies. Small-scale research project fund shall be
administered at department level.
Small-scale research projects can be of three types.
1. General small scale projects- These are small scale projects where the applicant
(Principal Investigator) can be any academic staff member. Senior staff (assistant
professor and above) are not encouraged to compete for these funds as they are expected
to draw funds from external sources. However, they can apply as co-PI only to work with
MSc students.
4
2. Young researcher small scale projects- these are small scale projects where the
applicant (PI) can be any academic member with an academic rank below assistant
professor. The aim of this project is to encourage young researchers who show promise in
improving their research experience and publication profile.
3. Female researcher- These are small scale projects where the applicant (PI) is a female
staff member with an academic rank of below assistant professor. The female researcher
projects are aimed at alleviating the very low participation of young female researchers in
research and publication, thereby empowering female researchers. Moreover, this fund
aims to provide young female researchers with opportunities to practice and develop
their research skills while working on a real life problem.
5
1. Large Scale Projects
1. 1. Submission Procedures All proposals should be prepared based on the Mekelle University proposal submission format
and should be prepared as follows. Proposals shall not exceed twenty pages and shall include the
following as annexes
a) CV's of all proponents (maximum of three pages per person). CV's should clearly show
the relevant publications, research and project management experiences of members.
b) A one page description of the roles and responsibilities of each of the proposed co-
applicants.
c) A one page description of the proposed post graduate student research.
1.2.ConstitutionofteamsLarge scale proposals should include at least four team members from at least three inter-
college/institute professionals, including two seniors (Assistant professor and above) and two
young or junior researchers (below assistant professors) and at least two post graduate students
who will be attached to the project for their postgraduate research. .
1.3.EvaluationcommitteeThe office of the Director for Research and Postgraduate Studies will invite relevant colleges/
institutes representing the following broad categories to assign technical experts for constituting
an evaluation team. The following procedures will be followed in the constitution of evaluation
team
1. Team members will be constituted by professionals at a level of assistant professor
and above and shall represent the following six broad areas of specializations
a. College of Social Sciences and Languages (CSSL)
b. College of Law and Governance (CLG)
c. College of Business and Economics (CBE)
d. College of Health Sciences (CHS)
e. College of Veterinary Medicine (CVM)
f. College of Dryland Agriculture and Natural Resources (CoDANR)
6
g. College of Natural and Computational Sciences (CNCS)
h. Ethiopian Institute of Technology - Mekelle (EIT-M)
i. Mekelle Institute of Technology
2. The evaluation team shall work together to evaluate each submitted proposals.
Relevant professionals might give explanations, about technicalities of a proposal, to
those not directly versed in the subject matter of a proposal in question. Finally each
and every member of the evaluation committed will give independent marks to each
of the proposals.
3. The evaluation committee shall be chaired by a member of the Research and Post-
Graduate Directorate (RGD) Office. Another member from the RGD office shall act
as a secretary. Both the chair and the secretary shall also give independent marks to
proposals.
4. Final marks for each proposal will be averages of total marks given by each of the
above mentioned nine members both for technical evaluation (Table 1) and public
defence evaluation (Table 2).
1.4.Evaluationprocedureandcriteria
1.4.1.AutomaticexclusioncriteriaProposals will first be screened according the following criteria. Those proposals with the
following limitations shall be excluded from further evaluation
1. Proposals that do not fulfil the formatting and submission requirement mentioned in
section 1.
2. Proposals that fail to fulfil the minimum technical feasibility requirement
3. Proposals whose PI's or member have been funded by any of the Mekelle University
research fund schemes (two years ago, the latest), but fail to show publications (from
these funds) in a reputable peer reviewed journal. Applicants should attach the
publication mentioning the registration number and fund from which such a
publication was produced
7
1.4.2.SpecificevaluationcriteriaThe nine members of the evaluation committee shall give independent marks to each submitted
proposal based on the technical evaluation criteria (Table 1). Proposals from each thematic area
will then be ranked according to their average marks given by each member of the evaluation
committee. Certain number of top ranked proposals (number shall depend on availability of
fund) will be invited for public defence.
During public defence (which will take 15 minutes presentation and ten minutes question and
answers), the evaluation committee will mark each proposal according to the public defence
evaluation criteria given on Table 1.2. Only the PI can do the public presentation. There will be
mandatory questions from the evaluation committee and other members of the audience may also
ask questions.
Finally, the results from the technical evaluation (Table 1.1) and those from the public defence
(Table 1.2) shall be converted into 70% and 30% of the final mark respectively. Certain number
of top ranked proposals (the number shall depend on availability of funds) will then be selected
for funding. Whatever number of proposals is possible to fund, according to the available budget,
any proposal that does not get a minimum 50% of the overall evaluation shall not be considered
for funding.
Evaluation committee will prepare important feedback (generic) to both accepted and rejected
proponents which will be directly communicated directly to the PI's.
Table 1.1. Detailed evaluation criteria and their relative weights
1
Technical feasibility and soundness of Rationale/conceptual framework,
objectives and methods * 20
Originality 8
Objectives and Rationale 5
Methods 7
2
Alignment and relevance to a specific thematic area (will depend on the
year, call etc) 5
8
3
Contribution and relevance to achieving climate resilient green economy,
reducing vulnerability to climate change, and contribution to climate
change adaptation and mitigation 4
4
Contributions and relevance for achieving good governance and social
equity 4
5
Contribution and relevance to improving food security for rural and urban
communities 4
6
Contributions, relevance and opportunities for involving and benefiting
civil society organizations, Industry and the public 4
7
Contribution and relevance towards enhancing the use of emerging
technologies (Bio-technology, ICT, nano-technology) 5
8
Prospects and opportunities for producing marketable output and fostering
entrepreneurship and commercialization 4
9
Prospects and contributions towards generation of new knowledge,
technology, ideas and build new capacity (attitude, skill knowledge) 5
10
Technical (subject matter), project management and publication experience
of PI and co-PI's (to be evaluated from submitted CV's) 8
Technical (subject matter) 2
Project management skills and experiences 3
Publications (numbers relative to opportunities, relevance of publications) 3
11
Prospects and opportunities for improving urban and rural employment
specially to disadvantages communities (landless youth, women,
pastoralists minorities etc), equitable development 4
12
Prospects and plan for producing scientific publications in high impact
factor peer reviewed international journals, policy and media briefs 5
13
Prospects and opportunities for involving graduate and undergraduate
student researchers 3
14
Opportunities and plans for involving post and undergraduate student
researchers 5
15 Opportunities and plans for using and upgrading Mekelle University's 5
9
research infrastructure and equipment
16 Language, formatting and presentation quality of the proposal 5
17 Matching of technical, financial and time plans 5
18
Affirmative action for involvement of female researchers (5 for >50%, 3
for 30%-50%, 2 for <30%, and 0 for 0%) 5%
Total (100%) 100
* Proposals that fail to fulfil the minimum 15% technical feasibility will be automatically
excluded from further evaluation
Table1.2. Evaluation criteria for public defense
SN Criteria Mark (%)
1 Explanation 10%
Articulatin of the problem 5
Articulation and explanation of proposed solution 5
2 Effort and devotion 15%
Confidence 5
Reflected commitment 5
Reflected effort in reviewing all related knowldege 5
3 Technical competence 15%
Grasp of technical concepts and ideas 5
Understanding local dimensions 5
Response to questions 5
4 Presentation 10%
Presentation style 3Time management 2
language 5
10
2.Medium‐ScaleProjects
2.1.SubmissionProceduresAll proposals should be prepared based on the Mekelle University proposal submission format
and should be prepared as follows. Proposals shall not exceed twenty pages and shall include the
following as annexes. Submissions will be made on the University e-research portal and will be
approved by the College Research and Community Services Council.
a) CV's of all proponents (maximum of three pages per person). CV's should clearly show
the relevant publications, research and project management experiences of members.
b) A one page description of the roles and responsibilities of each of the proposed co-
applicants.
c) A one page description of the proposed post graduate student research-
2.2.ConstitutionofteamsMedium scale proposals should include at least four team members from at least three inter-
college/institute professionals, including two seniors (Assistant professor and above) and two
young or junior researchers (below assistant professors) and at least two post graduate students
who will be attached to the project for their postgraduate research.
2.3.Evaluationcommitteethe evaluation of medium scale projects will be done in the same manner as that of large-scle
projects
2.4.Evaluationprocedureandcriteria
2.4.1.AutomaticexclusioncriteriaProposals will first be screened according the following criteria. Those proposals with the
following limitations shall be excluded from further evaluation
Proposals will first be screened according the following criteria. Those proposals with the
following limitations shall be excluded from further evaluation
1. Proposals that do not fulfil the formatting and submission requirement mentioned in
section 1.
2. Proposals that fail to fulfil the minimum technical feasibility requirement
11
3. Proposals whose PI's have been funded by any of the Mekelle University research
fund schemes (two years ago, the latest), but fail to show publications (from these
funds) in a reputable peer reviewed journal. Applicants should attach the publication
mentioning the registration number and fund from which such a publication was
produced
2.4.3SpecificevaluationcriteriaThe members of the evaluation committee, chaired by the college/institute CRCS head will give
independent marks to each of the proposals according to criteria in Table 2.1. Finally shortlisted
proposals with technical evaluation marks above 50% will be invited to a college level public
defence.
During the public defence, only PI's shall present a 20 minute presentation of their proposals
which will be followed by ten minute question and answer session. During the public defence,
every member of the evaluation committee shall give independent marks based on criteria listed
in Table 2.2
Finally, the results from the technical evaluation (Table 2.1) and those from the public defence
(Table 2.2) shall be converted into 70% and 30% of the final mark respectively. Certain top
ranked proposals (number will depend on the amount of fund) be selected for funding. Final
evaluation results of each of the proposals, shall be sent in an official letter from the office of
CRCS to the Office of Research and Post Graduate Director. Minutes of the evaluation
committee shall be attached to this letter.
Finally, based on results obtained from all colleges and institutes, the office of Research and
Postgraduate Directorate, shall decided on a list of proposals to be funded.
Table 2.1. Detailed evaluation criteria and their relative weights
1
Technical feasibility and soundness of Rationale/conceptual framework,
objectives and methods * 20
Originality 8
Objectives and Rationale 5
Methods 7
12
2 Alignment and relevance to a thematic area specified in a call 15
3
Relevance to any of the national policies for instance, currently to GTP-II
(climate change adaptation and mitigation, enhancement of use of
emerging technologies and achievement of good governance) 5
4
Contribution and relevance to improving food security for rural and urban
communities 5
5
Prospects and opportunities for producing marketable output and fostering
entrepreneurship and commercialization 7
6
Prospects and contributions towards generation of new knowledge,
technology, ideas and build new capacity (attitude, skill knowledge) 10
7
Technical (subject matter), project management and publication experience
of PI and co-PI's (to be evaluated from submitted CV's) 8
Technical (subject matter) 2
Project management skills and experiences 3
Publications (numbers relative to opportunities, relevance of publications) 3
8
Prospects and plan for producing scientific publications in high impact
factor peer reviewed international journals, policy and media briefs 5
9
Opportunities and plans for involving post and undergraduate student
researchers 5
10
Opportunities and plans for using and upgrading Mekelle University's
research infrastructure and equipment 5
11 Language, formatting and presentation quality of the proposal 5
12 Matching of technical, financial and time plans 5
13
Affirmative action for involvement of female researchers (5 for >50%, 3
for 30%-50%, 2 for <30%, and 0 for 0%) 5
Total (100%) 100
* Proposals that fail to fulfil the minimum 15% technical feasibility will be automatically
excluded from further evaluation
Table 2.2. Evaluation criteria for public defense
SN Criteria Mark (%)
13
1 Explanation 10%
Articulatin of the problem 5
Articulation and explanation of proposed solution 5
2 Effort and devotion 15%
Confidence 5
Reflected commitment 5
Reflected effort in reviewing all related knowldege 5
3 Technical competence 15%
Grasp of technical concepts and ideas 5
Understanding local dimensions 5
Response to questions 5
4 Presentation 10%
Presentation style 3Time management 2
language 5
14
3.Small‐ScaleProjects
3.1.SubmissionProceduresAll proposals should be prepared based on the Mekelle University proposal submission format
and should be prepared as follows. Proposals shall not exceed twenty pages and shall include the
following as annexes
a) A one page description of the roles and responsibilities of each of the proposed co-
applicants, including part of the project objectives which will be dealt by the postgraduate
student.
3.2.ConstitutionofteamsThe small-scale researches shall be of three types namely 1). General small scale-where the
applicant can be any academic staff, 2) Young researcher small-scale fund- is reserved for young
academic staff (below assistant professor) and 3) Female researcher small-scale fund- which is
reserved for only young female researchers (below assistant professors). The required team
composition for each proposal types is as follows
1. General small scale- one PI, with at least two co-PI's (one of whom a senior or junior
mentor) and an additional one postgraduate student
2. Young researcher small scale- a young PI (below assistant professor) with at least two
co-PI's (one of whom a senior or junior mentor) and an additional one postgraduate
student
3. Female researcher small scale- a young female PI (below assistant professor) with at least
two co-PI's (one of whom a senior or junior mentor) and an additional one postgraduate
student. Except for the junior or senior mentor, all the co-PI's and the postgraduate
students should be females.
15
3.3.Evaluationcommittee
3.3.1.EvaluationcommitteeforGeneralsmall‐scaleprojectsThe college/ Institute Research and Community Services Council heads, shall appoint one
evaluator (Assit. Prof. and above) from each of the departments. The team of evaluators chaired
by the college/institute CRCS head, shall work together to evaluate each submitted proposals.
Relevant professionals might give explanations, about technicalities of a proposal, to those not
directly versed in the subject matter of a proposal in question
3.3.2.Evaluationcommitteeforyoungresearchersmall‐scaleprojectsThese are similar to the general small scale projects in terms of the evaluation criteria, however,
since this specific program is aimed at supporting young researchers who are showing promise in
their research endeavors, there will no be quotas to each college, rather applications will be
evaluated at center level using the procedures followed for large-scale project evaluation
(Section 1.3)
3.2.3.Evaluationcommitteeforfemaleresearchersmall‐scaleprojectsThese are similar to the young researcher small sclae projects, but only focused on supporting
young female researchers who are showing promise in their research endeavors. The evaluation
of these projects will be similar to the procedures used for large-scale projects (Section 1.3)
3.3.Evaluationprocedureandcriteria
3.3.1.AutomaticexclusioncriteriaProposals will first be screened according the following criteria. Those proposals with the
following limitations shall be excluded from further evaluation
1. Proposals that do not fulfil the formatting and submission requirement mentioned in
section 1.
2. Proposals that fail to fulfil the minimum technical feasibility requirement
3. Proposals whose PI's have been funded by any of the Mekelle University research
fund schemes (two years ago, the latest), but fail to show publications (from these
funds) in a reputable peer reviewed journal. Applicants should attach the publication
16
mentioning the registration number and fund from which such a publication was
produced
3.4.Specificevaluationcriteria
3.4.1.Generalsmall‐ScaleandYoungresearchersmallscaleprojectsThe members of the evaluation committee, chaired by the will give independent marks to each of
the proposals according to criteria in Table 5. Finally shortlisted proposals with technical
evaluation marks above 50% will be invited to a college level public defence.
During the public defence, only PI's shall present a 20 minute presentation of their proposals
which will be followed by ten minute question and answer session. During the public defence,
every member of the evaluation committee shall give independent marks based on criteria listed
in Table 6
Finally, the results from the technical evaluation and those from the public defence shall be
converted into 70% and 30% of the final mark respectively.
The number of proposals to be funded from each college in this research category will depend on
the amount of research budget available for each college or institute. Therefore, the evaluation
committee will decide on how many of the top ranked proposals should be funded.
Final list of funded proposals, shall be sent in an official letter from the office of CRCS to the
Office of Research and Post Graduate Director. Minutes of the evaluation committee shall be
attached to this letter.
Table 3.1. Detailed evaluation criteria for general small scale and young researcher small scale
project proposals
Criteria Mark%
1
Technical feasibility and soundness of Rationale/conceptual framework,
objectives and methods * 15
Originality 5
Objectives and Rationale 5
17
Methods 10
2 Alignment and relevance to a thematic area specified in a call 15
3
Contribution and relevance to improving food security for rural and urban
communities 5
4
Contribution and relevance to any of the national GTP-II issues (climate
change, emerging technologies and good-governance) 5
5
Prospects and opportunities for producing marketable output and fostering
entrepreneurship and commercialization 5
6
Prospects and contributions towards building new capacity (attitude, skill
knowledge) e.g. the use of a software, tool, equipment etc 15
7
Prospects and plan for producing scientific publications, conference
proceedings and other forms of publications 10
8 Opportunities and plans for involving ndergraduate student researchers 5
9
Opportunities and plans for using and upgrading Mekelle University's
research infrastructure and equipment 5
10 Language, formatting and presentation quality of the proposal 5
11 Matching of technical, financial and time plans 5
12
Affirmative action for involvement of female researchers (5 for >50%, 3 for
30%-50%, 2 for <30%, and 0 for 0%) 5
Total (100%) 100
* Proposals that fail to fulfil the minimum 10% technical feasibility will be automatically
excluded from further evaluation
Table 3.2. Public defence evaluation criteria
SN Criteria Mark (%)
1 Explanation 10%
Articulation of the problem 5
Articulation and explanation of proposed solution 5
2 Effort and devotion 15%
18
Confidence 5
Reflected commitment 5
Reflected effort in reviewing all related knowledge 5
3 Technical competence 15%
Grasp of technical concepts and ideas 5
Understanding local dimensions 5
Response to questions 5
4 Presentation 10%
Presentation style 3Time management 2
language 5
3.4.2.Specificevaluationcriteriaforfemaleresearcherssmall‐scaleproposalsSince the objectives and the targeted groups of this scheme are different from the above
mentioned categories, there will be different set of specific criteria that will be used for
evaluation of female researcher small-scale criteria. These criteria are given in Table 3.3. for
technical evaluation and Table 3.4 for public defense.
Table 3.3. Detailed technical evaluation criteria for female researchers small-scale proposals
Criteria Mark%
1
Technical feasibility and soundness of Rationale/conceptual framework,
objectives and methods * 15
Originality 5
Objectives and Rationale 5
Methods 10
2 Alignment and relevance to a thematic area specified in a call 15
3
Contribution and relevance to improving food security for rural and urban
communities 5
4
Contribution and relevance to any of the national GTP-II issues (climate
change, emerging technologies and good-governance) 5
19
5
Prospects and opportunities for producing marketable output and fostering
entrepreneurship and commercialization 5
6
Prospects and contributions towards building new capacity (attitude, skill
knowledge) e.g. the use of a software, tool, equipment etc 15
7
Prospects and plan for producing scientific publications, conference
proceedings and other forms of publications 10
8 Opportunities and plans for involving undergraduate student researchers 5
9
Opportunities and plans for using and upgrading Mekelle University's
research infrastructure and equipment 5
10 Language, formatting and presentation quality of the proposal 5
11 Matching of technical, financial and time plans 5
12 Prospectus and opportunities for gender empowerment 5
Total (100%) 100
* Proposals that fail to fulfil the minimum 10% technical feasibility will be automatically
excluded from further evaluation
Table 3.4. . Evaluation criteria for public defense
SN Criteria Mark (%)
1 Explanation 10%
Articulation of the problem 5
Articulation and explanation of proposed solution 5
2 Effort and devotion 15%
Confidence 5
Reflected commitment 5
Reflected effort in reviewing all related knowledge 5
3 Technical competence 15%
Grasp of technical concepts and ideas 5
Understanding local dimensions 5
Response to questions 5
4 Presentation 10%
Presentation style 3Time management 2
20
language 5