reviewer orientation - ons...
TRANSCRIPT
Assigned Applications• Reviewers will “identify” the applications they would be able to review, based
on the match with their content and/or methods expertise, identifying any
conflicts of interest. The total number of applications per reviewer will
depend on the number of applications received. A Reviewer Proposal
Identification Worksheet, will be submitted by each reviewer with the
proposals identified for their review. The completed worksheet will be sent to
the ONS Foundation Research Department.
• The chair will finalize all assignments, ensuring that each application has a
primary, secondary and collateral reviewer.
• Assigned applications will be reviewed through a link to the online review
area.
• All components of the assigned applications should be read
• Major strengths and weaknesses should be identified for each criteria, using
the Reviewer Comment-Scoring Form
• A preliminary overall impact score is to be assigned as well as scores to
each of the five core criteria on the Reviewer Comment-Scoring form
Reviewer Comment-Scoring Form • A Reviewer Comment-Scoring Form will be used by the three
assigned reviewers (primary, secondary and collateral) for each
application
• Emails will be sent with the link to the online review website, the
reviewer’s passwords and other review materials needed.
• All reviewers should use the Reviewer Comment-Scoring form
when reviewing the applications assigned.
• The completed Reviewer Comment-Scoring Forms are to be
emailed to the ONS Foundation Research Department by a
designated date. These forms must be submitted on time as
they will be combined and emailed to the review team in
preparation of the review call.
Preparation of Critique Comments
• Use bulleted points to make succinct, focused
comments
• Short narratives may occasionally be appropriate,
but should be rare
• Focus on major strengths and weaknesses (ones
that impacted your overall rating of the application)
• Limit text to ¼ page per criterion
Reviewer Comment-Scoring
Form Layout
Criteria – Description of scoring criteria (i.e., significance,
investigators, etc.)
Score: ___
Strengths:
- Bulleted Comments
Weaknesses:
- Bulleted Comments
Scoring DescriptionsImpact Score Descriptor Additional Guidance on Strengths/Weaknesses
High
1 Exceptional Exceptionally strong with essentially no weaknesses
2 Outstanding Extremely strong with negligible weaknesses
3 Excellent Very strong with only some minor weaknesses
Medium
4 Very Good Strong but with numerous minor weaknesses
5 Good Strong but with at least one moderate weakness
6 Satisfactory Some strengths but also some moderate weaknesses
Low
7 Fair Some strengths but with at least one major weakness
8 Marginal A few strengths and a few major weaknesses
9 Poor Very few strengths and numerous major weaknesses
Minor Weakness: An easily addressable weakness that does not substantially lessen impact
Moderate Weakness: A weakness that lessens impact
Major Weakness: A weakness that severely limits impact
Scoring Criteria1. Significance - Does the project address an important problem or a
critical barrier to progress in the field? If the aims of the project are
achieved, how will scientific knowledge, technical capability, and/or
clinical practice be improved? How will successful completion of the
aims change the concepts, methods, technologies, treatments, services,
or preventative interventions that drive this field?
2. Investigator(s) - Are the PIs, collaborators, and other researchers well
suited to the project? If Early Stage Investigators or New Investigators,
or in the early stages of independent careers, do they have appropriate
experience and training? If established, have they demonstrated an
ongoing record of accomplishments that have advanced their field(s)? If
the project is collaborative or multi-PD/PI, do the investigators have
complementary and integrated expertise; are their leadership approach,
governance and organizational structure appropriate for the project?
Scoring Criteria (Continued)3. Innovation - Does the application challenge and seek to
shift current research or clinical practice paradigms by
utilizing novel theoretical concepts, approaches or
methodologies, instrumentation, or interventions? Are
the concepts, approaches or methodologies,
instrumentation, or interventions novel to one field of
research or novel in a broad sense? Is a refinement,
improvement, or new application of theoretical
concepts, approaches or methodologies,
instrumentation, or interventions proposed?
Scoring Criteria (Continued)4. Approach - Are the overall purpose, aims, strategy, methodology,
and analyses well-reasoned and appropriate to accomplish the
specific aims of the project? Are potential problems, alternative
strategies, and benchmarks for success presented? If the project is
in the early stages of development, will the strategy establish
feasibility and will particularly risky aspects be managed?
If the project involves clinical research, are the plans for
1) protection of human subjects from research risks, and 2)
inclusion of minorities and members of both sexes/genders, as well
as the inclusion of children, justified in terms of the scientific goals
and research strategy proposed?
Scoring Criteria (Continued)
5. Environment - Will the scientific environment in which the
work will be done contribute to the probability of success?
Are the institutional support, equipment and other physical
resources available to the investigators adequate for the
project proposed? Will the project benefit from unique
features of the scientific environment, subject populations,
or collaborative arrangements?
Where criteria are addressed
• Significance: Project narrative
• Investigator(s): Biographical sketch(es)
• Innovation: Innovation field, project
narrative and overall
• Environment: Facilities and resources
• Approach• Project narrative
• Protection of human subjects or animals
• Women and minority inclusion
• References
• Timetable
• Letters of support
• Instruments
• Consent form
• Miscellaneous field
• Project NarrativeLength: >$25,000 (12 pages) / $25,000 or less (6 pages)
• Purpose and specific aims
• Significance, framework and review of literature
• Preliminary work
• Methods and design
Design
Sample and settings
Experimental variables, if applicable
Instruments
Data collection schedule and procedures
Data analysis and interpretation
Additional Review Criteria
• As applicable, reviewers will consider additional items in
the determination of scientific and technical merit, but will
not give separate scores for these items.
• Responses for Protections for Human Subjects and/or
Vertebrate Animals are required for all applications
• A response for Inclusion of Women, Minorities and
Children is required for applications proposing Human
Subjects Research.
Additional Review Criteria
Protection for Human Subjects
Protections for Human Subjects – Did the application
describe how informed consent will be obtained and the
steps taken to protect participants’ rights or the welfare of
animals? Did the application identify any potential risks
associated with participation in the project?
Comments –
Data and Safety Monitoring Plan (Clinical trials only)
Comments -
Additional Review Criteria
Inclusion of Women, Minorities and Children
Inclusion of Women, Minorities and Children (Applicable Only for
Human Subjects Research) – Did the application address the inclusion
of women, minorities and children in developing a research design
appropriate to the scientific objectives of the study. Inclusion is
required unless a clear and compelling rationale shows that inclusion is
inappropriate with the respect to the health of the subjects or that
inclusion is inappropriate for the purpose of the study. Did the
application provide information on the composition of the proposed
study population in terms of sex/gender and racial/ethnic group and
provide a rationale for selection of such subjects in terms of the
scientific objectives and proposed study design.
Comments -
Additional Review Criteria
Vertebrate AnimalsVertebrate Animals – Did the application address the involvement of
live vertebrate animals as part of the scientific assessment according to
the following five points: 1) proposed use of the animals, and species,
strains, ages, sex, and numbers to be used; 2) justifications for the use
of animals and for the appropriateness of the species and numbers
proposed; 3) adequacy of veterinary care; 4) procedures for limiting
discomfort, distress, pain and injury to that which is unavoidable in the
conduct of scientifically sound research including the use of analgesic,
anesthetic, and tranquilizing drugs and/or comfortable restraining
devices; and 5) methods of euthanasia and reason for selection if not
consistent with the AVMA Guidelines on Euthanasia.
Comments (Required unless not applicable)
Additional Criteria
Budget and Period of Support(Not part of the scientific review)
Budget and Period of Support – Is the budget and the
requested period of support fully justified and reasonable in
relation to the proposed research. For more details, please
see Budget Information on the application.
Recommended budget modifications or possible overlap
identification -
Overall Impact Reviewers will provide an overall impact score to reflect their assessment of the
likelihood for the project to exert a sustained, powerful influence on the research
field(s) including consideration of the following five scored review criteria, and
additional review criteria. An application does not need to be strong in all categories
to be judged likely to have major scientific impact.Proposal #: _____
Principal Investigator: ________________
Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a
sustained, powerful influence on the research field(s) involved in consideration of the following five scored review criteria,
and additional review criteria. An application does not need to be strong in all categories to be judged likely to have major
scientific impact.
Overall Impact - After considering all of the review criteria, briefly summarize the significant
strengths and weaknesses of the application and state the likelihood of the project to exert a
sustained powerful influence on the field.
Preliminary Score: _____
Final Score: _____
Strengths –
Weaknesses -
Overall Impact ScoringImpact Score Descriptor Additional Guidance on Strengths/Weaknesses
High
1 Exceptional Exceptionally strong with essentially no weaknesses
2 Outstanding Extremely strong with negligible weaknesses
3 Excellent Very strong with only some minor weaknesses
Medium
4 Very Good Strong but with numerous minor weaknesses
5 Good Strong but with at least one moderate weakness
6 Satisfactory Some strengths but also some moderate weaknesses
Low
7 Fair Some strengths but with at least one major weakness
8 Marginal A few strengths and a few major weaknesses
9 Poor Very few strengths and numerous major weaknesses
Minor Weakness: An easily addressable weakness that does not substantially lessen impact
Moderate Weakness: A weakness that lessens impact
Major Weakness: A weakness that severely limits impact
Additional Comments to Applicant
Reviewers may provide guidance to the applicant or
recommend against resubmission without fundamental
revision.
Additional Comments to Applicant (Optional) – Please
provide any additional guidance to the application or recommend
against resubmission without fundamental revision
Online Overall Impact Scores
• After completing your review of each
application you will need to enter a
preliminary impact score in the online
review system by clicking on one of
the numerical ratings at the bottom of
each application reviewed
• Once you have selected your
preliminary impact score, click
“Submit Ratings” to finalize your
score
My Application Score
0 / 90.0
My Ratings
Overall Impact Score1 = Exceptional (High impact)
2 = Outstanding (High impact)
3 = Excellent (High impact)
4 = Very Good (Medium impact)
5 = Good (Medium impact)
6 = Satisfactory (Medium impact)
7 = Fair (Low impact)
8 = Marginal (Low impact)
9 = Poor (Low impact)
1 2 3 4 5 6 7 8 9
O O O O O O O O O
Prior to Review Conference Call• Completed worksheets are to be emailed to
[email protected] by the designated due date
• Primary, secondary and collateral comment-scoring forms are
combined by the ONS Foundation Research Department for each
application
• The combined forms for the applications each reviewer has been
assigned to review are emailed several days before the call for your
preparation for the call
• Primary, secondary and collateral preliminary impact scores are
averaged and the Preliminary Mean Scores are ranked and emailed
to all reviewers
• The Grant Review Team Chair contacts any reviewers with variant
scores and determines any proposals that are non-competitive and do
not need to be discussed
Logging In with the Temporary Password Provided
• Access the website (https://admin.closerware.net/gm_onsf/
page.jsp?pagename=evaluatorarea )
• Enter the username
provided in the email
• Enter the password
provided in the email• (please do not change the
password provided)
• Click “Login”
“My Applications”
• All of the applications you
have been assigned to review
will appear below the
“Applications Pending your
Review” area
• Click on the applicant’s name
to view the application
• Applications will remain in this
area until a score is assigned
My
Applications
Pending
Applications
Approved
Applications
Research Grants Program
(RE##): Type of Grant
Applicant Name (Username)
Applicant Name (Username)
Applicant Name (Username)
Admin
© Copyright 2003-2010 Closerware, LLC. All rights reserved.
Evaluator Extranet
Applications Pending
Your Review
Applications you have
reviewed
“Pending Applications”
• As you complete your review
of each application and the
score is assigned, the
application will move to the
“Pending Applications” area
• To view these applications
again, simply click on the
applicant name in that area to
access that application again,
as needed
My
Applications
Pending
Applications
Approved
Applications
Applicant Name (Username) Research Grants Program
(RE##): Type of Grant
Admin
© Copyright 2003-2010 Closerware, LLC. All rights reserved.
Evaluator Extranet
Application Label Grant: Grant Opportunity
Accessing Applications• The applicant name
and application
number will appear in
the blue area followed
by:
– Type of Grant
– User Queue Number
– Application Status
(Pending)
– Research Team
Evaluator Extranet
•Grant Opportunity Main >
•Grant Application Main >
•Grant Application
* Administrative Index
* Grant Opportunity Main
* Grant Application Main
* Grant Application
Grant Application
Created: 9/6/10 5:29:36 PM EDT
Grant Opportunity: Research Grants Program (RE##): Type of Grant
User Queue: #1
Application Status: Pending
Research Grants: Research Team
Research Team: Name Credentials Institution Res Team Role
Name Credentials Institution PI
Name Credentials Institution Co-investigator
Applicant Name (Username)
Conflict of Interest Review
• Check the names shown in the “Research Team”
area of the Applicant Login Worksheet to see if you
have any conflicts of interest with anyone on the
team or their institution
• If you identify anyone or any institution that you may
be perceived as having or have a conflict with,
notify [email protected] immediately
Uploaded Documents
Abstract: Abstract.pdf
Project Narrative: Project Narrative.pdf
Reference List: Reference List.pdf
Timetable: Timetable.pdf
Human Protection
Education Certificates
of Completion: Certificates.pdf
Support Letters: Letters of Support.pdf
Biographical Sketches: Biosketches.pdf
Instruments: Instruments.pdf
Itemized Budget: Budget Worksheet.pdf
Budget Justification: Justification.pdf
• Documents uploaded by the
applicant will appear as links
which can be opened and either
reviewed online or printed
• The uploads will be in PDF
formats
• The system will not allow the
applicant to upload more than
one document in any area (i.e.,
letters of support, biosketches,
instruments, etc.) so the
applicants have been asked to
scan the multiple documents into
one document and upload in a
PDF format.
Resubmission / Special Grants
Is this application a resubmission
from a previous ONS Foundation
research grant cycle:– Year of Grant
– Type of previous grant
• Resubmission – from a previously
non-funded ONS Foundation
research grant cycle - year and type
of grant will be shown
– Resubmission Cover Letter-uploaded
– The previous reviewer’s critiques will
have been added to the end of the
resubmission cover letter by the
Research Department.
• Special Grants ($25,000 or less
Only) – Applicant will indicate if they are
applying for one of the ONS Foundation special
grants
Special Grants ($25,000 or less Only):
– ONS/STTI Research Grant
– Trish Greene Pain Research Grant
– Certification, Education and Outcomes
Grant
Project Title and PI Information
Title of Project:
PI First Name:
PI Last Name:
PI Credentials:
PI Title:
PI Institution:
• Following the project title is
the principal investigator’s
name, title and institution
• Review both the research
team members (at the top
of the application page) as
well as the PI information to
be sure you do not have a
conflict of interest with this
applicant
IRB Submission
• Human Subjects?:
Yes / No / Pending
• Animal Subjects?:
Yes / No / Pending
• IRB to be submitted
upon notification of
funding?: Yes
• If human or animal subjects will be
included, the applicant must obtain IRB
approval either before or after funding
notification
• If submitted or approved, IRB Assurance
ID number and proof of approval or
submission will be uploaded in a PDF
format
• If submitting upon notification of funding,
the applicant will indicate this
Protection of Human Subjects and
Inclusion of Women and Minorities• Protection of Human Subjects or
Animals Used for Research:– How informed consent will be obtained
– Steps taken to protect participant’s
rights or the welfare of animals
– Potential risks associated with
participation in the project
• Women and Minority Inclusion in
Clinical Research:– Information on the composition of the
proposed population in terms of
sex/gender and racial/ethnic group
– Rationale for selection of subjects in
terms of scientific objectives and design
• The applicant is asked to
submit 1-2 paragraphs
on the following areas
– Protection of human
subjects or animals used
for research
– Women and minority
inclusion in clinical
research
There are currently no comments.
Innovation and Facilities and Resources
• Innovation– How the project challenges existing
paradigms or clinical practice
– How the project addresses an innovative
hypothesis or critical barrier to progress in the
field
– If applicable, how it develops or employs
novel concepts, approaches, methodologies,
tools or technologies in the area
• Facilities and Resources (Environment):
– Facilities and resources available to carry out
the project at all research sites such as:
• Computers
• Statistical and data management support,
• Office space
• Equipment, etc.
• The applicant is asked
to submit 1-2
paragraphs on the
following areas
– Innovation
– Facilities and
Resources
Implications for Practice
Implications for Practice and Research:
– Implications for oncology nursing
practice
– Identify future research that may
develop from this project
– Describe how this project will provide
the groundwork for seeking additional
funding in the future
– Describe when and how the study
findings will be disseminated
• The applicant is asked
to submit 1-2
paragraphs on the
Implications for
Practice and Research
Online Overall Impact Scores
• After completing your review of each
application you will need to enter a
preliminary impact score in the online
review system by clicking on one of
the numerical ratings at the bottom of
each application reviewed
• Once you have selected your
preliminary impact score, click
“Submit Ratings” to finalize your
score
My Application Score
0 / 90.0
My Ratings
Overall Impact Score1 = Exceptional (High impact)
2 = Outstanding (High impact)
3 = Excellent (High impact)
4 = Very Good (Medium impact)
5 = Good (Medium impact)
6 = Satisfactory (Medium impact)
7 = Fair (Low impact)
8 = Marginal (Low impact)
9 = Poor (Low impact)
1 2 3 4 5 6 7 8 9
O O O O O O O O O
Review Call Discussion
1) Primary, secondary, and collateral reviewers will begin by
stating their preliminary impact scores for the proposal.
2) Primary reviewer provides a brief summary of the project
3) Primary reviewer presents a brief 2 –3 minute critique
highlighting strengths and weaknesses in each criteria.
4) Next, Secondary Reviewer adds additional strengths and
weaknesses.
5) Collateral reviewer follows with any additional comments
not previously made.
Review Call Discussion
6) After the critiques have been presented, the review
team will discuss the proposal for up to 2 – 3 minutes.
7) Primary, secondary and collateral reviewers are then
asked if their preliminary impact scores stand as
submitted
8) All remaining reviewers will assign a final Overall
Impact Score to the application in private on the
online application review page.
9) Any concerns about the budget are discussed at this
time.
Comment Revisions
• You may have changes to your Reviewer Comment-
Scoring form after the conference call. Please make
these changes and e-mail the revised form to
[email protected] within 48 hours of the
conference call.
• If you do not have changes to your critiques, please
e-mail the ONS Foundation Research Department to let
them know.
• The chair or co-chair will read the critiques. You may be
contacted if there are any questions.
Summary Statements
• Overall impact scores of applications will be the
mean of scores entered by all eligible reviewers,
multiplied by 10
• Final scores will range from 10-90, in whole
numbers
• The primary reviewer’s summary for ALL
applications will include the criterion scores and
critiques posted by assigned reviewers
Recipient Selection• After the review all of the proposals, the group’s average
scores will be rank ordered and used to guide the
assignment of grant awards.
• The ONS Foundation Executive Director, the ONS
Research Director and the review team chair and co-chair
will identify the recommended impact score funding range
and the applications within that range.
• The group’s recommendations are presented to the ONS
Foundation Board of Trustees for their approval.
• Once the award and non-award letters are mailed, the list
of funded proposals will be emailed to you.