Grading system

• Two major categories– 20% Class attendance, participation, assignment submissions– 40% Quality and preparation of rough draft: Joshi– 40% Your peers and Professors

A 95-100% Outstanding: must fundAB 90-94 Excellent: high priorityB 85-89 Very Good: medium priorityBC 80-84 Good: low priorityC 75-79 OK: Some meritCD 70-74 BAD: Why did you waste my

time?D 65-69 Very BAD: Do not fund!

So far…• Melissa Roberts

Chee Hui Lee Danilelle Dusold Eric Minner Ee Lim Tan Eric Winder Kate Tay Lindsey Tuominen Louis Paladino Matthew Barron Meagan Harless Tara WaybrantWilfred Previant

• 6 have not given the rough draft which was originally due on March 6th

• Today is March 26, 2008

Review Process

Who decides the fate of your proposal?How?

Checklist

• Cover page• Table of content• Summary• Project description• References cited• Biographical sketches• Budget and budget justification• Current and pending support• Facilities and equipment

Put summary, PD and references together in the same sequence in a single file, paginate it and convert it into a pdf file. Email me one pdf file with your last name as file name by April 9, 2008

Now, let us presume you have submitted the proposal via Fastlane through your SRO

Who looks at it and how deep?

• Administrative assistants• Format compliance• eligibility• Complete set of documents• Assign to area/manager as you requested• Prepare list of individuals that are in conflict with you

• Program manager (also a Professor at some U)– Reads title and abstract– Confirms the program to which your proposal is

assigned or shifts to other programs– Decides upon the panel manager/panel members– NSF has no separate panel manager. Program

manager does the job as panel manager!

Panel manager

• Who is the panel manager? (Roll of dice)– Well-established, successful senior professor at some

renowned University– Works for one-two years in a row– Is happy with his/her own career, well connected– Has vision for the future research in the topic– Reads ALL abstracts and first pages of ALL proposals

(at least).

• Decides in consultation with program manager who could be panel members and ad hoc reviewers

Panel members

• Based on the topics of proposals submitted to a program, 10-12 panel members are invited.

• Some are the same as last year’s panel• Mostly they do not compete for funding in the

same year or have to leave discussion if any direct conflict of interest exists.

• Primary and secondary panel members• Each reads about 10-12 proposals in depth and

prepares reports for panel and submits electronically.

• Gets input from ad hoc e-reviewers

Ad hoc reviewers• About 6-10 specialists are

invited to review your proposal.

• They should have no direct conflict of interest.

• Done electronically via fastlane

• Use two major criteria– Intellectual merit– Broader impact

• They rate the proposal as excellent (e), very good (vg), good (g), fair (f), poor (p)

Next step…

• Not every ad hoc reviewer responds on time (~2 months)• May have personal bias against or for you or too busy• May be harsh or too mild (depending on where they are on ladder?)• They may be jealous of your success too!! You hope..• About 3-9 reviews get in and put in excel spread sheet• Panel member has to submit review first then they can see the rest• All such reviews are seen by the primary reviewer who makes up

his/her mind about fate of your proposal• Defends or destroys your proposal in the panel meeting• Secondary panel member keeps a check on primary member and

takes notes of panel discussions• Open process on hidden agendas!

After about 3 months after proposal submission

• Panel meets at Washington D.C. for 2 and half days• Each proposal gets less than ~15 minutes of discussion.• Someone has to passionately defend you!• “Wow” science goes at the top• Each proposal gets a rating (or other grades)

– Outstanding, highly meritorious, meritorious, non-competitive

• If your name is on the outstanding side, your chances of success are high and very low if you are in the fourth pile

• Next day or so, panel members go back read proposals again and change ratings, if necessary

Horror stories

• Simple methodology omitted and proposal tanked.• No publications from last grant, proposal rejected!• Panel members had bad experiences with the PI• The PI (postdoc of another well-funded researcher)

almost lost a grant but saved at the last minute by his fax declaring change of job

• Supporting evidence submitted just before the panel meeting did save a dying proposal

• Catch-22 situation! First, asked for preliminary data and then asked why you need more money?

• No clear broader impact statements and proof• Fan club reviewers so not funded

Program manager

• Decides the final ranking of proposals• Two more visiting program managers help• Has some freedom to move within the ranks• Decides how much money could be given• Calls or communicates with the PI• Negotiates what needs to be done and for how

much support $?• Sends declination letters and reviews• Answers your questions

What if you get a grant?

• Do Party but not forever!• You are among the top 5-10% researchers in

your field• Hire people and deliver the goods promised on

time• Publish profusely in high quality journals• Write more grants! Why?

– funding does not last for ever– 10% success rate– distribution of wealth principle

What if you do not get a grant?

• don't cry (OK, cry a little if you feel better)• pick up the pieces of your failed proposal

and restart your “grants writing” engine• get reviewer’s comments, read and get

angry then keep them in a drawer away from your view for a while..

• come back and read reviews again• talk to PM and your mentor/well wishers• resubmit until you succeed

1. Lack of new or original ideas. 2. Diffuse, superficial, or unfocused research plan. 3. Lack of knowledge of published relevant work. 4. Lack of experience in the essential methodology. 5. Uncertainty concerning future directions. 6. Questionable reasoning in experimental approach. 7. Absence of an acceptable rationale. 8. Unrealistically large/small amount of work. 9. Lack of sufficient experimental detail. 10. Uncritical approach.

Top ten reasons why funding is normally not awarded

http://www.hort.purdue.edu/rhodcv/hort652n/ho00005.htm

How will your proposal be evaluated for this FW5850 class?

• You will submit a single pdf file of your proposal to me, your advisor and your peers by email. It will contain only one page of summary, up to 15 pages of description and 3-4 pages of references.

• Please do this latest by April 9th, 2008 (no extensions possible)• Now, you will change your role. You will review (as an ad hoc

reviewer) all the proposals from your group except your own.• You will prepare reviews for each proposal in your group in the

prescribed format (I will email it to you and your advisors) • Bring two hard copies of your reviews to class on April 21st, 2008

and put your name on only one and give it me and the other will be put in respective student's folder

• Your advisors and I may also review them and we will add our reviews to the folder!

• All reviews given to the PI will be anonymous

Evaluation Criteria

• Intellectual merit– How important this proposal is for advancement of knowledge?– Qualification of PI and quality of proposal?– Creative and original concepts? – How well conceived and organized is this activity?– Sufficient resources available for this research?

• Broader impacts– Advance discovery and understanding– Can promote teaching and research integration– Diversity (gender, ethnicity, disability, geographical), if any *– Infrastructure development*– Dissemination of information obtained – What is the benefit to society?

ALL CRITERIA MAY NOT APPLY FOR EACH PROPOSAL!Focus more on the contents (summary, description) than the format!* Not important for your evaluation

Your review• One page• Please try to answer some of these questions

– Is the title concise and attractive?– Did the summary give you sufficient idea about the activity to be undertaken?– Did the first page sell you the project?– Is the idea sufficiently novel?– Will it advance science significantly?– Was the background given adequate (and not excessive)?– Were references cited in the text?– Were you clear about what is known, what is proposed and what will be achieved

by the end of project?– Were all necessary methods appropriately and briefly described?– Were there three-four objectives? Were they stand-alone?– Was proposal a pleasure to read? Were there any grammatical issues?– Was there a time-table? – Are the available facilities and equipment adequate for the project?– Is the reference style used uniform?– Were sufficient number of figures and tables used?– Did you like the project? Do you think it is a worth funding and well prepared

project?

Please do not just say “yes” or “no” but use these questions to build your review

Finally, please rate the proposal as excellent (E), very good (VG), good (G), fair (F) and poor (P). Please also provide suggestions for improvement.

Panel meeting on April 21st and 23rd, 2008

• You change your role again. Now, you are a panel member.

• There will be four panels = 4 peer groups• On April 21st, we will meet in the atrium area.• There will be four separate tables for panel

meeting, one per group• Each proposal will get 15-20 minutes discussion

(a bell will ring every 20 minutes)• You will go to the other table when your own

proposal is being discussed as shown in the next slide.

ENGINEERSBiomedical

Chemistry Biology

Me

Panel reviews

• Each panel member will first state their ratings and then discuss each proposal based on their own written reviews.

• Panel will prepare a one page report using three major categories: summary, strength, weakness (2-3 lines each) (see next slide).

• The panel will also give the final rating as Excellent (E), very good (VG), good (G), fair (F), poor (P)

• The panel report will be given to me at the end of your discussion and you will select a panel leader from the members within your own group to read the summary to whole class on April 23rd, the last day of this class.

Panel evaluation

• What is the main research topic of the proposal? (One-two sentences)

• What is the major strength of this proposal?

• What is the major weakness of this proposal (if any)?

• Summary statement (one or two lines)

Panel presentation (April 23rd 2008)

• You will select one leader from your group who will read all reports from your group after panel meeting to the whole class

• Each proposal will get ~two minutes

• Each report will have a specific final rating.

• Then you will enjoy your summer vacation!

Any questions?