broader impacts at nsf proposal and award policies and procedures guide (pappg) (9 bi categories)...
TRANSCRIPT
Suzi Iacono, HeadOffice of Integrative Activities
U MichiganMarch 15, 2018
Broader Impacts at NSF
National Science Foundation’s mission
“To promote the progress of science; to advance the national health, prosperity, and welfare; to
secure the national defense...”
$7.5BFY 2017 Enacted
93%funds
research,education and
related activities
50,000proposalsevaluated
11,000awardsfunded
2,000NSF-fundedinstitutions
359,000people NSFsupported
$1.2BSTEM
education
231NSF-funded
NobelPrize
winners
$100Mto seed
public/privatepartnerships
NSF by the numbers
Examples of NSF’s significant investments
U
3-D PRINTINGBREAKTHROUGH
LIGOGRAVITATIONAL WAVES
SBIR
FOUNDATION FORTHE INTERNET
BARCODESPOPULARIZED
EARLY WEBSEARCH
HL TAUDISCOVERY
AUTONOMOUS CARSOFTWARE
NSF Merit Review Criteria
• Intellectual Merit: The Intellectual Merit criterion encompasses the potential to advance knowledge; and
• Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.
NSF Strategic Plan 2018 - 2022
• STRATEGIC GOAL 2 Advance the capability of the Nation to meet current and future challenges.
• This goal flows from the latter part of the NSF mission statement—”to advance the national health, prosperity, and welfare; to secure the national defense; andfor other purposes.”
• This strategic goal echoes the “broader impacts” merit review criterion that was developed by the National Science Board. Within this strategic goal, NSF also seeks to advance the state of the practice of achieving broader impacts.
A Topic of Great Interest to NSF, NSB, and Congress: All Seeking to Clarify
1974 11 criteria for NSF review 1982 4 criteria (research competence, merit of the
research, utility, and effect on infrastructure)1986 NSF officially utilized “merit” review1996 Intellectual merit and broader impacts1998 Accountability for BI established 2000s “Archived” criteria (5 BI criteria)2010 America COMPETES Reauthorization (8 BI categories)2011 National Science Board report2016 NSF Proposal and Award Policies and Procedures
Guide (PAPPG) (9 BI categories)2017 American Innovation and Competitiveness Act (7 BI
categories)
Broader Impacts CategoriesAICA
The Foundation shall apply a broader impacts review criterion to identify and demonstrate project support of the following goals:
(1) Increasing the economic competitiveness of the United States.(2) Advancing of the health and welfare of the American public.(3) Supporting the national defense of the United States.(4) Enhancing partnerships between academia and industry in the United States.(5) Developing an American STEM workforce that is globally competitive through improved pre-kindergarten through grade 12 STEM education and teacher development, and improved undergraduate STEM education and instruction.(6) Improving public scientific literacy and engagement with science and technology in the United States.(7) Expanding participation of women and individuals from underrepresented groups in STEM.
Broader Impacts CategoriesArchived
• Advance discovery and understanding while promoting teaching, training, and learning, e.g., by training graduate students, mentoring postdoctoral researchers & junior faculty, involving undergraduates in research experiences, & participating in the recruitment, training, & professional development of K-12 mathematics & science teachers.
• Broaden participation of under-represented groups, e.g., by establishing collaborations with students & faculty from institutions & organizations serving women, minorities, & other groups under-represented in STEM.
• Enhance infrastructure for research and education, e.g., by establishing collaborations with researchers in industry & government labs, developing partnerships with international academic institutions & organizations, & building networks of U.S. colleges & universities.
• Broaden dissemination to enhance scientific and technological understanding, e.g., by presenting results of research & education projects in formats useful to students, scientists & engineers, members of Congress, teachers, & the general public.
• Benefits to society may occur, e.g., when results of research & education projects are applied to other fields of science & technology to create startup companies, to improve commercial technology, to inform public policy, & to enhance national security.
BI Original, AICA and PAPPGDifferences
• Overlaps and gaps• (5) BI Original = Advance Discovery category
+ while supporting college students/postdocs
+BUT no K – 12 mention• (7) AICA = STEM/Workforce development
+K through 12, but no 13 through post doc+no research infrastructure category
• (9) PAPPG = Separates students from STEM workforce development
+includes research infrastructure category
Why Broader Impacts?
• Since the beginning of NSF, more excellent projects than there are funds.
• Non-technical considerations come into play when there are many projects of quasi-equal scientific merit.
• Clear criteria are important–• Many changes over the years at NSF, in reducing criteria• NIH still uses a larger number of criteria.
How Does This Work?
• Since 1998, more strict accountability for BI at every stage of the merit review process.
• BI must be discussed and evaluated at each stage:• Proposal• Reviews• Panel Summary• Review analysis• Annual report & final report
• COVs are asked to assess BI treatment at each stage.
Merit Review Process
BI
BI BI BI
Merit Review Process
COV Reports on BI
• COVs, in general, have a very positive view of NSF’s merit review process and often offer suggestions for improvements.
• A majority of the COVs called for greater clarification about the meaning of the BI criterion.
• Since documentation was inconsistent, COVs are concerned about unnecessary randomness in the merit review process.
• A number of COVs suggest more extensive training for reviewers.
COV Comments
• Relative weighting of IM and BI is not consistent across the section and variations in expectations of what constitute quality BIs are seen in individual reviews, panel summaries and review analyses [SEP].
• There continues to be confusion (among PIs, reviewers, and COV members) about what sorts of activities constitute broader impacts, and whether they need to be specifically related to the research aspects of the proposal [CISE].
COV Comments
• There is evidence of a wide range of interpretation among both PIs and reviewers about what constitutes broader impact and how it should be weighted in the proposal ratings [CBET].
• The broader impact criterion appears to be used by reviewers more as a tie-breaker rather than a more substantial and equally weighted criterion [CBET].
• Most reviewers made an effort to report on both intellectual merit and broader impacts. However, what constitutes broader impacts and the relative weighting of these criteria in the review varies greatly [PHY-A].
Recent BI Analyses and Developments • Schienke, E.W., Tuana, N., Brown, D.A., Davis, K.J., Keller, K., &
Shortle, J.S. (2009). The role of the National Science Foundation Broader Impacts Criterion in enhancing research ethics pedagogy. Social Epistemology (317-336).
• Rothenberg, M. (2010). Making judgments about grant proposals: A brief history of the Merit Review Criteria at the National Science Foundation. Technology Innovations 12 (189 – 195).
• 2013: National Alliance for Broader Impacts (NABI) formed: a community of practice that fosters the development of sustainable and scalable institutional capacity and engagement in broader impacts activity.
• Watts, S.M., George, M., & Levey, D.J. (2015). Achieving broader impacts in the National Science Foundation Division of Environmental Biology. BioScience, 65(4), 397-407.
• Ongoing: Analyses of institutional data in NSF’s Office of Integrative Activities. 18
An Empirical Look at BI• A three-year process, so far
• Directorate-wide discussions• Internal staff• External ACs
• NSF Leadership Workshop in September 2015• Report to OMB in December 2015 + July 2016• Agency-wide strategic review in January – March 2016• Report to OMB in June 2017• Presentations at BI Summits, hosted by NABI
• Consensus hypotheses• Hypothesis 1: The application of BI has changed over time; we
should expect it to continue to evolve.• Hypothesis 2: The so-called COV “confusion” might be variation in
BI due to different award sizes, disciplines and solicitation language; inconsistent instructions from NSF; and/or some other reasons.
• Hypothesis 3: There are some common practices in panels, e.g., tie-breaking.
50.77% 54.30%
37.56%
11.34%8.84%
11.35%
13.41%17.18%
14.95%
9.90%6.69%
8.55%
14.59% 12.99%
27.57%
Advance Discovery Benefit To Society Broaden Participation Dissemination Enhance Infrastructure
Investment Type: Project Summary
RESEARCH RESEARCH INFRASTRUCTURE
EDUCATION
48.01% 49.27%
34.84%
8.42% 5.43%
13.73%
14.79%22.82%
17.22%
15.99%10.78%
13.34%
12.79% 11.69%20.87%
Advance Discovery Benefit To Society Broaden Participation Dissemination Enhance Infrastructure
RESEARCH RESEARCH INFRASTRUCTURE
EDUCATION
Investment Type: Panel Reviews
-8.00%
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
8.00%
AdvanceDiscovery
Benefit ToSociety
BroadenParticipation
Dissemination EnhanceInfrastructure
Discovery(Total:6124-Collected5448)
Learning(Total:845-Collected821)
Research Infrastructure(Total:607-Collected552)
Greater Emphasis in the Summary
Greater Emphasis in the Reviews
Investment Type: Differences Between Reviews and Project Summaries
Comments from Panel ReviewsSentences in panel summaries for awards with the phrase “Is A.” in proximity to a broadening participation term. • the team is world class and includes a (couple of?) young investigators (one is a
woman) working at an institution (plu) committed to diversity of color and s-e• the postdoctoral researcher to be supported by this grant, <PI_NAME>, is an
emerging woman scientist• the pi is also a woman in computational area of cognitive science, an under-
represented population, and has a track record of training students in underrepresented groups (minorities and students with financial need)
• the pi is a woman from an under-represented group, and so this project would help increase the visibility and retention of minority women in a stem field
• as a Hispanic woman, the pi is a member a woefully underrepresented group in the sciences, and she has a track record of mentoring younger emerging scientists
• however, as a woman, the pi is already making a significant impact as a role model for female students interested in science
• the graduate student involved with this work is a woman and, given her presentation at the recent sclerochronology conference -which earned her an award for the best student talk at the meeting - she is very well qualified and shows a great deal of potential for continuing a career as a female scientist
4.82% 3.56% 5.79%
38.50%
53.45%43.88%
25.20%
21.49%25.17%
6.64%
6.12%4.91%
15.15%7.19%
11.17%
2.55% 1.89% 2.42%7.16% 6.30% 6.66%
Advancing Health And Welfare Broaden Participation Developing Workforce
Economic Competitiveness Industry Participation National Security
Public Literacy
RESEARCH RESEARCH INFRASTRUCTURE
EDUCATION
Investment Type: AICA Project Summary
9.36% 5.80%10.94%
37.02%
61.01%45.54%
28.14%
21.10%
25.99%
5.20%
2.09%2.01%
13.11%4.01%
7.94%1.26% 0.91% 1.97%
5.91% 5.08% 5.61%
Advancing Health And Welfare Broaden Participation Developing Workforce
Economic Competitiveness Industry Participation National Security
Public Literacy
RESEARCH RESEARCH INFRASTRUCTURE
EDUCATION
Investment Type: AICA Panel Reviews
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
Advancing HealthAnd Welfare
BroadenParticipation
DevelopingWorkforce
EconomicCompetitiveness
IndustryParticipation
National Security Public Literacy
Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)Research Infrastructure(Total:607-Collected552)
Greater Emphasis in Reviews
Greater Emphasis in Summaries
Investment Type: Differences Between Reviews and Project Summaries
Differences Across Investment Types by Student Group
0.000
0.500
1.000
1.500
2.000
2.500
3.000
3.500
4.000
4.500
Doctoral student Graduatestudent
Masters student Phd student Post doctoral Studentassessment
Undergrad Undergraduateresearch
Undergraduatestudent
Infrastructure Research Education
50.39% 48.41% 52.00% 54.82%47.54%
56.63%
33.25%
9.91% 10.89% 6.15%
10.47%
14.60%4.26%
27.01%
16.33%11.09%
20.97%10.93%
11.63% 16.27% 17.75%
12.43%
7.62%
6.67%7.76% 10.88%
9.66% 8.66%
10.93%
21.98%14.21% 16.02% 15.34% 13.18% 13.32%
Advance Discovery Benefit to Society Broaden Participation Dissemination Enhance Infrastructure
BIO CSE EHR ENG GEO MPS SBE
Directorates: Project Summary
49.12%42.86%
49.10% 51.02%46.57%
51.77%
34.38%
6.12%10.59%
6.91%8.09%
10.53% 3.74%
20.66%
19.64%
13.16%
19.56% 11.92% 13.40% 18.12% 18.00%
17.48%
12.78%
10.40%14.91% 15.45%
15.78%12.62%
7.64%
20.61%14.03% 14.07% 14.04% 10.59% 14.34%
Advance Discovery Benefit to Society Broaden Participation Dissemination Enhance Infrastructure
Directorates: Panel Reviews
BIO CSE EHR ENG GEO MPS SBE
-8.00%
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
8.00%
Advance Discovery Benefit to Society Broaden Participation Dissemination Enhance Infrastructure
Greater Emphasis in Reviews
Greater Emphasis in Summaries
BIO CSE ENG GEO MPS SBEEHR
Directorates: Differences Between Reviews and Project Summaries
2.90% 4.36% 1.02% 4.34% 4.45% 2.55%
12.66%
44.52%35.50% 42.20% 32.83%
37.39%
51.53%
42.75%
32.76%
20.32%
28.24%
24.83%
33.44%
28.23%
6.39%
2.47%
6.03%
10.11%
9.18%
3.77%
3.81%
10.72%
4.97%
21.19%
9.46%22.50% 6.67%
7.84%
13.10%
0.36% 7.48%4.58% 2.28%
0.31%
0.79%
5.55%
12.01%5.13% 4.40% 4.04%
13.97%5.24% 8.83%
Advancing Health And Welfare Broaden Participation Developing Workforce
Economic Competitiveness Industry Participation National Security
Public Literacy
Directorates: AICA Project Summary
BIO CSE EHR ENG GEO MPS SBE
6.58% 7.35%2.01%
6.35%15.77%
6.20%
24.83%
41.77% 43.03%44.79% 32.99%
35.32%50.46%
40.55%
37.59%
20.29% 27.13%
24.68%
32.04%29.59%
7.59%
1.14%
3.04%6.19%
8.15%
1.71%2.52%
5.30%
2.77%
18.88%11.47% 23.67%
4.49%
7.01%
7.99%
3.75%2.46%
1.83%
0.41%
0.54%
2.20%
9.95%3.66% 5.94% 2.34%
10.26%3.68%
11.54%
Advancing Health And Welfare Broaden Participation Developing WorkforceEconomic Competitiveness Industry Participation National SecurityPublic Literacy
Directorates: AICA Panel Reviews
BIO CSE EHR ENG GEO MPS SBE
-8.00%
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
14.00%
Advancing Health And Welfare Broaden Participation Developing Workforce Economic Competitiveness
Industry Participation National Security Public Literacy
BIO CSE EHR ENG GEO MPS SBE
Greater Emphasis in Reviews
Greater Emphasis in Summaries
Directorates: Differences Between AICA Reviews and Project Summaries
50.39% 52%48.41% 49% 52.00%
56% 54.82% 54%47.54% 49%
56.63% 59%
33.25% 31%
9.91%11%
10.89%13% 6.15%
6% 10.47% 13%
14.60%17% 4.26%
5%
27.01% 34%
16.33%16%
11.09%11% 20.97%
16% 10.93% 9%11.63%
10%16.27% 14% 17.75%
15%
12.43%10%
7.62%7%
6.67% 7%7.76% 7% 10.88%
11% 9.66% 10% 8.66% 7%
10.93% 10%
21.98% 20%14.21% 15% 16.02% 16% 15.34% 13% 13.18% 13% 13.32% 13%
Advance Discovery Benefit to Society Broaden Participation Dissemination Enhance Infrastructure
EHR ENG GEO MPS SBEBIO CSE
Project Summaries: Differences Between Awards – Declines by Directorate
Award Decline Award Decline Award Decline Award Decline Award Decline Award Decline Award Decline
-8.00%
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
BIO CSE EHR ENG GEO MPS SBE
Advance Discovery Benefit to Society Broaden Participation Dissemination Enhance Infrastructure
Great Emphasis in Declines
Greater Emphasis In Awards
Project Summaries: Differences Between Awards – Declines by Directorate
49% 52%43%
50% 49%54% 51% 55%
47%52% 52%
58%
34% 38%
6%9%
11%
12%7%
6%8%
10%
11%
13%4%
4%
21%
26%
20%17%
13%
10% 20%16%
12%9%
13%
10%18%
13%18%
15%
17% 13%
13%10%
10% 10%15% 11% 15% 12% 16% 12% 13%
8%
8% 10%
21% 19%14% 13% 14% 16% 14% 14% 11% 14% 14% 13%
Advance_Discovery Benefit_To_Society Broaden_Participation Dissemination Enhance_InfrastructureEHR ENG GEO MPS SBEBIO CISE
Reviews: Differences Between Awards –Declines by Directorate
Award Decline Award Decline Award Decline Award Decline Award Decline Award Decline Award Decline
-8%
-6%
-4%
-2%
0%
2%
4%
6%
BIO CISE EHR ENG GEO MPS SBE
Advance_Discovery Benefit_To_Society Broaden_Participation Dissemination Enhance_Infrastructure
Greater Emphasis In Awards
Greater Emphasis In Declines
Reviews: Differences Between Awards –Declines by Directorate
-8.0000%
-6.0000%
-4.0000%
-2.0000%
0.0000%
2.0000%
4.0000%
6.0000%
8.0000%
Advance Discovery Benefit to Society Broaden Participation Dissemination Enhance Infrastructure
Greater Emphasis In Review
Greater Emphasis In Summary
Awards: Differences Between Reviews –Project Summaries
BIO CSE EHR ENG GEO MPS SBE
-10%
-8%
-6%
-4%
-2%
0%
2%
4%
6%
8%
Advance_Discovery Benefit_To_Society Broaden_Participation Dissemination Enhance_Infrastructure
Greater Emphasis In Review
Greater Emphasis in Project Summaries
BIO CSE EHR ENG GEO MPS SBE
Declines: Differences Between Reviews –Project Summaries
-8%
-6%
-4%
-2%
0%
2%
4%
BIO CSE EHR ENG GEO MPS SBE
Advance Discovery Benefit to Society Broaden Participation Dissemination Enhance Infrastructure
Greater Discussion In Decline Reviews Compared to Proposals
Greater Discussion In Award Reviews Compared to Proposals
Differences Between Review Focus Difference & Proposal Focus Difference Awards – Declines
• Differences across investment categories, directorates, and BI evolution; some similarities, too
• Across the investment categories, people are important BI targets• For PIs, the focus is often Students and BP• For Reviewers, the focus is often BP and Workforce Development
• But Research Infrastructure and Dissemination are also important• PIs pay more attention to research infrastructure, while the Reviewers care
about Dissemination more than PIs
• PIs seem to use more of the available categories than the reviewers
Take Away Messages
• Across the Directorates, again people are the main targets• When looking at differences between PIs and Reviewers:
• PIs are focused on students, benefits to society and research infrastructure
• Reviewers are looking at dissemination more than the PIs• For the awards,
• Greater emphasis by reviewers on BP and Dissemination than warranted in the proposals
• For the declines,• Greater emphasis by reviewers on the students and research
infrastructure than in the proposals• Ration of ratios differences
• Award reviews compared to proposals• BP and dissemination are the emphasis areas
• Decline reviews compared to proposals• Students are the main emphasis area
Take Away Messages (2)
• Categories: Even though all categories are used by someone, some are used more than others
• Some categories may not be well understood or are rarely possible• Some categories may be seen as more influential than others e.g., in
getting an award, or in tie-breaking in a panel
• Primary BI focus: Young people• Least BI focus: Benefits to society
• Perhaps, these are too far-fetched for some or the opposite, taken for granted for others
• Except for SBE and GEO
• PIs: seem to use all BI areas, but not evenly• Most common use is what is close at hand, most tangible
• Reviewers: uneven use of the categories, too• Most common use is focused on what might be the most influential
• How important is BI in NSF merit review?• Not a 50-50 proposition, like two independent criteria• Instead they need to be aligned with the capacity of the research team
• Can they achieve what they say they will achieve?
What Does This Mean?
• How can we increase understanding of what BI is/could be?• Continue analysis and presentations on BI• Include all stakeholders, i.e., PI, reviewers, and PD• Decide what actions to take if we don’t like what we see in the analysis?• Include some BI survey questions in bi-annual merit review survey, including from the
different points of view of PI and Reviewer• How can we continue to increase clarity about merit review?
• Develop guidance for PIs, Reviewers, NSF staff, and COVs• Expect heterogeneity across research communities and types of programs• Our understanding of BI will evolve• Make it clear that PIs and Reviewers sometimes diverge on what is most
important• Increase the opportunities for evidence-based approaches as well as novelty• Assess impact of merit review orientation pilot
• What is the role of the institution and collective impact models?• Work with the NABI on institutional and collective BI • IM: Publications :: BI: ??
Open Questions for BI
Thank You!
+Awarded proposals, panel summaries, and post-award reports were assigned to 1 of 5 categories
Five groups of NSF documents related to NSF FY 2013 awards:
• Project summaries (N=8,042)• Project descriptions (N=8,823)• Panel summaries (N=1,264)• Final Project reports (N=1,586)• Public outcomes reports
(N=1,812)
Documents analyzed• Analyzed BI content of all
documents• Each document was
assigned to one of five categories
Methods
+Why different N’s? Encoding issues; BI section cannot be “found” in project summary; panels not required for all proposals; not all awards expired yet.
Data Analytics
1.39% 2.01% 0.64% 2.32% 1.82% 1.04% 7.37%
28.06%
17.97%
34.12%
20.23% 18.49%27.34%
27.76%
2.89%
2.28%
3.86%
2.63%2.59%
2.75%
1.12%0.67%
1.59%
1.22%
3.51%0.48%
0.73%
6.32%
28.60% 45.22%
24.05%
34.69%40.17%
30.90%
29.66%
3.07%
10.66%
7.45%13.60%
3.22% 4.12%
8.28%
0.22%
3.76%
3.58%
1.38%
0.15% 0.42%
3.51%
7.42%
2.58%
3.46%2.44%
6.73%2.76%
5.58%27.66%
13.94%
21.62% 19.20%26.35%
29.95%
10.39%
$404,439,723 $ 56 1,4 83,13 4 $534,735,189 $389,213,939 $550,681,144 $152,791,355
BIO(TOTAL:1070-COLLECTED 1062)
CSE(TOTAL:1247-COLLECTED 1231)
EHR(TOTAL 1208-COL L ECTED 931)
ENG(TOTAL:1455-COLLECTED 1435)
GEO(TOTAL:978-COLLECTED 958)
MP S(TOTAL:1972-COLLECTED 1966)
SBE(TOTAL:698-COLLECTED 693)
DIRECTORATES: PAPPG PROJECT SUMMARY Advance Well-Being Broaden Participation Develop Workforce Economic Competitiveness Enhance Infrastructure
Industry Participation National Security Public Literacy Stem Education
1.21%0.49% 2.33% 2.01% 0.90%
7.28%
29.26%23.24%
33.30%
22.00%19.60%
28.59%
24.42%
1.24%
0.88%
1.91%
1.14%1.68%
1.59%0.63%0.25%
0.76%
0.47%
4.47%
0.23%
0.41%2.78%
21.25%
42.88%27.20%
29.10%36.69%
25.30%
35.92%1.89%
10.04%
8.31% 15.43%
2.39%3.92%
4.66%
2.00%
1.77%1.19%
0.22%0.30%
1.28%
6.79%
1.95%
4.31% 1.53%
5.45%2.06%
6.73%38.22%
17.05%22.23% 22.83%
31.72%36.93%
16.30%
$404,439,723 $56 1 ,4 83,13 4 $534 ,735,189 $389 ,213,939 $550 ,681,144 $152 ,791,355
B IO(TOTAL:1070-COLLECTED 957)
CSE(TOTAL:1247-COLLECTED 1048)
EHR(TOTAL 1208-COL L ECTED 931)
ENG(TOTAL:1455-COLLECTED 1316)
GEO(TOTAL:978-COLLECTED 863)
MP S(TOTAL:1972-COLLECTED 1837)
SBE(TOTAL:698-COLLECTED 629)
DIRECTORATES: PAPPG PANEL REVIEWSAdvance Well-Being Broaden Participation Develop Workforce Economic Competitiveness Enhance Infrastructure
Industry Participation National Security Public Literacy Stem Education
NSF Broader Impacts: Thematic Analysis
April 2017
50Image courtesy MIT Museum
Scientific progress is one essential key to our security as a nation, to our better health, to more jobs, to a higher standard of living, and to our cultural progress..
Little Difference between Proposed and Reported Activities
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Description Final ReportAdvance Discovery Benefit To Society Broadening Participation Dissemination Enhance Infrastructure
1,586 1,586
2.11% 1.17% 1.56%
22.59%
41.20%
22.28%
1.58%
1.11%
1.83%1.83%
0.35%29.93%
27.59%
46.89%
7.83%
2.60% 3.83%
0.76%
0.59%0.95%
3.53%
3.29%2.71%
29.85%22.11% 19.73%
$2,234,929,965 $161885448 $278,730,801
DISCOVERY(TOTAL:6124-COLLECTED6053)
LEARNING(TOTAL:845-COLLECTED844)
RESEARCH INFRASTRUCTURE(TOTAL:607-
COLLECTED602)
PANEL REVIEWS PAPPG FY 2014Advance Well-Being Broaden ParticipationDevelop Workforce Economic Competitiveness
Differences across Project Size
Differences between PI and Reviewer Perspectives
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Description PanelAdvance Discovery Benefit To Society Broadening Participation Dissemination Enhance Infrastructure
8,823 1,264
Significant differences between what PIs propose & panelists perceive.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Description Panel
Advance Discovery Benefit To Society Broadening Participation Dissemination Enhance Infrastructure
1,264 1,264
-0.4
%
-0.8
% -0.2
%
0.0% 0.2%
-0.1
%
-0.1
%
1.2%
5.3%
-0.8
%
1.8%
1.1% 1.3%
-3.3
%
-1.7
%
-1.4
%
-2.0
% -1.5
% -0.9
%
-1.2
% -0.5
%
-0.4
%
-0.8
%
-0.8
%
1.0%
-0.3
%
-0.3
%
-3.5
%
-7.4
%
-2.3
%
3.2%
-5.6
%
-3.5
%
-5.6
%
6.3%
-1.2
% -0.6
%
0.9%
1.8%
-0.8
% -0.2
%
-3.6
%
-0.1
%
-1.8
%
-1.8
%
-0.2
%
0.1%
-0.1
%
-2.2
%
-0.6
%
-0.6
%
0.9%
-0.9
%
-1.3
% -0.7
%
1.2%
10.6
%
3.1%
0.6%
3.6%
5.4%
7.0%
5.9%
-10.00%
-8.00%
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
Difference Between Reviews and Project Summary FY 2014 PAPPGAdvance Well-Being Broaden Participation Develop Workforce
Economic Competitiveness Enhance Infrastructure Industry Participation
National Security Public Literacy Stem Education
Greater Emphasis in Reviews
Greater Emphasis in Summaries
BIO(Total: 1070-
Collected 957)
CSE (Total: 1247 –
Collected 1048)
EHRTotal 1208 –
Collected 931)
ENG(Total: 1455 –
Collected 1316)
GEOTotal: 978 –
Collected 863)
MPS(Total: 1972 –
Collected 1837)
SBE(Total: 698 –
Collected 629)
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
Advancing HealthAnd Welfare
BroadenParticipation
DevelopingWorkforce
EconomicCompetitiveness
IndustryParticipation
National Security Public Literacy
Difference Between Reviews and Project Summaries FY 2014Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)Research Infrastructure(Total:607-Collected552)
AdvancingHealth And
Welfare
Broaden Participation
Developing Workforce
Economic Competitiveness
Industry Participation
National Security
Public Literacy
Alternate 2 of Previous: Added additional label text above bars
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
Advancing HealthAnd Welfare
BroadenParticipation
DevelopingWorkforce
EconomicCompetitiveness
IndustryParticipation
National Security Public Literacy
Difference Between Reviews and Project Summaries FY 2014Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)Research Infrastructure(Total:607-Collected552)
AdvancingHealth And
Welfare
Broaden Participation
Developing Workforce
Economic Competitiveness
Industry Participation
National Security
Public Literacy
Alternate 3 of Previous: Added
label and emphasis text
Greater Emphasis in Reviews
Greater Emphasis in Summaries
-8.0%
-6.0%
-4.0%
-2.0%
0.0%
2.0%
4.0%
6.0%
8.0%
Advance Well-Being
BroadenParticipation
DevelopWorkforce
EconomicCompetitiveness
EnhanceInfrastructure
IndustryParticipation
NationalSecurity
Public Literacy Stem Education
Difference In Focus Between Reviews and Proposal Summaries FY 2014
Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)Research Infrastructure(Total:607-Collected552)
-0.4
%
-0.8
% -0.2
%
0.0% 0.2%
-0.1
%
-0.1
%
1.2%
5.3%
-0.8
%
1.8%
1.1% 1.3%
-3.3
%
-1.7
%
-1.4
%
-2.0
%
-1.5
% -0.9
%
-1.2
% -0.5
%
-0.4
%
-0.8
%
-0.8
%
1.0%
-0.3
%
-0.3
%
-3.5
%
-7.4
%
-2.3
%
3.2%
-5.6
%
-3.5
%
-5.6
%
6.3%
-1.2
% -0.6
%
0.9%
1.8%
-0.8
% -0.2
%
-3.6
%
-0.1
%
-1.8
%
-1.8
%
-0.2
%
0.1%
-0.1
%
-2.2
%
-0.6
%
-0.6
%
0.9%
-0.9
%
-1.3
% -0.7
%
1.2%
10.6
%
3.1%
0.6%
3.6%
5.4%
7.0%
5.9%
-10.00%
-8.00%
-6.00%
-4.00%
-2.00%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
Difference Between Reviews and Project Summary FY 2014 PAPPG
Advance Well-Being Broaden Participation Develop Workforce
Economic Competitiveness Enhance Infrastructure Industry Participation
National Security Public Literacy Stem Education
Greater Emphasisin Reviews
Greater Emphasis in Summaries
CSE EHR ENG GEO MPS SBEBIO
-8.0%
-6.0%
-4.0%
-2.0%
0.0%
2.0%
4.0%
6.0%
8.0%
Advance Well-Being
BroadenParticipation
DevelopWorkforce
EconomicCompetitiveness
EnhanceInfrastructure
IndustryParticipation
NationalSecurity
Public Literacy Stem Education
Difference In Focus Between Reviews and Proposal Summaries FY 2014
Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)
Research Infrastructure(Total:607-Collected552)
Greater Emphasis in Summary
Greater Emphasis in Reviews