Reference Number: 09-2008 NIDCR

Facilitating and Experiencing Interdisciplinarity in Biomedical Research

Mid-course Evaluation of the Interdisciplinary Research Consortium Program:

an NIH Common Fund Program

Sue Hamann, PhD, Science Evaluation Officer National Institute of Dental and Craniofacial Research

National Institutes of Health 31 Center Drive, MSC 2190

Bethesda, MD 20892 &

Alina Martinez, EdD Abt Associates Inc. 55 Wheeler Street

Cambridge, MA 02138-1168

September 4, 2011

NIH Leadership of the Interdisciplinary Research Consortium Program Lawrence Tabak, DDS, PhD, Office of the Director

Patricia A. Grady, PhD, RN, FAAN, National Institute of Nursing Research Melissa Riddle, PhD, National Institute of Dental and Craniofacial Research

1

  

Facilitating and Experiencing Interdisciplinarity in Biomedical Research Evaluation of the Interdisciplinary Research Consortium Program

Table of Contents List of Appendices ……………………………………………………….… page 3 Executive Summary ……………………………………………………………….. 4

Section 1. Introduction to the Evaluation 1.A. Program description ……………………………………………………….. . 6 1.B. Review of the literature about evaluating interdisciplinary research ………. 7 1.C. Purpose of the evaluation, logic model, and key evaluation questions …..…. 9 1.D. Evaluation methods ………………………………………………………… 10 1.E. Evaluation costs and resources ……………………………………………… 10 1.F. Dissemination of evaluation findings ……………………………………….. 11

Section 2. Interviews of U54 (UL) Principal Investigators (See also Appendix 7) . 13 2.A. RFA and proposal development ……………………………………………. 13 2.B. Budget issues …………………………………………………………………14 2.C. Other NIH mechanisms that could fund IDRC …………………………….... 15 2.D. Self-assessment of leadership qualities ……………………………………… 16

2.D. i. Preparation and satisfaction with PI role ……………………….. 16 2.D. ii Generating and sustaining trust ………………………………… 17 2.D. iii Cultivating a shared vision ……………………………………... 18 2.D. iv. Bias toward risk taking and action ……………………………… 19

2.E. Self-assessment of methods for achieving important process goals for team science

2.E. i. Participatory goal-setting ……………………………………… 20 2.E. ii. Adequate feedback and communication ……………………….. 20

2.F. Interactions with NIH staff ………………………………………………… 21

Section 3. Observations of scientific meetings ……………………………………. 22

Section 4. Investigator surveys (See also Appendix 8) ……………………………. 24

Section 5: Trainee interviews and ratings (See also Appendix 9) …………………. 27 5.A.Characteristics of the participants …………………………………….. 27

5.B. Structural features of the interdisciplinary training experience ……… 27 5.C. Scholarly experiences and scientific productivity …………………… 28 5.D. Trainee responses to specific questions ……………………………… 28

5.D. i. What are the advantages of interdisciplinary training? ……. 28 5.D. ii. What impact is this interdisciplinary training likely

to have on your career? …………………………………… 29 5.D. iii. What message would you like to give Dr. Collins

about your training experience? …………………………… 30

2

  

Section 6. NIH Staff interviews (See also Appendix 10) …………………………. 31

3

  

Appendices

Appendix 1. Project abstracts for each of the Interdisciplinary Research Consortium

Appendix 2. eSPA Definitions of Key Indicators

Appendix 3. Abt Associates Inc Report Review of Relevant Interdisciplinary Research Evaluations and Literature Evaluation of the NIH Roadmap’s Interdisciplinary Research Work Group:

A Design and Feasibility Study

Appendix 4. Data collection instruments

Appendix 5. Consent forms

Appendix 6. OMB Clearance document

Appendix 7. Interviews with Principal Investigators

Appendix 8. Abt Associates Inc Report Interdisciplinary Research Work Group Evaluation: Survey of Consortia Investigators

Appendix 9. Abt Associates Inc Report Interdisciplinary Research Work Group Evaluation: Survey of Trainees

Appendix 10. Abt Associates Inc Report Interdisciplinary Research Work Group (IDRWG) Evaluation:

Summary of NIH Staff Interviews

4

  

  

Executive Summary

The RFA for the Interdisciplinary Research Consortium program (IDRC program) was issued in 2006. Nine consortia were funded, beginning in FY 2007. Each consortium received a center award (initially a U54), multiple R01 awards (the range was 2 to 10), and typically a mixture of P30s, T90s, R90s, and others. A marked difference between these projects and the typical NIH projects was that each project was assigned to multiple institutes or centers at NIH for grant administration, grant management, and grant oversight.

The primary program goals for the Interdisciplinary Research Consortium program follow: • Encourage the integration of different scientific disciplines to develop new intellectual

and technological approaches to complex health problems; • Support interdisciplinary approaches to solving significant and complex biomedical

problems, particularly those that have been resistant to traditional approaches; • Catalyze the creation of new disciplines.

The following key evaluation questions were chosen. • What management issues arose, both at the project (grantee) and program (NIH) level

and how were these resolved? • How did management of interdisciplinary research differ from management of other

biomedical research? • What structural features were put in place at the project level to support

interdisciplinarity of research and training? • What leadership qualities supported interdisciplinarity at the project level? • How did the experiences of investigators differ from their previous experiences? • How did the experiences of post-doctoral trainees differ from their previous experiences? • How did trainees predict that their interdisciplinary training would affect their careers?

The evaluation began in Year 2 (2009) of the project, and almost all data collection was completed by the end of Year 3 (2010). Multiple methods, qualitative and quantitative, were employed. Document reviews focused primarily on grant applications and annual progress reports from the grantees, as well as grantee documents such as manuscripts or other reports and published articles. Other reports from Program Officers at NIH were reviewed. Project Principal Investigators, some staff, and trainees participated in guided interviews. Some trainees participated in group interviews. Trainees also completed rating scales. Scientific meetings were observed. Meetings of NIH program officials were observed. Electronic surveys were sent to investigators.

The following interdisciplinary activities, behaviors, or outcomes were documented for the Investigators: • Frequent meetings with investigators from multiple disciplines to discuss research

methods dedicated to a specific, shared medical condition • Expansion of own research vocabulary and portfolio, consistent with Consortium

disciplines

5

  

• Increase in scholarly activity and scientific productivity in new disciplines • Co-mentoring of trainees, consistent with Consortium disciplines

• Sharing of resources

The following interdisciplinary activities, behaviors, or outcomes were documented for the Trainees: • Access to investigators from multiple disciplines • Training on equipment and methods from multiple labs • Expansion of knowledge in other fields

The apparent facilitators of interdisciplinarity at the project level follow: • Dedication to a single medical or public health problem, narrowly defined • Funding for an administrative core • Investigator interactions – face to face

o Research progress review & planning (specific experiments and methods) o Senior investigator meeting or seminar series

• Flexibility in funding to allow expansion of research plan, rapid response to research findings, and expansion of investigator team

• Scientific Advisory Boards and Annual Meeting, especially when combined with NIH site visit

• Shared equipment (non-IT)

The apparent inhibitors of interdisciplinarity at the project level follow: • Dedication to a single medical or public health problem, broadly defined • Geographic dispersion • Weak link between clinical activities and clinical research • Management of clinical trials • Temporally sequenced set of research activities such that many specific investigations

could not be initiated until other activities were completed

Note: Analysis of evaluation data was continuing at the date of the submission of the current report (September 4, 2011).

6

  

Section 1. Introduction to the Evaluation

1.A. Description of the program The Interdisciplinary Research Working Group (IRWG) of the NIH Common Fund (also known as the Roadmap fund) launched eight initiatives to overcome barriers to medical research that are inherent in single discipline approaches. The RFA for the Interdisciplinary Research Consortium program (IDRC program) was issued in 2006. Nine consortia were funded, beginning in FY 2007. Each consortium received a center award (initially a U54), multiple R01 awards (the range was 2 to 10), and typically a mixture of P30s, T90s, R90s, and others. A marked difference between these projects and the typical NIH projects was that each project was assigned to multiple institutes or centers at NIH for grant administration, grant management, and grant oversight. The average annual amount awarded in direct costs in FY 2007 was $2.8 million per consortium. Total direct costs for each consortium for five years were initially projected to be $14.6 million and total of all costs, $22.1 million.

The primary program goals for the Interdisciplinary Research Consortia follow: • Encourage the integration of different scientific disciplines to develop new intellectual

and technological approaches to complex health problems; • Support interdisciplinary approaches to solving significant and complex biomedical

problems, particularly those that have been resistant to traditional approaches; • Catalyze the creation of new disciplines.

Titles of the nine projects follow: Consortium for Neuropsychiatric Phenomics; Genomics Based Drug Discovery; Interdisciplinary Research Consortium in Geroscience; Interdisciplinary Research on Stress, Self-Control, and Addiction; NeuroTherapeutics Research Institute; Northwest Genome Engineering Consortium; Oncofertility Consortium: Fertility Preservation for Women; Systems-based Consortium for Organ Design and Engineering; and Taskforce for Obesity Research at Southwestern. Project abstracts from the original proposal of each of the Interdisciplinary Research Consortium are contained in Appendix 1.

Scientific productivity demonstrated by research publications in peer-reviewed science journals are a major metric of performance at NIH. Many institutes use an application developed at NIH, the electronic Scientific Portfolio Assistant (eSPA) to provide descriptive metrics, based on web-based searches, for NIH-funded projects. Following is a summary of eSPA metrics for each of the nine projects, based on approximately three years of performance. (Please see Appendix 2 for the eSPA explanations of the metrics.)

7

  

 

                                                        

Table 1. Portfolio Analysis from the NIAID eSPA

Research publications (total) **

Impact factor (Range)

Times cited (total)

Patents issued

Project 1* 33 2-23 194 0 Project 2 8 10-10 68 0 Project 3 68 1-24 1094 0 Project 4 73 1-27 366 0 Project 5 35 2-23 157 0 Project 6 20 4-27 78 0 Project 7 62 1-5 61 0 Project 8 38 2-24 711 0 Project 9 35 2-29 273 0

* Project numbers were assigned randomly; there is no meaning to the order of the projects. ** All indicators are based on Years 1‐March 2011; Research Publications; w/o self‐citation

1.B. Review of the literature about evaluating interdisciplinary research A major source of information for the evaluation came from the following source: National Academy of Sciences, National Academy of Engineering, and Institute of Medicine of the National Academies. Facilitating Interdisciplinary Research. The National Academies Press, Washington, DC, 2005. The book included definitions of interdisciplinary training and research, examples of interdisciplinary programs and projects, recommendations for evaluating interdisciplinary faculty, training programs, institutions, and research teams, and toolkits for evaluation.

An important distinction was made many times in this book between standard biomedical research quality and productivity indicators and the value-added indicators that should emerge from interdisciplinary research and training: “… a successful IDR program will have an impact on multiple fields or disciplines and produce results that feed back into and enhance disciplinary research. It will also create researchers and students with an expanded research vocabulary and abilities in more than one discipline and with an enhanced understanding of the interconnectedness inherent in complex problems.” (p.150) Similarly, “Many of the standard means for evaluating disciplinary research and teaching can also be applied to interdisciplinary research and teaching: the use of metrics, such as number of publications, citations of publications, and successful research grant proposals.... However, IDR can be expected to have measurable outcomes in multiple elements of technique, theory, and application” (p. 152).

Facilitating Interdisciplinary Research also contained recommendations about evaluating project leadership: “The commonest cause of underperformance of IDR is the failure of a team to gel or function collaboratively. That may happen for a variety of reasons: individual members may place the importance of their own work ahead of the team vision, devalue the contributions of other team members, or lack leadership. Other contributing causes of lower than expected

8

  

 

outcomes may be inadequate recognition for contributions to teams, low participation or understanding by senior staff members, inadequate time for participants to establish close working relationships, and insufficient funding.” (p.53) Results about leadership from a survey study cited in the book suggested that principal investigators can facilitate interdisciplinary research by “… increase[ing] leadership and team-forming activities … and develop[ing] and clearly stat[ing] their research goals and their overall vision ….” (p.77).

Finally, one of the recommendations in the section about evaluating programs, institutes, and centers spoke to the importance of evaluator site visits that include interviews and observations by the evaluator.

The National Science Foundation sponsored an evaluation of 71 projects awarded in the late 1990s from a program entitled The Knowledge and Distributed Intelligence projects (Reference: http://www.nsf.gov/cise/kdi/eval.html, also cited in Facilitating Interdisciplinary Research). Overall, the program met its goals and positive outcomes fell into 4 categories: new ideas; new tools; student training; and project outreach. Additional analyses were conducted to understand the predictors of success of the individual projects. A key finding was that “dispersion rather than multi-disciplinarity was the most problematic aspect of KDI projects. Projects with PIs in more universities were significantly less well coordinated and reported fewer positive outcomes. Coordination mechanisms that brought researchers together (such as holding a project-related conference or workshop) appeared to reduce the negative impact of dispersion.”

The following report was also informative in regard to the proposed methodology: Institute of Medicine, 2004.,NIH Extramural Center Programs: Criteria for Initiation and Evaluation. Components of the methodology recommended for evaluating center programs included the following:

Comparison of results achieved against expressed goals; Site visits, interviews, and surveys; Multiple sources of evidence.

Specifically in regard to training programs, recommended evaluation indicators were increased number of courses, seminars, and workshops offered and increases in the numbers of health professionals attending such trainings (Box 5-1, page 120).

Trochim et al., 2008 (Trochim WM, Marcus SE, Masse LC, Moser RP, Weld PC. The Evaluation of large research initiatives: a participatory integrative mixed-methods approach. Am J of Evaluation, (2008); 29 (1), 8-28.) made a series of recommendations for evaluating scientific entities, as follows:

Develop a comprehensive conceptual model Use participatory and collaborative evaluation approaches Incorporate integrative mixed methods

Integrate evaluation with existing reporting systems Adapt the evaluation to the initiative’s stage of development Develop standardized cross-initiative evaluation systems Utilize peer review approaches Address issues of causation and control Improve funding and organizational capacity for evaluation

9

  

 

  

Address management issues in large initiative evaluation (p.20 -25).

1.C Purpose of the evaluation, logic model, and key evaluation questions The evaluation activities reported herein continued and complemented the evaluation activities already completed for the Interdisciplinary Research Work group. The following written reports about the training components have been issued: Mid-course Review Meeting of Training Director, October 29, 2007; Mid-course Report of Training Initiatives Developed by the Interdisciplinary Research Work Group (IRWG) of the NIH Roadmap; and Interdisciplinary Research Training Director’s Retreat, April 29-30, 2007, Bethesda. Maryland. Also released on June 26, 2008, was a report prepared for the IRWG by Abt Associates: IRWG Process Evaluation Design, Evaluation of the NIH Roadmap’s Interdisciplinary Research Work Group: A Design and Feasibility Study (Appendix 3). Moreover, a pilot evaluation was completed for the Multiple PI Initiative.

There were multiple purposes for the evaluation. Evaluation of Common Fund initiatives was mandated by the NIH Office of the Director. Also, the evaluation provided information to NIH regarding the institutionalization of program and project management policies and procedures for multidisciplinary research and training. The evaluation also informed NIH recommendations for future I/C locations of interdisciplinary research and training programs. Finally, the evaluation documents for the biomedical research policy community and the evaluation community the ways in which interdisciplinarity are experienced, facilitated, and inhibited.

The evaluation began in Year 2 (2009) of the project, and almost all data collection was completed by the end of Year 3 (2010). In accordance with an outcomes logic model that identifies expected outcomes by the stage of the program (Table 2, below), the following key evaluation questions were chosen. • What management issues arose, both at the project (grantee) and program (NIH) level

and how were these resolved? • How did management of interdisciplinary research differ from management of other

biomedical research? • What structural features were put in place at the project level to support

interdisciplinarity of research and training? • What leadership qualities supported interdisciplinarity at the project level? • How did the experiences of investigators differ from their previous experiences? • How did the experiences of post-doctoral trainees differ from their previous experiences? • How did trainees predict that their interdisciplinary training would affect their careers?

10

  

 

                     

 

Table 2 : Outcome Logic Model for the Common Fund IDRW Short-term Outcomes

Intermediate- term outcomes

Long-term & final outcomes

Years 1-2 Years 3-5 Years 4-6 Outcomes of interest Outcomes of interest Outcomes of interest Program management Investigator Improved Project management development interventions Collaboration & new New or improved Emergence of a new organizational models methods, models, or field Communication theories Translation to Fidelity to application Research quality &

productivity Trainees develop IDR knowledge base and research skills

practice Dissemination Achievement of NIH project goals & objectives Health impact Trainees move to next phase in the IDR pipeline

1.D. Evaluation methods Multiple methods, qualitative and quantitative, were employed. Document reviews focused primarily on grant applications and annual progress reports from the grantees, as well as grantee documents such as manuscripts or other reports and published articles. Other reports from Program Officers at NIH were reviewed. Project Principal Investigators, some staff, and trainees participated in guided interviews. Some trainees participated in group interviews. Trainees also completed rating scales. Scientific meetings were observed. Meetings of NIH program officials were observed. Electronic surveys were sent to investigators. (Data collection instruments are included in Appendix 4.)

The Institutional Review Board of the contracted evaluator, Abt Associates, reviewed and approved data collection forms and consent documents. Informed consent (see Appendix 5) was obtained in writing for individual and group interviews. The electronic surveys were subject to the federal Paperwork Reduction Act, and consent was obtained from the Office of Management and Budget (see Appendix 6).

1.E. Evaluation costs and resources The evaluation cost was borne by the National Institute for Dental and Craniofacial Research (NIDCR) budget, which included Common Fund monies, and by evaluation set-aside funds, which are available across the NIH by competitive application from any of the Institutes or Centers. Internal funds of approximately $75,000 from the Office of the Director of NIDCR supported the NIH Principal Evaluator; travel costs of about $25,000 for her were paid from the Common Fund. Approximately $233,000 in set-aside funds was charged to the evaluation by one contracted evaluation firm, Abt Associates Inc.

11

  

 

The NIDCR Science Evaluation Officer was the Principal Evaluator. She has almost thirty years of experience as a program evaluator and human subjects researcher. She also has experience as an NIH-funded investigator and in conducting randomized, controlled, clinical trials. She received a PhD from The Ohio State University in Educational Policy and Leadership, with specialty areas of evaluation and statistics. She has been an active member of the American Evaluation Association since 1994.

The contracted evaluator, Abt Associates, was selected through a competitive Request for Quotation process managed by the National Heart, Lung, and Blood Institute at the NIH. Abt Associates was founded in 1965 and is a major contract evaluation firm, with a staff of more than 1,900 and gross revenue in FY 2010-2011 of $383 million. According to its public website, Abt Associates “… applies its expertise in research, consulting, technical assistance, data collection and medical and life sciences to a wide variety of problems in the public and private sectors. In the United States, Abt Associates has helped shape many important and complex public programs, including Medicaid, welfare reform, Head Start, crime reporting, and housing... Examples of Abt Associates' work include research, evaluation and technical assistance related to improving the effectiveness of government programs, the cost-effectiveness of environmental regulations, the usefulness of new education strategies, the efficiency of healthcare systems, the incidence and prevalence of diseases, and the effectiveness of customer satisfaction and market positioning strategies” (August 13, 2011, www.abtassociates.com). The Abt Project Director for the evaluation was Alina Martinez, EdD, who has led many evaluations of federal programs, including those for the National Science Foundation and the National Institutes of Health. Dr. Martinez was the Project Director for the prior feasibility study of the Common Fund’s Interdisciplinary Research program. She is a member of the American Evaluation Association.

The evaluation was planned and conducted in accordance with The Program Evaluation Standards, 2nd

Edition (The Joint Committee on Standards for Educational Evaluation, Sage Publications, Thousand Oaks, 1994) and the Guiding Principles for Evaluators of the American Evaluation Association.

An evaluation advisory committee was convened. Each of the nine projects was asked to participate on the committee, and two of the projects did so. The representative of one project was the Science Manager for her Consortium and is a PhD biologist; she was involved in the project evaluation from the beginning of the project and subsequently joined the American Evaluation Association. The second representative was the Evaluator for his Consortium; he is a PhD psychologist and an NIH Principal Investigator. He is also a member of the American Evaluation Association.

1.F. Dissemination of evaluation findings A briefing on initial findings was presented in January 2010 to the two NIH Institute Directors who were the Chairs of the Interdisciplinary Research Working Group, the NIH Consortium Program Coordinator, and the NIH Consortium Deputy Coordinator. A briefing on all major findings was presented to the same group of program leaders in September 2010. The current written report will be delivered to the program leaders listed above, the UL Principal Investigators of each project, and members of the Evaluation Advisory Committee. The full written report or portions of the written report will be submitted electronically to the NIH

12

  

                       

                                              

  

 

       

    

    

    

                                              

         

   

   

                                                            

         

   

   

                             

                      

              

          

    

    

    

    

        

        

  

  

      

         

                                  

    

        

 

      

               

        

          

 

   

evaluation set-aside program and posted on its public website and may also be submitted to other interested parties in the NIH community.

The Evaluation Advisory Committee is actively delivering information about the project level and program level evaluations to the evaluation community. Four papers from the Committee were presented at the annual meetings of the American Evaluation Association in 2010. The Committee continues to work together for further analysis of the local and national data. Dissemination activity for 2011 is detailed in the following table.

Table 3: Evaluation Presentations in 2011 Martínez A, Hamann S, Katz L. (2011) Interdisciplinary Training Funded by the National Institutes of Health: Findings from an Evaluation. Poster presentation. Annual Science of Team Science Conference, Chicago, IL

Martinez, Hamann, Katz

Science of Team Science Chicago April 11‐14

(1) Trainee interviews (2) Trainee surveys

Agoulnik, I. (2011) Social Network Analysis and Publications Review in the Evaluation of Team Science. Poster presentation. Annual Science of Team Science Conference, Chicago, IL.

Agoulnik Science of Team Science Chicago April 11‐14

Project data

Tebes, J. K. (2011). Evaluating the local context of interdisciplinary team science. Invited panel address on: Evaluation of Team Science: A Multilevel Systems Perspective. Annual Science of Team Science Conference, Chicago, IL.

Tebes Science of Team Science Chicago April 11‐14

Project data

Martínez A, Hamann S, Katz L. (2011). Interdisciplinary Research and Education Funded by the National Institutes of Health: Findings from an Evaluation. Conference abstract published at Science on FIRE: Facilitating Interdisciplinary Research and Education, Boulder, CO.

Martinez AAAS & the Colorado Initiative in Molecular Biotechnology

March 28‐29th

(1) Trainee interviews (2) Trainee surveys

Changing Scientific Engagement through Hamann, European Investigator Interdisciplinarity Martinez,

Tebes, Sociological Association

surveys, U54 PI interviews,

Status: Oral Presentation accepted Agoulnik Geneva,Sep 7‐10 Observations The Science of Team Science: Advances in Tebes, AEA, Anaheim CA Program and Evaluation of the NIH Interdisciplinary Agoulnik, Nov 2‐5 project data Research Consortium Program Martinez, Status: Oral presentation accepted Hamann Internal Evaluation of the NIH Interdisciplinary Research Consortium Program Status: Poster presentation accepted

Agoulnik AEA, Anaheim CA Nov 2‐5

Project data

13

  

Section 3. Interviews of U54 (UL) Principal Investigators

Each Consortium was organized by an administrative core, which was funded originally under a U54 mechanism and then under a UL mechanism. Herein, the terms will be used interchangeably. Each U54 had a named Principal Investigator; the Consortium could also choose to include a Co-Principal Investigator. Each U54 Principal Investigator participated in a 90-minute interview with the NIH Principal Evaluator or the Abt Project Director during Year 2 of the program. If the project had a Co-PI, the Co-PI was included in the interview. For one project, both the PI, who had recently been appointed to this position, and the Director of Sponsored Research, were interviewed (separately), although the latter responded only to questions about grant management. The interview questions were tailored to each project, but a general interview guide and consent form (Appendices 4,5) were sent to the respondents about one week prior to the site visit. Written consent was obtained in person by the Evaluator prior to each interview. A few weeks after each interview, a written transcript of the interview was emailed to the respondent, who edited as wished. The edited transcripts served as the raw data for each project Appendix 7).

In the following sections, the comments recorded, edited, and approved by the PIs are summarized and presented. The comments are organized by six observed topics:

o RFA and proposal development o Budget issues o Other NIH mechanisms that could fund IDRC o Self-assessment of leadership qualities o Self-assessment of methods for achieving important process goals for team

science o Interactions with NIH staff

Not every PI contributed to every topic. Comments that could not be categorized into one of the above observed topics were not summarized in this section but may be accessed in Appendix 6. Comments were edited to mask the identity of the speaker. Speaker’s comments appear in italics.

2.A. RFA and proposal development The PIs learned about the Program in a variety of ways but none appeared to have advanced notice, that is, notice prior to the release of the Program Announcement or RFA. The PIs were favorable as a whole toward the proposers’ meeting and the pre-application mechanism but mixed in their opinions of the utility of a lengthy planning process. The PIs advised that the full proposal required much work in a short amount of time.

Eight weeks was sufficient for the pre-application. We had to be direct and focused. I love grants that require a pre-application. It allows me to formulate my thinking, to distill it, to articulate my thoughts about IDR. Then most of the work for the final proposal was done. In terms of feedback about the pre-application, I got good feedback from the first Program

14

  

Coordinator, but the comments from the reviewers were even more useful. We addressed the reviewers’ comments in the full proposal.

I was at an NIH meeting, serving as a grant reviewer, and found out about the P20 Exploratory Planning Grants, the 22 funded planning grants, so I looked up the Interdisciplinary Research Program on the web. I found the Program Coordinator’s name and saw that he was doing a workshop, which I attended. I thought it was fascinating. Dr. Zerhouni spoke at the workshop, and I was impressed. Dr. Eric Lander was also there, and NIH did a very nice job with the workshop.

We did not have a planning grant, although another group at our university had one. The planning grant seemed to put some groups in jeopardy; if you didn’t do well on the P30, you wouldn’t get refunded.

The full proposal was an enormous document that was just due just before Christmas. How many groups could develop and submit a proposal under these conditions? We considered this part of the self-selection process. This was the biggest application in our history, and other proposals seemed easier after that.

We discovered the RFA by reviewing grant announcements, but we had no advance knowledge that the IDRC program was coming. From August 2006 through December 2006, I spent every waking minute putting the IDRC application together. I had to establish my credibility. There are not too many opportunities to put together a group like this and then work together.

2.B. Budget issues The PIs discussed several problematic budget experiences: paying fringe benefits for trainees, justifying carry-over, and inter-IC differences in funding that presented challenges, such as allowable ARRA mechanisms. There was also confusion about the structure of the Annual Progress Reports. Delays in getting prompt replies from Program Officers were frustrating. Mixed opinions about whether the U54 PIs should control the entire Consortium budget were observed, but the majority opinion was that the U54 PI should have had more control.

It does make it difficult that I cannot control all the money in the Consortium. I would have a lot more flexibility and could meet the needs of the Consortium much more quickly if all the money came to the U54 first and then was distributed by subcontracts. That is how my other U54 award works. One of the reasons we lost one member of the team was that he controlled his own money and was not as committed to the Consortium as to his own award. At the beginning of a project, the distributed financial allocation takes out the ability of the team to function as a team. Even the original PI thinks the funding should start at the U54.

We had quite serious carry-over at the end of Year 1 because of the nine month duration and also changes in our administrative structure. We thought it might be a problem to request the carry-over because it had to be justified. The U54 NIH Program Officer and the NIH project team have turned around our requests for carry-over and approval for expenses quickly. The problem we experienced was that the rules for P30 carry-over were different than the rules for U54. That meant that we had different rules for different components of the IRC. Sometimes I

15

  

felt that I was being jerked around by NIH staff. More guidance upfront about NIH expectations for fiscal management and reporting would have been helpful.

In regard to control of the Consortium budget, it would be preferable for it all to go to the U54. We need more flexibility than we had with the separate budgets. NIH said we would have flexibility but it was hard. With the cooperative agreement, it was hard to move the money. NIH restrictions on our ability to move money are tough. I’m okay that NIH does not allow me to be in charge of all the money, but it does create an issue. But I don’t want to deal with all the budgets. Restrictions on the end of the year carry-forward are also tough. I want to be able to do whatever need to be done.

It is not necessary for the U54 to control all the funds, and you should not have to use money as a whip. Investigators like to have control of their own money, so it is better to farm out the funds. There is utility in linked awards because funds may be re-allocated, although there has not been a big shift in aims for us, so we have not had to do major re-allocations.

A few things have been confusing. For example, we were asked by a grant administrator from NIDCR who reviewed the U54 budget and progress report why we were not spending more money on the collaborative aspects of the project. When we told him that funds for collaboration were in each of the R01 budgets and reports, he told us that he did not have access to these reports, and that seems like a problem. Also, there appeared to be a policy in Year 1 that we had to prepare separate progress reports for each award mechanism although we requested to do a single, unified report. We have not experienced any problems with carry-forward funds. We did have a lot of back and forth on our RL5 for inter-lab training over the issue of paying fringe benefits for trainees. I think the issue itself was new to some people. It took some time to get the clear answer that these benefits could not be paid from the award, and so the University picked up the cost.

2.C. Other NIH mechanisms that could fund the IDRC program The PIs shared the opinion the administrative and research mechanisms were essential to the progress of the Consortia. They articulated their positions that the observed interdisciplinary interactions were made possible because of the mechanisms that brought scientists together.

Our Consortium is not R01able. Here is our challenge. Anytime you start to bridge studies that span disorders or species, it will not fit into an R01. We would never have gotten 52 investigators together if there hadn't been a big prize at the end, that is, $25 million in funding. We could possibly fit our Consortium into a Center grant. We have two huge cores that bind together these projects because we needed explicit sharing of samples and core services. We could continue to be structured as a consortium or a large center

NIH has to fund the central core. Without a central core, you would lose the data sets. You would not have the interdisciplinary team encouraging you and facilitating your work and giving rationales for their suggestions. If you present your research at a professional meeting, you present the good stuff, and this is what I would hear if I saw my current team once a year at a meeting. But you can advance the science by looking at the problems and finding the solutions. That is why we need to be able to bring the investigators together regularly and frequently,

16

  

whether in person or electronically. We have invested a lot of resources in intra-project communication. The administrative core moves the scientific agenda forward by bringing the people together.

Could R01s be the only mechanism? Yes, but we would not be making as much progress as we have made if we had only R01s. The U54 brought us coordination to expand existing research efforts and increase interactions. With the U54, we could bring together people who were working in different areas. With the U54, we can share resources through the viral core and have supplies that are synergistic.

These investigators could do work under R01s, but the administrative piece could not be funded elsewhere. We are further along because we had the infrastructure in place. The group of investigators wants this to move forward, not because they are getting more money than an R01, but with an R01 they wouldn’t have the benefit of the ideas that are generated across the projects. I may have been one of the most skeptical people about interdisciplinary work. Interdisciplinary projects have to have a reason, they can’t be forced. There needs to be a balance between interdisciplinarity and productivity. There is no substitute for getting a bunch of very smart people together in a room; you would think it is easy, but it isn’t. These larger umbrella grants help bring those people together. Without the administrative piece, everyone would still interact, but it won’t be as regular and it won’t be as productive. The cores will go away. There is no substitute for this. The seminar, administration, weekly progress – none of those would exist without the bigger program. It is impossible to predict how this program will be in 5 years.

In the absence of the IDRC, I don’t think there would have been an incentive for all these people to come together and think about the problem of facilitating our biomedical problem. It was a huge benefit for the NIH mechanism to bring people together. The scientists probably could have gotten funding for their individual projects but would not have come together to think hard about how to apply their discipline to the shared topic.

2.D. Self-assessment of leadership qualities 2.D. i. Preparation and satisfaction with PI role The Principal Investigators reported that they were at least moderately prepared, by temperament and experience, to assume leadership roles. Most reported that their roles have evolved and several expressed interest in team leadership training. On the whole, they were satisfied with the experience and were willing to accept such a leadership role again, although there were concerns that the administrative requirements competed with their own scientific productivity. Several spoke of the importance of sharing responsibilities with a Co-Principal Investigator or a Project Coordinator. Selected individual comments follow.

I was as prepared as many people would be. It is a challenge for anyone to be the U54 leader because no one knows all the disciplinary areas. That is why we have a Co-Director; he complements my knowledge base. I do have a broad background in physiology and other basic sciences, in addition to my expertise in clinical sciences.

17

  

It would be really difficult if I was a control freak, if I was trying to control science I didn’t understand. The investigators understand that I am not trying to interfere with their science. It helps to have a good sense of humor.

I had no specific training in the leadership qualities that are associated with facilitating team progress. I have had leadership experience, especially in coordinating research and working with the government. I was completely unprepared to take that experience and turn it around to leading friends and peers and even some people who were senior to me. There is no natural authority for me because I am not the department Chair or the Dean. I adopted a somewhat passive leadership style for the IRC because I came from the middle of the ranks. I did have a lot of currency because I put the whole proposal together. I have employed encouragement, cajoling, inspiring, and educating about the worth of participants to develop the team.

I have no formal training in leadership, although I have had a leadership role here as a Center Director. The U54 leadership role is different. The PI and I are both feeling our way. We were taking a hands-off approach but making sure that things were moving forward. Now we see that some different leadership skills might be needed. To get this project moving even more, we might have to acquire different skills. We have let things unfold, but the EAB has asked us to consider how we navigate the Consortium to be as productive as possible.

It has been a joy to have this Consortium, to be the director. It’s fun to see the breadth of the work. It has allowed me to develop closer relationships with investigators I hadn’t worked with before. Working with the post docs and grad students is great. I wouldn’t trade the experiences with them for anything.

Would I do this all again? Yes, for sure. We would not be performing the same quality of research right now without the benefit of the IDR consortium model. Assembling a critical mass of investigators from different scientific disciplines has altered our thinking and has significantly influenced the design and analysis of experiments. As a scientist, I feel that I am actively learning, and that I am able to apply knowledge that I obtain from scientific meetings to our own research.

Being the U54 PI does carry more of an administrative burden than I had anticipated.

It takes of lot of work, and I did not envision all the work. The thing that drove me to do it was the possibility of creating something with real impact. This is motivating and exciting. I have learned a lot. The first years are probably the hardest because we have the most human studies. Would I be the U54 PI again? This will depend on how the next few years go. Now I have two R01s and a P50 in addition to the U54. I love the scientific leadership piece but I love the science part too, and I want to pursue my own R01s.

2.D. ii. Generating and sustaining trust All U54 Principal Investigators endorsed generating and sustaining trust as an important personal quality for leaders of interdisciplinary research teams. They reported being strong on this quality personally. The most common descriptions of generating and sustaining trust included openness and transparency in budget and administrative matters, commitment to the team while valuing

18

  

the individual, initial careful selection of team members, listening and conflict resolution skills. Selected individual comments follow.

If people don’t think you can follow through with data sharing and resource sharing, the team will fall apart. Following through, having good communication, letting people see that principles will be enforced, making sure people get together and talk – these are all part of generating and sustaining trust.

It was important to me that we protect junior investigators from having others use their data. Our database itself protects one investigator from taking someone else’s ideas or data without written permission.

Everybody has to trust everybody. I have to be able to create an altruistic environment. Whatever it takes to move money or makes things happen for the Consortium, that is what I have to do. Our group also started with a lot of trust. One of the reasons we lost one member of the team was that he controlled his own money and was not as committed to the Consortium as to his own award. The autonomy of the individual scientist is supported.

I needed to build trust within the team from the beginning. I’m pretty good at generating and sustaining trust. I listen to people and respect their wishes, but I also have to balance individual’s wishes against the team’s needs. The most important quality is to be open and direct and not to play politics, so people can count on you.

Part of our institutional environment is that our organization has selected people who are comfortable in their skins and area of expertise. Our scientists are smart people with strong expertise who have a high level of self-confidence but who also recognize that other smart people work here, and this facilitates trust.

I agree that generating and sustaining trust is really important for me to be a good leader in team science. The Speech of Trust, by Stephen M.R. Covey, is a book that I keep on my desk.

I generally put decisions to the group and gain consensus. I make sure that discussions about resources are transparent. Also, when I know that specific topics are going to be discussed by the team leadership, I make sure that all the requisite parties are in the room. I do not support hidden agendas. It can be hard to tell people no when they ask for something, but at times it is best done directly.

2.D. iii. Cultivating a shared vision All Principal Investigators agreed that cultivating a shared vision was important to their success as leaders. Most implied that they started with a vision and then encouraged others to share their initial vision, but several spoke about developing and the vision as a team. Representative individual comments follow.

Cultivating a shared vision is important to team science. To promote this within our team, I first got them excited about the science and the possibilities. The topic interested them. I approached each person with the framework and then I saw whether they would be excited. I showed them

19

  

their potential unique contribution. Then I explained what the others would do. In steps, I got people on board. The fact that I am a scientist in their area was helpful; it would have been different if I had been a senior administrator. Sharing data also cultivates a shared vision.

I try to attend and to participate in every meeting. I see one of my key roles as making sure to inject the larger consortium vision into all meetings... Sometimes I feel that I am the main proselytizer for “the cause,” although the other team leaders certainly share a common vision. A shared vision implies shared responsibility for the success of the program.

This grant made this program a priority for the school. So when it is a priority of the school, there are going to be resources devoted, not necessarily to individual investigators, but to grow the program.

I try to surface ideas in an open way, as a group activity. My job is to figure out how to concretize these ideas. Through our team interactions we develop a shared vision of where the science can go.

2.D. iv. Bias toward risk taking and action Some but not all the Principal Investigators endorsed a bias toward risk taking and action as an important leadership quality. Several noted a distinction between risk taking as it applied to scientific work and risk taking as it applied to administrative matters. Several interpreted risk-taking as evidence of self-confidence, which was endorsed. One Principal Investigator spoke about an inherent conflict between leading by consensus versus leading by action. Selected individual comments follow.

This quality is not like the other two. I am not a real risk taker. I am a cautious risk taker. I do want to see data and logic behind the risk. I am definitely action-oriented, but I am not one who changes my mind about scientific direction all the time, which would be confusing and demoralizing.

I have a weird combination of scientific risk taking and fiscal responsibility. I want the science to be as exotic as possible, and I am willing to take risks. I am not risk-taking with money.

I am somewhat neutral about this attribute, which might be important for some tasks but not for others and which has two parts that often conflict. In our team, we have a bias toward consensus, which is sometimes antithetical to speed. I intentionally choose consensus over action sometimes, and leading by consensus can be antithetical to risk-taking. We are about innovation and execution. It is okay to accept risk for innovation, but it is not okay to accept risk in execution. In a team, the team leader has to be a risk taker on the innovation side but not on the execution side. On the execution side, for example, we minimize risk by having goals, objectives, and tasks articulated for each team and by having time-driven metrics.

I am biased toward risk taking and action. For example, I went after an NIH Challenge grant even though I knew only a few would be funded. But I was hopeful, and I knew that writing the proposal would frame our work. We could dust off the proposal and submit it elsewhere, but we

20

  

got the award. You need to balance between being rigorous and being adventurous. You have to be both to be transformative.

2.E. Self-assessment of methods for achieving important process goals for team science 2.E. i. Participatory goal setting. There was not agreement among the Principal Investigators about participatory goal setting as an important team process. Many of the disagreements revolved around the need to define aims and goals in the original research proposal and the project and NIH expectations of commitment to these initial goals. Representative individual comments follow.

We haven’t needed to do this yet. In a consortium where the science is going to move you in a new direction, you have to be able to do this, but we’re still in an exploratory mode, so we don’t have to do this yet. Also, participatory goal setting could be important at times, but it depends on your philosophy. The Consortium is not a corporation. I would really have to believe that there was a crucial scientific direction that we should be moving toward before I asked everybody to change directions. Usually, we don’t have enough information to make a radical change.

Most of the strategic planning happened during the six weeks of writing the pre-application. Our overall aims were done at the level of the pre-application and all the work done today is consistent with that initial planning.

Initially I got the team together to plan the project, but now there are some groups that do not set goals together. For example, what the clinical team does is different from what the other scientists are doing. Even within these groups, though, we are not sitting around and saying what goal we are trying to achieve.

To develop the initial proposal, we had an iterative process around the framework. Each PI came up with aims, given the framework. Then we would work with those aims as a team and brainstorm. We built science around the model. We had never had this kind of scientific interaction before. There was a lot of divergent thinking with this kind of team.

2.E. ii. Adequate feedback and communication This item was interpreted to include feedback to the U54 Principal Investigator and feedback from the U54 Principal Investigator. Most of the respondents talked about methods of communication between them and other members of the team and stakeholders. Individual comments follow.

We have frequent meetings. Our web work also promotes communication. I try to provide direction and provide opportunities. Team members receive a lot of affirmation from me, publicly and privately. When a difficult conversation is needed, it is held in a timely fashion and privately.

It’s clearly good to have this. Most of the feedback I get comes from the PIs, especially if it is negative. The postdocs give feedback about didactics to another faculty and we work on these

21

  

issues at our monthly administrative meetings. I get feedback from students too because I work with the students frequently. I spend a lot of time with them, am exposed to their work, talk with them, make suggestions to the PIs. We have regular pot lucks at my house. I get almost no feedback from the Dean or people here, but I think they are happy; they think we are great. We get feedback from the Community Advisory Board and the External Scientific Advisory Board.

I try not to jump in and solve problems. Rather, I try to surface each person's feelings, find out what is bothering him, and let him express his own solution. When people on the team come to me with issues, I try to figure out what they are thinking and make sure that the solution does not harm anyone else. I have the stakeholders work together.

I’ve given feedback individually if it was needed, but we have not had a lot of problems. I help the scientists feel ownership of the consortium and the team. I bring up problems respectfully and try to be direct. I also use the Executive Committee for feedback and communication. I have to work with each PI based on his or her style. The culture here is competitive and independent. Living with that reality can be challenging. Some components have stayed within their own teams more than others. It takes constant monitoring and pushing to keep people to work in collaboration, to show their data to others. Scientific leadership is critical and requires constant framing and reframing of issues.

This depends on how one views a consortium. A poorly defined area for us is authorship. Among senior leadership we have no problems because we are all invested in doing beautiful work. It is more of an issue for post-docs and graduate students who need first authorship to advance their careers. These issues have to be settled case by case, based on the individual situation, and it can be a challenge. I emphasize to trainees that the referral letters you get from a lot of other PIs as a post-doc are really important too. Referral letters lose meaning if they don’t reflect the reality of performance.

2F. Interactions with NIH staff Following is a summary of the comments. 1. Initially, there was confusion about specific policies and procedures. Differences between ICs in policies regarding funding were problematic. There was inconsistent guidance from different NI H staff. Clear lines of authority and communication were lacking. Most of these problems resolved over time. 2. Satisfaction with NIH grant administration was 3. Satisfaction with individual Program Officers was moderate. Most interactions involved administrative issues. In general, the Program Officers provided low but acceptable level of science guidance to the Principal Investigators. 4. Interactions with Program Officers might be improved if the Program Officers received some training in managing team science projects. 5. Moderate satisfaction with the initial overall Program Coordinator was observed and high satisfaction with the current overall Program Coordinator was observed. 6. Several Principal Investigator stated that NIH was not well prepared to support team science (cross-IC) initially, and they were uncertain about whether NIH is well prepared now.

22

  

Section 4. Observations of Scientific Meetings

Structured observations were conducted at seven of the nine Consortia. An informal observation was conducted at an additional Consortium for development of the observation instrument (see Appendix 4); that Consortium was not re-visited. The remaining Consortium had an extensive observation schedule as part of its local evaluation and thus was excluded from this part of the national evaluation.

Consortia were asked to select an event for observation that would showcase interdisciplinary interactions among investigators: criteria were that the main purpose of the meeting was for Consortium investigators to discuss the Consortium’s own research and that the persons attending the meeting should reflect the interdisciplinarity of the Consortium. The preference of the NIH Principal Evaluator was to attend a recurring meeting. Most of the Consortia had monthly or more frequent meetings that met these criteria. For one Consortium, the annual debriefing session of the Scientific Advisory Board with the Consortium Investigators was observed. For another Consortium, a meeting scheduled as part of an annual retreat was observed. Meetings observed were of one to two hours in duration. Participants either provided written consent to participate in the observation or were informed prior to the meeting that the NIH Evaluator was present. Upon request, the meeting participants received comments from the Evaluator at the completion of the observation.

The number of participants at the observed meetings ranged from 10 to more than 24, with 20 or more present at most of the meetings. Persons attending included primarily faculty but also trainees and staff from the Consortium in question. Four of the meetings included faculty or staff from the sponsoring institution who were not formally part of the Consortium. At two of the meetings, members of the Scientific Advisory Board were present. At one meeting, NIH officials in addition to the Evaluator were present.

To measure whether the Consortium faculty attending each meeting represented the interdisciplinary mix put forward in the initial grant proposal for that Consortium, the most important disciplines named in the proposal were matched to the faculty present. For example, the five significant disciplines named in one original Consortium proposal were neuroscience, behavioral science, human genetics, pharmacology, and molecular and cell biology. The primary discipline of each faculty member who was proposed as a Consortium investigator was also named in the proposal. For six of the Consortia observed, five distinct disciplines were named; one Consortium named four disciplines. For five of the seven Consortia, faculty representing the named disciplines were present. For one Consortium, there was not representation from a behavioral pediatrician, although that was one of the key disciplines named. For another Consortium, three of the five most important disciplines were represented; absent were an oncologist and a medical ethicist.

Discussion topics were documented during the course of the meetings. Research design was discussed at all meetings, translational aspects of the research were discussed at six, and data analysis and research results were each discussed at five meetings. Research resources were

23

  

discussed at three meetings, data sharing at two, and manuscript development and publications or presentations were each discussed at one meeting.

In regard to apparent outcomes of the meeting, the number of Consortia for which the investigators stated specific intentions follows: • 6/7: modification of research design for an existing study series; • 5/7: resource sharing; • 4/7: follow-up with a new study from outside the investigator’s discipline; • 3/7: follow up with new study from the investigator’s own discipline.

The scientific literature about interdisciplinary research suggests that interdisciplinary actions differ in specific ways from other interactions (Reference). Following are five behaviors that have been hypothesized to characterize interdisciplinary interactions and their frequencies of occurrence during the observations: • 6/7: Unexpected findings • 6/7: Task orientation • 5/7: Divergent point of view • 3/7: Analogous thinking • 3/7: Supportive of innovation

Finally, the Evaluator provided global impressions of the interactions observed during the meetings on ten criteria hypothesized to be relevant to interdisciplinary interactions (Reference), using a 1 to 7 rating scale, with a higher score indicating a more positive rating. Global impressions were positive for all criteria, with the highest rating (6.7) observed for level of engagement, followed closely by a high rating for multidisciplinarity in the interactions.

Table 4: Global Impressions of Scientific Meetings of Investigators Dimension Rating scale Average

rating Low engagement/high engagement 1 2 3 4 5 6 7 6.7 Uni-disciplinary/multidisciplinary 1 2 3 4 5 6 7 6.3 Conflicted/harmonious 1 2 3 4 5 6 7 5.9 Speaker-dominated/100% participated 1 2 3 4 5 6 7 5.9 Competitive/cooperative 1 2 3 4 5 6 7 5.7 Isolated/cohesive 1 2 3 4 5 6 7 5.7 Unsupportive/supportive 1 2 3 4 5 6 7 5.4 Scientifically fragmented/ scientifically integrated

1 2 3 4 5 6 7 5.3

Not enjoyable/enjoyable 1 2 3 4 5 6 7 5.1 Non-productive/productive 1 2 3 4 5 6 7 5.0

24

  

Section 4. Investigator Surveys

A survey was distributed electronically to the 122 senior investigators (PI or Co-PI level) in the Consortium program, 105 of whom completed it. The survey includes scales relevant to team leadership, team science climate, interdisciplinary activities, and social network analysis. An overall 86% response rate was observed, and the response rates by Consortium follow. Please see Appendix 8 for the report prepared by Abt Associates. Selected findings follow.

Table 5: Percentage of respondents strongly agreeing or agreeing with each statement about team leadership.

Our Consortium PI(s) were successful in … Strongly agree

Agree

supporting innovative projects 66 % 24% cultivating a shared vision 56 35 generating and sustaining trust 56 32 distributing resources equitably 50 31 providing adequate feedback and communication 48 39 setting goals through team participation 45 42 making financial management transparent 41 39 structuring interdependence of members' tasks 39 40 resolving conflicts 37 40 structuring interdependence of members' rewards 35 34

Investigators gave overall favorable ratings of team leadership. The most positive ratings of their U54 PI(s) were observed for supporting innovative projects, followed by cultivating a shared vision, and generating and sustaining trust. The least agreement, although still high, was for structuring interdependence of members’ rewards and tasks and resolving conflicts.

25

  

Table 6: Percentage of respondents reporting engagement in specific scholarly activities within and outside their primary discipline during the last six months.

Within discipline

Outside discipline

Difference (within – outside)

Established links with colleagues that led to or may lead to collaborative studies

96 96 0

Were ready to collaborate with other investigators 99 98 1 Obtained new insight into your research through discussions with colleagues

96 92 4

Worked on research projects with other individuals

99 94 5

Designed a new collaborative study with colleagues

87 82 5

Read science journals or publications 100 94 6 Co-authored research articles or books with other individuals

90 76 14

Co-authored research proposals with other individuals

87 67 20

Participated in working groups or committees 93 71 22 Published research findings in a journal 91 65 26 Mentored graduate student(s) 79 45 34 Attended scientific meetings or conferences 97 63 34 Presented research findings at a scientific conference

90 52 38

All respondents endorsed high levels of scholarly activities within the last six months, consistent with their status as active, senior investigators. Scholarly activity within their primary disciplines ranged from 78% to 100% and outside, from 45% to 98%.

26

  

Table 7: Percentage of respondents strongly agreeing or agreeing with each statement about team science climate.

strongly agree

agree

Our investigators were committed to interdisciplinary collaboration 76% 19 % The consortium environment was collaborative rather than competitive 72 19 The complementary research interests and expertise in participating laboratories facilitated collaboration

70 24

Intellectual contributions made by more than one investigator were valued 70 28 The consortium leadership facilitated collaboration 68 24 Investigators were exposed to divergent points of view that positively affected thinking or research

64 31

Communication structures and processes facilitated interdisciplinary collaboration

62 28

Unexpected findings served as a source for new ideas and new directions for research

62 30

Investigators used cross-disciplinary analogies to communicate ideas or generate solutions to scientific challenges

55 35

Electronic connectivity of members' offices or labs facilitated collaboration 54 34 Work on high risk, high reward projects was encouraged 50 29 Physical proximity of members' offices or labs facilitated collaboration 35 25

Overall, the investigators reported positive perceptions of the team science climate at their Consortium. More than 95% agreed or agreed strongly that their investigators were committed to interdisciplinary collaboration, that intellectual contributions made by more than one investigator were valued, and that investigators were exposed to divergent points of view that positively affected thinking or research. The lowest level of agreement was about the physical proximity of offices or labs in facilitating collaboration. Because of the survey wording, we cannot determine whether the respondents did not agree because the offices and labs were far apart or because the proximity did not facilitate collaboration.

27

  

Section 5. Trainee Interviews and Rating Scales

Forty-two post-doctoral (PhD, MD) trainees were interviewed from six Consortia. Although pre-doctoral trainee interviews were also conducted and ratings were obtained from pre-doctoral students, the current report excludes data from the pre-doctoral trainees. Data collection instruments are included in Appendix 4. The interview guide and the rating scales provided the pilot work for an electronic survey of trainees who were not part of the Consortium program but were funded under other interdisciplinary training programs under the Common Fund. The final report, prepared by Abt Associates, of these electronic surveys is presented in Appendix 9.

5.A. Characteristics of the participants. Post-doctoral trainees from six of the nine Consortia were interviewed. The number of trainees from each Consortium ranged from three to eleven. At one site, the eleven participants were interviewed as a group; individual interviews were conducted elsewhere. Two held MDs; the remainder held PhDs. Their PhDs were from the basic sciences (biology, chemistry), behavioral and cognitive sciences (clinical psychology, neuroscience), and interdisciplinary fields (computational biology, molecular neurobiology). Seventeen had participated in a prior post-doctoral training experience. Most intended to pursue careers in academics upon completion of their post-doctoral training but many also were interested in careers in industry (pharmaceuticals, research institutes).

The primary reasons for choosing a traineeship with a Consortium follow: • To work with a specific investigator, N=27; • To gain an interdisciplinary training experience, N=15; • Consortium activities matched trainee’s research interests, N= 14.

5.B. Structural features of the interdisciplinary training experience. Trainees were asked to describe the interdisciplinary features of their Consortium training programs. Following is a list of the typically reported features: • Dual mentorships; • Structured interactions with senior investigators (seminar series, progress report

meetings) re: specific research projects, including trainee’s projects; • Informal interactions with senior investigators (accessibility); • Training outside trainee’s main lab (shared resources) on specific equipment or specific

research methods; • Lab meetings outside trainee’s main lab; • Structured reading and learning beyond trainee’s main field; • Contribution of knowledge and expertise from trainee’s main field to trainees and

investigators in other fields.

When asked to compare the frequencies of three different kinds of interactions with their mentors, the trainees endorsed formal discussions of the trainee’s own research project as the most frequent, formal interdisciplinary training experiences as the second most frequent, and formal collaborations (writing a paper or research proposal) as the third most frequent interaction.

28

  

5.C. Scholarly experiences and scientific productivity. Participants were asked to rate the frequencies of specific scholarly activities both during their Consortium experience and in the two years prior to their involvement with the Consortium. The largest increases in specific interdisciplinary activities during training follow: • Team research projects (80% now; 48% prior) • IDR outside a course setting (68% now; 30 % prior) • Courses with an IDR focus (64%, 36%) • Mentoring by faculty in multiple disciplines (77%, 50%) • Courses outside home department (57%, 32%)

In terms of scientific productivity during the training period, 22/42 (52%) of the post-doctoral trainees had shared authorship on a paper accepted for publication by a peer-reviewed science journal based on their work in the Consortium. Thirty (30/42, 71%) had either published or presented research findings at a national conference. Eleven (11/42,(26%) had written and submitted, usually as a member of a team, a grant proposal.

When presented with a list of possible positive and negative features of interdisciplinary research, the trainees expressed the strongest agreement with the following items: • IDR exposes me to new scientific approaches and paradigms; • IDR is a better was to explore a biomedical problem than single discipline research; • I am optimistic that IDR will lead to valuable scientific outcomes that would not have

occurred without that kind of collaboration.

5.D. Following are selected trainee responses (in italics) to specific questions.

5.D.i. What are the advantages of interdisciplinary training? I had an experience in the past, before coming to this Consortium, that was not interdisciplinary. We had a seminar that was supposed to be IDR, but it was single discipline; people would come and ask low-level questions. Here, people take the time to learn methods and constructs of other’s disciplines so that time is saved when we come together. Collaboration is more than sending someone a data set, which is what some people mean by collaboration. Here it is much more than that. My mentors had a relationship before I came.

The greatest advantage the training gave me was a crash course on Who’s Who in this institution. Our monthly work in progress meetings are the single greatest advantage. If I have a question or a scientific idea to discuss, I knew who to go to. The meetings gave me instantly a map of who to go to. That’s how the collaborations got started: I knew who to go to, how their work fit into my research.

I observed several differences between my first post-doc experience and my Consortium experience. Analytical chemists do not typically go into academia. Also, we are much more focused on labwork with the current post doc. Here, I took classes in developmental biology and grant writing that have been invaluable. I have been amazed by the willingness of the

29

  

investigators here to encourage and help me. I was encouraged to attend an international conference, and conference money was available for me.

You can go so broad, get into so many areas. I hadn’t typically looked at developmental psychology, but it came onto my radar here. Serendipitously, I came across other areas. I came here to learn fMRI, and I was also a little interested in [a specific topic]. When I got here, I started working with the U54 PI and now I can develop my work [on this topic].

5.D.ii. What impact is this interdisciplinary training likely to have on your career? I don’t think it will be difficult for me to find a job. I have friends and colleagues who have stayed in one field, so I can compare. I haven’t figured out yet just where I want to be. An IDR traineeship could be a problem if all my publications are not in the core journals of one field, but things are changing and I think I will find a place. If a department wanted someone who had published in the core journals only, I wouldn’t be attractive to that department but I wouldn’t want to go there anyway. Once I have gotten my own funding (K award) and maybe an R01 and articles in multiple journals, I should be able to find a job.

The interdisciplinary training will definitely help me. Before I was here, during my PhD studies, I always thought of doing a more focused study. Interdiscplinary training really helped me think about other approaches, how to apply techniques at a systems level. The translational aspects are also more apparent with an interdisciplinary approach. You can determine if the approach is right. This training really changed my way of thinking about what I want to do in my own lab, about the direction I want to go in. To make our lab interdisciplinary, we use transgenic animal models to study certain human disorders. We use electrophysiology to see what is going on in neurons of lab animals. It will be easy to find a university willing to support this kind of research.

I believe I am being prepared to be hirable by multiple departments (Internal Medicine, Pharmacology, Psychiatry). I am most prepared to teach cell signaling and endocrinology and less prepared in pharmacology and neuroscience. I am better prepared in terms of diverse skills (ability to frame a research question, employ specific research techniques, interact with a variety of investigators) than someone trained in a single discipline.

This Consortium is perfect for me because I was already committed to interdisciplinarity. The Consortium did not help me much with networking because I knew and was working with the Consortium people before I became a trainee, but I have met a lot of other investigators. I am in the middle of two disciplines, and this makes it harder for me to get a job. Interdisciplinary training is better for science but harder for me because I will need a joint appointment. The Consortium helps people who are in the biomedical field but not in the natural sciences.

I am looking for a job in academia now, but I am not well prepared to teach an undergraduate core course. It might have been better to go for a mainstream engineering degree and do a dissertation on bioengineering. There is no way to get teaching experience within the Consortium because the focus is on research. A biomedical PhD might be too specialized, an amalgamation.

30

  

The biggest advantage of the Consortium is the connections I am making for future collaborations. I think these connections will help me. I would like to be on the Intramural track at NIH. I have acquired lots of skills and should be attractive to multiple employers. As a researcher, if I can’t get one area of research funded, maybe I can get another area funded. It’s like being an ENT surgeon; if you can’t do rhinoplasties, you can do sinus surgery. I like to be a jack of all trades and enjoy knowing a fair amount about a lot of different fields.

The traineeship helps me. I am a post doc, so someday I have to write my own grant, be innovative. Because the training is interdisciplinary, there’s less likelihood that my research proposal will be outdated. There is lots of material to help me write a grant. I will be looking for a position in psychiatry or clinical psychology but still connecting with radiology, endocrinology, and genetics.

5.D. iii. What message would you like to give Dr. Collins about your training experience? I came here from a top tier research institution, but even in the simplest sense, in a single department, nothing came close to this. It boggles my mind. At my PhD institution, genetics stopped at the association. Here it goes on to the person, there is a clinical application.

You walk down the hall at any given time, and you will see investigators talking to each other. Everybody is on a first name basis. People from all different areas of campus come together every Thursday. It’s remarkable.

I love what I do and the people I work with. In the past, I dreaded having to do research, but my desire to do research started while I was working on my PhD. Now all I want to do is ask questions. I am always looking to do more work. My desire and passion to do work in all different areas of genetics was initiated by and developed with this Consortium.

It is critical that NIH fund interdisciplinary research training. To have this kind of training result in better career opportunities, there should be some kind of effort to sensitize the larger world about the advantages of hiring trainees like me who have broad, rather than deep, expertise. The trainees here are set to be research leaders. Leadership has to be taught, trainees have to be taught to be like our U54 PI. NIH has to have a vision of how the interdisciplinary research trainees are unique and will fit into the broader research community. It would be fun to be the head of a research institute where your whole job was to bring researchers together around a single topic, broadly defined, and facilitate their research.

31

  

Section 6. NIH Staff Interviews

Ten Program Officers were selected by the NIH Interdisciplinary Research Consortium Program Coordinator and invited to participate. All accepted and were interviewed by the (contracted evaluator) Project Director or staff. Individual, one hour guided interviews were conducted, following an interview guide developed by the NIH Principal Evaluator and the Abt Associates Project Director.

The complete report is contained in Appendix 10. Data are organized around the following topics: • Differences between managing interdisciplinary and single discipline grants • Training to manage interdisciplinary research (IDR) • NIH policies and procedures for managing IDR • Experience with Program Coordinators • Administrative issues with RL1 grants • Managing projects outside home IC • Alignment of IDRWG initiatives with home IC ‘s mission • Establishment of new collaborations • Predicted outcomes and long-term impact of IDR • Transitioning projects from Common Funds • Opinions about successful IDR

Concluding remarks from the report follow: “Overall, most POs think IDR is a novel and innovative approach to solving complex problems and agreed that the scope and scale of such projects would not be possible without cross-institute collaboration and use of Common Fund resources. The large majority are hopeful that the Interdisciplinary Research Consortium program will continue to be a trans-NIH focus; however, many provided suggestions and comments of ways that the program could be improved to facilitate management within NIH, as well as externally for the Consortium Principal Investigators, which could ultimately lead to better science. As one Program Officer emphasized, Roadmap has been a ‘great demonstration project; and now we really need to institutionalize it.’ ” (p. 12)

32