from logic model to data model: real and perceived barriers to research assessment
Post on 26-Jul-2015
69 Views
Preview:
TRANSCRIPT
Real and Perceived Barriers to Research Assessment
Parallel Session 1.2
ORCID-CASRAI JOINT CONFERENCEBARCELONA, SPAIN
18 MAY 2015
FROM LOGIC MODEL TO DATA MODEL
22015 ORCID – CASRAI Joint Conference Barcelona, Spain
32015 ORCID – CASRAI Joint Conference Barcelona, Spain
Barriers to Research Assessment
1. Program managers are not familiar with evaluation concepts or do not have the capacity to carry out evaluation
2. Evaluation is not top-of-mind at the program design stage
3. Insufficient operating budget is available to carry out evaluation
4. Data to support evaluation are not ready-made
5. Burden on the R&D community to support evaluation
42015 ORCID – CASRAI Joint Conference Barcelona, Spain
NIH/NCI/CSSI/OPSO
Established in 2009
$150M, $10M U54 Research
Grants
Physical Sciences – Oncology Centers Program (PS-OC)
• To unite the fields of physical science with cancer biology and oncology
• To develop trans-disciplinary teams and infrastructure
• To generate new knowledge and catalyze new fields of study
Program Goals
• Twelve centers were funded 2009-2014, U54 research grants
• 150 main investigators from the fields of physics, mathematics, chemistry, engineering, cancer biology and clinical oncology
• 110 Institutions involved across the US
PS-OC Network
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 5
6Confidential
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 7
Evolution of data available for to support PSO program management
Manual data extraction, organization, deduplication and visualization in Excel
Difficult to track individuals’ contributions over time
No ability to collectively search progress reports
Data are entered through an intuitive web-based interface
Identify data relationships
Data deduplication
Flexible, unified search
On-demand Bar, pie, and line charts and network graphs
One-click tabular exportfor data behind graphs or “report card” tables
Data can be visualized by network, center, person or time, for a total of over 100 possible charts and graphs
At-a-glance view of research output• Personnel +2,900• Publications
+2,300• Collaborations +1,900• Conferences
+4,200• Workshops +400
Manual Data Analysis Before
iTRAQR
Automated Data Analysis With iTRAQR
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 8
Deduplication
1. Collaboration2. Course3. Funding4. Meeting5. Patent6. Publication7. Training
Transition8. Workshop9. Exchange10. Project11. Person
The value of structured data: clarity, communication and change
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 9
Lessons learned for overcoming barriers
1. Start with your evaluation logic model and translate it to your data model
2. Think about the level of analysisa. Analyzing subprojects activities at outputs would have been impossible
without iTRAQR
b. Understanding people involved beyond key personnel
3. Data structure is key
4. Have a flexible approach to evaluation (adjust based on findings using initial data)
102015 ORCID – CASRAI Joint Conference Barcelona, Spain
Individual-Level• Publications• Patents• Grants (NIH, other)• Science Awards (innovative,
translational, training)• Clinical Trials• Conference presentations• Courses and workshops taught• Trainee disciplines
Center-Level• Cost, content and people
involved in research projects, pilot projects and cores
• Stage, content and people involved in collaborations
• Datasets, techniques, technologies and bio-specimens generated and utilized
• Enumeration and content of transdisciplinary team science activities
Network-Level• Cost, content and people
involved in trans-network projects and outside network pilot projects
• Stage, content and people involved in collaborations
• Datasets, techniques, technologies and bio-specimens generated and utilized
• People and centers involved in trainee exchanges
• Location and content of outreach activities
Inputs and Activities Outputs Outcomes Relative to Comparison Groups
Generated Robust Collaborations that Resulted in Significant Transdisciplinary Research • Accelerated the formation of a greater quantity of transdisciplinary
collaborations• Accelerated the creation of a greater quantity of field convergent research• Communicated effectively across disciplines to form optimal team sizes• Effectively contributed to team based activities and outreach
Connected Physical Sciences Perspectives with Clinical Research• Accelerated the formation of a greater quantity of collaborations among
physical and physician scientists• Reduced the time between the appearance of a physical sciences perspective
or technology to its application in translational research• Acted as key investigators leading a convergence of physical sciences
perspectives within translational research and motivating transdisciplinary translational research
Bridged Oncology Research Gaps• Accelerated the generation of innovative and impactful transdisciplinary
solutions to outstanding questions in oncology (e.g. integrated transdisciplinary datasets, technologies and bio-specimens, prominently positioned in citation networks and commercialized cancer-relevant patented technology)
Trained a New Generation of Transdisciplinary Scientists• Conducted a greater quantity of transdisciplinary training activities• Attracted a greater volume of training grant applications to the PS-OC
program• Graduated a greater quantity of transdisciplinary scientists
• Accelerated the trainee development path toward a career in physical sciences-oncology
Generated a Sustainable Transdisciplinary Infrastructure• PS-OC alumni sustained a transdisciplinary perspective by integrating team
science into their infrastructure and attracting new investigators to the field • Motivated the formation of other inter-/intra- national programs promoting
physical sciences perspectives in cancer research
PS-OC Program Logic Model: Dec 2013
Network-Level• Coordinate Expertise
Trans-network Projects Physical or virtual
infrastructure Integrative training Data Coordinating Center Research Contracts to further
support clinical translation, cross-validation and integration of datasets, techniques, technologies, bio-specimens
• Communicate with PS-OC and Broader Research Community
Center-Level• Primary leading physical
scientist and cancer researcher
• Research framework: 3-5 projects
• Shared Resources: 1-3 non-redundant core facilities
• Pilot Projects • Transdisciplinary lectures,
workshops, working groups, courses
Individual-Level• Research findings: pre-award
publications, grants, patents, clinical trials and business development
• Research discipline• Organization associations
(location, Title/Rank, department)
• Degrees received• Other demographics
Evaluation is only as good as the data available
• Program design and management is enhanced by early evaluation design and evaluation efforts
• Data are not infallible and should be part of a holistic evaluation approach as well as close engagement with program participants
• Data do not exist today to measure all of your program goals
• Careful consideration for what actions can be taken following the evaluation should help to prioritize data collection
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 12
Acknowledgments
Nicole Moore, ScD Program DirectorNCI Division of Cancer BiologyPhysical Sciences-Oncology
Unni Jensen, PhD Sr Scientific AnalystThomson Reuters
Jodi Basner, PhD Scientific AnalystThomson Reuters
132015 ORCID – CASRAI Joint Conference Barcelona, Spain
THANK YOU
Systems to link research to outputs and outcomes
152015 ORCID – CASRAI Joint Conference Barcelona, Spain
top related