educating future clinical research investigators educating future clinical research investigators :...
TRANSCRIPT
AN 8-WEEK CLINICAL RESEARCH
EDUCATION COURSE FOR NEUROLOGY RESIDENTS
Educating Future Clinical Research Investigators:
Jeanie McGee, MSHS, CCRC
Acknowledgements
Dissertation ChairWarren McDonald, PhD
Dissertation CommitteeAlireza Minagar, MD, FAANRebecca Nolan, PhD, MP
Special ThanksRobert Schwendimann, MDDebra E. Davis, MD
Background
The physician-investigator has been labeled as an endangered species (Armstrong, DeCherney, Leppert, Rebar, & Maddox, 2009; Goldhammer et al., 2009; Guelich, Singer, Castro, & Rosenberg, 2002; Johnson, Subak, Brown, Lee, & Feldman, 2010; Lowe et al., 2007; Raghavan & Sandham, 2007; Siemens, Punnen, Wong, & Kanji, 2010; Teo, 2009; Wyngaarden,1979)
Lack of research fundingLack of time and interestRegulatory requirementsLack of adequate exposure, education, and
training in research during residency (Armstrong et al., 2009; Clancy, 2008; Krousel-Wood, 2006; Teo, 2009)
Background (cont.)
Consequences:Delays in translational researchIndustry hesitance to invest time and moneyDecrease in competent and effective
mentorsDevastating effects on patient outcomes
Review of the Literature Medline (EBSCOHost) (Exploded with MeSH options)
Pubmed Google Scholar 2002 to 2012 (some historical literature) Terms:
research education, residency research education, clinical research education, self-perceived competency, residency education, and neurology residency education
Research during residency is a critical topic. Limited evaluation of research education during
residency, especially neurology
Review of the Literature (cont.)
Institutional neurology residency program accredited in 2005
3 residents have participated in research elective (out of 27 graduated)
Mostly used as study time, so revised to require research plan with specific topic and protocol – 0 residents participated since
(Dr. Davis, personal communication, August 28, 2012)
Problem Statement
This researcher will develop, implement, and evaluate an 8-week clinical research educational course for residents in a neurology department at a Louisiana university medical center to prepare neurology residents for participation as clinical research investigators by increasing knowledge of fundamental clinical research methodology, design, and analysis, as well as by increasing the residents' self-perceived competency in these areas.
First Major Aspect of Problem Statement
Improving the quality of patient care by filling the research education gapPhysicians with research interests and skills
needed for successful translation of new treatments and therapies
Solution: Require research education during residency training
Second Major Aspect of Problem Statement
National recognition of the problem National Institutes of Health (NIH)
○ Clinical Research Training Program – 1997○ Medical Research Scholars Program – 2012○ Resident Research Career Day - 2012○ Office of Clinical Research Training and Medical Education (OCRTME)
Summer internships, research electives for medical students, residents, fellows, and high school students
United States Food & Drug Administration (FDA)○ Critical Path Initiative - 2009○ Clinical Investigator Training Course – 2009
Accreditation Council for Graduate Medical Education (ACGME)○ Require curriculum that advances “"residents' knowledge of the basic
principles of research, including how research is conducted, evaluated, explained to patients, and applied to patient care" (ACGME, 2010, p.14).
Solution: Incorporate research education at the local level
Third Major Aspect of Problem Statement
Research education in neurology residencyOnly 10% of ACGME- accredited neurology
residency programs have a research requirementOver 50% of neurology residency programs require
research, but offer no specific rotation or educational curriculum, nor protected research time
If available, research rotation has shortest average rotation time (mean = 0.3; range = 0 – 2 months) (Schuh et al., 2009)
Solution: Implement a mandatory, structured clinical research curriculum for neurology
residents
Fourth Major Aspect of Problem Statement
Extinction of the clinical research investigatorDownward trend since late 1970s (Wyngaarden, 1979)
Currently, only 4% of physicians engage in research activities (CISCRP, 2012)
Need to preserve the integrity of clinical researchCNS therapeutics are the most prevalent area
addressed by pharmaceutical research (second only to oncology and immunomodulators, which are also common neurological therapies) (CISCRP, 2012)
Solution: Promote interest and encourage clinical research activity involvement during neurology residency
Fifth Major Aspect of Problem Statement Effects of research education for residents in order to build
research competency 87% of physicians believe statistics knowledge is important in patient
care delivery, but only 17% report their statistics knowledge to be sufficient (West & Ficalora, 2007)
88% report biostatistics and a strong research design foundation is necessary to effectively and accurately appraise published literature (West & Ficalora, 2007)
75% of medicine residents reported they understand all statistics reported in the literature, while their mean score was 41.4% correct responses when tested on statistical concepts found in the literature (Windish, Huot, & Green, 2007)
Solution:
Implement curriculum that includes evaluation that effectively assessing discrepancies in self-assessed
competency and actual
level of knowledge
Sub-Problems
Sub-Problem 1.Do neurology residents who have taken a prior elective or required clinical research course(s) have a higher level of self-perceived competency in fundamental clinical research methodology, design, and analysis than neurology residents who have not taken a prior elective or required clinical research course(s) as determined by correlating results of a pre-test measure of self-perceived competency in fundamental clinical research with resident prior exposure to research education?
Sub-Problems
Sub-Problem 2.Does the neurology resident’s future career plans have an impact on their level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Sub-Problems
Sub-Problem 3.Does the resident’s age influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Sub-Problems
Sub-Problem 4.Does the resident’s gender influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Sub-Problems
Sub-Problem 5.Does the resident’s post-graduate year level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Purpose of the Study
The purpose of this dissertation research was to evaluate the effectiveness of an 8-week clinical research education course designed for neurology residents. The research curriculum was designed to include topics significant to research methodology, design, and analysis.
Study Goals
to provide neurology residents in a single department at a university medical center in Louisiana information to increase the knowledge level of residents in the topic areas of research methodology, design, and analysis
Study Objectives (1) measure baseline levels of research knowledge
and residents' self-perceived level of competency in research,
(2) measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention,
(3) evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research, and
(4) determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure
Theoretical Framework
Cognitivist Approach and Competency-Based Model Jerome Bruner’s cognitivist approach
Self-discoveryLeveled learningPrior knowledgeInformation acquisition (derived from existing
knowledge)Information transformation (processing and analyzing
new information)Information evaluation (self-assessment of information
accuracy)
Theoretical FrameworkCognitivist Approach and Competency-Based
Model G. E. Miller’s framework for clinical
assessmentDoes
(Action)
Shows How (Performance)
Knows How
(Competence)
Knows
(Knowledgeable)
Research Design
Quantitative one-group pre-test post-test Dependent variables:
Resident level of knowledge in clinical research methods, design, and analysis
Resident self-perceived competency in clinical research methods, design, and analysis
Sample:Neurology residents in PGY II, III, and IV
training
Curriculum / Intervention Weekly sessions for 8 consecutive weeks 1 hour Session topics:
Introduction to Research: Basic Research Concepts and Terminology
The Research Question and HypothesisResearch Study Design and Sampling Introducing Variables and Scales of Measurement Introduction to BiostatisticsResearch Ethics, Privacy, and Responsible Study ConductBasic Regulatory Requirements for Clinical ResearchReview of Research Methodology, Design, and Analysis
(Overview)
Methodology Measure baseline levels of research knowledge and
residents' self-perceived level of competency in research
Measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention
Evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research
Determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure
Methodology (cont.) Measure baseline levels of research knowledge and
residents' self-perceived level of competency in research
Measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention
Evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research
Determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure
Instruments Developed by the researcher:
Evaluation Tool to Capture Neurology Resident Descriptive Data demographic assessment tool
Self-Assessed Level of Knowledge and Competency in Clinical Research (pre-assessment) (38 items – score 1 to 4 from least competent to most competent)
Level of Knowledge in Clinical Research Methodology, Design, and Analysis (pre-assessment) (50 multiple choice items with a possible score = 100%)
Self-Assessed Level of Knowledge and Competency in Clinical Research (post-assessment) (38 items – score 1 to 4 from least competent to most competent)
Level of Knowledge in Clinical Research Methodology, Design, and Analysis (post-assessment) (50 multiple choice items with a possible score = 100%)
Instruments
Considerations in the design of instruments:American Academy of Neurology (AAN)
evidence-based medicine Association of Clinical Research Professionals
(ACRP) Certified Clinical Research Investigator mock examination
Miller’s pyramid of competence
Data Collection
Collect demographic data Collect baseline self-competency data Collect baseline level of knowledge data Implement curriculum intervention Collect post self-competency data Collect post level of knowledge data
Data Analysis
Data coded and entered into Microsoft Excel Cleaned, verified Data moved to SPSS v. 21 for statistical
analysis
Predictions1. Residents with prior research education will have
higher mean scores on both knowledge and self-perceived competency at baseline
2. Residents with future career plans involving research and/or academia will have higher mean scores on both knowledge and self-perceived competency at baseline
3. Higher PGY levels will have higher mean scores on both knowledge and self-perceived competency at baseline
4. Male residents will have higher mean scores on both knowledge and self-perceived competency at baseline
5. Older will have higher mean scores on both knowledge and self-perceived competency at baseline
Data Analysis Descriptive statistics
Average age = 35.5 yearsPGY Levels:
○ PGY2: 30% (n=3)○ PGY3: 40% (n=4)○ PGY4: 30% (n=3)
70% reported prior research education (only 1 during medical school or residency)
Length of time of prior education: range = less than 1 week (n=3) to three years or more (n=1) (others were less than one month (n=1), 3m to 6m (n=1), 6m to 1y (n=1)
Most common topics:○ Developing and writing research question and hypothesis
(n=5)○ Research study design and sampling (n=5)
Data Analysis (cont.) Descriptive statistics
20% (n=2) reported career plans involving clinical research
20% (n=2) reported career plans involving basic science research
20% (n=2) reported career plans involving writing and publication
30% (n=3) reported career plans involving work at an academic medical center
20% (n=2) reported career plans NOT involving any type of research
30% (n=3) have not yet decided whether or not they will engage in research after residency
Data Analysis (cont.) Self-assessed Competency at Baseline
Mode = 1 “knows about the skill or concept or has heard about it, but is unable to do it”
Mode range 1 -2:○ “knows about the skill or concept or has heard
about it, but is unable to do it”○ "knows how to do a skill/concept, but has never
done it and is unable to show or explain how to do it“
No item had a mode of 3 or 4 at baseline ○ 3: “knows how, can show or explain how to do it, but has
never done it”○ 4: “knows how, has done, and can do at any time”
Data Analysis (cont.) Self-assessed Competency at Baseline
Possible total score = 152 (all 4s, representing highest level of competency)
Percentage scores calculated, mean = 63% at baseline (range = 42 – 125)
Data Analysis (cont.) Level of Knowledge at Baseline
Mean score = 53 (range = 36 – 70)
Resea
rch
Term
s
Resea
rch
Quest
ion &
Hyp
othe
sis
Study
Des
ign &
Sam
pling
Variab
les &
Sca
les o
f Mea
sure
men
t
Statis
tics
Ethics
, Priv
acy,
Res
pons
ibility
Regula
tory
Req
uirem
ents
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
52.546.7
53.8 48.845.6
61.7 62.0
Sub-Problem ResultsSub-Problem 1. Do neurology residents who have taken a prior
elective or required clinical research course(s) have a higher level of self-perceived competency in fundamental clinical research methodology, design, and analysis than neurology residents who have not taken a prior elective or required clinical research course(s) as determined by correlating results of a pre-test measure of self-perceived competency in fundamental clinical research with resident prior exposure to research education?
Pre-Knowledge Mean Pre-Competency Mean Percentage0
10
20
30
40
50
60
44.748.6
56.651.1
No Prior Research Educa-tion
Prior Research Education
Had an effect, but did not reach statistical significance.
Sub-Problem Results
Sub-Problem 2. Does the neurology resident’s future career plans have an impact on their level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Self-perceived competency: Career interest involving clinical research (F=9.142; p=.016)* Career interest involving basic science research (F=8.652; p=.019)* Career interest involving writing and publishing scientific
manuscripts (F = 8.652; p = .019)*
Level of knowledge: No statistically significant effect
Sub-Problem ResultsSub-Problem 3. Does the resident’s age influence level of knowledge in
fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Self-perceived competency: No statistically significant effect
Level of knowledge: No statistically significant effect
Sub-Problem ResultsSub-Problem 4. Does the resident’s gender influence level of knowledge
in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Self-perceived competency: No statistically significant effect, but males had a higher mean (67,
compared to 63 for females)
Level of knowledge: No statistically significant effect, but males had a higher mean
(54.57, compared to 49.33 for females)
Sub-Problem ResultsSub-Problem 5. Does the resident’s post-graduate year (PGY)
level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Self-perceived competency: No statistically significant effect
Level of knowledge: No statistically significant effect on overall score, but statistics
domain differences statistically significant (F=5.067; p=.044)*
PGY2 PGY3 PGY40
10
20
30
40
50
60
70
80
90
100
33.3
52.8 51.87 PreStats Domain
Sub-Problem ResultsSub-Problem 5. Does the resident’s post-graduate year (PGY)
level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?
Level of knowledge: No statistically significant effect on overall score
Pre-Knowledge Mean Pre-Competency Mean Percentage0.0
20.0
40.0
60.0
80.0
100.0
48.0
58.956.0 56.254.0
33.8
PGY2
PGY3
PGY4
Final Results Self-perceived competency baseline to follow-up
(t = -2.558; p =.034)*Baseline percentage mean = 48.4Follow-up percentage mean = 68.4Baseline modes = 1 – 2Follow-up modes = 3 – 4Four items: mode increased from 1: Does Not Know How
to 4 Knows How, Has Done, and Can Do at Any Time: ○ Item 11 “Identify the difference between descriptive and analytic
research study designs”○ Item 21 “Identify the measures of central tendency”○ Item 27 “Identify what information a confidence interval (CI) says
about a statistical result”○ Item 37 “Determine when a research study should be closed
through the local Institutional Review Board (IRB)”.
Final Results Self-perceived competency baseline to follow-up
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
0
1
2
3
4
Pre-Assessment
Post-Assessment
Final Results Level of knowledge baseline to follow-up
(t = -3.210; p=.012)* Baseline mean = 52.67 Follow-up mean = 65.78 Knowledge level scores increased from 12.5% to 94.44% per
resident
1 2 3 4 5 6 7 8 90
10
20
30
40
50
60
70
80
90
100
Pre-Knowledge Score
Post-Knowledge Score
Final ResultsLevel of knowledge baseline to follow-up by domains
Pre-Assessment
Means
Post-Assessmen
tMeans
Difference in Means
t = p =
Overall raw score 52.67 65.78 -13.11 -3.210 .012*
Terminology and concepts 54.17 63.89 -9.722 -1.360 .211
Research question and hypothesis
46.28 66.60 -20.38 -3.051 .016*
Study design and sampling 52.78 68.06 -15.28 -1.846 .102
Variables and scales of measurement
48.61 72.22 -23.61 -3.688 .006*
Statistics 43.21 50.61 -7.40 -1.260 .243
Ethics, privacy, and responsible study conduct
61.12 64.82 -3.69 -.449 .666
Basic regulatory requirements 62.22 80.00 -17.78 -2.530 .035*
Final ResultsSelf-perceived competency and level of knowledge
PreKnow PreComp PostKnow PostComp0
10
20
30
40
50
60
70
80
90
100
8.37% Difference
3.88% Difference
Final ResultsSub-problem 1. Differences in change in level of knowledge based on
prior research education (F = 13.44; p = .008)*
PreKnow Mean PostKnow Mean0
10
20
30
40
50
60
70
80
90
100
44.67
70.00
56.57
63.67
No Prior Research Education
Prior Research Education
No prior education:56.71% improvement
Prior education: 12.55% improvement
Final ResultsSub-problem 4. Differences in change in level of knowledge based on
gender (F=15.028; p=.006)*
Male:24.08% improvement
Female: 19.60% improvement
(no statistical difference)PreKnow Mean PostKnow Mean0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
54.57
67.71
49.33
59.00 Male
Female
Attendance Per resident
Range = 0 sessions attended to 8 attended (100%)
Mean total attendance = 66.7%
PGY2
PGY3
PGY4
0 10 20 30 40 50 60 70 80 90
53.2
58.3
87.5
Mean Attendance
(x2 (df = 2) = 10.33; p = .005)*
Attendance Per session
Range = 30% in attendance to 90% in attendanceMean = 60% in attendance
1 2 3 4 5 6 7 80.00
20.00
40.00
60.00
80.00
100.00
% of Neurology Residents in Attendance
Session 6. Ethics, Privacy, and Responsible Study Conduct:Lowest attendance and lowest mean percentage change from baseline
Statistically significant
improvement from baseline
Implications and Sustainability
Implications Level of knowledge and self-perceived competency BOTH
significantly increased regardless of inconsistent attendance Any exposure is better than no exposure Results suggest:
○ Residents still consider research careersPromote and prepare
○ Most residency programs serve a small numberIndividual improvements are important; educate and study
population anyway Sustainability:
Develop an environment fostering research Identify key players for dedication to continued education Departmental investment (time, space, personnel) to continue
education throughout residency
Limitations
Small sample size (n=10), although census sample
Conflicting responsibilities interrupt attendance
Recommendations for Future Research Potential benefits to providing a
curriculum longer than 8 weeks Collaborative projects among residency
training programs to determine effects of education in a larger sample
Consideration of “leveling” research courses and topics based on PGY level
Identified a great need for more intensive statistical education
Thank You
Any questions?