hodges research workshop - short version 2015...justifying implementing s modelling predicting s...
Post on 22-Jul-2020
2 Views
Preview:
TRANSCRIPT
2015-01-26
1
Research in Health Professions Education: Asking Good Questions – Using Appropriate Methods Brian Hodges, MD, PhD, FRCPC Professor, Scientist and Director The Wilson Centre, University of Toronto Richard and Elizabeth Currie Chair in Health Professions Education Research Vice President Education, University Health Netowrk
Overview
• What is research, what is not? • Asking a good research
question • Choosing an approach: the
cycle of research • Unearthing assumptions • Making a commitment
Acknowledgements
• Charlotte Ringsted
• Albert Scherpbier
• Doug Buller • Vicki LeBlanc
Why do research in medical education?
Discuss in pairs Write down 3 motivations for research 1.
2.
3.
Medical Education Research is rapidly developing as a viable and exciting research field
More Journals
More readers Higher quality
2015-01-26
2
More Association and Conferences
• To improve the experience, knowledge and competence of students and clinicians
• To improve the care of patients and the health care outcomes generally
• To deepen the field of health professions education research
Creating Increased Expectations
Health professions education research is undertaken:
• In naturalistic / opportunistic settings
• Without a theoretical base
• With little funding
• By isolated researchers
• Published in a dispersed fashion, difficult to find
Why so much poor quality research?
Bordage 2000
• The conceptualization and research methods used are limited and narrow
• Researchers are often isolated, in units with missions for teaching and administration rather than research
• There is little funding for medical education research
• Researchers unable to connect their work to a larger “conversation” in an academic discipline
Explanatory Factors
Albert 2005
1. Expand research questions and move from local studies to multi-institution and international studies
2. Replicate existing research in varied settings
3. Undertaken better controlled studies and support good qualitative research
4. Created stables sources of funding for medical education research
5. Formally train researchers
Solutions: According to Bordage
1. Intensify collaboration between PhD researchers and clinicians interested in education research
2. Support the diversification of disciplines which play a role in health professions education research
Solutions: According to Albert
2015-01-26
3
• 108 medical faculty with an interest in medical education: survey and focus groups
• 40% had fellowship or masters training
• Though most involved in a project, few had funding or published their work
• No significant differences between those with and without additional education training
A masters or fellowship is not enough!
Zibrowski 2008
Factors for success • Education research support • Enhancing colleague interaction • Ongoing development activities
Barriers • Protected time • Access to a context and support staff that
sustains research • Knowledge of research methodology (Does the
masters/fellowship actually teach research skills?)
Solutions according to Zibrowski
Goals of Education Research
Aims at building on existing knowledge and understanding of learning and education by
– Studying phenomena, interactions, interventions
– Formulating models, theories, and predictions
– Studying what works, why, how and for whom
Related approaches
• Evaluation, assessment, audit
• Study of local, concrete problems and phenomena
• Limitations: – Do not contribute to advancement of
knowledge or theory building
– Not part of a larger ”conversation”
Study of local, concrete problems
Evaluation ”What is the effect of this course in my
school regarding satisfaction, usefulness, learning, behaviour?”
Assessment ”What is the learning outcome of this
instruction or course assessed by MCQ or OSCE?”
Audit • ”How well is our programme working in
our department regarding ...?”
Research
• Starts with an idea, problem or case • Relates the idea to conceptual framework • Develops specific research questions • Employs research approach, design, and
instruments • Collects and analyse data • Interprets results • Discusses validity and generalisability (or
transferability) of results and practice implications
2015-01-26
4
A research program
• Problem
• Research question
• Study design
• Data and analysis
• Understanding refined
Two poles of medical education research
Theory driven research Deepen the knowledge and understanding of learning, teaching, training, education
Applied research Provide evidence of effect, justify initiatives, concepts, technology, expenditure
Players in the complex field Getting published, promoted, famous,
and socialised Albert AHSE 2007 Albert/Rees ME 2010 Slide: Ringsted 2010
Stokes Quadrant Model of Research Not bird watching…
A single study at Kirkpatrick level 1 (Satisfaction) – We created a course about #fv*7%d – 5 people took it – 3 of them filled out the evaluation form – They all thought it was great
Sweeping generalization
– We have shown it is feasible and acceptable to implement a course in #fv*7%d
– Medical schools around the world should created courses in #fv*7%d
Bohr Quality standards from the primary discipline
Pure applied research
Adv
ance
men
t of k
now
ledg
e
Applicability
Edison Quality standards for innovation - eg design based research
2015-01-26
5
Use-‐inspired basic research
Adv
ance
men
t of k
now
ledg
e
Applicability
Pasteur
Quality research = theory + utility
Eg. Learning in a Simulator
Practical Evaluation
Knowledge Building Research
Process Did the simulation learning program operate well / as intended?
What organizational factors enable the successful development and implementation of simulation learning?
Outcome Were the learning objectives met? Did students learn?
How does simulation-enhanced education impact learners’ self-efficacy and perceived readiness for practice?
From Practical Evaluation to Knowledge Building Research
Asking a Good Research Question
Don’t start with a tool or data
• “I’d like to do a survey” • “I want to do a study on OSCEs” • “I want to do something qualitative” • “We have a lot of teacher evaluation
data - I’d like to do some research on it”
• “My chair wants me to do some research on the clerkship”
The complexity of education
Learning task Nature
Complexity
Learner Characteristics
Experience Instruction
Self-study One-on-one Classroom
Setting School
Workplace
Organization Rules and regulations Selection
Teachers Characteristics Qualifications
Curriculum Content Format
Materials Assessment Content Format
Programming
Environment Milieu
Supportive infrastructure
Society Politics Culture
Evaluation Target Format
Consequence
Education Slide: C. Ringsted 2010
Define your problem
• Think of an idea, problem, issue in education that has been on your mind
• Write it down
• Are you passionate about this problem or issue?
• Explain the problem, and why it
interests you, to a neighbour
2015-01-26
6
Perform a systematic review of literature
Conceptual frameworks
• Relate your idea, problem, phenomenon to theories of learning and education (or sociology, anthropology, psychology, etc)
• Identify models that explain underlying mechanisms of a phenomenon or why and how an intervention might work
• Create a relevant framework for your study – Appropriate design and valid methods
Find a conceptual framework
• ”Conceptual frameworks represent ways of thinking about a problem or a study”
• ”Different frameworks will emphasise different variables and outcomes, and their inter-relatedness”
• ”Scholars are responsible for making explicit the assumptions and principles contained in the conceptual framework they use in their R&D projects”
Bordage 2009
Learn more about the problem
• What, if any, investigation of this problem have you done already? – Reading about it – Talking with an expert – Attend a lecture or workshop
• Has anyone researched this problem before?
• If you are to pursue a study about this problem, what will you need to know more about?
Generate an answerable question
Change your problem into a question
The nurses don’t allow the medical students to participate in deliveries
Our international graduates
have problems talking to patients
Students are resistant to
feedback about their OSCEs
Why do nurses not involve clinical clerks in the labour and delivery room?
In what ways do international
medical graduates have trouble with patient communication?
How do students perceive
feedback they receive from their OSCEs?
2015-01-26
7
Ensure your question can be studied
• Too vague: What is the meaning of life?
• Too ambitious What is the best curriculum to train a professional
physician for the 21st century?
• The PhD thesis How do methods of assessment of competence
used in countries around the world reflect their history and culture?
• Reword, refocus, redefine your question as needed
The Cycle of Research
Ringsted’s cycle of research
Experimental
studies Observa
tional
studies
Justifying
Implementing
Translational
studies
Predicting Modelling
Explorative
studies
Conceptual theoretical framework
What is the nature of the problem?
Experimental
studies Observa
tional
studies
Justifying
Implementing
Translational
studies
Predicting Modelling
Explorative
studies
Conceptual theoretical framework
Design Descriptive Qualitative
Psychometric
Descriptive studies
• Description – Phenomenon, novel intervention, assessment
method, administrative procedure, organisation, population of individuals
• Prevalence – High in young research disciplines – Medical
Education and Emergency Medicine
• Characteristics – Reports with or without outcome data – Make no comparisons
Conceptual theoretical framework
What is qualitative research?
The goal of qualitative research is: – the development of concepts which help us
to understand social phenomena – in natural (rather than experimental) settings – giving emphasis to the meanings,
experiences and views of all participants Pope and Mays BMJ, 1995
Qualitative methods use language-based
rather than numerically-based data
2015-01-26
8
When to use qualitative research?
1. Preliminary to quantitative research
2. Supplemental (triangulation) to quantitative data
3. To explore complex phenomena not amenable to quantitative research
Qualitative and quantitative are not opposites
• Often artificially seen as opposing poles • Different, but complementary ways of viewing
similar phenomena Eg. Diabetes and insulin compliance • Quantitative study: What is the relationship
between non-compliance (as measured by hemoglobin A1C) and disease progression?
• Qualitative study: Why are patients not compliant? How do they view disease progression? Do they see a relationship with compliance?
Qualitative methodology and method
Methodology • Historical
method • Ethnography • Phenomenology • Discourse
Analysis • Grounded
Theory
Method • Interviews • Focus Groups • Case Studies • Text Analysis • Observation
• Naturalistic / Interpretive – Study phenomena in
natural settings – Meanings participants
bring to phenomena • Specific to context studied • Individuals’ values involved: participants’ & researchers’
Data Collection
Data Management
Data Analysis
Data Sampling
Does it work in a lab?
Experimental
studies Observa
tional
studies
Justifying
Implementing
Translational
studies
Predicting Modelling
Explorative
studies
Conceptual theoretical framework
RCTs & Quasi- experimental
studies
Challenges with RCTs in education
• Samples – Size and representativeness
• Interventions – Definition of intervention
and control circumstance – Standardisation of intervention
• Measurement – Identification of valid outcome measure – Appropriate measurement points
2015-01-26
9
Measurement points
• Before measurement – Testing might have an effect on learning in
itself – The test might be remembered
• After measurement – Measurement immediate after intervention – Measurement after a pause, retention test – Measurement of transfer
• Another, but similar task • A more complex task • Another context
Experimental design
• Learning is a sustainable change in capacity to perform – Measure retention and/or transfer
• Be aware of the testing-effect Randomized design R Ob X1 Oa Or/Ot 4-group Solomon R Ob X2 Oa Or/Ot before-after R X1 Oa Or/Ot retention/transfer R X2 Oa Or/Ot design
Other concerns about RCTs
• Ethics, blinding, and feasibility – Matched groups, ’waitlist’ controls
• Validity – RCTs are the best design to find
out if an intervention works – The worst design to find out who
it works for and the long-term effects the intervention
Measurement
• Test-enhanced learning – Testing has an intrinsic effect on
memory
• Effect of stress on learning and performance
Roediger HL et al 2006. Kromann C et al. ME 2009 Kromann C et al. ME 2010
LeBlanc AM 2009. Harvey et al. ME 2010 Dieckmann et al. 2007 Stefanidis et al. 2010 Munch-Petersen/ Rosenberg UfL2008
Feedback and learning
• Training program: Vascular anastomosis, plastic model – Distributed over 1 week – Massed in 1 day
• Measurement points – Pre-test – Post-test – Retention 1 month after – Transfer, live animal
Pre Post Retention Transfer
Distributed vs. massed training
Glo
bal r
atin
g * * NS
Moulton 2006
Does it work in “real” settings?
Experimental
studies Observa
tional
studies
Justifying
Implementing
Translational
studies
Predicting Modelling
Explorative
studies
Conceptual theoretical framework
RCTs & Quasi- experimental
studies
Cohort studies Case-control Associational
2015-01-26
10
Observational studies
• Aims at predicting – effect, cause or consequence
– Cohort studies – Case-control studies – Associational studies
Observational studies
+ Exposure
- Exposure
Outcome measure
Cohort studies Start here
Case-control studies Start here
Cross-sectional studies Examine everything
at a certain time
IIIIIIII IIIIIIII
IIIIIIII IIIIIIII III
III
Examples
• Tamblyn et al. JAMA 2007 – Prospective cohort study
• Scores on communication, decision-making predict complaints in later practice
• Draycott et al. BJOG 2006 – Retrospective cohort study
• Effect of training on neonatal outcome
• Wayne et al. Chest 2008 – Retrospective case-control study
• Effect of ACLS training on performance standards
Example: Case-control study
Norman et al. Med Educ 2008
• RQ – Does PBL in medical
school predict results of doctors’ performance reviews in practice?
• Design – Retrospective study of
1166 Canadian doctors’ performance reviews
108, +PBL
857, - PBL
4%
5%
24%
28%
Result
Concern Excellent
Case-control studies
• Study the cause or consequence of differences that already exist between or among groups
• Useful for studies where – Outcome is categorical
• Yes/no or present/absent
– Prevalence of outcome is low • Rarely observed phenomenon
– Time delay until outcome is long • Years
Associational studies
• Aims at – Describing phenomena by
identifying the degree of an existing relationship among variables
– Clarify understanding and seek to predict a certain outcome
• However .... – Conceptual, theoretical
framework is crucial to interpretation of cause and effect
r=0.86
r=0.86
2015-01-26
11
Does it work in different settings?
Experimental
studies Observa
tional
studies
Justifying
Implementing
Predicting Modelling
Explorative
studies
Conceptual theoretical framework
Effect studies Implementation
Observation
Translational
studies
Translational research Sung JAMA 2003
• Political concern – Scientific discoveries fail to be translated
efficiently into tangible human benefit
• Three translational blocks (biomedicine)
Basic Research
From Lab to Human
Clinical Research
From study to practice
Happy Healthy People
Block 1
Block 2
Block 3
Effect studies
Practical clinical trials (PCT) Tunis JAMA 2003
• Approach is based on a need to make decisions • Select relevant alternative interventions to compare • Include a diverse population of study participants • Recruit participants from heterogeneous settings • Collect data on a broad range of outcomes related to
both persons and systems, including finances
Satisfaction Learning Behaviour Organization Outcome
Jump on and off wherever you like
Experimental
studies Observa
tional
studies
Justifying
Implementing
Translational
studies
Predicting Modelling
Explorative
studies
Conceptual theoretical framework
RCTs and Quasi- experimental
studies
Cohort studies Case-control Associational
Literature Qualitative
Psychometric
Effect studies Implementation
Observation
• What KIND of approach to you want to use to address your research question?
• Think about the previous slides, and see if you can identify where in the cycle you wish to be
• Chat with your neighbour about your thoughts
Unearthing Assumptions
2015-01-26
12
Where do your ideas come from?
• “It is known that..” • “Everyone agrees that…” • “Many studies have shown
that…” • “I believe that…”
Unearth your assumptions
What is the answer(s) to your question?
Write down what you think it might be (even if you are not sure - even if you just ‘suspect’ them to be true)
It’s OK to do this - really!
• Bias • Perspective • Reflexivity
Making a commitment
Make a commitment
• You must make a commitment to yourself
• Write down your research question…somewhere
• Put it somewhere you will see everyday
Post it somewhere you can see it!
2015-01-26
13
Advancing healthcare education and practice through research
www.thewilsoncentre.ca
References
Bordage G, Dawson B. Experimental study design and grant writing in eight steps and 28 questions. Med Educ. 2003;37:376-385.
Bordage, "Conceptual frameworks to illuminate and magnify", Med Educ 2009; 43:312-319
Beckman TJ, Cook DA. Developing scholarly projects in education: a primer for medical teachers. Med Teach. 2007;29:210-218.
Campbell et al, BMJ 2000. Framework for design and evaluation of complex interventions to improve health.
Cook DA. Avoiding confounded comparisons in education research. Med Educ. 2009;43:102-104.
Cook DA, Bordage G, Schmidt HG. Description, Justification, and Clarification: A Framework for Classifying the Purposes of Research in Medical Education. Med Educ. 2008;42:128-133.
References Eva KW, Lingard L. What's next? A guiding question for
educators engaged in educational research. Med Educ. 2008.
Morrison J. Developing research questions in medical education: the science and the art. Med Educ. 2002;36:596-597.
McGaghie WC, Bordage G, Shea JA. Problem Statement, Conceptual Framework, and Research Question. Acad Med. 2001;76:923-924.
Norman G. RCT = results confounded and trivial: the perils of grand educational experiments. Med Educ. 2003;37:582-584.
Norman & Eva ASME publ. 2008. Quantitative research methods in medical education.
Prideaux D, Bligh J. Research in medical education: asking the right questions. Med Educ. 2002;36:1114-1115.
top related