educational research: getting started making life better through research, education &...
TRANSCRIPT
Educational Research: Getting Started
Making Life Betterthrough Research, Education & Healthcare TM
Steven B. GoldinChief of Hepatobiliary and Pancreatic
SurgeryAssociate Professor of SurgeryVice Chair of Surgical Education
Chief of Hepatobiliary and Pancreatic Surgery
NO DISCLOSURES EXCEPT:Yes, they really are my
photos
NO DISCLOSURES EXCEPT:Yes, they really are my
photos
Outline1. Research: Why do it?2. What is educational research3. Research: How to get started4. Features of rigorous studies5. Pitfalls – validity and reliability6.Surveys7.Ethics, funding, resources
ASE – “No discipline can call itself a profession that does not give top priority to research. More specifically, the status of a given profession depends on the adequacy of the research activities, and the application of research findings to routine professional activities.” (Mouly 1978)
ASE – “No discipline can call itself a profession that does not give top priority to research. More specifically, the status of a given profession depends on the adequacy of the research activities, and the application of research findings to routine professional activities.” (Mouly 1978)
Research – why do it?
Faculty advancement
Teaching and research separate academic surgeons from those in community practice
Develop a national prominence/reputation
May be a funding source
Faculty advancement
Teaching and research separate academic surgeons from those in community practice
Develop a national prominence/reputation
May be a funding source
Research- why do it?
Importance of clinical productivity – how do you find the time to do a research project?
Do research with your daily activities
Data collection is common for all courses and rotations.
medical studentsresidentsfaculty
Innovation – what makes an academic center academic. What additional data would need to be collected to begin a research project?
Importance of clinical productivity – how do you find the time to do a research project?
Do research with your daily activities
Data collection is common for all courses and rotations.
medical studentsresidentsfaculty
Innovation – what makes an academic center academic. What additional data would need to be collected to begin a research project?
Introduction
Innovation and evaluation form the backbone of research ideas and projects
Many research projects are done with students, residents, and faculty
Steady supply of potential participants
Innovation and evaluation form the backbone of research ideas and projects
Many research projects are done with students, residents, and faculty
Steady supply of potential participants
Introduction
Traditionally empirical research (bench top) was viewed as the most valid method. Other forms of research (historical, phenomenological, introspective, qualitative, etc.) contribute to “educational research”
Empirical research is characterized by direct observation, objectivity, falsification, replicability, parsimony and quantification (scientific method)
Parsimony is defined as the adoption of the simplest assumption in the formulation of a theory or in the interpretation of data, especially in accordance with the rule of Ockham's razor
Traditionally empirical research (bench top) was viewed as the most valid method. Other forms of research (historical, phenomenological, introspective, qualitative, etc.) contribute to “educational research”
Empirical research is characterized by direct observation, objectivity, falsification, replicability, parsimony and quantification (scientific method)
Parsimony is defined as the adoption of the simplest assumption in the formulation of a theory or in the interpretation of data, especially in accordance with the rule of Ockham's razor
Introduction
Occam’s razor (Ockham’s razor) - “entities should not be multiplied unnecessarily.” The explanation of any phenomenon should make as few assumptions as possible, eliminating those that make no difference in the observable predictions of the explanatory hypothesis or theory
When competing hypotheses are equal in other respects, select the hypothesis that introduces the fewest assumptions and postulates the fewest entities while sufficiently answering the question
Occam’s razor (Ockham’s razor) - “entities should not be multiplied unnecessarily.” The explanation of any phenomenon should make as few assumptions as possible, eliminating those that make no difference in the observable predictions of the explanatory hypothesis or theory
When competing hypotheses are equal in other respects, select the hypothesis that introduces the fewest assumptions and postulates the fewest entities while sufficiently answering the question
Introduction
Evidence based medicine – interventions should be guided by data from rigorous studies was described at least 15 years ago
Evidence-Based Medicine (EBM) – excellent framework for evaluating the quality of clinical research
EBM is not focused on educational research, but provides useful criteria for different categories of research. (Treatment efficacy, accuracy of diagnostic tests, clinical guidelines, prognosis and risk, overviews and meta-analysis)
Evidence based medicine – interventions should be guided by data from rigorous studies was described at least 15 years ago
Evidence-Based Medicine (EBM) – excellent framework for evaluating the quality of clinical research
EBM is not focused on educational research, but provides useful criteria for different categories of research. (Treatment efficacy, accuracy of diagnostic tests, clinical guidelines, prognosis and risk, overviews and meta-analysis)
Evidence Based Medicine
Many EBM concepts are applicable to educational research and should be familiar to all educational researchers
Medical educators, likewise, should teach based on evidence
Many EBM concepts are applicable to educational research and should be familiar to all educational researchers
Medical educators, likewise, should teach based on evidence
Evidence Based Medicine
Educational research is driven by the process of documenting the steps of a project
This process is the same for all studies including bench research, studying teaching methods, or student attitudes
Educational research focuses on describing and documenting the transmission of knowledge, competency, and understanding about which teaching methods are best and can lead to better practice
Educational research is driven by the process of documenting the steps of a project
This process is the same for all studies including bench research, studying teaching methods, or student attitudes
Educational research focuses on describing and documenting the transmission of knowledge, competency, and understanding about which teaching methods are best and can lead to better practice
Introduction
All research involves – asking a question and obtaining an answer in a systematic, thoughtful way
Scholarly teaching involves routinely asking questions about daily educational activities
Research is answering these questions and scholarship is the dissemination of the research results
All research involves – asking a question and obtaining an answer in a systematic, thoughtful way
Scholarly teaching involves routinely asking questions about daily educational activities
Research is answering these questions and scholarship is the dissemination of the research results
Making it Scholarly
Descriptive: study describes what is going on or what exists
Relational: study looks at the relationships between two or more variables
Causal: study is designed to determine whether one or more variables causes or affects one or more outcome variable
Descriptive: study describes what is going on or what exists
Relational: study looks at the relationships between two or more variables
Causal: study is designed to determine whether one or more variables causes or affects one or more outcome variable
Types of Studies
1. Ask the question – state clear goals (hypothesis and alternate hypothesis)
2. Prepare (literature review, discussions, conferences)
3. Specify the techniques or interventions4. Test methods and ways of measuring
impact5. Pilot test the process and refine process6. Select the best method to answer the
question
1. Ask the question – state clear goals (hypothesis and alternate hypothesis)
2. Prepare (literature review, discussions, conferences)
3. Specify the techniques or interventions4. Test methods and ways of measuring
impact5. Pilot test the process and refine process6. Select the best method to answer the
question
Research Steps
7. Do the study
8. Capture and analyze data
9. Interpret findings and determine significance of the outcome
10.Negative results can be meaningful depending on the question
11.Reflective critique (thoughtful discussion, think about the implications of the findings and develop the next steps)
12.Disseminate the findings (article, abstract, presentation, workshop)
7. Do the study
8. Capture and analyze data
9. Interpret findings and determine significance of the outcome
10.Negative results can be meaningful depending on the question
11.Reflective critique (thoughtful discussion, think about the implications of the findings and develop the next steps)
12.Disseminate the findings (article, abstract, presentation, workshop)
Research Steps
Large numbers of student participants
Multi-institutional enrollment of participants
Having a control or appropriate comparison group
Measuring an objective outcome
Measuring some outcome at least 1 month after the intervention
Conduct the intervention more than once
Estimate statistical power
Large numbers of student participants
Multi-institutional enrollment of participants
Having a control or appropriate comparison group
Measuring an objective outcome
Measuring some outcome at least 1 month after the intervention
Conduct the intervention more than once
Estimate statistical power
Features of Rigorous Studies
Measure objective outcomes including any evaluation other than self-assessment by students including:
Tests of knowledgeCourse grades Objective structured clinical examinations or observations of standardized patient
Interaction assessment of performance with a real patient such as graded observation of patient interactions or patient feedback
Objective clinical outcomes such as blood pressure controlPerformance with mannequins, computer simulations, or laboratory animals
Psychological inventories
Measure objective outcomes including any evaluation other than self-assessment by students including:
Tests of knowledgeCourse grades Objective structured clinical examinations or observations of standardized patient
Interaction assessment of performance with a real patient such as graded observation of patient interactions or patient feedback
Objective clinical outcomes such as blood pressure controlPerformance with mannequins, computer simulations, or laboratory animals
Psychological inventories
Features of Rigorous Studies
Language
Types of questions (descriptive, relational, causal)
Time in research (cross-sectional vs. longitudinal with repeated measures or time series)
Types of relationships (correlational vs. causal)
Patterns of relationships (none, positive, negative)
Variables and attributes (independent and dependent, exhaustive, mutually exclusive)
Hypothesis and null hypothesis
Types of data (quantitative vs. qualitative)
Language
Types of questions (descriptive, relational, causal)
Time in research (cross-sectional vs. longitudinal with repeated measures or time series)
Types of relationships (correlational vs. causal)
Patterns of relationships (none, positive, negative)
Variables and attributes (independent and dependent, exhaustive, mutually exclusive)
Hypothesis and null hypothesis
Types of data (quantitative vs. qualitative)
Foundation
Theoretically significant research questions are different from other types of questions.
A research question is not a question:
of fact. (How many residents passed the in-service or their qualifying examinations?)
about a specific sample of subjects. (Do my students perform better with open-ended rather than multiple choice questions?)
about things that can not be measured. (Do lucky students have more success than unlucky students?)
Theoretically significant research questions are different from other types of questions.
A research question is not a question:
of fact. (How many residents passed the in-service or their qualifying examinations?)
about a specific sample of subjects. (Do my students perform better with open-ended rather than multiple choice questions?)
about things that can not be measured. (Do lucky students have more success than unlucky students?)
The Hypothesis
1. seek new knowledge
2. use the scientific method
3. build upon existing scientific knowledge
4. pick up where existing theories and data end
5. lead to specific hypotheses which can be tested
6. have an answer deducible from the results and provide information applicable not only to the subjects studied, but to a broad, definable population of subjects and settings
1. seek new knowledge
2. use the scientific method
3. build upon existing scientific knowledge
4. pick up where existing theories and data end
5. lead to specific hypotheses which can be tested
6. have an answer deducible from the results and provide information applicable not only to the subjects studied, but to a broad, definable population of subjects and settings
A good research question should:
Education research quality is linked to reliability and validity, which are the two major criteria for the quality of the measurement
Reliability involves the quality of measurement and the consistency or repeatability of the measures
Theories of validity involve each step which are interdependent
Education research quality is linked to reliability and validity, which are the two major criteria for the quality of the measurement
Reliability involves the quality of measurement and the consistency or repeatability of the measures
Theories of validity involve each step which are interdependent
Reliability and Validity
Studies may be
reliable but not valid
valid but not reliable
neither reliable nor valid
both reliable and valid
Consulting with a research design and statistics expert to avoid issues that may impact study reliability or validity
Studies may be
reliable but not valid
valid but not reliable
neither reliable nor valid
both reliable and valid
Consulting with a research design and statistics expert to avoid issues that may impact study reliability or validity
Reliability and Validity
General Surgeon Shortage
Surveys - References
Sage Publications, Inc; 2nd edition (November 12, 2002), 10 volumes, 1434 pages
Gall, Borg, and Gall. Educational Research: An Introduction (8th edition). Allyn and Bacon. 2006
One of the most important areas of measurement in applied social research.
Two types
Questionnaires
Interviews
Construction of the Survey
Types of questions
Decisions about question content
Decisions about question wording
Decisions about response format
Question placement and sequence
One of the most important areas of measurement in applied social research.
Two types
Questionnaires
Interviews
Construction of the Survey
Types of questions
Decisions about question content
Decisions about question wording
Decisions about response format
Question placement and sequence
Surveys
Often done because users think they are simple and quick ways to publications
Survey design has fallen into disrepute among editors, reviewers and consumers of medical research because they have been:
poorly designed
poorly developed
poorly administered
poorly analyzed
Reinforces the reconceived notion that surveys are not worth doing
Like experimental research, good survey research is very systematic and rigorous
Often done because users think they are simple and quick ways to publications
Survey design has fallen into disrepute among editors, reviewers and consumers of medical research because they have been:
poorly designed
poorly developed
poorly administered
poorly analyzed
Reinforces the reconceived notion that surveys are not worth doing
Like experimental research, good survey research is very systematic and rigorous
Surveys
The purpose of the study must be supported by a review of the literature and understanding of the problem. Then develop the research question, which will guide the development of the content. If an item does not help you answer a research question, do not include it on the survey
What information do you want from respondents?attitudes/beliefs?knowledge?behavior?attributes (demographic characteristics)?
The purpose of the study must be supported by a review of the literature and understanding of the problem. Then develop the research question, which will guide the development of the content. If an item does not help you answer a research question, do not include it on the survey
What information do you want from respondents?attitudes/beliefs?knowledge?behavior?attributes (demographic characteristics)?
Surveys
Structure the questions carefully
Unless the sole purpose of the survey is to solicit suggestions, probe memories, clarify positions and or vent frustrations, limit the number of open-ended questions
Open ended questions make it difficult to construct meaningful variables for statistical analysis, are difficult to collate, and difficult to decipher
Structure the questions carefully
Unless the sole purpose of the survey is to solicit suggestions, probe memories, clarify positions and or vent frustrations, limit the number of open-ended questions
Open ended questions make it difficult to construct meaningful variables for statistical analysis, are difficult to collate, and difficult to decipher
Surveys
A questionnaire should:1. look attractive/appealing to complete2. have questions laid out and be organized in such a manner as to make them as easy to complete as possible3. be organized with items appearing in some logical sequence4. begin with those questions that the respondent is likely to see as useful to the purpose of the study, or with those that are easy to answer and non-threatening5. place important items that are key to the study near the beginning
Items that are interesting and clearly relevant to the study will increase response rate. Length has a small effect on response rate.
Pilot the instrument
A questionnaire should:1. look attractive/appealing to complete2. have questions laid out and be organized in such a manner as to make them as easy to complete as possible3. be organized with items appearing in some logical sequence4. begin with those questions that the respondent is likely to see as useful to the purpose of the study, or with those that are easy to answer and non-threatening5. place important items that are key to the study near the beginning
Items that are interesting and clearly relevant to the study will increase response rate. Length has a small effect on response rate.
Pilot the instrument
Surveys
Principle of voluntary participationInformed consentRisk of harm – physical or psychologicalConfidentialityPrinciple of anonymityPerson’s right to service (no treatment control group)Institutional Review Board (IRB)
Students and residents are considered a vulnerable populationSocial and Behavioral Institutional IRB vs.
Biomedical IRB
Principle of voluntary participationInformed consentRisk of harm – physical or psychologicalConfidentialityPrinciple of anonymityPerson’s right to service (no treatment control group)Institutional Review Board (IRB)
Students and residents are considered a vulnerable populationSocial and Behavioral Institutional IRB vs.
Biomedical IRB
Ethics
Most surgical education research is conducted without the benefit of research grants
ASE Foundation – CESERT program (www.surgicaleducation.com)
National Board of Medical ExaminersEmergency Medicine FoundationAgency for Health Care Research and QualitySagesAmerican College of Surgeons (www.facs.org) lists funding agencies that may be willing to fund educational research
Most surgical education research is conducted without the benefit of research grants
ASE Foundation – CESERT program (www.surgicaleducation.com)
National Board of Medical ExaminersEmergency Medicine FoundationAgency for Health Care Research and QualitySagesAmerican College of Surgeons (www.facs.org) lists funding agencies that may be willing to fund educational research
Funding
ASE Educational Research Committee
Institutional Department of Medical Education
Departments of psychology, education, sociology, and public health
Biostatistician
ASE – Surgical Education Research Fellowship Program (SERF)
ASE Educational Research Committee
Institutional Department of Medical Education
Departments of psychology, education, sociology, and public health
Biostatistician
ASE – Surgical Education Research Fellowship Program (SERF)
Resources
Clinical demands are taking on larger levels of importance
Protected time is a luxury
Research is required to advance academically
Educational research is a viable option even when faced with high clinical demands
Clinical demands are taking on larger levels of importance
Protected time is a luxury
Research is required to advance academically
Educational research is a viable option even when faced with high clinical demands
Conclusions
Many educational research projects have significant limitations
Insufficient sample sizesLack of generalizability when interventions are
assessed at only 1 institution or are conducted only once
Lack of appropriate control groupsUse of subjective and non-validated instruments to assess outcomesAssessing only short-term outcomes
Many educational research projects have significant limitations
Insufficient sample sizesLack of generalizability when interventions are
assessed at only 1 institution or are conducted only once
Lack of appropriate control groupsUse of subjective and non-validated instruments to assess outcomesAssessing only short-term outcomes
Conclusions
Educational research challenges often relate to measurement issues, bias and individual differences
An understanding of key educational concepts in psychology, sociology, anthropology, psychometrics, sociometrics, and statistics. Collaboration is key!
Assuming a significant question has been clearly articulated, the quality of the study will depend upon the research design, data analysis, interpretation and results reporting
Educational research challenges often relate to measurement issues, bias and individual differences
An understanding of key educational concepts in psychology, sociology, anthropology, psychometrics, sociometrics, and statistics. Collaboration is key!
Assuming a significant question has been clearly articulated, the quality of the study will depend upon the research design, data analysis, interpretation and results reporting
Conclusions
Research Methods Knowledge Base. William M.K. Trochim. http://www.socialresearchmethods.net. Free on the internet
Surgical Educators’ Handbook. Curriculum Committee and Members of the Association for Surgical Education. ASE website. http://www.surgicaleducation.com/mc/page.do
Research Methods Knowledge Base. William M.K. Trochim. http://www.socialresearchmethods.net. Free on the internet
Surgical Educators’ Handbook. Curriculum Committee and Members of the Association for Surgical Education. ASE website. http://www.surgicaleducation.com/mc/page.do
References
Educational Research: Getting Started
Making Life Betterthrough Research, Education & Healthcare TM
Steven B. GoldinChief of Hepatobiliary and Pancreatic
SurgeryAssociate Professor of SurgeryVice Chair of Surgical Education
Chief of Hepatobiliary and Pancreatic Surgery