Conducting Rigorous Qualitative Research
VA HSR&D Methods Cyberseminar SeriesJane Forman, ScD MHS
Ann Arbor VA Center for Clinical Management Research
May 11, 2009
2
Objectives of the SeminarBriefly describe key uses and
features of qualitative research
Describe rigorous procedures for: Formulating research questions Research design Sampling Data Collection Data Analysis
3
How do we use qualitative research in the health sciences?• Understanding complex social processes difficult
to measure quantitatively (e.g., organizational culture)
• Communication studies
• E.g., patient-physician interactions• Implementation studies
• Formative evaluation
• Mechanisms underlying outcomes• Instrument development
• Qualitative exploration needed to develop survey items
4
Type of Study:
Qualitative Quantitative
Goal UnderstandingMeasurement; Determining associations
Research Process
Iterative & Emerging
Sequential & Fixed
Sampling Purposeful Representative
Data Collection
Open-ended Closed-Ended
Data Analysis
Primarily Inductive Deductive
Qualitative vs. Quantitative Methods Qualitative vs. Quantitative Methods
5
Engagement: Constant and Iterative
Research QuestionsSampling
Data CollectionAnalysis
Rigorous Procedures
7
Research Questions: Clearly defined and focused Nature of qualitative research questions
Explore, explain, describe, understand....experiences, processes, beliefs, perceptions...
Don’t think in terms of variables that you’re trying to relate to each other.
Do think in terms of identifying factors and understanding how they work.
Non-directional language
AVOID BEING OVERLY AMBITIOUS
What are the communication problems between nurses and physicians?
How do nurses perceive communication problems with physicians in the ICU?
How do nurses in ICUs at VA hospitals perceive communication with physicians during rounds?Sub-question: How do perceptions differ across types of ICUs?
Focus
Focus
8
What data sources and methods are appropriate?What can I do given resource constraints?
Research Questions
Data sources and methods
Justification Practicalities (eg resources, access, skills)
Ethical Issues
(Mason 2002)
Research design: Matching Data Sources and Methods to Research Question
Research design: Matching Data Sources and Methods to Research Question
9
Triangulation
Inclusion of multiple data sources and methods in a study.
Different methods and sources reveal different aspects of empirical reality, e.g., Comparing what participants say to what
you observe. Comparing the perspectives of people who
have different points of view
Test for consistency. Convergence strengthens validity. Understanding inconsistencies can be
illuminative and important.
10
Sampling
Sampling is a key strategic part of the detailed design that you develop before you begin your study.
You make sampling decisions during data collection based on what you find in your data.
Topics: Purposeful vs. representational sampling Types of purposeful sampling Sample size
11
Sampling: Representational vs. Purposeful
Representational in quantitative studies Goal: Enable generalizations from study
samples to populations.
Purposeful in qualitative studies Goal: To understand a phenomenon,
not to represent a population. Select of information-rich cases for
intensive study. ▪ Cases that will provide information needed to
address your research questions.
12
Selected Types of Purposeful Sampling
Typical caseExtreme or Deviant caseStratifiedMaximum variationHomogenousSnowball or ChainRandomConvenience
Source: M. Q. Patton, pp.230-246
13
Types of Purposeful Sampling: Examples Typical Case
Useful in understanding, e.g., how an intervention is experienced by a typical patient.
Extreme Case Useful in understanding unusual cases or
outliers, E.g., outstanding successes (Covey), notable failures
Often useful, though comparison, in understanding typical cases.
Stratified Vary on characteristics relevant to the topic
under study. Creates comparison groups Main purpose is to capture major variations
rather than identify common core themes.
14
Sample Size
Factors that affect sample size: Number of comparison groups (more
comparisons larger sample size). Detail, complexity and depth of data
(more detailed, complex, and in-depth smaller sample size)
Saturation Seeing nothing new in newly sampled units Data is sufficient to create a valid product Depends on judgment based on experience
There’s a tendency to over-sample in the health sciences.
▪ Investigators used to representative sampling ▪ Haven’t yet experienced the volume of qualitative data
generated and the intensity of data analysis
15
Data Collection
• Choose data collection technique(s)• There’s a tendency to design studies around
the data collection technique vs. matching techniques to research questions.
• Systematically translate research questions to data collection instruments (e.g., interview guide, observation protocol)
• Additional procedures to ensure rigorous and valid data collection
16
Choosing data collection techniques: Individual InterviewsWhen individual’s experience and
unique interpretation of it is of interest
Decision: How structured? The more structured the interview, the
less rich your data will be Key decision in study design
17
Structured, open-ended vs. semi-structured interviews Structured, open-ended
Fixed questions asked in a particular order
When less complex data is appropriate Easier to analyze, e.g., by question
Semi-structured Cover topics while following the
interviwee’s train of thought Generates rich, narrative data Data analysis resource intensive and
more highly interpretive
18
Semi-structured interviews: Skills and experience especially important
Challenging to conduct Simultaneous management of
intellectual and social dynamics Establishing rapport Listening skills “think[ing] on your feet…in ways
consistent with your research questions” (Mason, p.67)
▪ Constantly making choices about where to go next
19
Choosing data collection techniques: Focus groups Facilitated group interviews
When breadth of data generated is more important than depth.
When group interaction will help address your research question.
Myth – focus groups are easier and less time-consuming to conduct than individual interviews.
Focus groups and interviews can both be used in a single study. E.g., start with focus groups, then conduct
interviews to gather more in-depth information about selected topics
20
Choosing Data Collection Techniques: Observation
Often used to triangulate with data produced through other techniques.
Objects: e.g., daily routines, interactions, non-verbal behavior, physical space
Challenges Figuring out what you want to observe
and how▪ Can’t just go and “see what’s there”
Skills, training Resource constraints when combined
with other techniques
21
Systematic translation of research questions to data collection instruments
Process of developing instrument serves to sharpen your idea of what you want to get from your data.
Not a restatement of your research questions What is the best way to address them.
Use your understandings of the topic from your previous work, the literature (theoretical and/or empirical), and experience in living. Components of theoretical framework and/or
conceptual model you develop for your study.
22
Example: Semi-structured Interview Guide Lists the topics that are to be explored
in the interview, usually along with questions that suggest lines of inquiry.
Narrower components of each topic listed as probes under each topic, to guide interviewer to elicit more detail.
Most important for early interviews. Interview usually relies less and less on the
interview guide as the study goes on.
23
Additional procedures to ensure rigorous and valid data collection
Involvement of data collectors as early as possible in the study. Best if PI is involved in data collection
▪ Knows the most about what data will best address research questions.
▪ Particularly important when conducting sem-structured interviews.
Data collection training Monitoring quality
“Qualitative analysis transforms data into findings. No formula exists for that transformation. Guidance, yes. But no recipe.”
-- M.Q. Patton
Here’s some guidance.
25
Data Analysis
Goal: Create new knowledge from raw, unordered data. Make sense of your data.
Qualitative analysis requires considerably more than just reading to see what’s there.” (Patton, 2001)
It’s vital to develop a systematic approach to data, emphasizing study aims.
Data analysis is constant and iterative. Be engaged in data analysis the first piece of data is collected.
Focus data analysis early on, but be open to new topics to explore.
26
Data Analysis Procedures Memoing Coding scheme development
Coding agreement Coding of the data Applying codes and rearranging data by
code Summaries by units of analysis, by code Synthesis/Interpretation
Data Display Conclusion Drawing and Verification
27
Memoing
Create memos throughout the study Write anything that will help make sense
of the data. Get engaged in the data by recording early
thoughts and analytic hunches. Identify and sharpen key categories and themes Develop and record data interpretation and
findings. Document analytic procedures and
decisions. Audit trail that makes analysis process
transparent
28
Codes: Why use them?
Provide rigor to the analytic process.Allow rearrangement of the data into
analytically meaningful categories with study aims always in mind
Create a way to exhaustively identify and retrieve data out of a data set
Enable the researcher to see the bigger picture of data that is difficult to see in transcript form.
29
Codes: Where do they come from?
Deductive codes: Questions and categories generated during
the conceptual and design phases of the study▪ Research questions▪ Conceptual model▪ Interview guide▪ Unit(s) of analysis (e.g., clinic, ICU, provider
group) Inductive codes:
▪ The data.
30
Coding Scheme Development
Iterative approach2 or more researchers involved
Multiple perspectives Resource intensiveProduct is a codebook: Code title,
definition, example
31
Early development
Team members engage in high quality conceptualization through an iterative, negotiated process.
Produce an initial list of codes Who?
All team members who might be involved in coding or later stages of the analysis, including PI
How? Read and code transcripts independently Highlight text relevant and interesting test and
comment in the margins (preliminary coding) Test a priori codes Meet to share impressions of the data.
32
Further Development
Refine coding definitions, add or delete codes Continue iterative process using the initial
codes, usually with a subset of the research team. Independent coding, meeting
Produce codebook with “stable” codes Clear definitions; applied to text consistently by
coders Document changes in codes and code
definitions. Transparent and systematic process
33
Codebook Excerpt
CODE DESCRIPTION EXAMPLE
ENVIRONMENTAL CONTEXT [ENV]
Tree Node
General description of the environmental context not included in the other ENV codes.
External guidelines [GUIDE]
Child Context related to external entities that impacts the organization being interviewed and/or their ability to adopt IC practices. Reference to the role or influence of guidelines in getting practices adopted or implemented. Examples include JCAHO, CDC guidelines, Keystone initiative, and standards of practice reporting mandates 6/13/06. Also, references to relationships with outside entities.
"We've done a good bit with trying to work through JCAHO stuff and change our preps and things like that.""Keystone has really been instrumental in getting the resources we need."
Health system [HS]
Child Description of umbrella entities that are over the purview of the hospital. Examples include the VISN for the VA, a corporate office or sister entities in the case of privately owned hospitals that may be part of a network.
34
Coding agreement: establish coding reliability
Coding agreement: Two or more coders using the same codebook consistently apply the same codes to the same text segments.
35
Measurement statistic
Use quantitative measure of agreement to establish coding reliability
▪ Standard: Kappa statistic >= .7
Underlying philosophy: positivist view that bias introduced by human involvement in research must be minimized to increase the validity of research results.
36
Consensus approach: negotiated agreement processWhen discrepancies occur coders
discuss the rationale they used to apply particular codes to the data.
Explain their perspectives and justifications, how and why it differs from other team members’ perspectives, and reach consensus on how the data ultimately should be coded. Involves reflexivity: confronting your
assumptions, recognizing how they affect what you see
37
Consensus approach: underlying philosophy
Constructivist view that validity is derived from community consensus, through the social process of negotiation. Measuring coding agreement leads to
over-simplification that compromises validity
Reflexivity and reason-giving are more important than achieving a pre-specified level of agreement independently.
38
Which approach?
Factors to Consider Complexity of data and degree of
inference in analysis▪ More complex, higher inference – use
consensus approach Use of the data?
▪ E.g., if transforming qualitative to categorical data, agreement statistic appropriate.
Who is your audience?▪ Some journals expect agreement statistic,
although that’s changing
39
Applying codes and rearranging data
Apply codes to all of the text in the data set. When applying a code to a segment of
text, include text that will provide sufficient context so that its meaning can be discerned out of the context of the transcript.
Text is rearranged into code reports, which list all of the text to which each particular code has been applied.
40
Data Summaries
Descriptive and Interpretive Summaries Includes main points obtained from reading
code report, quotations to provide evidence for those points, an interpretive narrative at the code and case levels.▪ Case summaries grouped together so each can be
examined before making cross-case comparisons Integrate memos
Often done by analysts at the project manager level. Entire team, including those doing the
summaries, must understand which questions the team wants to answer, papers team wants to write
PI and team engagement throughout study
41
TranscriptsTranscriptsTranscripts
Alternative to traditional coding: Team-based Approach
Assign transcripts to pairs of analysts
Develop Memos (+ Quotes)
Large Group Discussion
Modify Memos(+Quotes)
Begin Case Analysis
Pre-existing frameworkor codebook
Initial Memos
Group Memos
Features:• Move more quickly into analysis & interpretation• Multi-disciplinary group process• Multiple perspectives coalesce through consensus• Foster reflexive thinking through deliberation• Efficient for principle investigator• Reduce volume of data going into analysis• Ensure validity
CaseTranscripts
42
Interpretation and synthesis
Data analysis products are further analyzed, interpreted, and synthesized to formulate results.
Identify patterns, formulate preliminary conclusions Tool: data display
▪ Helpful in looking across cases
43
Tool: Data Display
Leadership Provider Commitment
Resistance to Change
Champions
Site 1
Site 2
Site 3
Site 4
44
Ensuring validity of findings
Conclusion-drawing and verification Test conclusions by going back to the data to assess evidence. Look for data that supports alternative explanations Look for negative or deviant cases
▪ e.g., minority group in formative evaluation – why are they in minority? What about them or their situation is different and what does that tell us about the program we’re evaluating?
Ensure that findings are not driven by forgone conclusions or preconceived biases, and that findings are grounded in sound evidence. Reason-giving and examining assumptions in group discussions
Assess degree to which findings contribute to theory or practice, i.e., how useful are they
Review of rigorous procedures for conducting
qualitative research
46
Sustained Engagement
Sustained engagement of the whole team throughout the research process vital to maintaining rigor PI = main interpreter Data collectors Coders/analysts Use multiple coders and analysts Involvement of PI and other team members at
all stages Engage in reflexive discussion
47
Rigorous Procedures: Study Design and Sampling
Match data sources and methods to research questions
Best design given resource constraints
Triangulation of data sources and methods
Purposeful sampling strategy tailored to your study
48
Rigorous Procedures: Data Collection
Systematic translation of research questions into data collection tools
TrainingTranscription rules and
verification
49
Rigorous Procedures: Data Analysis
Systematic approach Constant and iterative Focused Rigorous code development Coding agreement process Document procedures, changes in
codebook, analytic decisions Interpretation and synthesis involving
reflexive discussion to ground findings in sound evidence
Conclusion verification
5050
Michael Q. Patton, Qualitative Research & Evaluation Methods, 3rd ed., Sage, 2002
John Creswell, Qualitative Inquiry and Research Design, Sage
Jennifer Mason, Qualitative Researching, Sage, 2002
Margarete Sandelowski, series of articles on qualitative research in Research in Nursing and Health.
Catherine Marshall and Gretchen B. Rossman, Designing Qualitative Research, 4th Ed., Sage, 2006
Interviewing: Robert S. Weiss, Learning from Strangers, The Free Press, 1994
Focus Groups: David L. Morgan and Richard A. Krueger, The Focus Group Kit, Sage, 1998
Data Analysis: Forman and Damschroder, Coffey and Atkinson
ResourcesResources