designing and conducting mixed methods studies
TRANSCRIPT
DESIGNING AND CONDUCTING MIXED METHODS STUDIES
Beth Angell and Lisa Townsend
Workshop for the 2011 Society for Social Work and Research annual meeting
Overview of Workshop
Definitions and terminology of MM ResearchPhilosophical AssumptionsMixed Methods: Nuts and BoltsBreakPlanning your mixed methods study
SamplingData Collection
Data AnalysisEvaluating Mixed Methods StudiesRepresenting Mixed Methods ResearchExamplesQ and A and Technical Assistance
Resources
Creswell & Plano Clark (2011) Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications, Inc.
Teddlie & Tashakkori (2009) Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Los Angeles: Sage Publications, Inc.
Consensus Definition of MM Research
“Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the purpose of breadth and depth of understanding and corroboration”
Johnson et al. (2007).
Key terminology
Qualitative (QUAL) component and Quantitative (QUAN) component are often referred to as strands
Philosophical Issues
Quantitative Tradition
Philosophical underpinnings: positivism/post‐positivism
Deductive logic
Data are represented numerically
Associated terms: survey research, probability sampling, experimental and quasi‐experimental designs, descriptive and inferential statistics
Qualitative Tradition
Philosophical underpinnings: constructivism
Inductive logic
Data are represented textually or pictorially
Associated terms: grounded theory, ethnography, case studies, purposive sampling, categorical vs. contextualizing strategies, trustworthiness, credibility
Mixed Methods Tradition
Philosophical underpinnings: pragmatism
Both deductive and inductive
Data are represented both numerically and textually/pictorially
Associated terms: concurrent (parallel) and sequential mixed designs, triangulation, data conversion, inference quality
Philosophical debates about mixed methods
Incompatibility thesis – fundamental differences between QUAN and QUAL approaches are so great that methods cannot be mixed
Pragmatism – what is the best way to answer a research question; both methods offer different ways of answering research questions
Mixed Methods: Nuts and Bolts
Characteristics of MM Studies
Involves collection and analysis of qualitative and quantitative data in ways that are:
rigorous framed epistemologically/theoretically
The methods are mixed by ordering them sequentially merging them embedding one strand within the other
Combines the data within the context of a single study or research programEncapsulates the strands within an overall research design that guides the study as a whole
Research Questions that Call for MM
Exploring the meaning of a construct or phenomenon from more than one perspective
Explanation of anomalous findings or getting behind the mechanism of action of an effect
Theory development followed by testing/extension
Measure development using grounded concepts
Augmenting evaluation studies with better understanding of intervention implementation
Ways That Designs Vary
Level of interaction between strands
Relative priority of strands
Timing or pacing of each strand
Point of interface (at which point in the research process are the strands mixed?): during interpretation, data analysis, data collection?
Research stance, epistemology
Typology of Mixed Methods Designs
Convergent parallelExplanatory sequentialExploratory sequentialEmbeddedCaveat: evolving field with evolving language
Adapted from Creswell & Plano Clark (2011)
Convergent Designs
QUAN and QUAL strands are conducted separately yet concurrently and merged at the point of interpretation
Equal priority given to each strand
Used to form a more complete understanding of a topic, or to validate or corroborate quantitative scales
Convergent Parallel Design
QUAL
Data Collection
Compare or Relate
Interpretation/Meta‐
Inference
QUAN
Data Collection
QUAL
Data Analysis
QUAN
Data Analysis
Convergent Parallel Design Example: Conceptual Adequacy of the Convergent Parallel Design Example: Conceptual Adequacy of the Drug Attitude Drug Attitude Inventory for YouthInventory for Youth
QUANData
Collection
QUANData
Analysis
QUALData
Collection
QUALData
Analysis
Design Decisions Design Decisions
Choice of instrumentQuestionnaireRating scaleSampling ConvenienceRepresentative
Semi-structured interviewsYouth SEMIParent SEMI
Compare Contrast
Mixed Methods Question: Can prediction of youth attitudes Mixed Methods Question: Can prediction of youth attitudes toward psychotropic treatment be improved by knowledge about thetoward psychotropic treatment be improved by knowledge about thefactor structure of the DAI in youth and their subjective experifactor structure of the DAI in youth and their subjective experiences ences of treatment?of treatment?
DemographicsYouth DAIParent DAIAdherence RatingsClinical Scales (CDRS,YMRS,CBCL)
Choice of methodInterviewEthnographyFocus groupSamplingPurposiveConvenienceSetting
Metainference
Townsend, Floersch, & Findling, 2010
Explanatory Sequential Design
Methods are implemented sequentially, (QUAN →QUAL)
Used when researcher wishes to use qualitative findings to help interpret or contextualize quantitative results
QUAL Data Collection and Analysis
Interpretation/Meta‐
Inference
Follow up with
QUAN Data Collection
and Analysis
Explanatory Sequential Example: ACT Social Network Study
ACT Randomized Trial : No Social Network Effects
QUAN Analysis of Social Network Predictors
QUAL study of RCT Subsample
QUAN Data Collection
and Analysis
QUAL and QUAN Data Collection and Analysis
Interpretation/Meta‐
Inference
Interpretation/Meta‐
Inference
Angell & Test, 2002; Angell, 2003
Exploratory Sequential Design
Methods are implemented sequentially, (QUAL →QUAN)
The QUAL strand is considered exploratory, to be followed by further testing and verification during the QUAN phase
Qualitative Data Collection and Analysis
InterpretationBuilds toQuantitative
Data Collection and Analysis
Exploratory Sequential Design Example: Measuring Procedural Justice (PJ) in Police Encounters
Review of existing instruments and literature led to research question: Do existing PJ instruments capture features of contacts between police and citizens with mental illness?
QUAL strand:•Consumer interviews•Analysis of discrete encounters using grounded dimensional analysis
Interpretation: PJ experiences are a) contextualized by negative expectations and b) sensitive to small gestures of humanity
QUAN strand•Instrument development•Cognitive interviewing•Expert review•Survey of consumers using final instrument (PCES)•EFA and Rasch Analysis
Interpretation: PCES predicted reactions to police encounter (resistance, cooperation)
Watson, Angell, Vidalon, & Davis (2010)
Embedded Design
Researcher conducting either a QUAL or QUAN study embeds a smaller strand of the other method, as an enhancement
Secondary strand can be concurrent or sequential
Qualitative or Quantitative Design
QUAL or QUAN Data Collection and Analysis
QUAL or QUAN Data Collection and Analysis (before, during, or after)
Interpretation/Meta‐Inference
Embedded Design Example: CTI Evaluation and Fidelity Study
RCT of Critical Time Intervention (CTI) for Men Leaving Prison. QUAN Data Collection, n=220
Fidelity/Process StudyQUAL data collection, n = 24
Data Collection Decisions:InterviewsFocus GroupsFieldnotesRecord abstractionSampling criteria
MM Research Questions: In what ways is Critical Time Intervention modified or adapted when used with a population of recently released prisoners? What processes contribute to the adapted program’s level of effectiveness?
Data Analysis Decisions:Coding (open, selective, axial) of interviews and documents/Narrative analysis?Draine, Angell, Barrenger, & Kriegel (in progress)
Variations on the MM Designs
Multiphase format
Multilevel format
Monostrand Conversion (not truly mixed methods): conversion of QUAL data to QUAN or QUAN data to QUAL, without additional strands
Transformative stance
Questions?
Break
Planning Your Mixed Methods Study
Sampling: General Considerations
Strategy chosen should be appropriate to each respective strand
Balance between saturation of phenomenon or theory (qualitative goal) and representativeness (quantitative goal)
Sampling Strategies (Teddlie and Tashakkori, 2010)
Parallel mixed methods sampling (parallel use of probability and purposive strategies, either concurrently or with a time lapse).
One sample may be a subset of the otherBoth studies may use same total sample
Sequential mixed methods sampling (information from the first sample is used to draw the second)
Multilevel mixed methods sampling: using probability and purposive sampling techniques at different levels of analysis (e.g., clinicians and clients)
Data Collection(Teddlie & Tashakkori, 2009)
Self‐Report TechniquesInterviewsQuestionnairesAttitude ScalesPersonality inventoriesProjective instruments
Observational MethodsParticipant observation, non‐participant observation
SociometrySocial network analysis
Secondary Data AnalysisArchival analysisMeta‐analysis
Multiple Modes of Data Collection (Tashakkori & Teddlie, 1998)
Data Analysis
Quantitative Data Analysis
Descriptivesummarizing data, looking for trends and patterns; means, frequencies, measures of variability
Inferentialhypothesis testing, inferences about a population characteristic; significance tests (χ2,t, F), multiple regression, ANOVA, MANOVA, MANCOVA, hierarchical linear modeling, time‐series, event history
Qualitative Data Analysis
Often ongoing during data collection (e.g., purposive sampling, modification of interview questions, etc.)
Grounded theoryThematic analysisNegative case analysis
FRACTURING VS. CONTEXTUALIZINGCategorical strategies: produce categories that facilitate comparisons; e.g., constant comparative methodContextualizing strategies: interpret narrative data in the context of the whole text, focusing on interconnections between statements, events, etc.; e.g., phenomenology
SIMILARITY VS. CONTRAST
Mixed Methods Data Analysis(Creswell and Plano Clark, 2011)
QUAN+QUAL = converge results CONVERGENT DESIGNQUAN → qual = explain results SEQUENTIAL EXPLANATORY DESIGNQUAL → quan = generalize findings SEQUENTIAL EXPLORATORY DESIGNQUAN (+qual) = enhance experiment EMBEDDED DESIGNTRANSFORMATIVE DESIGN – uses a transformative theoretical perspective to advocate for social change, address social injustice, or give voice to marginalized/underrepresented group.MULTIPHASE DESIGN – a program of research that involves several studies; can have combinations of sequential and concurrent designs
Mixed Methods Data Analysis(Creswell & Plano Clark, 2011)
Convergent parallel: merged data analysis for purposes of comparing results
Collect and analyze QUAL and QUAN dataStrands are analyzed independently (could be qualitizing/quantitizing strategies also)How will the two strands be compared?How will they be represented?
Explanatory: connected data analysis to explain findingsCollect and analyze quantitative data; derive second research questionDesign and conduct qualitative researchAnalyze qualitative data for answers to secondary research questionLink results from both strands – how do qualitative results explain quantitative findings?
Convergent Parallel Design: Data Analysis of the Drug Attitude Inventory
QUANData
Collection
QUANData
Analysis
QUALData
Collection
QUALData
Analysis
Univariate descriptivesBivariate correlationsStructural Equation ModelingFactor analysisParallel Analysis(SPSS, LISREL)
In vivo codesIntermediate codesSuperordinate codesPeer reviewConstant comparative approach(Atlas TI)
Compare Converge
Can prediction of youth attitudes toward psychotropic treatmentCan prediction of youth attitudes toward psychotropic treatment be be improved by knowledge about the factor structure of the DAI in improved by knowledge about the factor structure of the DAI in youth and their subjective experiences of treatment?youth and their subjective experiences of treatment?
Does the factor structure of the DAI in adults fit the youth data?
If not, what is the factor structure of the DAI in youth?
How well do DAI items correlate with one another?
Do they measure a single construct or multiple constructs?
Are there elements ofyouth medication experience that the DAI does not capture?
Mixed Methods Data Analysis(Creswell & Plano Clark, 2011)
Exploratory: connected data analysis to generalize findings
Collect and analyze qualitative data; use qualitative data to design quantitative componentCollect and analyze quantitative dataLink results from both strands: how do quantitative results extend qualitative findings?
Embedded design: merged (concurrent design) or connected (sequential design) analysis
Collect and analyze primary data set; decide how embedded data will be used and where they should be incorporated into the primary analysisAnalyze secondary data set dictated by where it is embedded in the larger designHow do the embedded findings integrate with the primary study findings?
Evaluating Mixed Methods Studies
Mixed Methods Validation Framework (VF)(Dellinger & Leech, 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010)
Five elements:Foundational element
Quality of literature review and theory baseConstruct validation
Validity of QUAN, QUAL, and mixed elementsInferential consistency
Consistency of links between various strands of the study (see table on following slide)
Utilization/historical elementWhether and how the study’s findings went on to be used in future work
Consequential elementSocial acceptability and consequences of study findings
Construct Validation(Dellinger & Leech, 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010)
Representing Mixed Methods Data(Creswell & Plano Clark, 2011)
Side‐by‐side comparison
Joint comparison
Merged category/theme display
Writing the article(Creswell, 2003)
Introduction – explicit integration of both paradigms from the outset
Literature review – integration of inductive/deductive reasoning, why the literature needs this type of study
Posing the research question – what are the questions and why do they call for two paradigms?
Methods – present both methodologies, in their respective languages, integrated under the umbrella of the research question
Results – present results of both modes of data collection
Discussion – role of meta-inference
Findings from DAI Study
(Townsend, Floersch, & Findling, 2010)
Example I
Study Flowchart
QUANData
Collection
QUANData
Analysis
QUALData
Collection
QUALData
Analysis
DemographicsYouth DAIParent DAIDecision-Making ScalesAdherence RatingsClinical Scales (CDRS,YMRS,CBCL)
Univariate descriptivesBivariate correlationsStructural Equation ModelingFactor analysisParallel Analysis(SPSS, LISREL)
Semi-structured interviewsYouth SEMIParent SEMIBrief Parent SEMI
In vivo codesIntermediate codesSuperordinate codesPeer reviewConstant comparative approach(Atlas TI)
Compare Converge
Can prediction of youth attitudes toward psychotropic treatmentCan prediction of youth attitudes toward psychotropic treatment be be improved by knowledge about the factor structure of the DAI in improved by knowledge about the factor structure of the DAI in youth and their subjective experiences of treatment?youth and their subjective experiences of treatment?
Parallel Data Reduction Strategies
@stigma {0-56}
&Desire_for_Normality {0-0} &Crazy_Identity {0-0} &Educating_Others {0-0}
+want_to_live_normal_life_without_meds {1-0}
+nobody_else_takes_meds {2-0}
+for_crazy_people {2-0}
+Labeled_psycho {2-0}
+did_bipolar_slide_show_at_school {1-0}
+I_like_explaining_meds_to_people {1-0}
Structural Equation Model OneDAI Original Factor Structure
RMSEA .061 [ideal = <.05 (Kaplan, 2000)]
CFI .925 NNFI .913
[ideal = >.95 (Kaplan, 2000)]df 258
X2 420.38
Exploratory Factor AnalysisFactor Selection Criteria:
Maximum likelihood estimationEigenvalue > 1.0Minimum item loading > .30Retained 4+ items>4 items but differentiated well from other factorsQualitative data indicate retention of items/factors is justified
EFA OneTwo factors were not interpreted further because they were each comprised of only one item (items 8 and 13) and had values > 1.0. (Heywood cases)
EFA Two‐28 items (Supplemented by Parallel Analysis)Two factors retained, accounting for 36.61% of the variance in DAI score.Factor labels:
“Positive Feelings toward Medication”“Negative Feelings toward Medication”
Four items did not load on any component (10R, 11R, 20R, and 30)Cronbach’s alpha = .889Youth DAI correlated positively with youth self‐reported adherence (r = .205, p<.05)DAI showed no significant correlations with clinical outcome measures (CBCL Competence and Symptom Scales).
Qualitative Analysis Methods
Constant comparative approach – analysis of in vivo codes followed by intermediate level coding and synthesis into higher order superordinate concepts.
@stigma {0-56}
&Desire_for_Normality {0-0} &Crazy_Identity {0-0} &Educating_Others {0-0}
+want_to_live_normal_life_without_meds {1-0}
+nobody_else_takes_meds {2-0}
+for_crazy_people {2-0}
+Labeled_psycho {2-0}
+did_bipolar_slide_show_at_school {1-0}
+I_like_explaining_meds_to_people {1-0}
Summary of Qualitative Themes
DAI‐related concepts
Positive feelings
Negative feelings
Health/Illness ModelHealth/Illness Model
Internal LOC
External LOC
Relapse PreventionRelapse Prevention
Concepts not represented in the DAI
Balanced responsibility
Ambivalence
Change over Time
Adherence if effective
Expectations
Inclusion in treatment decisions
Autonomy
Stigma
Side‐by‐Side Comparison Example: Conclusions Section
In summarizing the findings of the adolescent qualitative and quantitative analyses, it is apparent that the seven dimensions found in the original DAI employed with adults are present in teens’ conceptualizations of their views of medication. However, adolescents may think about those dimensions, such as side effects and relapse prevention, differently than adults do. These findings are supported by both the EFA and qualitative results. EFA findings indicate that side effects differentiated into specific cognitive effects as well as other harms for youth. Adolescents did not appear to view relapse prevention as the adults did, instead evaluating their perceived need for medication in relation to their symptom stability in forming their attitudes toward medication.
The qualitative data highlighted the presence of eight additional themes not reflected in the original factor structure of the DAI, including important themes such as personal autonomy, inclusion in medication decision‐making, the experience of stigma, and level of personal responsibility vs. external control over behavior. These themes highlight that medication experiences and the formation of attitudes toward pharmacological treatment for youth are complex and multiply‐determined, rendering it difficult to represent adolescent attitudes toward medication fully with a single instrument. These findings point to the potential value of creating individualized, qualitative assessment tools to capture youth experiences with medication rather than relying on a single quantitative measure or set of subscales.
Quantitative Findings Qualitative Findings
Discussion section highlights divergent results
Joint Data Display ExampleHighlights Convergence of Factor Analytic and Thematic Analysis Results
Positive Feelings Factor Loading
2 – good outweighs bad .474
21 – thoughts are clearer .789
26 – happier on meds .487
29 – in better control .598
Qualitative Themes Quotations
Emotional “I can either not take it and be a grouch, or take it and be happy. So I would much rather take it and be happy.”
Cognitive “Sitting down, paying attention to the teacher, not talking. Just paying attention and doing what I need to do.”
Physical “I gain energy to want to do my work and listening and focus and try to do my best and give 110%.”
Negative Feelings Factor Loadings
5R – take b/c of pressure from others
.315
14R – medication is slow‐acting poison
.714
19R – I’d rather be sick than taking medications
.658
Qualitative Themes Quotations
Emotional “I don’t like sometimes I feel like out of my body or just like not myself, more anxious sometimes.”
Cognitive “Well I had it told me the Topamax or whatever causes short‐term memory loss, so I assume it’s the Topamax or whatever.”
Physical “Make me sick. Make me get the bubble guts and stuff. And I wish that there was a medicine that didn’t have side effects.”
Merged Data Display ExampleCounts of quotations represented graphically to demonstrate convergence
with factor analyses
“positive” and “negative” factors supported by all forms of QUAN and QUAL evaluation
Distribution of DAI-Related Quotations across Respondents
0
5
10
15
20
25
30
35
40
4 7 10 12 13 14 16 17 18 20 22 23 24 25 27 29 30 32 38 41
Respondent
Num
ber o
f Quo
tatio
ns
PositiveNegativeIllness ModelILOCELOCHarmRelapse Prev.
Number of adolescent quotations linked to original DAI factors
Positive Subjective
Feeling
Negative Subjective
Feeling
Health/Illness Model
External Locus of Control
Internal Locus of Control
Harm/Toxicity
Relapse Prevention
103 31 21 20 10 48 34
Example II
Forensic Assertive Community Treatment (FACT) Evaluation (Angell & Watson, in progress)Key mixed method questions:
How is ACT modified in the context of prison reentry?How does the agency’s recovery‐oriented mission shape the translation of ACT to FACT?How does engagement of consumers occur and to what extent is leverage involved?What are the unique features of client‐provider relationships in FACT?How does engagement relate to rates of success (avoidance of reincarceration)?
Overall Design: Convergent Parallel
Sample=21 adults with mental illness leaving prison and entering FACT program
QUAN Data Collection
QUAL Data Collection
Panel Design•Client reported outcomes assessed at BL, 1, 3, 6, 9, 18 mos• staff reported outcomes monthly
Ethnographic approach• Participant and passive observation•Interviews with consumers and staff• Record abstractionsQUAN
Data AnalysisQUAL
Data Analysis
Within‐Subject Cross‐Data Comparisons
Interpretation/Meta‐Inference
Analysis Strategies
Sample=21 adults with mental illness leaving prison and entering FACT program
QUAN Data Collection
QUAL Data Collection
QUAL• Descriptive•Pre‐Post Comparisons• Outcome differences by risk level
QUAL• open and focused coding• creation of synthetic participant narratives
QUAN Data Analysis
QUAL Data Analysis
Within‐Subject Cross‐Data Comparisons
Interpretation/Meta‐Inference
Mixed Analysis:Comparing phenomena•Across data type•Across time•Across subjects
Q and A/Technical Assistance