A travel guide into the fantastic journey into Qualitative Evidence Synthesis

Karin HannesMethodology of Educational Sciences Research Group

Faculty of Psychology and Educational SciencesKU Leuven, Belgium

Scientific advancements lunch seminars locationNottingham University, UK, 23 November 2014

THE (NOT SO PLEASANT) EXPERIENCE THAT CHANGED MY VIEW ON SYSTEMATIC REVIEWS

Part 1

Meet EmmaBorn the 6th of October 2010

Meet EmmaBorn the 6th of October 2010Little sister of Door and Polle

MAIN QUESTION:How do I get rid of the extra pounds I gained after having delivered a couple of children?

Meet EmmaBorn the 6th of October 2010Little sister of Door and Polle

‘BIASED’ TRANSLATION:What interventions have proven to be effective to loose weight post-partum?

Evidence of what?• Evidence of ‘effectiveness’: the extent to

which an intervention, when used appropriately, achieves the intended effect.

If you don’t know the answer to your question

Where would you go look for it?

I’d go to the Cochrane or Campbell library!

• Women who exercised did not lose significantly more weight than women in the usual care group.

• Women who took part in a diet or diet plus exercise program, lost more weight than women in the usual care.

• There was no difference in the magnitude of weight loss between diet and diet plus exercise group.

• The interventions seemed not to affect breastfeeding performance adversely.

The answer to my question

A study in the Journal of the American College of Nutritian found that those who ate cereals where lower in weight compared to those who ate meat and eggs, bread or skipped breakfast.

The answer to my question

SIMPLE logical reasoning:

• IF a diet helps to loose weight after pregancy• IF cereals have proven to work well as a diet• THEN the consumption of cereals will lead to

weight loss after pregnancy!• Right?

Wrong effect!And then you panic…

… Or you’d go and dig a little deeper!

You’d go and dig a little deeper!

Study 1 (Thornton)• Husbands and female relatives were

the primary sources of emotional, instrumental and informational support.

• Holistic health beliefs and the opinions of others strongly influenced the mothers in their perception about the need to remain healthy.

• Absence of mothers, female relatives, friends to do child care, companionship for exercise and advice about food were barriers limiting women’s ability to maintain healthy practices

You’d go and dig a little deeper…

Study 2 (Setse)• Postpartum depression• Desire to loose weight• Cost of weight programs• Negative impact of media

covering celebrity post partum weight loss

• Family behaviors that promote unhealthy eating

• Impact of child care facilities on ability to exercise

You’d go and dig a little deeper

Conclusion study 1 (Thornton)We need community-based, family oriented programs to increase the chance of successful weight reduction.

Conclusion study 2 (Setse)Weight loss interventions should address the psychological effects of childbearing, affordability and perceptions of body image. They should incorporate family-centred approaches.

Variability in effects

Non-avoidable Individual differences

Evidence of ‘effectiveness’: the extent to which an intervention, when used appropriately, achieves the intended effect.

Variability in effects

Non-avoidable Context!

TEXT MESSAGE: **Honey, I'm running late - please put the chicken on the stove. Love you!**

Cynthia Lum, 2011

(Most) commonly read conclusions of recent systematic reviews of effectiveness…

• More research is needed.• There was too much heterogeneity to pool

statistically.• The results are inconclusive or inconsistent.

Complex interventions

Variability in effects

SIMPLE logical reasoning:

• IF an intervention or program has been proven to work in experimental or ideal conditions…

• THEN there is no valid reason not to adopt it in practice

• Right?

Wrong!

Variability in effects

More advanced logical reasoning:

• IF an intervention or program has been proven to work in experimental conditions…

• IF the intrinsic qualities of the program are good…• THEN it may still fail to work in real practice

(or at least in some regions or situations or populations or……)

• Because the context in which it is applied variesand some aspects of programs are not automatically transferable to other contexts

Type 3 error: an error related to process and implementation aspects of the intervention or program.

If we want to make the results of our systematic review more relevant to end-users…

What sort of roads can we travel?

OPTION 1‘Variable-oriented’ approach

• Create variables that capture discrete aspects of implementation (dose, fidelity etc.) and quantitatively assess these factors in relation to intervention outcomes.

OPTION 2‘Theory-oriented’ approach

• Include an explicit theory or model of how an intervention contributes to a set of specific outcomes through a series of intermediate results” and “include an explanation of how the program’s activities contribute to the results.

OPTION 3A ‘Qualitative’ or a ‘Mixed

method’ synthesis approach

• Using insights from qualitative studies or Combining insights from quantitative and qualitative studies in one overall synthesis. The qualitative insights will allow you to evaluate potential barriers and facilitators toward an intervention or program

OPTION 1‘Variable-oriented’ approach

• Create variables that capture discrete aspects of implementation (dose, fidelity etc.) and quantitatively assess these factors in relation to intervention outcomes.

OPTION 2‘Theory-oriented’ approach

• Include an explicit theory or model of how an intervention contributes to a set of specific outcomes through a series of intermediate results” and “include an explanation of how the program’s activities contribute to the results.

OPTION 3A ‘Qualitative’ or a ‘Mixed

method’ synthesis approach

• Using insights from qualitative studies or Combining insights from quantitative and qualitative studies in one overall synthesis. The qualitative insights will allow you to evaluate potential barriers and facilitators toward an intervention or program

Simple interventions Complicated interventions Complex interventions or situations

OPTION 3:BUYING INTO METHODS FOR QUALITATIVE EVIDENCE SYNTHESIS

Part 2

Conceptualisation of reviews

Configuration AGGREGATION

Metaphors honestly stolen from Gough and Thomas, 2012

Conceptualisation of Qualitative evidence synthesis

“A systematic empirical inquiry into meaning”Systematic

• Planned• Ordered/structured

• “Reconstructed logic of science”

Empirical

• Depends upon the world of experience.

• Builds on what we can capture with our senses.

Inquiry into meaning

• Developing a more complex picture of a phenomenon or situation.

• Rich, Deep, Thick, Textured, Insightful, …

Shank, 2006

‘The process or result of building up separate elements, especially ideas, into a connected whole, especially a

theory or system’ (Oxford English Dictionary)

“A systematic inquiry into meaning”

If ‘configuring’ evidence is considered as the new hype…

Then what sort of questions can we answer using qualitative research

insights in our review?

Questions• Evidence of ‘effectiveness’: the extent to which an intervention,

when used appropriately, achieves the intended effect.• Evidence of ‘feasibility’: the extent to which an intervention is

practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context.

• Evidence of ‘appropriateness’ the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given.

• Evidence of ‘meaningfulness’: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population.

Heyvaert M, Hannes K, & Onghena P. Conducting systematic and comprehensive literature reviews: The Mixed Methods Research Synthesis approach. (2015). In: Mixed method research series, N. V. Ivankova & V. Plano-Clark, In development.

Questions• “A synthesis of evidence from qualitative research can explore questions

such as how do people experience illness, why does an intervention work (or not), for whom and in what circumstances … what are the barriers and facilitators to accessing health care, or what impact do specific barriers and facilitators have on people, their experiences and behaviours? … Evidence from qualitative research can help with interpretation of systematic review results by aiding understanding of the way in which an intervention is experienced by all of those involved in developing, delivering or receiving it; what aspects of the intervention they value, or not; and why this is so. These types of qualitative evidence can provide insight into factors that are external to an intervention including, for example, the impact of other policy developments, factors which facilitate or hinder successful implementation of a programme, service or treatment and how a particular intervention may need to be adapted for large-scale roll-out (Roen 2006).”

Noyes J, Popay J, Pearson A, Hannes K, Booth A. Chapter 20: Qualitative research and Cochrane reviews. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.

Now that we know what a QES can answer…

What methodological pathways are available to us?

Confusing Terminology, Variety of Choices

• Qualitative Systematic Review

• Qualitative Meta-Synthesis

• Qualitative Research Synthesis

Qualitative Evidence Synthesis

• A huge big…

Methodological pathways

Meta-ethnography

Critical Interpretive synthesis

Meta-aggregation

Thematic synthesis

Framework synthesis

Meta- Grounded TheoryMeta-

narrative Ecological triangu-lation

Bad Reasons for Choosing Method

• Frequency of Use of Method (e.g. Meta-Ethnography)

• Popularity/”Sexiness” of Method (e.g. Realist Synthesis)

• What a friend/ colleague/ mentor has used (once!)

• Bad experiences of others (may have been inappropriate!)

To use a Sledgehammer to crack a Nut, or…It doesn’t have to be Meta-Ethnography!

To use a cannon to kill flies!

Good Reasons for Choosing Method

• Type of Question• Aim/purpose and Intended Output• Extent of Description versus Interpretation

(level of depth)+ type of data• Epistemology (your positional stance)

Other Considerations: Methodological Expertise in Team

Available Resources

The method is related to the aim and anticipated end product of the review

– Bring together separate findings into an interpretive explanation that is greater than the sum of the parts (Meta ethnography)

– Critically approach the literature in terms of deconstructing research traditions or theoretical assumptions (CIS)

– Produce theories or models that are based on phenomena involving processes of contextualised understanding and action (Grounded theory)

– Respond to a review need for evaluating an intervention’s appropriateness, acceptability and effectiveness (Thematic analysis)

– Summarize evidence in order to develop lines of action for practice and policy (Meta-aggregation)

– Unpicking the mutually interdependent relationships between persons and environments, by formulating patterns 'With this intervention, these outcomes occur with these population foci and in these settings ( Ecological triangulation)

– Bring together research of widely different designs and paradigms (Meta-narrative)

Noyes & Lewin, 2010 http://cqrmg.cochrane.org/supplemental-handbook-guidance

Methodological pathways

Methodological pathways

Noyes J & Lewin S. Chapter 6: Supplemental Guidance on Selecting a Method of Qualitative Evidence Synthesis, and Integrating Qualitative Evidence with Cochrane Intervention Reviews. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group, 2011. Available from URL http://cqrmg.cochrane.org/supplemental-handbook-guidance

So if we know what our aim is and what we anticipate as an end-product we can start ‘configurating’?

Or what else is important to consider?

The researcher’s worldview

Qualitative ScienceQualitative Inquiry

Meta-ethnography

Critical Interpretive synthesis

Meta-aggregation

Thematic synthesis

Framework synthesis

Meta- Grounded TheoryMeta-

narrative Ecological triangu-lation

The researcher’s worldview

Meta-narrative

Critical interpretive synthesis

Meta-ethno-graphy

Grounded theory

Thematic synthesis

The JBI meta-aggregative approach

Framework synthesis

Ecological triangulation

Subjective idealism

Subjective idealism

Objective idealism

Objective idealism

Critical realism

Critical realism

Critical realism

Scientific realism

Based on Barnett-Page and Thomas, 2009

Idealist Realist

There is no shared reality independent of multiple alternative human constructions

There is a world of collectively shared understandings:

Qualitative ScienceQualitative Inquiry

Knowledge of reality is mediated by our perceptions and beliefs

It is possible for knowledge to approximate closely an external reality

Spencer, 2003

Qualitative Scientists

“To see means” Masters of the window• Clarity• Neutrality

“What you see, is what you get.” Transparancy

Qualitative Scientists

quality of the study

• Check the credibility of the findings in terms of an accurate display of people’s voices

• Check the means to correct for the impact of the researcher on the findings

• Check whether the conclusions are grounded in the data

Before using it for decision making processes

Qualitative Inquirists

“TO see means” MASTERS OF THE LANTERN

• To step in someone else’s shoes

• To explore the dark corners or gaps in our knowledge base

• To go beyond what has been reported in the primary studies

• To problemize existing literature

“Shed light where there has been no light before” ILLUMINATION

Qualitative Inquirists

QUALITY OF THE STUDY• the process of

systematically examining research evidence to assess its relevance and utility for the story line to be developed

before using it to inform a decision

Choosing the right approach to inform policy and practice?

Purpose/Aim

Breadth and depth

Epistemology

Time and resources

Idealist Realist

CONTEXT

So now that we have a good sense of what qualitative evidence synthesis is…

It probably does not make a lot of sense to conduct mixed method

reviews, does it?

Mixing apples with oranges!

Coming up!

• Critique: Mixing apples with oranges again!

Of course it mixes apples and oranges. In the study of fruit nothing else is sensible. Comparing apples to oranges is the only endeavor worthy of true scientists. Comparing apples to apples is trivial. (Gene Glass, 2000)

To conclude• The advantage of a travel guide is that it can be put to use as well

as forgotten… In brief, a travel guide always addresses potential barriers on the road. It also offers suggestions rather than imposing itself on the reader. My talk was meant to be a first explorative guide for researchers and practitioners to the bumpy territory of Qualitative Evidence Synthesis (QES).

• Although QES is a concept for which many roads already have been developed, there is no doubt that those who choose to explore it shall have to travel slowly, often by their own means, on small and potentially difficult roads, pay for their own costs and pack as little as possible.

• Be prepared for a potential delay!Free translation based on:

Hannes K. Understanding the Evidence-Based Movement: a context-specific study on obstacles to implementing Evidence-Based Practice. Doctoral thesis in Medical Sciences, 2008: KU Leuven

Thanks to

• The (ex)-colleagues of the Cochrane Qualitative Research and Implementation Methods GroupJane Noyes, Andrew Booth, Janet Harris, Angela Harden, Jenny Popay, Craig Lockwood, Allan Pearson, Margaret Cargo, James Thomas, Kate Fleming, Ruth Garside

• The colleagues from the EPPI-centre, London• The (ex) colleagues from the Campbell Process and

Implementation Methods Group:Michael Saini, Patricia Rogers

Karin.Hannes@ppw.kuleuven.bePSI, Methodology of Educational Sciences Research Group

KU LeuvenTiensestraat 102

3000 LeuvenBelgium