Appraisal, Extraction and Pooling of Qualitative Data and Text

- Evidence from qualitative studies, narrative and text.

JBI/CSRTP/2013-14/0003

Introduction• Recap of Introductory

module– Developing a question (PICo)– Inclusion Criteria– Search Strategy– Selecting Studies for Retrieval

• This module considers how to appraise, extract and synthesize evidence from qualitative studies and text.

Program Overview

Program Overview

Session 1: Introduction to Qualitative Evidence Synthesis

Qualitative Research is…• “…a situated activity that locates the observer in the world. It

considers a set of interpretive, material practices that make the world visible…[it] involves an interpretive, naturalistic approach to the world. This means qualitative researchers study things in their natural settings, attempting to make sense of, or to interpret, phenomena in terms of the meanings people bring to them” (Denzin & Lincoln, 2000)

Qualitative research and health care• Health care is a people centered process concerned with

health and healing– Health care derives much of it’s knowledge from a range of

biomedical sciences

• However this does not supply all the knowledge that is necessary to provide holistic care for the patients and clients

• Qualitative researchers use humanistic frameworks to examine ordinary activities of everyday life

Qualitative Research Findings as Evidence for Practice

• Qualitative evidence is of increasing importance in health services policy, planning and delivery.

• It can play a significant role in: – understanding how individuals / communities perceive health,

manage their own health and make decisions related to health service usage;

– increasing our understandings of the culture of communities and of health units;

– Informing planners and providers;– evaluating components and activities of health services that

cannot be measured in quantitative outcomes.

Application of Qualitative Research to Practice

• As with quantitative research, results from a single study only should not be used to guide practice

• The findings of qualitative research should be synthesized in order to develop recommendations for practice

Research Paradigms

• Serve as a model/world view or philosophy that structures knowledge and understanding

• Define what is to be observed and scrutinised• Prioritise questions• Inform the interpretation of results

Dominant Paradigm 1: Positivist/Empirical-Analytical

• Assumptions– There is an external, real world of objects apart from

people, which is comprehensible to us– Researchers can compare their claims against this

objective reality

• Seeks to predict and control

Dominant Paradigm 2: Interpretive• Assumptions

– Reality as we know it is constructed/produced inter-subjectively (between people).

– Meaning and understanding are developed socially and experientially.

– Researchers’ values are inherent in research. – Findings of knowledge claims are created– All interpretations are located in a particular context,

setting and moment.• Seeks to understand

Dominant Paradigm 3: Critical• Assumptions

– Examines issues of power and envisions new opportunities– Main task social critique, whereby the restrictive and alienating

conditions of the status quo are brought to light– Common assumptions with Interpretivism; however, moves past

describing meaning to emancipation and change. • Seeks to critique and emancipate

Qualitative Methodologies

Interpretive Methodoligies

Phenomenology• Assumes that people come to know a phenomena

through their experience of it• Focuses on INDIVIDUAL meaning• Emphasizes a focus on people's subjective

experiences and interpretations of the world. • Understand how the world appears to others.

Ethnography• Study of Culture/specific cultural group.

– What is the way of life of this group of people?– Everyday life is worth of study.

• Focuses on SOCIAL meaning.• Researcher immerses self in cultural group.

– Field Work - primarily participant observation and interview data

• Focuses on the context of communities• Used increasingly in healthcare to explore the relationship

between health and culture.

Grounded theory• Develop theory grounded in real world;• Searches and generates theoretical explanations

from observations of the world (induction);• The constant collection and iterative analysis of data

to enable theory to emerge; and• Data are not constrained by predetermined

theoretical framework, but define boundaries of inquiry.

Critical Methodologies

Action Research• Basis in critical social science;• Researchers interact with the

participants to achieve change;• Often community-based;

– used with implementation studies in healthcare; and

• Treats the individual as an autonomous being, capable of exercising agency

Discourse analysis• Discourse – to talk, converse; hold forth in speech or

writing on a subject;• Discourse as patterns of ways of representing

phenomena in language;• Application of critical thought to social situations and

the unveiling of (hidden) politics within socially dominant or marginalised discourses.

Qualitative data analysis• Is based on principles of thematic analysis for most

methodologies– A process of identifying and presenting meaning in the

form of themes, metaphors or concepts;• there is wide variability in the terms used and how they are

defined;

– Not suited to all methodologies (i.e. discourse analysis)

Qualitative data analysis

• Three strategies:– Data analysis as a separate step following data collection– Data analysis occurs simultaneously with data collection– Data collection and analysis are ‘staged’

Congruity between Paradigm, Methodology and Methods

Group Work 1• In pairs, work through the Group Work 1 task in the

Workbook.• Reporting back

Session 2: The Systematic Review of Qualitative Evidence

Trends and Debates within the International Literature

Approach Purpose Evidence of Interest

Search Strategy

Critical Appraisal

Data Extraction

Method of Synthesis

Outcome Software Available

Narrative Synthesis

To summarise two or more papers in narrative form

Generic Unspecified/Selective

Not specified Not Specified

Unspecified Review article-type report

NO

Meta-ethno-graphy

The generate new knowledge/theory use processes of interpretation

Findings of qualitative research studies

Not comprehen-sive or exhaustive; seeks saturation – theoretical sampling

Opposed; all studies included as each may provide insight into the phenomena of interest

Extraction of key concepts

Refutation-al synthesis;Reciprocal translation;Line of argument synthesis.

Higher order interpretat-ion of study findings

NO(Although QARI can be used)

Meta-synthesis: approaches

Approach Purpose Evidence of Interest

Search Strategy

Critical Appraisal

Data Extraction Method of Synthesis

Outcome Software Available

Realist Synthesis

To develop and test program theories

Generic; but more suited to qualitative research studies

Not comprehen-sive or exhaustive; seeks saturation

Not specified Extracts positive and negative mechanisms/strategies that influence change

Tacit testing of implicit theories; building theory.

A transfer-able theory on “what works, for whom, in which circum-stances”

NO

Thematic analysis

To aggregate findings of 2 or more studies

Findings of qualitative research studies

Not comprehen-sive or exhaustive; seeks saturation – theoretical sampling

Not specified Extracts of major/recurrent themes in literature

Aggregation of themes/metaphors/categories

A summary of findings of primary studies under thematic headings

NO(Although QARI can be used

Approach Purpose Evidence of Interest

Search Strategy

Critical Appraisal

Data Extraction

Method of Synthesis

Outcome Software Available

Content analysis

To analyse then summarise content of papers.;occurrences of each theme counted and tabulated

Generic Usually comprehen-sive with predeterm-ined search strategy

Not specified

Content extracted then coded using extraction tool designed to aid reproducibility

coded data categorised under thematic headings; also counted and tabulated

A summary of findings, and their rate of occurrence, of primary studies under thematic headings

YES

Meta-synthesis/Meta-aggregation

To aggregate the findings of included studies

Findings of qualitative research studies

Comprehen-sive; detailed search strategy at protocol stage required

Required, using standard-ised critical appraisal instrument

Extraction of findings PLUS data that gives rise to finding using data extraction instrument

Aggregation of findings into categories; and of categories into synthesised findings

Synthesised findings that inform practice or policy in the form of a standardised chart

YES

Meta-synthesis: worked examples

Meta aggregationMeta ethnography

Meta aggregation

• A structured and process driven approach to systematic review drawing on the classical understandings and methods associated with systematic review of quantitative literature as practiced by the Cochrane Collaboration

Meta aggregation

• Based on an a-priori protocol– Established, answerable question– Explicit criteria for inclusion– Documented review methods for searching, appraisal,

extraction and synthesis of data

Meta aggregation• Explicitly aligned with:

– Philosophy of pragmatism• Delivers readily useable findings• Informs decision making at the clinical or policy level

– Transcendental phenomenology• Looks for common or “universal” essences of meaning• Attempts to “bracket” pre-understandings of the reviewer

Transcendental Phenomenology

• Based on the philosophic traditions of Husserlian phenomenology:– the intuitive examination of essences that have immediate

validity;– Seeks to avoid undue influence of the reviewer on the text;– Seeks to generate practice level theory that has explanatory

power for policy or practice– Seeks to preserve the intended meaning of text

The needs and experiences of people with a diagnosis of skin cancer: a

systematic review Janet Barker, Arun Kumar, Wendy Stanton and Fiona Bath-Hextall, JBI Library of

Systematic Reviews, 2011; 9(4):104-121

Inclusion Criteria• Participants

– Adults with a diagnosis of skin cancer.

• Phenomena of Interest – The needs and experiences of people with skin cancer.

• Context– Hospital or community-located dwellers.

• Types of studies – All qualitative studies that described or analyzed the needs and experiences

with people who had been diagnosed with skin cancer.

Methods• Each paper was assessed independently by two

reviewers for methodological quality. The internal validity (quality) of research papers was assessed using JBI-QARI.

• There were no disagreements between the two reviewers and therefore a third reviewer was not required.

Results

Results ( cont/…)

• A total of 12 findings were extracted from the 2 included qualitative papers

• These findings were aggregated into 4 categories on the basis of similarity of meaning.

• The categories were synthesized to generate 2 synthesized findings.

Results/cont..

• The 2 studies included in the review used a qualitative approach but neither specified a particular methodological approach.

• Both studies were conducted in the United Kingdom (UK), using semi structured interviews.

Recommendations arising• There is a real need to increase knowledge of skin cancer so

that people do not delay in seeking medical help as early diagnosis can dramatically improve both prognosis and the patient experience since early lesions are treated more simply compared with larger or neglected lesions.

• Health professionals caring for these patients need to understand the psychosocial concerns of this patient group in order to design services appropriately and to provide patients with the support they need and information that they can easily understand.

Meta-Ethnography• Explicitly aligned with philosophy of interpretivism• Searches for new meaning• Focuses on multiple realities

Hermeneutic Interpretivism• Is the basis for meta-ethnography

– Interpretation of text for its inner meaning– Focuses on the reviewers interpretive skills– Seeks to re-interpret the published literature– Seeks to generate new, mid level theoretical

explanations– Seeks engagement between reviewer and text

Meta ethnography

• From within the social sciences to develop theories from existing ethnographic data,

• Iterative development of emic interpretations, • Incorporates 7 phases, these can be aligned to the

systematic review process, but there is no requirement to do so.

Meta ethnography• Identify the research interest• Decide which studies to include• Read the studies• Determine how the studies are related• Translate the studies in to each other• Synthesize the translations• Express the synthesis

Meta ethnography: 3 stages of synthesis

• First order interpretations– Themes, metaphors or concepts identified

• Second order interpretations– The researchers interpretation of how the identified

concepts relate to each other

• Third order interpretations– Seeks to encompass themes within each other

Interpretation Second Order Interpretation:

• Reciprocal– Like interpretations are brought together “this one is like that one..”

• Refutational– Competing discourses; must also have a relationship that can be

explored

Third Order Interpretation:

• Line of argument– Asks what do the parts infer about the whole

Patient adherence to Tuberculosis treatment

Salla A. Munro, Simon A. Lewin, Helen J. Smith, Mark E. Engel1, Atle Fretheim, Jimmy Volmink. ‘Patient Adherence to Tuberculosis Treatment: A Systematic Review of Qualitative Research’.

PLoS Medicine July 2007.

Inclusion criteria

• Patients, carers or health professionals delivering DOTs

• Perceptions of adherence was the phenomena of interest

• Context was patients undergoing DOTs therapy for TB• Qualitative studies on the perception of adherence

Methods

• Studies were assessed using a checklist• Data extracted using a standardized form• Synthesis was reciprocal and line of argument

Results

• Following screening, 44 papers were included• First order interpretations 8• Second order interpretations 6• Third order interpretations 4• Expression of the synthesis: visual model

Recommendations• Increase visibility of TB programs in the community, which may

increase knowledge and improve attitudes towards TB• Provide more information about the disease and treatment to patients

and communities• Increase support from family, peers, and social networks• Minimize costs and unpleasantness related to clinic visits. Increase

flexibility/ patient autonomy

STEP 2: CATEGORIES

STEP 3: SYNTHESISED FINDINGS

FIRST ORDER ANALYSIS

SECOND ORDER INTERPRETATION

THIRD ORDER INTERPRETATION

QARI METAGGREGATION

META ETHNOGRAPHY

Session 3: Critical Appraisal of Qualitative Evidence

Why Critically Appraise?

• Combining results of poor quality research may lead to misleading understandings of issues explored

1004 references

832 referencesScanned Ti/Ab

172 duplicates

117 studiesretrieved

715 do not meetIncl. criteria

82 do not meetIncl. criteria

35 studies forCritical Appraisal

The Critical Appraisal Process• Every JBI review must set out to use an explicit

appraisal process. Essentially:– A good understanding of philosophical perspectives,

methodologies, methods is required in appraisers; and– An agreed instrument/checklist should be used.

Validity, Reliability and Generalizability

• Judging the quality of qualitative research remains deeply contested

Analogous criteria for paradigmatic assumptions

Quantitative Qualitative

Reliability Dependability

Internal Validity Credibility

External Validity Transferability

Reliability/dependability

• Appropriateness of methodology, methods and implementation of the research methods, regardless of paradigm

• The focus of dependability is on achieving consistent quality rather than repeatability.

Internal validity/credibility• Credibility addresses whether a finding has been

represented correctly– Assessment of credibility is multi-dimensional, including

goodness of fit and representativeness, – Credibility is auditable - the process may be based upon

researcher confirmation, member checks, peer checks, second researcher analysis, or observation.

External validity/transferability• Findings are not generalizable in the quantitative sense of

the word – generalization is “narrowly conceived in terms of sampling and

statistical significance.”– “qualitative research is directed toward naturalistic or idiographic

generalizations, or the kind of generalizations made about particulars”

– Schofield (1990) describe qualitative metasynthesis as “cross-case generalizations created from the generalizations made from, and about, individual cases.” Sandelowski et

al(1997)

Critical Appraisal• General acceptance of need for quality• Ongoing debate around the role of appraisal• Particular focus of debate on role of scales and sum

scores in appraisal• In practice, appraisal instruments for qualitative

research tend to focus on establishing the degree to which the evidence applies to practice (transferability) rather than internal validity (credibility)

The premise

• Methods applied during the systematic review of qualitative evidence should firstly be congruent with the universally accepted process of systematic review.

• The characteristics of a systematic review might be debated in terms of the detail, but there is general acceptance of a series of steps, stages or processes

The Joanna Briggs Approach• Establishing credibility in qualitative research requires:

– A standardized approach that is clearly defined, articulated, and applied, and

• Takes in to consideration the relationship between:– Methodology and methods– Methods and data– Data and conclusions

The Qualitative Assessment and Review Instrument (QARI)

• QARI provides a systematic process mirroring that of systematic reviews of quantitative research whilst sensitive to the nature of qualitative data.

• a significant proportion of qualitative work is not designed to address questions or issues of an immediate practical nature

• this is consistent with the traditions of many qualitative methodologies

• JBI focuses on evidence to improve global health, reviews with a clinical or policy focus are encouraged.

Critical Appraisal in JBI-QARI

Critical Appraisal of Interpretive Research

• In interpretive and critical inquiry validity relates to the rigor of the process of inquiry– limiting bias to establish validity in the appraisal of

quantitative studies is antithetical to qualitative approaches to inquiry

• QARI incorporates a checklist to appraise rigor generically

• Read the 2 papers provided and then undertake critical appraisal of the paper using the JBI Critical Appraisal Checklist for Interpretive and Critical Research.

• Report Back• Enter the details into JBI QARI

Group Work 2

Session 4: Data Extraction

Study data and Data Extraction

Data most frequently extracted

1004 references

832 referencesScanned Ti/Ab

172 duplicates

117 studiesretrieved

715 do not meetIncl. criteria

82 do not meetIncl. criteria

35 studies forCritical Appraisal

26 studies incl.in review

Considerations in Data Extraction • Source - citation and contact details• Eligibility - confirm eligibility for review• Methodology - approach to the inquiry• Methods - how data were collected• Phenomena of Interest - the described experience or activity• Setting - specific context of study eg, hospital• Geographical location - region or country• Culture - cultural characteristics of setting or participants• Participants - demographic data such as gender, age etc• Data analysis - analytic approach taken, eg thematic analysis etc

Extracting Findings• The units of extraction in this process are specific findings (and illustrations from the

text that demonstrate the origins of the findings);• In Meta-aggregation a finding is defined as: A conclusion reached by the

researcher(s) and often presented as themes, metaphors, findings, concepts, conclusions etc;

• Some papers may present themes and sub themes (or findings of the qualitative analysis at different levels); for such papers the reviewer will need to decide which of these levels represent authentic findings; however, only findings of one level should be extracted from a specific paper.

• No matter what level is decided, the actual verbatim words of the researcher must be extracted.

Extracting Illustrations• Meta-aggregation requires reviewers to also extract an

illustrative excerpt that the researcher presents in support of that particular finding;

• It is only necessary to extract one such supporting illustration;• The actual verbatim words the researcher uses as the

illustration must be extracted.

Assigning a Level of Plausibility1 to Qualitative Evidence

1 Currently, the term “credibility” is used in QARI; the next version will refer to plausibility

Levels of Plausibility- Qualitative• Unequivocal - relates to evidence beyond reasonable doubt• Plausible - those that are, albeit interpretations, plausible in light of

data and theoretical framework. • Not Supported - when 1 nor 2 apply and when most notably findings

are not supported by the data– Should not be included in synthesis to inform practice

Session 5: Data Analysis and Meta-synthesis

General Analysis - What Can be Reported and How

– What phenomena of interest has been evaluated– The meaningfulness/appropriateness/feasibility of the

experience/activity– Contradictory findings and conflicts– Limitations of study methods– Issues related to study quality– The use of inappropriate definitions– Specific populations excluded from studies– Future research needs

Meta-Synthesis

• Analysis and synthesis of qualitative studies • Based on processed data• Aim of meta-synthesis

– is to assemble findings; – categorize these findings into groups on the basis of similarity in

meaning; – aggregate these to generate a set of statements that adequately

represent that aggregation.

Synthesis of the Results of Qualitative Studies

• Differing methodologies, such as phenomenology, ethnography or grounded theory, can be mixed in a single synthesis of qualitative studies because the synthesis is of findings and not the data.

• This is a critical assumption of meta-aggregation.

As qualitative studies differ from RCT, meta-synthesis utilizes a different approach from that used during meta-analysis.

Both approaches provide an understanding based on populations, settings and circumstances

Unlike meta-analysis, meta-synthesis deals in multiple realities and so provides but one interpretation of the phenomenon.

Meta Synthesis vs Meta Analysis

Executing Meta Synthesis using QARI

Assembling the findings

• Once findings are extracted from all included papers;– The reviewer transitions from a focus on papers to a

focus on assembling all of the findings– This is executed inside JBI-QARI

Creating Categories

• Categorization involves repeated, detailed examination of the assembled findings• The reviewer identifies groups of findings on the basis of similarity in meaning to

create categories– A category “Name” is the element of analysis that appears in the synthesis and

is therefore a full description that conveys the whole, inclusive meaning of the category; that comprehensively represents the meanings embodied in the findings it encompasses

– A Category “Summary” is a brief explanatory statement created by the reviewer to assist interpretation and auditabililty.

Allocating Findings to Categories

Synthesized Findings• In meta-aggregation a synthesized finding is an overarching

description of a group of categorized findings that allow for the generation of recommendations for practice.

Creating Synthesized findings• Meta-aggregation of categories involves repeated, detailed examination of the

assembled categories• The reviewer identifies groups of categories on the basis of similarity in meaning

– A Synthesized Finding “Name” is the element of analysis that conveys the whole, inclusive meaning of the categories; and that comprehensively represents the meanings embodied in the findings it encompasses

– A synthesized finding “Name” must also draw a conclusion or claim and communicate the probability of that conclusion or claim

– A Synthesized Finding “Summary” is a brief explanatory statement created by the reviewer to assist interpretation and auditabililty.

Creating Synthesized Findings– Can be stated propositionally as “if-then” statements

• e.g. “If a patient is awaiting a final diagnosis, their relatives will sometimes feel as if they are not involved”

– Preference for the indicatory form (i.e. suggesting or demonstrating the expedience or advisability of action - emphasizing the probability of the claim)

• “Relatives of patients awaiting a final diagnosis of brain death may feel as if they are not involved if strategies to include them are not pursued”.

Allocating categories to synthesized findings

Session 6: QARI Trial

Day 2

Program Overview

Overview

• This module considers how to appraise, extract and synthesize evidence from qualitative studies and text and opinion papers.

Session 7: The systematic review of text and opinion

Text, Expert Opinion and Discourse– Narrative and text (e.g. review papers, discussion papers and policy

documents) present information that arises out of experience or observation. Much of the resources drawn on to develop policy are narrative and textual in nature rather than the outcomes of formal research

– The opinions of leaders, professional organizations and learned bodies draw on the collective wisdom of experienced experts and are often referred to for guidance in relation to policy making and professional practice

• Expertise is highly regarded and is linked to the ability of an expert to “have to hand” information in a given area and is associated with the possession of large amounts of knowledge and fluency in applying this knowledge

• Discourse refers the verbal interchange of ideas that is grounded in language and in the context within which it occurs. Discourse in the professional and public domains is a source of knowledge that can be used to inform policy and clinical decision making.

Text, Expert Opinion and Discourse as Evidence for Policy and Practice

• Narrative, opinion, expertise and discourse often represent the best available evidence in areas where research is limited, or where the knowledge that is needed is generally generated through policy-making or other processes rather than through formal research

• This kind of knowledge cannot be ignored as legitimate sources of evidence for policy and practice

• While this kind of evidence is not a product of “good” science it is empirically derived and mediated through the cognitive processes of health professionals or policy makers who have typically been trained to be analytical

• The superior quality of evidence derived from research is not denied rather, in its absence, it is not appropriate to discount opinion as non-evidence

The Narrative, Opinion and Text Assessment and Review Instrument (NOTARI)

Critical Appraisal of Evidence arising out of Opinion and Text

Critical Appraisal of Text and Opinion• This focus on limiting bias to establish validity in the

appraisal of quantitative studies is not possible when dealing with text and opinion

• The assessment of validity within the JBI-NOTARI process focuses on:

examining the opinion; identifying the credibility of the source of the opinion; establishing the motives that underlie the opinion; and locating alternative opinions that give credence to it or, conversely,

question it.

Critical Appraisal of Evidence arising out of Text and Opinion

Report Back

Extracting ConclusionsExtracting Conclusions

• The units of extraction in this process are specific

conclusion stated by the author/speaker and the

text that demonstrate the argument or basis of the

conclusion

Meta-SynthesisMeta-Synthesis

• The aim of meta-synthesis is to assemble conclusions; categorise these conclusions into groups on the basis of similarity in meaning; and to aggregate these to generate a set of statements that adequately represent that aggregation. These statements are referred to as synthesised findings - and they can be used as a basis for evidence based practice.

Session 8: NOTARI Trial

Session 9: Protocol Development

Session 10: Protocol Presentations

Session 11: Assessment