Comparative Analysis of Ten Assessment Grids of Quality Criteria

in Qualitative Research Marie Santiago-Delefosse, Professor of Health Psychology

Christine Bruchez, FNS Junior Researcher University of Lausanne

Agenda

ü Literature review: the facts ü Scientific relevance of qualitative methods in Health

Sciences ü Facts and questions in Social Health Sciences ü Preliminary research: quality criteria and

comparison of 10 different grids ü Discussion and provisional conclusions ü Our research (2011-2014): work in progress

Literature review: the facts Qualitative methods:

ü Heterogeneous approach consisting of dynamic and contextualised models ü  Cross-disciplinary among traditional areas of Human Sciences

Yet, they share several aspects that allow epistemological and ethical unity to a certain extent: ü  Refusal of « methodolatry » (= methods should be associated with the analysis of

concrete and unique situations) (Bruner, 1991; Chamberlain, 2000); ü  Interest on reflexivity and meaning of human activity (studies on narrative, texts or

visual data collected in « everyday life » situations) ü  They consider the researcher’s involvement as part of the research process ü  Questions the social and political values underpinning research (Psychology as a

science is thus not « value free ») ü  Suggests that the value of research is to be evaluated with respect to its contribution

to knowledge as much as its social and political contribution to a certain culture

Scientific relevance of qualitative methods for Health Sciences

Important development of qualitative methodologies in Social and Human Sciences and in the areas of health (Medicine, Sociology, Anthropology, and Psychology) ü  Increasing requests from medical and nursing studies ü Allows studying complex phenomena conceived in concrete

dynamics ü  Increasing demands in training: breakthrough technological

advances and huge increase in cost of health

Need to reorganize knowledge

ü  Better training for researchers and peer-review experts ü  Better comprehension of the specific traits that regard qualitative

work ü  Better acknowledgement of publishing and diffusing knowledge ü  Better enlightenment of debates related to the establishment of

Research Commissions of Ethics in Health Sciences

Facts and Questions in Social Health Sciences

FACTS ü  Growing visibility and acknowledgement among the scientific world (healthcare services, peer-

reviewed journals) ü  Significant increase of qualitative research and publications since 1980:

-  Abundant literature on procedures and evaluation of qualitative studies -  Symposia, networks, committees and expert groups, and websites -  Financing organisms and experts of scientific journals

SET OF PROBLEMS ü  Expert evaluators and editors encounter difficulties in evaluating qualitative research ü  Difficulty for researchers in publishing qualitative studies (demand for specific evaluation of this

type of research) ü  Difficulties in reaching a consensus on common evaluation criteria that can be applied in health

areas ü  Authors lack of information or are misinformed about these criteria

DEBATES AMONG QUALITATIVE RESEARCHERS

ü  Main consensus suggests remaining realistic about evaluation procedures ü  Suggesting experts the use of lists that may help the evaluation of qualitative research publications

(however, abundance of guidelines and grids)

Preliminary Research (2011) Quality criteria in qualitative research : comparison of 10 existing grids

MATERIAL FOR PRELIMINARY RESEARCH (2011) ü  Choice of 10 different grids among 5 areas of Health (Medicine, Public Health,

Nursing Studies, Psychology/Psychiatry, and Methodology) ü  Grids include guidelines on doing, conducting, and evaluating qualitative research in

Health Sciences. METHODOLOGY and COMPARATIVE ANALYSIS ü  Thematic content analysis of grids based on:

-  number of presented criteria -  different types of proposed criteria -  extent to which their concepts are defined

ü  Comparing existing grids for each Health area (two by two) ü  Showing theoretical backgrounds underpinning the criteria: (common and divergent)

GENERAL HYPOTHESIS ü  The variability of criteria and the differential importance given reflect the qualitative

research approach adopted by the authors of these grids

7

Grids 1-2 : Medicine

Inui & Frankel, 1991, J Gen Intern Med Cohen & Crabtree, 2008, Annals of Family Medicine

l Clear and important research question l Appropriate sampling strategy, data sources, data gathering, and analysis methods l Rich, coherent, salient, strong, and trustworthy inferences and conclusions l Evidence of data archiving and appropriate data management l External audit review, member checking/review l Evidence of investigator self-reflection (role and perceptions of research from inception) l Investigator journal keeping l Seeking out and analyzing deviant or negative cases

l Importance of the research l Use of appropriate & rigorous methods l Clarity and coherence of the research report l Importance of establishing validity or reliability l Importance of verification or reliability l Importance of reflexivity l Carrying out ethical research

Bio-medical approach “Rigor relies mainly on establishing equivalent epistemological rules of objectivity and researcher neutrality

or critical distance from data” (Meyrick, 2006)

G1 G2

8

Grids 3-4 : Public Health

Boulton, Fitzpatrick et al., 1996, Journal of Eval Clin Practice

Seale & Silverman, 1997, European Journal of Public Health

l Introduction Clear aims and appropriateness of a qualitative approach l Sample & generalizability Description of criteria for selection, clear method of recruitment, description of the sample characteristics of sample, adequate final sample l Methods of data collection Adequate description of fieldwork and methods of data collection, systematic and sensitive collection of data, careful recordings and storage of data l Data analysis Adequate description of processes of analysis, providing evidence in support of the analysis, presenting sufficient original material, representative supporting materials, evidence of efforts to establish reliability l Discussion Setting study in a broader context

l Understand context of study by examining relevant historical, cultural & political artifacts l Data collection Record data objectively and comprehensibly (audiotapes, videotapes, different levels of detail in the transcription of data) Ensure representativeness of cases, combination of quanti-quali methods, representative (random) + theoretical sampling l Analysis Support qualitative generalization by counts of events (quasi-statistics); test hypothesis, including consideration of deviant case analysis, analytic induction and grounded theory (support generalization); use of computer programmes, ensuring systematic analysis of representative instances of data

Health services research “Highlights the power of qualitative research to examine ontological differences and generate theory. However,

quality criteria from this sector tend to replicate some of the assumptions underlying quantitative rigour. A variety of approaches reflects the multi-disciplinary make-up of the field; a number of quality frameworks have been presented

alongside each other.” (Meyrick, 2006)

G3 G4

9

Grids 5-6 : Nursing Studies Cesario & Santa-Donato, 2002, J Obst Gyn Neonat Nursing Horsburgh, 2003, J Clin Nurs

l  I: Descriptive Vividness (essential information, clarity, credibility, adequate length of time, validation with participants)

l  II: Methodological Congruence -  Rigor in documentation (phenomenon

identified, philosophical base explicited, questions, aims, assumptions identified, literature, methods, data collection and analysis)

-  Procedural rigor (right questions, steps described, sufficient data, time spent, selection...)

-  Ethical rigor (participants informed of their rights, informed consent, rights protected...)

-  Confirmability (description of data collection process, possibility of other researcher arriving at similar conclusions...)

l  III: Analytical Preciseness (interpretive theoretical statements correspond with findings, propositions dev. during the study verified by data? Conclusions based on the data gathered?...)

l  IV: Theoretical Connectedness (concepts adequately defined/validated by data, conceptual map derived from data, clear connection of data-framework,…)

l  V: Heuristic Relevance (intuitive recognition, relationship to existing body of knowledge, applicability...)

l  Subjective meaning: participants’ accounts as data base, evaluation of reliance of specific sources of data or approaches used in their collection

l  Participant ‘validation’: individual participant’s perception identified and clarified*

l  Description of context : Social context, structures, settings and frameworks

l  Lay knowledge : Participants’ and ‘experts’ perspectives are of equivalent importance

l  Flexibility: Adaptation and redesign in the writing up of research, variability rather than standardization

l  Sampling: Initial sampling decisions should be purposive**

l  Generalizability : Situational rather than statistical***

Nursing studies “Consideration for practical utility of qualitative research emerges through key themes of perception,

transparency, sensitivity to context” (Meyrick, 2006)

G6 G5

10

Grids 7-8 : Psychiatry, psychology

Drapeau, 2004, Pratiques psychologiques Meyrick, 2006, J Health Psychology

l  Internal validity/Internal acceptability/credibility Representative observations of reality, a strong long-term

commitment of researcher with the participants, constant re-evaluation of hypothesis and interpretations, access to a natural environment, triangulation of sources and methods, reflexivity, feedback to participants, plausibility, use of appropriate literature

l  External validity/transferability Generalize to other objects/contexts, saturation, complexity, subtlety

of individual experience, exact description of sample, good description of participants, context, social conditions of the study, definition of used terms and constructs

l  Fidelity/reliability/constancy/internal coherence Cross-checking by other researchers, in-depth description, long-term

commitment in the field, consideration of incidents, consistency of results, feedback from the participants, low level of inference in descriptors, possibility of reproducing the study

l  Objectivity/reliability Presenting complete overview of phenomenon, including the analysis

of the researcher’s influence in its construction l  Deep or wide view In-depth study of the phenomenon, wide enough to include

epiphenomena yet not restricted to pre-established categories, reach a balance between these two tendencies, being flexible and rigorous.

l  Researcher’s epistemological and theoretical stance

Clearly stated, objective and reflexive l  Methods Clear aims and objectives, methods appropriate to research

question l  Sampling Detailed description of sample, representative of themes and

groups; detailed theoretical sampling and description of sampling strategy l  Data collection Transparent, justified change in focus, description of

systematic analytic framework, data collection/analysis responsive to data l  Analysis Transparent pathway data to conclusions; description of

systematic process, complete, examination of all cases, theories + deviance. Triangulation (of method, source, sample, research). Internal validation (reflexivity, qualitative audit trail data to conclusions) and external validation (multiple coding) l  Results and conclusions Findings grounded in data (illustrated). Respondent validation.

Applicability: transferable, sufficient detail to judge, generalizable and representative

Psychology, health psychology Key themes of sensitivity to context, transparency and researcher reflexivity

G7 G8

11

Grids 9-10 : Methodology publications

Leininger, 1994, in Morse, Critical Issues in Qualitative Research Studies

Patton, 1990-99, Qualitative Evaluation and Research Methods

l  Credibility Prolonged observation, engagement

l  Confirmability Direct and often repeated affirmations of what

has been heard, seen, or experienced l  Meaning in context

Data understandable with reference to the context

l  Recurrent patterning instances, sequences of events or behaviors

that tend to be patterned and occur repeatedly or in similar ways over time

l  Saturation Exhaustive exploration of a phenomenon,

apparent redundancy l  Transferability

Findings could be translated to other contexts, situations, and settings, reserving the specific meanings that emerged from the study

l  Data analyzed with attention to issues of reliability, validity and triangulation

l  Researchers credible: training, experience, and track record

l  Analysis creative and methodical l  Sufficient detail reported to allow others

judging quality l  Credibility, rigorous techniques and

methods for gathering high-quality data l  Rigorous, iterative sampling l  Examination of rival explanation l  Bias: analysis of negative or deviant cases l  Triangulation l  Care with regard to transferability

How to Guides “Methodology guides tend to place emphasis on gaining greater understanding of how to carry out qualitative research

enabling informed judgments on quality” (Meyrick, 2006)

G9 G10

12

Discussion

ü  The semantical field of evaluation criteria in qualitative research remains problematic -  Either this field is precise and describes the different steps to be followed in research (but

then mirrors the quantitative approach methodology this is, with a neo-positivist epistemology)

-  Or, this field remains broad and its criteria ambiguous (which still remains to be clarified)

ü  Grids present multiple structures -  Lists of concepts or objectives that are aimed -  Description of steps to follow in a research plan -  « Bizarre blend » of concepts and steps of a research plan

ü  The more the grids remain close to quantitative criteria -  …the more theses describe the steps to follow according to a research plan -  …the more theses focus only on methodology

ü  The more the grids recommend new evaluation categories, the more these include criteria. However, this profusion becomes difficult to manage semantically, since the focus is on criteria closely linked to an epistemological stance, values and a specific theoretical standpoint.

13

Provisional conclusions Reproducing debates between Physical and Natural Sciences and Social and Human Sciences ���

(the “sciences of the mind”) 1. Grids can be classified along the following continuum:

a) An almost quantitative conception of evaluation criteria for qualitative research §  Defined criteria mirror quantitative approaches and are rather poorly or misadapted §  Description of steps in research according to the quantitative model §  Important focus on methodology §  Lack of criteria that refer to theoretical background, analysis, epistemological stance, values

b) Conceptions which consider that evaluation of quality in qualitative research must involve: §  Loosely defined criteria, however, often numerous §  Reference to researchers training and to their positioning in the research process §  Lack or minimal description of stages §  Insistence on values, epistemology, and theoretical analysis

2. This division is to be found inside disciplines

ü  Methodology issues cannot solve existing epistemological and theoretical debates ü  But work is still needed to classify both, grids and criteria, by placing these on the basis of

the authors’ epistemology and theoretical perspectives ü  This explicitation allows determining the relevance of the use, qualities and limitations of the

grids, according to a specific theoretical and epistemological questioning

14

The research (2011-2014) Work in progress (1) Project funded by the “Fonds National de la Recherche Suisse”

“Quality of Qualitative Research in the Health Sciences: which Evaluation Criteria?” Material for research (2011-2014) ü  Collection of 68 grids (currently being updated) ü  Most numerous publications in Medicine and in Public Health (32), followed by Nursing (14),

Methodology guidelines (13), Psychiatry/Psychology (6), Midwife Journals/Health Education(3)

Aim of the research : identify theories/epistemologies underpinning the grids through: ü  Criteria suggested by the authors ü  2 methodological papers per author (identify theoretical foundations) ü  Institutional affiliation of authors, ü  Testing procedure of the grids by an Assessment Panel composed of users (Expert

evaluators of scientific journals, Experts belonging to international research organizations, Users in Ethics Committees, Editorial Directors, etc.

First Findings = Wide variety ü  The elements concerning the analysis of the 10 grids should apply to all grids ü  Number of criteria: between 4 and over 30 (either precise or not) ü  Different typologies: different weight attached to steps in research (methodology,

complementarity of methods, foundings of the research, positioning of the researcher) ü  Little consideration of ethical criteria, often mistaken with “values”

Qualitative research is the best means of explaining a number of singular facts peculiar to human beings and social systems which ‘other methods cannot reach’ (Pope & Mays, 2005)

References British Medical Journal, n° 311, July 1995. URL : http://www.bmj.com/cgi/content/full/311/6996/2 Bruchez, C., Fasseur, F. & Santiago-Delefosse, M. (2004). Visibilité et diffusion de la recherche qualitative en psychologie à partir des sites Internet: une étude préliminaire. Pratiques Psychologiques, 3, 231-241. Castro, D., Blanchet, A., Santiago-Delefosse, M. & Haas, V. (2003). Processus d'expertise scientifique et priorité des experts. Pratiques Psychologiques, 4, 49-63. Castro D. & Santiago-Delefosse M. (2002). Les communications scientifiques en psychologie. Paris: L'esprit du temps. Cesario, S., Morin, K. & Santa-Donato, A. (2002). Evaluating the level of evidence of qualitative research. J Obstet Gynecol Neaonatal Nurs, 31(6), 708-714. Chapple, A., Ziebland, S. & McPherson, A. (2004). Stigma, shame, and blame experienced by patients with lung cancer: a qualitative study. BMJ, 328, 1470. Cohen, D.J., & Crabtree, B.F. (2008). Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med, 6(4), 331-339. Flick, U. (2007). Managing quality in qualitative research. Thousand Oaks: Sage. Fox, B.H. (1995). The role of psychosocial factors in cancer incidence and prognosis. Oncology, 9, 245-253. Grbich, C. (1999). Qualitative Research in Health. An Introduction. London: Sage. Green, J. & Thorogood, N. (2004). Qualitative Methods for Health Research. London: Sage. Guba, E.G. & Lincoln, Y. S. (1981). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches. San Francisco, CA: Jossey-Bass. Horsburgh, D. (2003). Evaluation of qualitative research. J Clin Nurs, 12(2), 307-312. Jeannin, P. & Santiago-Delefosse, M. (2003). Evaluer la recherche à travers les revues de psychologie : enseignements quantitatifs et qualitatifs. In Congrès National, Société Française de Psychologie, Actes du Colloque, Poitiers, 94. Kuper, A., Reeves, S. & Levinson, W. (2008). An introduction to reading and appraising qualitative research. BMJ, 337, 404-407. Leininger, M. (1994). Evaluation criteria and critique of qualitative research studies. In J. M. Morse (Ed.), Critical issues in qualitative research methods. Thousand Oaks, CA: Sage. Mays, N. & Pope, C. (2000). Qualitative research in health care: assessing quality in qualitative research. BMJ, 320, 50-52. Miles, M.B. & Huberman, A.M. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage Publications.

Morse, J.M. & Field, P.A. (1995). Qualitative research methods of health professions (2nd ed). Thousand Oaks, CA: Sage. Murray, M. & Chamberlain, K. (1999). Qualitative Health Psychology. Theories and Methods. London: Sage. Patton, M.Q. (1999). Enhancing the quality and credibility of qualitative research. Health Services Research, 34(5), 1189-1208. Pickler, R.H. (2007). Evaluating qualitative research studies. Journal of Pediatric Health Care, 21, 195-197. Popay,J., Rogers, A. & Williams, G. (1998). Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research, 8(3), 341-351. Pope, C. & Mays, N. (1995). Qualitative Research. Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ, 311, 42-45. Rowan, M., & Huston, P. (1997). Qualitative research articles: information for authors and peer review. Can Med Assoc J, 157(10), 1442-1446 Ryan, F., Coughlan, M. & Cronin, P. (2007). Step-by-step guide to critiquing research. Part 2: qualitative research. British Journal of Nursing, 16(12), 738-744. Sandelowski, M. (1997). To be of use: Enhancing the utility of qualitative research. Nursing Outlook, 45, 125-132. Santiago-Delefosse, M. (2002a). Psychologie de la santé: Perspectives qualitatives et cliniques. Liège: Mardaga Santiago-Delefosse, M. (2002b). Les méthodes de recherche qualitatives et la psychologie de la santé. In G.N. Fischer (Ed.), Traité de psychologie de la santé, (pp. 187-207). Paris: Dunod Santiago-Delefosse, M. (2004). Evaluer la qualité des publications. Quelles spécificités pour la recherche qualitative? Pratiques Psychologiques, 3, 243-254. Santiago-Delefosse, M. (2006). L’activité du chercheur entre objectivisme et subjectivisme. Un révélateur de la tension entre théorie et terrain? In Paillé (Ed.), La méthodologie qualitative, postures de recherche et travail de terrain (pp. 201-223). Paris: Albin Michel. Santiago-Delefosse, M. (2007). Perspectives critiques en Psychologie de la Santé. L’exemple des recherches en Oncologie. Nouvelle Revue de Psychosociologie, 4, 9-21. Santiago-Delefosse, M. & Rouan, G. (Eds.) (2001). Méthodes qualitatives en psychologie. Paris: Dunod Seale, C. & Silverman, D. (1997). Ensuring rigour in qualitative research. European Journal of Public Health, 7, 379-384. Yardley, L. (2000). Dilemmas in qualitative health research. Psychology and Health, 15, 215-228. Yardley, L. (2008). Demonstrating Validity in Qualitative Psychology. In J.A. Smith (Ed), Qualitative Psychology. A Practical Guide to Research Methods (2nd ed). London : Sage.

Criticisms against Qualitative Research

ü  Lack of rigor in the sampling procedure

ü  Lack of fidelity and constancy/consistency in data

ü Not reproducible

ü  Lack of explicitation concerning validation procedures

-  Encouragement to combine qualitative and quantitative findings

ü  Lack of comparison (difficulty to generalize)