January 2003

Judging research quality

24

The Psychologist Vol 16 No 1

How do we recognisegood research?P SYCHOLOGY has evidently

been vexed by qualitative research.Articles in The Psychologist and

elsewhere have passionately arguedwhether it will save the discipline from the perils of positivism, or herald itsbanishment from the community ofsciences into subjectivity and obfuscation(e.g. Gillett, 1995; Morgan, 1996). Ofcourse, qualitative methods have long beenemployed by other disciplines, and are nowbeing quietly embraced by medicine(Macnaughton, 1996; Mays & Pope, 1995).

Nevertheless, the vehemence of thisdebate in psychology is not surprising.After all, this is the science whosepractitioners challenge not only eachother’s results but also their methods.However, to accept the debate as justanother instance of the tensions inherent in psychology would be to miss anopportunity. Rather than becominginvolved in the debate, this article will takea novel look at it and will argue that issuesit has helped expose can clarify ourthinking about how we evaluate research in general – quantitative and qualitative.

‘Methodologism’ in evaluatingresearchDisregarding the lofty epistemologicalpositions that conventionally define thequalitative–quantitative debate, this articleinstead reflects a view that debate is bettergrounded in behaviour: psychologists’judgements of research quality. It emergesfrom my own need to assess the quality ofresearch in the course of learning to writeand review qualitative papers and tosupervise and examine qualitative theses,against a background of quantitativeresearch experience.

Criteria for judging quantitative researchare familiar to psychologists, and can bereliably operated by examiners or refereesanywhere. They can ask, for instance: Weregroups allocated randomly? Was the samplebig enough? Were appropriate statisticsused? Notice that these judgements areabout methods, or procedures. In effect,and without necessarily thinking in these

terms, the examiner or referee is applyingan epistemology of ‘methodologism’;research is good if it has been conductedaccording to certain methods.

There are growing signs ofmethodologism in judgements aboutqualitative research in psychology, too.Articles describing how such researchshould be done (Elliott et al., 1999; Stiles,1993; Turpin et al., 1997) are sometimesused by writers and referees alike aschecklists of methods to be ticked off; forinstance, whether a tape-recorder was used,a field diary kept or an audit trail left. Justas methodologism sustains the quantitativetradition in psychology, it is beginning toshape its qualitative work.

The limits of methodologismUnfortunately, methodologism is a limitedepistemology. It is a forlorn belief thatquality can be guaranteed simply byfollowing procedures. One problem is thatmany things that are called methods arenot. For instance, aspiring qualitativeresearchers sometimes write that they‘used’ triangulation. However, this termrefers to an approach whereby researchersaddress their subject from differentperspectives or with different data – or with different methods. So it is a categorymistake to regard it as a method. Turningprinciples like this into qualitative‘methods’ on a checklist has been criticisedbefore (Barbour, 2001; Chapple & Rogers,1998; Reicher, 2000). Its absurdity is clearwhen researchers write that they ‘used’grounded theory or discourse analysis. A spade or a computer program or astatistical technique can be ‘used’ but anepistemological perspective, a way ofthinking – which is what discourse analysisand grounded theory are – cannot.

It might be objected that ‘using’a technique is just a manner of speech. Butthe words people use are the best guide wehave to what they mean. Moreover, wordsshape, as well as express, what peopleunderstand, so new researchers who followguidance to ‘use’ a technique might be lesslikely to seek to justify the quality of theirwork in a more robust way.

A serious limitation of methodologismis that it asserts rather than justifies theascendancy of methods. When pressed todefend their allegiance to, for instance,controlled experimental design orinterpretative analysis, researchers mightcite the need for objectivity or forembedding meaning in the context. Butthese are themselves essentially methodsand so the justification is circular.Feyerabend (1975, 1978) has pointed outthis inherent circularity: justification for thevalue of scientific methods cannot logicallyemerge from the methods themselves.Natural scientists accepted the methods ofnatural science, not because they couldprove their value, but because scientists andsponsors liked what they saw the methodsdeliver.

When pressed, methodologists mightstate an epistemological position to justifyparticular methods. Indeed, somequalitative researchers argue thatinvestigators should be explicit about theepistemological basis of their work. Thereare two problems here. First,epistemologies describe the use to whichmethods can be put, not which methodscan be used. For instance, qualitativemethods can serve a researcher whobelieves that research discovers underlyingreality as easily as they serve one whobelieves that researchers’ interaction withresearch participants constructs reality.

PETER SALMON discusses anarchism, methodologism

and the quantitative vs. qualitative debate.

Secondly, it is a rare researcher who thinksthrough an epistemological position beforechoosing a method. Such positions aremore often post-hoc rationalisations ofwhat has been done. In reality, researchersuse methods for historical, ideological orpractical reasons – they use the methodsthey have learned to use and that they canuse. A more realistic and robust approachto evaluating research quality is needed,instead of the unholy alliance ofmethodologism and retrospective‘epistemologism’.

An epistemology of qualityAs well as identifying inherent limitationsof methodologism, Feyerabend (1975,1978) described an epistemology, groundedin scientists’ behaviour, that extends toevaluating quality. His ‘anarchist’epistemology states not that rules areunnecessary but that all methodologieshave limits, so the value of a methodologycan be tested only by research that violatesit. Scientific progress has resulted,

therefore, not from allegiance tomethodological rules but from breakingthem. Therefore the anarchist scientistmight choose (or create) any of an infinitearray of methods.

This leaves the question of how thechoice is made in any instance.Feyerabend’s view is that the anarchistscientist is playful rather than precious withmethodology. Methods are chosen simplybecause they interest the researcher orbecause the researcher values their products.For example, when I and colleaguesrecently set out to study paediatric patients’pain, we soon noticed that our informalobservations of nurse–patient interactiontold us much more interesting things aboutpain than the questionnaires that weintended to use ever could. We could notdescribe our observations

using any methodsthat we knew,

but by learning and adapting some ideasfrom qualitative methodology we were ableto report what we thought that it wasimportant to say (Byrne et al., 2001). Gubaand Lincoln (1982) argued that thefundamental axioms of quantitative andqualitative enquiry are arbitrary: whether tobe quantitative or qualitative in any specificstudy should be decided by ‘fit’ with thephenomenon being studied. Guba andLincoln did not, however, follow their ownlogic as far as did Feyerabend to infer thatthis decision is subjective. Feyerabend’sideas provide an epistemological rationalefor views that have emerged recently inpsychology: that ‘good’ research is playful,that research that slavishly followsmethodological rules stultifies thediscipline, and that real scientific progressresults from imagination, creativity andcommon sense, rather than merelydeduction and induction (Rennie, 2000;Robinson, 2000).

Of course, we have to go further thanthis, because what is interesting, fun orvaluable or seems to ‘fit’ for one researchermight leave another bored, uninspired ordissatisfied. Feyerabend’s anarchist viewcannot tell us how to decide what is goodresearch. Instead, it directs us to be honestand explicit about how we already decidethis. It warns us that these judgements areinherently subjective and intuitive andreflect values that are broader than science.Therefore, a realistic debate about the waysin which psychological research should beevaluated will need to transcend parochialissues about, for example, qualitative vs.quantitative methods.

The test of this argument is whetherapplying it to the way we evaluate researchin psychology can expose subjective valuesthat, although normally neglected, doinfluence our judgements and so meritrecognition and debate. The followingsection proposes some values of this kindthat recent debate about qualitative vs.quantitative methods has helped to identify,which apply equally to both kinds ofresearch. Readers might object that thevalues to be described are arbitrary orpartial, and have been neither deducedfrom an epistemological position norinduced from a factor or thematic analysis.That is the point. They illustrate anepistemology that is based on subjectivityand behaviour: What do psychologistsreally look for when evaluating research?

The researcher should not try tomislead Since Rosenthal (1966)demonstrated investigator effects on

January 2003

25

The Psychologist Vol 16 No 1

Judging research quality

Breaking the methodological rules – Feyerabend’s anarchist view of scientific progress

KATE

GRE

Y

research outcomes, the reality that resultsof psychological research reflect theresearcher as much as the researched hasbeen inescapable. The scientist is aparticipant in, rather than observer of,the field of study (Potter, 1996). Somequalitative researchers celebrate this stanceby arguing that researchers’ experience,situation and motivation are integral totheir research.

However, we still seek science ratherthan the biases of politics or rhetoric.Unfortunately, the boundary of science isnot clearly defined, whether by the familiarpractices that claim to ensure objectivityand reproducibility of quantitative research,or by the emphasis in qualitative researchon disclosure of researchers’ situation andperspective. Thus, reproducibility ofquantitative findings is not assured byanalytic techniques (Sohn, 1998) and israrely tested in practice (Potter, 1996),and qualitative researchers’ emphasis onreflexivity can be merely a rhetoricaldevice for persuasive effect (Yardley,1997). Therefore, the boundary betweenscience and other kinds of persuasionremains ill defined (Potter, 1996), eventhough we apply it daily. The consumer of research is therefore left to make a moral as much as technical judgementthat the researcher has not been ‘too’biased.

Methods should be rigorous Likeother disciplines, psychology values‘discipline’ for its own sake. Researchersare expected to follow rules andconventions that make their task difficultand distinguish social science fromjournalism. ‘Sloppy’ research is deprecated.

Quantitative research is easily evaluatedin this respect. There is a well-establishedmethodological and statistical frameworkfor recognising rigour, at the pinnacle ofwhich is the randomised controlled trial(even though this design does notguarantee clinical generalisability, scientificadvance or replicability; see Holmberg etal., 1999; Sohn, 1998).

There are now many sources of similarrules and conventions that offer comparablerigour in qualitative research (Elliott et al.,1999; Mays & Pope, 1995; Turpin et al.,1997), and qualitative researcherssometimes cite strenuous ‘methods’ such as purposive sampling, grounded theory,multiple coding, triangulation andrespondent validation. Regarded purely as methods, however, each of these candegrade rather than enrich research(Barbour, 2001).

Adhering to demanding conventionsdefines what it is to be ‘a discipline’; it isinescapable that research is judged againstsuch conventions. Sometimes, adoptingparticular conventions expresses anepistemological position. However,acceptance that we judge the level of‘discipline’ for its own sake would allowmore realistic debate about the importanceof these conventions. It would also permitmore general acknowledgement thatresearch should be judged in its own terms(Reicher, 2000) and that authors shouldeven choose how their work should bejudged (Devers, 1999).

Analytic work should be done Weexpect to see that a researcher has notmerely recorded and meticulously reported

data, but that these data have beenanalysed. Qualitative researchers havewell-developed approaches to ensuring thatan analysis is developed until it ‘works’ –for example by achieving coherence andorganisation or empowering the reader orparticipants (Stiles, 1993).

Paradoxically, it is in evaluatingquantitative research that the role ofanalysis perhaps needs more debate. In theearly days of quantification, when statisticswere hard work and used preciously, datawere reduced to a few statistics that werethe analysis. Now, complex statisticalprocedures are available at a mouse-clickand researchers can generate more statisticsthan they have data, which imposes newdemands for further, post-statisticalanalysis. Often the problem is ignored, andthe reader drowns in a sea of correlationsor comparisons. Perhaps quantitativeresearchers need to debate taking a leaf outof the qualitative researcher’s book, forexample by seeking ways of organisingresults that indicate coherence or that focusselectively on findings that will ‘empower’readers or participants. But such is the lureof the computer that some qualitativeresearchers also assume that citing aqualitative analysis computer program willreassure the reader that analysis has beendone. In the absence of clear theoreticalrationale and analytic thought, it will nomore achieve this than SPSS will ‘do’a quantitative analysis.

It should be possible to know whenwork is worthless Quantitativeresearchers are well used to stating aims astestable hypotheses. In reality, they areoften not hypotheses, where they are writtenafter the data have been analysed.Nevertheless, aims written like this are notpurely rhetorical. They expose the coherenceof the finished work to scrutiny, exposewhether it has a message, and allow thereader to judge the work; aims that are trivial,unachievable or discordant with the designinvite judgment that the work was futile.

New qualitative researchers aresometimes less helpful in inviting thereader to judge whether their studies havebeen successful or worthless. The aimfrequently stated by students to ‘gain a deeper understanding of’ a phenomenonillustrates one potential problem. The word‘gain’ defines the arbiter of success as theresearcher (rather than reader), and‘deeper’ is an ill-defined metaphor. Bycontrast, aims to ‘describe’, ‘show how’,‘develop some concepts to understand’,‘develop a theory about’ or even ‘convey a more elaborate understanding about’a subject might more readily empower the reader to judge whether they wereachieved. Students and examiners needdebate about how we expect to be able tojudge that qualitative research has failed,just as being open about the charade ofhypotheses would allow more realisticdebate about what we seek fromquantitative research.

The work should matter to others,not just the researcher The argumentin the previous section assumes that, bydefinition, research is not a self-indulgentactivity for the researcher. It has to matterto others. Indeed, because research qualitycannot come from adherence tomethodological rules, it must instead beagreed by an audience – a scientificcommunity (Rennie, 1998). This viewleads to two further considerations forevaluating research.

The first is whether it is clear what theaudience can take from it. Quantitativeresearchers rarely address this issueexplicitly, often sheltering behind the‘generalisability’ of their findings.However, there is a need for greaterhonesty about the products of theirresearch. Even Cronbach (1975), whosealpha-coefficient is widely regarded as thestamp of permanence and generalisabilityon a questionnaire, warned that socialphenomena are too context-bound topermit generalisation. Distinguishing

January 2003

Judging research quality

26

The Psychologist Vol 16 No 1

‘Often the problem isignored, and the reader

drowns in a sea ofcorrelations or comparisons’

scientific from statistical inference(Holmberg et al., 1999) is not the solution.Even if results are used to build and testtheory rather than directly to generaliseimplications for a broader population, thetheory’s generalisability is at issue. Onecomplication in evaluating the message of quantitative research is that it is almostinvariably written up in a theory-testingway which, as noted above, is oftenpretence. Given the exploratory nature ofmuch statistical analysis, and the habitualdisregard of Type 1 errors, perhapsquantitative research should more openlybe judged according to its ability toproduce and develop ideas rather than testthem.

Qualitative researchers are quick todisown generalisability, but are often lessclear about what stands in its place.Accepting that work is ‘exploratory’ doesnot go far enough; it describes themotivation of the researcher, not what thereceiver might take from it. Similarly,reference to ‘transferability’ in qualitativeliterature ducks the question unless it isclear what is being transferred. There is noshortage of possibilities, although they arenot always made explicit. Conceptsdeveloped in a study can equip researchersor practitioners to think or act differently infuture. Findings can help to identify newhypotheses. Findings can simply beincompatible with prevailing assumptions;what transfers is the need to rethink thoseassumptions. The practical orientation ofmedicine might explain why qualitativeresearch has been accepted with so littlecontroversy – clinicians can simply judgefor themselves when a finding is useful(Corbin, 1998; Macnaughton, 1996).Indeed, it has been proposed that inmedical research synthesis the message’simportance should be weighted, not just themethod’s rigour (Edwards et al., 1998).

A second consideration arises fromacknowledging that research is for others.If this is true, work that never reaches itsintended audience – because it is notpublished or presented – or work that oncedisseminated is never read or understood orhas no impact on anything or anyone, couldnot be regarded as research.

It is hardly controversial that the viewsof a scientific audience define what countsas research. It is less readily accepted thatthis recognition is ultimately a social act;that is, what the audience accepts is what it agrees to accept. Acknowledging thismight make explicit some interestingproblems in evaluating research. Forinstance, if dissemination to an audience is

a component of the research process, canstudents’ research be said to be completeand examinable before they havedemonstrated their ability to communicateit effectively, for instance by writing it inpublishable form for a specified audience?

Who counts as a legitimate ‘scientific’audience, and how big does it have to be?

Changing the terms of debatePsychologists’ debate about qualitativeresearch presents a big opportunity. At lastwe are looking at how to evaluate research.The lessons to be learned are not just – oreven mainly – for qualitative researchers.Indeed, the qualitative–quantitativedistinction becomes trivial in the context of the issues raised here. There is an

opportunity to become much more openabout the subjective and intuitive criteriathat we use to judge all research.

Many readers will disagree witharguments presented in this article, and fewwill regard the issues raised as exhaustive.That is to the good, because the aim wasnot to reveal yet another checklist of truthsto prescribe what research should be.Instead the intention was to promote a debate that will expose the subjective and non-scientific values that guide ourevaluation of scientific research. Explicitexamination of the criteria by whichresearch is assessed is central to thestrength of a discipline (Devers, 1999).Psychology needs a debate that is groundedin the untidy and intuitive reality of whatwe do when we evaluate research, ratherthan in the more ordered andoversimplified logic of what we write tojustify what we do.

■ Professor Peter Salmon is in theDepartment of Clinical Psychology at theUniversity of Liverpool. E-mail:psalmon@liv.ac.uk.

January 2003

27

The Psychologist Vol 16 No 1

Judging research quality

Barbour, R. (2001). Checklists forimproving rigour in qualitativeresearch:A case of the tailwagging the dog? British MedicalJournal, 322, 1115–1117.

Byrne,A., Morton, J. & Salmon, P.(2001). Defending againstpatients’ suffering:A qualitativeanalysis of nurses’ responses tochildren’s postoperative pain.Journal of Psychosomatic Research,50, 69–76.

Chapple,A. & Rogers,A. (1998).Explicit guidelines for qualitativeresearch:A step in the rightdirection, a defence of the ‘soft’option, or a form of sociologicalimperialism? Family Practice, 15,556–561.

Corbin, J.M. (1998).Alternativeinterpretations:Valid or not?Theory and Psychology, 8, 121–128.

Cronbach, L. (1975). Beyond the twodisciplines of scientificpsychology. American Psychologist,30, 116–127.

Devers, K.J. (1999). How will weknow ‘good’ qualitative researchwhen we see it? Beginning thedialogue in health servicesresearch. Health Services Research,34, 1153–1188.

Edwards,A.G., Russell, L.T. & Stott,N.C.H. (1998). Signal versus noisein the evidence base formedicine:An alternative tohierarchies of evidence? FamilyPractice, 15, 319–322.

Elliott, R., Fischer, C.T. & Rennie D.L.(1999). Evolving guidelines forpublication of qualitative researchstudies in psychology and relatedfields. British Journal of ClinicalPsychology, 38, 215–229.

Feyerabend, P. (1975). Against method.London:Verso.

Feyerabend, P. (1978). Science in a freesociety. London:Verso/NLB.

Gillett, G. (1995).The philosophicalfoundations of qualitativepsychology. The Psychologist, 8,111–114.

Guba, E.G. & Lincoln,Y.S. (1982).Epistemological andmethodological bases ofnaturalistic enquiry. EducationalCommunication and TechnologyJournal, 30, 233–252.

Holmberg, L., Baum, M. & Adami, H.O.(1999). On the scientificinference from clinical trials.Journal of Evaluation in ClinicalPractice, 5, 157–162.

Macnaughton, R.J. (1996). Numbers,scales, and qualitative research.The Lancet, 347, 1099–1100.

Mays, N. & Pope, C. (1995). Rigourand qualitative research. BritishMedical Journal, 311, 109–112.

Morgan, M. (1996). Qualitativeresearch:A package deal? ThePsychologist, 9, 31–32.

Potter, J. (1996). Representing reality:Discourse, rhetoric and socialconstruction. London: Sage.

Reicher, S. (2000).Against

methodolatry: Some commentson Elliott, Fischer, and Rennie.British Journal of Clinical Psychology,39, 1–6.

Rennie, D.L. (1998). Grounded theorymethodology:The pressing needfor a coherent logic ofjustification. Theory and Psychology,8, 101–119.

Rennie, D.L. (2000). Grounded theorymethodology as methodologicalhermeneutics. Theory andPsychology, 10, 481–502.

Robinson, D.N. (2000). Paradigms and‘the Myth of Framework’: howscience progresses. Theory andPsychology, 10, 39–47.

Rosenthal, R. (1966). Experimentereffects in behavioral research. NewYork:Appleton-Century-Crofts.

Sohn, D. (1998). Statistical significanceand replicability:Why the formerdoes not presage the latter.Theory and Psychology, 8, 291–311.

Stiles,W.B. (1993). Quality control inqualitative research. ClinicalPsychology Review, 13, 593–618.

Turpin, G., Barley,V., Beail, N., Scaife, J.,Slade, P., Smith, J.A. et al. (1997).Standards for research projectsand theses involving qualitativemethods: Suggested guidelines fortrainees and courses. ClinicalPsychology Forum, 108, 3–7.

Yardley, L. (1997). Material discourses ofhealth and illness. Routledge,London.

References

‘Qualitative researchersare quick to disown

generalisability, but are oftenless clear about what stands

in its place’