ma honey tale of two cultures

8/2/2019 Ma Honey Tale of Two Cultures

1/24

Society for Political Methodology

A Tale of Two Cultures: Contrasting Quantitative and Qualitative ResearchAuthor(s): James Mahoney and Gary GoertzReviewed work(s):Source: Political Analysis, Vol. 14, No. 3, Special Issue on Causal Complexity and Qualitative

Methods (Summer 2006), pp. 227-249Published by: Oxford University Press on behalf of the Society for Political MethodologyStable URL: http://www.jstor.org/stable/25791851 .

Accessed: 11/02/2012 16:13

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

Oxford University Press and Society for Political Methodology are collaborating with JSTOR to digitize,

preserve and extend access to Political Analysis.

http://www.jstor.org
http://www.jstor.org/action/showPublisher?publisherCode=ouphttp://www.jstor.org/action/showPublisher?publisherCode=polmethhttp://www.jstor.org/stable/25791851?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/25791851?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=polmethhttp://www.jstor.org/action/showPublisher?publisherCode=oup


2/24

Advance Access publication June 13, 2006 PoliticalAnalysis 2006) 14:227-249doi:10.1093/pan/mpj017

A Tale of Two Cultures: Contrasting Quantitative andQualitative Research

JamesMahoneyDepartments of Political Science and Sociology,

Northwestern University, Evanston, IL 60208-1006e-mail: [email protected] (corresponding author)

Gary GoertzDepartment of Political Science,University fArizona, Tucson,AZ 85721

e-mail: [email protected]

The quantitative and qualitative research traditions can be thought of as distinct culturesmarked by differentvalues, beliefs, and norms. Inthis essay, we adopt thismetaphor towardthe end of contrasting these research traditions across 10 areas: (1) approaches to explanation, 2)conceptions fcausation, (3)multivariatexplanations,4) quifinality,5) copeand causal generalization,6)case selection, 7)weightingbservations, 8) substantivelyimportant cases, (9) lack of fit, nd (10) concepts and measurement. We suggest that an

appreciation of the alternative assumptions and goals of the traditions can help scholarsavoid misunderstandings and contribute to more productive "cross-cultural" communication inpolitical science.

IntroductionComparisons of the quantitative and qualitative research traditions sometimes call tomind religious metaphors. In his commentary for this issue, for example, Beck (2006)likens the traditionsto theworship of alternativegods. Schrodt (2006), inspiredby Brady's(2004b, 53) prior casting of thecontroversy in termsof theologyversus homiletics, ismoreexplicit: "while this debate is not in any sense about religion, its dynamics are bestunderstood as though itwere about religion.We have always known that, it just neededtobe said."We prefer to thinkof the two traditionsas alternative cultures.Each has itsown values,beliefs, and norms. Each is sometimes privately suspicious or skeptical of the otherthoughusually more publicly polite. Communication across traditions tends tobe difficultand marked bymisunderstanding. When members of one tradition offer their insights toAuthors' note: Both authors contributed equally to this article.Mahoney's work on thisproject is supported by theNational Science Foundation (GrantNo. 0093754). We would like to thankCarles Boix, Bear Braumoeller, DavidCollier, Scott Desposato, Christopher Haid, Simon Hug, Benjamin I. Page, Charles C. Ragin, Dan Slater, DavidWaldner, Lisa Wedeen, and theanonymous reviewers forcomments on earlier drafts.We also thank students at the2006 Arizona State Qualitative Methods Training Institute for feedback on a presentation of much of thismaterial.? The Author 2006. Published byOxfordUniversityPress on behalf of theSociety forPoliticalMethodology.All rightsreserved.For Permissions,please email: [email protected]

227


3/24

228 JamesMahoneyandGaryGoertzmembers of the other community, the advice is likely tobe viewed (rightlyorwrongly) asunhelpful and even belittling.As evidence, consider the reception of Ragin's The Comparative Method: MovingBeyond Qualitative and Quantitative Strategies (1987) and King, Keohane, and Verba'sDesigning Social Inquiry: Scientific Inference inQualitative Research (1994). AlthoughRagin's book was intended tocombine qualitative and quantitativemethods, itwas writtenfrom theperspective of a qualitative researcher, and it became a classic in the field ofqualitative methodology. However, statisticalmethodologists largely ignoredRagin's ideas,and when theydid engage them,theirtonewas oftenquite dismissive (e.g.,Lieberson 1991,1994; Goldthorpe 1997). For itspart, the famous work ofKing, Keohane, and Verba wasexplicitly about qualitative research, but it assumed thatquantitative researchers have thebest tools formaking scientificinferences,and hence qualitative researchers should attemptto emulate these tools to thedegree possible. Qualitative methodologists certainly did notignore thework ofKing, Keohane, andVerba. Instead, theyreactedby scrutinizing thebookin great detail, pouring over each of its claims and sharply criticizingmany of its conclusions (e.g., see theessays inBrady and Collier 2004).In this essay, we tell a tale of these two cultures.We do so from theperspective ofqualitative researcherswho seek to communicate with quantitative researchers. Our goal istocontrast theassumptions and practices of the two traditionstoward theend of enhancingcross-traditioncommunication. Like Brady and Collier (2004), we believe thatqualitativeand quantitative scholars share the overarching goal of producing valid descriptive andcausal inferences.Yet, we also believe that these scholars pursue differentspecific researchgoals, which in turnproduce different orms about researchpractices.Hence, we emphasizehere to a greater degree thanBrady and Collier the ckstinctiveness in basic goals andpractices in thetwo traditions.Having said this,however,we wish to stress that ur intentionisnot to criticize eitherquantitative or qualitative researchers. In fact,we argue throughoutthat the dominant practices ofboth traditionsmake good sense given theirrespective goals.We adopt a criterial approach (Gerring2001) to thinking bout differences between thetwo traditionsand contrast them across 10 areas: (1) approaches to explanation, (2) conceptions of causation, (3)multivariate explanations, (4) equifinality, (5) scope and causalgeneralization, (6) case selection practices, (7) weighting observations, (8) substantivelyimportantcases, (9) lack of fit,and (10) concepts andmeasurement. There are certainlyother differences across the two traditions,1but our experience has been that theseareas are especially important in generating misunderstandings andmiscommunication.Table 1provides a guide to thediscussion thatfollows.Before proceeding, we should note that our discussion presents a stylized view ofboth qualitative and quantitative research. Our characterizations are intended todescribedominant norms and practices. It is easy to find examples of research in one traditioninwhich theanalyst carries out practices thatcharacterize theother tradition.Likewise, aswith all cultures, therewill be some individualswho have fairly strongattachments tobothtraditions.However, we suggest thatmost researchers in political science will locatethemselves predominantly inone column of Table 1.We should also be upfrontthat ur comparison of the two traditionsfocuses centrallyonissues related tocausal analysis.We do not consider qualitative research cultures inpolitical

^ome other potential differences concern level ofmeasurement, type of probabilistic approach, understandingsof time, importance of path dependence, and rationales forbeing concerned about omitted variables. Quantitativeand qualitative methodologies may also have affinities fordistinct theoretical orientations, a topicwhich we donot explore here.


4/24

Tableontrastingualitativenduantitativeesearch

Criterion

Qualitative

Quantitative

Approachesoxplanationxplainndividualases;causes-of-effects"pproach

2 3 4 5 6 7

ConceptionsfausationMultivariatexplanations

Equifinality

Scope andgeneralizationCaseelectionracticesWeightingbservationsSubstantivelymportantases

Lackfit

Conceptsndeasurement

Necessaryndufficientauses;athematicalogicINUSausation;ccasionalndividualffects

Coreoncept;ewausalaths

AdoptarrowcopeovoidausaleterogeneityOrientedowardositiveasesnependentariable;

no0,0,0)ases

Theoryvaluationensitiveondividualbservations;oneisfitanavenmportantmpactSubstantivelymportantasesustexplained

NonconformingasesrexaminedloselyndxplainedConceptsenterfttention;rroreadsooncept

revision

Estimateverageffectfndependentariables;

"effects-of-causes"pproach

Correlationalauses;robability/statisticalheoryAdditiveausation;ccasionalnteractionermsAbsentoncept;mplicitlyargeumberfausalaths

Adoptroadcopeo aximizetatisticaleveragend

generalization

Randomelectionideally)nndependentariables;

allasesnalyzed

Allbservationsrerioriquallymportant;verall

patternfitsrucial

SubstantivelymportantasesotivenpecialttentionNonsystematicausalactorsrereatedsrror

Measurementndndicatorsenterfttention;rrorsmodelednd/orewndicatorsdentified

ro


5/24

230 James Mahoney and Gary Goertzscience?such as descriptive case studies,critical and postmodern theories, nd some strandsof interpretive nalysis?in which causal analysis isnot the leading goal. Consideration ofthese research orientations would necessitate new columns and new criteria inTable 1.1 Approaches toExplanation

A core goal of qualitative research is the explanation of outcomes in individual cases.For example, qualitative researchers attempt to identify the causes ofWorld War I, exceptional growth inEast Asia, theend of theCold War, thecreation of especially generouswelfare states, and therise of neopopulist regimes. A central purpose of research is toidentifythecauses of these specific outcomes foreach and every case thatfallswithin thescope of the theoryunder investigation.By startingwith cases and their outcomes and thenmoving backward toward thecauses, qualitative analysts adopt a "causes-of-effects" approach to explanation. Goodtheoriesmust ideally explain the outcome in all the cases within the population. Forinstance, Skocpol's (1979) famous theory is intended to explain adequately all cases ofsocial revolution among agrarian-bureaucratic states thatwere not formally colonized, theuniverse ofwhich corresponds toFrance, Russia, andChina. The assessment of the theory,in turn, s based primarily on how well it succeeds at this research objective.From the qualitative perspective, this approach to asking and answering questions isconsistentwith normal science as conventionally understood. For example, researchers inthefields of evolutionarybiology and astronomyoften seek to identifythecauses of partic

ular outcomes. Indeed,most natural scientistswould find itodd that theirtheoriescannot beused toexplain individual events.Questions such as "Why did the space shuttleChallengerexplode?" are a request fora cause of an effect.When testifying n front fCongress, RichardFeynman did not think thisquestion tobe nonsensical or nonscientific (Vaughan 1986).In contrast, statistical approaches to explanation usually use theparadigm of the controlled experiment.2With a controlled experiment, one does not know theoutcome untilthe treatmenthas been applied. Indeed, one might say that thewhole point of theexperiment is to observe the effect (if any) of the treatment.Statistical approaches attempt to reproduce theparadigm of the controlled experimentin the context of an observational study.Although there are important and well-knowndifficulties inmoving fromcontrolled experiment toobservational study (e.g., theabsenceof true randomization andmanipulation), forour purposes thecrucial point is that statistical researchers follow the "effects-of-causes" approach employed in experimentalresearch. In particular,with a statistical research design, one seeks toestimate theaverageeffect of one ormore causes across a population of cases. The explanation of specificoutcomes inparticular cases is not a central concern. Hence, quantitative researchersformulatequestions such as "What is theeffectof economic development on democracy?"or "What effect does a given increase in foreign direct investment have on economicgrowth?" They do not normally ask questions such as "Was economic crisis necessaryfor democratization in the Southern Cone of Latin America?" or "Were high levels offoreign investment in combination with softauthoritarianism and export-orientedpoliciessufficientfor the economic miracles in South Korea and Taiwan?"

Methodologists working in the statistical traditionhave seen clearly the differencebetween thecauses-of-effects approach, inwhich theresearch goal is to explain particular2For example, Angrist, Imbens, and Rubin (1996, 144) assert that, "causal inference in statistics, going back atleast towork by Fisher (1981,1925) andNeyman (1923) on agricultural experiments, is fundamentally based onthe randomized experiment (see also Kempthorne 1952 and Cox 1958)."


6/24

A Tale ofTwoCultures 231outcomes, and the effects-of-causes approach, inwhich the research goal is to estimateaverage effects. In general, however, theyhave expressed skepticism about thecauses-ofeffects approach. For example, Holland responded to comments on his article as follows:

Imust disagree with Glymour's paraphrasing ofmy (i.e., Rubin's) analysis, however, andwith thecounterfactual analysis of causation of Lewis described by Glymour. I believe that there is anunbridgeable gulf between Rubin's model and Lewis's analysis. Both wish togive meaning to thephrase "A causes B.n Lewis does thisby interpreting "A causes Bn as "A is a cause ofB? Rubin'smodel interprets "A causes BT as "the effectofA is (Holland 1986b, 970)

King, Keohane, andVerba (1994) followHolland quite closely, and theyexplicitly definecausality in termsof theeffects-of-causes approach.3 They do not consider or discuss thecauses-of-effects approach to explanation.We can see clearly thedifferencesbetween these two approaches to explanation whenwe consider research on a particular topic.For instance, scholars fromeither traditionmaystarttheirresearch with a general question such as "What causes democracy?" To addressthisquestion, however, researcherswill typically translate it intoa new question accordingto the norms of their culture. Thus, qualitative researchers will rephrase the researchquestion as "What causes democracy in one or more particular cases?" Quantitativeresearcherswill translate itdifferently: "What is theaverage causal effect of one ormoreindependent variables on democracy?"The distinctionbetween causes of effectsand effectsof causes arises several times in thesymposium on Brady and Collier (2004) in this special issue. For example, Beck (2006) inhis contribution believes it is essential tobe clear "whether our interest s infinding somegeneral lawlike statements or in explaining a particular event." In the case of Stokes's(2001) and Brady's (2004a) work, he concedes that "the qualitative analysis ishelpful forunderstanding one specific case," but his basic view advocates looking foreffects acrosslargepopulations. Likewise, Shively (2006) suggests thatscholarswho work with a smallnumber of cases "devote their effortspredominately to process-tracing, not to quasistatisticalgeneralization." His view of causation too is from theeffects-of-causes tradition.Much misunderstanding between the two traditionsseems toderive from thesedifferentapproaches to explanation. Quantitative researchersmay have difficultyappreciating theconcern of qualitative researchers with explaining outcomes in particular cases. Forexample, the idea thatSkocpol (1979) would reallywant towrite a whole book that isprimarily an effort to explain the occurrence of social revolution within a scope thatincludes as positive cases only France, Russia, and China may seem puzzling within thestatistical culture. "Real science should seek to generalize about causal effects,"mightbe a typical reaction.Yet, from thequalitative perspective, science can precisely be used inservice of explaining outcomes inparticular cases.We believe thatboth approaches are of value; in fact, they complement one another.Ideally, an explanation of an outcome in one or a small number of cases leads one towonder if the same factors are atwork when a broader understanding of scope is adopted,stimulating a larger-Afnalysis inwhich the goal is less to explain particular cases andmore to estimate average effects.Likewise, when statistical results about the effects ofcauses are reported, it seems natural to ask if these results make sense in termsof thehistory of individual cases; one wishes to tryto locate the effects in specific cases. This3In contrast toHolland, Dawid (2000) does not go so faras toreject thecauses-of-effects approach. Instead, he treatsitas a special case of causation. Interestingly, in his response to a series of comments from several distinguishedstatisticians, he expresses surprise that his analysis of causes of effects provoked so little discussion since hethought itwould be controversial. "I am surprised at how little of the discussion relates tomy suggestions forinference about 'causes of effects,'which I expected tobe themost controversial" (Dawid 2000,446).


7/24

232 James Mahoney and Gary Goertzcomplementarity is one reason why mixed-method research is possible (for recent discussions of mixed-method research strategies, see George and Bennett 2005; Lieberman2005; Coppedge forthcoming).2 Conceptions of CausationIn order to explain outcomes inparticular cases, qualitative researchers often thinkaboutcausation in termsof necessary and/or sufficient auses. The adoption of this understanding of causation can be seen clearly in thekinds of comparative methods employed byqualitative researchers.Mill's methods of difference and agreement, explanatory typologies, andRagin's qualitative comparativemethods are all predicated inoneway or anotheron necessary and/or sufficient causation (see Ragin 1987, 2000; Mahoney 2000; Goertzand Starr 2003; Elman 2005; George and Bennett 2005).

From thequalitative perspective, the assessment of necessary and/or sufficientcausation seems quite natural and fully consistent with logic and good science. For example,classical qualitativemethodologists?such asWeber (1949), Honore andHart (1985), andAron (1986), infactgoing back toDavid Hume?think about causation in individual casesin termsof a necessary condition counterfactual: if >X then->KX is a cause of Y becausewithout X, Y would not have occurred. This approach to causation corresponds to thepreference of most qualitative analysts for expressing their theories using logic andset-theoretic terms.Likewise, as various methodologists point out, thisunderstanding ofcausation is common in historical explanation:

If some eventA is argued tohave been the cause of a particular historical event B, there seems tobeno alternative but to imply that a counterfactual claim is true?if A had not occurred, the event Bwould not have occurred. (Fearon 1996, 40; see also Gallie 1955, 161; Nagel 1961, 581-2)When the scope of theirtheoryencompasses a small ormedium N9 qualitative researchers often adopt the "INUS" approach tocausation (Mackie 1980; Ragin 1987, 2000).4 AnINUS cause is neither individually necessary nor individually sufficientfor an outcome.Instead, it is one cause within a combination of causes thatare jointly sufficient for anoutcome. Thus, with this approach, scholars seek to identify combinations of variablevalues that are sufficient for outcomes of interest. The approach assumes that distinctcombinations may each be sufficient, uch that therearemultiple causal paths to the sameoutcome (this is sometimes called equifinahty; see subsequently). Research findingswith INUS causes can often be formally expressed throughBoolean equations such asY = (AAND B AND C) OR (CAND D AND E).The situation is quite different on the quantitative, statistical side. Here the analysttypically seeks to identifycauses that,on average, affect (e.g., increase or decrease) thevalues on an outcome across a large population. For convenience, we call this the correlational approach to causation. More formally,one can define this approach to causationfora single case in termsof a counterfactual: thedifferencebetween the treatment T) andcontrol (C) for the same unit, i.Using theframework and notation ofKing, Keohane, andVerba (1994), we have for an individual case:

Causal effect= yj - yf; T?treatment, C?control. (1)This equation representswhat King, Keohane, and Verba (1994, 78-9) call the "realizedcausal effect" forunit i (Dawid 2000 calls this the "individual causal effect").Unlike the4An INUS condition is "an insufficientbut nonredundant part of an unnecessary but sufficient [combination ofconditions]" (Mackie 1980, 62).


8/24

ATale ofTwoCultures 233logical and set-theoretic focus of qualitative research, thequantitative approach uses anadditive criterion todefine cause: yj - yf.When thequantitative approach moves from the individual case tomultiple cases, theunderstanding of causal effectas an (unobservable) contrastbetween control and treatmentforan individual observation becomes the causal effect formultiple observations throughthecomparison of groups, inotherwords, overmany units / 1,... ,Af. gain using thebasic notation ofKing, Keohane, and Verba:

Mean causal effect= \iT- \f\ T?treatment, C?control. (2)Instead of they, in equation (1) for an individual case, in equation (2) we have n,whichrepresents themean of the group of cases receiving T or C. Not surprisingly,King,Keohane, and Verba refer to themean causal effect as (3.5 his is variously called the"mean causal effect" (Holland 1986a), "average treatment ffect" (Sobel 2000), "averagecausal response" (Angrist and Imbens 1995), or "average causal effect" (Dawid 2000).Thus, the statistical approach replaces the impossible-to-observe causal effect of T ona specific unitwith thepossible-to-estimate average causal effect of T over a populationof units (Holland 1986a, 947). Hence, it is an easy step toconsider causal effectsas beingthe Ps one estimates in statisticalmodels.Given these differentconceptualizations of causation, there is real potential formisunderstanding and miscommunication. In fact, the kinds of hypotheses developed in thetwo traditions are not always commensurate. For example, considerWaldner's (1999)hypotheses about statebuilding and economic development inTurkey, Syria, Taiwan, andKorea: low levels of elite conflictand a narrow state coalition are both necessary fora developmental state; a developmental state in turn is necessary and sufficientfor sustainedhigh growth. It is not clear how a scholarworking within the statistical frameworkwouldevaluate orunderstand these causal claims. Possibly, shewould translate thehypotheses intoa language that she is familiarwith. Thus, shemight assume thatWaldner hypothesizes that(1) elite conflict and coalitional shallowness are positively associated with thepresence ofa developmental stateand (2) a developmental state ispositively associated with economicdevelopment. ButWaldner does not in fact develop (or necessarily agree with) thesehypotheses; his argument focuses on necessary and sufficient auses, and itcannot be unproblematically translated into the language of correlational causation.The reaction of statistical researchers to the qualitative approach to causation isoften one of profound skepticism. This skepticismmay be grounded in the belief thatthere are no necessary and/or sufficientcauses of social phenomena, that these kinds ofcauses make untenable deterministic assumptions, or that these kinds of causes must bemeasured as dichotomies.6 Statistical researchersmay thereforechoose todismiss out ofhand qualitative hypotheses thatassume necessary/sufficientcausation. Alternatively, assuggestedwith theWaldner example, theymay choose to reinterpretthemas representingimplicit correlational hypotheses.Our view is that it is amistake to reject in toto alternative understandings and definitions of cause. For one thing, thereare in fact differentmathematical models for representing the idea of cause within each tradition.For example, within the statistical tradition,one does nothave todefine causal effects inadditive terms.Rather, asDawid (2000) notes,

5Actually King, Keohane, and Verba (1994) use P to refer to themean causal effect forunit i,which we wouldnotate as (3,.^ot surprisingly, qualitative researchers have responded systematically to these kinds of concerns (e.g., Goertzand Starr 2003; Mahoney 2004). See also below.


9/24

234 James Mahoney and Gary Goertzone could use yj/yf or \og(yf/yf). Also, as Braumoeller (2006) suggests, one couldmodel causal effects as appearing in thevariance rather than themean. In thequalitativetradition,one could think of causation in singular cases in termsof sufficiencywithoutnecessity: "a [covering, scientific] law has theform IF conditions Cl, C2, ..., Cn obtain,THEN always E" (Elster 1999,5) or "everygeneral proposition of the form 'C causes E' isequivalent to a proposition of the form 'whenever C, thenE'" (Ayer 1946, 55). Moregenerally, given that theories regularly posit alternative notions of cause, scholars shouldbe open toworking with differentconceptions of causation. Although to some thismayseem self-evident, the tendency inpolitical science has too often been todismiss certainunderstandings of causation or to use methods thatassume an understanding that is notcongruentwith the theoryunder investigation (see, e.g., Hall 2003).

3 Multivariate ExplanationsIn all causal research, the desire to explain leads to a multivariate focus. In qualitativeresearch, this can be seenwith theassumption that individual events do not have a cause;ratherone must include a variety of casually relevant factors. In quantitative research,of course, one normally assumes that it is impossible to estimate average effectswithoutcontrolling for relevant variables.Yet the typicalmultivariate model of each tradition varies in quite importantways.Take perhaps themost common, modal, model in each tradition:

Equation (3) represents a typical set-theoreticBoolean model based on the INUS approach to causation. In this equation, the * symbol represents the logical AND, the+symbol represents the logical OR, the = symbol indicates sufficiency and implies alogical if-then statement, and lowercase letters indicate thenegation of a variable. Theequation identifies two different combinations of variables that are sufficient for theoutcome. By contrast, equation (4) is a standard statisticalmodel that includes an interaction term.The ways inwhich these two equations are similar and differentare not obvious. Forexample, one might believe that the equations are different in that thequalitative modelnecessarily assumes dichotomous variables, whereas thequantitative one does not.However, equation (3) can be readily estimated with continuously coded variables (Ragin2000).7 Likewise, one might assume that the lack of an error term in the qualitativeequation means that themodel must be tested under deterministic assumptions. In fact,however, themodel could be tested using one of several procedures that have beendeveloped over the last 10 years for analyzing probabilistic necessary and sufficientcauses (e.g., Dion 1998; Braumoeller and Goertz 2000; Ragin 2000; Scott R. Eliasonand Robin Stryker,unpublished manuscript).There are real differences between the two equations. In thequalitative tradition,oneoften focuses primarily on the impact of combinations of variables and only occasionallyfocuses on theeffects of individual variables. Indeed, unless a variable is a necessary cause

F=(A*?*c) + (A*C*D*?) (3)Y = P0 Ml + P2*2 P3*3 + *X2+ B. (4)

7Many works of qualitative analysis at least implicitly employ continuous measurement. For a receding andreanalysis of Skocpol (1979) with continuous fuzzy-set variables, see Goertz andMahoney (2005).


10/24

ATale ofTwoCultures 235or individually sufficientfor an outcome, thequalitative researcherwill usually make noeffort to estimate its net effect.For example, in equation (3) the qualitative researcherwould certainly point out that variable A is necessary for the outcome. But itmakesvirtually no sense to ask, "what is the effect of cause CT Because C sometimes hasa positive effect and sometimes a negative effectdepending on the other variable valueswith which it appears, asking about itsnet effect is not a fruitfulapproach. Likewise,B matters in thepresence ofA and c but in other settings it has no effecton theoutcome.Hence, it isnot useful togeneralize about the overall effect ofB without saying somethingabout the context (i.e., other variable values) inwhich B appears.In thequantitative tradition, y contrast,one ismore likely tobe focused on estimatingthe effect of individual causes, i.e., the individualXt. For example, in the causal modelrepresented by equation (4), one is centrally concerned with estimating thenet effect ofeach individual variable. To be sure, one can include interaction terms in a statisticalmodel (as we have done). Nevertheless, recent articles on themethodology of statisticalinteraction terms (Braumoeller 2003, 2004; Brambor, Clark, and Golder 2006; Clark,Gilligan, andGolder 2006; see also Achen 2005b) illustrate that the individual effect approach continues tobe thenorm in statistics as actually practiced in the social sciences.Typically, when scholars use interactiontermsthey stillask about the individual impact ofX (see Braumoeller 2004 forexamples and critique).8When scholars not familiarwith qualitativemethods see a Boolean model like equation(3), theymay tryto translate it into the familiar termsof interaction effects.This is nota completely unreasonable view (Clark,Gilligan, andGolder's article in this special issuedefends at length this translation), for the logical AND is a first ousin ofmultiplication.However, a good statisticianwould almost never actually estimate equation (3). To estimatethemodel, statisticalpractice suggests that ne should include all lowerorder termssuch asA, AB, AC, andAD. Although thereare very good statistical reasons for thispractice, inBoolean models these reasons do not exist because one isdealing with logic and set theory.In fact, the logicalAND inequation (3) isnot the same asmultiplication inequation (4).Nor is the logical OR in equation (3) the same as addition in equation (4).We believethata failure to recognize these differences contributes to substantial confusion acrossthe two traditions. In particular, it causes quantitative scholars tobelieve thata Booleanmodel is a set of interaction terms that could easily be translated into statistical language(e.g., King, Keohane, and Verba 1994, 87-9; Seawright 2005).One way to illustrate the point is by considering the set-theoretic underpinnings ofnecessary and sufficient auses (seeRagin 2000; Goertz and Starr 2003). With a necessarycause, all cases where theoutcome is present are contained within a largerpopulation ofcases where thenecessary cause is present. Thus, cases inwhich a necessary cause ispresent are a superset, and the Y = 1 cases are a subset of this superset.With a sufficientcause, all cases where the sufficientcause is present are contained within the largerpopulation where theoutcome ispresent.Hence, cases where a sufficient ause ispresentare one subset of a larger superset of Y = 1 cases.This set-theoretic logic ensures that there is a consistent relationship at the supersetand subset levels for findings that are expressed with the logical AND. For instance,8We are also aware that some statistical methodologists have suggested that quantitative practice would beimproved if analysts were to focus on a smaller number of independent variables, exploring carefully theirinteractions, rather than including all possibly relevant independent variables. These same methodologists maysuggest that researchers might profit by focusing on particular subsets of cases rather than the population asa whole. This advice pushes quantitative research in the direction of standard qualitative practice, see, e.g.,Achen (2005a), Clarke (2005), and Ray (2005).


11/24

236 James Mahoney and Gary Goertzsuppose fora population thatwe have a Boolean model such as Y = (A * b * c) + (A *Q.Since A is a necessary cause of Ffor thispopulation, then itmust be a necessary cause forany subset of thepopulation. For a substantive example, take theclassic democratic peacehypothesis: democratic dyads do not fightwars. The hypothesis can be phrased in termsofa necessary cause: nondemocracy (i.e., nondemocratic dyads) is a necessary cause ofwar.Since the set ofwar dyads is a subset of all nondemocratic dyads, thishypothesis willremain trueforany subset ofwar dyads. Likewise, if thecombination A * b * c is sufficientfor theoutcome in thepopulation, then itmust be sufficientfor theoutcome in any subsetof thepopulation. Of course,A* b* cmight not be present in all subsets (e.g., theA *Cone). But thepoint is that ifA * b * c is present in a subset, thenY will also be present.In short,findings thatapply at thepopulation levelmust as amathematical fact also applyto any subset of thepopulation.

The logical approach of qualitative research can be contrasted with the relationshipbetween populations and subsets in statistical research. Imagine that in a statistical studythe impact ofXx is stronglypositive in thepopulation. Does thismean that x cannot havea stronglynegative impact fora particular subset of cases? The answer, of course, is "no."The impact ofX\ as one moves froma superset to subsets is always contingent in statisticalmodels; there is no mathematical reason why X\ could not be negatively related to theoutcome in particular subsets, i.e., the stability of parameter estimates is a contingentphenomenon.9 Similarly, the estimate of theparameter Pi^Xi *X2 could change dramaticallywhen moving from thewhole population toa subset. In short,what is amathematicaltruth nBoolean logic?the consistency of necessary/sufficientcausal relationships acrossa superset and its subsets?is a contingent relationship in the parameter estimates ofstatisticalmodels.The twomodels represented in equations (3) and (4) are thus inmany ways difficulttocompare, which points toreal differences across the traditions.But from theperspectiveof a dialogue between cultures, it isbetter tounderstand the differences than tofightoverwho is rightor better. Indeed, the logic and set theorythat form thebasis of thequalitativeview of cause and causal complexity are not more or less rigorous than the probability andstatisticsused by quantitative scholars.We thereforesee the two approaches as each viablefor social science research.

4 EquifinalityAnother indicator of differences between the qualitative and quantitative traditions isthe importance or lack thereof attributed to the concept of "equifinality" (George andBennett 2005). Also referred to as "multiple, conjunctural causation" or just "multiplecausation," the concept of equifinality is strongly associated with the qualitative comparative analysis approach developed by Ragin (1987), and itplays a key role in howmany qualitative scholars think about causal relationships. In contrast, discussions ofequifinality are absent inquantitativework. If one were to read only large-Af uantitativework, theword "equifinality" (or its synonyms)would notbe part of one's methodologicalvocabulary.

Equifinality is the idea that there remultiple causal paths to thesame outcome. In termsofmultivariate explanations, as we have seen, equifinality is expressed using the INUS

9The assumptions associated with unit homogeneity and unit independence, e.g., Stable Unit Treatment ValueAssumption (see Brady and Seawright 2004 for a nice discussion), are designed to prevent this parameterinstability from occurring. In practice, parameter instability remains a real possibility.


12/24

A Tale ofTwoCultures 237approach. In equation (3), e.g., thereare two causal paths (A *B * c) OR (A *C *D *E)\eitherone is sufficientto attain theoutcome.We thinkthatmuch of thediscussion of equifinality inappropriatelyviews itsdistinctiveaspect as the representation of causal paths throughcombinations of variable values, thefact that causal paths are "conjunctural" innature. Ifone focusesmainly on thiscomponentusing a statisticalperspective, as do King, Keohane, and Verba (1994, 87-9), onemaybelieve thatequifinality is simply a way of talking about interaction terms.What actuallymakes equifinality distinctive inqualitative work is the fact thatthereareonly a few causal paths to a particular outcome. Each path is a specific conjunction offactors, but there are not very many of them.Within the typicallymore limited scopeconditions of qualitative work (see below), the goal is to identify all the causal pathspresent in thepopulation.

In contrast, implicit in statistical models such as equation (4) are thousands, ifnotmillions, of potential paths to a particular outcome. The right-hand side of the statisticalequation essentially represents aweighted sum, and as long as thatweighted sum isgreaterthan the specified threshold?say ina logit setting?then theoutcome should (on average)occur.Within this framework, therewill be countless ways that theweighted sum couldexceed the threshold.One has equifinality in spades.In qualitative research, analysts will normally assign cases to causal paths. Sincetheoverall research goal is toexplain cases, one does so by identifyingthe specific causalpath that ach case follows. For example, Hicks, Misra, andNg (1995) conclude that thereare three separate paths to an earlywelfare state, and theiranalysis allows one to identifyexactlywhich cases followed each of the threepaths (see also Esping-Andersen 1990). Inqualitative research, these causal paths can play a key organizing role forgeneral theoretical knowledge. To cite another example,Moore's (1966) famous work identifiesthreedifferentpaths to themodern world, each defined by a particular combination of variables,and the specific countries that follow each path are clearly identified.10Within quantitative research, it does not seem useful to group cases according tocommon causal configurations on the independent variables. Although one could do this,it isnot a practicewithin the tradition.Again, thegoal of research here isnot toexplain anyparticular case but rathertogeneralize about individual causal effects. In thiscontext,onespeaks about thepopulation as awhole and does not discuss theparticular pathways thatindividual cases follow to arrive at their specific values on thedependent variable.5 Scope and Causal GeneralizationIn qualitative research, it is common for investigators todefine the scope of theirtheoriesnarrowly such thatinferences are generalizable toonly a limited range of cases. Indeed, insome qualitative works, the cases analyzed in the study represent the full scope of thetheory.By contrast, in quantitative research, scholars usually define their scope morebroadly and seek tomake generalizations about large numbers of cases. Quantitativescholars often view thecases they analyze simply as a sample of a potentiallymuch largeruniverse.

The narrower scope adopted in qualitative analysis grows out of the conviction thatcausal heterogeneity is thenorm for large populations (e.g., Ragin 1987,2000). Qualitative10Given that equifinality often organizes causal generalization in qualitative research, it is not surprising thatMackie's (1980) chapter on INUS models is called "causal regularities.'' With an INUS model, each case maybelong to a larger set of cases that follow the same causal path. INUS models thus form a series of theoreticalgeneralizations.


13/24

238 James Mahoney and Gary Goertzresearchers assume thatas thepopulation size increases, evenmodestly, thepotential forkey causal relationships to be missing from ormisspecified in their theories increasesdramatically. These researchers thusbelieve that the addition of each case to the analysisstands a good chance of necessitating substantialmodifications to the theoreticalmodel,even though themodel works perfectlywell for theoriginal cases analyzed. Insofar asthesemodifications produce major complications, qualitative researchers believe that it isbetter todevelop an entirely separate theoryfor the additional cases. For example, Skocpoldevelops separate theories for the great historical social revolutions and for themorecontemporary social revolutions in theThirdWorld (Skocpol 1979; Goodwin and Skocpol1989).As we saw in theprevious section, causal generalization inqualitative research oftentakes the form of specifying a few causal paths thatare each sufficientfor theoutcome ofinterest.Given this approach, expanding the scope of a study can easily risk introducingcausal heterogeneity. Itmight be that thenew cases do not fit the current set of causalpaths. In termsof equation (3), one has two causal paths (A* B* c) OR (A *C *D * ?),and enlarging the scope mightmean that thenew cases require theaddition of a thirdorfourth causal path. It can also arise that thenew cases make existing causal paths problematic, even though theyare sufficientfor theoutcome of interest in the original casesanalyzed. For example, thepath (A * B * c) may not be sufficient for the outcome ofinterest in thenew cases.

Research practices are quite different in the quantitative tradition.Here of courseresearchers need tohave a largenumber of observations tousemost statistical techniques,which may encourage a broad understanding of theory scope. But more importantly,thevery conception of causation used in quantitative researchmeans that the concerns ofcausal heterogeneity are cast indifferentterms. In particular, ifyour goal is toestimate anaverage effectof some variable or variables, theexclusion of certain variables associatedwith new cases is not a problem as long as assumptions of conditional independencestill hold.11 Independent variables that are important for only a small subset of casesmay be appropriately considered "unsystematic" and relegated to theerror term.12 ence,in quantitative research,where adequate explanation does not require getting the explanation right for each case, analysts can omit minor variables to say somethingmoregeneral about thebroader population.One key implication of thisdifference is thatcausal generalizations inqualitative workare more fragile than those in large-Af tatistical analyses. Statistical analyses are oftenrobust andwill not be dramatically influenced bymodest changes in scope or population.But in qualitative research, heterogeneity of various sorts (e.g., concepts, measurement,andmodel) poses amajor problem, which in turnmakes qualitative scholars particularlylikely torestrict thedomain of theirargument.This implication is themirror image ofwhatwe saw in the last section.Whereas findings fromqualitative research tend to be morestable thanfindings fromquantitative research when one moves from a superset toparticular subsets, quantitative findings tend tobemore stable thanqualitative findingswhenonemoves froma subset toa superset.These differences are important, ut theyshould notform thebasis forcriticism of either tradition; theyare simply logical implications of thekinds of explanation pursued in the two traditions.11Of course, some statisticalmethodologists do not believe that these assumptions usually hold outside of naturalexperiments (e.g., Lieberson 1985; Freedman 1991). Yet this concern raises a separate set of issues thatare bestdebated fromwithin the statistical tradition itself.12In this sense, the error termof a typical statistical model may contain a number of variables that qualitativeresearchers regard as crucial causes in individual cases.


14/24

A Tale ofTwoCultures 2396 Case Selection PracticesQualitative researchers usually starttheir research by selecting cases where the outcomeof interestoccurs (these cases are often called "positive" cases). This practice is not surprisingwhen we recall that theirresearch goal is the explanation of particular outcomes.If you want to explain certain outcomes, it is natural to choose cases that exhibit thoseoutcomes. Although sometimes qualitative researchersmay only select positive cases,quite commonly they also choose "negative" cases to test their theories (seeMahoneyand Goertz 2004).In quantitative research, by contrast, researchers generally select cases without regardfor theirvalue on thedependent variable. In fact, forwell-understood reasons, selectingcases based on theirvalue on thedependent variable can bias findings in statistical research(e.g.,Heckman 1976). Quantitative researchers thereforeideally try ochoose populationsof cases throughrandom selection on independent variables.These basic differences tocase selection have stimulated debate across thetwo traditions.In the late 1980s and early 1990s, Achen and Snidal (1989) andGeddes (1991) criticizedqualitative research designs on the subject of selecting on the dependent variable. Thisforeshadowed thewell-known discussion of the issue by King, Keohane, and Verba (1994),which was especially critical of research designs that lack variance on thedependent variable (i.e., no-variance designs). By the late 1990s, a number of scholars responded to thesecriticisms.Regarding no-variance designs, methodologists pointed out that fthehypothesisunder consideration postulates necessary causes, as is common inqualitative research, thedesign is appropriate (e.g.,Dion 1998; Braumoeller andGoertz 2000; Ragin 2000; Harvey2003).13 Likewise, othermethodologists (e.g., Collier, Mahoney, and Seawright 2004)insisted thatwithin-case analysis, which relies on causal-process observations (discussedbelow), provides substantial leverage for causal inferenceevenwhen theN equals 1.Nevertheless, inmany researchdesigns, qualitative scholars include negative outcome cases forthepurposes of causal contrastand inference (e.g., Skocpol also examines sixnegative caseswhere social revolution did not occur in addition toher threepositive cases).To highlight other differences to case selection in the two traditions, an example ishelpful. InTable 2, thereare two independent variables and one dependent variable; allvariables aremeasured dichotomously. In a standard experimental design, one can manipulate cases such that they assume the four possible combinations of values on theindependent variables and then observe theirvalues on thedependent variable. In statistical analysis, the selection of a largenumber of cases without regard for theirvalue on thedependent variable has the effectof approximating this experimental design.In the typical small-Afstudy,however, thereare two characteristics thatare somewhatdistinctive.The first s that thereare usually very few cases of 1on thedependent variable;in termsofTable 2, the tophalf of the table ismuch less populated than thebottom half.This is truebecause thepositive cases of interest (i.e., cases where Y = 1) inqualitativeresearch are generally rare occurrences (e.g.,wars, revolutions, growthmiracles), whereasthenegative cases (e.g., nonwars, nonrevolutions, non-growth miracles) are potentiallyalmost infinite in size. Of course, the same can be true in experimental or statisticalresearchwhen analysts study rare events (e.g., seeGoldstone et al. 2000; King and Zeng

13Although there ismostly consensus on thispoint, Braumoeller and Goertz (2000) argue thatno-variance designsdo not permit one to distinguish trivial from nontrivial necessary causes. For a different view, see Seawright(2002), who argues for the use of "all cases" and not merely those where Y = 1when testing necessarycondition hypotheses.


15/24

240 James Mahoney and Gary GoertzTable 2 Case selection

Yi X27=1 1 10 11 00 0Y = 0 1 10 11 00 0

2001), though as a generalization we can say that the studyof exceptional outcomes ismore common inqualitative research.The second and more importantdistinctive traitof qualitative analysis is that in theheavily populated bottom half, the (0,0,0) cell (inbold type in thetable)where both causesand theoutcome are absent is particularly heavily populated and problematic. In practice,qualitative researchers rarely choose cases (or case studies) from the (0,0,0) cell.A practical reason why is that the (0,0,0) cases are so numerous and ill-defined that it isdifficult to select only a few for intensive analysis, whereas selecting a large number ofthese cases is not a realistic option. By contrast, in a statistical analysis, having a lot ofcases is desirable, and computation of statistical results is not hindered but helped byhaving many cases in each cell.Another problem confronting thequalitative scholar is that the (0,0,0) cases are lessuseful for testingtheorieswhen compared tocases takenfrom theother cells. For example,assume that the causal model being tested inTable 2 isY = Xi AND X2. Negative cases inthe (0,1,1) cell are extremelyuseful because theydisconfirm or at least count against thistheory (i.e., both causes are present, but the outcome is absent); hence, qualitative researchers are highly attuned tofinding these cases. Likewise, negative cases in the (0,1,0)and (0,0,1) cells help qualitative researchers illustrate how Xi andX2 are not individuallysufficientfor theoutcome. But the (0,0,0) cases provide less leverage for causal inference(Braumoeller andGoertz 2000). In fact, inmost of these cases, theoutcome of interest isnot even possible and thus thecases are regarded as irrelevant(Mahoney andGoertz 2004).In short,one will almost never see a qualitative scholar doing a case studyon an observation from the (0,0,0) cell.In contrast, in quantitative research, increasing variance reduces the standard errorand thus is pursued when possible. Within a statistical framework, one would normallywish to include cases distant from 1on the independent variables, such as cases from the(0,0,0) cell. Given a largeN, random selection on the independent variables would bea good way of accomplishing this.In these importantways, the two traditionsdiffer inhow they approach case selectionon both thedependent and independent variable sides of the equation. Yet, we are convinced thatboth traditionshave good reasons fordoing what theydo. Ifyour goal is toestimate average causal effects for large populations of cases, itmakes sense to avoidselecting on the dependent variable. Likewise, itmakes sense to include all types ofnegative cases and treat themas equally importantfordrawing conclusions about causaleffects.But ifyour goal is to explain outcomes inparticular cases, it does notmake senseto select cases without regard for theirvalue on the outcome. Nor does itmake sense


16/24

ATale ofTwoCultures 241to treat all negative cases that lack the outcome of interest as equally relevant to theanalysis.

7 Weighting ObservationsQualitative researchers are in some ways analogous to criminal detectives: they solvepuzzles and explain particular outcomes by drawing on detailed factgathering, experienceworking with similar cases, and knowledge of general causal principles. From the standpoint of this "detective" method (Goldstone 1997; see also Van Evera 1997, chap. 2;McKeown 1999; George and Bennett 2005), not all pieces of evidence count equallyfor building an explanation. Rather, certain observations may be "smoking guns" thatcontribute substantially to a qualitative researcher's view that a theory is valid. By thesame token,much like a detective whose initial hunch about a murder suspect can beundermined by a single new piece of evidence (e.g., an air-tight libi), a new fact can leadqualitative researchers to conclude thata given theory is not correct even though a considerable amount of evidence suggests that it is. For qualitative researchers, a theory isusually only one critical observation away from being falsified.And yet, researcherssometimes build enough evidence to feel quite confident that the theory is valid andthatno falsifyingevidence will ever be found.Also like detectives, qualitative researchers do not view themselves as approachingobservations ina theoreticallyneutralway (Goldstone 2003; Western 2001). Rather, theseresearchers in effect ask: "Givenmy prior theoreticalbeliefs, how much does thisobservation affect these beliefs?" When testing some theories, a single piece of data can radically affect posterior beliefs. The crucial data could show that a key variable wasincorrectlymeasured, and when correctlymeasured, the theoryno longermakes sense.We see thiswith the theorythatheld thatChina performedbetter than India on key socialindicators before 1980 because of itshigher level ofGDP per capita.When researchersintroduced a newmeasure of economic development, which addressed problems with theprevious GDP per capita estimate and showed similar levels of development in the twocountries, thewhole theorywas called intoquestion and rejected (Dreze and Sen 1989).The decisive data need not involve ameasurement problem. For instance, consider thetheory that the combination of a weak bourgeoisie, a divided peasantry, and a powerfullanded elite is sufficientfor fascism in interwarEurope (Moore 1966). This theory ischallenged by theobservations thatpowerful landed elites in the fascist cases either couldnot deliver large numbers of votes or were actually supporters of liberal candidates(Luebbert 1991, 308-9). When one takes this information into consideration, the theoryseems deeply problematic, despite thefact it isplausible inotherways (forotherexamples,seeMcKeown 1999).

By contrast,quantitative scholars generallymake no assumptions that some pieces ofevidence?i.e., particular observations?should countmore heavily than others. Rather,quantitative researchers usually weight a priori all observations equally. They thenwork toestablish a pattern of conforming observations against a null hypothesis.With this approach, a single observation cannot lend decisive supportor criticallyundermine a theory;only a pattern ofmany observations can bolster or call intoquestion a theory.Statisticalresults thatdraw too heavily on a few specific observations (oftenoutliers) are suspect.These differentuses of data correspond to the distinction ofBrady and Collier between"causal-process" and "data-set" observations (2004, 252-5). A data-set observation issimply a row ina standard rectangulardata set and isordinarilywhat statistical researcherscall a case or an observation. Data-set observations provide analytic leverage because they


17/24

242 James Mahoney and Gary Goertzshow or do not show statistically significantpatterns of association between variablesas well as allow for the estimation of the size of effects.By contrast, "A causal-processobservation is an insight or piece of data thatprovides information about context or

mechanism and contributes a differentkind of leverage in causal inference. It does notnecessarily do so as partof a larger, systematized arrayof observations ... a causal-processobservation may be like a 'smoking gun.' Itgives insight into causal mechanisms, insightthat s essential tocausal assessment and is an indispensable alternative and/or supplementto correlation-based causal inference" (Brady and Collier 2004, 252-3). Causal-processobservations are crucial for theory testing in a qualitative setting precisely because onesorts through the data with preexisting theoretical beliefs (including common sense).Like Brady andCollier, we believe thatboth kinds of evidence can be useful.We wouldsimply add that causal-process observations are especially useful when one seeks toexplain specific outcomes inparticular cases, whereas data-set observations are especiallyhelpfulwhen one wishes togeneralize about average causal effects fora large population.Thus, ifyour goal is toexplain particular outcomes, itmakes sense tomove back and forthbetween theory nd thedata; itdoes notmake sense tocarryout a single pass of thedata orto avoid all expost model revisions (though researchersmust still be sensitive to simplyfittinga theory to thedata). By contrast, ifone seeks to estimate average causal effects,one should normally assume a more strict differentiationbetween theoryand data andone should notmove as freelyback and forthbetween theoryand data (though specification searches and other data probes may be consistent with good practice). The upshot isthatquantitative researchers should not primarily seek out causal-process observationsanymore thanqualitative researchers should primarily studydata-set observations.8 Substantively Important CasesQualitative and quantitative scholars have differentperspectives on what constitutes an"important" case. In a typical large-Afnalysis, thereare no ex ante importantcases. Eachcase carries equal weight. Ex post one can and should examine outliers and observationsthat have large leverage on the statistical results. And techniques have long existed foridentifyingand analyzing these kinds of cases (e.g., Bollen and Jackman 1985).In contrast, just as was true for specific pieces of evidence, qualitative scholars donot necessarily treat all cases as equal; some cases are more "important" than others.For example, in thequalitative tradition,researchers explicitly pursue "most likely," "leastlikely," and "critical" case study research designs (Przeworski and Teune 1970; Collier1993; George and Bennett 2005). These kinds of research designs assume that theresearchcommunityhas prior theoreticalknowledge thatmakes certain cases especially interestingand theoretically important.In addition, because qualitative researchers are interested in individual cases, theyare aware of and concerned with cases thatare regarded

assubstantively important.Here"substantively important"means of special normative interestbecause of a past or currentmajor role in domestic or internationalpolitics. For example, qualitative scholars mighthave serious doubts about a theory of American elections that failed miserably forCalifornia and New York even if itworked well for some smaller states. In thefield ofinternational relations, scholars in security studies believe that the ability of realism toexplain the end ofCold War is absolutely crucial. For some social constructivists, in fact,a failure of realism to explain this single case represents amajor strikeagainst thewholeparadigm. Realists seem to agree andwork hard to explain the end ofCold War (there isamassive literatureon thisdebate; see, e.g., theexchange between Brooks andWohlforth


18/24

A Tale of Two Cultures 2432000, 2002, and English 2002). Our view is thatqualitative researchers almost instinctivelyunderstand the requirement of getting the "big" cases rightand worry when it isnot met.

The general point isnicely illustratedwith Goldstone's discussion of theconsequencesforMarxist theoryof a failure toadequately explain theFrench Revolution: "Itmight stillbe that theMarxist view held in other cases, but finding that it did not hold inone of thehistoricallymost importantrevolutions (that is, a revolution in one of the largest,mostinfluential,and most imitated states of the its day and frequent exemplar forMarxisttheories)would certainly shake one's faith in thevalue of the theory" (2003,45-6). Withinthequantitative framework,by contrast, the French Revolution does not count extra forfalsifying theory. fmany other cases conform, thenonconformity of theFrench Revolution is not a special problem (or at least no more of a problem than, say, the BolivianRevolution would be).The qualitative concern with important cases is puzzling for a quantitative scholar.From thisperspective, there is no real reason why substantivelyor historically importantcases are thebest ones when evaluating a theory. t could well be thatan obscure case hasthekey characteristics needed to test a theory. In addition, it is unclear why importantcases should count formore in evaluating theories. If theoretical and empirical scopestatementsare important?which we believe theyare in both qualitative and quantitativeresearch?then it would be better to explain more cases than to evaluate the theoryprimarily against what might be very important,but idiosyncratic, cases.

9 Lack of FitIn qualitative research, the investigator is normally quite familiarwith each case underinvestigation.As a consequence, a particular case that oes not conform to theinvestigator'scausal model is not simply ignored. Instead, the researcher seeks to identifythe specialfactors that ead thiscase tofollow a distinctive causal pattern.These special factorsmay notbe considered part of the central theoreticalmodel, but theyare explicitly identified anddiscussed. The qualitative researcher thereforeseeks tounderstand exactlywhy theparticular case did not conform to the theoretical expectation (Ragin 2004,135-8).By contrast, in quantitative research, the failure of a theoreticalmodel to explainparticular cases is not a problem as long as themodel provides good estimates of parameters for thepopulation as awhole. Many idiosyncratic factorsmay matter forparticularcases, but these factors are not importantformore general theory, nd therefore theyarenot of great concern.14The exclusion of idiosyncratic factors does not bias theparameterestimates of themodel given thatthese factors are oftennot systematically correlatedwith

14Theview of statistical researchers on this issue is nicely captured by theone effort ofKing, Keohane, and Verbatodiscuss causal explanation foran individual case. The authors describe a research project inwhich thegoal isto evaluate the effect of incumbency on elections. King, Keohane, and Verba realize that other "nonsystematic"variables might come into play, but these are relegated to the error termand are of no particular interest:

[W]e have argued that social science always needs to partition the world into systematic andnonsystematic components_To see the importance of thispartitioning, thinkabout what wouldhappen ifwe could rerun the 1998 election campaign in the Fourth District of New York, witha Democratic incumbent and a Republican challenger. A slightly different total would result, dueto nonsystematic features of the election campaign?aspects of politics that do not persist fromone campaign to the next, even if the campaigns begin on identical footing. Some of these nonsystematic featuresmight include a verbal gaffe, a surprisingly popular speech or position on anissue_We can therefore imagine a variable thatwould express thevalues of theDemocratic voteacross hypothetical replications of this same election. (King, Keohane, and Verba 1994, 79)


19/24

244 James Mahoney and Gary Goertzerror termsspecified in themodel. Moreover, the lack of fitof a theoreticalmodel may bedue not simply to omitted variables but also to randomness and nonsystematicmeasurement error?problems which again do not bias results.

These different approaches to dealing with a lack of fitprovide ample ground formisunderstandings. Qualitative researchers believe that prediction error "should beexplained, rather than simply acknowledged" (Ragin 2004, 138). Given thisbelief, theymay be troubledby statisticalmodels thatexplain only a small portion of thevariation ofinterest, eaving therest to theerror term.Theymay ask, "What are thevarious factors thatcomprise the error term?" If the overall fit of the statisticalmodel is not very good, theymay be unconvinced by theargument that theerror termcontains onlyminor variables (ormeasurement error or inherentrandomness). For theirpart, statistical researchersmay beperplexed when qualitative researchers spend a great deal of energy attempting to identifyfactors atwork innonconforming cases. They may wonder, "Why use up valuable timeonresearch that does not lead to generalizable findings?" Indeed, theymay view the effortof fully explaining theoutcome of interestas a deterministic trapor a Utopian goal.Yet, we are convinced thatwhen one appreciates thedifferentresearch goals pursued byqualitative and quantitative analysts, it is hard to condemn either viewpoint. Ifyou reallywant toestimate average causal effects,you should not be in thebusiness of tryingtohuntdown each causal factor thatmight affectoutcomes inparticular cases. But ifyou reallywant to explain outcomes inparticular cases, itmakes good sense tobe in this business.

10 Concepts and MeasurementIt is common in qualitative analysis for scholars to spendmuch time and energy developing clear and precise definitions forconcepts thatare central to theirresearch. They doso because theyare concerned with conceptual validity, and theybelieve that the failure toaddress this concern is a major source ofmeasurement error.When analyzing multiplecases, these researchers especially tryto avoid conceptual stretchingor thepractice ofapplying a concept to cases for which it is not appropriate (Sartori 1970; Collier andMahon 1993). Debates aboutmeasurement validity in this research traditionare thereforeoften focused on the logical structure nd content of specific concepts (see Gerring 2001;Goertz 2006).In quantitative research, by contrast, the focus is less onmeasurement errorderivingfrom the definition and structureof concepts. Instead, this research tradition ismoreconcerned with operationalization and theuse of indicators. For quantitative researchers,measurement errortypicallyoccurs at the level of indicators,not the level of concepts, andmethodological discussions of measurement error therefore concentrate on modelingmeasurement error and modifying indicators with little concern for concept revision.In fact, some (though certainlynot all) quantitative researchers would go so far as to saythata concept is defined by the indicators used tomeasure it,a position thatqualitativeresearcherswould almost never endorse.We can see these differences clearly in comparative research on democracy. In thequalitative research tradition,debates over the (mis)measurement of democracy oftenfocus on the stretchingof this concept to cases that are not really democracies (or arespecial kinds of democracies). Solutions to theproblem are proposed at the conceptuallevel?e.g., developing appropriate subtypes of democracy thatwill simultaneously allowresearchers to capture diverse forms of democracy and avoid stretching the concept(Collier and Levitsky 1997). By contrast, discussions about the (mis)measurement ofdemocracy in quantitative research are concerned with theproperties of indicators and


20/24

A Tale ofTwo Cultures 245the statisticalmeasurement model, including error (e.g., Bollen 1980, 1993; Bollen andPaxton 1998). It is standard in this research tradition tobelieve thatmany measurementproblems result from theuse of poor or biased indicators of democracy.

These differences contribute to skeptical views across the traditions. For example,qualitative researchers sometimes believe that the indicators used in statistical researchare simplisticmeasures thatomit key elements (or include inappropriate elements) of theconcept being studied (Coppedge 1999;Munck and Verkuilen 2002; Bowman, Lehoucq,andMahoney 2005). They may feel that statistical indicators do notmeasure the samethingacross diverse contexts and thus that significantunrecognized conceptual heterogeneity is present inquantitative research.This skepticismultimately emanates from thegoal of qualitative researchers todevelopadequate explanations of each particular case, which means thattheymust trytomeasureall key variables correctly for each case. In the qualitative tradition, in fact, scholarsactively discuss and debate the scoring of particular variables for specific cases. Thestakes of such discussions may be high, for theoryfalsificationmight occur with a changein the value of one or a small number of variables. In qualitative research, in short,measurement errorneeds tobe addressed and eliminated completely, ifpossible. Indicatorsthaton average do a reasonable job ofmeasurement will be problematic because theywill incorrectlymeasure many particular cases.For quantitative researchers, by contrast,measurement error is something that isunavoidable but not devastating so long as it can be adequately modeled. Systematicmeasurement error (i.e., bias) is of course importantand procedures exist to identify it(e.g., Bollen and Paxton 1998). And when systematicmeasurement error is discovered,quantitative researchers will normally seek out better indicators for the concept beingmeasured or betterways tomodel error.But it is stilloftenquite possible togenerate goodestimates of average causal effectswhen nonsystematicmeasurement error is present.Given thesedifferences, it is appropriate to speak of two separate strands in themethodological literatureon measurement error inpolitical science: a qualitative strand thatfocuses on concepts and conceptual validity and that is centrally concerned with eliminatingmeasurement error,and a quantitative strand thatfocuses on indicators andmeasurementvalidity and that seeks tomodel measurement error and avoid systematic error.Both literaturesare hugely influentialwithin theirrespective cultures, but cross-culturalcommunication between the two is relatively rare (though seeAdcock and Collier 2001;Goertz 2006).ConclusionComparing differences inqualitative and quantitative research in contemporary politicalscience entails traversing sensitive ground. Scholars associated with either tradition tendto react defensively and in exaggerated ways to criticisms or perceived mischaracterizations of their assumptions, goals, and practices. The possibilities formisunderstandingaremanifold.

Misunderstanding is enhanced by thefact that the labels "quantitative" and "qualitative" do a poor job capturing the real differences between the traditions.Quantitativeanalysis inherently involves the use of numbers, but all statistical analyses also relyheavily onwords for interpretation. ualitative studies quite frequentlyemploy numericaldata; many qualitative techniques in fact require quantitative information.Although wehave no illusions about changing prevailing terminology,we believe thatbetter labels fordescribing the twokinds of research analyzed

here would be statisticsversus logic, effect


21/24

246 James Mahoney and Gary Goertzestimation versus outcome explanation, or population-oriented versus case-orientedapproaches.This article is not as an effortto advise researchers about how they should carry outwork within their tradition. It is also not an effortto criticize research practices?withinthe assumptions of each tradition, the research practices we have described make goodsense.We thushope that scholars will read this articlewith thegoal of learningmore abouthow the "other side" thinks about research.We especially hope that scholarswill not readthe article with thegoal of notinghow theassumptions of theother side are deeply flawedfrom within theirown culture. Given the differentassumptions and research goals underlying the two traditions, it necessarily follows thatwhat is good advice and goodpractice in statistical researchmight be bad advice and bad practice inqualitative researchand vice versa. In this framework, it isnot helpful to condemn research practices withouttaking into consideration basic research goals.Misunderstandings across the two traditions are not inevitable. Insofar as scholarsare conversant in the language of the other traditionand interested inexploring a peacefuland respectful dialogue, theycan productively communicate with one another.We hopethatour listing of differences across the two traditionsmight contribute to this kind ofproductive communication.

ReferencesAchen, Christopher H. 2005a. Let's put garbage-can regressions and garbage-can probits where they belong.Conflict Management and Peace Science 22:327-39.-. 2005b. Two cheers for Charles Ragin. Studies in Comparative International Development 40:27-32.Achen, Christopher H., and Duncan Snidal. 1989. Rational deterrence theory and comparative case studies.World Politics 41:143-69.Adcock, Robert, and David Collier. 2001. Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review 95:529-46.Angrist, Joshua D, and Guido W. Imbens. 1995. Two-stage least squares estimation of average causal effects in models with variable treatment intensity. Journal of the American Statistical Association 90:431^12.Angrist, Joshua D., Guido W. Imbens, and Donald Rubin. 1996. Identification of causal effects using instrumental variables. Journal of the American Statistical Association 91:444?55.Aron, Raymond. 1986 (1938). Introduction a la philosophie de Vhistoire: essai sur les limites de Vobjectivite

historique. 2nd ed. Paris: Gallimard.Ayer, Alfred Jules. 1946. Language, truth and logic. New York: Dover.Beck, Nathaniel. 2006. Is causal-process observation an oxymoron? Political Analysis 10.1093/pan/mpj015.Bollen, Kenneth A. 1980. Issues in the comparative measurement of political democracy. American SociologicalReview 45:370-90.

-. 1993. Liberal democracy: Validity and method factors in cross-national measures. American Journal ofPolitical Science 37:1207-30.Bollen, Kenneth A., and Robert W. Jackman. 1985. Regression diagnostics: An expository treatment of outliers

and influential cases. Sociological Methods and Research 13:510-42.Bollen, Kenneth A., and Pamela Paxton. 1998. Detection and determinants of bias in subjective measures.American Sociological Review 63:465-78.Bowman, Kirk, Fabrice Lehoucq, and JamesMahoney. 2005. Measuring political democracy: Case expertise,data adequacy, and Central America. Comparative Political Studies 38:939-70.Brady, H. E. 2004a. Data-set observations versus causal-process observations: The 2000 U.S. presidentialelection. In Rethinking social inquiry: Diverse tools, shared standards, ed. H. E. Brady and D. Collier.Lanham, MD: Rowman and Littlefield.-. 2004b. Doing good and doing better: How far does the quantitative template get us? In Rethinkingsocial inquiry:Diverse tools, shared standards, ed. H. E. Brady and D. Collier. Lanham, MD: Rowman andLittlefield.


22/24

ATale ofTwoCultures 247Brady, H. E., and David Collier, eds. 2004. Rethinking social inquiry:Diverse tools, shared standards. Lanham,MD: Rowman and Littlefield.Brady, H. E., and Jason Seawright. 2004. Framing social inquiry: From models of causation to statistically basedcausal inference. Paper presented at the annual meetings of the American Political Science Association.Brambor, T.,W. Clark, andM. Golder. 2006. Understanding interaction models: Improving empirical analyses.Political Analysis 14:63-82.Braumoeller, Bear F. 2003. Causal complexity and the study of politics. Political Analysis 11:209-33.-. 2004. Hypothesis testing and multiplicative interaction terms. International Organization 58:807-20.-. 2006. Explaining variance: Exploring the neglected second moment. Political Analysis 10.1093/pan/

mpj009.Braumoeller, Bear R, and Gary Goertz. 2000. The methodology of necessary conditions. American Journal ofPolitical Science 44:844-58.Brooks, Stephen G., and William C. Wohlforth. 2000. Power, globalization, and the end of the Cold War:

Reevaluating a landmark case for ideas. International Security 25:5-53.-. 2002. From old thinking tonew

thinking in qualitative research. International Security 26:93-111.Clark, W. R., M. J.Gilligan, and M. Golder. 2006. A simple multivariate test for asymmetric hypotheses.Political Analysis 10.1093/pan/mpj018.Clarke, Kevin A. 2005. The phantom menace: Omitted variable bias in econometric research. Conflict Man

agement and Peace Science 22:341-52.Collier, David. 1993. The comparative method. InPolitical science: The state of thediscipline II, ed. A. Finifter.

Washington, DC: American Political Science Association.Collier, David, and Steven Levitsky. 1997. Democracy with adjectives: Conceptual innovation in comparativeresearch. World Politics 49:430-51.Collier, David, and James E. Mahon, Jr. 1993. Conceptual stretching revisited: Adapting categories in comparative analysis. American Political Science Review 87:845-55.Collier, David, JamesMahoney, and Jason Seawright. 2004. Claiming toomuch: Warnings about selection bias.InRethinking social inquiry:Diverse tools, shared standards, ed. H. E. Brady and D. Collier. Lanham, MD:Rowman and Littlefield.Coppedge, Michael. 1999. Thickening thinconcepts and theories: Combining large-N and small in comparative

politics. Comparative Politics 31:465-76.Dawid, P. 2000. Causal inferencewithout counterfactuals (with discussion). Journal of theAmerican StatisticalAssociation 95:407-50.Dion, Douglas. 1998. Evidence and inference in the comparative case study.Comparative Politics 30:127-45.Dreze, Jean, and Amartya Sen. 1989. Hunger and public action. Oxford: Clarendon Press.Elman, Colin. 2005. Explanatory typologies in qualitative studies of international politics. International Orga

nization 59:293-326.Elster, J. 1999. Alchemies of the mind: rationality and the emotions. Cambridge: Cambridge University Press.English, Robert D. 2002. Power, ideas, and new evidence on the Cold Wars end: A reply to Brooks andWohlforth. International Security 26:70-92.Esping-Andersen, G0sta. 1990. The threeworlds of welfare capitalism. Cambridge: Polity Press.Fearon, James D. 1996. Causes and counterfactuals in social science: Exploring an analogy between cellularautomata and historical processes. InCounter/actual thought experiments inworld politics, ed. P. Tetlock andA. Belkin. Princeton, NJ: Princeton University Press.Freedman, David A. 1991. Statistical models and shoe leather. In Sociological methodology, ed. P. Marsden.San Francisco: Jossey-Bass.Gallie, W. 1955. Explanations in history and the genetic sciences. Mind 64:160-80.Geddes, Barbara. 1991. How the cases you choose affect the answers you get: Selection bias in comparative

politics. In Political Analysis, ed. J.A. Stimson. Vol. 2. Ann Arbor: University ofMichigan Press.George, Alexander L., and Andrew Bennett. 2005. Case studies and theory development in the social sciences.

Cambridge, MA: MIT Press.Gerring, John. 2001. Social science methodology: A criterialframework. Cambridge: Cambridge University Press.Goertz, Gary. 2006. Social science concepts: A user's guide. Princeton, NJ: Princeton University Press.Goertz, Gary, and JamesMahoney. 2005. Two-level theories and fuzzy-set analysis. Sociological Methods andResearch 33:497-538.Goertz, Gary, and Harvey Starr, eds. 2003. Necessary conditions: Theory, methodology, and applications.

Lanham, MD: Rowman and Littlefield.Goldstone, J.A. 1997. Methodological issues in comparative macrosociology. Comparative Social Research16:121-32.


23/24

248 JamesMahoneyandGaryGoertz-. 2003. Comparative historical analysis and knowledge accumulation in the study of revolutions. In

Comparative historical analysis in the social sciences, ed. J.Mahoney and D. Rueschemeyer. Cambridge:Cambridge University Press.

Goldstone, J.A., T. R. Gurr, B. Harff,M. A. Levy, M. G. Marshall, R. H. Bates, D. L. Epstein, C. H. Kahl, P. T.Surko, J.C. Ulfelder, and A. N. Unger. 2000. State failure task report: Phase III findings. McLean, VA:Science Applications International Corporation.

Goldthorpe, J. 1997. Current issues in comparative macrosociology: A debate on methodological issues.Comparative Social Research 16:1-26.

Goodwin, J., and Theda Skocpol. 1989. Explaining revolutions in the contemporary Third World. Politics andSociety 17:489-509.

Hall, Peter A. 2003. Aligning ontology and methodology in comparative research. In Comparative historicalanalysis in the social sciences, ed. J.Mahoney and D. Rueschemeyer. Cambridge: Cambridge UniversityPress.

Harvey, Frank P. 2003. Practicing coercion: Revisiting successes and failures using Boolean logic and comparative methods. InNecessary conditions: Theory, methodology, and applications, ed. G. Goertz and H. StartNew York: Rowman and Littlefield.Heckman, James J. 1976. The common structureof statisticalmodels of truncation, sample selection and limited

dependent variables and a simple estimator for such models. Annals of Economic and Social Measurement5:475-92.Hicks, Alexander M., JoyaMisra, and T. N. Ng. 1995. The programmatic emergence of the social security state.American Sociological Review 60:329-50.Holland, Paul W. 1986a. Statistics and causal inference. Journal of theAmerican Statistical Association 81:945-60.

-. 1986b. Statistics and causal inference: Rejoinder. Journal of the American Statistical Association81:968-70.Honore, Tony, and Herbert Lionel Adolphus Hart. 1985. Causation in the law. 2nd ed. Oxford: Oxford UniversityPress.King, Gary, Robert O. Keohane, and Sidney Verba. 1994. Designing social inquiry: Scientific inference in

qualitative research. Princeton, NJ: Princeton University Press.King, Gary, and Langche Zeng. 2001. Logistic regression in rare events data. Political Analysis 9:137-63.Lieberman, Evan S. 2005. Nested analysis as a mixed-method strategy for comparative research. AmericanPolitical Science Review 99:435-52.Lieberson, Stanley. 1985.Making itcount: The improvement of social research and theory.Berkeley: Universityof California Press.-. 1991. Small Ns and big conclusions: An examination of the reasoning in comparative studies based ona small number of cases. Social Forces 70:307-20.-. 1994. More on the uneasy case for using Mill-type methods in small-N comparative studies. SocialForces 72:1225-37.Luebbert, Gregory M. 1991. Liberalism, fascism, or social democracy: Social classes and thepolitical origins of

regimes in interwar Europe. New York: Oxford University Press.Mackie, John Leslie. 1980. The cement of the universe: A study of causation. Oxford: Oxford University Press.Mahoney, James. 2000. Strategies of causal inference in small-N research. Sociological Methods and Research28:387-424.

-. 2004. Comparative-historical methodology. Annual Review of Sociology 30:81-101.Mahoney, James, and Gary Goertz. 2004. The possibility principle: Choosing negative cases in comparativeresearch. American Political Science Review 98:653-69.McKeown, Timothy J. 1999. Case studies and the statistical worldview: Review ofKing, Keohane, and Verba's

Designing Social Inquiry. International Organization 53:161-90.Moore, Barrington 1966. The social origins of dictatorship and democracy: Lord and peasant in themaking ofthemodern world. Boston: Beacon Press.Munck, Gerardo L., and JayVerkuilen. 2002. Conceptualizing andmeasuring democracy: Evaluating alternativeindices. 35:5-34.Nagel, Ernest. 1961. The structure of science: Problems in the logic of scientific explanation. New York:

Harcourt, Brace and World.Przeworski, Adam, and Henry Teune. 1970. The logic of comparative social inquiry.New York: JohnWiley andSons.Ragin, Charles C. 1987. The comparative method: Moving beyond qualitative and quantitative strategies.

Berkeley: University of California Press.


24/24

A Tale ofTwoCultures 249-. 2000. Fuzzy-set social science. Chicago: University of Chicago Press.-. 2004. Turning the tables: How case-oriented research challenges variable-oriented research. In Re

thinking social inquiry: Diverse tools, shared standards, ed. H. E. Brady and D. Collier. Lanham, MD:Rowman and Littlefield.Ray, James Lee. 2005. Constructing multivariate analyses (of dangerous dyads). Conflict Management andPeace Science 22:277-92.Sartori, Giovanni. 1970. Concept misformation in comparative politics. American Political Science Review64:1033-53.Schrodt, Philip A. 2006. Beyond the linear frequentist orthodoxy. Political Analysis 10.1093/pan/mpj013.Seawright, Jason. 2002. Testing for necessary and/or sufficient causation: Which cases are relevant? (with

discussion) Political Analysis 10:178-93.-. 2005. Qualitative comparative analysis vis-a-vis regression. Studies in Comparative InternationalDevelopment 40:3-26.

Shively,W. Phillips. 2006. Case selection: Insights from Rethinking Social Inquiry. Political Analysis 10.1093/pan/mpj007.

Skocpol, Theda. 1979. States and social revolutions: A comparative analysis of France, Russia, and China.Cambridge: Cambridge University Press.

Sobel, Michael E. 2000. Causal inference in the social sciences. Journal of theAmerican Statistical Association95:647-51.Stokes, Susan C. 2001. Mandates and democracy: Neoliberalism by surprise in Latin America. Cambridge:

Cambridge University Press.Van Evera, Stephen. 1997. Guide tomethods for students of political science. Ithaca, NY: Cornell UniversityPress.Vaughan, Diane. 1986. The Challenger launch decision. Chicago: University of Chicago Press.Waldner, David. 1999. State building and late development. Ithaca, NY: Cornell University Press.Weber, Max. 1949. Objective possibility and adequate causation in historical explanation. The methodology ofthe social sciences. New York: Free Press.Western, Bruce. 2001. Bayesian thinking about macrosociology. American Journal of Sociology 107:352-78.

ma honey tale of two cultures

Documents