evidentiary pluralism as a strategy for research and evidence-based practice in rehabilitation...

Evidentiary Pluralism as a Strategy for Research and Evidence-BasedPractice in Rehabilitation Psychology

Jalie A. TuckerUniversity of Alabama at Birmingham

Geoffrey M. ReedInternational Union of Psychological Science

Objective: This article examines the utility of evidentiary pluralism, a research strategy that selectsmethods in service of content questions, in the context of rehabilitation psychology. Hierarchical viewsthat favor randomized controlled clinical trials (RCTs) over other evidence are discussed, and RCTs areconsidered as they intersect with issues in the field. RCTs are vital for establishing treatment efficacy, butwhether they are uniformly the best evidence to inform practice is critically evaluated. Conclusions: Theauthors argue that because treatment is only one of several variables that influence functioning, disability,and participation over time, an expanded set of conceptual and data analytic approaches should beselected in an informed way to support an expanded research agenda in which therapeutic andextratherapeutic influences on rehabilitation processes and outcomes is investigated. The benefits ofevidentiary pluralism are considered, including those that help close the gap between the narrowerclinical rehabilitation model and a public health disability model.

Keywords: evidence-based practice, evidentiary pluralism, rehabilitation psychology, randomized con-trolled trials

Over the past two decades, medicine followed by public healthand the mental health professions have embraced the evidence-based practice (EBP) movement, which seeks to bring health carepractices and policies in line with the best scientific knowledge(Brownson, Baker, Leet, & Gillespie, 2003; Goodheart, Kazdin, &Sternberg, 2006; Institute of Medicine [IOM], 2001, 2005;Jenicek, 2003; Norcross, Beutler, & Levant, 2006; Victoria,Habicht, & Bryce, 2004; Westen, Novotny, & Thompson-Brenner,2004). Rising health care costs and the growth of managed care inthe United States in the late 20th century energized the early EBPmovement, as did the rapid growth of the scientific knowledgebase. The EBP movement is now an international movement inwhich “evidence” serves as a guiding force in health care practiceand policy. Although the concept of evidence remains at the heartof the movement, the nature and scope of the relevant evidencecontinue to evolve as EBP has expanded from its origins in acute

medical care and drug therapies to encompass care for chronichealth conditions and mental and substance use disorders. Thepresent article is about that evolution and how it has affectedrehabilitation psychology, sometimes in adverse ways, and why abroadened conception of evidence and the methods used to obtainit may be necessary for advancing the field.

The early EBP movement emphasized the restrictive use oftreatments that were “empirically validated” by a minimum of tworandomized controlled clinical trials (RCTs) and used the findingsas the basis for guidelines and manuals for best practices (Agencyfor Health Care Policy and Research, 1993; Chambless et al.,1996; Sackett, Strauss, Richardson, Rosenberg, & Haynes, 2000).RCTs came to be viewed as the “gold standard” of evaluationresearch and as superior evidence compared with nonrandomizedobservational studies of treatment effects and other forms of clin-ical research (Sacks, Chalmers, & Smith, 1982; Westen et al.,2004). Table 1 shows the well-known “evidence hierarchy” (e.g.,Jenicek, 2003, p. 34) that resulted from this view of researchquality as it informs practice. Strength of evidence is ranked fromhigh to low (Levels I–V) on the basis of the presumed capacity tosupport causal inferences. Experimental research, as epitomized bythe RCT, is at the top. Later elaborations placed systematic re-views and meta-analyses of RCTs above individual RCTs in thehierarchy (Public Health Information & Data Tutorial, 2006). Theemphasis on RCTs contrasts with the broader view of evidencecommon in public health practice, which values observational andquasi-experimental studies in addition to RCTs (Brownson et al.,2003).

The public health perspective notwithstanding, experimentalmethods have been increasingly applied to health-related researchquestions, often without much regard for whether the methodssuited the questions at hand. The scope of acceptable methodolo-gies has narrowed, and many scientific journals now require treat-ment-related research to be reported in line with the “Consolidated

Jalie A. Tucker, Department of Health Behavior, University of Alabamaat Birmingham School of Public Health; Geoffrey M. Reed, InternationalUnion of Psychological Science, Madrid, Spain.

Geoffrey M. Reed is now at the Department of Mental Health andSubstance Abuse, World Health Organization, Geneva, Switzerland.

Preparation of this manuscript was supported in part by U. S. NationalInstitute on Alcohol Abuse and Alcoholism Grant R01 AA08972 and U. S.National Institute on Drug Abuse Grant R21 DA R21-DA021524. We aregrateful to the following colleagues for taking the time to communicatewith us about the ideas in this article as they apply to rehabilitationpsychology (though they should be held blameless as to its content):Susanne Bruyere, John Corrigan, David Keer, Don Lollar, David Roth,Marcia Scherer, Susan Stark, and Edward Taub.

Correspondence concerning this article should be addressed to Jalie A.Tucker, Department of Health Behavior, School of Public Health, Univer-sity of Alabama at Birmingham, 1665 University Boulevard, 227 RPHB,Birmingham, AL 35294. E-mail: [email protected]

Rehabilitation Psychology2008, Vol. 53, No. 3, 279–293

Copyright 2008 by the American Psychological Association0090-5550/08/$12.00 DOI: 10.1037/a0012963

279

Standards of Reporting Trials,” or CONSORT, statement (Moher,Schulz, & Altman, 2001), which includes a checklist in whichrandomization and other typical RCT features figure prominently.The original CONSORT statement was influenced heavily byRCTs for pharmacological treatments and has recently been elab-orated to accommodate additional features common to nonphar-macological (e.g., behavioral) treatments (Boutron, Moher, Alt-man, Schulz, & Ravaud, 2008). Although these methodologicaldirectives have helped advance the knowledge base for practice,important research questions that are not amenable to experimen-tation have been underresourced or gone unaddressed. Disciplinesand fields of inquiry with content questions that are not well suitedto experimentation using RCTs are at risk for marginalization,including rehabilitation psychology.

Reed and Eisman (2006) argued that EBP can best be under-stood in the context of contemporary patterns of organized healthcare delivery. EBP is a “public idea,” meaning an idea that bothdescribes a public problem—in this case, the undeniable failings inthe U.S. health care system—and suggests the wisdom of a par-ticular response—that health professionals should practice in waysthat are more consistent with research findings (Tanenbaum,2005). EBP has been used to justify the lay management ofprofessional behavior, which is the central operating principle ofmanaged care, in ways that reduce health care costs, typicallythrough strategies such as limiting access to services and increas-ing out-of-pocket consumer costs. Although championed in somequarters (Carpinello, Rosenberg, Stone, Schwager, & Felton, 2002;Chorpita et al., 2002; see Tanenbaum, 2005), there is a dearth ofevidence that use of EBP principles by managed care or otherhealth systems results in better and cheaper care (Reed & Eisman,2006; Westen et al., 2004). There also has been remarkably littlechallenge to the idea that serving managed care in this way shouldbe a core aspect of the scientific agenda.

Historically, rehabilitation psychology has been allied with andinfluenced by medical rehabilitation. Both fields have been af-fected by the growing dominance of EBP and the emphasis itplaces on RCTs as the most desirable form of evidence. Fuhrer(2003), a former director of the National Center for MedicalRehabilitation Research (NCMRR), characterized this state ofaffairs as reflecting “the zealotry of some advocates . . . who wouldonly admit evidence from RCTs in deciding about a treatment’seffectiveness or who, as reviewers of funding applications, insistthat only the worthy outcomes research design is the RCT” (p.

S11). As a lower status field among medical specialties, rehabili-tation medicine has been vulnerable to the charge that its scientificfoundation is weak due to a relative paucity of RCTs.

Increasingly, arguments have been advanced for a shift from asharply hierarchical or vertical view of evidence to a broaderhorizontal conception that recognizes the limitations of RCTs foraddressing some research questions that are central to promotingbest practices and policies (e.g., Brownson et al., 2003; Concato,Shah, & Horwitz, 2000; Horn, DeJong, Ryser, Veazie, & Teraoka,2005; Leichsenring, 2004; Reed & Eisman, 2006; Tucker & Roth,2006). These arguments endorse the core assumption of EBP thatevidence can improve health care practice and policy but havecritically considered what constitutes evidence and how to evalu-ate its quality and applicability. Tucker and Roth (2006), forexample, argued against using fixed “rules of evidence”—such asdesign, statistical, and effect size requirements—in a content-freefashion. Such features matter in evaluating research quality, but,

can limit flexibility, creativity, and diversity if rigidly applied and thuscan work at cross-purposes with knowledge development . . . . The valueof different forms of evidence derives more fundamentally from thedevelopmental state of theory, research, and practice in a given field. (p.919)

In this article, we take up these issues as they pertain to reha-bilitation psychology and advance an argument for evidentiarypluralism, an approach that entails a more balanced and diverseevidentiary base for practice compared with conventional hierar-chies that favor RCTs over all other forms of evidence. The termwas introduced by Tucker and Roth (2006) and refers to selectingmethodologies on the basis of the needs of a given researchquestion, rather than posing questions within the constraints ofmethodology. The approach should help advance an interdiscipli-nary approach to science and practice in rehabilitation psychologyand builds on similar analyses in other medical areas (Concato etal., 2000), behavioral medicine (Spring et al., 2005), psychophar-macology (Blacker & Mortimore, 1996), mental health (Leichsen-ring, 2004; Reed & Eisman, 2006; Westen et al., 2004), substanceabuse (Tucker, 1999; Tucker & Roth, 2006), public health (Vic-toria et al., 2004), health promotion (World Health Organization[WHO], 1998), and consumer behavior (Seligman, 1995) fields.

There is no doubt that RCTs are invaluable for addressingquestions about treatment efficacy. However, the design has oftenunrecognized limitations that can make it suboptimal for studyingother important issues that are intrinsic to psychosocial treatmentsand behavior change processes and are important for improvingthe reach and effectiveness of practice. Some questions are betteraddressed—and sometimes can only be addressed—with nonex-perimental designs or require a combination of the two approaches(Brownson et al., 2003). In the present article, we challengereaders to evaluate critically and select different research strategiesand tactics on the basis of their strengths and weaknesses inrelation to specific theoretical or empirical questions.

The article is organized as follows: We first describe the history,advantages, and limitations of the RCT and then consider howthese issues intersect with rehabilitation psychology, includingshared concerns with other areas and concerns specific to thefield’s content focus and knowledge development. We illustrateconceptual issues and research questions in rehabilitation psychol-ogy that cannot be addressed satisfactorily with RCTs and require

Table 1Hierarchical classification of evidenceinforming treatment efficacy

Ranking Evidence

I. Randomized controlled clinical trials.II. Well-designed trials without randomization.

Nonrandomized trials or those with highalpha and beta errors.

III. Analytical observational studies.IV. Multiple time series or place comparisons,

uncontrolled (“natural”) experiments.V. Expert opinions, descriptive occurrence

studies, case reports, case series reports.

Note. From Jenicek (2003, p. 34).

280 TUCKER AND REED

alternative methods that often are multivariate, longitudinal, andobservational. We conclude that an informed evidentiary pluralismcan help advance linkages between rehabilitation science, clinicaltreatment, and public health practice and that psychologists shouldappreciate and help others to learn that not all research questionswith practice implications will fit the RCT model.

History, Benefits, and Limitations of RCTs

Relative to the natural sciences, the social and behavioral sci-ences are recent developments of the 20th century and, despitetremendous advances in some areas, the state of knowledge de-velopment remains uneven and somewhat limited. This is partic-ularly the case in the behavior change arena in which the deter-minants and mechanisms of change remain poorly understood,even though tremendous pressures exist to develop efficacioustreatments for a wide range of health, mental health, and othersocietal problems that involve behavior. On occasion, experimen-tal research has been premature, unfeasible, or unethical due to theunderdeveloped state of knowledge or due to the nature of thephenomena of interest, despite the appeal of experimentation froma causal inferential perspective.

Philosophers of science (e.g., Kuhn, 1970; MacKenzie, 1977;Pepper, 1942/1970) have argued convincingly that approaches todata collection and theory construction should vary depending onthe state of knowledge development in a given field. When ascience is young, as is the case in many behavioral sciences,descriptive data collected using sound measures are often mostuseful for advancing knowledge and supporting the inductivedevelopment of concepts and theory. As a science matures, adeductive approach to theory construction and hypothesis testing isbetter supported and facilitative of knowledge development. Thisgenerally suggests that a good match between developmental stageand scientific methods will advance a particular field, whereas amismatch will slow, if not misdirect, knowledge development. Butin either case, the scientific method should not be reduced to orconfused with mere experimentation, and experimentation is notalways possible or desirable (e.g., consider the natural science ofastronomy). Although experiments often support causal inferencesmost readily, other methods of data gathering can suggest andilluminate causal relationships. For example, the field of epidemi-ology, the seminal science of public health, developed throughcareful observational research on patterns of epidemics that couldnot be studied experimentally using random assignment. This workresulted in a range of nonexperimental methods that have preciserules of inference on the basis of their variable strengths andlimitations, and nonexperimental methods continue to figureheavily in evidence-based public health practice (Brownson et al.,2003).

Despite these arguments for matching methodologies with thestate of knowledge development, applied research areas with pub-lic health impact, including rehabilitation psychology, continue tobe pushed toward devising the “best” treatment or policy thatreduces costs and measurably improves short-term health out-comes. This has understandably resulted in heavy resource allo-cation for the development of efficacious therapies, and the logic,design, and methods of the RCT have been prominent in thisundertaking. RCTs have contributed much to the knowledge baseand will remain central to evaluating treatment efficacy, but they

are less well suited for investigating other applied questions. Thissection describes the characteristics, benefits, and limitations of theRCT following the discussion of others (e.g., Horn et al., 2005;Leichsenring, 2004; Rosen, Manor, Engelhard, & Zucker, 2006;Tucker, 1999; Tucker & Roth, 2006) and lays a foundation for thelater consideration of longitudinal and observational methods thatcan be used in place of, or in combination with, RCTs. Suchmethods are needed to address applied research questions, such ashow treatment effects interact with participant and contextualfeatures to influence the variable course of chronic health andbehavioral health outcomes.

History of the RCT

RCTs developed in the 1940s to establish the safety and efficacyof new drug therapies. The design was rapidly and widely adopted(Concato et al., 2000), including as part of the U.S. Food and DrugAdministration (FDA) process to evaluate new drug therapies forhumans after drugs showed promise in preclinical trials withanimals (Meadows, 2002). The four-phase FDA process includessmall studies with healthy humans (N ! 20–80) to evaluate drugsafety (Phase 1); moderate (N ! several hundred) and then large(N ! several hundred to several thousand) sample studies toevaluate treatment efficacy for persons with a disease or at risk fora disease before a drug is approved (Phases 2 and 3, respectively);and large studies that extend the populations, dosages, and long-term effects of a drug’s therapeutic use after it receives FDAapproval (Phase 4). Power is increased across phases to detect bothtreatment effects and uncommon adverse events that may not befound using smaller samples. The RCT is now widely used toevaluate medical and psychosocial treatments for a wide range ofhealth and behavioral disorders, in addition to pharmacotherapies(Boutron et al., 2008). As discussed next, this extension is basedon assumptions that fit well with some research questions but notwith others.

Benefits of the RCT

The RCT is a powerful research tool when applied and executedproperly. Much of its appeal lies in the simplicity and benefits ofrandomization. Random assignment to experimental and controlconditions is thought to “neutralize” potential confounding vari-ables, both measured and unmeasured, known and unknown, bydistributing their effects evenly across treatment and controlgroups (Horn & Gassaway, 2007). Potential confounding variablesinclude participant characteristics, self-selection processes, andother contextual features that surround intervention delivery. Fol-lowing randomization, between-group differences observedshortly after treatment administration support confident inferencesthat the treatment, and not an associated selection process orconfounding contextual factor, caused the observed differences.Although maintenance of treatment gains may be clinically desir-able, it is not essential for establishing causality.

A cardinal feature of the RCT is its emphasis on maximizinginternal validity in order to separate and detect the “true” effectof a treatment independent of other possible influences onoutcomes. In addition to the benefits of randomization, thefollowing procedures are often used to aid detection of treat-ment-produced effects apart from other sources of systematic

281SPECIAL ISSUE: EVIDENTIARY PLURALISM

and error variance: (a) use of large, homogeneous samples with“pure” forms of the condition targeted for treatment; (b) pre-randomization “run-in” periods to select participants who willcomply with the treatment protocol; (c) monetary incentives orfree treatment for study enrollment or compliance to assure thattreated participants are adequately “dosed”; (c) double-blindprocedures to minimize placebo and expectancy effects; (e)manualized treatment protocols with frequent fidelity checks;(f) standardized outcomes assessment; and (g) wait list orminimal treatment control conditions to assess effects due to thepassage of time (“spontaneous” remission).

When used to address an appropriate research question, awell-executed RCT provides stronger evidence of a causalrelationship between treatment and outcomes than nonexperi-mental research, such as the retrospective case-control or pro-spective cohort studies that are common in epidemiology. Thus,in the early EBP hierarchy (see Table 1), nonexperimentaldesigns were placed at lower levels of the hierarchy, bothbecause of their nonexperimental status and because they werethought to entail more risk of selection biases that could affectinferences about treatment efficacy (Sacks et al., 1982). Con-cerns about selection bias, or the tendency for membership intreatment and control groups to be determined by nonrandomfactors that also influence outcomes, appeared to be valid forsome conditions. For example, decades of conventional wisdomand early observational studies had suggested health benefits ofhormone replacement therapy (HRT) in postmenopausalwomen, including a reduction in the risk of stroke. However, alarge RCT of HRT efficacy (Wassertheil-Smoller et al., 2003)showed that some forms of HRT significantly increased the riskof ischemic stroke in generally healthy postmenopausal women.Similar discrepancies have been found about the benefits of betacarotene and vitamin A on lung cancer and cardiovasculardiseases (Omenn et al., 1996), with observational studies sug-gesting benefits and experimental studies failing to find them.Such examples are cited as illustrative of the “stainless steelrule of evaluation”; that is, the more rigorous the research, thelower the chances of positive findings (Rosen et al., 2006).

In spite of these situations in which selection biases appear tohave clouded inferences about treatment effects, RCTs and obser-vational research have produced similar results in many otherhealth care areas. Comprehensive meta-analytic reviews havefound similar effect sizes for RCTs and well-conducted, nonran-domized cohort and case-control studies that used current controlgroups, selection criteria, and measures similar to those used in theRCTs (Benson & Hartz, 2000; Concato et al., 2000; Shadish, Navarro,Matt, & Phillips, 2000). The findings from the two designs convergedacross a wide range of treatments and conditions, and, when differ-ences were observed, effect sizes tended to be lower, not higher, in thequasi-experimental studies (Shadish et al., 2000). The long-standingconcern that observational studies without random assignment willoverestimate treatment effects because of selection bias appears to beexaggerated, especially when sound research methods appropriate tothat approach are used.

Limitations of the RCT

A more accurate conclusion about the relative merits of well-conducted observational research and RCTs is that both contribute

to the evidence base for practice, and confidence about causalinferences and generalizability is high when the findings converge.However, the RCT is not well suited to study how outcomes ofinterest are influenced by powerful extratherapeutic variables thatmay interact with treatment effects over time. As discussed next,this expanded scope of inquiry is essential for understandingrehabilitation processes and outcomes.

Understanding and promoting change in the real world. Al-though RCTs establish treatments as efficacious under controlledconditions that maximize internal validity, there often are trade-offs between internal and external validity (Shadish, 2002). Con-cerns about the generalizability of RCT findings to usual practicesettings led to increasing emphasis during the 1990s on treatmenteffectiveness studies as part of the necessary research base forpractice (Blacker & Mortimore, 1996; Seligman, 1995; Tucker,1999). As summarized in Table 2, effectiveness studies evaluatewhether treatments established as efficacious in RCTs generalizeto real-world practice settings and also investigate provider adop-tion and patient utilization patterns and processes. The studysamples better represent the heterogeneity of patient populationsand disorders, including co-morbid conditions. Although their useentails greater risk to internal validity, effectiveness studies tend tohave greater external validity because treatment processes andoutcomes in usual care environments using a broader patient mixare studied.

Concerns about how methodological features of the RCT wereshaping the knowledge base for practice fueled this expansion ofthe applied research agenda. Foremost among the concerns is thefact that treatments are never randomly prescribed in the real world(Tucker & Roth, 2006). Complex social, economic, health careaccess, and health status variables influence service utilizationprocesses and patterns across a wide range of health and behav-ioral health problems (Andersen, 1995; Mechanic, 1978; Morrisey,1992; Tucker & King, 1999). But by virtue of random assignment,RCTs create an artificial service delivery environment that elimi-nates, by design, the role that treatment selection factors have ondetermining who is likely to enter a treatment program or howtreatment selection and other contextual variables may influencetreatment outcomes. This trade-off between experimental controland real-world conditions is of minor concern when the treatmentunder study does not depend much on patient motivation, compli-ance, active behavior change, or the environmental circumstancesthat surround treatment delivery. But the more these extra-thera-peutic dimensions matter in determining outcomes, as they sooften do in rehabilitation, the more poorly suited RCTs are forinvestigating the nature of linkages among the relevant variablesets. For example, RCTs are well suited for studying the effects ofrelatively passive interventions, such as a surgical procedure thatinvolves little or no postoperative adjustment or rehabilitation bythe patient. But RCTs are not a good design choice when theself-selection and motivational components of complying withcare matter to its success, as often occurs in a lengthy rehabilitationprocess. Achieving the desired treatment effect requires moreactive participation on the part of the patient, for example, byadhering to a home physical therapy program, regularly releasingpressure to prevent pressure sores, properly using and maintainingcatheters and other equipment, or not allowing well-meaning fam-ily members to offer a level of assistance that interferes withattainment of maximal independence.

282 TUCKER AND REED

It is well established that treatment seeking and treatment out-comes commonly depend on individuals’ motivation for change;the use of a change process that is consistent with their values,capabilities, and life circumstances; and social network reactionsand support (Miller & Rollnick, 2002; Tucker & King, 1999). TheRCT, however, treats these person attributes and contextual factorsas potential confounding variables, and whatever effect they mayhave is included in the estimate of the within-group error varianceterm (Tucker & Roth, 2006). Some studies measure such variablesand assess their effects on outcomes statistically, but usually this isdone to remove this source of variability from the error term ratherthan to understand behavior change and the role of treatment inpromoting it. But treatment seeking and treatment engagement arethemselves complex processes that are influenced by characteris-tics of individuals and environments and have important relation-ships with outcomes (Tucker & King, 1999). Furthermore, thehealth problems of concern to rehabilitation often wax and wane orfollow a deteriorating course that requires multiple treatments andtreatment modalities over time (e.g., multiple sclerosis, amyotro-phic lateral sclerosis). Understanding the factors that lead to treat-ment entry at the appropriate point in a disease process in order tobenefit maximally from treatment is essential for preventing sec-ondary complications and further disability.

Sample limitations. A second shortcoming of the RCT is thecommon use of restricted, unrepresentative samples that do notreflect the heterogeneous case mix found in usual care settings orin the larger population with the problem of interest, whether ornot health services are sought. Conducting RCTs using carefullyselected patients who lack complicating co-morbidities or othercontaminating characteristics is not mandated by the RCT meth-odology. Nevertheless, such homogeneous patient groups arethought to provide the best opportunity to observe treatment ef-fects that are not attenuated by the effects of the multiple contrib-uting factors and other conditions that have been eliminated by theparticipant eligibility criteria. In rehabilitation, co-morbidities areprevalent (e.g., hypertension, diabetes), and people who have themtypically are excluded from RCTs. This may bias effect sizeestimates of RCTs in the direction of suggesting stronger treatmenteffects than would be observed in usual practice settings with more

heterogeneous and diverse patient groups (Blacker & Mortimore,1996).

Another concern is the increasingly common use of stricteligibility criteria for sample recruitment, and how these criteriamay skew the knowledge base for treatment, sometimes withadverse health outcomes. Horn et al. (2005) discussed such anexample involving an influential RCT that demonstrated thebenefits of the drug spironolactone, a potassium-sparing di-uretic, to treat congestive heart failure (CHT; Pitt et al., 1999).In the 18 months that followed, there was a four-fold increasein the number of sprionolactone prescriptions, followed by atripling of hospital admissions and deaths from dangerous ele-vations of potassium among CHF patients (McMurray &O’Meara, 2004). The problem was that many CHF patients whowere prescribed sprionolactone would have been excluded fromthe RCT that established efficacy because of its stringent se-lection criteria. In short, the trial’s selection criteria did notadequately take into account real-world practice and, in thiscase, resulted in serious negative outcomes.

Unless interpreted carefully, results of RCTs can be misleadingas evidence for practice because of the use of nonrandom ornonrepresentative samples. The samples are further biased becauseRCTs only include people willing to participate in a clinical trialthat involves random assignment to a promising new treatment orto a usual care or other control condition. This very small propor-tion of the already small proportion of people with a given problemwho will seek and use a prescribed treatment may have unusualcharacteristics. A related concern with relevance to rehabilitationstudies is that RCTs are often conducted in university-based ter-tiary care hospitals that serve those too sick to be treated in theirlocal medical facilities.

Treatment fidelity versus individualization. A third artificialityof the RCT emanates from procedures aimed at achieving hightreatment “fidelity” by standardizing research and treatment expe-riences across participants. This approach is appropriate when theevaluation target is an acute surgical or pharmacological interven-tion that would benefit from fine adjustments in technique ordosing. But when the treatments and disorders are distributedthrough time and are context dependent, as they often are in

Table 2Efficacy Versus Effectiveness Studies of Treatment Effects

Variable Efficacy studies Effectiveness studies

Purpose Detect the “true effects” of treatment by reducingextraneous variance

Assess treatment effectiveness in usual care settings

Typical characteristics homogeneous samples heterogeneous samplesrandom assignment, blinded circumstances leading to and surrounding caremanualization, standardized treatment, fidelity checks studied

compliance measured, not forcedBetter suited for

studyingtechnical interventions with acute effects (e.g.,

surgery, pharmacotherapy)interventions with effects that are fairly independent

psychosocial problems and interventionsdistributed through time (e.g., functionalrecovery after injury)

of contextinterventions amenable to fine adjustments to

improve outcomes

problems and interventions that are influenced bycontext

care studied as a process, not just as an outcomeValidity and inference Higher internal validity but lower generalization

potential and capacity to inform science-to-practicelinkages

Lower internal validity but higher generalizationpotential and capacity to inform science-to-practice linkages

Note. Adapted from Tucker (1999, p. 25).


rehabilitation and mental health care, rigid treatment standardiza-tion risks insensitivity to the dynamic nature and controlling vari-ables of change. It also is difficult to achieve in the clinicalrehabilitation context. Most therapists will do whatever they can tohelp their patients function as well as possible, regardless ofwhether it is part of a research protocol, and patients may seeothers receiving beneficial procedures and ask for them, or theymay try to replicate them on their own. Such provider and treat-ment contamination effects may lead to biased estimates of treat-ment effects in RCTs (Boutron et al., 2008).

Although not well studied, rehabilitation outcomes also arelikely influenced by the “common factors” known to influenceoutcomes in psychotherapy (e.g., therapeutic alliance, patient ex-pectations), which generally are more influential than the technicalattributes of different psychotherapies (Norcross, 2002). The abil-ity to form a therapeutic alliance and to motivate patients toparticipate in lengthy, difficult, tedious, and even painful treat-ments is at least as important in the rehabilitation context (Purtilo& Haddad, 2007). Such factors cannot be studied in manualizedRCTs in which therapists are required to respond in standard waysthat are insensitive to variations in client responses. These com-mon research practices promote high treatment fidelity and internalvalidity but may contribute to suboptimal outcomes compared withtreatments that are more individualized.

Role of theory. A final concern about RCTs is that the use ofrandom assignment can convey a false impression that there is areduced need for the guidance of theory in research (Tucker &Roth, 2006). The random assignment feature of the RCT appearsto empower atheoretical treatment evaluation research because theinvestigator does not have to consider, measure, or model thecovariates of treatment seeking, treatment selection, and otherextra-therapeutic influences on outcomes over time. This is thecase even if the treatment itself is well grounded in theory, andserious missteps can occur in the treatment development andvalidation process if the mechanisms of therapeutic action areunknown. As Spring et al. (2005) noted, theoretical research aboutmechanisms of action typically are a preliminary step in the designof biomedical interventions (e.g., based on preclinical animalmodel research). But for psychosocial treatments, concerns aboutthe mechanisms of action typically occur after, not in advance of,RCT demonstrations of treatment efficacy.

In rehabilitation psychology, theory development often has beenlargely bypassed in favor of evaluating the effectiveness of inter-ventions, products, devices, and environmental modifications. Therisks are considerable for advancing the knowledge base, devel-oping interventions, and understanding the mechanisms of inter-vention action (Fuhrer, 2003; Hollon, 2006). Atheoretical RCTsmay indicate that an intervention has an effect but do not revealhow or why. Furthermore, because many processes important todisability adjustment and participation outcomes (see Elliott &Warren, 2007) cannot be manipulated or subjected to randomassignment, theory is especially needed to guide intervention de-velopment and the derivation of associated predictions that can beinvestigated in quasi-experimental research. The disconnection ofresearch in rehabilitation psychology from larger theoretical per-spectives on behavior change has meant that most available dataare descriptive and do not follow from theoretical propositions thatcould be used to predict behavior or guide intervention develop-ment (Elliott, 1994; Elliott & Frank, 2000).

An excellent example of a theoretically based program of re-search that is relevant to rehabilitation psychology grew out ofTaylor’s (1983) theory of cognitive adaptation, which proposedthat psychological beliefs, such as the ability to find meaning inone’s experience, a sense of control, and optimism, helped pre-serve mental health in the face of traumatic or life-threatening(health) events. Although the positive beliefs expressed often wereinconsistent with medical evidence and therefore could be consid-ered unrealistic or illusory, their presence was associated withpositive adjustment, not with mental health problems (Taylor &Brown, 1988). For example, Taylor’s cognitive theory predictedthat unrealistically optimistic HIV-positive men would not be lesslikely than their more realistic counterparts to engage in safer sexpractices, contrary to the opposite prediction of another theory(Weinstein, 1984). A study in which these divergent predictionswere tested supported Taylor’s theory (Taylor et al., 1992).

What makes this line of research powerful and persuasive wasthe ability to make specific, theoretically based predictions aboutwhat would be observed at each step (see Taylor, Kemeny, Reed,Bower, & Gruenewald, 2000). The prospective studies that com-prise this body of work were methodologically rigorous and con-trolled for many potential confounds, but none were RCTs. Thismay seem obvious given that no treatment was being evaluated,but our point is that this kind of longitudinal research is bettersuited for the theory building and evaluation that seem to beamong the more significant needs to advance rehabilitation psy-chology. At a later stage, theoretically based treatments can bedesigned and evaluated using RCTs. We return to this issue laterwith another example involving constraint-induced movementtherapy.

Summary

The very features of RCTs that enhance internal validity oftenpreclude the study of the multiple determinants and processesinvolved in behavior change and that contribute to long-termoutcomes (Seligman, 1995; Tucker, 1999; Westen et al., 2004).RCTs are an excellent choice when treatments are likely to beefficacious regardless of contextual or self-selection factors andcan produce quick benefits that logically can be attributed to thetreatment and not to extratherapeutic events and circumstances thatoccur during the interval surrounding treatment participation. Thedesign is less suitable for investigating how such contextual vari-ables or treatment selection factors may affect intervention out-comes or for studying the often lengthy process of behavior changeand the role of changing health states in that process. For thesereasons, treatment effectiveness studies have developed as a com-plementary research strategy to the RCT and are aimed at increas-ing external validity. Interventions established as beneficial inefficacy research are evaluated under usual care conditions using abroader patient mix.

Placing RCTs at the top of the evidence hierarchy tends to drivethe development of a knowledge base for practice that favorsshort-term acute care over chronic disease management and reha-bilitation. This is because the kinds of treatments and disorders thatcan be readily investigated using RCTs tend to be acute medicaltreatments like surgery and drug therapy for health problems thatcan be diagnosed and treated in a single, time-limited course ofcare (Reed, 2006). The design is not as well suited for studying the

284 TUCKER AND REED

long-term functional outcomes of treatments for chronic health andbehavioral health problems that have recurring periods of remis-sion and relapse and that are influenced by environmental andsocial factors, the very conditions that characterize much of reha-bilitation. In addition, selecting variables for manipulation, assess-ment, or statistical control should be a deliberative process guidedby theory and prior research as well as by practical and ethicalconstraints on what variable domains can and cannot be manipu-lated experimentally.

Advancing the Applied Knowledge Base forRehabilitation Psychology

We now return to the question of what kinds of evidence willadvance the knowledge base for practice in rehabilitation psychol-ogy, in light of the state of development of theory and research.We begin by selectively highlighting the field’s accomplishmentsand then discussing issues in rehabilitation that are difficult toapproach with RCT methodologies.

Research Accomplishments That Inform Practice

Rehabilitation has made remarkable accomplishments during its50-year history, and consistent with the field’s multidisciplinaryvision, rehabilitation psychologists have been integrally involvedin the inception, operation, and direction of the broader medicalrehabilitation enterprise and in the development of its researchbase (Elliott & Frank, 2000). Some of this research figures prom-inently in a recent IOM report (2007) titled “The Future of Dis-ability in America.” For example, psychologists were instrumentalin the development and direction of the Model Systems of reha-bilitation care for spinal cord, traumatic brain, and severe burninjuries, funded since the 1970s by the National Institute onDisability and Rehabilitation Research (NIDRR) in the U.S. De-partment of Education. The Model Systems include uniform mul-tisite data collection and have made substantial contributions toresearch and practice (IOM, 2007).

These and other initiatives have provided abundant, unequivocalevidence that psychological factors play a powerful role in reha-bilitation processes and outcomes. This includes overall outcomesof quality of life and life satisfaction (e.g., Rosenberg, Plakeney, etal., 2006; Tate, Kalpakjian, & Forchheimer, 2002; Whiteneck,Forchheimer, & Krause, 2007) and specific outcomes such asdepression, substance use, and suicide risk (e.g., Bombardier,Richards, Krause, Tulsky, & Tate, 2004; Rosenberg, Robert, et al.,2006; Tate, Forchheimer, Krause, Meade, & Bombardier, 2004).Behavioral, psychological, and social factors generally account formore variance in adjustment outcomes than the specific character-istics of an injury or a disability (Elliot & Frank, 1996; Elliott &Warren, 2007; Gatchel, Polatin, & Mayer, 1995).

There also is a strong neuropsychological research tradition inrehabilitation, particularly in relation to traumatic brain injury.This work has focused on the relationships among neuropsycho-logical performance, cognitive and neurobehavioral rehabilitation,and outcomes (e.g., Atchison et al., 2004; Niemeier, Kreutzer, &Taylor, 2005) as well as related issues such as the ability to makemedical decisions (e.g., Marson et al., 2005). The vocational/occupational research tradition in rehabilitation psychology goes

back to the field’s inception. This research has produced a richbody of knowledge concerning predictors of vocational outcomesamong rehabilitation patients and other people with disabilities andhas produced programs that support successful occupational per-formance, return to work, and job stability (e.g., Walker, Marwitz,Kreutzer, Hart, & Novack., 2006; cf. Elliott & Leung, 2005).

Although it is not always feasible or desirable to dismantlepsychological interventions from other components of rehabilita-tion care, they have generally fared quite well when examinedspecifically in a rehabilitation context (Elliott & Jackson, 2005;Elliott & Warren, 2007). Psychological interventions have beenshown to make specific and substantial contributions in the contextof rehabilitation programs for chronic pain (Fordyce, 1976;Gatchel, 2005), serious mental illness (Corrigan, Mueser, Bond,Drake, & Solomon, 2007), and cardiovascular disease (Linden,Stossel, & Maurice, 1996; Mendes de Leon et al., 2006). Psychol-ogists also have been heavily involved in the development ofpsychologically based assessment models to support an optimalmatch between persons with disabilities and assistive technologies(Scherer, Coombs, Merbitz, & Merbitz, 2006). On a more systemiclevel, psychologists have contributed extensively to the develop-ment, dissemination, and implementation of the WHO (2001)International Classification of Functioning, Disability, and Health(ICF; Reed et al., 2005), including a classification for children andyouth (Lollar & Simeonsson, 2005). Despite some acknowledgedimperfections, the ICF is emerging as the dominant model ofdisability (IOM, 2007), and it is increasingly well represented inthe rehabilitation psychology literature (Elliott & Warren, 2007;Hendershot & Lollar, 2007).

Unresolved Issues Related to the Poor Fit WithRCT Methodologies

In spite of these many accomplishments, psychological researchin rehabilitation has frequently been limited by being embedded inlarger projects framed by the perspective and research needs ofmedical rehabilitation. Elliott and Frank (2000) observed thatrehabilitation psychologists have often used their research skills insupport of overall medical rehabilitation, but, in so doing, have“ignored the need to demonstrate the worth and cost-effectivenessof psychological interventions while nurturing medical databases”(p. 646). They criticized rehabilitation psychology for failing toadvance and evaluate psychological models of behavior change,which they say has unwittingly advanced a medical model and hasfallen short with respect to providing the evidence demanded intoday’s health care environment for psychological interventions. Inaddition, because much knowledge in rehabilitation psychologydoes not derive from RCTs, when the evidentiary criteria of theEBP movement are applied, the unsurprising and circular conclu-sion is that evidence for practice is deficient. We next discuss theways in which EBP methodological directives have constrainedresearch in rehabilitation psychology and suggest alternative meth-ods for knowledge development.

Shared concerns with other areas of psychological practice.The evidentiary criteria of EBP emphasizing RCTs are morecongenial to investigating acute drug and somatic treatments,thereby favoring them over psychological treatments in the evi-dence base. Rehabilitation psychology and other areas of psycho-logical practice suffer from this bias, which shifts favorable con-


clusions in the direction of drug and somatic treatments whenevidence is reviewed to determine which practices qualify asevidence-based. Powerful financial incentives also operate to builda research base for pharmacological treatments, which are heavilyfinanced by commercial interests (Reed & Eisman, 2006), and, toa lesser degree, for costly assistive devices and surgical proce-dures. These incentives include FDA approval, marketplace prof-its, and substantial hospital billings. This has contributed to thestrong bias noted by the IOM (2007) in favor of body-levelinterventions in rehabilitation research and practice. Comparablefinancial incentives do not exist for lower technology aspects ofrehabilitation such as psychological, occupational, and physicaltherapy. As a result, the corresponding research base has grownmuch more slowly.

Negative findings for somatic treatments are almost never pub-lished. This has contributed to further distortion in the evaluationof the research base, because strength of evidence in EBP isgenerally equated with the number of positive RCTs. Furthermore,many EBP advocates view the double-blind RCT as the idealclinical trial, including in rehabilitation research (Tate, Findley,Dijkers, Nobunaga, & Karunas, 1999), and some have gone so faras to endorse the randomized blinded trial to eliminate any possi-bility of investigator bias (Hulley & Cummings, 1988). Obviously,most psychological treatments cannot be studied double-blind, andfrequently it is impossible to conduct a single-blind trial.

These factors interact with other preconceptions and biases inthe medical community in evaluating evidence for practice, evenwhen research supports psychological treatments. For example, acomparison of cognitive therapy and medication for depressedpatients in which the cognitive therapy appeared to have beenpoorly implemented concluded that its effects were inferior tothose of medication (Elkin et al., 1989). But when psychologicaltreatments for depression have been well delivered, they havecompared favorably with medications (Hollon, Jarrett et al., 2005;Hollon, Thase, & Markowitz, 2002) and have had longer lastingeffects (Hollon, DeRubeis et al., 2005). Nonetheless, influentialguidelines for depression treatment, primarily developed by psy-chiatrists, emphasized pharmacotherapy over psychological treat-ment (e.g., National Institute for Clinical Excellence, 2004).

Although there are several RCTs of postacute rehabilitationservices as practiced in usual treatment environments (e.g., Lang-horne & Duncan, 2001), rehabilitation as an enterprise sharesmany of the characteristics that make psychotherapy difficult tostudy using RCTs (Seligman, 1995). First, rehabilitation is gener-ally not of fixed duration. In RCTs, the treatment typically stopsafter a certain interval or number of sessions, regardless of how theperson is doing. Second, rehabilitation is self-correcting. If onetechnique is not working, then another technique or modality istried. In RCTs, treatment is confined to specified techniques,administered in a standardized manner. Third, in rehabilitation,patients are generally active. They participate in the selection oftreatments and goals, which is considered desirable, if not neces-sary for treatment success. This contrasts with the passive processof random assignment and acquiescence to whatever treatment isoffered as part of an RCT. Fourth, individuals undergoing reha-bilitation typically have multiple problems, and treatment is gearedtoward relieving parallel and interacting difficulties. Patients inRCTs are typically selected to have a single problem, and co-morbid conditions are often a basis for exclusion. Finally, reha-

bilitation is concerned with the improvement of general function-ing as well as the amelioration of specific presenting symptoms,which are generally the focus of RCTs.

Special concerns for rehabilitation psychology. Further char-acteristics of rehabilitation are difficult to study using RCTs. First,the time horizons for many of rehabilitation’s intended effects areextremely long. RCTs typically involve short follow-up periods(e.g., 3–6 months), and inferences about efficacy are based onchanges observed during or soon after treatment administration.When patients are followed for longer periods that more fullycapture the intended goals of rehabilitation, it is increasinglydifficult to attribute observed differences in outcomes to the ex-perimental manipulation rather than to other environmental factorsthat cannot be controlled. In addition to inevitable changes inpatients’ life contexts, different phases of the rehabilitation processand recovery trajectory typically involve different needs and goals,and individuals with disabilities may seek and receive many treat-ments and services over many years. Because of the problems thatlong time horizons and environmental interactions pose for RCTsof a single acute episode of care, both procedurally and withrespect to causal inferences, participation outside of the treatmentcontext is almost never used as a measure of rehabilitation out-comes. Yet, it is nearly universally endorsed by consumers, healthprofessionals, and funding agencies as the ultimate goal of reha-bilitation (Heinemann, 2005).

One way to deal with this state of affairs is to enroll a largesample that is selected on the basis of the functional rather thanphysical nature of disability and then follow them for many yearsusing a core assessment battery that is supplemented intermittentlyby measures tied to the expected trajectory of recovery and thatinclude global endpoint measures of participation. For example,Whiteneck and colleagues (2004) conducted a large longitudinalstudy of 2,726 individuals with spinal cord injuries in which theyinvestigated the role of environmental factors in long-term adjust-ment as assessed at 1, 5, 10, 15, 20, and 25 years after initial injury.Environmental factors were more strongly related to life satisfac-tion than to societal participation. Such longitudinal cohort studiescould be supplemented intermittently with brief randomized trialsof short-term interventions with expected time-limited effects thatare based on normative or individual recovery patterns and reha-bilitation needs. Although lingering exposure effects from onesuch treatment to the next would be a concern, treatment exposurewould be deliberately sequenced and the effects measured, ratherthan occurring in an uncontrolled, but probably nonrandom fashionas patients seek out additional services during the lengthy recoveryprocess.

Second, rehabilitation research often encounters difficulties ob-taining sufficiently large, homogenous samples to adequatelypower RCTs when eligibility is based on diagnosis because baserates of many conditions are low. As discussed in a later section onconceptual issues, this issue may be resolved somewhat by adopt-ing a “disability model” of rehabilitation that focuses more onfunctional capacity than on the nature of the physical diagnosis,injury, or avenue into the health care system. Use of the ICF(WHO, 2001) to define samples on the basis of functional statusrather than diagnosis offers a potential means to obtain sufficientsample sizes for RCTs.

Third, defining and selecting an appropriate control group inRCTs can be challenging in rehabilitation studies. The ethical

286 TUCKER AND REED

doctrine of clinical equipoise in RCTs requires that (a) there isevidence-based disagreement or uncertainty about best practices,and (b) no participant is “randomized to an intervention known tobe inferior to one of the treatments under investigation or theestablished, scientifically validated standards of care” (Miller &Brody, 2007, p. 153). This is a complex determination, and onethat has come to favor RCTs that compare a “usual treatment” withan experimental treatment; wait list, minimal, or no-treatmentcontrol groups have fallen into disfavor on ethical grounds. How-ever, clinical equipoise is particularly difficult to achieve in reha-bilitation RCTs because there often is no “usual treatment” againstwhich an experimental treatment can be compared. Natural historydata are generally lacking, especially concerning initial trauma andinjury (e.g., stroke, spinal cord injury), so the time frames ofnatural recovery and intervening markers of longer term adjust-ment and outcomes that could serve as benchmarks against whichto assess further benefits due to rehabilitation are unknown. Whencoupled with the problems of defining control groups, the lack ofnatural history data makes it is difficult to conduct treatmentefficacy studies in rehabilitation psychology that fulfill the meth-odological and ethical requirements of sound RCTs.

Collectively, these methodological issues lead in a direction thatfavors acute medical and pharmacological treatments with short-term outcomes and biological endpoints, and thus may be partic-ularly troubling for rehabilitation psychology. The IOM (2007)characterized research on rehabilitation and disability as “highlyskewed toward basic and clinical research, with inadequate supportfor research on the physical, social, and other environmentalcontributors to disability and insufficient attention to the evalua-tion of interventions to minimize activity limitations and partici-pation restrictions” (p. 5).

Conceptual issues. In addition to the constraining influence ofRCT methodologies, two unresolved conceptual issues appear tobe limiting the advance of the field. The first concerns the diverg-ing research agendas of what we refer to as the dominant clinicalrehabilitation model and an emerging public health-oriented dis-ability model. The clinical rehabilitation model defines its subjectmatter according to who enters specific systems for rehabilitationcare and views the overarching purpose of rehabilitation researchas being to improve systems of care and patient outcomes (e.g.,cost-effectiveness, quality of life, health status, physical function-ing). Evaluation of specific interventions using RCTs is whollyconsistent with this view, even though carrying out such studiesmay be challenging because of the multifactorial nature of theinterventions and the temporally distant nature of many desiredrehabilitation outcomes.

In contrast, the disability model defines the target populationusing an explicit set of criteria based on the associated functionand goals (e.g., eligibility for health benefits, access to publicfacilities). Because the definition of disability depends on itsfunction, no definition is universally accepted (Lollar & Crews,2003). Disability and morbidity are not synonymous (Chamie,1995), and people identified as disabled may not consider them-selves disabled or experience functional limitations or health prob-lems. The disability model underlies the 2007 IOM report and isclosely aligned with a public health model aimed at preventingsecondary conditions (Lollar, 1999), which is any condition towhich a person with a primary disability is more susceptible (e.g.,medical, mental, family problems). Preventing secondary condi-

tions is part of the Healthy People 2010 agenda (Department ofHealth and Human Services, 2000).

Although the disability model research agenda includes a rolefor the systematic evaluation of interventions, it directs attentiontoward new research questions, such as predictors of positiveadjustment among people with disabilities and the nature of theadjustment process (e.g., Dunn & Dougherty, 2005; Elliott, Ku-rylo, & Rivera, 2002), reducing employment barriers to peoplewith disabilities (e.g., Bruyere, Erickson, & VanLooy, 2004),understanding how people with disabilities experience their iden-tity and treatment in society (e.g., Olkin & Pledger, 2003), andaddressing the fact that many common psychology measures havenot been normed on people with disabilities and may requiresignificant adaptation (e.g., Olkin, 2002). Its expanded view ofinterventions includes population-based public health interven-tions and environmental modifications, in addition to individualtreatments. Research guided by a disability model is more likely tobe based on functional than on diagnostic status, which is impor-tant because specific diagnoses are not good predictors of second-ary conditions (Lollar, 1997; Ravesloot, Seekins, & Walsh, 1997).Interventions are likely to be seen in the context of an unfoldingprocess rather than as isolated treatments with immediate out-comes. The research agenda that proceeds from the disability modelextends beyond treatment efficacy studies that are the foundation of anarrow view of the evidentiary requirements for EBP, and it thusseems better suited than the rehabilitation model to guide much ofcontemporary research in rehabilitation psychology.

A final issue concerns outcome criteria in rehabilitation re-search. Full participation in society has long been viewed as theultimate goal of rehabilitation (Heinemann, 2005; IOM, 2007;Pope & Tarlov, 1991; WHO, 2001; Wright, 1980), including aspart of the Americans With Disabilities Act (1990). Althoughparticipation is differentiated from activity performance alone andis distinguishable empirically (Cardol, De Jong, & Ward, 2002;Van Brakel et al., 2006), far more attention has gone into devel-oping measures of body functions and activity performance inrehabilitation contexts than into measures of societal participation(Heinemann, 2005). It is much easier to measure performance ofdaily tasks such as drinking, eating, bathing, and walking than tomeasure the longer term and more global outcomes that tend to bemost important to people with disabilities or who are undergoingrehabilitation following an injury or in the context of a chronic illness,such as: Will I be able to work? Will I be able to have closefriendships? Can I manage living on my own? Can I have an intimaterelationship? Can I be a good father? These outcomes are of moreinterest to society as well. However, significant method developmentwork remains before rehabilitation research can focus on outcomesrelated to participation (Heinemann, 2005). The development of suchmeasures is complicated by the fact that participation outcomes oftenare temporally distant from specific rehabilitation interventions. Yet,this effort is essential if research is to address the outcomes thatindividuals with disability say are most important to their lives.

Revising the Evidence Hierarchy to Promote Knowledgefor Practice

The preceding discussion highlights conceptual issues and em-pirical questions that cannot be fully addressed using RCTs. This


section summarizes recent expansions of the evidence base forpractice in ways that extend its reach and effectiveness in real-world settings. Alternative research designs are described thatinvestigate naturalistic components of care seeking and behaviorchange either separately or in conjunction with experimentallymanipulated treatment variables. Such research complements theevidence base for practice provided by treatment efficacy studiesand leads to an expansion of the dimensions of research that arerelevant to evaluating the utility of evidence for informing prac-tice.

Revisions to Study Natural Variation and to ModelTreatment Selection

Several recent critiques have placed greater value on studies oftreatment seeking and treatment effectiveness in the natural envi-ronment (Horn, 2006; Horn et al., 2005; Leichsenring, 2004;Seligman, 1995; Shadish, 2002; Tucker & Roth, 2006). Well-conducted naturalistic studies are considered an undervaluedsource of evidence that are better than RCTs for addressing certainresearch questions that are vital to practice. As Horn et al. (2005)noted, such methodological alternatives “. . . do not replace RCTs,but rather provide additional sources of systematic outcomes in-formation . . . . that answer questions in the real world, wheremultiple variables and factors can affect the outcomes” (p. S14).“Unlike RCTs, [such] studies do not experimentally alter thetreatment regimen to evaluate the outcomes associated with aparticular intervention, but take advantage of natural variation inpatients and care processes occurring in routine practice . . .”(Horn, 2006, p. 2732). Leichsenring (2004) similarly emphasizedthat RCTs of treatment efficacy and naturalistic studies of treat-ment effectiveness serve different purposes and thus require dif-ferent criteria to judge research quality. Noting that “psychother-apy is not a drug that . . . works equally under different conditions”(p. 144), he presented an evidence scheme that placed highestvalue on prospective quasi-experimental studies with high clinicalrepresentativeness in which treatments, patients, and therapistscommon in practice were used.

Tucker and Roth (2006), in the context of research on treatmentsfor substance use disorders, specifically emphasized the value ofmultivariate longitudinal studies of treatment, person, and contex-tual variables that influence outcomes. The fundamental goal is tomodel the multiple influences on health and behavior, includingbut not limited to interventions, and to make valid inferences thatapply to larger populations. The multivariate longitudinal approachoverlaps with and extends beyond the RCT. Such studies canaccommodate experimentally manipulated independent variablesand naturally occurring change in variables that are not manipu-lated (Singer & Willett, 2003). Specific multivariate methods, suchas mediation modeling, can be applied to both experimental in-vestigations (Shrout & Bolger, 2002) and longitudinal designs(Judd, Kenny, & McClelland, 2001).

Unlike in RCTs, in the multivariate longitudinal model, random-ization is not used to avoid the important theoretical work requiredfor model building, even though randomization may be useful forproviding targeted experimental control of a key predictor variable(Tucker & Roth, 2006). In a well-developed longitudinal model,randomization decisions are based on theory and research in thefield and on the ease with which a variable is amenable to con-

trolled manipulation. Randomization can be used to strengthenselect causal pathways in an overall multivariate longitudinalmodel that includes experimentally manipulated and naturally oc-curring pathways. But most associations will be among the pathsthat are naturally occurring without experimental control. Infer-ences to clinical populations thus may actually be strengthened byavoiding unnecessary experimental control, a point echoed byHorn (2006).

Use of multivariate longitudinal models is relevant to rehabili-tation psychology because treatment is only one variable of manythat contributes to longer term outcomes. The approach puts treat-ment in context, allows for the study of participation as a longerterm outcome, and is compatible with the disability model thatfocuses on the person in context, not just on the person as a patientin the health care system. Treatment can be investigated as anindependent or dependent variable. The latter focus is an importantcomplement to treatment efficacy research because treatment seek-ing, engagement, and outcomes are complex, interrelated pro-cesses that are multidetermined (e.g., by patient motivation, socialsupport, and health care access, quality, and monetary cost). RCTs,however, preclude the study of treatment selection factors and howthey may interact with treatment and other contextual factors toproduce long-term outcomes.

Finally, Borckardt and colleagues (Borckardt et al., 2008), follow-ing Kazdin (1982) and others, argued anew for the direct relevance toclinical practice of practitioner-generated case-based time series de-signs with baseline measurement. They contended that this design“qualifies as a true experiment and that it ought to stand alongsidemore common groups designs (e.g., the . . . RCT) as a viable approachto expanding our knowledge” (p. 77). Relegating case studies in tototo the lowest level of the evidence hierarchy misses important dis-tinctions between sound and unsound single-case designs.

Dimensions for Evaluating the Utility of Evidence forInforming Practice

In their extension of the evidence hierarchy to include multi-variate longitudinal studies of treatment, person, and contextualvariables that influence outcomes, Tucker and Roth (2006) iden-tified five dimensions of research to guide judgments about evi-dentiary value. The first dimension is whether the aim of the studyis to obtain descriptive data to generate hypotheses for futureresearch, or whether the study is designed to test hypotheses and toinfer from sample data estimates of effects in the larger population.Although often undervalued, the descriptive or exploratory func-tions of research are very important in the early stages of knowl-edge development in a given field (MacKenzie, 1977).

A second dimension is whether the research is cross-sectional orlongitudinal. Three or more waves of data are necessary to esti-mate both true change trajectories and measurement error compo-nents separately for each individual participant. Simple two-wavestudies with one baseline and one posttreatment assessment gen-erally do not qualify as longitudinal designs (Singer & Willett,2003).

The third dimension is whether the study analyses and statisticalmodels are bivariate or multivariate. Multivariate refers to ananalysis with more than one predictor or independent variables (asin multidetermined outcomes) or with more than one outcomebeing modeled simultaneously. Multivariate longitudinal designs

288 TUCKER AND REED

are suitable for evaluating mediation and moderation as mecha-nisms by which treatments have causal effects (Kazdin, 2007;Kraemer, Stice, Kazdin, Offord, & Kupfer, 2001). In contrast,simple bivariate analyses in such correlation coefficients or one-way analyses of variance can provide little information aboutmultiple determinants of behavior or about the independence orcodependence of multiple outcomes.

The fourth dimension is whether the research goal is to modelnaturally occurring variability or to test the effects of experimen-tally manipulated variability. The RCT is the primary method usedfor the latter purpose, but most behavioral scientists wish eventu-ally to understand and explain phenomena in the natural environ-ment. This dimension directly affects and informs clinical practice.The controlled experiment contributes to this overall goal, but it isnot the final step in an expanded conception of evidence.

A final dimension of research is its potential public healthsignificance. Public health approaches accept that behavioral in-terventions tend to have positive, but modest effect sizes and thatoutcomes are multidetermined. Instead of seeking to maximizetreatment benefits for individuals (“clinical significance”), publichealth approaches seek to reach more of the affected populationwith beneficial services. Because population impact is a functionof intervention effect size and the reach of the intervention into thepopulation in need (Abrams & Emmons, 1997), a less powerfulintervention that is used by many can have greater impact onpopulation health than a more powerful and costly interventionreceived by relatively few.

These are not new ideas, but they have been slow to influencethe applied research agenda in rehabilitation psychology and otherareas of psychological practice. A population perspective high-lights the need for research to increase access to services and tostudy treatment utilization as a dependent variable (as in healtheconomics) and consumer preferences for services, in addition tostudying treatment as an independent variable (the efficacyagenda). Consumer input from persons with disabilities is under-developed in the design and study of rehabilitation services (Olkin& Pledger, 2003) and deserves greater attention.

Revisiting the Purpose of RCTs: Establishing Causality orStandards of Care?

If one entertains, as we have done, that the RCT is not alwaysthe most informative evidence for practice, then interesting newquestions about its scientific utility emerge, which we take up inthis final section. In defending RCTs as the best form of evidencefor effective practice, Hollon (2006) stated, “if you can get a pig tofly, then you do not need a control group (or statistics) to tell youthat you have had an effect. Pigs do not fly of their own accordand, if you can overcome the laws of God and nature, then youhave done something” (p. 98). Those who style themselves to bethe most die-hard of empiricists might disagree and point out thatreplicability of the effect and ruling out alternative causes wouldbe necessary to reach conclusions about the treatment, whichwould best be addressed using controlled research designs. What ismore relevant to the present discussion, however, is that Hollon’sproposition reflects a common view in psychology that the primarypurpose of RCTs is to establish causality.

In the context of much medical research, RCTs frequently do

not have the establishment of causality as their primary purpose(Spring et al., 2005). The rise of RCTs as the gold standard ofevaluation research was based on its use to establish the safety andefficacy of drug therapies as part of the FDA process (Concato etal., 2000). By the time a Phase 1 clinical trial begins, basicresearch to establish the mechanism of a drug’s effect has beenfinished. The issue, then, is whether the causal effect observed inpreclinical animal research will translate into therapeutic efficacyin humans with specific clinical conditions. The research questionsconcern drug dosage and the range of clinical presentations forwhich the drug is effective.

From a medical point of view, the main purpose of RCTs is toestablish standards of care for which health care providers areprofessionally and legally accountable (Reed & Eisman, 2006). Inestablishing practice standards, although teaching a pig to flymight indeed be a noteworthy accomplishment, RCTs would stillbe necessary to establish that the process was safe for the pig,which training techniques and dosages were required to achievethe effect, for what sort of pigs the training worked, and, eventu-ally, the long-term effects of flying on pigs. This has generally notbeen the psychological approach to RCTs. Instead, RCTs are oftenexpected to carry the burden both of demonstrating therapeuticefficacy and of elucidating the mechanism of the effect (e.g., viathe design of a control condition that contains everything but theputative “active ingredient”). Questions about dosing and the rangeof clinical applicability are addressed in a very general sense ineffectiveness studies, but they are rarely the focus of systematicinquiry.

Constraint-induced movement therapy (CIMT) is an example inrehabilitation psychology in which use of RCTs to establish stan-dards of care is highly appropriate, but only as one phase in abroader program of research. CIMT was based on neuroscienceresearch with adult monkeys with surgical somatosensory deaffer-entiation that demonstrated neural plasticity beyond what had beenthought possible in adult mammals (Taub, 1980; Taub, Uswatte, &Elbert, 2002). Taub’s animal model on which CIMT is based canbe seen as parallel, if not superior, to the information that would beavailable about the potential therapeutic mechanism of a new drugprior to Phase 1 FDA testing. On this basis, CIMT was developedas a new physical rehabilitation program for persons with extrem-ity mobility problems due to central nervous system damage fromcauses such as brain injury or stroke (Taub, Uswatte, Mark, &Morris, 2006). CIMT has three components: (a) daily repetitive,task-oriented training of the impaired extremity or function usingshaping principles for 2–3 weeks; (b) constraining patients so theyhave to use the impaired extremity or function during wakinghours; and (c) a “package” of behavioral methods for transferringtherapeutic gains to the real world.

CIMT has been found to substantially improve upper limbmobility deficits in patients with mild to moderately severe chronicstroke (Taub et al., 1993). At the time, conventional wisdom wasthat stroke patients would show little improvement in motor func-tion 6 months after stroke onset, and certainly not after 1 year,regardless of the therapy administered. Remaining motor deficitswere considered permanent, and many patients are still told that.The mean chronicity of Taub et al.’s (1993) patients was over 4years. In rehabilitation, therefore, this is the logical equivalent ofteaching a pig to fly.

Evidence for the efficacy of CIMT continues to grow, including


RCTs and replications by other researchers (Taub, Uswatte, Kinget al., 2006; Wolf et al., 2006). Research on CIMT over the nextdecade will likely continue to rely on RCTs, but not becausecausality is at issue. Clear evidence of causality comes from thelarge body of preclinical animal research and is buttressed byextensive clinical experience with chronic stroke and other condi-tions that are the focus of CIMT. The primary goal, then, of theRCT research will be to establish standards of care (see Taub &Uswatte, 2006), addressing such questions as: What is the neces-sary duration and intensity of CIMT? Are all three componentsnecessary? With what range of patients and conditions is CIMTeffective? Additional questions that cannot be addressed by RCTsalone include: How cost-effective is CIMT? What type of trainingis necessary to perform CIMT? What patient characteristics mod-erate the effects of CIMT?

Addressing these questions, especially those concerning cost,are necessary (though perhaps not sufficient) to persuade healthinsurance plans to cover CIMT, which can be seen as an expen-sive, intensive service for a large group of people who previouslyhad been classified as unable to benefit from further treatment(Taub & Uswatte, 2006). Denials of coverage have been ostensiblybased on the CIMT’s “experimental” status, even though it enjoysa stronger evidence base than exists for many other coveredpractices.

Conclusions

As a part of graduate training, psychologists learn researchmethodology on the basis of principles of logical reasoning andcritical thinking. This is a tradition we value enormously. Butwhen the concern is the development of research on clinicalinterventions, there may be considerable liability to the psycho-logical view of RCTs as being devoted to establishing causalityand elucidating mechanisms, which are often better addressed byother research approaches. Although weighing different strategieswith which to investigate a given research question imposes asignificant scholarly burden and risks “evidentiary dilution” in thehands of uncritical researchers, psychologists should recognize,and help others to learn, that not all research questions—and not allforms of treatment—will fit the RCT model.

The field of rehabilitation psychology seems particularly illserved by overly restrictive methodologies. As discussed earlier,this is due to the field’s developmental status and the complexnature of questions the field must address to be of value to patientsin rehabilitation care, to individuals with disabilities, and to soci-ety. Clearly, the RCT has a central role in outcome research relatedto treatments in which rehabilitation psychologists participate.However, we believe that it is imperative that rehabilitation psy-chologists resist following the dictates of a narrow construction ofthe EBP movement that has elevated research using the RCT andthe standardized treatment manual above all else (Tucker & Roth,2006). In this respect, we concur with a meta-analysis of method-ologically diverse studies of medical treatments (Concato et al.,2000) that concluded: “The popular belief that only randomized,controlled trials produce trustworthy results and that all observa-tional studies are misleading does a disservice to patient care,clinical investigation, and the education of health care profession-als” (p. 1892).

We contend that positive developments will come from an

informed and critical evidentiary pluralism and that this will helppromote the development of an interdisciplinary research base forrehabilitation that includes psychology in a leadership role. Theefficacy-effectiveness distinction highlights how different empiri-cal questions are better addressed using different methods andhow, when considered together, they can expand the scope andimpact of health-related research. If both are components of thefield’s research portfolio, then the trade-offs between internal andexternal validity are more balanced in the body of work. Indiscussing field and quasi-experiments that did and did not involverandomization, Shadish (2002) observed that while “. . . nonran-domized experiments continue to be . . . tarred with a general brushof inferiority” (p. 3), . . . “across a program of research, all validitytypes are a high priority, and by the end of a program of research,each should have had its turn in the spotlight” (p. 10).

Additional research questions require attention if the field is touse findings concerning contextual, psychological, and treatmentinfluences on functioning and disability over time. We have dis-cussed how this suggests an urgent need for theoretical develop-ment in rehabilitation psychology and the development of validindicators of participation as a key outcome in rehabilitation andbroader efforts to improve the lives of people with disabilities.Wider adoption of a multivariate longitudinal approach will havea salutary effect in this regard. It will help to close the gap betweena public health oriented disability model and the narrower clinicalrehabilitation model by enhancing the ability to conceptualizetreatment as one of several variable classes that influence func-tioning, disability, and participation over time.

There will be costs to this endeavor (Tucker & Roth, 2006).Investigators may need to upgrade their statistical skills to takeadvantage of modern data analytic capabilities and to communi-cate effectively about these approaches. Some approaches willrequire larger samples, longer studies, more interdisciplinary col-laboration, and more multicenter investigations in order to accruesufficient numbers of participants, given the relatively low baserates of the conditions of concern to rehabilitation psychology,even when functionally defined. We believe the payoffs from thesedevelopments will be substantial. When combined with the exist-ing achievements of rehabilitation psychology, they can help po-sition the field to strengthen linkages between science and practicethrough conceptually guided research on the multidetermined andlongitudinal process of adaptation and change among personsliving with disabilities.

References

Abrams, D. B., & Emmons, K. M. (1997). Health behavior and healtheducation: The past, present, and future. In K. Glanz, B. K, Rimer, &F. M. Lewis (Eds.), Health behavior and health education: Theory,research, and practice (pp. 453–478). San Francisco: Jossey-Bass.

Agency for Health Care Policy and Research. (1993). Clinical PracticeGuideline Development (AHCPR Publication No. 93–0023). Rockville,MD: U.S. Department of Health and Human Services.

Americans With Disabilities Act. (1990). Pub. L. No. 101–336, 42U.S.C.A. § 12101. Retrieved September 11, 2007, from http://www.ad-a.gov/pubs/ada.htm

Andersen, R. M. (1995). Revisiting the behavioral model and access tomedical care: Does it matter? Journal of Health and Social Behavior, 36,1–10.

Atchison, T. B., Sander, A. M., Struchen, M. A., High, W. M., Roebuck,

290 TUCKER AND REED

T. M., Contant, C. F., et al. (2004). Relationship between neuropsycho-logical test performance and productivity at 1 year following traumaticbrain injury. Clinical Neuropsychology, 18, 249–265.

Benson, K., & Hartz, A. J. (2000). A comparison of observational studiesand randomized, controlled clinical trials. New England Journal ofMedicine, 342, 1878–1886.

Blacker, C. V. R., & Mortimore, C. (1996). Randomized controlled trialsand naturalistic data: Time for a change? Human Psychopharmacology,11, 353–363.

Bombardier, C. H., Richards, J. S., Krause, J. S., Tulsky, D., & Tate, D. G.(2004). Symptoms of major depression in people with spinal cord injury:Implications for screening. Archives of Physical Medicine and Rehabil-itation, 85, 1749–1756.

Borckardt, J. J., Nash, M. R., Murphy, M. D., Moore, M., Shaw, D., &O’Neil, P. (2008). Clinical practice as a natural laboratory for psycho-therapy research: A guide to case-based time-series analysis. AmericanPsychologist, 63, 77–95.

Boutron, I., Moher, D., Altman, D. G., Schulz, K. F., & Ravaud, P. (2008).Extending the CONSORT statement to randomized trials of nonphar-macologic treatment: Explanation and elaboration. Annals of InternalMedicine, 148, 295–309.

Brownson, R. C., Baker, E. A., Leet, T. L., & Gillespie, K. N. (2003).Evidence-based public health. New York: Oxford University Press.

Bruyere, S. M., Erickson, W. A., & VanLoy, S. (2004). Comparative studyof workplace policy and practices contributing to disability nondiscrim-ination. Rehabilitation Psychology, 49, 28–38.

Cardol, M., De Jong, B. A., & Ward, C. D. (2002). On autonomy andparticipation in rehabilitation. Disability and Rehabilitation, 24, 970–974.

Carpinello, S. E., Rosenberg, L., Stone, J., Schwager, M., & Felton, C. J.(2002). New York State’s campaign to implement evidence-based prac-tices for people with serious mental disorders. Psychiatric Services, 53,153–155.

Chambless, D. L., Sanderson, W. C., Shoham, V., Johnson, S. B., Pope,K. S., Rits-Christoph, P., et al. (1996). An update on empirically vali-dated therapies. The Clinical Psychologist, 49, 5–18.

Chamie, M. (1995). What does morbidity have to do with disability?Disability and Rehabilitation, 17, 323–337.

Chorpita, B. F., Yim, L. M., Donkervoet, J. C., Arensdorf, A., Amundsen,M. J., McGee, C., et al. (2002). Toward large-scale implementation ofempirically supported treatments for children: A review and observa-tions by the Hawaii empirical basis to services task force. ClinicalPsychology: Science and Practice, 9, 165–190.

Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlledtrials, observational studies, and the hierarchy of research designs. NewEngland Journal of Medicine, 342, 1887–1892.

Corrigan, P. W., Mueser, K., Bond, G. R., Drake, R. E., & Solomon, P.(Eds.). (2007). Principles and practice of psychiatric rehabilitation: Anempirical approach. New York: Guilford Press.

Department of Health and Human Services. (2000). Healthy people 2010.Washington, DC: Author.

Dunn, D. S., & Dougherty, S. B. (2005). Prospects for a positive psychol-ogy or rehabilitation. Rehabilitation Psychology, 50, 305–311.

Elkin, I., Shea, M. T., Watkins, J. T., Imber, S. D., Sotsky, S. M., Collins,J. F., et al. (1989). National Institute of Mental Health Treatment ofDepression Collaborative Research Program: General effectiveness oftreatments. Archives of General Psychiatry, 46, 971–982.

Elliott, T. (1994). A place for theory in the study of psychological adjust-ment among persons with neuromuscular disorders: A reply to Livnehand Antonak. Journal of Social Behavior and Personality, 9, 231–236.

Elliott, T., & Frank, R. G. (1996). Depression following spinal cord injury.Archives of Physical Medicine and Rehabilitation, 77, 816–823.

Elliott, T. R., & Frank, R. G. (2000). Afterward: Drawing new horizons. InR. G. Frank & T. Elliott (Eds.), Handbook of rehabilitation psychology(pp. 645–653). Washington, DC: American Psychological Association.

Elliott, T., & Jackson, W. T. (2005). Cognitive-behavioral therapy inrehabilitation psychology. In A. Freeman (Editor-in-Chief). Encyclope-dia of cognitive behavior therapy (pp. 324–327). New York: SpringerScience and Business Media.

Elliott, T., Kurylo, M., & Rivera, P. (2002). Positive growth following anacquired physical disability. In. C. R. Snyder & S. Lopez (Eds.), Hand-book of positive psychology (pp. 687–699). New York: Oxford Univer-sity Press.

Elliott, T., & Leung, P. (2005). Vocational rehabilitation: History andpractice. In W. B. Walsh & M. Savickas (Eds.), Handbook of vocationalpsychology (3rd ed., pp. 319–343). New York: Erlbaum.

Elliott, T. R., & Warren, A. M. (2007). Why psychology is important inrehabilitation. In P. Kennedy (Ed.), Psychological management of physicaldisabilities: A practitioner’s guide (pp. 16–39). London: Routledge Press.

Fordyce, W. (1976). Behavioral methods for chronic pain and illness. St.Louis, MO: Mosby.

Fuhrer, M. J. (2003). Overview of clinical trials in medical rehabilitation:Impetuses, challenges, and needed future directions. American Journalof Physical Medicine and Rehabilitation, 82(Suppl.), S8–S15.

Gatchel, R. J. (2005). Clinical essentials of pain management. Washington,DC: American Psychological Association.

Gatchel, R. J., Polatin, P. B., & Mayer, T. G. (1995). The dominant role ofpsychosocial risk factors in the development of chronic low back paindisability. Spine, 20, 2702–2709.

Goodheart, C. D., Kazdin, A. E., & Sternberg, R. J. (Eds.). (2006).Evidence-based psychotherapy: Where practice and research meet.Washington, DC: American Psychological Association.

Heinemann, A. W. (2005). Putting outcome measurement in context: Arehabilitation psychology perspective. Rehabilitation Psychology, 50,6–14.

Hendershot, G. E., & Lollar, D. (2007). The International Classification ofFunctioning: Facilitating integrative healthcare. In I. A. Serlin & M. A.DiCowden (Eds.), Whole Person Healthcare—Volume 1: Humanizinghealthcare (pp. 233–248). Westport, CT: Praeger.

Hollon, S. D. (2006). What qualifies as evidence on which to judgeeffective practice? Randomized clinical trials. In J. C. Norcross, L. E.Beutler, & R. F. Levant (Eds.), Evidence-based practices in mentalhealth (pp. 96–105). Washington, DC: American Psychological Asso-ciation.

Hollon, S. D., DeRubeis, R. J., Shelton, R. C., Amsterdam, J. D., Salomon,R. M., O’Reardon, J. P., et al. (2005). Prevention of relapse followingcognitive therapy vs. medications in moderate to severe depression.Archives of General Psychiatry, 62, 417–422.

Hollon, S. D., Jarrett, R. B., Nierenberg, A. A., Thase, M. E., Trivedi, M.,& Rush, A. J. (2005). Psychotherapy and medication in the treatment ofadult and geriatric depression: Which monotherapy or combined treat-ment? Journal of Clinical Psychiatry, 66, 455–468.

Hollon, S. D., Thase, M. E., & Markowitz, J. C. (2002). Treatment andprevention of depression. Psychological Science in the Public Interest, 3,39–77.

Horn, S. D. (2006). Performance measures and clinical outcomes. Journalof the American Medical Association, 296, 2731–2732.

Horn, S. D., DeJong, G., Ryser, D. K., Veazie, P. J., & Teraoka, J. (2005).Another look at observational studies in rehabilitation research: Goingbeyond the Holy Grail of the randomized controlled trial. Archives ofPhysical Medicine and Rehabilitation, 86(Suppl. 2), S8–S15.

Horn, S. D., & Gassaway, J. (2007). Practice-based evidence study designfor comparative effectiveness research. Medical Care, 45(Suppl. 2),S50–S57.

Hulley, S. B., & Cummings, S. R. (1988). Designing clinical research.Baltimore: Williams & Wilkins.

Institute of Medicine. (2001). Crossing the quality chasm: A new healthsystem for the 21st century. Washington, DC: National Academies Press.

Institute of Medicine. (2005). Improving the quality of health care for


mental health and substance use conditions. Washington, DC: NationalAcademies Press.

Institute of Medicine. (2007). The future of disability in America. Wash-ington, DC: National Academies Press.

Jenicek, M. (2003). Foundations of evidence-based medicine. New York:Parthenon Publishing Group.

Judd, C. M., Kenny, D. A., & McClelland, G. H. (2001). Estimating andtesting mediation and moderation in within-subjects designs. Psycholog-ical Methods, 6, 115–134.

Kazdin, A. E. (1982). Single-case research designs: Methods for clinicaland applied settings. New York: Oxford University Press.

Kazdin, A. E. (2007). Mediators and mechanisms of change in psycho-therapy research. Annual Review of Clinical Psychology, 3, 1–27.

Kraemer, H. C., Stice, E., Kazdin, A., Offord, D., & Kupfer, D. (2001).How do risk factors work together? Mediators, moderators, and inde-pendent, overlapping, and proxy risk factors. American Journal of Psy-chiatry, 158, 848–856.

Kuhn, T. S. (1970). The structure of scientific revolutions (2nd ed.).Chicago: University of Chicago Press.

Langhorne, P., & Duncan, P. (2001). Does the organization of postacutestroke care really matter? Stroke, 32, 268–274.

Leichsenring, F. (2004). Randomized controlled versus naturalistic studies:A new research agenda. Bulletin of the Menninger Clinic, 68, 137–151.

Linden, W., Stossel, C., & Maurice, J. (1996). Psychosocial interventionsfor patients with coronary artery disease: A meta-analysis. Archives ofInternal Medicine, 156, 745–752.

Lollar, D. J. (1997). Secondary conditions among adolescents with spinabifida. European Journal of Pediatric Surgery, 7(Suppl. 1), 55.

Lollar, D. (1999). Clinical dimensions of secondary conditions. In R. J.Simeonsson & L. N. McDivitt (Eds.), Issues in disability and health: Theroles of secondary conditions and quality of life (pp. 41–50). ChapelHill: University of North Carolina, FPG Child Development Center.

Lollar, D. J., & Crews, J. E. (2003). Redefining the role of public health indisability. Annual Review of Public Health, 24, 195–208.

Lollar, D., & Simeonsson, R. (2005). Diagnosis to function: Classificationfor children and youths. Journal of Developmental Behavior and Pedi-atrics, 26, 323–330.

MacKenzie, B. B. (1977). Behaviourism and the limits of scientific method.Atlantic Highland, NJ: Humanities Press.

Marson, D. C., Dreer, L. E., Krzywanski, S., Huthwaite, J. S., Devivo,M. J., & Novack, T. A. (2005). Impairment and partial recovery ofmedical decision-making capacity in traumatic brain injury: A 6-monthlongitudinal study. Archives of Physical Medicine and Rehabilitation,86, 889–895.

McMurray, J. J., & O’Meara, E. (2004). Treatment of heart failure withspironolactone: Trial and tribulations. New England Journal of Medi-cine, 351, 526–528.

Meadows, M. (2002). The FDA’s drug review process: Ensuring drugs aresafe and effective. FDA Consumer (FDA Publication No. 02–3242).

Mechanic, D. (1978). Medical sociology (2nd ed.). New York: Free Press.Mendes de Leon, C. F., Czajkowski, S. M., Freedland, K. E., Bang, H., Powell,

L. H., Wu, C., et al. (2006). The effect of a psychosocial intervention andquality of life after acute myocardial infarction: The Enhancing Recovery inCoronary Heart Disease (ENRICHD) clinical trial. Journal of Cardiopul-monary Rehabilitation and Prevention, 26, 9–13.

Miller, F. G., & Brody, H. (2007). Clinical equipoise and the incoherenceof research ethics. Journal of Medicine and Philosophy, 32, 151–165.

Miller, W. R., & Rollnick, S. (2002). Motivational interviewing: Preparingpeople for change (2nd ed.). New York: Guilford Press.

Moher, D., Schulz, K. F., & Altman, D. (2001). The CONSORT statement:Revised recommendations for improving the quality of reports of par-allel-group randomized trials. Journal of the American Medical Associ-ation, 285, 1987–1991.

Morrisey, M. (1992). Price sensitivity in health care: Implications for

health care policy. Washington, DC: National Federation of IndependentBusiness Foundation.

National Institute for Clinical Excellence. (2004). Depression: Manage-ment of depression in primary and secondary care. Retrieved June 16,2008, from http://www.nice.org.uk/pdf/CG023

Niemeier, J. P., Kreutzer, J. S., & Taylor, L. A. (2005). Acute cognitive andneurobehavioural intervention for individuals with acquired brain injury:Preliminary outcome data. Neuropsychological Rehabilitation, 15, 129–146.

Norcross, J. C. (2002). Psychotherapy relationships that work: Therapists,contributions and responsiveness to patients. New York: Oxford Uni-versity Press.

Norcross, J. C., Beutler, L. E., & Levant, R. F. (Eds.). (2006). Evidence-based practices in mental health: Debate and dialogue on fundamentalquestions. Washington, DC: American Psychological Association.

Olkin, R. (2002). Making research accessible to participants with disabil-ities. Journal of Multicultural Counseling & Development, 8, 130–137.

Olkin, R., & Pledger, C. (2003). Can disability studies and psychology joinhands? American Psychologist, 58, 296–304.

Omenn, G. S., Goodman, G. E., Thomquist, M. D., Balmes, J., Cullen,M. R., Glass, A., et al. (1996). Effects of a combination of beta-caroteneand vitamin A on lung cancer and cardiovascular disease. New EnglandJournal of Medicine, 334, 1150–1155.

Pepper, S. C. (1970). World hypotheses. Berkeley: University of CaliforniaPress. (Original work published 1942)

Pitt, B., Zannad, F., Remme, W. J., Cody, R., Castaigne, A., Perez, A., etal. (1999). The effect of spironolactone on morbidity and mortality inpatients with severe heart failure. New England Journal of Medicine,341, 709–717.

Pope, A. M., & Tarlov, A. R. (Eds.). (1991). Disability in America: Towarda national agenda for prevention. Washington, DC: National AcademiesPress.

Public Health Information & Data Tutorial. (2006). Hierarchy of evidence.Retrieved February 15, 2008, from http://phpartners.org/tutorial/04-ebph/2-keyConcepts/4.2.7.html

Purtilo, R. B., & Haddad, A. M. (2007). Health professional and patientinteraction (7th ed.). Philadelphia: W. B. Saunders.

Ravesloot, C., Seekins, T., & Walsh, J. (1997). A structural analysis ofsecondary conditions of primary physical disabilities. RehabilitationPsychology, 42, 3–16.

Reed, G. M. (2006). What qualifies as evidence of effective practice:Clinical expertise. In J. C. Norcross, L. Beutler, & R. F. Levant (Eds.),Evidence-based practices in mental health: Debate and dialogue on thefundamental questions (pp. 13–23). Washington, DC: American Psycho-logical Association.

Reed, G. M., & Eisman, E. (2006). Uses and misuses of evidence: Man-aged care, treatment guidelines, and outcomes measurement in profes-sional practice. In C. D. Goodheart, A. E. Kazdin, & R. J. Sternberg(Eds.), Evidence-based psychotherapy: Where research and meet (pp.13–25). Washington, DC: American Psychological Association.

Reed, G. M., Lux, J. B., Bufka, L. F., Trask, C., Peterson, D. B., Stark, S.,et al. (2005). Operationalizing the International Classification of Func-tioning, Disability and Health (ICF) in clinical settings. RehabilitationPsychology, 50, 122–131.

Rosen, L., Manor, O., Engelhard, E., & Zucker, D. (2006). In defense ofthe randomized controlled trial for health promotion research. AmericanJournal of Public Health, 96, 1181–1186.

Rosenberg, L., Robert, R., Thomas, C., Holzer, C., Blakeney, P., & Meyer,W. (2006). Assessing potential suicide risk of young adults burned aschildren. Journal of Burn Care and Rehabilitation, 27, 779–785.

Rosenberg, M., Plakeney, P., Robert, R., Thomas, C., Holzer, C., & Meyer,W. (2006). Quality of life of young adults who survived pediatric burns.Journal of Burn Care and Rehabilitation, 27, 773–778.

Sackett, D. L., Strauss, S. E., Richardson, W. S., Rosenberg, W., &

292 TUCKER AND REED

Haynes, R. B. (2000). Evidence based medicine: How to practice andteach EBM (2nd ed.). London: Churchill Livingstone.

Sacks, H., Chalmers, I., & Smith, H., Jr. (1982). Randomized versushistorical controls for clinical trials. American Journal of Medicine, 72,233–240.

Scherer, M. J., Coombs, F. K., Merbitz, N. H., & Merbitz, C. T. (2006).Policy issues in evaluating and selecting assistive technology and otherresources for persons with disabilities. In K. J. Hagglund & A. W.Heinemann (Eds.), Handbook of applied disability and rehabilitationresearch (pp. 215–242). New York: Springer.

Seligman, M. E. P. (1995). The effectiveness of psychotherapy: TheConsumer Reports study. American Psychologist, 50, 965–974.

Shadish, W. R. (2002). Revisiting field experimentation: Field notes for thefuture. Psychological Methods, 7, 3–18.

Shadish, W. R., Navarro, A. M., Matt, G. E., & Phillips, G. (2000). Theeffects of psychological therapies under clinically representative condi-tions: A meta-analysis. Psychological Bulletin, 126, 512–529.

Shrout, P. E., & Bolger, N. (2002). Mediation in experimental and non-experimental studies: New procedures and recommendations. Psycho-logical Methods, 4, 422–445.

Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis:Modeling change and event occurrence. New York: Oxford UniversityPress.

Spring, B., Pagoto, S., Kaufmann, P. G., Whitlock, E. P., Glasgow, R. E.,Smith, T. W., et al. (2005). Invitation to a dialogue between researchersand clinicians about evidence-based behavioral medicine. Annals ofBehavioral Medicine, 30, 125–137.

Tanenbaum, S. J. (2005). Evidence-based practice as mental health policy:Three controversies and a caveat. Health Affairs, 24, 163–173.

Tate, D. G., Findley, T., Jr., Dijkers, M., Nobunaga, A. I., & Karunas, R. B.(1999). Randomized clinical trials in medical rehabilitation research. Amer-ican Journal of Physical Medicine and Rehabilitation, 78, 486–499.

Tate, D. G., Forchheimer, M., Krause, J. S., Meade, M., & Bombardier,C. H. (2004). Patterns of alcohol and substance use and abuse in personswith spinal cord injury: Risk factors and correlates. Archives of PhysicalMedicine and Rehabilitation, 85, 1837–1847.

Tate, D. G., Kalpakjian, C. Z., & Forchheimer, M. (2002). Quality of lifeissues in individuals with spinal cord injury. Archives of PhysicalMedicine and Rehabilitation, 83(Suppl. 2), S18–S25.

Taub, E. (1980). Somatosensory deafferentation research with monkeys:Implications for rehabilitation medicine. In L. P. Ince (Ed.), Behavioralpsychology in rehabilitation medicine: Clinical applications (pp. 371–401). New York: Williams & Wilkins.

Taub, E., Miller, N. E., Novack, T. A., Cook, E. W., Fleming, W. C.,Nepomuceno, C. S., et al. (1993). Technique to improve chronic motordeficit after stroke. Archives of Physical Medicine and Rehabilitation,74, 347–354.

Taub, E., & Uswatte, G. (2006). Constraint-Induced Movement therapy:Answers and questions after two decades of research. NeuroRehabilita-tion, 21, 93–95.

Taub, E., Uswatte, G., & Elbert, T. (2002). New treatments in neuroreha-bilitation founded on basic research. Nature Reviews Neuroscience, 83,228–236.

Taub, E., Uswatte, G., King, D. K., Morris, D., Crago, J., & Chatterjee, A.(2006). A placebo-controlled trial of Constraint-Induced Movementtherapy for upper extremity after stroke. Stroke, 37, 1045–1049.

Taub, E., Uswatte, G., Mark, V. W., & Morris, D. M. (2006). The learnednonuse phenomenon: Implications for rehabilitation. Europa Medico-physica, 42, 241–255.

Taylor, S. E. (1983). Adjustment to threatening events: A theory ofcognitive adaptation. American Psychologist, 38, 1161–1173.

Taylor, S. E., & Brown, J. D. (1988). Illusion and well-being: A socialpsychological perspective on mental health. Psychological Bulletin, 110,193–210.

Taylor, S. E., Kemeny, M. E., Aspinwalk, L. G., Schneider, S. C., Rodri-guez, R., & Herbert, M. (1992). Optimism, coping, psychological dis-tress, and high-risk sexual behavior among men at risk for AIDS.Journal of Personality and Social Psychology, 63, 460–473.

Taylor, S. E., Kemeny, M. E., Reed, G. M., Bower, J. E., & Gruenewald,T. L. (2000). Psychological resources, positive illusions, and health.American Psychologist, 55, 99–109.

Tucker, J. A. (1999). Changing addictive behavior: Historical and contem-porary perspectives. In J. A. Tucker, D. M. Donovan, & G. A. Marlatt(Eds.), Changing addictive behavior: Bridging clinical and public healthperspectives (pp. 3–44). New York: Guilford Press.

Tucker, J. A., & King, M. P. (1999). Resolving alcohol and drug problems:Influences on addictive behavior change and help-seeking processes. InJ. A. Tucker, D. M. Donovan, & G. A. Marlatt (Eds.), Changingaddictive behavior: Bridging clinical and public health perspectives (pp.97–126). New York: Guilford Press.

Tucker, J. A., & Roth, D. L. (2006). Extending the evidence hierarchy toenhance evidence-based practice for substance use disorders. Addiction,101, 918–932.

Van Brakel, W. H., Anderson, A. M., Mutatkar, R. K., Bakirtzief, Z.,Nicholls, P. G., Raju, M. S., & Das-Pattanayak, R. K. (2006). TheParticipation Scale: Measuring a key concept in public health. Disabilityand Rehabilitation, 28, 193–203.

Victoria, C. G., & Habicht, J.-P., & Bryce, J. (2004). Evidence-basedpublic health: Moving beyond randomized trials. American Journal ofPublic Health, 94, 400–405.

Walker, W. C., Marwitz, J. H., Kreutzer, J. S., Hart, T., & Novack, T. A.(2006). Occupational categories and return to work after traumatic braininjury: A multicenter study. Archives of Physical Medicine and Reha-bilitation, 87, 1576–1582.

Wassertheil-Smoller, S., Hendrix, S. L., Limacher, M., Heiss, G., Kooper-berg, C., Baird, A., et al. (2003). Effect of estrogen plus progestin onstroke in postmenopausal women: The Women’s Health Initiative: Arandomized trial. Journal of the American Medical Association, 289,2673–2684.

Weinstein, N. D. (1984). Why it won’t happen to me: Perceptions of riskfactors and susceptibility. Health Psychology, 3, 431–457.

Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004). Empiricalstatus of empirically supported psychotherapies: Assumptions, findings,and reporting in controlled clinical trials. Psychological Bulletin, 130,631–663.

Whiteneck, G., Forchheimer, M. B., & Krause, J. S. (2007). Quality of lifeand health in the last years after spinal cord injury. Topics in Spinal CordInjury Rehabilitation, 12, 77–90.

Whiteneck, G., Meade, M. A., Dijkers, M., Tate, D. G., Bushnik, T., &Forchheimer, M. B. (2004). Environmental factors and their role inparticipation and life satisfaction after spinal cord injury. Archives ofPhysical Medicine and Rehabilitation, 85, 1793–1803.

Wolf, S. L., Winstein, C. J., Miller, J. P., Taub, E., Uswatte, G., Morris, D.,et al. (2006). Effect of Contraint-Induced Movement Therapy on upperextremity function 3 to 9 months after stroke: The EXCITE randomizedclinical trial. Journal of the American Medical Association, 296, 2095–2104.

World Health Organization. (1998). Health promotion evaluation recom-mendations to policy makers. Report of the WHO European WorkingGroup on health promotion evaluation. Geneva, Switzerland: Author.

World Health Organization. (2001). International classification of func-tioning, disability, and health. Geneva, Switzerland: Author.

Wright, G. N. (1980). Total rehabilitation. Boston: Little, Brown.

Received October 9, 2007Revision received May 19, 2008

Accepted May 30, 2008 !


evidentiary pluralism as a strategy for research and evidence-based practice in rehabilitation...

Documents