The Technology of Measurement Feedback Systems

Leonard Bickman, Susan Douglas Kelley, and Michele AthayVanderbilt University

Usual care in the community is far from optimal. Sufficient evidence exists that dropoutrates are significant, treatment is effective for only a small proportion of clients, and thetranslation of evidence-based treatments to the real world is problematic. Technologyhas been shown to be helpful in health care for improving the effectiveness oftreatment. A relatively new technology being used in mental health are measurementfeedback systems (MFSs). MFSs are particularly applicable to couple and familypsychology because of their ability to provide information on the multiple perspectivesinvolved in treatment. The Contextualized Feedback Systems (CFS), developed atVanderbilt University, is used as an example of what can be accomplished with anMFS. The advantages and limitations of this technology are described as well as theanticipated reimbursement requirements that mental health services will need.

Keywords: measurement feedback system, technology, outcome monitoring, mental healthtreatment

Why Clinicians Need Technology

Couples and family therapists face uniquechallenges compared with many other mentalhealth professionals. Whereas many mentalhealth professionals are primarily focused onone individual (the “patient”), couples and fam-ily therapists are focused on the entire familysystem. This means that the therapist must in-tegrate and interpret multiple sources of infor-mation and different points of view from indi-viduals with different idiographic experiences(e.g., symptom patterns, overt behaviors, andcovert personality dynamics) within the contextof their specific social, cultural, and environ-mental factors (Ridley, Tracy, Pruitt-Stephens,Wimsatt, & Beard, 2008). Only when this com-plete clinical picture is appreciated can the ther-apist know how best to treat the family. Yet this

picture is not static, it is ever moving. There-fore, this picture must be continually reassessedso that the therapist can make informed deci-sions about therapeutic goals and appropriatetreatments on an ongoing basis. Thus, the abil-ity of the therapist to obtain this information andthen accurately integrate and interpret it directlyinfluences the resulting clinical decisions andtreatment outcomes. Indeed, as Ridley andShaw-Ridley (2009) stated, “Clinical judgmentis foundational because positive therapeuticoutcomes hinge on establishing reasonablegoals and selecting appropriate treatments, andappropriate treatment selection hinges on soundjudgment and accurate decision making” (p.401).

Despite the ongoing and complex task ofgenerating an accurate clinical picture of a dy-namic family system, the efficacy of couple andfamily psychology (CFP) is well established(Sexton, Datachi, Evans, LaFollette, & Wright,in press). However, CFP is not immune to thechallenges faced by all areas of psychotherapy.These challenges include high rates of prema-ture termination of treatment (i.e., dropouts) andthe gap between research and practice. Thesehave direct influences on the quality and out-comes of clinical care. For example, not only ispremature dropout a costly waste of mentalhealth resources (Garfield, 1994) but outcomesfor clients who dropout early in therapy are

Leonard Bickman, Susan Douglas Kelley, and MicheleAthay, Center for Evaluation and Program Improvement,Vanderbilt University.

This paper was partially supported by a grant from theNational Institute of Mental Health (5 R01 MH087814) toL. B.

Correspondence concerning this article should be ad-dressed to Leonard Bickman, PhD, Center for Evaluationand Program Improvement Peabody #151, 230 AppletonPlace, Nashville, TN 37203-5721. E-mail: leonard.bickman@vanderbilt.edu

Couple and Family Psychology: Research and Practice © 2012 American Psychological Association2012, Vol. 1, No. 4, 274–284 2160-4096/12/$12.00 DOI: 10.1037/a0031022

274

similar to those who never begin therapy (Stark,1992). Dropout rates in individual therapiesrange from 20% to 50% (Swift, Greenberg,Whipple, & Kominiak, 2012; Wierzbicki &Pekarik, 1993), with comparable dropout ratesin couples and family therapy (Masi, Miller, &Olson, 2003).

The gap between research and practice is alsoa well-known challenge for improving the ef-fectiveness of mental health care (Kazdin,2008). This can be demonstrated by the diffi-culty in translating efficacious interventions de-veloped in laboratory settings into real worldpractice (i.e., “usual care” or UC). For example,using data collected from �6,000 adult clientsreceiving UC, treatment was successful withonly 20% of the clients while typical lab-basedstudies showed about three times that level ofeffectiveness (Hansen, Lambert, & Foreman,2002). There is less research investigating thetranslation of interventions from the lab to thefield in CFP. This is often attributed to increas-ing trends of couple and family therapiststoward being eclectic and integrative in theirapproaches (Pinsof & Wynne, 2000). Thus,multiple models or treatment approaches areused in an effort to help clients, rather thanrelying on a single structured approach as oftenused in lab studies. Additionally, large casel-oads and the little time between clinical sessionsto complete required paperwork make it diffi-cult to integrate new interventions into practice.

One such intervention is the use of outcomemonitoring and feedback for all practices asrecommended by the 2006 American Psycho-logical Association (APA) Presidential TaskForce on Evidence-Based Practice. Althoughsome research with couple therapy has shownthe effectiveness of incorporating ongoing as-sessment into clinical work (Anker, Duncan, &Sparks, 2009; Reese, Toland, Slone, & Nor-sworthy, 2010), more research is needed on howto readily incorporate this type of measurementand feedback into clinical work (Teachman etal., 2012). The pressure to do this may increaserapidly given that the incorporation of treatmentprogress instruments is likely to become a re-quirement for reimbursement in the future(Stewart & Chambless, 2008).

Given the complex tasks that couples andfamily therapists face in treating dynamic fam-ily systems, how can we facilitate their deliveryof effective treatment while also addressing the

challenges inherent in the field of psychother-apy that limit the quality and effectiveness ofmental health care? We believe that well de-signed technology has the potential to providethe tools necessary to do both, particularly withregard to outcome monitoring and feedback.

Technology in Mental Health Care

Although numerous definitions exist, tech-nology often refers to tools and machines thatare used to solve real-world problems in aneasier and/or more efficient manner. Althoughthis definition may include simple tools such asa hammer or lever, technology is now increas-ingly associated with scientific knowledge andmachines, in particular computers. Within men-tal health care, technologies include any proto-col, system, treatment, or intervention that isused to prevent, diagnose, monitor, or treatmental health conditions. In other words, a tech-nology is any innovation that improves thequality of mental health care delivery. Thus, thisdefinition includes innovations such as manual-ized treatments as well as advances that useelectronic devices.

Despite being relatively new, the integrationof electronic devices such as computers, cellphones or handheld devices (e.g., PDAs or tab-lets) into mental health services is increasing.For example, e-health or Telemental Health sys-tems use the Internet to provide access to mentalhealth providers from long distances. From thecomputer, clinicians and patients have clinicalsessions where all assessment and treatment canbe administered through the Internet (Chris-tensen & Hickie, 2010; Yuen, Goetter, Herbert,& Forman, 2012; King, Bickman, Shochet, Mc-Dermott, & Bor, 2010). Although this deliveryof services is still emerging in the mental healthfield, there is some evidence of its effectivenessfor improving patient outcomes for severalmental health conditions (e.g., see Germain,Merchand, Bouchard, Guay, & Droui, 2010;Griffiths, Farrer, & Christensen, 2010; Nelson,Barnard, & Cain, 2006; Yuen et al., 2010). Forexample, cell phones and handheld devices arebeing used to enhance the treatment of psychi-atric disorders (Ehrenreich, Righter, Rocke,Dixon, & Himelhoch, 2011; Granholm, Ben-Zeev, Link, Bradshaw, & Holden, 2011). Forexample, Granholm et al. (2011) found thatsending regular text messages to cell phones

275MEASUREMENT FEEDBACK SYSTEMS

provided to individuals with schizophrenia re-sulted in a significant increase in medicationadherence. Similarly, several studies reviewedby Ehrenreich et al. (2011) found increasednumbers of participants who remained abstinentfrom smoking by using cell phones and com-puters to send text messages, voicemail mes-sages, and e-mails. However, there is little re-search with these new technologies in CFP.

The integration of electronic technologiesinto mental health care is relatively recent but isincreasing at a rapid rate. Some may even callthis a new era for mental health services (Chris-tensen & Hickie, 2010; Kazdin, 2012; Ro-theram-Borus, Swendeman, & Chorpita, 2012).Smartphone technology is being applied tomental health in many ways. In their search,Luxton, McCann, Bush, Mishkind, and Reger(2011) found hundreds of unique mobile phoneapplications (i.e., “apps”) in the BlackBerryApp World site (http://appworld.blackberry.com/webstore/) that were specifically associatedwith behavioral health. And this number was onthe rise. These apps were designed for variouspurposes such as clinical assessment, symptommonitoring, psychoeducation, resource location,tracking treatment process, skills training andcommunication with health care providers.

There are a multitude of possibilities for in-tegrating such technology into mental healthcare, and researchers are working to keep upwith these rapid developments to evaluate theireffectiveness. In the next section, we describemeasurement feedback systems (MFSs), a tech-nology that has the ability to improve clinicalcare while also addressing the challenges facedby the field concerning the integration of ongo-ing outcome monitoring and feedback into prac-tice as well as dropout rates.

Measurement Feedback Systems: TheTechnology of Delivering Information

As described by Bickman (2008b), an MFS iscomposed of two components. The first is ameasure or a battery of comprehensive mea-sures that are administered regularly throughouttreatment to collect ongoing information con-cerning the process and progress of treatment.In CFP it is critical that this information iscollected from all participants, including thetherapist. The second component of an MFS isthe presentation of this information to provide

timely and clinically useful feedback to clini-cians. This feedback provides information aboutwhat does and does not seem to be working sothat clinicians can be more responsive to theneeds of the client or family by continuing,discontinuing, or altering treatment plans.MFSs have been used successfully to improveoutcomes in treatment with couples (Morten,Anker, Duncan, & Sparks, 2009), in treatmentwith families (Bickman, Kelley, Breda, De An-drade, & Riemer, 2011), and in general healthcare (Carlier et al., 2012).

In current research, the focus is often on thefirst aspect of the MFS technology, the mea-sures. Questions arise on what to measure, whento measure it, who to include as reporters, andso forth These are indeed important and neces-sary components of a MFS—the resulting feed-back is obviously limited by the type and va-lidity of the measures that are used to producethe feedback. In addition to having acceptablepsychometric properties, measures used forfeedback throughout treatment should also bebrief, sensitive to change, and clinically rele-vant (Kelley & Bickman, 2009). These are thebasic measurement properties for any high qual-ity implementation of outcome monitoring in-tended to inform mental health treatment thatincorporates frequent administration of mea-sures throughout treatment. Although therehave been important advances in measurementdevelopment and testing, measurement technol-ogy (psychometrics) has been with us for �100years.

Despite the importance of the measures in theMFS, it is the second component focused on thedelivery of feedback that has the potential toaddress the challenges faced in the mentalhealth care field in its quest to improve clinicalservices. An MFS uses electronic technology totransform measurement data into informationthat can influence treatment as it occurs. Inother words, information electronically enteredinto an MFS generates automatic feedback thatis available to the clinician for immediate use.This immediate information can then be incor-porated into the treatment planning for the verynext clinical session or even for use in thecurrent session.

There are two key benefits of an electronictechnology over a nontechnological solution (e.g., paper and pencil) for promoting the integra-tion of ongoing outcome monitoring and feed-

276 BICKMAN, KELLEY, AND ATHAY

back into practice. First, technology providesease of access to collect and use data in thetreatment process. The use of computers andhandheld computing devices (e.g., tablets) al-lows for real-time feedback with the electronicadministration of measures, automated calcula-tion of scores, interpretation of results, and im-mediate presentation of feedback. Second, tech-nology makes it possible to achieve integrationof data-informed decision-making throughoutall aspects of clinical workflow. We call this the“golden thread,” a concept based on best prac-tices in clinical documentation systems thatfacilitates linking individual client needs, treat-ment goals, therapeutic interventions, and mon-itoring of client responses (Lloyd, 2011). AnMFS makes it possible to spread the “goldenthread” of clinical information through all lev-els of an organization with real time usableinformation immediately available, from aggre-gated data for executive planning at the agencyleadership level to specific session data thatinforms what the clinician addresses in the nextencounter with a client. Importantly, all the dataare accessible in one location. However, thisdoes not mean that all information is availableto all users. Instead, an MFS that uses a permis-sion-based login allows users to access actionsand data tailored to their role in the clinicalworkflow, restricting or allowing access tofunctional or administrative aspects as well aslevels of aggregation of data. For example,clinic directors could view data aggregatedacross treatment programs or service settings;supervisors could view data aggregated for cli-nicians that they supervise; clinicians couldview data for only those clients they treat; andclients could have access only to functions re-lated to administration of their own accountsand completion of measures.

The primary purpose of an MFS is to providefeedback that is used to inform clinical practice.In other words, it is actionable, able to be usedto influence clinician behavior. A general ruleof thumb for any type of feedback to be action-able is that it must be considered by the recip-ient to be relevant to the issue at hand anddelivered in a way that incorporates an under-standing of how cognitive processes and learn-ing styles influence responses to feedback (Can-non & Witherspoon, 2005). In addition, emerg-ing guidelines for the design and evaluation ofmental health technologies emphasize a user-

centered and evolving design process (Doherty,Coyle, & Matthews, 2010). User-centered de-sign involves not only engaging end users (e.g.,clinicians, clients, agency directors) in the de-velopment of the functions and interface of theMFS but also incorporating theoretical princi-ples relevant to the use of information to pro-mote clinician behavior change (Riemer, Rosof-Williams, & Bickman, 2005; Sapyta, Riemer, &Bickman, 2005). An evolving design processentails the application of continuous quality im-provement to incorporate end user experienceinto ongoing development of the electronicMFS itself and accompanying support for op-eration and implementation processes.

In sum, MFSs aim to improve clinical prac-tice by using valid and reliable measures toprovide immediate feedback for cliniciansconcerning key process indicators and mea-sures that reflect progress in treatment. Anadditional advantage of an MFS is the abilityto aggregate information instantly for clinicalsupervisors and agency leadership for in-formed decision-making. However, for anMFS to be effective, feedback must be action-able and the system must be user-centered andadopt an evolving design process such that itis able to meet the different and changingneeds of organizations and users. As is de-scribed in the next section, these principleswere considered when the authors developedan MFS called Contextualized Feedback Sys-tems (CFS) over the last decade.

Description of Contextualized FeedbackSystems (CFS)

CFS is a quality improvement tool that assistsclinicians and supervisors in assessing changeas a client progresses. It was designed to en-hance clinical effectiveness and improve orga-nizational efficiency by providing feedback toinform clinical sessions, supervision, programplanning, and professional development. CFS isa web-based Software as a Service (SaaS) ap-plication that was specifically designed to beeasy to operate and provide new information tothose who treat individuals with behavioralhealth problems. Thus, CFS seamlessly inte-grates real-time clinical feedback about indica-tors of treatment progress (e.g., symptoms andfunctioning, life satisfaction) and process (e.g.,motivation for treatment, therapeutic alliance)

277MEASUREMENT FEEDBACK SYSTEMS

as reported by the clinician, client, and otherrespondents (e.g., caregivers). These indicatorsare acquired through the completion of mea-sures included in the Peabody Treatment Prog-ress Battery (PTPB). The PTPB is a compre-hensive set of brief clinical measures that havedemonstrated strong psychometric properties inlarge samples of youth, and their respectivecaregivers and clinicians, receiving mentalhealth treatment (Bickman et al., 2007, 2010;Bickman & Athay, 2012).

CFS has undergone �10 years of researchand development and can be used to support thepromotion of evidence-based practice in com-munity settings. A hallmark of CFS is its flex-ibility that allows ongoing tracking of any rel-evant data for individual clients and aggregationof information across units and time. Further-more, measures and resulting feedback can becontextualized to fit the quality improvementneeds of any organization with varied clientpopulations and/or treatment models. In addi-tion, CFS provides accountability via imple-mentation feedback with performance bench-marks and a rapid-response data system focusedon clinicians and programs.

Evidence-Based Effectiveness

CFS is the only MFS with demonstrated ef-fectiveness in youth mental health settings(Bickman et al., 2011). In a large randomizedtrial, results showed that youth whose clinicianswere randomly assigned to receive feedbackimproved significantly faster according to dataprovided from the clinician, youth, and care-giver respondents. Furthermore, there was adose–response relationship between feedbackviewing and clinical outcomes. The more re-ports viewed by the clinician, the faster clinicalimprovement the youth made, based on datafrom the clinician, the youth, and the caregiver.

Theory-Based Development

Research clearly indicates that progress intreatment is multidimensional, and that feed-back information is needed not only on tradi-tional outcomes such as a reduction in symp-toms, but also on the treatment processes thatmediate such outcomes. As such, CFS is basedon a theory of change and grounded in psycho-logical and organizational research. The feed-back

approach applies several behavior change theo-ries, including goal theory, cognitive disso-nance theory, attribution theory, and strength-based self-efficacy theory (Riemer et al., 2005;Sapyta et al., 2005). In addition, the feedbackitself is informed by research on common fac-tors in psychotherapeutic treatment (Kelley,Bickman, & Norwood, 2010). Common factorsare related to the impact of client and cliniciancharacteristics, relationship factors (e.g., thera-peutic alliance), and other ingredients of thetherapeutic process that are common to anypsychotherapy, regardless of treatment modalityor model.

User-Centered and Evolving Design Process

CFS was developed as a collaborative effortbetween mental health services researchers andpractitioners (Riemer, Kelley, Casey, & Taylor-Haynes, 2012). The research–practice partner-ship was essential to the development of themeasures, the “look and feel” of the computeruser interface, the determination of needed ad-ministrative functions and controls, and the de-sign of the feedback reports. Software applica-tion development is a complex and expensiveongoing process, with the most recent version(3.0) representing a substantial revision that in-tegrates user feedback to improve incorporationin the therapist workflow. In addition, the con-tent of support materials and processes hasevolved over time to include preadoption con-sultation, pre- and postimplementation support,online operations support, and ongoing qualityimprovement support. This was done to ensureCFS could have a sustainable effect throughoutthe organization. Live and web-based trainingand ongoing coaching support the use of CFS inclinical, supervisory, applied research, andmanagement activities. Quarterly quality assur-ance reports and learning collaborative telecon-ferences are provided beyond the first year ofintensive support to reinforce advanced integra-tion of data-informed decision-making and toshare ongoing learning of promising implemen-tation strategies. This is an undeveloped re-search area and we are just beginning to estab-lish empirical principles that go beyond thelimited effectiveness of traditional technical as-sistance and continuing education.

CFS technology is evolving and improving.For example, our earlier versions suffered from

278 BICKMAN, KELLEY, AND ATHAY

implementation problems associated with allevidence-based treatments. For example, thera-pist adherence in completing questionnaires andviewing reports was low and thus effectivenesssuffered. In the latest version of CFS, we antic-ipate that we ameliorated some of these prob-lems by displaying adherence rates that can beobserved by the therapist, supervisor, and man-agement. Thus, each therapists’ adherence iseasily tracked. Second, we are able to integratethe outcome measures used in CFS into thetypical reporting requirements (e.g., session-based documentation) each therapist must com-plete in order to be reimbursed. By makingadherence more transparent and visible as wellas building feedback into the required reporting,we expect adherence to be greatly improved.

Learning Organization Support

CFS is designed to assist in organizationalchange by sustaining a continuous learning en-vironment, and by encouraging and facilitatingthe appropriate use of empirically based mentalhealth interventions. The mental health carefield, like much of the medical community,lacks the electronic infrastructure to efficientlygenerate and collect clinical data. In commen-tating on medical care, Lindor (2012) notes thatthe absence of a method to gather clinical datain usual care is a major barrier to the develop-ment of evidenced based treatments. This limitsthe ability of provider organizations to learnhow to improve their services, minimizing theirability to contribute knowledge to the field as awhole. To develop and understand an interven-tion’s real world effectiveness, there must be aneasy mechanism to efficiently collect and ana-lyze clinical data from real world clients. CFSprovides that mechanism. New measures can beintroduced into the system at nominal cost, andmeasures can be revised and improved as newdata are gathered. Additionally, CFS allows thetesting of new interventions in the real worldwith essentially no cost associated with datacollection. Moreover, the system includes a ran-dom assignment generator that supports the or-ganization in the design of relatively easy toimplement randomized clinical trials. In thisway, service providers can help fill the gaps inour knowledge by identifying effective treat-ments that take into account the context, thera-pist, and client characteristics. An MFS such as

this provides an easy and inexpensive technol-ogy for ongoing data collection as part of usualclinical practice and forms the infrastructure fordata collection from all participants.

MFSs in Couples and Family Therapy

MFSs appear to have a positive effect onclient outcomes for both families and couples;however, the number of research projects isvery small. It is a new technology in need of agreat deal more research. Despite this, there isgreat potential for MFSs in couples and familytherapy, particularly for addressing the generalchallenges facing the field of psychotherapydiscussed previously, namely significant drop-out rates and the gap between research andpractice. In this section, we will briefly discusshow an MFS can address these challenges spe-cifically within the unique setting of couple andfamily therapy. Given the authors extensiveknowledge about the development, use, and po-tential of CFS, we will use this specific MFS fordemonstration purposes.

Challenge of Dropout

As previously discussed, premature dropoutfrom therapy is costly for the mental healthsystem and client’s clinical outcome. Althoughthere are many factors that relate to prematuredropout, one factor consistently found is thequality of the relationship between the clientand the therapist (i.e., therapeutic alliance). Re-search has demonstrated that good therapeuticalliance is positively correlated with lowerdropout rates (Kazdin, Holland, & Crowley,1997; Shields, Sprenkle, & Constantine, 1991).Given the greater complexity couples andfamily therapists face in establishing good thera-peutic alliance with each member in the dynamicfamily system, research suggests a potentiallygreater risk for premature dropout comparedwith individual therapy (Johnson & Wright,2002). The challenge in establishing good ther-apeutic alliance is especially apparent whenconsidering that each individual in a couple orfamily begins treatment with several presentingproblems, different levels of motivation, de-grees of interest, goals, and beliefs about howchange should occur (Rait, 2000).

Although measures of symptom severity areoften the only focus of clinical feedback, CFS

279MEASUREMENT FEEDBACK SYSTEMS

was specifically designed to incorporate a widevariety of measures relevant to clinical care.These can include measures assessing therapeu-tic alliance, motivation for treatment, and ex-pectations about therapy. By collecting suchinformation from each participant in treatment,CFS creates feedback that alerts the clinician topotential issues or discrepancies that may needto be addressed with individual clients or coupleor family. Although we recognize that cliniciansare continuously gathering similar informationinformally (i.e., through interactions with cli-ents, observation of body language, etc.), CFSenhances the clinician’s ability to make appro-priate clinical judgments by highlighting rele-vant information and tracking it over time, re-ducing the cognitive burden of the clinician.Thus, the clinician is better equipped to addressissues surrounding therapeutic alliance and pre-vent premature dropout.

Challenge of the Research and PracticeGap

The other challenge faced by the general fieldof mental health services is the gap betweenresearch and practice. Two ways this gap ismanifested are the difficulties in translating in-terventions from laboratory settings into realworld practice and incorporating outcome mon-itoring into clinical care. First, as discussedpreviously, it is common that treatments devel-oped and evaluated in a laboratory setting (i.e.,evidence-based treatments) do not display sim-ilar results in real world settings. Using an MFSsuch as CFS can provide the information nec-essary to get a clearer picture about what treat-ment are effective and for whom in UC (Bick-man, 2008a). For example, CFS has been pairedwith Functional Family Therapy (FFT) to ex-plore the effectiveness of FFT in UC settingsand help to understand the conditions underwhich it works best. Because CFS was notdeveloped to be treatment specific, it can beused with any evidence-based treatment or com-bination of treatments. This is especially usefulin the field of CFP where therapists report usingmultiple different approaches and techniques.

The other manifestation of the research–practice gap is the difficulty in incorporatingoutcome monitoring in UC. Despite the increas-ing evidence supporting the use of outcomemonitoring for improving client outcomes (e.g.,

Anker et al., 2009) and the frequent calls for itsuse in all practices (e.g., APA Presidential TaskForce on Evidence-Based Practice, 2006), workis needed to assist therapists in adopting andimplementing it in their practice. CFS was de-veloped as an automated, computerized, andcontextualized user-friendly and user-centeredsystem that can be integrated into a large varietyof clinical settings. In consideration of the tasksalready required by therapists and support staff,as well as the limitations of time, CFS measureswere designed to collect maximum informationin a limited amount of time (i.e., the last 5 minof a session) and are able to be completedelectronically (by computer, iPad, etc.) to elim-inate the need for manual data entry. Addition-ally, feedback was designed to be deliveredelectronically and presented in an easy-to-understand manner that highlights potentiallycritical clinical information (provides “alerts”)yet allows for the clinician to view any and alldata. This is accomplished with a “dashboard”system that displays critical and summary infor-mation at a glance with the ability of the clini-cian to select vivid icons to explore desiredinformation in more detail.

Technology, Therapy, MFSs, and theFuture

MFSs such as CFS have demonstrated a pos-itive effect on client outcomes for both individ-ual therapy settings as well as couple and familytherapy settings. Many of the features of MFSsare theoretically and conceptually compelling,such as the need for feedback for individual(e.g., clinician) learning and the urgency in cre-ating large-scale data infrastructures to aid inorganizational learning. Simply put, an MFSprovides the means to collect data from multipleperspectives relevant to CFP frequently through-out treatment, produce real-time feedback toinform treatment as it occurs, and provide ag-gregated data that can be used to build practice-based evidence within real world communitysettings. However, we are well aware of the keyaspects of MFSs that are unknown at this time.Therefore, MFSs are not being presented aspanaceas.

There are many limitations associated withMFSs because of the small amount of researchbeing conducted with them. For one, we do notknow how therapists actually use the feedback

280 BICKMAN, KELLEY, AND ATHAY

in practice. There is little research that ties thefeedback alerts given by most MFSs to actionsthat the clinician can or should take. For exam-ple, we know that the correlations among ayouth, a caregiver, and a therapist on ratings ofclient severity are low, but it is not clear whatclinicians should do with this information. Sim-ilarly, from our own research on therapeuticalliance (Bickman, 2012), we know that theclinician’s estimate of the youth’s and caregiv-er’s rating of therapeutic alliance has a lowcorrelation with the clinician’s estimate. Whenis it appropriate for the clinician to address suchdiscrepancies? Does merely knowing the dis-cordance help the clinician? An MFS providesnew knowledge to clinicians, but we need fur-ther research on effective clinical strategiesneeded to apply this specific and client-centeredknowledge to clinical practice. Fortunately,most MFSs can be used to productively exploresuch questions.

Similar to any other intervention, the effec-tiveness of an MFS is dependent on the successof implementation. As noted above, in version3.0 of CFS, key indicators of the quality ofimplementation are visible on every dashboardto aid in monitoring. Additionally, separate andmore detailed implementation reports are avail-able to therapists, supervisors, and manage-ment. But this information must be used by theprovider organization for it to be effective. Or-ganizations that will not or cannot supportchange and the use of technology are unlikely tobe successful in implementing an MFS. Simplyput, it won’t work if it is not used. As anemerging technology, there is a lack of researchon the most effective design that promotes bothadherence and integration of feedback intoclinic workflow. We need more research on theapplication of user-centered design to promoteevidence-based evaluation of MFSs.

Although there are now several commerciallyavailable MFSs that have shown positive effectson clinical outcomes, the adoption of MFSs islimited by the lack of incentives for deliveringeffective services. Without such incentives,MFSs will be limited to the providers who canfind funding and who share a vision with thedevelopers of MFSs about how to improve men-tal health services. Significant financial incen-tives are being offered in by the Centers forMedicare & Medicaid Services (CMS) for elec-tronic health records that include “meaningful

use” of these records. However, mental healthservices are only tangentially eligible for thesefinancial incentives at this time although itshould be noted that pressure is being appliedto extend the incentives to mental healthprofessionals.

Meaningful use is being implemented in stag-es. Stage I does not require the comprehensivefeedback provided by many MFSs. However,the CMS Electronic Health Record IncentivePrograms stage 2 requirement for meaningfuluse scheduled for implementation in 2014 willrequire Clinical Quality Measures (CQMs).CMS has recommended nine core CQMs forboth adult and pediatric populations. The CQMsfor both eligible professionals and hospitalsmust cover three of six National Quality Strat-egy domains: Patient and family engagement,Patient safety, Care coordination, Populationand public health, Efficient use of health careresource, and Clinical processes/effectiveness.We anticipate changes in policy and funding inmental health services that will follow CMS’sapproach, which demands more performanceindicators from mental health providers thansimply how many clients have been seen. Thetechnology of MFSs will help us meet thoseanticipated requirements.

References

Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009).Using client feedback to improve couple therapyoutcomes: A randomized clinical trial in a natural-istic setting Journal of Consulting and ClinicalPsychology, 77, 693–704. doi:10.1037/a0016062

APA Presidential Task Force on Evidence-BasedPractice. (2006). Evidence-based practice in psy-chology. American Psychologist, 61, 271–285.doi:10.1037/0003-066X.61.4.271

Bickman, L. (2008a). Why don’t we have effectivemental health services? Administration and Policyin Mental Health and Mental Health Services Re-search, 35, 437– 439. doi:10.1007/s10488-008-0192-9

Bickman, L. (2008b). A measurement feedback sys-tem (MFS) is necessary to improve mental healthoutcomes. Journal of the American Association ofChild and Adolescent Psychiatry, 47, 1114–1119.doi:10.1097/CHI.0b013e3181825af8

Bickman, L. (2012). Integrating functional familytherapy and CFS into a community based mentalhealth system “Contextualized Feedback Sys-tems.” Paper presented at the American Psycho-logical Association Convention, Tampa, FL.

281MEASUREMENT FEEDBACK SYSTEMS

Bickman, L., & Athay, M. M. (Eds.). (2012) Youthmental health measurement [special issue]. Admin-istration and Policy in Mental Health and MentalHealth Services Research, 39, 3–12.

Bickman, L., Athay, M. M., Riemer, M., Lambert,E. W., Kelley, S. D., Breda, C., . . . Vides deAndrade, A. R. (Eds). (2010). Manual of thepeabody treatment progress battery (2nd ed.).[Electronic version]. Nashville, TN: VanderbiltUniversity. Available from http://peabody.vander-bilt.edu/ptpb/

Bickman, L., Kelley, S. D., Breda, C., de Andrade,A. R., & Riemer, M. (2011). Effects of routinefeedback to clinicians on mental health outcomesof youths: Results of a randomized trial. Psychiat-ric Services, 62, 1423–1429. doi:10.1176/appi.ps.002052011

Bickman, L., Riemer, M., Lambert, E. W., Kelley,S. D., Breda, C., Dew, S. E., Brannan, A. M., &Vides de Andrade, A. R. (Eds). (2007). Manual ofthe Peabody Treatment Progress Battery [Elec-tronic version]. Nashville, TN: Vanderbilt Univer-sity. http://peabody.vanderbilt.edu/ptpb/

Cannon, M. D., & Witherspoon, R. (2005). Action-able feedback: Unlocking the power of learningand performance improvement. The Academy ofManagement Executive, 19, 120–134. doi:10.5465/AME.2005.16965107

Carlier, I. V. Meuldijk, D., Van Vliet, I. M., VanFenema, E., Van der Wee, N., & Zitman, F. G.(2012), Routine outcome monitoring and feedbackon physical or mental health status: Evidence andtheory. Journal of Evaluation in Clinical Practice18, 104 –110. doi:10.1111/j.1365-2753.2010.01543.x

Christensen, H., & Hickie, I. B. (2010). Using e-health applications to deliver new mental healthservices. Medical Journal of Australia, 192, S53–S56. Available from https://www.mja.com.au/

Doherty, G., Coyle, D., & Matthews, M. (2010).Design and evaluation guidelines for mental healthtechnologies. Interacting with Computers, 22,243–252. doi:10.1016/j.intcom.2010.02.006

Ehrenreich, B., Righter, B., Rocke, D. A., Dixon, L.,& Himelhoch, S. (2011). Are mobile phones andhandheld computers being used to enhance deliv-ery of psychiatric treatment? A systematic review.The Journal of Nervous and Mental Disease, 199,886–891. doi:10.1097/NMD.0b013e3182349e90

Garfield, S. L. (1994). Research on client variables inpsychotherapy. In A. E. Bergin, & S. L. Garfield(Eds.), Handbook of psychotherapy and behaviorchange (4th ed., pp. 190–228). New York: JohnWiley & Sons.

Germain, V., Marchand, A., Bouchard, S., Guay, S.,& Drouin, M. (2010). Assessment of the therapeu-tic alliance in face-to-face or videoconferencingcognitive-behavioral therapy for PTSD delivered

via videoconferencing technology. Behavior Mod-ification, 31, 856–866. doi:10.1177/0145445507302125

Granholm, E., Ben-Zeev, D., Link, P. C., Bradshaw,K. R., & Holden, J. L. (2011). Mobile assessmentand treatment for schizophrenia (MATS): A pilottrial of an interactive text-messaging interventionfor medication adherence, socialization, and au-diotory hallucinations. Schizophrenia Bulletin, 38,414–425. doi:10.1093/schbul/sbr155

Griffiths, K. M., Farrer, L., Christensen, H. (2010).The efficacy of internet interventions for depres-sion and anxiety disorders: A review of random-ized controlled trials. Medical Journal of Austra-lia, 192 (Suppl 11), S4 –S11. Available fromhttps://www.mja.com.au/

Hansen, H. B., Lambert, M. J., & Forman, E. M.(2002). The psychotherapy dose-response effectand its implications for treatment delivery ser-vices. Clinical Psychology: Science and Practice,9, 329–343. doi:10.1093/clipsy.9.3.329

Johnson, L. N., & Wright, D. W. (2002). RevisitingBordin’s theory on the therapeutic alliance: Impli-cations for family therapy. Contemporary FamilyTherapy: An International Journal, 24, 257–269.doi:10.1023/A:1015395223978

Kazdin, A. (2012). Advancing new frontiers withclinical psychological science. APS Observer, 25,14–16, 34.

Kazdin, A. E. (2008). Evidence-based treatment andpractice: New opportunities to bridge clinical re-search and practice, enhance the knowledge base,and improve patient care. American Psychologist,63, 146–159. doi:10.1037/0003-066X.63.3.146

Kazdin, A. E., Holland, L., & Crowley, M. (1997).Family experience of barriers to treatment andpremature termination from child therapy. Journalof Consulting and Clinical Psychology, 65, 453–463. doi:10.1037/0022-006X.65.3.453

Kelley, S. D., & Bickman, L. (2009). Beyond out-comes monitoring: Measurement feedback sys-tems in child and adolescent clinical practice. Cur-rent Opinion in Psychiatry, 22, 363–368. doi:10.1097/YCO.0b013e32832c9162

Kelley, S. D., Bickman, L., & Norwood, E. (2010).Evidence-based treatments and common factors inyouth psychotherapy. In B. L. Duncan, S. D.Miller, B. E. Wampold, & M. A. Hubble (Eds.),The heart and soul of change: Delivering whatworks in therapy (2nd ed., pp. 325–355). Wash-ington, DC: American Psychological Association.doi:10.1037/12075-011

King, R., Bickman, L., Shochet, I., McDermott, B., &Bor, B. (2010). Use of the internet for provision ofbetter counselling and psychotherapy services toyoung people, their families and carers. Psycho-therapy in Australia. 17, 62–70.

282 BICKMAN, KELLEY, AND ATHAY

Lindor, R. A. (2012). Comment: Advancing evi-dence-based medicine by expanding coverage withevidence development, Jurimetrics, 52, 209–237.

Lloyd, D. (October 27, 2011). Reviewing and apply-ing state and federal audit reports to enhanceknowledge base regarding audit expectations,trends and results. Webinar for the National Coun-cil. Available from www.nationalcouncil.org. Re-trieved on September 26, 2012.

Luxton, D. D., McCann, R. A., Bush, N. E., Mish-kind, M. C., & Reger, G. M. (2011). mHealth formental health: Integrating smartphone technologyin behavioral healthcare. Professional Psychology:Research and Practice, 42, 505–512. doi:10.1037/a0024485

Masi, M. V., Miller, R. B., & Olson, M. M. (2003).Differences in dropout rates among individual,couple, and family therapy clients. ContemporaryFamily Therapy: An International Journal, 25, 63–75. doi:10.1023/A:1022558021512

Morten, G., Anker, M. G., Duncan, B. L., & Sparks,J. A. (2009). Using client feedback toimprove cou-ple therapy outcomes: A randomized clinical trialin a naturalistic setting. Journal of Consulting andClinical Psychology, 77, 693–704. doi:10.1037/a0016062

Nelson, E. L., Barnard, M., & Cain, S. (2006). Fea-sibility of telemedicine intervention for childhooddepression. Counseling and Psychotherapy Re-search, 6, 191–195. doi:10.1080/14733140600862303

Pinsof, W. M., Wynne, L. C. (2000). Toward prog-ress research: Closing the gap between family ther-apy practice and research. Journal of Marital andFamily Therapy, 26, 1– 8. doi:10.1111/j.1752-0606.2000.tb00270.x

Rait, D. S. (2000). The therapeutic alliance in couplesand family therapy. Psychotherapy in Practice, 56,211–224. doi:10.1002/(SICI)1097-4679(200002)56:2

Reese, R. J., Toland, M. D., Slone, N. C., & Nor-sworthy, L. A. (2010). Effect of client feedback oncouple psychotherapy outcomes. PsychotherapyTheory, Research, Practice, Training, 47, 616–630. doi:10.1037/a0021182

Ridley, C. R., Tracy, M. L., Pruitt-Stephens, L.,Wimsatt, M. K., & Beard, J. (2008). Multiculturalassessment validity. In L. A. Suzuki, & J. G.Ponterotto (Eds.), Handbook of multicultural as-sessment: Clinical, psychological and educationalapplications (3rd ed., pp. 22–33). New York: JohnWiley & Sons.

Ridley, C. R., & Shaw-Ridley, M. (2009). Clinicaljudgment accuracy: From meta-analysis tometatheory. The Counseling Psychologist, 37(3),400–409. doi:10.1177/0011000008330830

Riemer, M., Kelley, S. D., Casey, S., & TaylorHaynes, K. (2012). Developing effective research-

practice partnerships for creating a culture of evi-dence-based decision making. Administration andPolicy in Mental Health and Mental Health Ser-vices Research. Special Issue: Research PracticePartnerships, 39, 248–257. doi:10.1007/s10488-011-0368-6

Riemer, M., Rosof-Williams, J., & Bickman, L.(2005). Theories related to changing clinicianpractice. Child and Adolescent Psychiatric Clinicsof North America, 14, 241–254, viii. doi:10.1016/j.chc.2004.05.002

Rotheram-Borus, M. J., Swendeman, D., & Chorpita,B. F. (2012). Disruptive innovations for designingand diffusing evidence-based interventions. Amer-ican Psychologist, 67, 463– 476. doi:10.1037/a0028180

Sapyta, J., Riemer, M., & Bickman, L. (2005). Feed-back to clinicians: Theory, research and practice.Journal of Clinical Psychology, 61, 145–153. doi:10.1002/jclp.20107

Sexton, T. L., Datachi Phillips, C., Evans, L., LaFol-lette, J., & Wright, L. (in press), The effectivenessof couple and family-based clinical interventions.In M. Lambert (Ed). Handbook of psychotherapyand behavior change. New York: Wiley.

Shields, C. G., Sprenkle, D. G., & Constantine, J. A.(1991). Anatomy of an initial interview: The im-portance of joining the structuring skills. AmericanJournal of Family Therapy, 19, 3–18. doi:10.1080/01926189108250829

Stark, M. M. (1992). Dropping out of substanceabuse treatment: A clinically oriented review.Clinical Psychology Review, 12, 93–116. doi:10.1016/0272-7358(92)90092-M

Stewart, R. E., & Chambless, D. L. (2008). Treat-ment failures in private practice: How do psychol-ogists proceed? Professional Psychology: Re-search and Practice, 39, 176–181. doi:10.1037/0735-7028.39.2.176

Swift, J. K., Greenberg, R. P., Whipple, J. L., &Kominiak, N. (2012). Practice recommendationsfor reducing premature termination in therapy.Professional Psychology: Research and Practice,43, 379–387. doi:10.1037/a0028291

Teachman, B. A., Drabick, D. A. G., Hershenberg,R., Vivian, D., Wolfe, B., Goldfried, M. R. (2012).Bridging the gap between clinical research andclinical practice: Introduction to the special sec-tion. Psychotherapy, 49, 97–100. doi:10.1037/a0027346

Wierzbicki, M., & Pekarik, G. (1993). A meta-analysis of psychotherapy dropout. ProfessionalPsychology: Research and Practice, 24, 190–195.doi:10.1037/0735-7028.24.2.190

Yuen, E. K., Goetter, E. M., Herbert, J. D., & For-man, E. M. (2012). Challenges and opportunitiesin internet-mediated telemental health. Profes-

283MEASUREMENT FEEDBACK SYSTEMS

sional Psychology: Research and Practice, 43,1–8. doi:10.1037/a0025524

Yuen, E. K., Herbert, J. D., Forman, E. M., Goetter,E. M., Juarascio, A. S., Rabi, S. J., . . . Bouchard,S. (2010, August). Using Skype videoconferencingand Second Life virtual environments to deliveracceptance-based behavior therapy for social anx-

iety disorder. Paper presented at the 44th AnnualConvention of the Association for Behavioral andCognitive Therapies, San Francisco, CA.

Received October 2, 2012Revision received October 19, 2012

Accepted October 25, 2012 �

284 BICKMAN, KELLEY, AND ATHAY