informing best practice in mental health: using feedback ... · informing best practice in mental...
TRANSCRIPT
Informing best practice in mental health: Using feedback to improve clinical outcomes.
Elizabeth A Newnham, BSc (Hons)
School of Psychology
The University of Western Australia
Year of submission: 2009
This thesis is presented for the degree of Doctor of Philosophy at
The University of Western Australia
2
Table of Contents
Abstract...........................................................................................................................................4
Publications Arising from this Thesis............................................................................................6
Author Contributions......................................................................................................................7
Acknowledgments..........................................................................................................................8
Section One: General Introduction
General Introduction..................................................................................................................10
Setting the scene..............................................................................................................10
Thesis aims and outline...................................................................................................11
Chapter One: Bridging the gap between best evidence and best practice in mental health.......16
Introduction.....................................................................................................................17
What treatment by whom.................................................................................... ............17
Is most effective...............................................................................................................20
For this individual...........................................................................................................22
With that specific problem...............................................................................................26
Implementing changes in practice...................................................................................27
New directions for the scientist practitioner...................................................................28
Section Two: The clinical value of routine outcome measures in psychiatric care
Foreword.......................................................................................................................................33
Chapter Two: Evaluating the clinical significance of responses by psychiatric inpatients
to the mental health subscales of the SF-36....................................................................38
Foreword.......................................................................................................................................45
Chapter Three: The subscale structure and clinical utility of the Health of the Nation
Outcome Scale.................................................................................................................53
3
Section Three: The effectiveness of progress monitoring and feedback in psychiatric
care
Foreword.......................................................................................................................................63
Chapter Four: Client-focused research: New directions in outcome assessment......................68
Chapter Five: Monitoring treatment response and outcomes using the World Health
Organization’s Wellbeing Index in psychiatric care.......................................................74
Chapter Six: Patient monitoring and feedback in psychiatric care reduces depressive
symptoms.........................................................................................................................80
Section Four: General Discussion
General Discussion...................................................................................................................104
Findings.........................................................................................................................104
Can we identify poor responders in psychiatric care?.....................................104
Is it possible to track patients’ progress during psychiatric care?..................107
What effect does feedback have on clinical outcomes?....................................112
Future research.............................................................................................................114
How does feedback work?................................................................................114
How can information technology best be used to benefit mental healthcare?.117
General conclusions......................................................................................................120
4
Abstract
Physical healthcare uses a suite of tools for measuring response to treatment. However,
reliable systems of regular patient monitoring are rare in mental healthcare. Mental health
services often measure a treatment response from pre- to post- therapy, yet measurement
between those occasions is less common. This omission is problematic since arguably there is a
need for an alarm system in psychotherapy (Andrews & Page, 2005). A substantial minority of
patients do not experience reliable change following treatment, and a small proportion
deteriorates (Hansen, Lambert, & Forman, 2002; Newnham, Harwood, & Page, 2007). Without
monitoring, it is not always possible to know which patients are progressing poorly. Since the
publication of Howard and colleagues’ (1996) proposal that patient progress be monitored
routinely during therapy and the results fed back to clinicians to direct treatment, this
monitoring regime has garnered attention in the United States and Europe (Lambert, 2007; Lutz,
et al., 2006). Findings in outpatient psychotherapy have demonstrated that providing real-time
feedback on patient progress to clinicians and patients significantly improves clinical outcomes
for those patients demonstrating a negative response to treatment (Harmon et al., 2007; Lambert
et al., 2001; Lambert et al., 2002). What is not yet apparent is how these processes would
generalize to inpatient and day patient (i.e. patients attending hospital for a whole day of
treatment) psychiatric care. Inpatients often present with greater severity and are treated in an
intensive setting. Therefore they will experience therapy differently to outpatients, may have
many treatments simultaneously and be managed by a team of professionals (usually under the
leadership of a medical practitioner). The thesis aims to contribute to the field of best practice
by describing the development and evaluation of a program for monitoring patient progress in a
hospital-based psychiatric setting. It was hypothesized that presenting feedback during therapy
to clinicians and patients would significantly improve wellbeing and significantly reduce
symptom distress for those patients at risk of poor outcome. Deviations from this expected
pattern would highlight possible differences between inpatient and outpatient care.
5
To develop an appropriate system for monitoring patient progress, it was important to
first define clinically significant recovery in inpatient psychiatric care, and provide criteria for
clinicians to judge outcome in routine practice (Newnham, Harwood, & Page, 2007). Second, a
quick and easy-to-administer system of progress monitoring and real-time feedback was
developed to enhance treatment decision making (Newnham, Hooke, & Page, 2009). Third, the
system was evaluated to determine clinical effectiveness. Using the World Health
Organization’s Wellbeing Index, a program for monitoring patient progress and providing
feedback to clinicians and patients was established at Western Australia’s largest private
psychiatric service. The sample consisted of 1308 consecutive inpatients and day patients whose
primary diagnoses were predominantly depressive (67.7%) and anxiety (25.9%) disorders.
Feedback to patients and clinicians was effective in reducing depressive symptoms (F (1,649) =
6.29, p<.05) for those patients at risk of poor outcome, but not effective in improving wellbeing
(F (1,569) = 1.14, p>.05). The findings support the use of progress monitoring and feedback in
psychiatric care to improve symptom outcomes, but raise questions about changes in wellbeing
during psychotherapy. The effectiveness study was conducted as a historical cohort trial,
consistent with quality improvement efforts, and replication with a randomized controlled
design is warranted. Feedback of progress information appears to be an important process
within psychotherapy, and further investigation of the means by which clinicians and patients
use that information is necessary.
6
Publications Arising from this Thesis
Chapter 1
Newnham, E. A. & Page, A. C. (In press). Bridging the gap between best evidence and best practice in mental health. Clinical Psychology Review. doi:10.1016/j.cpr.2009.10.004
Chapter 2
Newnham, E. A., Harwood, K. E. & Page, A. C. (2007). Evaluating the clinical significance of responses by psychiatric inpatients to the mental health subscales of the SF-36. Journal of Affective Disorders, 98, 91-97.
Chapter 3
Newnham, E. A., Harwood, K. E. & Page, A. C. (2009). The subscale structure and clinical utility of the Health of the Nation Outcome Scales. Journal of Mental Health, 18, 326-334.
Chapter 4
Newnham, E. A., & Page, A. C. (2007). [Invited essay] Client-focused research: New directions in outcome assessment. Behaviour Change, 24, 1, 1-6.
Chapter 5
Newnham, E. A., Hooke, G. R. & Page, A. C. (In press). Monitoring treatment response and outcomes using the World Health Organization’s Wellbeing Index in psychiatric care. Journal of Affective Disorders. doi:10.1016/j.jad.2009.1006.1005
Chapter 6
Newnham, E. A., Hooke, G. R. & Page, A. C. (under review). Patient monitoring and feedback in psychiatric care reduces depressive symptoms.
7
Author Contributions
Chapter 1: Elizabeth Newnham (90%) and Andrew Page (10%) developed the design of the
manuscript; Elizabeth Newnham wrote the first draft; and both authors contributed to and approved of the final manuscript.
Chapter 2: Elizabeth Newnham (75%) and Andrew Page (20%) designed the study; Elizabeth Newnham conducted the statistical analyses and wrote the first draft; Elizabeth Newnham, Kate Harwood (5%) and Andrew Page all contributed to and approved of the final manuscript.
Chapter 3: Elizabeth Newnham (70%) and Andrew Page (25%) designed the study and conducted the statistical analyses; Elizabeth Newnham wrote the first draft; Elizabeth Newnham, Kate Harwood (5%) and Andrew Page all contributed to and approved of the final manuscript.
Chapter 4: Elizabeth Newnham (80%) and Andrew Page (20%) contributed to the manuscript design; Elizabeth Newnham wrote the first draft; and both authors contributed to and approved of the final manuscript.
Chapter 5: All authors contributed to the study design. Geoff Hooke (10%) managed the data collection. Elizabeth Newnham (75%) and Andrew Page (15%) undertook the statistical analyses and Elizabeth Newnham wrote the first draft of the manuscript. All authors contributed to and have approved of the final manuscript.
Chapter 6: All authors contributed to the study design and data collection. Elizabeth Newnham (70%) and Andrew Page (20%) conducted the statistical analyses and Elizabeth Newnham wrote the first draft of the manuscript. Elizabeth Newnham, Geoff Hooke (10%) and Andrew Page contributed to and have approved of the final manuscript.
8
Acknowledgments_______________________________________________________
I feel very fortunate to have had the opportunity to write a PhD thesis, and more so, to love the research field even more now than when I started. I have many people to thank for this.
First, my deepest gratitude to Professor Andrew Page for his wisdom, guidance, and humour. I would never have embarked upon a PhD without his encouragement, and his support has been constant. Andrew made the process of writing a PhD exciting and interesting, and inspired me to commence a scientific career. I look forward to many more years of collaborative work together.
Thank you to Perth Clinic and the Clinical Improvement Team. I very much appreciate the generous support of Moira Munro and the clinical staff, particularly the CBT therapists. I am indebted to Greg Beard and Paul Hussein, and the administration staff whose assistance helped the project run so smoothly. I am very proud to have worked with such an impressive and effective team.
A special thank you to Geoff Hooke, who helped to implement every research plan, provided inspired discussion, and noticed every new hairstyle.
In particular, I would like to thank the many patients who have contributed to this research.
This research was supported by a number of grants. I would like to acknowledge the Medicare Private Safety and Clinical Improvement Incentive Pool. The research was also supported by a University Postgraduate Award; UWA Completion Scholarship; Geoffrey Kennedy Postgraduate Research Travel Award; a UWA Grant for Research Student Training; and travel grants from the Graduate Research School and School of Psychology at UWA.
I wish to thank Professor Mike Anderson who read a draft of this thesis and whose comments improved it.
I was privileged to share an office with three very bright and entertaining PhD students who have become my close friends. Thank you Pia, Emma and Karina.
To my wonderful girlfriends for their constant encouragement and entertaining distractions. A special thank you to Alex, Heather, Liz, Sarah and Madeleine.
I am deeply grateful to my partner Willis Samson, for his unwavering support and love. Thank you for listening to every idea, for your wise advice, for travelling the world to support me, and for making me laugh. You have been my pillar of strength.
Thank you to my family. To my sister Prudence, for her friendship and humour. To my brother Richard, for the technical support, political debates, and strength when I needed it. To my father, John, who is the most wonderful mentor; and my mother, Susie, my closest confidante; who supported and celebrated every achievement. Thank you. My love of science is the product of being raised in such a close family, for whom scientific debate is regular dinner table discussion.
9
Section One General Introduction
10
General Introduction__________________________________________
Monitoring patient response is integral to modern medical care. Most areas of
healthcare are aided by the use of an instrument, such as the thermometer, that provides clear
and reliable information on the patient’s health status. A thermometer alerts clinicians to
changes in a patients’ health, and thus complements clinical judgment of progress and recovery.
As yet, there is no such instrument for gauging health status in psychiatric care. Yet the
rationale is clear. A mental health thermometer would complement current clinical practice by
assessing patient response to treatment and alerting clinicians to unexpected changes.
Accordingly, clinicians would be empowered to act upon subtle (or substantial) deviations in
the patient’s progress during their treatment.
Development of a mental health thermometer would involve three steps (Stritzke &
Page, 2009). The first is the definition of recovery: a cut-off point that would indicate positive
response or conversely, poor response to treatment. The second involves the generation of
trajectories that illustrate the patient’s expected response to treatment. Deviation from this
pattern of improvement would thus highlight poor response and the need for intervention. The
third step is the implementation of a feedback system that alerts clinicians to poor response,
providing an opportunity for change with minimal interference to the therapeutic process
(Stritzke & Page, 2009).
The current thesis comprises a series of studies that outline the process described above.
To develop and implement a mental health thermometer in clinical practice, recovery was
defined, expected response trajectories were created, and a feedback system implemented; all
within a hospital-based psychiatric service. In doing so, the research program aimed to extend
findings in patient-focused research and contribute to bridging the gap between science and
clinical practice.
Setting the Scene
The widening gap between science and practice is particularly problematic because of
the sheer scale of the level of mental health need in Australia. One in five adults (20%) report
11
experiencing a mental health problem in the previous twelve-month period (Australian Bureau
of Statistics, 1998; Slade et al., 2009). Depressive disorders account for the greatest level of
burden of disability for both males and females in Australia (Mathers, Vos, & Stevenson, 1999),
and are a leading cause of disability burden on a global scale. Great costs, both personal and
financial (Kessler & Frank, 1997), are associated with depressive disorders. Effective evidence-
based treatments for depression are widely available, but not all patients improve during therapy
(Newnham, Harwood, & Page, 2007), and relapse rates are high (Daniels, Kirkby, Hay, Mowry,
& Jones, 1998). A cost-effective instrument that highlights poor response during therapy thus
has the potential to improve clinical practice and consequently, treatment outcomes.
Thesis aims and outline
The primary aim of the present thesis is to develop and assess the effectiveness of a
system for monitoring patient progress and providing feedback within a psychiatric inpatient
and day patient population. Second, the thesis looks to explore new directions in outcome
assessment in the field of mental health, and contribute to the area by critically assessing the
psychometric and clinical validity of routinely used outcome measures, and defining criteria for
recovery in psychological health care.
The thesis is comprised of four parts that address the primary concern of improving the
effectiveness of treatment in psychiatric services. Section One outlines the state of evidence in
mental health outcome assessment. Chapter 1 consists of a review of the literature in
psychotherapy outcomes research and provides a context for patient-focused methodologies,
with an aim to bridge the gap between science and practice. Section Two addresses the use of
outcome assessment measures in psychiatric care, and specifically their clinical value in routine
practice. The specific aims are to determine:
1. That recovery can be defined using clinical significance methodologies with outcome
measures commonly used in Australia, such as the SF-36, for the benefit of practising
clinicians.
2. A factor structure that is both psychometrically sound and clinically meaningful for the
Health of the Nation Outcome Scale.
12
The section comprises two published manuscripts that investigate the psychometric
properties and clinical function of two measures, the Medical Outcomes Short Form (SF-36)
and the Health of the Nation Outcome Scales (HoNOS). Both have been recommended for use
as outcome assessment measures in routine clinical practice in Australia by the Consumer
Outcomes Project Advisory Group advising the Commonwealth Government (Stedman,
Yellowlees, Mellsop, Clarke, & Drake, 1997). However, despite this recommendation,
assessment of their psychometric properties and clinical validity in psychological settings in
Australia has been minimal. The papers comprising Section One (Newnham et al., 2007;
Newnham, Harwood, & Page, 2009) attempt to expand upon the Advisory Group’s guidance by
providing clinically meaningful information on recovery criteria and appropriate scoring
methods for the measures, so that they may be more readily used in routine practice.
The third section of the thesis comprises three papers, of which two are published and one
has been submitted for publication. These papers outline the development and evaluation of a
system for monitoring patient wellbeing and providing feedback to clinicians and patients
within an inpatient and day patient setting. Specifically, the following hypotheses are
investigated:
1. The World Health Organization Wellbeing Index (WHO-5) is an appropriate and
clinically useful monitoring instrument for use with a psychiatric sample, in that is it
reliable, sensitive to change in time-limited treatment, and early scores predict final
outcome.
2. Monitoring patient progress using the WHO-5, and providing individualized feedback
to clinicians and patients during group therapy will improve (i) wellbeing and (ii)
symptom relief as measured on a series of standardized self report and clinician rated
outcome assessment instruments.
Chapter 4 (Newnham & Page, 2007) reviews the literature in the area of patient-focused
research and highlights the challenges remaining. Chapter 5 (Newnham, Hooke, & Page, 2009)
extends previous work conducted with the WHO Wellbeing Index with an investigation of the
psychometric properties of the WHO-5 within a psychiatric setting, finding it to be a reliable
and valid measure, appropriate for use as a monitoring and outcome assessment tool in mental
13
health services. The final study (Newnham, Hooke, & Page, submitted) reports a non-
randomized effectiveness trial that demonstrated that providing feedback on patient progress in
psychiatric care was effective in reducing depressive symptoms, but did not improve wellbeing
for those at risk of poor outcome. This trial addresses two issues that have not been previously
examined in the mental health literature: the implementation a freely available monitoring
system within a psychiatric hospital setting, and the use of feedback within a group therapy
format. The impact of both features is outlined in Chapter 1.
The Discussion (Section Four) contextualizes the findings of the current thesis within
the field of patient-focused research and clinical outcomes assessment in mental health.
Changes in wellbeing during psychotherapy are examined, and new directions for the
assessment of feedback processes are outlined. The thesis therefore contributes to a growing
field of evidence-based practice by providing an innovative and sustainable means by which
clinicians can assess and improve their practice.
14
References
Australian Bureau of Statistics. (1998). Mental Health and Wellbeing: Profile of Adults,
Australia, 1997 Canberra: ABS Cat No. 4326.0.
Daniels, B. A., Kirkby, K. C., Hay, D. A., Mowry, B. J., & Jones, I. H. (1998). Predictability of
rehospitalisation over 5 years for schizophrenia, bipolar disorder and depression.
Australian and New Zealand Journal of Psychiatry, 32, 281-286.
Kessler, R. C., & Frank, R. G. (1997). The impact of psychiatric disorders on work loss days.
Psychological Medicine, 27, 861-873.
Mathers, C., Vos, T., & Stevenson, C. (1999). The burden of disease and injury in Australia.
Canberra: Australian Institute of Health and Welfare.
Newnham, E. A., Harwood, K. E., & Page, A. C. (2007). Evaluating the clinical significance of
responses by psychiatric inpatients to the mental health subscales of the SF-36. Journal
of Affective Disorders, 98, 91-97.
Newnham, E. A., Harwood, K. E., & Page, A. C. (2009). The subscale structure and clinical
utility of the Health of the Nation Outcome Scale. Journal of Mental Health, 18, 326-
334.
Newnham, E. A., Hooke, G. R., & Page, A. C. (2009). Monitoring treatment response and
outcomes using the World Health Organization's Wellbeing Index in psychiatric care.
Journal of Affective Disorders, doi:10.1016/j.jad.2009.1006.1005.
Newnham, E. A., Hooke, G. R., & Page, A. C. (submitted). Patient monitoring and feedback in
psychiatric care reduces depressive symptoms.
Newnham, E. A., & Page, A. C. (2007). Client-focused research: New directions in outcome
assessment. Behaviour Change, 24(1), 1-6.
Slade, T., Johnston, A., Teesson, M., Whiteford, H., Burgess, P., Pirkis, J., et al. (2009). The
Mental Health of Australians 2. Report on the 2007 National Survey of Mental Health
and Wellbeing. Canberra: Department of Health and Ageing.
15
Stedman, T., Yellowlees, P., Mellsop, G., Clarke, R., & Drake, S. (1997). Measuring Consumer
Outcomes in Mental Health. Canberra, ACT: Department of Health and Family
Services.
Stritzke, W. G. K., & Page, A. C. (2009). Electronic patient monitoring in mental health
services. In A. Dwivedi (Ed.), Handbook of Research on Information Technology
Management and Clinical Data Administration in Healthcare (pp. 87-103). Hershey,
New York: IGI Publishing.
16
Chapter One_________________________________________________
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
Section Two The clinical value of routine outcome measures in psychiatric care.
33
Foreword
Clinical psychology is now building upon the processes of data collection and
assessment, to improve the utility of outcome evaluation programs and their interpretation.
Thus, outcomes information must be presented in a way that enables its routine use by
practising clinicians (Clancy & Eisenberg, 1998). With automated or at least easily calculated
interpretation methods available, clinicians will be able to integrate outcomes information into
clinical work without the usual challenges to time, resources and the therapeutic relationship
(Hatfield & Ogles, 2004). When such methods are available to clinicians, outcome assessment
takes on a much greater significance not only for the assessment of treatments or services, but
also for individual patients.
The measurement of recovery is not new. Reports dating from the 1920s outline
attempts by clinicians to classify the outcomes of their patients as ‘cured’, ‘improved’, ‘much
improved’ and ‘uncured’. The Berlin Psychoanalytic Institute took care to collect clinician-rated
data on the proportion of patients in each category (Bergin & Lambert, 1978), thus providing
some indication of therapeutic effectiveness. Despite the methodological flaws (Ogles, Lunnen,
& Bonesteel, 2001), the early data signals an attempt to address a dilemma that has troubled the
field since its origins – the definition and measurement of psychological recovery. More
recently, a rise in the use of statistical methodologies to capture “significant” change, has
included the employment of an alpha criterion to determine reliable improvement, or effect size
calculations to address the magnitude of change. However, the question remained – has the
client experienced a meaningful improvement as a result of the treatment?
Thus the status quo in outcomes assessment is not sufficient. Providing an averaged
group score (or even an individual’s assessment or outcome score) is not adequate information
for meaningful interpretation of progress or useful for treatment planning. It is not sufficient
because average scores can mask wide individual differences. The average may also not be the
best measure of central tendency for a particular individual or sub-group. Fortunately, given the
progress made in the large-scale collection of outcomes data, we now have an exciting
34
opportunity to provide clinicians (and also patients) with more meaningful information about a
particular patient’s treatment progress.
The development of clinical significance methodologies (Jacobson & Truax, 1991)
enables the categorization of outcome according to the individual patient’s pathway of recovery.
Utilizing scores at two points in time (for inpatients this is often admission and discharge) we
can determine whether a patient has made change large enough to be statistically reliable, and
whether that change has involved a clinically meaningful reduction in symptoms. Alternatively,
it enables knowledge of whether a patient’s functioning has deteriorated over the time period.
Recent advances have been made in calculating the criteria for recovery on widely and routinely
used measurement tools such as the Medical Outcomes Questionnaire Short Form (SF-36;
Ware, Snow, Kosinski, & Gandek, 1993) and the Health of the Nation Outcome Scale (HoNOS;
Wing et al., 1998) which will provide clinicians with a much richer level of interpretation of
patients’ outcomes (see Newnham, Harwood, & Page, 2007; Parabiaghi, Barbato, D'Avanzo,
Erlicher, & Lora, 2005).
The provision of individualized outcome data empowers patients and positions them as
co-managers of their own healthcare (Calman, 1998). It enables a dialogue between clinician
and patient about treatment progress and expectations, such as whether the patient has made a
meaningful recovery and should consider termination of therapy, or an objective indication that
expected progress has not been achieved. Whilst increasing accountability for service providers,
the open discussion of outcomes data may also increase responsibility on the patient’s part so
that they may reflect upon their own treatment progress and put necessary changes into action.
Only with the appropriate and psychometrically valid assessment of recovery, will people be
able to make informed choices about their mental healthcare.
The mental health subscales of the SF-36 are widely and routinely used at a national
and international level, and their use as a measure of outcome is mandatory in the psychiatry
private sector in Australia. It was anticipated that the following study (Newnham et al., 2007)
would provide clinicians with information specific to their individual patient’s recovery. Further
to this, assessment of a patient group’s clinically significant recovery provides valuable
information on treatment effectiveness, which is potentially useful for service providers, health-
35
care managers, and administrative and legislative policy makers. Criteria outlined for the mental
health subscales of the SF-36 provide clinicians with a simple formula for determining a
patient’s level of improvement. By entering a patient’s pre- and post-treatment scores and
determining whether they have passed the level of statistically significant and meaningful
change, clinicians are positioned to assess the progress of their caseload and to provide valid
information on their own effectiveness.
An important factor in encouraging the use of outcome assessment by clinicians and
service providers is the ease of application. So in practical terms, how would the clinician assess
and interpret their patient’s change over the course of treatment? The first step is to subtract the
patient’s admission score from their discharge score, and divide the answer by the subscale’s
Standard Error of Difference (as outlined in Newnham et al., 2007). If the answer is greater than
1.96, then the patient has achieved statistically reliable change during the therapy period.
Additionally, if the patient’s discharge score is greater than or equal to the subscale’s cut-off
point (also illustrated in Newnham et al., 2007); then the patient has made clinically significant
change. With this information, clinicians have the opportunity to make decisions about the
treatment plan, their patient’s progress, or their own effectiveness with ease.
Chapter 2 also illustrates the rate of recovery for inpatients receiving psychiatric
treatment in Australia. While the majority of patients made a reliable improvement, a notable
proportion experienced a negative outcome as measured on the SF-36 mental health subscales.
It is important to identify patients who respond poorly to treatment so that action can be taken to
address this issue. Findings in outpatient therapy suggest that a significant proportion of patients
do not achieve reliable improvement in treatment (Hansen, Lambert, & Forman, 2002), however
a corresponding assessment of recovery rates has not yet been conducted with inpatient
samples. Chapter 2 illustrated rates of no change in therapy that ranged from 43.6% to 56.6%;
and that approximately 2% of inpatients deteriorated during treatment (Newnham et al., 2007).
This finding indicates that a significant proportion of patients are not experiencing the
improvement expected from treatment, and that a need exists for an intervention that addresses
poor response prior to the termination of therapy (in this case, discharge from hospital). Thus
36
Chapter 2 marks the first step in identifying poor responders in psychiatric care, so that their
progress may be addressed during treatment.
37
References
Bergin, A. E., & Lambert, M. J. (1978). The evaluation of therapeutic outcomes. In S. L.
Garfield & A. E. Bergin (Eds.), The handbook of psychotherapy and behavior change
(2nd ed.). New York: John Wiley.
Calman, K. C. (1998). Potential for health. Oxford: Oxford University Press.
Clancy, C. M., & Eisenberg, J. M. (1998). Outcomes research: Measuring the end results of
health care. Science, 282(5387), 245-246.
Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). The psychotherapy dose-response
effect and its implications for treatment delivery services. Clinical Psychology: Science
and Practice, 9, 329-343.
Hatfield, D. R., & Ogles, B. M. (2004). The use of outcome measures by psychologists in
clinical practice. Professional Psychology: Research and Practice, 35(5), 485-491.
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining
meaningful change in psychotherapy research. Journal of Consulting and Clinical
Psychology, 59, 12-19.
Newnham, E. A., Harwood, K. E., & Page, A. C. (2007). Evaluating the clinical significance of
responses by psychiatric inpatients to the mental health subscales of the SF-36. Journal
of Affective Disorders, 98, 91-97.
Ogles, B. M., Lunnen, K. M., & Bonesteel, K. (2001). Clinical significance: History, application
and current practice. Clinical Psychology Review, 21, 421-446.
Parabiaghi, A., Barbato, A., D'Avanzo, B., Erlicher, A., & Lora, A. (2005). Assessing reliable
and clinically significant change on HoNOS: A method for displaying longitudinal data.
Australian and New Zealand Journal of Psychiatry, 39(719-724).
Ware, J. E., Snow, K. K., Kosinski, M., & Gandek, B. (1993). SF-36 Health Survey: Manual
and Interpretation Guide. Boston: The Health Institute, New England Medical Centre.
Wing, J. K., Beevor, A. S., Curtis, R. H., Park, S. B. G., Hadden, S., & Burns, A. (1998). Health
of the Nation Outcome Scales (HoNOS): Research and development. British Journal of
Psychiatry, 172, 11-18.
38
Chapter Two_________________________________________________
39
40
41
42
43
44
45
Foreword
The expansion of health outcomes research and implementation of evaluation programs
in service settings is changing the face of psychological care. The use of routine outcome
assessment in practice benefits clinical, ethical and scientific domains (Slade, 2002; Stedman,
Yellowlees, Mellsop, Clarke, & Drake, 1997). Clinically, evaluating patient progress has the
potential to inform practice and treatment planning; whilst the assessment of outcomes ensures
the highest quality of care – an ethical responsibility for all clinicians. The evaluation of
treatment has traditionally been conducted with efficacy and effectiveness research. Efficacy
research provides an indication of therapeutic benefit for the average patient under ideal
conditions in randomized controlled trials. If the treatment is shown to be efficacious, its
effectiveness within a clinical setting with less controlled variables may then be investigated.
Whilst efficacy research maximises internal validity at the cost of external validity,
effectiveness research strengthens a treatment’s potential to generalise to other clinical settings.
Although efficacy and effectiveness studies complete one side of the scientist-
practitioner equation, the ongoing evaluation of ordinary practice is an important
complementary process. Now that the age of outcome assessment is maturing, we have the
opportunity to use that information not only for the assessment of practice, but also to improve
the services we provide.
However, despite advances, there are a range of difficulties that hinder the assessment
of outcomes in mental health. First, defining a positive outcome may be problematic. It’s not
always the case that the clinician and patient agree upon a desired outcome for therapy (Slade,
2002), despite a common goal of optimal functioning for the individual. For example, a
clinician may encourage a patient to reduce their alcohol intake, which may not align with the
patient’s treatment goals. Second, different types of outcome can be desynchronous and
different indices may change at different rates during intervention. The phase model (Howard,
Lueger, Maling, & Martinovich, 1993) proposes that the remediation of hope occurs first,
followed by a reduction in symptoms, and then a gradual rehabilitation and improvement in
quality of life. Changes in patients’ wellbeing, symptomatic distress and life functioning at
46
different sessions across the duration of therapy have been shown to be consistent with the
three-phase model (Howard et al., 1993). To complicate things further, for many patients, some
things get better while others may get worse (Trauer, 1998). For example, medication may
improve mood, but at the risk of unpleasant side effects. Capturing this complex progression of
wellbeing is not possible in a single measure, and requires significant consideration when
deciding upon the timing of assessments. Multiple assessments or organizing administration at
clinically meaningful points during therapy are essential for the appropriate evaluation of
outcomes.
Third, there are a number of reasons why a patient may not improve, not all of which
equate to a negative outcome. Occasionally, as may be the case in the treatment of
schizophrenia, the aim of therapy may be to maintain a level of functioning, or to slow a decline
in functioning, and thus the expected outcome will not be recovery. Additionally, the purpose of
hospitalization may be to monitor a patient’s wellbeing whilst changing medication, to keep
them safe from suicide attempts, to remove them from a dangerous environment or to reduce
risk to others. In each of these cases, an improvement in symptom scores may not reflect all
facets of an outcome.
These issues highlight the significance of collecting outcome data from a variety of
sources. A clinician’s perspective of the patient’s progress will differ from the patient’s
perspective and vice versa. It would be insufficient to measure only one viewpoint and assume
an objective assessment of outcome. Therefore, to complement the collection of patient self-
report data, the use of clinician-rated measures has been widely encouraged (Stedman et al.,
1997).
Enthusiasm about evidence-based practice and accountability in health care has driven a
vast improvement in outcome assessment methods and measures. In Australia, the first national
mental health strategy involved a systematic review of outcomes data collection policies and
practices (Andrews, Peters, & Teesson, 1994), the result of which was a set of recommendations
for specific measures to be used in routine care. This review was followed by an independent
field-assessment of the recommended measures and procedures (Stedman et al., 1997). Despite
shifting the focus towards developing optimal methods of outcome evaluation; the consultancy
47
provided specific suggestions for the use of measures when assessing treatment outcomes in
mental health. Of six measures proposed, two were endorsed for routine use in clinical practice:
the mental health subscales of the SF-36 and the Health of the Nation Outcome Scale (Stedman
et al., 1997). Where service providers had an outcomes collection system already established,
continuance was encouraged. However, for those not already collecting outcomes data, the
implementation of a program using the SF-36 subscales and the Health of the Nation Outcome
Scale (HoNOS) in combination was advised. Accordingly, both patient and clinician
perspectives would be captured and the data available could contribute to outcome evaluation,
as well as treatment planning and quality improvement efforts. Yet despite enthusiasm for using
the HoNOS to complement patient health status data, the field report asserted that at the time,
the HoNOS was a new measure, and further assessment of its psychometric properties and
clinical utility were required (Stedman et al., 1997).
For an outcome measure to provide meaningful and valid data, Slade (2002) has
proposed a number of criteria that must be met: (1) the measure assesses a desired outcome, a
process issue, problem severity or enables comparison between staff and patient views; (2) it is
specifically designed for a mental health population; (3) it is freely available; (4) brief to
administer; and (5) the measure has peer-reviewed published evidence of sound psychometric
properties.
The HoNOS assesses severity and breadth of presenting problems, as well as
functioning and environmental factors that may influence wellbeing; and when used in
combination with self-report measures, allows for comparison between staff and patient views.
This provides an important level of information not only about the patient’s difficulties, but also
their insight into those difficulties. Created by the Royal College of Psychiatrists for the purpose
of assessing national health status in the United Kingdom (College Research Unit, 1996), the
HoNOS meets criteria for specific mental health assessment. It is freely available, quick to
administer (less than 15 minutes) and to increase reliability of the measure, training is required
for its use. Since its development, the psychometric properties of the HoNOS have been an issue
of great debate. In accordance with the authors’ intentions (Wing et al., 1998), the items are
heterogenous and cover a wide range of symptoms and behaviours, resulting in low internal
48
consistency (Preston, 2000). However, despite this, evidence suggests that the HoNOS is a valid
and useful measure in clinical practice (Hansen & Kingdon, 2006; Kisely, Campbell, Crossman,
Gleich, & Campbell, 2007; Page, Hooke, & Rutherford, 2001; Wing, Lelliott, & Beevor, 2000).
The HoNOS’ value has certainly been reflected in its use. The use of the HoNOS in
outcomes studies and clinical assessment reports from the United Kingdom (Wing et al., 2000),
Europe (Lora, Bezzi, & Erlicher, 2007; Parabiaghi, Barbato, D'Avanzo, Erlicher, & Lora, 2005),
the United States (Kisely et al., 2007), Australia (Pirkis, Burgess, Kirk, Dodson, & Coombs,
2005) and New Zealand (Theuma, Read, Moskowitz, & Stewart, 2007) are a testament to the
need for a clinician’s judgment of treatment outcome when evaluating services and treatments.
The collection of HoNOS data is mandatory within both private and public psychiatric services
in Australia, and yet very little has been published to support its use and enable meaningful
interpretation by clinicians, service providers and government policy makers.
Further criteria originally identified by a committee of experts assembled by the
National Institute of Mental Health in the United States of America (NIMH; Ciarlo, Brown,
Edwards, Kiresuk, & Newman, 1986) refer to the measure’s use on a wider scale. They
proposed that an effective outcome measure should have the capacity to be understood by non-
professional audiences, that feedback should be easily provided and interpreted, that it should be
useful in clinical services and be compatible with clinical theories and practices (Newman &
Ciarlo, 1994). HoNOS data have been used in practice to provide information to the patient
about their treatment progress or outcome, changes that have been made or progress still
expected, and what areas have been influenced by the intervention (Page et al., 2001). On a
larger scale, HoNOS data are collected across all private psychiatric hospitals in Australia, and
these data are used to inform service providers and policy makers (Morris-Yates & The
SPGPPS Data Collection and Analysis Working Group, 2000). The Private Mental Health
Alliance, (PMHA, formerly SPGPPS) provides quarterly reports on service effectiveness which
allows for comparison between services with casemix accounted for. HoNOS data are integral
to this reporting, as they provide a clinician’s perspective on patient outcomes. Accordingly,
HoNOS results have the capacity to be provided as feedback for patients, clinicians, third-party
payors, and policy makers at all levels of government.
49
However, at present, the feedback of results can be problematic. The current models for
subscale structure and subsequent scoring methods of the HoNOS have received much criticism
(Preston, 2000; Trauer, 1999) and do not appear to provide patient information consistent with
clinical theory and practice. The twelve items were designed to provide information on distinct
and disparate contributing factors. They range from rating aggressive, agitated behaviours to
activities of daily living and the availability of appropriate accommodation. It appears that the
College Research Unit (1996) developed the conventional four subscales upon the basis of item
similarity (Trauer, 1999). In contrast, Trauer (1999, 2000) has investigated the HoNOS’ utility
in largely psychotic samples and proposed a novel five-subscale structure based on these
findings. Despite its demonstrated validity within some patient groups, it is unlikely that a
subscale structure developed within a psychotic sample will provide an adequate fit for a largely
mood disordered patient group. Depression is the most common mental health problem reported
in Australia, and now accounts for the greatest burden of disease for all mental disorders,
followed by substance abuse and anxiety disorders respectively (Mathers, Vos, & Stevenson,
1999). Depressive disorders affect a substantial proportion of the population and thus
appropriate outcome evaluation for patients with depression, substance use and anxiety is
important information for patients, families, health-care providers, and the public. It is vital that
the assessment of outcome for these patients cater to their clinical characteristics in order to
provide a valid interpretation of the data. It is for this reason that the current study aimed to
investigate the factor structure and interpretation of the HoNOS within a predominantly mood
and anxiety disordered population. Previously, despite administration and data collection
requirements for mental health services in Australia, insufficient empirical evidence has been
available to assess the HoNOS’ validity in psychiatric samples. The following study (Newnham,
Harwood, & Page, 2009) illustrates the most appropriate means for scoring and interpreting the
HoNOS within a predominantly mood-disordered psychiatric sample.
50
References
Andrews, G., Peters, L., & Teesson, M. (1994). The measurement of consumer outcome in
mental health. Canberra: Australian Government Publishing Service.
Ciarlo, J. A., Brown, T. R., Edwards, D. W., Kiresuk, T. J., & Newman, F. L. (1986). Assessing
mental health treatment outcome measurement techniques (DHHS Pub. No. (ADM)86-
1301). Washington, DC: Superintendent. of Documents., U.S. Government Printing
Office.
College Research Unit. (1996). Health of the Nation Outcome Scales: Report on Research.
London: Royal College of Psychiatrists.
Hansen, L., & Kingdon, D. (2006). Rating Suicidality in Schizophrenia: Items on Global Scales
(HoNOS and CPRS) Correlate with a Validated Suicidality Rating Scale (InterSePT).
Archives of Suicide Research, 10(3), 249-252.
Howard, K. I., Lueger, R. J., Maling, M. S., & Martinovich, Z. (1993). A phase model of
psychotherapy: Causal mediation of outcome. Journal of Consulting and Clinical
Psychology, 61, 678-685.
Kisely, S., Campbell, L. A., Crossman, D., Gleich, S., & Campbell, J. (2007). Are the Health of
the Nation Outcomes Scales a valid and practical instrument to measure outcomes in
North America? A three-site evaluation acorss Nova Scotia. Community Mental Health
Journal, 43(2), 91-107.
Lora, A., Bezzi, R., & Erlicher, A. (2007). Estimating the prevalence of severe mental illness in
mental health services in Lombardy (Italy). Community Mental Health Journal, 43(4),
341-357.
Mathers, C., Vos, T., & Stevenson, C. (1999). The burden of disease and injury in Australia.
Canberra: Australian Institute of Health and Welfare.
Morris-Yates, A., & The SPGPPS Data Collection and Analysis Working Group. (2000). A
National Model for the Collection and Analysis of a Minimum Data Set with Outcome
Measures for Private Hospital-based Psychiatric Services. Canberra: Commonwealth
of Australia.
51
Newman, F. L., & Ciarlo, J. A. (1994). Criteria for selecting psychological instruments for
treatment outcome assessment. In M. E. Maruish (Ed.), The use of psychological testing
for treatment planning and outcome assessment (pp. 98-110). Hillsdale, NJ: Lawrence
Erlbaum Associates Inc.
Newnham, E. A., Harwood, K. E., & Page, A. C. (2009). The subscale structure and clinical
utility of the Health of the Nation Outcome Scale. Journal of Mental Health, 18, 326-
334.
Page, A. C., Hooke, G. R., & Rutherford, E. M. (2001). Measuring mental health outcomes in a
private psychiatric clinic: Health of the Nation Outcome Scales and Medical Outcomes
Short Form SF-36. Australian and New Zealand Journal of Psychiatry, 35, 377-381.
Parabiaghi, A., Barbato, A., D'Avanzo, B., Erlicher, A., & Lora, A. (2005). Assessing reliable
and clinically significant change on HoNOS: A method for displaying longitudinal data.
Australian and New Zealand Journal of Psychiatry, 39(719-724).
Pirkis, J., Burgess, P., Kirk, P., Dodson, S., & Coombs, T. (2005). Review of standardised
measures used in the National Outcomes and Casemix Collection. Version 1.2.
Canberra: National Mental Health Strategy.
Preston, N. (2000). The Health of the Nation Outcome Scales: Validating factorial structure and
invariance across two health services. Australian and New Zealand Journal of
Psychiatry, 34, 512-519.
Slade, M. (2002). What outcomes to measure in routine mental health services, and how to
assess them: a systematic review. Australian and New Zealand Journal of Psychiatry,
36, 743-753.
Stedman, T., Yellowlees, P., Mellsop, G., Clarke, R., & Drake, S. (1997). Measuring Consumer
Outcomes in Mental Health. Canberra, ACT: Department of Health and Family
Services.
Theuma, M., Read, J., Moskowitz, A., & Stewart, A. (2007). Evaluation of a New Zealand early
intervention service for psychosis. New Zealand Journal of Psychology, 36(3), 136-145.
Trauer, T. (1998). Issues in the assessment of outcome in mental health. Australian and New
Zealand Journal of Psychiatry, 32, 337-343.
52
Trauer, T. (1999). The subscale structure of the Health of the Nation Outcome Scales (HoNOS).
Journal of Mental Health, 8, 499-509.
Trauer, T. (2000). Update from Down Under. British Journal of Psychiatry, 176, 392-395.
Wing, J. K., Beevor, A. S., Curtis, R. H., Park, S. B. G., Hadden, S., & Burns, A. (1998). Health
of the Nation Outcome Scales (HoNOS): Research and development. British Journal of
Psychiatry, 172, 11-18.
Wing, J. K., Lelliott, P., & Beevor, A. S. (2000). Progress on HoNOS. British Journal of
Psychiatry, 176, 392-395.
53
Chapter Three________________________________________________
54
55
56
57
58
59
60
61
62
Section Three The effectiveness of progress monitoring and feedback in psychiatric care.
63
Foreword
The National Mental Health Strategy outlines a cycle of quality improvement in mental
health service, by which data are collected and analysed, and the resulting findings inform and
improve practice. Considerable progress has been made in each of these individual areas;
however, a full completion of the cycle is rarely realized in practice. The recent establishment of
large-scale outcome assessment programs in health care (e.g. CORE Partnership, 2007;
Lambert, 2007) have made significant progress in the collection and assessment of data. A
multitude of evaluation measures that are both psychometrically sound and clinically useful
have been reviewed in the scientific literature and adopted in routine practice. Further to this,
data collection at an individual (Lambert, 2007), service-wide, state and national level
(Department of Health and Ageing, 2008) has provided findings that have the potential to
inform practice and policy. Increasingly useful and meaningful methods of interpretation are
being developed for use by professionals and the public alike (e.g. Newnham, Harwood, &
Page, 2007, 2009; Parabiaghi, Barbato, D'Avanzo, Erlicher, & Lora, 2005). We are now in a
position to answer the most important question in psychotherapy: Is this treatment working for
this particular patient?
In a seminal work, Howard and colleagues (1996) outlined the three central questions in
clinical psychology research. First, does this treatment work? Second, does this treatment work
in real-practice settings? Third and most importantly, a question that has been largely neglected
within psychological science is whether this treatment is working for a particular patient. The
first question has been answered by randomised controlled trials evaluating the efficacy of
psychological treatments. Effectiveness research answers the second question. And the third
question, of particular interest to clinicians and patients, is the subject of patient-focused
research.
Clinical significance has broadened our understanding of psychotherapy outcomes, and
demonstrated that not all patients achieve reliable and meaningful change, despite use of
evidence-based treatments (Lambert, Hansen, & Bauer, 2008). In fact, some patients deteriorate
64
over the course of treatment (Lambert & Ogles, 2004). Clearly, receiving this information at the
end of therapy is not useful for that particular patient, and thus feedback on treatment progress
in real time is required. At an individual level, patient progress information enhances the
opportunity for change during the course of therapy.
The ongoing monitoring of individual patients’ progress and the feedback of that
information to clinicians and patients significantly improves outcome in outpatient therapy
(Harmon, Hawkins, Lambert, Slade, & Whipple, 2005; Harmon et al., 2007; Hawkins, Lambert,
Vermeersch, Slade, & Tuttle, 2004; Lambert, Hansen, & Finch, 2001; Lambert, Whipple et al.,
2001; Lambert et al., 2002). This process of providing feedback to clinicians about their
patient’s level of improvement, in comparison to their expected level of improvement, is a new
development in clinical and research strategy. Based upon the assertion that the foundation of
change is knowledge (Calman, 1998), and in a therapeutic partnership which requires
continuous revision and reassessment, patient-focused research provides education as to a
patient’s level of progress, contrasted against their expected improvement. With a goal to
complement clinical judgment, feedback provides an avenue for assessment of treatment and the
patient’s relationship to change.
However, despite the promising findings demonstrated for outpatient, individual
therapy (Lambert, 2007), a program for patient-focused research has not yet been developed for
use within an inpatient setting. Further to this, there is potential for its use in a group-therapy
format, and this has not been assessed either. Accordingly, the aim of the current project was to
develop and evaluate a patient-focused research program within a group-format inpatient and
day patient psychiatric setting. The first paper, (Newnham & Page, 2007), describes patient-
focused research and its role within psychotherapy research and practice. The manuscript was
published with a view to inform CBT therapists of recently developed methodologies as well as
the potential benefits of patient monitoring. Notably, patient monitoring is a suitable vehicle for
conducting research within one’s practice, and in turn, using those findings to inform process.
The second paper in this section (Newnham, Hooke, & Page, 2009), assesses the
psychometric properties of the World Health Organization’s Wellbeing Index (WHO-5) within
a psychiatric inpatient setting, and illustrates its clinical validity as a monitoring measure. The
65
final paper (Newnham, Hooke, & Page, submitted), evaluates the effectiveness of the wellbeing
monitoring program conducted within an inpatient psychiatric setting, and demonstrates the
potential for patient-focused research to improve treatment outcomes in hospital-based settings.
The effectiveness trial is unique not only in its format and applicable patient-group, but
also in its method of evaluation. Other trials of patient-focused research have evaluated
effectiveness with comparison against the same measure (Hawkins et al., 2004; Lambert,
Whipple et al., 2001). The current feedback trial assessed effectiveness not only on the same
measure, the WHO-5, but also against psychometrically valid measures of function and
psychological distress. Having illustrated the sound properties and clinical validity of the self-
rated mental health subscales of the SF-36, and the clinician-rated HoNOS, these, together with
the Depression and Anxiety Stress Scales were ample measure of the program’s effect.
66
References
Calman, K. C. (1998). Potential for health. Oxford: Oxford University Press.
CORE Partnership. (2007). Is initial overall CORE-OM score an indicator of likely outcome? In
CORE Partnership Occasional Paper, No 1. CORE IMS: Rugby.
Department of Health and Ageing. (2008). 2007-08 Annual Report. Canberra: Commonwealth
of Australia.
Harmon, C., Hawkins, E. J., Lambert, M. J., Slade, K., & Whipple, J. L. (2005). Improving
outcomes for poorly responding clients: The use of clinicial support tools and feedback
to clients. Journal of Clinical Psychology, 61(2), 175-185.
Harmon, C., Lambert, M. J., Smart, D. W., Hawkins, E. J., Nielsen, S. L., Slade, K., et al.
(2007). Enhancing outcome for potential treatment failures: Therapist/client feedback
and clinical support tools. Psychotherapy Research, 17(4), 379-392.
Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K., & Tuttle, K. (2004). The effects
of providing patient progress information to therapists and patients. Psychotherapy
Research, 14, 308-327.
Howard, K. I., Moras, K., Brill, P., Martinovich, Z., & Lutz, W. (1996). Evaluation of
psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist,
51, 1059-1064.
Lambert, M. J. (2007). Presidential address: What we have learned from a decade of research
aimed at improving psychotherapy outcome in routine care. Psychotherapy Research,
17(1), 1-14.
Lambert, M. J., Hansen, N. B., & Bauer, S. (2008). Assessing the clinical significance of
outcome results. In A. M. Nezu & C. M. Nezu (Eds.), Evidence-based Outcome
Research: A Practical Guide to Conducting Randomized Controlled Trials for
Psychosocial Interventions. New York: Oxford University Press.
Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Patient-focused research: Using patient
outcome data to enhance treatment effects. Journal of Consulting and Clinical
Psychology, 69, 159-172.
67
Lambert, M. J., & Ogles, B. M. (2004). The efficacy and effectiveness of psychotherapy. In M.
J. Lambert (Ed.), Bergin and Garfield's Handbook of Psychotherapy and Behaviour
Change (5 ed., pp. 139-193). New York: Wiley.
Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S. L., & Hawkins, E.
J. (2001). The effects of providing therapists with feedback on patient progress during
psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49-68.
Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L.,
et al. (2002). Enhancing psychotherapy outcomes via providing feedback on patient
progress: A replication. Clinical Psychology and Psychotherapy, 9, 91-103.
Newnham, E. A., Harwood, K. E., & Page, A. C. (2007). Evaluating the clinical significance of
responses by psychiatric inpatients to the mental health subscales of the SF-36. Journal
of Affective Disorders, 98, 91-97.
Newnham, E. A., Harwood, K. E., & Page, A. C. (2009). The subscale structure and clinical
utility of the Health of the Nation Outcome Scale. Journal of Mental Health, 18, 326-
334.
Newnham, E. A., Hooke, G. R., & Page, A. C. (2009). Monitoring treatment response and
outcomes using the World Health Organization's Wellbeing Index in psychiatric care.
Journal of Affective Disorders, doi:10.1016/j.jad.2009.1006.1005.
Newnham, E. A., Hooke, G. R., & Page, A. C. (submitted). Patient monitoring and feedback in
psychiatric care reduces depressive symptoms.
Newnham, E. A., & Page, A. C. (2007). Client-focused research: New directions in outcome
assessment. Behaviour Change, 24(1), 1-6.
Parabiaghi, A., Barbato, A., D'Avanzo, B., Erlicher, A., & Lora, A. (2005). Assessing reliable
and clinically significant change on HoNOS: A method for displaying longitudinal data.
Australian and New Zealand Journal of Psychiatry, 39(719-724).
68
Chapter Four_______________________________________________
69
70
71
72
73
74
Chapter Five_________________________________________________
75
76
77
78
79
80
Chapter Six__________________________________________________
Patient monitoring and feedback in psychiatric care reduces depressive symptoms.
Abstract
Background: To date, the monitoring of patient progress using standardized assessments
has been neglected in hospital-based psychiatric care. Findings in outpatient psychotherapy
have demonstrated clinically significant benefits for providing feedback to the sizeable minority
of patients who were otherwise unlikely to experience positive outcome (Lambert, 2007).
However, a similar system for presenting feedback on patient progress has not yet been assessed
within psychiatric inpatient settings. The current study aimed to develop and evaluate the
effectiveness of a feedback system suitable for use in psychiatric services. Methods: In a
nonrandomized trial, 1308 consecutive inpatients and day patients, whose diagnoses were
primarily depressive and anxiety disorders according to ICD-10 criteria, completed the World
Health Organization’s Wellbeing Index (WHO-5) routinely during a ten-day cognitive
behavioral therapy group. The first cohort (n=461) received treatment as usual. The second
cohort (n=439) completed monitoring measures without feedback, and for patients in the third
cohort (n=408), feedback on progress was provided to both clinicians and patients midway
through the treatment period. Results: Feedback was effective in reducing depressive symptoms
(F(1,649)=6.29, p<.05) for those patients at risk of poor outcome, but not effective in improving
wellbeing (F(1,569)=1.14, p>.05). Conclusions: Similar to outpatient settings, feedback
appears to be beneficial for improving symptom outcomes but further time may be required for
wellbeing to be affected. The current findings may be generalized to patient samples that exhibit
largely depressive disorders, and rigorous follow-up is warranted.
81
Psychiatric care, unlike physical healthcare, does not yet have the instruments available
to reliably monitor patient health and alert clinicians to a negative response to treatment.
Physical healthcare relies upon a suite of tools, such as the thermometer, to guide treatment.
Monitoring complements and informs the treatment process by providing the clinician with
quick, easy to interpret feedback on a patient’s response. Unfortunately, psychiatric care is in
the early phase of developing standard monitoring instruments that may alert clinicians to a
poor treatment response and signal a need to evaluate and possibly alter therapy.
This absence is concerning because relatively high rates of negative outcome occur in
mental health care. Despite the widespread use of evidence-based treatments, a large proportion
of patients still fail to demonstrate reliable or clinically meaningful improvement. Estimations
of deterioration rates have been as high as 23%, and up to 40% of patients show no change as a
result of therapy, as illustrated in both clinical settings (Lambert et al., 2002; Newnham,
Harwood, & Page, 2007; Parabiaghi, Barbato, D'Avanzo, Erlicher, & Lora, 2005) and
randomized controlled trials (Hansen, Lambert, & Forman, 2002), where a highly controlled
treatment is assessed under optimal conditions and with homogenous patient groups. It appears
that there is no typical pattern of change for all patients (Lutz, Stulz, & Köck, 2009), and that
despite the success of efficacy and effectiveness studies in identifying valuable treatments; a
large minority of patients are not benefiting from psychological therapy. One response to these
findings has been to seek better and more refined treatments. Another, not mutually exclusive
response has been to highlight the need to complement the delivery of psychological treatment
with additional progress information to improve rates of clinical recovery. By monitoring
progress during the therapy period, clinicians have the opportunity to improve outcomes in real
time for the benefit of each particular individual (Lambert, Harmon, Slade, Whipple, &
Hawkins, 2005; Lutz, 2003; Page & Stritzke, 2006).
Monitoring programs suitable for outpatient psychological therapy have been developed
in the United States and Europe, with notable success. In the US, Lambert and colleagues have
developed a program of feedback to alert therapists and patients to deviations from expected
response (Lambert, 2007). Administration of the 45-item Outcomes Questionnaire (OQ-45) at
each therapy session allows for the ongoing monitoring of progress, mapped against the
82
patient’s expected response trajectory. The expected trajectory is a function based upon the
patient’s severity at intake. Five studies (Harmon et al., 2007; Hawkins, Lambert, Vermeersch,
Slade, & Tuttle, 2004; Lambert et al., 2001; Lambert et al., 2002; Whipple et al., 2003), of
which four were randomized controlled trials, have demonstrated that providing feedback on
patients’ treatment response improved rates of clinically significant outcomes, but perhaps more
importantly, reduced negative outcomes from 20% to 8%. Those studies that have demonstrated
a significant improvement in outcomes have also revealed cost-efficient benefits for the use of
ongoing monitoring. Whilst the minority of cases that were identified as ‘not on track’ were
provided two to three sessions extra, the majority were ‘on track’ and received one session less
(Lambert, 2007). Thus progress monitoring and feedback appears to improve treatment
outcomes in an efficient and cost-effective manner.
However, one issue not addressed by the studies to date relates to the quantity of
psychotherapy provided. Although participants were randomly assigned to conditions, the
number of sessions of therapy delivered varied across conditions. As mentioned, the cost-
effectiveness of the different numbers of sessions received could be highlighted, but it does
mean that the research has not revealed that the same benefits would be observed if patients in
the feedback and non-feedback groups received the same quantity of therapy. Consequently,
research is needed in which the amount of treatment delivered is constant across feedback and
non-feedback groups to address this issue.
At present, monitoring patient response has been illustrated as an important addition to
clinical practice and research within the scientific literature; however it has been greatly
underutilized in practice. There are a number of arenas in which it would be beneficial to
monitor treatment response more closely. The argument is particularly compelling for fields in
which early response to treatment is useful information, such as drug trials; or the monitoring of
patient health is particularly important, such as suicide risk management or treatment for
trauma. Time-intensive psychotherapy and inpatient psychiatric care also require ongoing
monitoring but as yet, an appropriate and efficient monitoring system for short-term assessment
has not been identified. This is a surprising reality, given that Lambert rightly heralds the
ongoing monitoring of patient progress a clinician’s ethical responsibility (Lambert, 2007).
83
Despite the temptation to extend the methods to all areas of mental healthcare, of the
models that have been developed for the ongoing monitoring of treatment response (Lambert,
2007; Lutz et al., 2005) none operate over a brief time-frame. For instance, treatment offered in
inpatient settings is often intensive and brief, being measured in days, as opposed to outpatient
psychotherapy that takes place over weeks. Although a number of health status measures are
available for monitoring treatment response, all measure progress within a timeframe of 1 week
to 3 months, and most fail to demonstrate appropriate psychometric features for individual
monitoring (Lambert & Hawkins, 2004; McHorney & Tarlov, 1995). Accordingly, a system of
monitoring appropriate for settings in which changes may occur in days, not weeks, is required.
In addition, current programs have been developed within outpatient samples, and
therefore the outcomes may not necessarily generalize to an inpatient group setting (Newnham
& Page, 2007). Inpatient care differs in that it can be more intensive, the population may
experience a greater level of disturbance, psychotherapy is delivered in the context of a variety
of other interventions (e.g., pharmacotherapy, ongoing nursing care) and patients are often
discharged with the expectation that further improvement will occur with ongoing community
care. Furthermore, while the average length of stay in psychiatric facilities differs between
nations, Australia, like the United States, tends to have relatively short admissions (Page,
Hooke, & Rampono, 2005). It is therefore important to develop a monitoring program that has
the capacity to provide feedback within a shorter time-frame.
The current study comprised the development and evaluation of a monitoring system
suitable for use in an acute psychiatric setting. The World Health Organization’s Wellbeing
Index (WHO-5; Bech, Gudex, & Johansen, 1996) is a five-item self-report measure of positive
wellbeing that has performed reliably and sensitively in psychiatric samples (Newnham, Hooke,
& Page, 2009). The monitoring program involved the routine administration of the WHO-5 as a
measure of patient progress, evaluated within a Cognitive Behavioral Therapy group at a private
psychiatric hospital. The aim of the study was to assess the effectiveness of monitoring patient
progress, using that information as feedback for clinicians and patients. It was anticipated that
patients receiving feedback on their progress during group therapy would exhibit significantly
improved outcomes on measures of wellbeing and symptom distress at completion of the group.
84
Method
Research setting and participants
Participants were recruited from a 98 bed, private psychiatric hospital in Australia.
Eligible participants were English speaking inpatients or day patients who were participating in
the hospital’s two week cognitive behavioral therapy program. The sample comprised of 1308
patients who participated in the trial; 408 who received feedback on their WHO-5 scores, 439
completed the WHO-5 routinely but did not receive feedback on scores until completion of the
group, and a control group of 461 who were not administered the WHO-5 while participating in
the CBT treatment program.
Participants were diagnosed by their treating psychiatrist according to ICD-10-AM
criteria (National Centre for Classification in Health Publications, 2002) and primary diagnoses
consisted mostly of mood (67.7%), anxiety (25.9%), and substance use (3.0%) disorders. Ages
ranged from 16 years to 76 years with a mean of 39.8, and 63.1% were female. There were no
significant differences between groups in diagnoses (F (2,1302) = 1.77, p > .05), sex (F
(2,1305) = .476, p > .05) or age (F (2,1305) = .295, p > .05). The investigation was carried out
in accordance with the Declaration of Helsinki. The University of Western Australia Human
Research Ethics Committee approved the study protocol prior to commencement, and patients
provided informed consent as part of the routine admission procedure at the hospital.
Monitoring and Outcome Measures
The World Health Organization’s Wellbeing Index (WHO-5; Bech et al., 1996) is a 5-
item scale of positive wellbeing. Originally designed to screen for depression in diabetic
samples (Bech et al., 1996), the WHO-5 has also demonstrated reliability and validity in
screening for depression in primary care (Löwe et al., 2004) and elderly samples (Bonsignore,
Barkow, Jessen, & Heun, 2001). Further to this, the psychometric properties and clinical utility
of the WHO-5 has warranted its use as a measure for monitoring patient progress and treatment
outcomes in psychiatric care (Newnham et al., 2009). The Index is rated on a five point scale,
and for the purpose of the current study, wellbeing was measured over the past day rather than
previous two weeks (Newnham et al., 2009). Scores thus range from 0 to 25 with higher scores
indicating more positive wellbeing.
85
The Mental Health subscales of the Medical Outcomes Questionnaire Short Form (SF-
36; Ware, Snow, Kosinski, & Gandek, 1993), are a reliable self-report measure of patient
mental health status. They comprise of Vitality, Social Function, Role Emotion and Mental
Health scales, which have previously demonstrated validity as measures of patient outcomes in
psychiatric care (Newnham et al., 2007). The scales have sound internal consistency, exceeding
0.8 for each of the scales (Scott, Tobias, Sarfati, & Haslett, 1991), together with strong content
and construct validity (Newnham et al., 2007; Scott et al., 1991; Ware et al., 1993).
The Depression Anxiety Stress Scale (DASS-21; Lovibond & Lovibond, 1995a) is a 21-
item self-report measure of psychopathology. A short form of the of the 42 item scale, the
DASS-21 has strong internal consistency (Crawford & Henry, 2003) sound construct validity
(Lovibond & Lovibond, 1995b), and a cleaner factor structure (Antony, Bieling, Cox, Enns, &
Swinson, 1998). It is rated on a 5-point scale and high scores indicate more severe
psychopathology.
The Health of the Nation Outcome Scale (HoNOS; Wing et al., 1998) is a 12-item
measure of patient mental health that covers a heterogeneous range of presenting issues,
including aggression and agitated behavior, hallucinations, depressed mood, difficulties with
activities of daily living, social relationships and housing issues. The HoNOS is completed by
therapy staff, psychiatric nurses or the treating psychiatrist, all of whom have completed a
training program for its administration. The Scale measures health as reported for the previous
two weeks when rated at admission and over the preceding 72 hours when rated at discharge.
All items on the scale are rated from 0 (no problem) to 4 (severe problem); thus low scores
indicate healthier functioning.
The SF-36, DASS-21 and HoNOS were administered as routine clinical practice at
admission and discharge for each patient.
Trial Design
The trial comprised a historical cohort design so that between January 2005 and March
2006, patients were not administered the WHO-5; from April 2006 to July 2007 the WHO-5
was routinely administered during treatment, but feedback was not presented to the patient or
clinician until the final day of therapy; and between August 2007 and January 2009, participants
86
completed the WHO-5 routinely during therapy and were provided with feedback on their
progress at Day 5 and Day 10. The cohorts that were administered the WHO-5 were matched
according to the severity of their WHO-5 score at Day 1 of treatment.
Group Treatment
The Cognitive Behavioral Treatment Program is a closed group of 6-8 members that
runs from 9am to 5pm over 10 working days. Each group is run by two therapists, and covers
depression and anxiety management, cognitive disputation, behavioral activation and
experiments, identification and modification of negative core beliefs, self esteem,
communication skills, stress management, and dealing with setbacks. Each group member sets
personal therapy goals and participates in homework tasks. The treatment program has
demonstrated effectiveness comparable to randomized controlled trials (Page & Hooke, 2003).
Monitoring Intervention
Prior to the commencement of the group therapy, on days 1 (Monday), 3 (Wednesday),
5 (Friday), 7 (Tuesday), and 9 (Thursday) of the program, patients completed the WHO-5.
Occasionally, if a day was missed, the questionnaire would be completed on the following day.
For those in the No Feedback cohort, scores were graphed and provided to patients on Day 10
(Friday), where they were given the opportunity to discuss their scores with the therapist.
Using the data from the No Feedback group’s wellbeing scores, expected treatment
response cures were generated. The sample was divided into five groups according to severity
of wellbeing scores at Day 1, so that each group consisted of 20% of the sample. For each
group, means and standard deviations were calculated for each measurement point, which
illustrated a dose-response curve across the ten days of therapy. A log linear curve was
generated, one standard deviation around the mean for each day, for each group (see Table 1).
This curve became the trajectory of expected response against which each patient’s actual
scores were mapped (see Figure 1).
Consistent with the procedure of administration for No Feedback, those in the Feedback
condition completed the WHO-5 on days 1 (Monday), 3 (Wednesday), 5 (Friday), 7 (Tuesday),
and 9 (Thursday) of the program. According to the patient’s score at Day 1, an expected
treatment response curve was generated, and their actual scores mapped against it. This graph
87
and an accompanying explanation was provided as feedback to the clinician and patient at Day
5 (Friday of the first week of therapy) and Day 10 (Friday of the final week of therapy).
Feedback graphs were distributed during group, and clinicians spent time discussing what the
graphs meant, exploring the meaning of fluctuations in scores, and discussing the opportunity
that feedback provides to examine one’s progress and re-assess the treatment goals. Clinicians
were not given specific directions on the use of feedback but clinical management meetings
were held during the trial to assess adherence to protocol and discuss the clinical management
of cases.
Day 5 was considered the most appropriate point for feedback for clinical and evidence-
based reasons. Regression analyses (see Newnham et al., 2009) deemed Day 5 to be the point at
which Day 9 scores could be best predicted and it provided sufficient time to modify the
treatment plan. Further to this, it was suggested by the clinical management team that providing
feedback on a Friday morning presented the optimal opportunity to review the week’s work, set
homework exercises for the weekend ahead, and evaluate and devise treatment goals for the
second week of therapy.
Results
Equivalence of Groups
To compare the effectiveness of providing feedback, groups were matched according to
severity at Day 1 on the WHO-5. This was calculated by comparing each cohort according to
their expected trajectory of response groupings. Accordingly, for the analyses, each condition
had a sample size of 379, with no significant differences in severity between groups at Day 1.
To reduce the skew of the data, a square root of WHO-5 scores at each day was taken.
This score was used for all further WHO-5 analyses. Quality control charts revealed three
therapy groups with outlying scores (i.e. three standard deviations beyond the mean) which
were removed from the sample for further analyses.
Definition of Alarm
Consistent with Lambert’s research (Harmon et al., 2007; Lambert et al., 2001; Lambert
et al., 2002; Whipple et al., 2003), the data were analysed according to whether a person was
deemed ‘on track’ at the point of feedback. Following the feedback processes used, those who
88
were deemed to be ‘not on track’ at Day 5 (alarm cases) were patients whose Day 1 scores were
below 12 (thus within the unwell range at admission to the CBT Program), and whose scores
fell below the expected trajectory of improvement at Day 5.
Intervention Outcomes
The distribution of raw scores was inspected and means and standard deviations were
calculated for each cohort across all outcome measures (see Table 2). The data suggests that on
average, patients in all cohorts move from the ‘unwell’ range to the ‘well’ range on the WHO-5
(Newnham et al., 2009) and SF-36 (Newnham et al., 2007) as a result of therapy.
Patient self-report outcomes
To assess the effectiveness of feedback in reducing patients’ symptoms and improving
wellbeing, a series of repeated-measures ANOVAs were conducted with the outcome measure
as the dependent variable, and condition and alarm status as between group factors. A
significant improvement over time was evident for both conditions so that on average, patients
improved in wellbeing as a result of therapy (F (1,569) = 237.1, p<.05). However, there was no
significant difference in wellbeing scores at Day 9 between feedback conditions for those
patients on track or not on track (F (1,569) = 1.14, p>.05). Thus feedback to staff and patients
about scores on the WHO-5 did not significantly improve patients’ wellbeing by Day 9.
In contrast, feedback was of benefit on some measures of symptom distress. Again,
significant improvement over time was evident for all conditions on the Depression (F (1,649) =
438.5, p<.05), Anxiety (F (1,649) = 305.2, p<.05), and Stress (F (1,649)=421.2, p<.05) scales
of the DASS-21. A significant interaction was evident, arising because those patients who were
not on track and received feedback exhibited relatively greater improvements in Depression
scores on the DASS-21 (F (1,649) = 6.29, p<.05). However no significant difference resulting
from feedback was evident for any patients on the Anxiety (F (1,649) = .496, p>.05) or Stress
(F (1,649) = .628, p>.05) subscales of the DASS-21.
Those patients not on track who received feedback exhibited a significant improvement
on the Vitality subscale (F (1,639) = 5.53, p<.05), and Role Emotion subscale (F (1,635) = 4.11,
p<.05) of the SF-36. Yet no significant interaction was evident for Mental Health (F (1,639) =
2.28, p>.05) or Social Function subscales (F (1,643) = 1.94, p>.05). The results suggest that
89
measures of depressive symptoms, such as vitality and emotion, demonstrate a significant
improvement for those patients who received feedback when at risk of poor outcome.
The clinical validity of the results was investigated using the Jacobson and Truax
(1991) criteria for clinical significance (Newnham et al., 2009). Consistent with the finding that
feedback does not affect outcomes for those patients on track during therapy, positive outcomes
on the Vitality subscale of the SF-36 remained constant across feedback conditions (40.8%
achieved reliable improvement without feedback and 40.2% with feedback). However, the
provision of feedback reduced deterioration rates from 5.3% in the No Feedback cohort to 3.3%
in the Feedback cohort.
In contrast, no significant change in rates of clinically significant improvement or
deterioration was evident for the DASS-21 Depression subscale. Positive outcomes remained
constant across cohorts (51.3% achieved reliable improvement without feedback and 51.1%
with feedback). Similarly, 48.7% of participants exhibited a negative outcome (no change or
deterioration) without receiving feedback, compared to 48.9% with feedback. Deterioration
rates showed no significant change across cohorts.
The calculation of clinical significance was not appropriate for the Role Emotion
subscale due to the limited variability in potential scores (Newnham et al., 2007).
Clinician-rated outcomes
When examining the clinician-rated scores, again a significant effect of improvement
over time was evident so that patients were regarded as much improved following therapy (F
(1,615) = 639.7, p<.05). Yet, no effect of feedback was evident on HoNOS total scores
regardless of predicted outcome (F (1,615) = 3.20, p>.05). Interestingly, the data depict an
overall elevation of HoNOS scores for those patients not on track regardless of whether
feedback is provided, but no significant effect is evident (F (1,615) = 3.15, p=.08).
To investigate the hypothesis that feedback improves the convergence of clinician-rated
scores with patient self-report, correlations were conducted between HoNOS scores and the
other outcome measures (see Table 3). There is some indication of convergence between
clinician-ratings and self-report as the correlations reliably increase over time, however there is
no significant change in correlations across cohorts.
90
Discussion
The current study aimed to assess the effectiveness of monitoring patient progress and
providing feedback about wellbeing in an inpatient and day patient psychiatric setting. It was
proposed that those patients who received feedback on their response to treatment would exhibit
significant improvements in symptom relief and wellbeing, when compared to those who did
not receive feedback. The study found, unexpectedly, that providing feedback on wellbeing did
not significantly improve patients’ wellbeing scores at conclusion of the group. This result
represents a divergence from previous findings in patient-focused research that have illustrated
large treatment effects for feedback when measured against the same questionnaire at outcome
(Hawkins et al., 2004; Lambert et al., 2001; Lambert et al., 2002; Whipple et al., 2003) and
therefore warrants further consideration.
There are a number of plausible reasons for this discrepancy. It may be the case that
wellbeing is a construct less susceptible to the effects of feedback, or that it requires a longer
period of time to realise significant change. The stages of change outlined in the Phase Model
(Howard, Lueger, Maling, & Martinovich, 1993) suggest that an improvement in general
positive wellbeing and function occur in a later stage of therapy, following periods of
remoralization and remediation of symptoms. Thus, the time-intensive program of measurement
required for this study, as opposed to the weekly points of measurement depicted in previous
studies, may not capture changes in wellbeing required to show a significant improvement
resulting from feedback.
Another reason may be that the WHO-5 scores illustrate a ceiling effect which may
indicate that a large proportion of the sample move into the ‘well’ range early in therapy and
thus their difficulties are not captured by the measure. To test this hypothesis, the effects of
feedback were assessed on two measures of symptom distress.
In an extension of previous studies in the field of patient-focused research, the
effectiveness of feedback in reducing patients’ presenting symptoms was assessed on two
convergent measures: the DASS-21 and the mental health subscales of the SF-36. Traditionally,
the effects of feedback have been investigated using a single measure for monitoring and
outcome assessment (Harmon et al., 2007; Lambert et al., 2001; Lambert et al., 2002; Whipple
91
et al., 2003). Analyses revealed that those patients who were not on track for improvement in
therapy, but received feedback on their progress demonstrated a significant improvement in
depression, vitality and role emotion scores. For a predominantly depressive disordered group,
these treatment gains are of great consequence. The gains suggest that providing depressed
patients with feedback that they are not improving as expected presents an opportunity for a
crucial change in direction. Importantly, these findings demonstrated clinical impact in that
deterioration rates in therapy reduced from 5.3% to 3.3% when feedback was provided. This
finding complements and extends Lambert and colleagues’ work in patient-focused research
that has illustrated the significant benefits of providing feedback to clinicians and patients
during therapy (Harmon et al., 2007). Thus an improvement in depressive symptoms is evident
for those patients demonstrating poor progress when the feedback and non-feedback groups
received the same quantity of therapy.
In contrast, feedback did not significantly improve outcomes as measured on the broad
mental health and social function subscales of the SF-36, or the anxiety and stress subscales of
the DASS-21. Thus, the benefits of feedback appear limited to the symptoms which were the
principal reasons for treatment. This pattern of results is consistent with previous findings that a
reduction in depressive symptoms may not necessarily generalize to improvements in general
function (Bolton et al., 2007; Hirschfeld et al., 2000; Lin et al., 2000). Furthermore, the
improved outcomes associated with feedback were not evident when measured on the clinician-
rated HoNOS. Interestingly, a trend in the data suggested that clinicians rated ‘alarm’ patients
worse off, regardless of whether they received feedback, which suggests that they are attuned to
their patients’ symptoms, despite the low correlations between clinician ratings and patient self-
report outcomes.
The current study aimed to extend the field of patient-focused research by assessing the
use of feedback in an inpatient and day patient setting, using a group therapy format.
Accordingly, a number of discrepancies exist between the current research design and previous
work that may account for the alternative findings. First, for the current study, feedback was
provided within a group format. Previous attempts to provide feedback within a group format
have not used individualized feedback, and have failed to demonstrate any clinical benefit
92
(Davies, Burlingame, Johnson, Gleave, & Barlow, 2008). Group treatment differs from
individual treatment in that the individual is able to process their own response to treatment in
the context of others’ progress. Group therapy offers a dynamic environment in which a
patient’s lack of change may be highlighted or normalized, and achievements encouraged,
depending on the progress of the group in which they are treated. Thus the process of feedback
within group settings may be a qualitatively difference experience for patients and clinicians,
and this requires further examination.
Second, inpatient and day patient therapy differs to outpatient therapy in that it is time-
limited, the therapy provided is intensive, and the severity of the sample is likely to be greater,
which may limit the utility of feedback in this setting. Lambert and colleagues’ work suggests
that treatment for those patients deemed on track during therapy concludes sooner than those
not on track when feedback is provided (Lambert, 2007). Thus when the dose of therapy is
fixed, it appears that feedback presents the opportunity to attend more closely to those patients
not responding.
Third, the current study employed the WHO-5 as a measure of patient progress, which
is a freely-available, quick, reliable and valid measure of individual progress and outcome in
psychiatric settings (Newnham et al., 2009); but has a ceiling effect. The low cut-off for
movement into the healthy range means that a large proportion of patients’ distress is not
captured by the measure, and similarly, the range of movement is restricted. It is recommended
that future patient-focused research includes the measurement of symptoms which may be more
sensitive to acute changes. In addition to this point, outcome on the WHO-5 was measured at
the beginning of Day 9, which does not accurately reflect the patient’s health status at
conclusion of the group (end of Day 10). Thus, the symptom measures portray outcome more
precisely.
The present study was conducted with a historical cohort design, and thus patients were
not randomized to conditions. The reliable finding that progress monitoring and feedback
improves outcomes has been demonstrated in a series of randomized controlled trials (Harmon
et al., 2007; Hawkins et al., 2004; Lambert et al., 2001; Whipple et al., 2003), and therefore the
current study moved to extend the patient-focused methodology to a novel setting. Whilst the
93
study design reflects the real-world nature of the setting, conclusions cannot be drawn on the
reliability of improvements in symptom reduction among inpatients and day patients without
rigorous follow-up.
Patient-focused research has the potential to bridge the scientist-practitioner gap and
improve patient outcomes (Howard, Moras, Brill, Martinovich, & Lutz, 1996; Lutz, 2003). The
current findings provide limited support for this proposition. Patient monitoring and feedback
presents an opportunity to improve safety issues, increase the reliability of outcome
measurement, and foster a more collaborative relationship between clinician and patient. In this
regard, the patient is informed about their progress in relation to their expected progress, and
accordingly, empowered to make decisions about their treatment management; an objective that
should be highlighted more frequently in psychotherapy research.
Acknowledgments
The authors would like to thank Mrs Moira Munro, Perth Clinic, for ongoing support and
assistance. This study was supported by a grant from the Medibank Private Safety and Clinical
Improvement Incentive Pool, and a PhD Completion Scholarship awarded by the University of
Western Australia.
94
References
Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric
properties of the 42-item and 21-item versions of the Depression Anxiety and
Stress Scales (DASS) in clinical groups and a community sample. Psychological
Assessment, 10, 176-181.
Bech, P., Gudex, C., & Johansen, K. S. (1996). The WHO (Ten) Well-being Index: Validation
in Diabetes. Psychotherapy and Psychosomatics, 65, 183-190.
Bolton, P., Bass, J., Betancourt, T., Speelman, L., Onyango, G., Clougherty, K. F., et al. (2007).
Interventions for depression symptoms among adolescent survivors of war and
displacement in Northern Uganda. Journal of the American Medical Association,
298, 519-527.
Bonsignore, M., Barkow, K., Jessen, F., & Heun, R. (2001). Validity of the five-item WHO
Wellbeing Index (WHO-5) in an elderly population. European Archives of
Psychiatry and Clinical Neuroscience, 251(Supplement 2), II27-II31.
Crawford, J., & Henry, J. D. (2003). The Depression Anxiety Stress Scales (DASS): Normative
data and latent structure in a large non-clinical sample. British Journal of Clinical
Psychology, 42, 111-131.
Davies, D. R., Burlingame, G. M., Johnson, J. E., Gleave, R. L., & Barlow, S. H. (2008). The
effects of a feedback intervention on group process and outcome. Group Dynamics:
Theory, Research, and Practice, 12, 141-154.
Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). The psychotherapy dose-response
effect and its implications for treatment delivery services. Clinical Psychology:
Science and Practice, 9, 329-343.
Harmon, C., Lambert, M. J., Smart, D. W., Hawkins, E. J., Nielsen, S. L., Slade, K., et al.
(2007). Enhancing outcome for potential treatment failures: Therapist/client
feedback and clinical support tools. Psychotherapy Research, 17(4), 379-392.
95
Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K., & Tuttle, K. (2004). The effects
of providing patient progress information to therapists and patients. Psychotherapy
Research, 14, 308-327.
Hirschfeld, R. M. A., Montgomery, S. A., Keller, M. B., Kasper, S., Schatzberg, A. F., Moller,
H. J., et al. (2000). Social functioning in depression: A review. Journal of Clinical
Psychiatry, 61, 268-275.
Howard, K. I., Lueger, R. J., Maling, M. S., & Martinovich, Z. (1993). A phase model of
psychotherapy: Causal mediation of outcome. Journal of Consulting and Clinical
Psychology, 61, 678-685.
Howard, K. I., Moras, K., Brill, P., Martinovich, Z., & Lutz, W. (1996). Evaluation of
psychotherapy: Efficacy, effectiveness, and patient progress. American
Psychologist, 51, 1059-1064.
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining
meaningful change in psychotherapy research. Journal of Consulting and Clinical
Psychology, 59, 12-19.
Lambert, M. J. (2007). Presidential address: What we have learned from a decade of research
aimed at improving psychotherapy outcome in routine care. Psychotherapy
Research, 17(1), 1-14.
Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing
feedback to psychotherapists on their patients' progress: Clinical results and
practice suggestions. Journal of Clinical Psychology, In Session 61(2), 165-174.
Lambert, M. J., & Hawkins, E. J. (2004). Measuring outcome in professional practice:
Considerations in selecting and using brief outcome instruments. Professional
Psychology: Research and Practice, 35(5), 492-499.
Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S. L., & Hawkins, E.
J. (2001). The effects of providing therapists with feedback on patient progress
during psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49-
68.
96
Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L.,
et al. (2002). Enhancing psychotherapy outcomes via providing feedback on patient
progress: A replication. Clinical Psychology and Psychotherapy, 9, 91-103.
Lin, E. H. B., VonKorff, M., Russo, J., Katon, W., Simon, G. E., Unützer, J., et al. (2000). Can
depression treatment in primary care reduce disability? A stepped care approach.
Archives of Family Medicine, 9, 1052-1058.
Lovibond, S. H., & Lovibond, P. F. (1995a). Manual for the Depression Anxiety Stress Scales
(2nd ed.). Sydney: Psychology Foundation.
Lovibond, S. H., & Lovibond, P. F. (1995b). The structure of negative emotional states:
Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck
Depression and Anxiety Inventories. Behavior Research and Therapy, 33, 335-343.
Löwe, B., Spitzer, R. L., Grafe, K., Kroenke, K., Quenter, A., Zipfel, S., et al. (2004).
Comparative validity of three screening questionnaires for DSM-IV depressive
disorders and physician's diagnoses. Journal of Affective Disorders, 78, 131-140.
Lutz, W. (2003). Efficacy, effectiveness, and expected treatment response in psychotherapy.
Journal of Clinical Psychology, 59(7), 745-750.
Lutz, W., Leach, C., Barkham, M., Lucock, M., Stiles, W. B., Evans, C., et al. (2005).
Predicting change for individual psychotherapy clients on the basis of their nearest
neighbors. Journal of Consulting and Clinical Psychology, 73(5), 904-913.
Lutz, W., Stulz, N., & Köck, K. (2009). Patterns of early change and their relationship to
outcome and follow-up among patients with major depressive disorders. Journal of
Affective Disorders, doi:10.106/j.jad.2009.2001.2019.
McHorney, C. A., & Tarlov, A. R. (1995). Individual-patient monitoring in clinical practice:
Are available health surveys adequate? Quality of Life Research, 4, 293-307.
National Centre for Classification in Health Publications. (2002). The International Statistical
Classification of Diseases and Related Health Problems, Tenth Revision,
Australian Modification (ICD-10-AM). Geneva: National Centre for Classification
in Health Publications.
97
Newnham, E. A., Harwood, K. E., & Page, A. C. (2007). Evaluating the clinical significance of
responses by psychiatric inpatients to the mental health subscales of the SF-36.
Journal of Affective Disorders, 98, 91-97.
Newnham, E. A., Hooke, G. R., & Page, A. C. (2009). Monitoring treatment response and
outcomes using the World Health Organization's Wellbeing Index in psychiatric
care. Journal of Affective Disorders, doi:10.1016/j.jad.2009.1006.1005.
Newnham, E. A., & Page, A. C. (2007). Client-focused research: New directions in outcome
assessment. Behaviour Change, 24(1), 1-6.
Page, A. C., & Hooke, G. R. (2003). Outcomes for depressed and anxious inpatients discharged
before or after group cognitive behavior therapy: A naturalistic comparison.
Journal of Nervous and Mental Disease, 191(10), 653-659.
Page, A. C., Hooke, G. R., & Rampono, J. (2005). A methodology for timing reviews of
inpatient hospital stay. Australian and New Zealand Journal of Psychiatry, 39, 198-
201.
Page, A. C., & Stritzke, W. G. K. (2006). Clinical Psychology for Trainees: Foundations of
Science-Informed Practice. Cambridge, UK: Cambridge University Press.
Parabiaghi, A., Barbato, A., D'Avanzo, B., Erlicher, A., & Lora, A. (2005). Assessing reliable
and clinically significant change on HoNOS: A method for displaying longitudinal
data. Australian and New Zealand Journal of Psychiatry, 39(719-724).
Scott, K. M., Tobias, M. I., Sarfati, D., & Haslett, S. (1991). SF-36 health survey reliability,
validity and norms for New Zealand. Australian and New Zealand Journal of
Public Health, 23, 401-406.
Ware, J. E., Snow, K. K., Kosinski, M., & Gandek, B. (1993). SF-36 Health Survey: Manual
and Interpretation Guide. Boston: The Health Institute, New England Medical
Centre.
Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E.
J. (2003). Improving the effects of psychotherapy: The use of early identification of
treatment failure and problem solving strategies in routine practice. Journal of
Counseling Psychology, 58, 59-68.
98
Wing, J. K., Beevor, A. S., Curtis, R. H., Park, S. B. G., Hadden, S., & Burns, A. (1998). Health
of the Nation Outcome Scales (HoNOS): Research and development. British
Journal of Psychiatry, 172, 11-18.
99
Figure 1.
Example feedback graph of WHO-5 scores for a patient responding to feedback at day 5.
100
Table 1
Log linear upper and lower scores for the expected treatment response trajectories for feedback.
WHO-5 Score at Admission Day 1 Day 3 Day 5 Day 7 Day 9
0-3 Upper 3.43 7.66 10.13 11.89 13.25
Lower 1.64 3.97 5.33 6.30 7.05
4-6 Upper 6.65 10.54 12.81 14.43 15.68
Lower 4.95 6.97 8.15 8.99 9.64
7-9 Upper 8.79 12.22 14.23 15.66 16.76
Lower 7.37 8.62 9.35 9.87 10.27
10-13 Upper 12.20 14.45 15.77 16.71 17.43
Lower 10.27 11.11 11.61 11.96 12.23
14-25 Upper 17.54 18.85 19.62 20.17 20.59
Lower 14.73 14.65 14.60 14.57 14.55
101
Table 2
Means and standard deviations (in parentheses) for each cohort across all outcome measures.
Measure Control No Feedback Feedback
WHO-5 Day 1 8.10 (5.15) 8.12 (5.31)
Day 9 13.21 (5.93) 12.98 (6.07)
DASS-21
Depression (Pre) 26.04 (12.92) 25.28 (13.17) 24.13 (12.69)
Depression (Post) 13.08 (11.29) 13.86 (11.07) 12.40 (10.79)
Anxiety (Pre) 20.00 (12.00) 19.24 (11.36) 19.57 (11.94)
Anxiety (Post) 11.41 (9.77) 11.41 (9.37) 10.92 (9.37)
Stress (Pre) 26.93 (11.38) 26.29 (11.23) 26.03 (11.19)
Stress (Post) 15.95 (10.47) 16.37 (10.67) 15.21 (10.07)
SF-36 Mental Health (Pre) 42.00 (20.68) 43.46 (20.65) 43.69 (20.98)
Mental Health (Post) 62.21 (19.92) 64.27 (19.90) 65.70 (19.49)
Vitality (Pre) 32.00 (21.81) 33.39 (21.68) 33.94 (22.69)
Vitality (Post) 48.88 (22.33) 51.01 (22.24) 52.77 (22.10)
Role Emotion (Pre) 28.25 (36.77) 26.55 (35.89) 26.54 (35.41)
Role Emotion (Post) 57.05 (41.23) 58.40(41.47) 60.40 (40.16)
Social Function (Pre) 38.71 (27.00) 40.33 (26.12) 40.47 (26.81)
Social Function (Post) 61.73 (27.39) 61.82 (27.69) 65.58 (25.86)
HoNOS Total (Pre) 11.17 (5.50) 10.32 (4.53) 12.15 (3.89)
Total (Post) 5.44 (4.06) 5.13 (3.39) 6.65 (4.05)
102
Table 3
Correlations between HoNOS post-treatment scores and patient self-report outcome measures across cohorts.
WHO-5 Day 1
WHO-5 Day 9
Depression Anxiety Stress Role Emotion
Vitality Mental Health
Social Function
HoNOS
Score
Control - - .385** .350** .326** -.276** -.297** -.390** -.334**
No Feedback -.218** -.397** .467** .366** .444** -.366** -.373** -.441** -.482**
Feedback -.275** -.476** .518** .485** .477** -.434** -.417** -.549** -.498**
** p<.01
103
Section Four General Discussion
104
General Discussion____________________________________________
Providing optimal healthcare is a dynamic process. It is vital that we act upon the best
scientific information available to improve patient care, with an ongoing openness to new
initiatives and findings so that the bridge between science and practice is continually
maintained. In doing so, we must implement systems of quality improvement for psychological
services, and incorporate new advances in knowledge, techniques and technologies as they
become accessible. At a treatment level, this evidence-based evolution of clinical practice is
epitomized in patient-focused research – the process of monitoring patient progress and
providing feedback on the patient’s treatment response. The ongoing assessment of patients’
health status allows clinicians to act upon deviations from expected response, and accordingly
alter the course of treatment. It enables the ongoing incorporation of knowledge with a view to
improve patient outcomes.
The current thesis outlined a program of patient-focused research, and in doing so,
aimed to address three important questions in psychotherapy research:
1. Can we identify poor responders in psychiatric care?
2. Is it possible to track patients’ progress during psychiatric care?
3. What effect does feedback have on clinical outcomes?
To address these questions, a series of studies investigated the validity of existing measures for
the evaluation of clinical outcomes in Australia, before designing and implementing a
monitoring system appropriate for use in psychiatric care, and evaluating its effectiveness. The
specific findings from these studies are outlined below.
Findings_______________________________________________________________
Can we identify poor responders in psychiatric care?
To improve treatment outcomes, it is important to first define recovery and conversely,
identify those patients who do not respond well to treatment. Various estimates of deterioration
rates have been published (Boisvert & Faust, 2003; Hansen, Lambert, & Forman, 2002;
105
Lambert & Ogles, 2004) however, the corresponding values have not been identified for
inpatient hospital-based psychiatric care. The detection of patient groups that are not
progressing well in therapy would enable clinicians to alter future treatment.
Outcomes are routinely measured in Australia via administration of the mental health
subscales of the SF-36 and the Health of the Nation Outcome Scale (HoNOS) (Morris-Yates &
Page, in press; Stedman, Yellowlees, Mellsop, Clarke, & Drake, 1997). Chapter 2 (Newnham,
Harwood, & Page, 2007) employed the Jacobson and Truax (1991) method for calculating
clinical significance, to determine criteria for recovery on the SF-36 for psychiatric inpatients
and day patients. A reliable change index (RCI) was calculated, as well as cut-off scores for
each subscale to convey the point at which a patient has moved from the ‘unwell’ to the ‘well’
range of scores. This information allows clinicians to determine the outcome for each patient, or
graph the pattern of outcomes for all of their patients. Application of the criteria to a sample of
inpatients in Western Australia revealed that between 35.7% and 41.8% recover, 5.5% to 12.8%
improve, 43.6% to 56.6% show no change, and up to 2.1% deteriorated. Thus almost 60% of
patients showed no improvement on one measure of mental health (Vitality) as a result of
inpatient treatment.
Deterioration rates identified in the psychiatric sample are significantly lower than those
shown for outpatient samples for a number of possible reasons. First, the inpatient sample
exhibits more severe psychopathology, and so it is more difficult to illustrate a decline in health
when many people are close to the measurement instrument’s ceiling. Second, inpatient
psychiatric treatment is intensive, and in Australian private psychiatric centres, average length
of stay is 20 days (Morris-Yates & Page, in press), while the hospital where the present research
was conducted is shorter at 12 days (Page & Hooke, 2009). It is therefore difficult to compare
these statistics with those for outpatient therapy which comprises weekly sessions measured in
hours rather than days.
The findings of Chapter 2 (Newnham et al., 2007) are directly applicable to clinical
practice in a variety of ways. Clinicians and mental health services could use the data to
evaluate outcomes and benchmark against other services. This process encourages optimal
clinical practice by enhancing transparency and accountability, highlighting individual
106
differences, and enabling a more meaningful approach to evaluating treatment effectiveness.
There are however, a number of different methods for calculating clinical significance
(Hageman & Arrindell, 1999; Hsu, 1989; Speer, 1992) and these would produce alternative
results. The Jacobson and Truax method was used in conjunction with recommendations for its
routine use in clinical practice (Lambert, Hansen, & Bauer, 2008), and convergent findings with
measures of quality of life and psychopathology outlined in Chapter 2 provide validity for this
approach. However, when assessing outcomes in mental healthcare, it important to consider a
variety of perspectives.
Fortunately, in Australian mental health services, self-reported outcomes are
complemented by clinician-rated scores. The HoNOS provides clinically-meaningful
information on a variety of health and psychosocial domains, and is widely used as a measure of
treatment outcome (Wing, Lelliott, & Beevor, 2000). Due to the diversity of the items, a total
aggregate score or individual items are often used to interpret the HoNOS; however, a new
parsimonious and clinically-useful subscale structure has emerged (Newnham, Harwood, &
Page, 2009). The third chapter of the thesis outlined the psychometric and clinical validity of the
HoNOS modified four-factor scoring model, which is applicable to psychiatric settings that treat
patients with predominantly depressive symptoms. The modified four factor model
demonstrated greater sensitivity to change, higher internal consistency, parsimony, and a better
fit to both the sample in which it was developed, and a second test sample, than previously
proposed models. In addition, the new model should facilitate the swift scoring of HoNOS
questionnaires, and the straightforward interpretation of the scores, enhancing its clinical utility.
The findings of Chapter 3 are important because the HoNOS is widely used both nationally and
internationally (e.g. Eager, Trauer, & Mellsop, 2005; Parabiaghi, Barbato, D'Avanzo, Erlicher,
& Lora, 2005; Pirkis, Burgess, Kirk, Dodson, & Coombs, 2005; Wing et al., 2000), and the
scores provide a different perspective on treatment outcomes, encompassing the clinician’s view
of a patient’s health status. As outlined in Chapter 6, the HoNOS exhibits a small to moderate
correlation with self-report measures of health status, thus providing more information than self-
report alone.
107
The HoNOS accounts for clinical significance in its scoring procedure (i.e., a score of 2
or more on any item indicates clinically significant distress or dysfunction), and Parabiaghi and
colleagues (2005) have investigated the use of clinical significance with longitudinal HoNOS
data; thus the issue of clinical significance was not further investigated for the current thesis.
In summary, the first section of the thesis outlines the clinical value of two routinely
used measures of outcome which provide not only a valid means for assessing treatment
effectiveness, but also enable benchmarking across services. Chapters 2 and 3 described
recovery criteria and scoring methods respectively; that would facilitate the use of patients’
scores to best inform treatment outcomes. In doing so, use of the SF-36 mental health subscales
and the HoNOS was validated in psychiatric outcome assessment. Further to this, a group of
poor responders could be identified using the clinical significance calculations outlined in
Chapter 2. This group of patients clearly requires intervention during therapy to address poor
treatment progress. The findings of Section One thus enable the valid and rigorous assessment
of whether a feedback intervention in psychiatric care improves clinical outcomes.
Is it possible to track patients’ progress during psychiatric care?
The second chapter of the thesis (Newnham et al., 2007) depicts the small but
noteworthy proportion of poor responders in psychiatric care, and highlights the level of need
for new innovations that complement effective treatments and improve outcomes for those
particular patients. This finding is further supported by the calculations of clinically significant
outcomes for wellbeing scores described in Chapter 5 (Newnham, Hooke, & Page, 2009).
Building upon the strong foundations for patient-focused research outlined by Michael Lambert,
Wolfgang Lutz and their colleagues (e.g. Howard, Moras, Brill, Martinovich, & Lutz, 1996;
Lambert, 2007; Lutz et al., 2006), a system for monitoring patient progress in a psychiatric
hospital setting was developed. Whilst incorporating the processes that have demonstrated
clinical effectiveness in a number of randomized controlled trials (Harmon et al., 2007;
Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004; Lambert et al., 2001; Lambert et al.,
2002; Whipple et al., 2003), the system was designed to address the nuances of a novel clinical
setting. The hospital setting required a program that demonstrated reliability, validity and
108
sensitivity to change over a short time-frame (due to the intensive nature of inpatient and day
patient care), it had to be freely-available, quick to administer, and demonstrate robust
psychometric properties. Chapter 5 (Newnham, Hooke et al., 2009) outlines the process of
development and Chapter 6 (Newnham, Hooke, & Page, submitted), the implementation of the
monitoring program.
The present research program employed the World Health Organization’s Wellbeing
five-item Index (WHO-5; Bech, Gudex, & Johansen, 1996) to monitor patients’ wellbeing
across a ten-day cognitive behavioural therapy group. The WHO-5 is a freely-available measure
of positive wellbeing that demonstrated reliability, validity and sensitivity to change in a
psychiatric sample (Newnham, Hooke et al., 2009). The ongoing monitoring of patient progress
was implemented in the hospital in a historical cohort design, consistent with quality
improvement efforts. Given the literature on implementation science, it was anticipated that
clinicians would raise concerns about the system’s impact on the therapeutic process, as well as
concerns that it would affect the amount of control clinicians’ may have over the treatment plan,
and burden them with extra tasks (Grizenko, Poinsier, Mercier, Perreault, & Latimer, 2000).
To address these points, we worked towards giving clinicians a voice in the design and
implementation of the program. This transpired in seven main actions.
1. Prior to the implementation of the feedback trial, we ran a series of presentations at
the hospital, to describe the rationale and methods of the program, for therapists,
nursing staff, doctors and the executive board, to ensure that all contributors to the
treatment team were aware of the program, had an opportunity to ask questions and
make suggestions for improvement. The presentations were followed up with
handouts that comprised timelines, a rationale statement, a review of the literature
(Chapter 4: Newnham & Page, 2007), and example patient graphs. This facilitated
communication between therapists, nurses and doctors about the feedback graphs
and their impact potential on the treatment plan.
2. The feedback trial was initiated in a small sample comprising only the CBT group at
the hospital. Two CBT groups run concurrently, with two therapists each.
Accordingly, we were working with only four therapists (with little turnover during
109
the trial period) and thus clinical concerns could be attended to quickly and often
among the therapists themselves. Starting small enabled us to establish an effective
and efficient system.
3. The timing of feedback was decided upon with the assistance of empirical data and
clinical opinion. The findings highlighted in Chapter 5 indicated that Day 5
Wellbeing scores (i.e., at the mid-point of therapy) significantly predicted outcome,
and the possibility of feedback on Day 5 was discussed with clinicians in a series of
meetings. Therapy staff agreed that Day 5 (Friday of the first week of therapy)
would be an optimal point to provide feedback as it presented an opportunity to
discuss the week’s progress, set homework tasks for the weekend, and validate or
modify the treatment goals for the coming week. Accordingly, clinicians were
encouraged to take greater control over the use of the feedback graphs, and this
promoted their engagement in the research trial.
4. The monitoring program was run first as a pilot trial which provided the opportunity
for clinicians to raise any concerns about the methodology and have them addressed
prior to the commencement of the research trial.
5. Once the trial was underway, regular clinical management meetings were held for
the therapy staff to discuss any concerns or difficulties they had experienced with
providing feedback to patients. Early in the trial period, clinicians raised concerns
about telling patients that they were ‘not on track’, fearing that it would dishearten
patients and cause them to feel as though they had “failed”. The clinical management
meetings were thus employed to normalize these concerns, reaffirm the rationale and
previous findings in the area (e.g. Harmon et al., 2007; Lambert, 2007) and problem
solve difficult cases and patient queries. A patient’s resistance to feedback may
highlight their concerns about the accuracy or appropriateness of the information,
and discussions about its interpretation are likely to enrich the therapeutic process
(Claiborn & Goodyear, 2005). Thus, the process of feedback is important, even if the
content is rejected; and this sentiment was emphasized with therapists. A team
approach was used to communicate ideas in the clinical management meetings, so as
110
to avoid directive guidance of the clinical staff. The meetings also provided an
opportunity to ensure adherence to protocol throughout the trial period.
6. The use of information technology such as touch-screen computers to administer the
WHO-5 enabled reliable and immediate data entry and feedback of results. There is
potential for information technology to play a large role in the implementation of
quality improvement programs in healthcare, and this will be expanded in the Future
Directions section of the General Discussion. However, even without the use of
technology to enable monitoring, reliance on administrative assistance is beneficial
in lessening the burden on clinicians’ time.
Accordingly, the measures taken to involve clinicians in the design of the monitoring
program and engage them in the implementation, significantly enhanced the management of the
research trial. In a reflection of the clinicians’ involvement, and the monitoring program’s
impact on clinical outcomes, it has since been extended to the other therapy groups offered by
the hospital. Tailoring the feedback program to the particular setting in which it is to be
implemented is vital. Without the inclusion of staff perspectives in quality improvement
programs and health services research, implementation within health settings can be difficult, if
not impossible (Grol & Grimshaw, 2003).
Chapters 5 and 6 describe the first program for individual feedback conducted with
group therapy. Graphs were created for each patient in the feedback cohort, depicting their
progress scores across the duration of therapy, from which they could assess their rate of
improvement and make changes to their treatment plan. Feedback was provided as individual
graphs, accompanied by a written description. Therapists complemented this process in session
by describing potential shapes of change and what they might mean, and provided an
opportunity for patients to share their views on their own progress, or ask questions, should they
wish to. Further opportunity to discuss the graphs privately with the therapists was available to
all patients. Therapists reported spending between five and thirty minutes discussing feedback
in session, dependent upon the level of interest within the group.
The wellbeing monitoring program demonstrated strong psychometric properties within
a psychiatric setting, as described in Chapter 5. Despite not having been used in psychiatric care
111
previously, the WHO-5 exhibited reliability, convergent validity with measures of
psychopathology and mental health, and specificity to depressive symptoms at discharge.
Clinical significance calculations revealed a cut-off score appropriate for the definition of
recovery in a psychiatric sample. The findings outlined in Chapter 5 enable the straightforward
execution of the wellbeing monitoring system in new settings. They highlight benchmarks for
clinicians to judge progress and outcome for individual patients when using the widely available
WHO-5 in mental healthcare.
Expected treatment response curves were created for the purpose of tracking patient
progress and alerting clinicians to “alarm” responses during therapy. These trajectories were
calculated according to the log-linear curves of the standard deviation around the group mean,
for each of five severity groups. The simplicity of this approach has advantages and limitations.
It provided clear and meaningful guides to treatment progression for clinicians and patients in
the current study, and it may be easily replicated in other data samples, however, sophistication
in prediction of patient profiles and accuracy of modelling were sacrificed. The modelling of
patient progress data would be greatly enhanced by multilevel modelling (Tasca & Gallop,
2009), or the nearest neighbour approach outlined by Lutz and colleagues (2005), which would
generate a more accurate depiction of the patient’s expected trajectory during the course of
therapy. With the addition of more data on progress in psychiatric care, these methods should be
explored in the future.
One limitation with the present research is that the WHO-5 is a measure of positive
wellbeing and thus it does not capture a full range of psychopathology or distress that patients
may present with. It is not intended to be a comprehensive and all-encompassing measure of
health status, rather, a perspective to complement clinical judgment. Initially, we had hoped to
achieve the same benefits that had been achieved with symptom measures, with a succinct and
quick-to-administer positive wellbeing assessment. However, it appears that incorporation of
both symptom and wellbeing information would provide a more comprehensive picture of
patient progress. Nonetheless, it was hoped that the monitoring process would help patients to
verbalise issues that they may not otherwise raise in therapy (perhaps in particular, group
112
therapy). Thus, the feedback is not an objective measure of the patient’s health, but a
supplement to the therapeutic process.
Chapter 5 therefore suggests that it is possible to track patient progress during group
psychotherapy, and that this information can be assembled as feedback, which leads to the
question of whether this process is effective in improving outcomes for those patients who do
not respond to treatment.
What effect does feedback have on clinical outcomes?
The clinically significant improvements resulting from progress feedback have been
well-documented in randomised controlled trials conducted in outpatient psychotherapy
(Harmon et al., 2007; Hawkins et al., 2004; Lambert et al., 2001; Lambert et al., 2002; Whipple
et al., 2003). It is therefore time to extend these findings to novel settings, such as psychiatric
hospital services, and examine their effectiveness. The sixth chapter outlines the effectiveness of
feedback in a sample exhibiting three characteristics not previously examined: Australian
patients seen in a hospital setting serving inpatients and day patients attending group therapy.
Extension of the feedback methodology to an inpatient setting resulted in a significant
improvement in clinical outcomes. Those patients who were not on track at Day 5 demonstrated
significantly greater improvements in depression, vitality and role emotion when given
feedback on their progress. Further to this, the rate of deterioration in vitality was reduced from
5.3% to 3.3%. The finding that the treatment of depression can be complemented by monitoring
and feedback to assist poor responders is noteworthy for mental healthcare. In Australia,
depression is the most common mental health difficulty reported (Mathers, Vos, & Stevenson,
1999) and it is costly to treat, with an estimated 48% of patients rehospitalized within five years
(Daniels, Kirkby, Hay, Mowry, & Jones, 1998). By improving outcomes specifically for those
patients who do not appear to respond well to psychotherapy; the costs (both financial and
personal) of depression in Australia could be greatly reduced.
Conversely, feedback did not improve outcomes when assessed on broad measures of
mental health and wellbeing, nor measures of anxiety and stress. There are three potential
reasons for why no change was evident in wellbeing scores. First, the final point of
113
measurement on the WHO-5 was at the beginning of Day 9, a rating of the previous 24 hours.
Accordingly, patients’ status over Day 8 of the ten-day group is unlikely to be representative of
their final outcome, and thus the other measures (SF-36 and the DASS-21, administered on Day
10 of group) present a more typical illustration of outcome. Second, the WHO-5 demonstrates a
ceiling effect in the present sample, so that many patients may have moved into the “well” range
early in therapy, and their distress not identified on a measure of positive wellbeing. Third, it
may be that wellbeing is less receptive to the effects of feedback. Information concerning a
patient’s progress may not be sufficient to effect change in their general wellbeing. Rather, the
feedback appears to elicit changes in mood and vitality, and more time may be required to
improve general functioning and quality of life as measured on the WHO-5 and mental health
subscale of the SF-36. Similarly, no significant benefits of feedback were evident on measures
of anxiety and stress. This may reflect the smaller proportion of patients presenting with
primary diagnoses of anxiety or stress, but could also indicate that these symptoms are more
difficult to improve with feedback. Further investigation into the effects of feedback for patient
samples exhibiting mostly anxiety disorders may provide more detail on its effectiveness within
that population.
Inpatient group psychotherapy is an effective means of treating depression (Koesters,
Burlingame, Nachtigall, & Strauss, 2006). The American Psychological Association’s Clinical
Practice Guidelines for Group Psychotherapy (Bernard et al., 2008) recommend the ongoing
monitoring of patient progress, and use of feedback to improve outcomes. However, despite
recommending a battery of tests (Strauss, Burlingame, & Bormann, 2008) that may be used with
groups to assess process, engagement and outcome, no evidence for use of monitoring in group
psychotherapy was cited. Feedback has been trialled in group outpatient psychotherapy on one
occasion previously, with little impact on clinical outcomes (Davies, Burlingame, Johnson,
Gleave, & Barlow, 2008). Indeed, the provision of group process and cohesion feedback to
group members was ineffective in improving outcomes, and harmful to those in high conflict
groups. However, Davies and colleagues (2008) did not provide individualized feedback, and it
may be the disconnect between feedback and treatment goals that failed to elicit an
114
improvement in outcomes. Thus, Chapter 6 highlights the first trial of feedback in a group
format to make a positive impact on treatment outcomes.
Group therapy is subject to different dynamics to those that play out in individual
therapy. The cohesion of the group and individuals’ engagement in the group process appear to
interact with feedback processes (Davies et al., 2008), however it is unclear what role these
therapeutic factors play. It may be the case that highly cohesive groups encourage the use of
feedback, or discourage it; or that a person’s progress in relation to the rest of the group may
alter their interpretation of individualised scores. Accordingly, the qualitative processes
involved in providing progress feedback with group psychotherapy are an intriguing area
requiring investigation.
Australian mental healthcare is moving towards an era of improvements based on
outcomes information (Australian Mental Health Outcomes and Classification Network, 2005).
Psychiatric care accounts for a significant proportion of mental healthcare, and private hospitals
account for one quarter of all psychiatric overnight admissions (Morris-Yates & Page, in press).
Of these, the majority of admissions are presentations of depressive disorders (Morris-Yates &
Page, in press). The findings of the thesis suggest that monitoring patient progress is not only
possible in Australian private psychiatry; the process improves clinical outcomes for those
exhibiting depressive symptoms. The wider application of this work is an important addition to
evidence-based practice. However, it is unclear as yet whether monitoring progress is possible
with all patient groups. Those patients experiencing difficulties with insight into their mood or
behaviours (including those with psychotic symptoms or personality disorders), may not be able
to provide valid information on their treatment progress. Further to this, it is not clear how
feedback is interpreted by patients, and this raises questions about whether it would be as
beneficial for other patient samples. In fact, very little is known about the process of feedback,
or how it is received and used to alter behaviour.
Future research_________________________________________________________
How does feedback work?
115
Feedback communicates information about a behaviour that has occurred, and
influences the probability and nature of its recurrence (Claiborn & Goodyear, 2005). Thus
feedback is intended to shape behaviour, through the recipient’s acceptance, partial acceptance
or rejection of the information. While feedback is a routine process inherent in all domains of
psychotherapy, the provision of formalized feedback to patients, with a view to improve the
treatment offered, is a relatively new technique. The feedback program outlined in Chapters 5
and 6 (Newnham, Hooke et al., 2009; Newnham et al., submitted) used descriptive and
interpretive feedback outlining the individual patient’s pattern of progress, and how it related to
similar patients’ progress. Very little research has been conducted within psychotherapy to
inform the necessary components required for feedback to be useful; and the way feedback is
interpreted has rarely been investigated within clinical samples. However, the few studies that
have examined the effects of various feedback formats provide some clues as to the
characteristics required for effective behavioural change (Claiborn & Goodyear, 2005).
For feedback to be optimally received, it should be clear and provide information that is
useful and of personal relevance to the receiver. It should come from an influential source, who
is judged to be not only an authority, but also trustworthy (Sapyta, Reimer, & Bickman, 2005;
Strong & Matross, 1973). Thus the therapeutic relationship is of great importance to the process
of change. Without a trusting and secure relationship, feedback may not be well received, and
without a shared curiousness, modification of the treatment plan may not be considered
(Claiborn & Goodyear, 2005). Despite this, the course of feedback does not require immediate
and total acceptance of the information, in fact, critical interpretation of the message is the key
to active participation and thus insight into one’s mood and behaviours. Another important
factor suggested to be critical in the effectiveness of feedback is the accompaniment of progress
information with a suggested course of action. Davies and colleagues (2008) proposed that their
group feedback program may not have worked because feedback was not individualized, and
did not identify possible options for enacting improvement. The study outlined in Chapter 6
(Newnham et al., submitted) responded to both issues. By providing individualized feedback on
wellbeing status (rather than group cohesion) and encouraging the use of this information in
116
treatment planning and evaluation of progress, outcomes were significantly improved for those
patients who were unlikely to respond otherwise.
The contextual feedback intervention theory (Sapyta et al., 2005), proposed a model of
feedback that encompasses a self-directed learning theory. It is argued that a number of features
within the therapeutic encounter interact with feedback in order to exact successful behaviour
change. First, the clinician (and patient) must be motivated to reach a goal, and feedback should
highlight their position in relation to attaining it. The goal must be both attractive and realistic.
Considerable planning is required to determine the most appropriate and useful feedback
content, direction and format, so that it meets the needs of the recipient and can be rapidly
processed. Further to this, Sapyta and colleagues (2005) highlight the importance of cognitive
dissonance in the process of behaviour change, but are quick to point out that dissonance does
not necessarily create change in the intended direction. There are a number of ways to reduce
cognitive dissonance, and reduced motivation and engagement in the therapeutic process are
potential outcomes. Continued commitment to the goal, personal responsibility for the outcome,
and a sense of control over the process are all important factors in optimizing the effects of
feedback within clinical settings (Sapyta et al., 2005).
An area yet to receive research attention is how feedback is interpreted, and what
factors influence its interpretation. As highlighted in Chapter 5 (Newnham, Hooke et al., 2009),
mood biases attention to and interpretation of negative information (Mathews, Ridgeway, &
Williamson, 1996). Research conducted with university students demonstrated that those with
dysphoric mood were more likely than those with non-dysphoric mood to attend to negative
messages, and were more suggestible, aligning their answers in a negative direction
(MacFarland & Morris, 1998). It is likely that negative messages fit with a confirmatory bias for
those with low mood (Claiborn & Goodyear, 2005), but this does not explain why feedback
suggestive of “alarm” status helps to turn progress around and improve outcomes. Feedback is
most successful when the information highlights a discrepancy between the current behaviour
and expected or desired outcome (Sapyta et al., 2005). Further clinical research that explores
how feedback interacts with a receiver’s mood, motivation to change, expectations for therapy
and self-esteem is undoubtedly warranted.
117
Further research is also necessary for investigating the qualitative experience of group
psychotherapy members who receive feedback. Group psychotherapy is inherently a social
experience, and implicit feedback occurs continuously at all levels of dynamic interaction.
However, the formal monitoring of progress and individualized feedback to group members
adds a complexity not yet examined. It is unclear whether members compare their progress with
others (in that particular group), and whether this impacts upon their personal interpretation. In
addition, motivation and expectations for therapy may impact upon one’s interpretation of
feedback. Feedback programs have now demonstrated effectiveness in outpatient and inpatient
settings, and with individual and group therapies. Thus, future research should investigate the
qualitative experience of feedback and how it is used by clinicians and patients, in order to
further our understanding of feedback processes and how they can be best used to improve
patient care.
How can information technology best be used to benefit mental healthcare?
The recent surge in information technology calls for the integration of electronic
apparatus in health service delivery and research. To match the evolution of information
technology, access to communication networks has become widespread and more than two
thirds of Australian households have internet access (Australian Bureau of Statistics, 2008). The
implications for health promotion and service delivery stem from American research that
suggests that 80% of internet users report searching for health information, and that this
information has influenced their decisions about healthcare access and interactions with their
doctor (Atienza et al., 2007). While many businesses have adopted recent technologies to
improve efficiency (such as online shopping, telephone and internet banking, and online check-
in and touch-screen kiosks at airports), health services are slowly adapting information and
communication technologies (ICT) to improve access and costs for patients and providers
(Griffiths, Farrer, & Christensen, 2007; Marks, 2004). Considerable opportunities exist for the
use of ICT in assessment and diagnosis, monitoring of patient health, complementing
interventions and the audit of clinical outcomes in mental health.
118
The current thesis adapted touch-screen technology to monitor patients’ wellbeing
during therapy, and provide real-time feedback. The benefits of using touch-screens in the
therapy rooms included: (1) increased interest and engagement with the monitoring process for
patients, because they can see what happens to the questionnaires they complete; (2) greater
reliability and efficiency for the process of data collection, as data are automatically submitted
and collated within an electronic database; (3) reducing the burden on clinical staff to administer
measures; (4) increased security with data encryption, a ‘kiosk’ running mode, and a Wyse “thin
client” terminal that can be locked in a cabinet; (5) greater efficiency in producing feedback
graphs; (6) the amount of missing data is reduced because the touch-screen requires completed
responses to previous items in order to move to the next screen; and (7) the automated growth of
a health database for future clinical audit and research evaluation. The use of touch-screens and
an electronic data network within the clinic also led to a rapid expansion of services integrated
into the technology, such as monitoring group attendance, creating outcome graphs (using the
SF-36 mental health subscales) and the integration of financial management data. Accordingly,
touch-screen technology was widely and enthusiastically adopted within a psychiatric setting,
and this raises the question of how new information technologies can best complement clinical
practice in future.
To date, ICT has been adopted in diagnosis and assessment of patient health, in both
physical and psychological healthcare. An abundance of self-evaluation measures of
psychopathology are now available on the internet, either through healthcare provider or
independent websites (Atienza et al., 2007); and electronic assessment has extended to the use
of touch-screens in waiting rooms (Hunter, Travers, & McCulloch, 2003), and interactive-voice-
response technology available through telephones (Stritzke & Page, 2009). High diagnostic
correspondence has been demonstrated between assessments conducted with interactive-voice-
response systems and clinician-administered structured clinical interviews (Kobak et al., 1997),
and the procedure has potential to improve efficiency and costs per patient (Stritzke & Page,
2009).
Technological advancements have also been incorporated into psychological
interventions. Telepsychiatry is particularly valuable in Australia where regional and remote
119
patients are disadvantaged due to great geographical isolation and difficulty with professional
recruitment, but is not yet being employed to its full potential (McLaren, 2005). Computer-
aided therapies such as the VirtualClinic supported by the Clinical Research Unit for Anxiety
and Depression (Titov, Andrews, & Schwencke, 2008), and the internet-based MoodGYM
program for adolescents (O'Kearney, Kang, Christensen, & Griffiths, 2009) are becoming
increasingly available and have demonstrated effectiveness at lower cost to the patient (Marks,
2004). In addition, doctors are increasingly using the applications available on smart-phones
such as the BlackBerry and Apple’s iPhone to provide immediate and accurate information for
patients, and to communicate with wider treatment teams regarding patient health (i.e. by
sending patient data such as patient charts that can be acted upon immediately) (Bhanoo, 2009).
Internet-based monitoring of patients’ progress is the next step in eHealth, and has the potential
to highlight a patient’s decline in health more quickly than current care procedures. However,
despite the benefits in cost efficiency and ease of use, there is considerable potential for
technological advancements to do harm as well as good.
The continuing advancement of technological improvements to health service provision
will require collaborations between IT specialists, industry partners, government and researchers
(Atienza et al., 2007). It is important that electronic systems are used to engage patients in their
healthcare decisions and promote collaborative management of the treatment plan, rather than
alienate them. The risk of alienation stems from the potential for ICT to increase the differential
of power between doctor and patient, increase the distance in communication between them,
and reduce the patient’s confidence in their own critical assessment. Technological
advancements are only a useful addition to healthcare if the clinician and patient are
comfortable using it. It is vital that technology is used to complement evidence-based practice,
not replace it. There is also a need for further randomized controlled trials conducted by
investigators not commercially invested in the technology. Despite concerns that an increasing
reliance on technology in health will widen the “digital divide,” recent employments of touch-
screen programs in Australia suggest otherwise.
Touch-screen technology provides an easy and interesting way to both deliver and
solicit health information. While the installation and programming of touch-screens incurs a
120
cost, their use can generate an overall saving in time for both health and administrative staff.
Touch-screen use improves reliability in data collection, enables the quick dissemination of
educational material, engages people in their health management, and allows clinical staff to
address more complex concerns. One example in a class of possible applications is the use of
touch-screens in remote communities. Providing health education through touch-screens set-up
in remote health clinics is a step towards reducing health disparities between Indigenous and
non-Indigenous Australians. By supplying health information through a touch-screen kiosk
adapted with local Indigenous voices and easy-to-access buttons, Hunter and colleagues (2003)
were able to engage a remote Indigenous community in their health education. This technology
has great potential for mental health promotion and monitoring, and provides the opportunity to
address unmet need in currently disadvantaged groups – the most exciting application of health
technologies yet.
Thus the current thesis outlines the novel application of touch-screen technology to
progress monitoring, which together with advancements in assessment and intervention
highlights the ongoing integration of health and technology for the improvement of health
services.
General conclusions_____________________________________________________
There is a very clear need for the extended application of monitoring and feedback
methodologies in mental healthcare. The current thesis outlined the level of need for
interventions to complement evidence-based practice, and illustrated the implementation,
validation, and evaluation of a wellbeing feedback program that significantly improved
outcomes for poor responders in psychiatric care. Feedback was effective in improving patient
outcomes for depressive symptoms, but further work is required to determine the effects on
wellbeing, and how feedback is processed and experienced by patients and clinicians. The
current findings justify the application and rigorous evaluation of monitoring systems in other
clinical settings.
Feedback to patients and clinicians appears to improve the therapeutic process for those
unlikely to improve as expected. This process is not only useful at a treatment level, but reflects
121
an important course of action, influential at many levels of mental healthcare. Quality
improvement developments highlight a cycle of feedback for patients and clinicians, but also to
the organization, to policy advisors and administrators, the scientific community, and the public.
An ongoing communication between all interest groups should complement healthcare
improvement and inspire future research questions. In doing so, the patient may be empowered
in their experience of mental healthcare, and this should continue to be the goal for
psychotherapy research.
122
References
Atienza, A. A., Bradford, W. H., Baker, T. B., Abrams, D. B., Rimer, B. K., Croyle, R. T., et al.
(2007). Critical issues in eHealth research. American Journal of Preventive Medicine,
32, S71-S74.
Australian Bureau of Statistics. (2008). Household Use of Information Technology, Australia,
2007-08: Cat. No. 8146.0.
Australian Mental Health Outcomes and Classification Network. (2005). Reporting Framework
for the National Outcomes and Casemix Collection, Version 1.0. Brisbane, Queensland:
Australian Government Department of Health and Ageing.
Bech, P., Gudex, C., & Johansen, K. S. (1996). The WHO (Ten) Well-being Index: Validation
in Diabetes. Psychotherapy and Psychosomatics, 65, 183-190.
Bernard, H., Burlingame, G. M., Flores, P., Greene, L., Joyce, A., Kobos, J. C., et al. (2008).
Clinical practice guidelines for group psychotherapy. International Journal of Group
Psychotherapy, 58, 455-542.
Bhanoo, S. N. (2009). New tool in the MD's bag: A smartphone. The Washington Post, p. 6,
Boisvert, C. M., & Faust, D. (2003). Leading researchers' consensus on psychotherapy research
findings: Implications for the teaching and conduct of psychotherapy. Professional
Psychology: Research and Practice, 34, 508-513.
Claiborn, C. D., & Goodyear, R. K. (2005). Feedback in psychotherapy. Journal of Clinical
Psychology, 61, 209-217.
Daniels, B. A., Kirkby, K. C., Hay, D. A., Mowry, B. J., & Jones, I. H. (1998). Predictability of
rehospitalisation over 5 years for schizophrenia, bipolar disorder and depression.
Australian and New Zealand Journal of Psychiatry, 32, 281-286.
Davies, D. R., Burlingame, G. M., Johnson, J. E., Gleave, R. L., & Barlow, S. H. (2008). The
effects of a feedback intervention on group process and outcome. Group Dynamics:
Theory, Research, and Practice, 12, 141-154.
123
Eager, K., Trauer, T., & Mellsop, G. (2005). The performance of routine outcome measures in
adult mental health care. Australian and New Zealand Journal of Psychiatry, 39(728-
731).
Griffiths, K., Farrer, L., & Christensen, H. (2007). Clickety-click: E-mental health train on
track. Australasian Psychiatry, 15, 100-108.
Grizenko, N., Poinsier, B., Mercier, C., Perreault, M., & Latimer, E. (2000). Developing a
patient care monitoring system. [Letter to the Editor]. The Canadian Journal of
Psychiatry, 45, 88-89.
Grol, R., & Grimshaw, J. (2003). From best evidence to best practice: Effective implementation
of change in patients' care. Lancet, 362, 1225-1230.
Hageman, W. J., & Arrindell, W. A. (1999). Enhancing clinically significant change: Increment
of precision between individual and group level of analysis. Behaviour Research and
Therapy, 37, 1169-1193.
Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). The psychotherapy dose-response
effect and its implications for treatment delivery services. Clinical Psychology: Science
and Practice, 9, 329-343.
Harmon, C., Lambert, M. J., Smart, D. W., Hawkins, E. J., Nielsen, S. L., Slade, K., et al.
(2007). Enhancing outcome for potential treatment failures: Therapist/client feedback
and clinical support tools. Psychotherapy Research, 17(4), 379-392.
Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K., & Tuttle, K. (2004). The effects
of providing patient progress information to therapists and patients. Psychotherapy
Research, 14, 308-327.
Howard, K. I., Moras, K., Brill, P., Martinovich, Z., & Lutz, W. (1996). Evaluation of
psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist,
51, 1059-1064.
Hsu, L. (1989). Reliable changes in psychotherapy: Taking into account regression toward the
mean. Behavioral Assessment, 11, 459-467.
Hunter, E., Travers, H., & McCulloch, B. (2003). Bridging the information gap: IT and health in
Indigenous populations. Australasian Psychiatry, 11(Supplement), S51-S56.
124
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining
meaningful change in psychotherapy research. Journal of Consulting and Clinical
Psychology, 59, 12-19.
Kobak, K. A., Taylor, L. V. H., Dottle, S. L., Greist, J. H., Jefferson, J. W., Burroughs, D., et al.
(1997). A computer-administered telephone interview to identify mental disorders.
Journal of the American Medical Association, 278(905-910).
Koesters, M., Burlingame, G. M., Nachtigall, C., & Strauss, B. (2006). A meta-analytic review
of the effectiveness of inpatient group psychotherapy. Group Dynamics: Theory,
Research, and Practice, 10, 146-163.
Lambert, M. J. (2007). Presidential address: What we have learned from a decade of research
aimed at improving psychotherapy outcome in routine care. Psychotherapy Research,
17(1), 1-14.
Lambert, M. J., Hansen, N. B., & Bauer, S. (2008). Assessing the clinical significance of
outcome results. In A. M. Nezu & C. M. Nezu (Eds.), Evidence-based Outcome
Research: A Practical Guide to Conducting Randomized Controlled Trials for
Psychosocial Interventions. New York: Oxford University Press.
Lambert, M. J., & Ogles, B. M. (2004). The efficacy and effectiveness of psychotherapy. In M.
J. Lambert (Ed.), Bergin and Garfield's Handbook of Psychotherapy and Behaviour
Change (5 ed., pp. 139-193). New York: Wiley.
Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S. L., & Hawkins, E.
J. (2001). The effects of providing therapists with feedback on patient progress during
psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49-68.
Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L.,
et al. (2002). Enhancing psychotherapy outcomes via providing feedback on patient
progress: A replication. Clinical Psychology and Psychotherapy, 9, 91-103.
Lutz, W., Lambert, M. J., Harmon, S. C., Stultz, N., Tschitsaz, A., & Schurch, E. (2006). The
probability of treatment success, failure and duration - What can be learned from
empirical data to support decision making in clinical practice? Clinical Psychology and
Psychotherapy, 13, 223-232.
125
Lutz, W., Leach, C., Barkham, M., Lucock, M., Stiles, W. B., Evans, C., et al. (2005).
Predicting change for individual psychotherapy clients on the basis of their nearest
neighbors. Journal of Consulting and Clinical Psychology, 73(5), 904-913.
MacFarland, W. L., & Morris, S. J. (1998). Are dysphoric individuals more suggestible or less
suggestible that nondysphoric individuals? Journal of Counseling Psychology, 45, 225-
229.
Marks, I. (2004). Psychiatry in the future: Information technology can pull mental health care
into the 21st century. Psychiatric Bulletin, 28, 319-320.
Mathers, C., Vos, T., & Stevenson, C. (1999). The burden of disease and injury in Australia.
Canberra: Australian Institute of Health and Welfare.
Mathews, A., Ridgeway, V., & Williamson, D. A. (1996). Evidence for attention to threatening
stimuli in depression. Behavior Research and Therapy, 34, 695-705.
McLaren, P. (2005). Information and communications technology in mental health care. Journal
of the Royal Society of Medicine, 98, 232-234.
Morris-Yates, A., & Page, A. C. (in press). Routine measurement of outcomes by Australian
private hospital–based psychiatric services. In T. Trauer (Ed.), Outcome measurement
in mental health. Cambridge: Cambridge University Press.
Newnham, E. A., Harwood, K. E., & Page, A. C. (2007). Evaluating the clinical significance of
responses by psychiatric inpatients to the mental health subscales of the SF-36. Journal
of Affective Disorders, 98, 91-97.
Newnham, E. A., Harwood, K. E., & Page, A. C. (2009). The subscale structure and clinical
utility of the Health of the Nation Outcome Scale. Journal of Mental Health, 18, 326-
334.
Newnham, E. A., Hooke, G. R., & Page, A. C. (2009). Monitoring treatment response and
outcomes using the World Health Organization's Wellbeing Index in psychiatric care.
Journal of Affective Disorders, doi:10.1016/j.jad.2009.1006.1005.
Newnham, E. A., Hooke, G. R., & Page, A. C. (submitted). Patient monitoring and feedback in
psychiatric care reduces depressive symptoms.
126
Newnham, E. A., & Page, A. C. (2007). Client-focused research: New directions in outcome
assessment. Behaviour Change, 24(1), 1-6.
O'Kearney, R., Kang, K., Christensen, H., & Griffiths, K. (2009). A controlled trial of a school-
based internet program for reducing depressive symptoms in adolescent girls.
Depression and Anxiety, 26, 65-72.
Page, A. C., & Hooke, G. R. (2009). Effects of increased inpatient group psychotherapy on
outcomes, readmission rates, and satisfaction levels. Psychiatric Services, 60, 426-428.
Parabiaghi, A., Barbato, A., D'Avanzo, B., Erlicher, A., & Lora, A. (2005). Assessing reliable
and clinically significant change on HoNOS: A method for displaying longitudinal data.
Australian and New Zealand Journal of Psychiatry, 39(719-724).
Pirkis, J., Burgess, P., Kirk, P., Dodson, S., & Coombs, T. (2005). Review of standardised
measures used in the National Outcomes and Casemix Collection. Version 1.2.
Canberra: National Mental Health Strategy.
Sapyta, J., Reimer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research and
practice. Journal of Clinical Psychology, 61, 145-153.
Speer, D. C. (1992). Clinically significant change: Jacobson & Truax (1991) revisited. Journal
of Consulting and Clinical Psychology, 60, 402-408.
Stedman, T., Yellowlees, P., Mellsop, G., Clarke, R., & Drake, S. (1997). Measuring Consumer
Outcomes in Mental Health. Canberra, ACT: Department of Health and Family
Services.
Strauss, B., Burlingame, G. M., & Bormann, B. (2008). Using the CORE-R battery in group
psychotherapy. Journal of Clinical Psychology: In Session, 64, 1225-1237.
Stritzke, W. G. K., & Page, A. C. (2009). Electronic patient monitoring in mental health
services. In A. Dwivedi (Ed.), Handbook of Research on Information Technology
Management and Clinical Data Administration in Healthcare (pp. 87-103). Hershey,
New York: IGI Publishing.
Strong, S. R., & Matross, R. P. (1973). Change processes in counseling and psychotherapy.
Journal of Counseling Psychology, 20, 25-37.
127
Tasca, G. A., & Gallop, R. (2009). Multilevel modeling of longitudinal data for psychotherapy
researchers: I. The basics. Psychotherapy Research, online, i1-9.
Titov, N., Andrews, G., & Schwencke, G. (2008). Shyness 2: Treating social phobia online:
Replication and extension. Australian and New Zealand Journal of Psychiatry, 42, 595-
605.
Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E.
J. (2003). Improving the effects of psychotherapy: The use of early identification of
treatment failure and problem solving strategies in routine practice. Journal of
Counseling Psychology, 58, 59-68.
Wing, J. K., Lelliott, P., & Beevor, A. S. (2000). Progress on HoNOS. British Journal of
Psychiatry, 176, 392-395.