Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

E-Mailed Standardized Cognitive BehaviouralTreatment of Work-Related Stress: A Randomized

Controlled Trial

Jeroen Ruwaard2, Alfred Lange1, Manon Bouwman2, Janneke Broeksteeg2 andBart Schrieken2

1Department of Clinical Psychology, University of Amsterdam, Amsterdam, TheNetherlands, and 2Interapy BV, Amsterdam, The Netherlands

Abstract. The aim of this study was to assess the effects of a 7-week standardized cognitivebehavioural treatment of work-related stress conducted via e-mail. A total of 342 people applied fortreatment in reaction to a newspaper article. Initial screening reduced the sample to a heterogeneous(sub)clinical group of 239 participants. Participants were assigned randomly to a waiting listcondition (n562), or to immediate treatment (n5177). A follow-up was conducted 3 years afterinception of the treatment. The outcome measures used were the Depression Anxiety Stress Scales(DASS-42) and the Emotional Exhaustion scale of the Maslach Burnout Inventory – General Survey(MBI-GS). Fifty participants (21%) dropped out. Both groups showed statistically significantimprovements. Intention-to-treat analysis of covariance (ANCOVAs) revealed that participants inthe treatment condition improved significantly more than those in the waiting control condition(0.001vp(0.025). In the treatment group, the effects were large to moderate (0.9 (stress)>d>0.5(anxiety)). The between-group effects ranged from d50.6 (stress) to d50.1 (anxiety). At follow-up,the effects were more pronounced, but this result requires replication in view of high attrition atfollow-up. The results warrant further research on Internet-driven standardized cognitivebehavioural therapy for work-related stress. Such research should include the direct comparisonof this treatment with face-to-face treatment, and should address the optimal level of therapistcontact in Internet-driven treatment. Key words: treatment manual; Internet-driven therapy; chronicstress; burnout; stress management; cognitive behaviour therapy; Interapy; E-health

Received 10 February, 2005; Accepted 3 April, 2007

Correspondence address: Alfred Lange, Department of Clinical Psychology, University of Amsterdam,Roetersstraat 15, 1018 WB Amsterdam, The Netherlands. E-mail: a.lange@uva.nl

Globalization, organizational restructuring,and the universal adoption of informationtechnology have transformed the demands onemployees in many organizations (Sparks,Faragher, & Cooper, 2001). The pressure toadapt to rapid technological and organiza-tional demands has contributed to an increasein the prevalence of chronic stress amongemployees. Despite considerable efforts tocounter this trend, the rates of work incapa-city and sick leave due to work-related stressare high in the Netherlands compared withother European countries (Schaufeli &Kompier, 2001).

Stress is not necessarily harmful, i.e. adegree of stress is necessary to achieve desiredlevels of performance (Van Doornen, 2001).However, it may become a serious problem ifpeople are exposed to high levels of stress overextended periods of time. Work-related stressis defined as excessive chronic stress caused bya mismatch between work-related demandsand available coping skills (Health & SafetyExecutive, 2001). It is characterized by avariety of negative emotional and physiologi-cal reactions to aspects of the work itself, tothe environment, and to the organization(Levi & Levi, 1995). Common psychological

# 2007 Taylor & Francis ISSN 1650-6073DOI 10.1080/16506070701381863

Cognitive Behaviour Therapy Vol 36, No 3, pp. 179–192, 2007

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

reactions to work-related stress are depres-sion, anxiety, and emotional exhaustion(Cooper, Dewe, & O’Driscoll, 2001).

Cognitive behavioural therapy (CBT) andrelaxation therapy are moderately effective inreducing work-related stress (Van der Klink,Blonk, Schene, & Van Dijk, 2001). In CBT,clients are instructed how to address thecauses of their stress by changing their copingskills with regard to demands and pressures.In relaxation therapy, clients learn to lowertheir physiological arousal. In a meta-review,Van der Klink et al. (2001) estimated theeffects of CBT and relaxation therapy to bed50.7 and d50.4, respectively.

The implementation of these interventions,however, requires trained professionals, whoare relatively few in number (World HealthOrganization, 2001). In addition, clients maybe hesitant to seek help, due to limitedmobility, travelling distance, or a lack ofawareness concerning their condition, or thepossibility of treatment. Finally, fear ofstigmatization may discourage potential cli-ents from seeking help. Given the obstacles toseeking face-to-face therapy, there is anincreasing interest in methods to provideevidence-based treatment using the Internet(Barak, 1999; Emmelkamp, 2005; Newman,2004). Online therapy may reduce the presentburden on the available mental healthresources, and may provide a viable alter-native for people who face geographical,physical, psychological and/or financial bar-riers in seeking traditional, face-to-face care.

Critics of Internet-driven therapy havequestioned its feasibility and have expressedethical concerns (e.g. Bauer, 2000; Bonger,1988). However, at present, the main questionconcerning online therapy is not whether, butrather how exactly, it should be undertaken.Over the years, the accumulation of studiesevaluating online therapy has resulted in abody of results. Reviews to date indicate thatonline therapy provides an effective alterna-tive to existing methods for early intervention(e.g. Griffith & Christensen, 2006; Wantland,Portillo, Holzemer, Slaughter, & McGhee,2004). However, these reviews also showmany challenging, open-ended issues thatremain to be addressed. For example, theavailable evidence on the effectiveness ofonline therapy is limited to a relatively smallnumber of disorders, and the effectiveness is

found to vary considerably across studies (e.g.Spek et al., 2006). Furthermore, dropout-ratestend to be high. Wantland et al. (2004)reported an average dropout-rate of 21%.

Studies vary greatly in the degree oftherapist involvement. Some studies reliedentirely on self-help material (e.g. Klein &Richards, 2001). Others explicitly involved apersonal therapist, and included explicitstructuring of the remote client-therapistinteraction to facilitate the desired changes.The involvement of a therapist allows forpersonalization of the interventions andencourages the development of a therapeuticalliance (Knaevelsrud, Jager, & Maercker,2004). In a meta-analysis of Internet-basedinterventions for anxiety and depression, Speket al. (2006) found that the effects of inter-ventions with therapist support were consid-erably larger than the effects of interventionswithout therapist support.

Zetterqvist, Maanmies, Strom, andAndersson (2003) tested the efficacy of e-mailed self-help for stress management withminimal therapist contact. The treatmentinvolved relaxation training, development ofproblem-solving and time-management skills,cognitive restructuring, and behavioural exer-cise. Compared with a waiting list controlgroup, participants in the programme dis-played significantly greater improvement onthe Perceived Stress Scale (PSS: S. Cohen,Kamarck, & Mermelstein, 1983) and theHospital Anxiety and Depression Scale(HADS: Zigmond & Snaith, 1983). Thereported summary statistics suggest large tomoderate (controlled) effect sizes of d50.6(PSS), d50.5 (HADS Anxiety) and d50.8(HADS Depression). Unfortunately, 26% ofthe participants dropped out, and intention-to-treat analyses failed to reveal significantdifferences between the 2 groups.

Based on principles of online therapydeveloped earlier for the treatment of post-traumatic stress (Lange, Rietdijk, et al., 2003),we designed a standardized treatment ofwork-related stress, which can be conductedvia e-mail. The treatment is based uponinterventions that have proved their worth inclinical trials and clinical practice (De Jong &Emmelkamp, 2000; Lange, Richard, Gest, DeVries, & Lodder, 1998). The treatment in-volves psycho-education, awareness training,applied relaxation, cognitive restructuring,

180 Ruwaard, Lange, Bouwman, Broeksteeg and Schrieken COGNITIVE BEHAVIOUR THERAPY

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

positive self-verbalization, social skills train-ing and time-management. The content ofthe treatment is similar to that of Zetterqvistet al. (2003). In contrast to the Zetterqvistet al. study, where client-therapist interac-tion was incidental and not standardized,the treatment manual includes explicit struc-turing of the client-therapist interaction.Specifically, it specifies the timing, nature,and frequency of the therapist instructionsand feedback.

Below, we present the results of a rando-mized controlled trial that we conducted toassess the effects of our Internet-based stan-dardized CBT on work-related stress. Wecompared the effects of treatment with thoseobserved in a waiting list control group. Wehypothesized that, in comparison to the wait-ing-list, the treatment would substantiallyreduce levels of stress. We also expected thetreatment to reduce stress-related complaints,in particular depression, anxiety, and emo-tional exhaustion. In addition, we report theresults of a 3-year follow-up study in which weassessed the long-term effects among thosewho had completed treatment.

MethodDesignThe study comprised a waiting-list controlledpre-post trial. Participants were randomlyassigned to 2 groups. One group started the7-week treatment immediately (experimentalgroup), while the other started it after 7 weeks(waiting list control group). Three years afterthe completion of the pre-test, a follow-upstudy was conducted among the clients of theexperimental group who had completed thetherapy, in order to determine the long-termeffects of the intervention.

ParticipantsRecruitment. A national Dutch quality news-paper published an interview with 1 of theauthors about Internet-driven therapy. Thearticle announced the current study, andreferred interested readers to a website. Thissite provided psycho-education on work-related stress, explained the purpose anddesign of the study, and contained anapplication form.

Screening. Respondents were screened bymeans of self-report questionnaires withrespect to the following exclusion criteria:age v18 years, heightened risk of dissociationor psychosis, suicidal ideation, drug abuse,use of neuroleptic medication, and concurrentother treatment. In addition, respondentswere required to download, print, and returna signed informed consent form. Excludedrespondents were referred to other mentalhealth institutions.Sample. Of the 342 respondents, who appliedfor the treatment, 65 did not complete thescreening and 38 met the exclusion criteria(see Figure 1 for details). The final sample(n5239) comprised 143 females (60%) and 96males (40%), aged between 22 and 60 years(M544, SD58). Most of the participants werehighly educated (84% completed tertiaryeducation), and reported a wide variety ofprofessional occupations. The majority (80%)were in full-time paid employment. Sevenparticipants were unemployed, because ofwork-related stress. Ten participants sufferedfrom stress in unpaid jobs (housewives,volunteers, and students). In 167 (70%) ofthe participants, the level of stress was abovethe clinical cut-off on the stress outcomemeasure (described in the measures section).On average, the duration of the reportedsymptoms was 30 months (SD534, range51–180 months). A number of participants (38%)was on partial or full sick leave. All partici-pants attributed their complaints to their job,although many (65%) reported additional(unspecified) personal issues.Randomization. One month after the publica-tion of the newspaper article, participantswere randomly assigned to the 2 experimentalgroups by means of the random numbergenerator procedure of SPSS 10. Assuming acorrelation of r50.5 between a pre-test andpost-test measurement, a significance level of0.05 would require 49 participants per groupto provide an 80% probability to detect amedium (d50.5) effect size (J. Cohen, 1988).Because we had a fairly large sample ofparticipants, we assigned 3 times as manyparticipants to the immediate treatment con-dition, while retaining a large enough sample-size in the control condition to ensuresufficient statistical power to detect treatmenteffects. Thus, 177 participants (74%) wererandomly assigned to immediate treatment,

VOL 36, NO 3, 2007 E-mailed CBT of work-related stress 181

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

and 62 (26%) to the waiting list controlcondition. Table 1 provides an overview ofthe characteristics of the 2 groups. To checkthe randomization, x2 tests or t-tests (whereappropriate) were conducted with respect tothe outcome measures, gender, age, maritalstatus, education, work contract, work years,working hours, duration of symptoms, andsick leave status. As expected, no significantdifference was found (t(238)50.26–1.75,p50.08–0.80; x250.002–4.1, p50.13–0.96).

ProcedureSetting. Communications took place entirelythrough e-mail, without any face-to-facesession. Participants underwent the treatment

at home. Incidentally, telephone contact wasnecessary (e.g. to inquire about the reasons fordrop-out when e-mails remained unanswered).Privacy. Several procedures secured the priv-acy of the participants. An e-mail server wasset up exclusively for the study and located ata professional Internet host. The server,running Linux OS, was protected by a firewalland remotely administered through anencrypted communication channel. The thera-pists worked from a central computer-roomthat was accessible only with proper author-ization. Each therapist received a privatepassword-protected account to login to thecomputers and to the e-mail server.Participants were informed of mail-clientextensions to secure their e-mail, and received

Figure. 1. Participant flow in the randomized controlled trial (RCT).

182 Ruwaard, Lange, Bouwman, Broeksteeg and Schrieken COGNITIVE BEHAVIOUR THERAPY

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

help in configuring these extensions onrequest.Materials. Participants needed a personalcomputer connected to the Internet and ane-mail account. For screening and outcomemeasurement, validated questionnaires weredelivered in the body text of e-mails and indocuments that were attached to these e-mails. The attachments were formatted in acommon cross-platform computer format(Rich Text Format).Treatment. The treatment comprised 7 phases,as described in detail below. Each phaserequired approximately 1 week to complete.The schedule allowed the participants to adjustthe pace of completion within limits to theirown situations. Treatment integrity was guar-anteed by a manual that specified each step ofthe treatment in detail (e.g. the order and thenature of the assignments, the contents of thepsycho-education, and the timing of thetherapist feedback). The manual also provideddefault text templates for feedback andinstructions. The therapists tailored these textsto the specific needs of their clients. They wererequired to email their feedback after 1 work-ing day. This provided the therapists withample time to think through their feedback, orto discuss difficult cases with colleagues or asupervisor. The treatment manual specified 10feedback-moments. Composing the feedback

took approximately 30 minutes. Thus, a fulltreatment took approximately 5 hours oftherapist time.Therapists. The therapists were 25 doctoralstudents in clinical psychology and 1 post-graduate student, aged between 22 and 42years (M526, SD54.7). They had followedadvanced courses in CBT. Additional trainingtaught the therapists how to tailor the feed-back and instruction templates of the manualto the needs of their clients, how to increasemotivation by adopting a stimulating empa-thetic attitude, how to avoid the pitfalls ofelectronic, text-based communication (e.g.Brennan & Ohaeri, 1999), and how to makeuse of the asynchronous nature of the com-munication to enhance the quality of thefeedback (e.g. by discussing cases with oneanother or the supervisor). Once a week, asenior specialist in Internet-driven CBT super-vised the therapists. The supervisor assignedparticipants to therapists on the basis of theavailability of the therapists.Post-test and follow-up. Immediately aftertreatment, participants received the post-testmeasurement (the outcome questionnairesand an evaluation questionnaire). Those whodid not complete treatment received a tele-phone call to inquire about the causes ofdropout. Dropouts were also asked to com-plete the post-test measurements. Three years

Table 1. Characteristics of participants.

Characteristics

Control Treatment Total

n562 n5177 n5239

DemographicGender: female 68% 42 57% 101 60% 143Age (M, SD) 42 9 44 8 44 8Education: tertiary 83% 50 85% 144 84% 194Marital status: married 27% 17 18% 32 21% 49

living together 42% 26 54% 96 51% 122Unmarried 21% 13 21% 38 21% 51Divorced/widow(er) 10% 6 6% 11 7% 17

WorkEmployment: full-time 77% 48 81% 140 80% 188Years in current profession (M, SD) 9 9 11 9 10 9Working hours per week (M, SD) 35 13 36 13 36 13

ComplaintsMonths with complaints (M, SD) 26 29 32 36 30 34Sick leave: no 56% 35 65% 114 63% 150Partial 23% 14 12% 21 15% 35Full 21% 13 23% 41 23% 54

VOL 36, NO 3, 2007 E-mailed CBT of work-related stress 183

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

after the pre-test measurements, participants,who had completed treatment, were invited bye-mail to complete the outcome questionnaires.

Cognitive behavioural treatmentPhase 1. Inducing awareness: monitoring andwriting. This phase comprised 1 week ofmonitoring stress and 2 writing assignmentsto increase participants’ awareness of somatic,cognitive, emotional and behavioural signs ofstress. Participants kept records of stressfulevents: they described the situation, associatedfeelings and thoughts, and rated the degree ofstress on a 10-point scale. Subsequently, theyconveyed their ratings graphically andreflected on the results. In addition, theparticipants spend 2 periods of 45 minuteswriting about stressful events, in the presenttense and in as great a detail as possible(Lepore & Smyth, 2002; Alford, Malouff, &Osland, 2005).Phase 2. Relaxation. First, participants weretaught muscle relaxation or mental relaxation,depending on their own preference.Subsequently, they practised 6 times a day.In addition, they were encouraged to takeshort breaks during their work to reduce theiroverall pace, and to engage in physicalexercise. Participants who reported sleepingcomplaints received instructions how toregain healthy sleeping habits: i.e. not sleepingduring the day, not sleeping too long duringthe night, avoiding stressful activities (e.g.stressful conversations) before going to bed,and not trying to force themselves to sleep(Morin, 1993).Phase 3. Worrying, rumination, and challen-ging dysfunctional thoughts. Participants weretaught to recognize events that trigger worry-ing and ruminating. They were instructed tolimit ruminating to predetermined, fixedmoments, rather than indulging in it at allhours of the day (McKay, Davis, & Fanning,1997). When they started worrying andruminating spontaneously, they wereinstructed to make a brief note, then readtheir previously generated self-instructioncard to determine the next fixed moment forruminating. Furthermore they were requiredto focus on their notes and to challengedysfunctional (black and white) thinking,overgeneralization, catastrophic thinking,self-blaming, and neglect of positive aspects

(Beck, 2005). They did this 3 times a week for20 minutes.Phase 4. Positive self-verbalization. Partici-pants were taught to focus on their positivequalities by using the technique of positiveself-verbalization (Lange, Richard, Gest, DeVries, & Lodder, 1998; Lange, Richard,Kiestra, & Van Oostendorp, 1997). Theywrote an essay on their positive traits andsummarized this on a small card. Theparticipants were encouraged to read thesummary on the card frequently, and read italoud.Phase 5. Positive assertiveness/social skillstraining and behavioural experiments. Partici-pants received a short social skills training.They were encouraged to reflect on thereciprocity of the interactions with theirfellow-workers. Subsequently, they receivedinstructions on how to experiment with newstrategies to improve negative reciprocalbehaviours. They were, for instance, taughtto communicate wishes in a positive andconstructive manner, to give compliments,and to express their interest in the well-beingof others. If they questioned the validity ofthese strategies, the therapist helped to createbehavioural experiments, in which the newapproaches could be put to test in a simpleand realistic manner.Phase 6. Time management. Participants weremade aware how they managed the demandsof both their work and their private life. In theevening, participants reflected on the activitiesof the day, after which they planned theactivities for the next day. The participantslearned how to set priorities, to reject requeststhat they found too demanding, to plan timefor relaxation and self-reward, and to avoidunrealistic scheduling. They were encouragedto take their time to decide consciouslywhether to start an activity immediately,postpone it, or to pass it on to someone else.Phase 7. Future, reintegration, and relapseprevention: the toolkit. Participants reflectedon symptoms that might signal relapse. Theywere encouraged to formulate their personal‘‘relapse prevention toolkit’’, using the tech-niques they had mastered in therapy. Theywere encouraged to print the toolkit, andplace it in a visible place at home as a symbolof what they had mastered. Participants whowere partially or fully on sick leave receivedsuggestions for gradual reintegration.

184 Ruwaard, Lange, Bouwman, Broeksteeg and Schrieken COGNITIVE BEHAVIOUR THERAPY

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

Screening measuresDemographic characteristics, substance abuseand medication. The participants completed amultiple-choice questionnaire to record age,gender, marital status, education, work con-tract, work years, working hours, duration ofsymptoms, sick leave status and perceivedcauses of stress. In addition, alcohol depen-dency and drug abuse was registered. Toassess the use of neuroleptics, participantsanswered open-ended questions concerningmedicine brands and prescribed doses.Dissociation. Risk of dissociation wasassessed using the 5-item SomatoformDissociation Questionnaire (SDQ-5)(Nijenhuis, Spinhoven, Van Dyck, Van derHart, & Vanderlinden, 1997). The internalconsistency of this self-report instrument isgood (Cronbach’s a50.80). A cut-off of 8(used in this study) results in good discrimina-tion between groups of patients and non-patients (sensitivity50.94; specificity50.96).Psychosis. Participants, who scored above thecut-off value of 13 on the Screening Devicefor Psychotic Disorder (SDPD) (Lange,Schrieken, Blankers, Van de Ven, & Slot,2000) were excluded. The cut-off value of 13was established in the Dutch norm group. The7-item self-rate inventory (Cronbach’sa50.82) is a good predictor of psychoticepisodes. Agreement between self-report in agroup of 33 schizophrenic patients and theratings by their clinicians is high (r50.85)(Lange et al., 2000).Suicidal ideation. This was measured by aninventory similar to that of Joiner et al.(2003). It comprises 6 multiple-choice ques-tions, including ‘‘Do you currently have plansto end your life?’’ and ‘‘Are you currentlyfeeling desperate?’’ Participants were excludedif they had a history of suicide attempt(s)within the past 2 years ago, or if there wasimmediate risk of suicide.

Outcome measuresThe primary outcome measure was the Stresssubscale from the Depression Anxiety StressScales (DASS-42: Lovibond & Lovibond,1995; Dutch version: De Beurs, Van Dyck,Marquenie, Lange, & Blonk, 2001).Secondary measures were the Depressionand Anxiety subscales from the DASS andthe Emotional Exhaustion subscale of the

Dutch version of the Maslach BurnoutInventory – General Survey (MBI-GS:Maslach, Jackson, & Leiter, 1996; Dutchversion by Schaufeli & Van Dierendonck,2000).DASS. The DASS is a self-report instrumentthat assesses depressive symptoms, physicalanxiety (fear), and mental stress (nervoustension). It contains 42 items, 14 per subscale,that relate to the experience of symptoms inthe past week. The items are measured on a 4-point scale ranging between 0 (‘‘did not applyto me’’) to 3 (‘‘applied to me very much, ormost of the time’’). Higher scores denote lessfavourable conditions.

All subscales of the Dutch adaptation arecharacterized by good internal consistencies(Cronbach’s a between 0.94 and 0.97, presentsample: 0.86–0.92). Test-retest reliabilities forthe Depression, Anxiety and Stress scale arer50.75, r50.89, and r50.79, respectively (DeBeurs et al., 2001). Clinical cut-off scores ofc512 and c55 for depression and anxiety arerecommended by Nieuwenhuijsen, De Boer,Verbeek, Blonk, & Van Dijk (2003). Thecorresponding percentile score of stress (14)observed in a large (n51771) non-clinicalsample in the UK (Crawford & Henry, 2003)was used as the clinical cut-off in this study.MBI-GS Emotional Exhaustion. The MBI-GSis a self-report questionnaire to assess burnoutacross professional occupations. TheEmotional Exhaustion subscale assesses emo-tional fatigue, i.e. the feeling of being ‘‘worn-out’’. The scale contains 5 items that aremeasured on a 7-point scale scored from 0 to6, where higher scores indicate higher levels ofemotional exhaustion. The internal consis-tency of the subscale ranges between0.84(a(0.90 (present sample: a50.84). The8-month test-retest reliability is satisfactory(0.58(r(0.85). Well-established clinical cut-off scores are unavailable for the Exhaustionscale. To indicate burnout, Brenninkmeijerand Van Yperen (2003) recommend a cut-offscore of 2.67, which was used in the presentstudy. With this cut-off score, they found afalse positive rate of 9.1% and a false negativerate of 13.8%.

Methods of analysisIntention-to-treat. The analyses were on anintention-to-treat basis and included all

VOL 36, NO 3, 2007 E-mailed CBT of work-related stress 185

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

participants. Dropouts, who did not completethe post-test measurements, were assumed tohave gained nothing. Their pre-test scores alsoserved as their post-test scores.Statistical significance. One-way analyses ofcovariance (ANCOVAs) (using the pre-testscores as a covariate) were conducted to testthe difference in means of the 2 groups atpost-test (a50.05). The statistical significanceof the within-group pre-post gain scores wereassessed using paired t-tests.

The assumptions of ANCOVA were met.The homogeneity of the regression coefficientsin the 2 groups was confirmed by non-significant interactions between the covariates(pre-test scores) and experimental condition.The distributions of the outcome variableswere approximately normal, and the varianceacross the experimental groups was found tobe homogeneous.Effect size. To express the magnitude of theeffects, gain scores on the outcome measureswere standardized to Cohen’s d’s (J. Cohen,1988), representing the number of standarddeviations separating the 2 means. Pointestimates and 95% confidence intervals of dwere determined both for the within- and thebetween-group effects following a proceduredescribed in detail by Robey (2004). Inthis procedure, between-group effect sizesare calculated using the pooled standarddeviation, and confidence intervals areapproximated on the basis of the centralt-distribution.Clinical relevance. We tested the higher prob-ability of statistically reliable improvementand recovery after treatment compared withthe control group with Fisher’s exact tests,and expressed the difference in this probabil-ity between the treatment and the controlgroup in terms of odds ratios (Hillis &Woolson, 2002). We used the ReliableChange Index (RCI) to test the significanceof individual improvement (Jacobson &Truax, 1991; a50.05; critical RCI51.96; ‘‘nochange’’ and ‘‘deterioration’’ were pooled intoa single ‘‘unimproved’’ category). Participantswere considered recovered if they reliablyimproved from a pre-test score above thecut-off to a post-test score below cut-off.Therefore, participants scoring below cut-offat pre-test were excluded from the recoveryanalysis.

Follow-up. Participants who had completedthe treatment were invited by e-mail tocomplete the follow-up measures on a spe-cially constructed website. These participantsinclude members of the treatment group andmembers of the control group, who followedthe treatment at a later date. Pre-test tofollow-up data of those who participated inthe follow-up were analysed using repeatedmeasures ANOVAs with time of measurementas the within factor. Differences between themeans at the times of measurement weretested for significance using Bonferroniadjustments to keep the family-wise Type Ierror at a50.05.

ResultsMain outcomeDropout. Fifty participants (21%) droppedout from the study, 49 from the treatmentgroup and 1 from the control group. Of these50, 25 (50%) dropped out without providing areason (see Figure 1). Twenty-two dropouts(44%) completed the post-test. None of thevariables used to check the randomizationwere predictive of dropout.Statistical significance. Table 2 shows theresults of the pre-test and post-test measure-ments on intention-to-treat basis. As hypothe-sized, the treatment group improvedstatistically significant with respect to stress(t(176)511.76, pv0.001), and the secondaryoutcome measures (all paired t testspv0.0001). However, the improvement ofthe waiting list control group also reachedstatistical significance with respect to stress((t(61)52.14, pv0.036), and all secondaryoutcome measures, except DASS depression(t(61)51.46, p50.15). Nonetheless, theANCOVAs showed that the treatment groupimproved significantly more than the controlgroup, on both stress and the secondarymeasures (see Table 2).Effect size. The within-group effects in thetreated group were large to moderate, rangingbetween d50.9 (stress) and d50.5 (anxiety,exhaustion). Because the control group alsoimproved, the effect sizes of treatment ascompared with no treatment were smaller: thebetween effects ranged between d50.6 (stress)and d50.1 (anxiety) (see Table 2).Clinical relevance. Table 3 shows theimprovement and recovery rates in the 2

186 Ruwaard, Lange, Bouwman, Broeksteeg and Schrieken COGNITIVE BEHAVIOUR THERAPY

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

experimental groups. In the treatment group,with respect to stress, 53% of the participantsreliably improved, and half the clinical sub-group recovered from clinical stress. The odds

of improvement and recovery were signifi-cantly higher in the treatment condition thanin the waiting list. With respect to thesecondary measures, the effects varied.

Table 2. Randomized controlled trial results (intention-to-treat): treatment (n5177) vs waiting list control(n562).

Measurea

Pre Postb,c Within ESd Between ES ANCOVA

M SD M SD d 95%CI d 95%CI F1,236 p

DASS StressTreatment 19.4 8.0 12.0 8.5 0.9 ¡0.2 0.6 ¡0.3 23.9 v0.001Control 19.7 8.0 17.6 9.3 0.2 ¡0.2

DASSDepressionTreatment 11.8 6.7 7.8 6.6 0.6 ¡0.2 0.4 ¡0.3 10.0 0.002Control 12.1 7.3 10.7 7.3 0.2 ¡0.3

DASS AnxietyTreatment 8.2 6.2 5.2 5.2 0.5 ¡0.1 0.1 ¡0.3 5.1 0.025Control 9.6 6.7 7.3 5.5 0.4 ¡0.2

MBI Em.ExhaustionTreatment 3.1 1.2 2.5 1.3 0.5 ¡0.1 0.3 ¡0.2 8.8 0.003Control 3.4 1.2 3.2 1.2 0.2 ¡0.2aDASS5Depression Anxiety Stress Scales; MBI5Maslach Burnout Inventory, General Survey. DASS scoresare mean sum scores, MBI scores are mean average scores.bThe means and standard deviations reflect the raw data, whereas the F-statistics reflect the results of the test ofthe difference in post-test means after pre-test adjustments (ANCOVA).cPre-test scores were carried forward for participants who did not complete the post-test measurement (n528).dES5effect size: Cohen’s d point estimate and 95% confidence interval (d295%CI to d+95%CI).ANCOVA5analysis of covariance; 95%CI595% confidence interval.

Table 3. Clinical relevance: reliable improvement and recovery.

Measure

Improved Recovered

n % ORb pc cnd c%

d n % ORb pc

DASS StressTreatment 93 53 2.9 v0.001 124 70% 62 50 5.1 v0.001Control 17 27 43 69% 7 16

DASS DepressionTreatment 64 36 3.3 0.001 78 44% 41 53 3.5 0.007Control 9 15 29 47% 7 24

DASS AnxietyTreatment 55 31 1.1 0.448 110 62% 38 35 1.9 0.097Control 18 29 41 66% 9 22

MBI Em. ExhaustionTreatment 62 35 1.8 0.047 106 60% 36 34 2.6 0.024Control 14 23 43 69% 7 16aDASS5Depression Anxiety Stress Scales; MBI5Maslach Burnout Inventory, General Survey.bOdds ratio (OR): the ratio of the odds of improvement/recovery in the treatment group compared with the oddsin the control group.cThe p-values represent results of Fisher’s exact tests of the observed 262 tables.dColumn cn and c% represent the number and percentage of participants that scored in the clinical range at pre-test.

VOL 36, NO 3, 2007 E-mailed CBT of work-related stress 187

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

Concerning depression, the effects were simi-lar to the effects on stress. The effects weresmaller, but statistically significant, withregard to emotional exhaustion. With respectto anxiety however, the differences were notsignificant. Concerning anxiety, there wassubstantial improvement and recovery in thecontrol group that was not compensated bymuch additional improvement and recovery inthe treatment group.

Long-term follow-upParticipants. After approximately 3 years (themean number of months to follow-up was 34,SD52.1), participants who had completedtreatment (n5167) were invited to completethe follow-up questionnaires. Of these 167, 73could not be traced. Of the remaining 94participants, 63 (67%) completed the follow-up questionnaires.

A few differences were found betweenparticipants who completed the follow-upand those who did not. The former were morehighly educated (x254.9; p50.034). In addi-tion, their treatment length was shortercompared with those who did not participatein the follow-up (t(117)53.3, p50.001). Atpre-test and post-test, the 2 groups did notdiffer on stress, anxiety or depression(t(164)50.27–1.0, p50.30–0.78). However,the mean pre-test score on emotional exhaus-tion of participants completing the follow-up(M53.4) was higher than that of participantswho did not (M53.0; t(164)52.47, p50.014).At post-test this difference was not statisti-cally significant (t(157)51.19, p50.23).Long-term effects. Table 4 shows the meansand variances of the mean scores of the

participants at pre-test, post-test, and the 3-year follow-up. There was no relapse: theeffects became more pronounced. The 3-yearfollow-up means show significantly lesspathology than the post-test means. In termsof effect size, the gains from pre-test to follow-up were large (1.3 (anxiety)(d(1.8 (stress)).

With respect to stress, the main effect oftime was significant, F(2, 126)579.90,pv0.0001. The pre-post improvement(pv0.001) was maintained, and was greaterat the 3-year follow-up (pre-test to follow-up:pv0.001; post-test to follow-up: p50.020).The same pattern emerged with respect todepression (F(2, 126)560.73, pv0.0001; post-test to follow-up: p50.002) and emotionalexhaustion (F(2, 120)551.68, pv0.0001; post-test to follow-up: pv0.001). However, withrespect to anxiety, the effects remained con-stant (F(2, 126)549.52, pv0.0001; post-test tofollow-up: p50.34).Effects of other treatments. Between the post-test and the 3-year follow-up, 41% (n526) ofthe participants received additional treatment(psychotherapy, other forms of counselling, ormedication). Those who followed additionaltherapy tended to display more complaints atfollow-up compared with those who did notundergo additional treatment, although thedifference was statistically significant only withrespect to emotional exhaustion (t(60)53.1,pv0.001)). At pre-test and post-test, thegroups did not differ on the outcome measures.

Treatment satisfactionAfter completing treatment, participants ratedthe overall value of the treatment on a scale of1 to 10 with an average of 7.6 (SD51.0;Range: 5–10). The different treatment phases

Table 4. Results of the 3-year follow-up (n563).

Measurea

Pre Post FU Pre-FU ES

M SD M SD M SD d CI.95

DASS Stress 19.6 7.6 10.1 7.9 7.3 6.6 1.8 ¡0.3DASS Depression 11.6 6.5 5.6 5.8 3.3 4.1 1.5 ¡0.3DASS Anxiety 8.6 5.2 3.8 3.6 3.0 3.7 1.3 ¡0.3MBI Emotional

Exhaustion3.4 1.4 2.5 1.3 1.7 1.0 1.6 ¡0.3

aDASS5Depression Anxiety Stress Scales; MBI5Maslach Burnout Inventory, General Survey. DASS scoresare mean sum scores, MBI scores are mean average scores.FU5follow-up.

188 Ruwaard, Lange, Bouwman, Broeksteeg and Schrieken COGNITIVE BEHAVIOUR THERAPY

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

were consistently rated favourably. Increasingawareness through writing received the high-est mean rating (M58.1; SD51.4), relaxationwas rated lowest (M57.1; SD51.8).

The participants rated various aspects ofthe relationship with their therapists on 3-point multiple-choice response scales. Overall,the relationship was rated as pleasant (88%)and personal (75%), and was perceived tohave grown during treatment (57%). Sixty-eight percent indicated that they had notmissed face-to-face contact.

Mediating variablesComparison of the outcome in completers ofthe treatment group and participants of thecontrol group who completed the treatment ata later date revealed no significant differences.Therefore, the treatment completers in bothgroups were pooled (n5167) to obtain morepower for the detection of significant pre-dictors of outcome in multiple regression.However, no significant mediating variableswere found. After controlling for pre-treat-ment stress-levels, post-treatment stress levelswere not predicted by gender (F(1,163)50.52,p50.47), age (F(1,153)50.45, p50.45), educa-tion (F(1,156)50.72, p50.40), working years(F(1,157)50.004, p50.95), working hours(F(1,157)50.12, p50.73), duration of symp-toms (F(1,110)50.10, p50.75) or length oftreatment (F(1,115)50.001, p50.97). Similarresults were obtained with the secondarymeasures.

DiscussionCompared with no treatment, e-mailed stan-dardized CBT moderately reduced stress andinduced small to moderate improvements ondepression and emotional exhaustion. Fiftypercent of the participants had recovered fromclinical stress following treatment. The effectson anxiety appeared to be small. However,after 3 years the (uncontrolled) effects werelarge on all outcome measures. Participantswere highly satisfied with their treatment andtheir therapists.

Evaluation of effect sizesThe substantial improvement of the partici-pants in the waiting list control groupconsiderably deflated the effects betweentreatment and control group. The improve-ment observed in the waiting-list control

group is comparable to that observed in otherstudies (Lange, Van de Ven, & Schrieken,2003; Zetterqvist et al., 2003). Zetterqvist et al.(2003) provided several explanations for theseimprovements, including the beneficial effectsof pre-testing and the prospect of treatment.In the current study, the recruiting article andthe website provided psycho-education thatmay have had beneficial effects. Possibly, theparticipants in the control condition benefitedfrom the information they received, combinedwith the knowledge that they would start theactive treatment in the near future. However,these explanations remain speculative. Theuse of different types of control groups mayshed light on the source of the effects in thecontrol group.

Notwithstanding the improvement in thecontrol group, the observed between groupeffects are similar to those reported in theliterature. The (controlled) effects on stress(d50.6) and depression (d50.4) compare wellwith the effects of face-to-face CBT (d50.7and d50.3 respectively; Van der Klink et al.,2001). In addition, the results corroborateresults obtained by Zetterqvist et al. (2003) ina RCT of an Internet-driven stress-reductionprogram. In that study, the effect sizes werehigh, but intention-to-treat analyses failed toreveal statistically significant between effects.In contrast, all between effects in the presentstudy were significant on an intention-to-treatbasis. This was perhaps due to the largersample size and the use of a more powerfulstatistical technique.

MaintenanceThe uncontrolled 3-year follow-up showed ahigh impact of the treatment with large effectsizes after 3 years, on stress and depression,and on emotional exhaustion and anxiety.This is consistent with our findings in thestudy on post-traumatic stress (Lange, Van deVen, & Schrieken, 2003), in which effects alsoproved to be stable. We may conclude thatparticipants, who complete this type ofInternet-driven CBT, have a high chance tonot relapse, and may even enjoy furtherimprovements. These results are compelling,as the further improvements were notexplained by any additional treatment, whichsome participants had sought in this 3-yearperiod.

VOL 36, NO 3, 2007 E-mailed CBT of work-related stress 189

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

LimitationsSeveral aspects of this study limit the general-izability of the results. We will discuss the self-selection of the participants, the differenttime-frame between measurements in theexperimental groups, and the dropout andattrition at follow-up.

Self-selection most clearly manifested itselfin the high educational level of the partici-pants. The high educational level is probablydue to the fact that most participants wererecruited through an article in a qualitynewspaper that is favoured by the welleducated. Yet, high education seems to berather typical of people who participate inInternet-based therapy (Andersson et al.,2006; Carlbring, Westling, Ljungstrand,Ekselius, & Andersson, 2001). Until therelation between education and Internet ther-apy is better understood, caution should beexercised in generalizing the present results toless well-educated populations.

Participants in the treatment condition werepost-tested when they completed treatment.We expected them to complete treatmentwithin the planned 7 weeks, but the averagetreatment took substantially longer (16weeks). Consequently, the experimentalgroup, on average, was post-tested severalweeks after the control condition. Therefore,the between-effects may have been con-founded by (uncontrolled) effects of the merepassage of time. However, if present, thiseffect was probably small. Exploratory ana-lyses did not reveal a significant effect oftreatment duration on the outcome variables.Nevertheless, to avoid such interpretativeissues in our current studies, we now multiplythe protocol time by 1.5 to estimate the actuallength of treatment.

During the study, a considerable number ofparticipants dropped out. Although theobserved dropout rate of 21% is common inonline therapy (Wantland et al., 2004), it isdisconcerting that most dropout occurred inthe treatment group. No significant predictorsof dropout were found, and many dropoutsdid not provide a reason for terminating theirparticipation. However, therapists noticedthat participants often experienced problemsin combining work and treatment demands.Perhaps some of the exercises required toomuch time for some participants or wereotherwise too demanding. Thus, the treatment

may have presented a practical burden interms of time and effort that was notexperienced by control group participants.

Finally, the observed long-term reductionof complaints is promising, but was uncon-trolled, and suffered considerable attrition.The long-term follow-up needs replication.

E-mail vs websiteIn this study, the website was used only forbasic psycho-education and recruitment.Communication between clients and therapiststook place though e-mails. At present, thetreatment is delivered completely through astructured, database-driven website. This hasseveral advantages over e-mail exchanges.Privacy is ensured by transparent encryptiontechniques and login procedures. Further,therapists and clients now use a graphicalinterface to access treatment elements, whichenhances access to the treatment dossier andprocess overview. Furthermore, the websiteprovides real-time calculation of results andmultimedia interfaces that support and facil-itate the execution of the exercises. Preliminary(unpublished) findings suggest that the website-driven manual decreases the number of drop-outs to a mere 15%, and that it enhances theimprovement rates. It would be interesting todirectly test the results of an e-mail vs website-driven treatment protocol for chronic stress.

Client-therapist relation and treatmentintegrityKnaevelsrud & Maercker (2006) and Spek etal. (2006) observed that positive relationshipsbetween therapists and clients are quitefeasible in Internet-driven therapy. Thisobservation is supported by the positiveevaluation of our participants of their thera-pists. Yet, this study does not permit conclu-sions regarding the effect of working allianceor the amount of therapist involvement inInternet-driven treatment. Given that somestudies investigating Internet-based self-helpwith minimal therapist contact report reason-able results (e.g. Carlbring, Furmark, Steckzo,Ekselius, & Andersson, 2006), future researchshould focus explicitly on the role of thetherapist: How much therapist contact isoptimal? What is the effect of working alliancein Internet-based treatment?

Internet-based treatment with standar-dized therapist contact may have distinct

190 Ruwaard, Lange, Bouwman, Broeksteeg and Schrieken COGNITIVE BEHAVIOUR THERAPY

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

advantages over face-to-face treatment. InInternet-driven treatment it is easier to ensuretreatment integrity than in manualized face-to-face treatment. In the present manual muchattention is devoted to small details to ensurethat the therapists use the manual in theintended way. The manual even specifies themotivating attitude that the therapists arerequired to adopt and convey to their clients.Furthermore, the weekly supervisions wereintense, and exploited the full details of theexchanges between the therapists and theirclients. The detailed manual provided therelatively young and inexperienced therapistswith guidance and support in treating theirclients. As a consequence, their results were ona par with the results of experienced thera-pists. Apart from the effects on clients, it isinteresting to investigate the effects ofInternet-driven manuals in the education andtraining of therapists.

Acknowledgements

The authors thank the Dutch Foundation ofMental Health (NFGV) in Utrecht for thegrant they awarded us to initiate this project.

ReferencesAlford, W. K., Malouff, J. M., & Osland, K. S.

(2005). Written emotional expression as acoping method in child protective servicesofficers. International Journal of StressManagement, 12, 177–187.

Andersson, G., Carlbring, P., Holmstrom, A.,Sparthan, E., Furmark, T., Nilsson-Ihrfelt, E.,Buhrman, M., & Ekselius, L. (2006). Internet-based self-help with therapist feedback and invivo group exposure for social phobia: arandomized controlled trial. Journal of Consul-ting and Clinical Psychology, 74, 677–686.

Barak, A. (1999). Psychological applications on theinternet: a discipline on the threshold of a newmillennium. Applied & Preventive Psychology, 8,231–245.

Bauer, C. (2000). Limiting factors on the transfor-mative powers of e-mail in patient-physicianrelationships: a critical analysis. HealthCommunication, 12, 239–259.

Beck, A. (2005). The current state of cognitivetherapy: a 40-year retrospective. Archives ofGeneral Psychiatry, 62, 953–959.

Bonger, B. (1988). Clinician, microcomputers, andconfidentiality. Professional Psychology:Research and Practice, 19, 286–289.

Brennan, S. E., & Ohaeri, J. O. (1999). Why doelectronic conversations seem less polite? Thecosts and benefits of hedging. In Proceedings,

International Joint Conference on WorkActivities, Coordination, and Collaboration(WACC 1999) (pp. 227–235). San Francisco,CA: ACM.

Brenninkmeijer, V., & Van Yperen, N. (2003). Howto conduct research on burnout: advantagesand disadvantages of a unidimensionalapproach in burnout research. Occupationaland Environmental Medicine, 60, 16–20.

Carlbring, P., Furmark, T., Steckzo, J., Ekselius,L., & Andersson, G. (2006). An open study ofinternet-based bibliotherapy with minimal con-tact via email for social phobia. ClinicalPsychologist, 10, 30–38.

Carlbring, P., Westling, B., Ljungstrand, P.,Ekselius, L., & Andersson, G. (2001).Treatment of panic disorder via the Internet: arandomized trial of a self-help program.Behavior Therapy, 32, 751–764.

Cohen, J. (1988). Statistical Power Analysis for theBehavioral Sciences (2nd edn). Hillsdale, NJ:Erlbaum.

Cohen, S., Kamarck, T., & Mermelstein, R. (1983).A global measure of perceived stress. Journal ofHealth and Social Behavior, 24, 385–396.

Cooper, C. L., Dewe, P. J., & O’Driscoll, M. P.(2001). Organizational Stress: a Review andCritique of Theory, Research, and Applications.Sage Publications, Thousand Oaks.

Crawford, J. R., & Henry, J. D. (2003). TheDepression Anxiety Stress scales (DASS):Normative data and latent structure in a largenon-clinical sample. British Journal of ClinicalPsychology, 42, 111–131.

De Beurs, E., Van Dyck, R., Marquenie, L. A.,Lange, A., & Blonk, R. W. B. (2001). DeDASS; een vragenlijst voor het meten vandepressie, angst en stress. (The DASS; aquestionnaire for depression, anxiety andstress.). Gedragstherapie, 34, 35–53.

De Jong, G. M., & Emmelkamp, P. M. (2000).Implementing a stress management training:comparative trainer effectiveness. Journal ofOccupational Health Psychology, 5, 309–320.

Emmelkamp, P. M. (2005). Technological innova-tions in clinical assessment and psychotherapy.Psychotherapy & Psychosomatics, 74, 336–343.

Griffith, K. M., & Christensen, H. (2006). Reviewof randomised controlled trials of internetinterventions for mental disorders and relatedconditions. Clinical Psychologist, 10 (1), 16–29.

Health & Safety Executive (2001). Work-RelatedStress: A Short Guide. Sudbury: HSE Books.

Hillis, S. L., & Woolson, R. F. (2002). Analysisof categorized data: use of the odds ratio asa measure of association. In M. T. Tsuang &M. Tohen (Eds), Textbook in PsychiatricEpidemiology (2nd edn, pp. 35–63). Hoboken,New Jersey: J. Wiley & Sons Inc.

Jacobson, N. S., & Truax, P. (1991). Clinicalsignificance: a statistical approach to mean-ingful change in psychotherapy research.Journal of Consulting and Clinical Psychology,59, 12–19.

Joiner, T. E., Steer, R. A., Brown, G., Beck, A. T.,Pettit, J. W., & Rudd, M. D. (2003).

VOL 36, NO 3, 2007 E-mailed CBT of work-related stress 191

Dow

nloa

ded

By:

[Lin

kopi

ng U

nive

rsity

] At:

16:0

2 30

Nov

embe

r 200

7

Worstpoint suicidal plans: a dimension ofsuicidality predictive of past suicide attemptsand eventual death by suicide. BehaviourResearch and Therapy, 41, 1469–1480.

Klein, B., & Richards, J. C. (2001). A brief internet-based treatment for panic disorder. Behaviouraland Cognitive Psychotherapy, 29, 113–117.

Knaevelsrud, C., Jager, J., & Maercker, A. (2004).Internet-psychotherapie: Wirksamkeit undbesonderheiten der therapeutischen beziehung(Online-therapy: Effectiveness and peculiaritiesof the therapeutic relationship). Verhaltensthe-rapie, 14, 174–183.

Knaevelsrud, C., & Maercker, A. (2006). Does thequality of the working alliance predict treat-ment outcome in online psychotherapy fortraumatized patients? Journal of MedicalInternet research, 8, e31.

Lange, A., Richard, R., Gest, A., De Vries, M., &Lodder, L. (1998). The effects of positiveselfinstruction; a controlled trial. CognitiveTherapy and Research, 22, 225–236.

Lange, A., Richard, R., Kiestra, J., & vanOostendorp, E. (1997). Cognitive treatmentthrough positive self-verbalization; a multiplecase study. Behavioural and Cognitive Psycho-therapy, 25, 161–171.

Lange, A., Rietdijk, D., Hudcovicova, M., Van deVen, J.-P., Schrieken, B., & Emmelkamp, P. M.G. (2003). Interapy: a controlled randomizedtrial of the standardized treatment of post-traumatic stress through the internet. Journal ofConsulting and Clinical Psychology, 71,901–909.

Lange, A., Schrieken, B., Blankers, M., Van deVen, J.-P., & Slot, M. (2000). Constructie envalidatie van de Gewaarwordingenlijst (GL);een hulpmiddel bij het signaleren van eenverhoogde kans op psychosen (Constructionof the Screening Device of Psychotic DisorderSDPD; an instrument for detecting a heigh-tened likelihood of psychosis). DirectieveTherapie, 20, 162–173.

Lange, A., van de Ven, J.-P., & Schrieken, B.(2003). Interapy: treatment of posttraumaticstress via the internet. Cognitive BehaviourTherapy, 32, 110–124.

Lepore, S., & Smyth, J. (2002). The Writing Cure;How Expressive Writing Promotes Health andEmotional Well Being. Washington: APABooks.

Levi, L., & Levi, I. (1995). Guidance on Work-Related Stress. Spice of Life, or Kiss of Death?.Luxembourg: Office for Official Publications ofthe European Communities.

Lovibond, S. H., & Lovibond, P. F. (1995). Manualfor the Depression Anxiety Stress Scales.Sydney, Australia: Psychology Foundation.

Maslach, C., Jackson, S. E., & Leiter, F. (1996).Maslach Burnout Inventory: Manual (3rd edn).Palo Alto: Consulting Psychologists Press.

McKay, M., Davis, M., & Fanning, P. (1997).Thoughts & Feelings: Taking Control of YourMoods and Life. Oakland, CA: New HarbingerPublications.

Morin, C. M. (1993). Insomnia; PsychologicalAssessment and Management. NY: GuilfordPresss.

Newman, M. G. (2004). Technology in psychother-apy: an introduction. Journal of ClinicalPsychology, 60, 141–145.

Nieuwenhuijsen, K., De Boer A., G., Verbeek, J.H., Blonk, R. W., & Van Dijk, F. J. (2003). TheDepression Anxiety Stress Scales (DASS):detecting anxiety disorder and depression inemployees absent from work because of mentalhealth problems. Occupational Environment, 60,77–82.

Nijenhuis, E. R. S., Spinhoven, P., Van Dyck, R.,Van der Hart, O., & Vanderlinden, J. (1997).The development of the somatoform dissocia-tion questionnaire (SDQ-5) as a screeninginstrument for dissociative disorders. ActaPsychiatrica Scandinavica, 96, 311–318.

Robey, R. R. (2004). Reporting point and intervalestimates of effect-size for planned contrasts:fixed within effect analyses of variance. Journalof Fluency Disorders, 29, 307–341.

Schaufeli, W. B., & Van Dierendonck, D. (2000).Utrechtse Burnout Schaal: handleiding (TheUtrecht Burnout Scale: manual.). Lisse: Swets& Zeitlinger.

Schaufeli, W. B., & Kompier, M. A. J. (2001).Managing job stress in the Netherlands.International Journal of Stress Management, 8,15–34.

Sparks, K., Faragher, B., & Cooper, C. L. (2001).Well-being and occupational health in the 21stcentury workplace. Journal of Occupational andOrganizational Psychology, 74, 489–509.

Spek, V., Cuijpers, P., Nyklicek, I., Riper, H.,Keyzer, J., & Pop, V. (2006). Internet-basedcognitive behaviour therapy for symptoms ofdepression and anxiety: a meta-analysis.Psychological Medicine, 20, 1–10.

Van der Klink, J. J. L., Blonk, R. W. B., Schene, A.H., & Van Dijk, F. J. H. (2001). The benefits ofinterventions for work-related stress. AmericanJournal of Public Health, 91, 270–276.

Van Doornen, L. (2001). Psychofysiologie vanburnout (psychophysiology of burnout). In C.A. L. Hoogduin, W. B. Schaufeli, C. P. D. R.Schaap & A. B. Bakker (Eds), Behande-lingsstrategieen bij burnout (pp. 40–50).Houten: Bohn Stafleu van Loghum.

Wantland, D. J., Portillo, C. J., Holzemer, W. L.,Slaughter, R., & McGhee, E. M. (2004). Theeffectiveness of web-based vs. non-web-basedinterventions: a meta-analysis o behaviouralchange outcomes. Journal of Medical InternetResearch, 6, e40.

World Health Organization (2001). Atlas: MentalHealth Resources in the World 2001. Geneva:WHO.

Zetterqvist, K., Maanmies, J., Strom, L., &Andersson, G. (2003). Randomized controlledtrial of Internet-based stress management.Cognitive Behaviour Therapy, 32, 151–160.

Zigmond, A. S., & Snaith, R. P. (1983). TheHospital Anxiety and Depression Scale. ActaPsychiatrica Scandinavia, 67, 361–370.

192 Ruwaard, Lange, Bouwman, Broeksteeg and Schrieken COGNITIVE BEHAVIOUR THERAPY