taz kids club evaluation report, february, 2012€¦ · taz kids club evaluation report, february,...
TRANSCRIPT
1
Taz Kids Club Evaluation Report, February, 2012
Allison Matthews1 & Clare Nicholls2
Lecturer/Research Fellow, School of Psychology, University of Tasmania
Research Assistant, School of Psychology, University of Tasmania
Corresponding Author:
Allison Matthews, BSc (Hons), PhD
Lecturer/Research Fellow
University of Tasmania
School of Psychology
Private Bag 30, Hobart, 7001
Email: [email protected]
2
ContentsExecutive Summary ................................................................................................................................. 4
Part 1 – Evaluation of Existing Outcome Data .................................................................................... 4
Part 2 – Recommendations for Future Taz Kids Club Evaluations ...................................................... 5
Evaluation of existing outcome measures ...................................................................................... 5
Suitability of existing outcome measures for evaluation of camps ................................................ 7
Data collation and coding ............................................................................................................... 7
Potential avenues for future research in this area (value adding) ................................................. 8
Taz Kids Club Evaluation –Terms of Contract ....................................................................................... 10
Background: ...................................................................................................................................... 10
Goals of evaluation: .......................................................................................................................... 10
Scope of evaluation:.......................................................................................................................... 10
Part 1 ‐ Evaluation of existing outcome measures ............................................................................... 11
Background on the Taz Kids Club program ....................................................................................... 11
Taz Kids Clubs ................................................................................................................................ 11
PATS Program................................................................................................................................ 12
Champs Camps .............................................................................................................................. 12
Consumer outcomes and Key Performance Indicators .................................................................... 13
Expected Consumer Outcomes ..................................................................................................... 13
Key Performance Indicators .......................................................................................................... 13
Service Delivery statistics .................................................................................................................. 13
Evaluation Methodology ................................................................................................................... 15
Outcome measures ....................................................................................................................... 15
Participants and Procedure ........................................................................................................... 16
Data Analysis ................................................................................................................................. 16
Results ............................................................................................................................................... 17
Analysis of missing data ................................................................................................................ 17
Taz Kids Club Program ................................................................................................................... 18
Champs Camp Program ................................................................................................................ 18
PATS Program................................................................................................................................ 18
Discussion.......................................................................................................................................... 25
Taz Kids Club: Kids ......................................................................................................................... 25
Taz Kids Club: Adolescents ............................................................................................................ 25
PATS program ............................................................................................................................... 26
Champs Camps. ............................................................................................................................. 26
3
Data Limitations ............................................................................................................................ 27
Summary/Conclusions ...................................................................................................................... 28
Part 2 –A review of Outcome Measures relevant for the Evaluation of COPMI Programs .................. 29
Qualitative feedback on existing Taz Kids Club outcome measures ................................................. 29
Literature review: COPMI evaluations and outcome measures ....................................................... 30
Introduction .................................................................................................................................. 30
Risk Factors ................................................................................................................................... 30
Resilience & Protective Factors .................................................................................................... 31
Review of previous COPMI program evaluations ......................................................................... 32
Review of Outcome Measures relevant to COPMI Interventions ................................................ 36
References ............................................................................................................................................ 46
Appendix A – Kids Problems, Connections and Coping Scale ............................................................... 50
Appendix B – Coping Measures ............................................................................................................ 52
Appendix C – Connectedness Measures ............................................................................................... 58
Appendix D – Mental Health Literacy Measures .................................................................................. 61
Appendix E – Self‐esteem measures ..................................................................................................... 66
Appendix F – Resilience measures ........................................................................................................ 71
Appendix G – Affect measures .............................................................................................................. 74
4
ExecutiveSummary
Part1–EvaluationofExistingOutcomeData The Taz Kids Clubs and Champs Camps are primary prevention programs designed to reduce risk
factors and increase protective factors among children of parents with a mental illness (COPMI).
The Taz Kids Club is an 8‐week peer support program where children attend weekly sessions (1‐
2 hours) after school hours. These are fun based sessions where learning about mental illness is
combined with games and creative projects.
In response to the challenge of engaging older teens in the Kidz Club program, Anglicare
implemented the Paying Attention to Self (PATS) program in 2012 for adolescents (aged 12‐17)
who are referred to the Taz Kids program.
Champs Camps are 3 day/2 night school holiday camps that allow respite for young people,
encourage resilience, provide opportunities to make new friends, and provide peer support.
The present evaluation included Kids (aged 7‐11) and adolescents (aged 12‐17) who completed
pre‐ and post‐intervention questionnaires after participating in the Taz Kids Club, PATS, or
Champs Camp programs in 2011 or 2012.
Outcome measures included the Kids Coping scale (comprised of emotion‐focussed, problem‐
focussed, and social support seeking sub‐scales), and the Kids Connections and Kids Problems
scales (both comprised of within‐family and outside‐family sub‐scales).
Data was provided to the authors in excel spread sheets and was checked against raw data
where possible. To aid statistical analysis, pre‐ and post‐intervention data was matched for each
participant where there was sufficient information to do so.
584 children took part in the Taz Kids Clubs (n=238) and Camps (n=346) between 2011 and
2012. Both pre‐ and post‐intervention data were available for 244 participants (97 from clubs
and 147 from camps) and samples sizes for some analyses were restricted due to missing data
within the questionnaire. There are a range of possible reasons to account for this missing data
which are discussed below and in the body of the report.
Given that the data available for the present evaluation represented less than one‐half of all of
the potential data available for collection during this time, the present findings should be
interpreted with caution due to possibility of sampling bias.
From pre‐ to post‐intervention, Kids (aged 7‐11 years) who completed the Taz Kids Club
program showed significant increases on the Coping scale (particularly social support‐seeking)
and the Connections scale (particularly within‐family connections), and an increase in the
number of friends reported. This suggests that the Taz Kids Club program has the potential to
produce positive outcomes that are consistent with the goals of the program for this age group.
An unexpected finding was the increase in problems reported by kids in the program, a result
which may be due to increased sensitisation or awareness of problems within the family as a
result of the intervention.
In contrast, there were no changes on outcome measures for adolescents (aged 12‐17). While
these findings should be interpreted with caution due to the small sample size, they have
implications for the suitability of the Taz Kids Club program and the associated outcome
measures for this age group.
As there were only two PATS programs conducted since its implementation in 2012, the sample
size (n=10) was not adequate to fully evaluate the outcomes of the program and further
research with larger sample sizes is required.
5
With regard to the Champs Camp program, there was no evidence for any changes on outcome
measures from pre‐ to post‐intervention for kids or adolescents which may be related to the
evaluation methodology and the measures used rather than the efficacy of the program. For
example, Anglicare have collected qualitative data which indicates that the camp program has
made a positive difference to children and families that have participated.
Part2–RecommendationsforFutureTazKidsClubEvaluationsFollowing a review of the COPMI literature and examination of existing outcome measures and
associated data the following key points and recommendations can be made in relation to future
evaluations of the Taz Kids Club/Champs Camp program:
Evaluationofexistingoutcomemeasures
Relationshipbetweentheprogramaims/goalsandtheoutcomemeasures The main key performance indicator (as per the service agreement between Anglicare and
DHHS) relevant to the current evaluation is the following: ‘Document how the program made a
difference to the individual’s protective factors, knowledge and understanding of mental illness
and sense of connectiveness to peers, family and community”.
The most commonly targeted protective factors by COPMI interventions include self‐esteem,
knowledge and understanding of mental illness (mental health literacy), coping skills and social
connectedness (Reupert & Maybery, 2009).
The current outcome measures used to evaluate the Taz Kids Club programs assess social
connectedness, problems and coping skills but do not assess mental health literacy and other
protective factors such as self‐esteem or resilience.
o Recommendation: Outcome measures which assess knowledge and understanding of
mental illness (mental health literacy) and self‐esteem should be included in future
evaluations.
The Rosenberg Self‐esteem inventory has been used in previous COPMI evaluations
and may be suitable for use in the Taz kids Club programs.
COPMI evaluations have typically developed mental health literacy measures that are
consistent with the content of the specific program. While the development of a
suitable measure is beyond the scope of the present evaluation, it may be possible for
a Masters of Counselling student within the School of Psychology to conduct a
research project around this in 2013.
Very few evaluations of similar COPMI programs have included a measure such as the Problems
scale used in the present evaluation and the Problems scale is not included as a
recommendation on the National COPMI website.
Increased rather than decreased problems were found among Kids (aged 7‐11) who completed
the the Taz Kids program (possibly due to increased sensitisation to or awareness of problems
within the home). A reduction (or an increase) in problems is not necessarily a main aim of
COPMI programs such as the Taz Kids Club, rather, such programs aim to increase children’s
resilience and ability to cope with and respond to problems which may occur.
In addition, there were large amounts of missing data for this scale (due to participant’s not
providing answers to particular questions) and qualitative feedback from staff suggests that
children may not feel comfortable answering these questions.
o Recommendation: Given the weak link between the Problems scale and the aims of the
intervention it is recommended that this scale is omitted from future evaluations.
6
Suitabilityofexistingmeasuresforadolescents Although there was a small sample size of adolescents in the present outcome evaluation, it is
possible that the null findings for adolescents are related to the appropriateness of the Taz Kids
Club content and/or the outcome measures that are being used to evaluate the program. In
response to previous concerns around program content for adolescents, the PATS program has
now been implemented for adolescents who are referred to the Taz Kids program. However,
the existing outcome measures are currently being used to evaluate this program.
o Recommendation: outcome measures which are more appropriate to the content of the
PATS program and to the developmental age of adolescent participants should be
implemented. Given the higher reading level and cognitive ability of adolescents’, slightly
longer instruments with better psychometric properties could be chosen.
Although the development of a suitable questionnaire is beyond the scope of the
present evaluation, some suggestions can be found below and in the body of the
report and there may be an opportunity for a Masters of Counselling student within
the School of Psychology to follow this up as a research project in 2013.
EvaluationoftheexistingCopingskillsmeasureo The Kids Coping scale (KCS) has below average psychometric properties and while it has been
shown to distinguish between two main forms of coping (problem and emotion focussed), other
research suggests that this distinction is too simplistic. In addition, the emotion focussed scale
measures both distraction and avoidance within the same scale (constructs which may be more
and less adaptive respectively).
o While the social support seeking sub‐scale does not emerge as a clear factor in psychometric
research, it was this subscale in which kids (aged 7‐11) showed an improvement in the present
evaluation, and increasing social support and peer‐networks is a major goal of Taz Kids Club
programs.
o Despite its limitations, the brevity and simplicity of the KCS is a major strength in the
assessment of younger children.
o Recommendation: Given the brevity of the existing Coping scale, it may be suitable to
retain it for future evaluations of the Taz kids Club program with the younger age group
(aged 7‐11).
o Recommendation: Given the psychometric shortcomings of the scale and evidence that
older adolescents engage in a wider range of coping strategies, consideration should be
given to adopting a more comprehensive and psychometrically valid measure of coping
for adolescents (aged 12‐17).
EvaluationoftheexistingSocialconnectednessmeasure The current Social Connectedness scale (questions 13‐19) has not been validated
psychometrically, has been modified from the original scale, and largely assesses connections
within the family but does not adequately assess connections outside of the family (due to
omission of all but two of these items from the original scale).
The current instructions do not include an explanation as how to answer the questions if one
does not have brothers or sisters or has not seen their parent/sibling in the last few weeks
which may have contributed to the large amount of missing data observed for this scale.
A strength of the current Connections scale is its relative simplicity and brevity. In addition, Kids
(aged 7‐11) reported an improvement in within‐family connections from pre‐ to post‐
intervention in the present evaluation.
7
o Recommendation: Further consideration should be given to the appropriateness of the
current Connections measure for kids. Specifically, items to measure outside family
connections, and the original instructions and response options (smiley faces) could be
re‐included (see original scale here).
o Recommendation: consideration should be given to adopting a more comprehensive and
psychometrically valid measure of social connectedness for adolescents.
The three additional connections questions which are not included in the sub‐scale score
(questions 10‐12) were considered problematic by staff and there was a large amount of
missing data for these questions suggesting they may have been unacceptable for children.
However, kids (aged 7‐11) did report an increase in the number of friends (question 10)
between pre‐ and post‐intervention for the Club program.
o Recommendation: Questions 10‐12 (or at least Questions 11‐12) of the Connections scale
could be omitted.
Suitabilityofexistingoutcomemeasuresforevaluationofcamps The suitability of the existing outcome measures for evaluation of camps is a major concern due
to the short time frame (1‐2 days) between the pre‐ and post‐measures and the limited
potential for any changes to occur during this period of time. As such, the following
recommendations are made in relation to the outcome measures used for camps.
o Option 1: One option is to send out the post‐intervention questionnaire at a later time
point (e.g., 4‐week follow) with a reply paid envelope. However, given the experience of
low return rates in previous follow‐up surveys with this population, option 2 may be
preferable.
o Option 2: Measures are chosen which focus on characteristics which have the potential
to change during the children’s time at camp. Such measures should be aligned with the
goals/aims of the camps and could include constructs related to self‐esteem, self‐
efficacy, emotional well‐being, optimism, or positive affect.
While the development of a suitable measure is beyond the scope of the present
evaluation, it may be possible for a Masters of Counselling student within the School
of Psychology to conduct a research project around this in 2013.
Datacollationandcoding The interpretation of the current evaluation findings was limited due to missing data. Thus it is
recommended that, where possible, steps are taken to increase completeness of data sets.
o Recommendation: Steps to improve data fidelity could include: improving the
instructions on the questionnaire (e.g., explaining how to respond if one does not have
brothers or sisters or has not seen their parent/sibling in the last few weeks), stressing
the importance that all questions are completed, systematically going through a
standardised script when administering the questionnaire (particularly with younger
groups) including reading aloud the questions to account for differences in
reading/cognitive ability, and checking that questionnaires are complete when handed
in.
Given current knowledge of evaluation methodology and statistical analysis, the following
recommendations regarding data collation and coding can be made. Please note that the
authors are willing to liaise with the service provider to develop new spread sheets for collating
data and also to assist in meaningful statistical analysis of such data:
8
o Recommendation: Data should be coded so that pre and post intervention data can be
matched across individuals. Comparisons are best made across a sample of people who
contribute data at both pre‐ and post‐intervention. This means each child acts as their
own control in terms of assessing change from baseline.
Where there is missing data (either pre or post), it may be best if this data is excluded
in subsequent analyses so that it does not artificially affect the results. That said, all
data should be collated initially to allow description of the data available.
o Recommendation: For the purposes of methodological rigour and to satisfy the
assumptions of particular statistical analyses (e.g., ANOVA), it is important to ensure true
independence of results. That is, it is necessary to know whether a particular child has
completed previous clubs/camps. This could be achieved by using a unique code
whenever data is entered into databases. This would ensure that repeat consumers
could be easily identified when data is collated across multiple interventions.
An example of a unique code is the participant’s initials followed by their birth month
and year (e.g., AM0577). Including the participant’s name and exact birthdate would
achieve the same purpose; however, a unique code has the additional advantage of
de‐identifying data and ensuring anonymity.
Potentialavenuesforfutureresearchinthisarea(valueadding) The present program represents an opportunity to conduct interesting and useful research in
the COPMI area. This could include the identification of correlates of intervention success
(positive change) and risk factors for non‐success/change. Such research would allow program
facilitators to identify individuals at greater risk and to focus and tailor interventions
accordingly. Through peer‐reviewed publication, such research would also help to provide an
evidence base for use by other health professionals in this area.
o Recommendation: Research to investigate correlates of program success would be
possible with the systematic collection of contextual data. The collation of such
contextual information would also be useful for describing the characteristics of
consumers for general reporting purposes. While it is believed that some of this
information is currently collected on Taz Kids Club referral forms, the information needs
to be systematically collated and linked to the outcome data for individual children. This
would be possible through entering the data into a single database or using a unique
code (see above) to link data within separate referral and outcome databases.
Relevant contextual information could include: referral pathway, demographics
such as which parent(s) has a mental illness, the parents’ diagnosed mental illness,
the current living arrangements of the child (e.g., in care of parent or other
arrangement), urban/rural/remote location.
The following points may not be relevant (or feasible to consider) within the context of the Taz
Kids program but may be methodological issues to consider if peer‐reviewed publication of
evaluation findings is a priority.
o Demonstration that any gains are maintained over a longer period of time gives stronger
and more compelling evidence of program efficacy. This is generally the gold standard for
evaluation methodology. One suggestion would be to implement an additional follow up
period where participants are sent the evaluation questionnaire (with a reply paid
envelope) at 4‐weeks post intervention (resulting in pre‐, post, and follow‐up measures for
9
each participant). While this would enhance scientific rigour, the decision to employ such
an approach would need to consider the potential for a low response rate at follow‐up, an
issue identified in previous attempts to collect follow‐up data from this population.
o Greater scientific rigour can also be demonstrated by employing a control group who
receive ‘no intervention’ or who receive another unrelated intervention. This allows any
changes on outcome measures to be attributed to the intervention rather than the passage
of time or some other factor such as developmental trajectories. At best, the control group
would have similar characteristics to the intervention group (i.e., COPMI) and would
complete the outcome measures within the same timeframe. An issue with implementing
this methodology in the field is the ethics of ‘not providing a treatment’ to the control
group. This could be countered if there was an existing wait‐list to enter the program and
therefore the control group received the intervention at a later date.
o Given the requirements of many peer‐reviewed scientific journals, it may be necessary to
obtain ethics approval from the Human Research Ethics Committee at the University of
Tasmania if evaluation data were to be published. The authors would be willing to facilitate
this process if required.
10
TazKidsClubEvaluation–TermsofContract
Background:Children of parents with mental illness (COPMI) have a heightened risk of experiencing a
range of psychosocial difficulties (e.g., behavioural, social, academic) and have a higher risk
of developing mental health problems themselves. ‘Taz Kids Clubs’ is an early intervention
program which provides support and educational opportunities for young people in
Tasmania who have a parent or relative with a mental illness. To date there has been no
systematic evaluation of the Taz Kids Clubs program in terms of its effectiveness as an early
intervention for this target group. The present research aims to investigate the effects of
the intervention on outcome measures (e.g., problems, connections and coping) among
children who have participated. In addition, consideration will be given to the
appropriateness of the existing outcome measures and recommendations will be made for
consideration in further evaluation research.
Goalsofevaluation:1) To investigate the effectiveness Taz Kids Clubs program in terms of changes on
existing pre‐ and post‐intervention outcome measures.
2) To evaluate the usefulness of the evaluation measures currently administered as
part of the Taz Kids Clubs program and make recommendations regarding potential
outcome measures for future evaluative research.
Scopeofevaluation:1) Undertake a review of the literature in this area; collate and analyse existing data
and provide an outcome evaluation of the program based on these existing
measures.
2) Examine the program in terms of its key aims and goals (including qualitative
discussions/focus groups with key staff members); Examine whether the existing
measures accurately assess outcomes which relate to the key aims and goals of the
program; write an evaluation report including recommendations for improvements
to existing outcome measures for consideration in further evaluative research.
11
Part1‐Evaluationofexistingoutcomemeasures
BackgroundontheTazKidsClubprogramThe Taz Kids Clubs program is supported by the Tasmanian government Kids in Mind Tasmania
Initiative. These primary prevention programs are designed to reduce the risk factors and increase
the protective factors for children of parents with a mental illness (COPMI). Anglicare Tasmania Inc.
was first contracted by the Tasmanian Department of Health and Human Services (DHHS) to deliver
Taz Kids Clubs and Champs Camps in 2008.
The target group for Taz Kids Clubs and Champs Camp programs are children of parents with a
diagnosed mental illness aged 18 years or less. In addition to the Taz Kids clubs and Champs Camps,
a Parenting Support Program provides practical support to the parents of the children involved in
the clubs and camps, and day trips (e.g., bowling, cable hang gliding, picnics, laser skirmish, craft
days, sailing) are also organised for children who cannot attend camp and so that children can
reconnect with friends previously made at camps or clubs and have a short, one day respite.
Referral to the Taz Kids Club/Champs Camp program occurs through a variety of ways. Estimates
provided by key personnel indicate that the usual referral pathways to the program are as follows:
20% self‐referral (this includes siblings who are now old enough to join Taz Kids), 20% school
workers, 10% medical staff (e.g., Mental Health/GPs), 10% internal Anglicare services, 40% Other
service providers (e.g., Mission, Gateway, Salvation Army, Headspace etc). Some children participate
in both the Club and Camp programs and/or attend multiple Clubs/Camps.
The main focus of the present evaluation is on the existing measures that are employed to evaluate
the Taz Kids Club and Champs Camp programs. As such a brief outline of these programs is given
below.
TazKidsClubsThe Taz Kids Club is an 8‐week peer support program where children attend weekly sessions (1‐2
hours) after school hours. Occasionally the program is run over 6 weeks with Sessions 1 and 2 and
Sessions 7 and 8 combined. The 8‐session program is based on the ‘Taz Kidz Club manual’ which was
developed by Christine Handley (Nurse Clinician Lecturer) and Angela Josephs (Clinical Psychologist)
(Handley & Josephs, 2002) and funded by Commonwealth Reform and Incentive Funding. The
session plans in the manual were based on the ‘Kidz Club’ program developed by Sue O’Rourke
(Social Worker) and Jane O’Sullivan (Child Therapist) from the Child and Youth Mental Health
Service, Mater’s Children Hospital, Brisbane, Australia.
A brief outline of the Taz Kids Club sessions is given below. These are fun based sessions where
learning about mental illness is combined with games and creative projects. This is done in a safe
environment where questions are encouraged and support is given to explore difficult issues. The
evaluation outcome measures are currently administered in the first and last session of the program.
12
Session 1 ‐ Getting to know each other and an explanation of what’s ahead; Making a ‘tool’ box and folder; completion of pre‐intervention questionnaire Session 2 – The myths and facts of mental illness: education and discussion about mental illness Session 3 – Discussing symptoms of mental illness and coping strategies for dealing with these symptoms (Making pizzas) Session 4 – Identifying and coping with stress (includes a relaxation session) Session 5 – Types and explanations of treatments for mental illness Session 6 –Families, supports and family responsibilities: Exploration of the role of family members when a parent/relative is unwell focussing on the responsibility often assumed by young people and identifying the supports available to young people. Session 7 – Feelings explored Session 8 – Discussion about stigma and how to reduce its effects; completion of post‐intervention assessment; certificate hand out and party time with parents attending.
PATSProgramThe Taz Kids program was initially developed for children aged 7‐12 years, but in recent years it was
delivered to both kids (7‐12) and adolescents (12‐17). In response to the challenge of engaging older
teens in the Kidz Club program, Anglicare implemented the Paying Attention to Self (PATS) program
in 2012 for adolescents who are referred to the Taz Kids program. This program has been initiated in
collaboration with local high schools, with some schools allowing the program to be facilitated in
school curriculum time. In 2012, there was one program delivered in the North/North‐west and one
program delivered in the South of the state.
The PATS program was developed in 1997 as a response to an identified lack of support and
resources for adolescents (aged 12‐18 years) whose parents have a mental illness (Hargreaves et al.,
2005). PATS aims to provide young people with the opportunity to share their experiences and be
supported by others in similar situations. The activities in PATS aim to increase knowledge of mental
illness, improve help‐seeking and coping behaviours and improve their sense of connection to peers,
family and community. Each PATS group comprises 6‐8 adolescents who meet weekly for 8 weeks.
Activities include discussions, games, role‐plays, art therapy, guest speakers and social outings with
the overall focus being on healthy thinking and coping strategies. The topics addressed in PATS
include: Understanding mental illness, improving relationships with parents, communication and
problem solving skills, dealing with the stigma of mental illness, and developing strategies to cope
with parental mental illness and to stay mentally healthy themselves.
ChampsCampsChamps Camps are 3 day/2 night school holiday camps that allow respite for young people,
encourage resilience, provide opportunities to make new friends, and provide peer support. Champs
Camps provide a range of support and activities creative and outdoor activities, challenging activities
and psycho‐education. Currently pre‐ and post‐intervention measures are administered on day 1 and
on the final day (day 2 or 3) of the program. In 2013 a new format for camps is being implemented
with a shortened time frame of 2 days/1 night.
13
ConsumeroutcomesandKeyPerformanceIndicatorsAs per the service agreement between Anglicare Tasmania Inc. and The Tasmanian Department of
Health and Human Services (DHHS), the following expected outcomes and key performance
indicators were identified in relation the Taz Kids Clubs /Champs Camps.
ExpectedConsumerOutcomes Strengthen the protective factors and reduce the risk factors for adverse mental health
outcomes among children of parents with mental illness
Increase children and young people’s knowledge and understanding of mental illness and
ways to maintain their own wellbeing
Improve a sense of connection to peers, family and community
KeyPerformanceIndicators The achievement of the purpose of the Funding agreement and/or consumer outcomes will
be monitored in by reference to the following key performance indicators:
Using the COPMI evaluation principles:
o Provide the number of children and adolescents participating in each Club or Camp
and regional spread of the programs.
o Document how the program made a difference to the individual’s protective factors,
knowledge and understanding of mental illness and sense of connectiveness to
peers, family and community.
o Evidence of a robust relationship with the National COPMI initiative
The focus of the present evaluation is the second key performance indicator (i.e., changes to the
individuals’ protective factors, knowledge and understanding of mental illness and sense of
connectiveness to peers, family and community). However, given that the number of children and
adolescents participating in the Taz Kids Club/Champs Camp programs is also a key performance
indicator specified in the service agreement, a summary of service provision data between Jan‐June,
2011 and July‐Dec, 2012 is given below.
ServiceDeliverystatisticsService delivery data was available to the authors in the form of bi‐annual reports between Jan‐June,
2011 and July‐Dec, 2012. Table 1 shows the number of clubs, camps and day trips completed during
these time periods. Tables 2 and 3 give a breakdown of the number of camps and clubs delivered in
the North‐North‐West and South of the state the number of children and adolescents who
participated. Based on the date supplied, there were a total of 238 children (177 kids, 61
adolescents) who took part in Clubs (Table 2), and 346 children (255 kids, 91 adolescents) who took
part in Camps (Table 3) state‐wide in the two‐year period between 2011 and 2012.
Table 1. Number of Clubs/Camps/Day trips and total number of children state‐wide, 2011‐2012
Reporting period
Total clubs Total camps Total day trips (No. Kids & adolescents)
Total children (Total participations)
July‐Dec 2012 6 x Kids, 1 x PATS 3 7 (114K, 32A) 208 (313)
Jan‐June 2012 5 x Kids, 1 x PATS 7 7 (95K, 31A) 325
July‐Dec 2011 6 x Kids 3 7 (80K, 30A) 215
Jan‐June 2011 4 x Kids 4 6 (26K, 28A) 130
Note: K=Kids, A=adolescent
14
Table 2. Number of Taz Kids clubs and total number of children/adolescents in the North/Northwest and South of Tasmania, 2011‐2012
North/North‐West South
Reporting period
No. clubs No. of kids No. of Adolescents
Pre/post evaluations
No. of clubs No. of kids No. of Adolescents
Pre/post evaluations
July‐Dec 2012
3 x Kids 46 8 54/54 3 x Kids, 1x PATS
27 4 31/31
Jan‐June 2012
2 x Kids, 1 x PATS
23 11 20/21 3 x Kids 18 6 13/18
July‐Dec 2011
3 x Kids 17 5 18/16 3 x Kids 18 6 24/15
Jan‐June 2011
3 x Kids 28 12 19/17 1 x Kids 0 9 9/9
Table 3. Number of Champs Camps and total number of children/adolescents in the North/Northwest and South of Tasmania, 2011‐2012
North/North‐West South
Reporting period
No. camps No. of kids No. of Adolescents
Pre/post evaluations
No. camps No. of kids No. of Adolescents
Pre/post evaluations
July‐Dec 2012
2 43 19 53/56 1 19 1 20/20
Jan‐June 2012
4 46 18 40/58 3 49 11 0/0
July‐Dec 2011
1 (+ 1 combined S/N/NW)
22 7 29/29 1 (+ 1 combined S/N/NW)
27 3 30/30
Jan‐June 2011
3 41 20 41/41 1 8 12 14/14
15
EvaluationMethodology
OutcomemeasuresA copy of the outcome questionnaire currently used in the Taz Kids Club program can be
found in Appendix A of the present report and a description of the measures contained in the
questionnaire is provided below.
Kids Coping Scale. The Kids Coping Scale (Maybery, Steer, Reupert, & Goodyear, 2009a) is a
9‐item scale which consisting of three‐subscales: problem focused coping (PFC), emotion focused
coping (EFC) and social support seeking (SSS). The 4‐item PFC scale included items such as ‘You tried
to think of different ways to solve to the problem’, the three item EFC scale included items such as
‘You avoided the problem or where is happened’ (EFC), and the 2‐item SSS subscale consisted of
items such as ‘You asked someone to help’ (SSS). Responses were made on a three point Likert scale,
from ‘Never (0)’, ‘Sometimes (1)’, to ‘A lot (2)’. Scores for each item were summed and averaged by
the number of items to form the total score, and mean scores for each of the three sub‐scales were
also calculated.
Kids Connections and Kids Problems Scales. The Kids Connections and Kids Problems Scales
(Maybery, Reupert, & Goodyear, 2006a) were designed to measure the key positive relationships
(Connections) and key negative relationships for children (Problems), both within and outside the
family. For the 7‐item Connections scale, participants rated how often they had a good time with
various people in the last few weeks (e.g., ‘Time spent with your Mum’ and ‘Good time spent with
your sisters’). For the 7‐item Problems scale, participants rated how often they had bad times with
various people (e.g., You had problems with your Dad’ and ‘You had problems with a friend’). In the
present research, responses were made on a three point likert scale, from ‘Never (0)’ ‘Sometimes
(1)’ to ‘A lot (2)’. According to the procedure described by (Maybery, et al., 2006a), the total
Connections or Problems score was calculated by summing responses to all items and averaging by
the number of items. The first five items of each scale was summed and averaged by the number of
items to form a ‘within family connections/problems’ score and the remaining two items were
summed and averaged to form an ‘outside family connections/problems’ score.
It should be noted that the Problems and Connections scales used for the Taz Kids Club
program differed from the original scales used in previous research (Maybery et al., 2006) which
may affect the reliability/validity of the measures. The original Connections and Problems scales had
10 and 11 questions respectively and included more questions relating to connections/problems
outside of the family. In addition, the original scales included more detailed instructions on how to
respond, and responses were made on a four point Likert scale with smiley faces for the Kids
Connections scale ranging from ‘None/Did not happen’, to ‘All the time’, and sad faces for the
Problems scale ranging from ‘No/No problem’ to ‘Major problem’.
Three questions regarding connections/friendships were also included in the Taz Kids Club
questionnaire. These questions required a written number response and are not included in the Kids
Connections score. The items were ‘How many friends do you have?’, ‘In the last month, how many
times have you gone to a friend’s house to play?,’ and ‘In the last month, how many times have you
had a friend over to play?’. According to the method previously adopted by Maybery, et al. (2006a),
responses above 20 were recoded to an upper limit of 20 to control for outliers. Where a response
other than a number was provided (e.g., the word ‘lots’), this was treated as missing data.
16
ParticipantsandProcedureFor both the Taz Kids Club and Champs Camp programs, participants completed the
evaluation questionnaire on the first day and last day of the program. Thus the pre‐ to post‐
intervention period was 8 weeks for the Clubs and 2‐3 days for the Camps. Pre‐and post‐evaluation
data was supplied by Anglicare in both excel data files and in paper questionnaire format. Where
possible, data collated in excel files was checked against the original questionnaire data. As a result
of this process, some data inconsistencies (possibly due to data entry error) were identified and
changed to be consistent with raw data. The pre‐ and post‐intervention data were originally coded
as separate entries in the excel data files. In order to provide an accurate statistical evaluation of
outcome measures, it was necessary to match the pre‐ and post‐evaluation data for each individual.
This process was difficult as due to inconsistencies (e.g., dates, locations, and participant names) in
the data files supplied.
According to service provision data (see Tables 2 and 3), 584 children took part in the Taz
Kids Clubs (238) and Camps (346) between 2011 and 2012. For this two year period, there were a
total of 325 unique data entries available for the present evaluation (representing 55% of the total
potential data). Of these, 81 participants were excluded from analysis due to missing data at either
pre‐ or post‐intervention time points, resulting in 244 participants (and 41% of total potential data)
where both pre‐ and post‐intervention data could be matched (97 from clubs and 147 from camps).
The break‐down of this data in terms of program (camp vs. club), age group (kids vs. adolescents),
location (N/NW vs. South), and sex (male vs. female) is provided in Table 4 below. The mean age of
Kids was 9 years (SD=1.4, range 6‐11) and the mean age of adolescents was 13 years (SD=1.2, range
12‐16).
Table 4. Number of participants in the Taz Kids program evaluation sample (n=244)
North/North‐west South
Kids Adolescents Kids Adolescents
Clubs Total n 32 12 35 6
Male 16 8 17 4
Female 16 4 18 2
Camps Total n 82 33 22 10
Male 40 12 9 4
Female 42 21 13 6
Note: 12 adolescents who participated in Taz Kids Clubs received the PATS program
DataAnalysisFor over one‐half (53%) of the 244 participants, there was missing data on at least one of the
26 questions on the pre‐ or post‐intervention questionnaires. A summary of the missing data is
available in the results section. Given that mean values were calculated for each of the sub‐scales
(coping, connections, problems), participants with missing data were excluded from analysis. To
maximise the data available, these data were excluded per analysis (i.e., sub‐scale). The total
number of participants available per analysis (n) is given in the results section below. Data were
analysed separately for each program type (Taz Kids vs. camp vs. PATS), each age group (kids vs.
adolescents), and each subscale (coping, connections, problems). Paired sample t‐tests were
conducted to examine whether there were statistical significant differences between pre‐ and post‐
intervention scores for each of these categories. Statistical significance was denoted at the alpha
level of .05.
17
Results
AnalysisofmissingdataTable 5 shows the number of data points available for each of the 26 questions (and for each
of the 3 scales) for kids and adolescents. There was over 95% of data available for each individual
question on the Coping scale, and 92% of data complete in total. The data available for the
Connections and Problems scales was considerably lower, representing between 64% and 69% of the
total potential data points. There was also considerable missing data for questions 10‐12 (extra
connections questions), such that between 76‐82% of data was available. This was particularly
evident for Kids, where one‐quarter of the data was missing for questions 11 and 12.
Table 5. Summary of data available for analysis (n and total % of data points) for kids and
adolescents at pre‐ and post‐intervention.
Question Kids (n=171) Adolescents (n=73)
Pre (n) Post (n) Total % Pre (n) Post (n) Total %
Total Coping Scale 161 154 92 68 66 92
1. Try to think of different ways to solve problem
171 170 100 73 73 100
2. Don’t want to think about the problem 168 169 99 71 72 98
3. You think about what others might do 170 169 99 71 72 98
4. You try your best to make things better 169 168 99 71 71 97
5. You avoid the problem or where it happened
166 169 98 72 71 98
6. You ask someone to help 167 169 98 72 73 99
7. You try hard to fix the problem 165 168 97 72 72 99
8. You do things to stop thinking about it 167 167 98 72 72 99
9. If it is your fault, you say sorry 164 170 98 72 72 99
Connections (extra questions)
10. How many friends do you have? 151 131 82 58 58 79
11. In the last month, how many times have you gone to a friend’s house to play?
142 121 77 61 58 82
12. In the last month, how many times have you had a friend over to play?
138 123 76 61 57 81
Total Connections Scale 131 105 69 51 45 65
13. Time spent with your Mum 168 165 97 70 70 96
14. Time spent with your Dad 167 160 96 70 72 97
15. Good time spent with your Brothers 160 157 93 62 65 87
16. Good time spent with your Sisters 160 153 92 67 69 93
17. Time spent with a special Grandparent or other relative
160 156 92 64 63 87
18. Time spent with your friend 168 165 97 71 70 97
19. Another grown up? 141 135 81 59 61 82
Total Problems Scale 130 98 67 52 42 64
20. You had problems with your Mum 167 169 98 70 68 95
21. You had problems with your Dad 160 156 92 70 69 95
22. You had problems with your Brothers 157 154 91 66 65 90
23. You had problems with your Sisters 155 150 89 67 66 91
24. You had problems with a Grandparent (or other relative)
158 150 90 63 63 86
25. You had problems with a friend 165 156 94 72 70 97
26. You had problems with another person?
143 133 81 72 54 86
18
TazKidsClubProgramThe total mean scores on the Coping, Connections, and Problems scales (and each of their
respective sub‐scales) are shown at pre and post‐intervention for both Kids (Figure 1) and
adolescents (Figure 2) who participated in the Taz Kids Club program. Mean scores on ‘additional
connections questions’ are shown in Figures 3 and 4 for Kids and Adolescents respectively.
Kids (aged 7‐11). Statistical analysis showed that for Kids there was a significant increase
from pre‐ to post‐intervention in the total Coping scale score, t(60)=‐2.49, p=.016 (Figure 1a), the
total Connections score, t(43)=‐2.60, p=.013 (Figure 1b), and the total Problems score, t(36)=‐2.53,
p=.016 (Figure 1c). Within the Coping scale, a significant increase was found for the Social support
seeking sub‐scale, t(60)=‐3.15, p=.003, but not for the Problem or Emotion based coping sub‐scales.
Within the Connections scale a significant increase was found for Within‐family connection scores,
t(43)=‐3.03, p=.004, but not Outside‐family connection scores. Similarly, for the Problems scale, a
significant increase was found for Within‐family problems scores, t(36)=‐2.49, p=.018, but not
Outside‐family problems scores. For the additional Connections questions (Figure 3), Kids in the club
program showed a statistically significant increase in the number of friends between pre‐ and post‐
intervention, t(39)=‐3.52, p=.001. The number of visits to friends’ and friends’ to visit also tended to
increase over time but did not reach statistical significance.
Adolescents (aged 12‐17). Statistical analysis did not reveal any significant changes from
pre‐ to post‐intervention on any of the scales or sub‐scales for adolescents.
ChampsCampProgramThe total means scores on the Coping, Connections, and Problems scales (and each of their
respective sub‐scales) are shown at pre and post‐intervention for both Kids (Figure 5) and
adolescents (Figure 6) who participated in the Champs Camp program. Mean scores on additional
connections questions are shown in Figures 7 and 8 for Kids and Adolescents respectively.
Kids (aged 7‐11). Statistical analysis did not reveal any significant changes from pre‐ to post‐
intervention on any of the scales or sub‐scales for kids.
Adolescents (aged 12‐17). Statistical analysis did not reveal any significant changes from
pre‐ to post‐intervention on any of the scales or sub‐scales for adolescents.
PATSProgramFigure 9 shows the mean scores on coping scale for adolescents (n=10) who completed the
PATS program in 2012. There were no statistically significant changes from pre‐ to post‐intervention.
Only five of the ten PATS participants filled in all questions on the Connections sub‐scale and just
three filled in all questions on the Problems sub‐scale which is not sufficient data for statistical
analysis.
19
Taz Kids Club Program – Kids (aged 7‐11)
(a)
(b)
(c)
Figure 1. Mean Coping (a, n=61), Connections (b, n=44), and Problems(c, n=37) scale scores for kids
(aged 7‐11) in the Taz Kids Club Program (* indicates statistically significant differences, p<.05).
0.0
0.5
1.0
1.5
2.0
Total copingscore
Problem‐basedcoping
Emotion‐basedcoping
Social‐supportbased coping
Mean
score
Pre Post
0.0
0.5
1.0
1.5
2.0
Total connectionsscore
Within‐familyconnections
Outside‐familyconnections
Mean
score
Pre Post
0.0
0.5
1.0
1.5
2.0
Total problemsscore
Within‐familyproblems
Outside‐familyproblems
Mean
score
Pre Post
20
Taz Kids Club Program – Adolescents (aged 12‐17)
(a)
(b)
(c)
Figure 2. Mean Coping (a, n=18), Connections (b, n=13), and Problems(c, n=12) scale scores for
adolescents (aged 12‐17) in the Taz Kids Club Program (* indicates statistically significant differences,
p<.05).
0.0
0.5
1.0
1.5
2.0
Total copingscore
Problem‐basedcoping
Emotion‐basedcoping
Social‐supportbased coping
Mean
score
Pre Post
0.0
0.5
1.0
1.5
2.0
Total connectionsscore
Within‐familyconnections
Outside‐familyconnections
Mean
score
Pre Post
0.0
0.5
1.0
1.5
2.0
Total problemsscore
Within‐familyproblems
Outside‐familyproblems
Mean
score
Pre Post
21
Figure 3. Mean number of friends (n=40), visits to friends (n=34)’, and friends’ to visit (n=32) for kids
(aged 7‐11) in the Taz Kids Club Program (* indicates statistically significant differences, p<.05).
Figure 4. Mean number of friends (n=15), visits to friends’ (n=11), and friends’ to visit (n=10) for
adolescents (aged 12‐17) in the Taz Kids Club Program (* indicates statistically significant differences,
p<.05).
0
2
4
6
8
10
12
Number friends Visits to friends Friends to visit
Mean
numbner
Pre Post
0
2
4
6
8
10
12
Number friends Visits to friends Friends to visit
Mean
numbner
Pre Post
22
Champs Camps ‐ – Kids (aged 7‐11)
(a)
(b)
(c)
Figure 5. Mean coping (a, n=93), connections (b, n=61), and problems(c, n=61) scale scores for kids
(aged 7‐11) in the Champs Camp Program (* indicates statistically significant differences, p<.05).
0.0
0.5
1.0
1.5
2.0
Total copingscore
Problem‐basedcoping
Emotion‐basedcoping
Social‐supportbased coping
Mean
score
Pre Post
0.0
0.5
1.0
1.5
2.0
Total connectionsscore
Within‐familyconnections
Outside‐familyconnections
Mean
score
Pre Post
0.0
0.5
1.0
1.5
2.0
Total problemsscore
Within‐familyproblems
Outside‐familyproblems
Mean
score
Pre Post
23
Champs Camps ‐ – Adolescents (aged 7‐11)
(a)
(b)
(c)
Figure 6. Mean Coping (a, n=37), Connections (b, n=27), and Problems(c, n=27) scale scores for
Adolescents (aged 12‐17) in the Champs Camp Program (* indicates statistically significant
differences, p<.05).
0.0
0.5
1.0
1.5
2.0
Total copingscore
Problem‐basedcoping
Emotion‐basedcoping
Social‐supportbased coping
Mean
score
Pre Post
0.0
0.5
1.0
1.5
2.0
Total connectionsscore
Within‐familyconnections
Outside‐familyconnections
Mean
score
Pre Post
0.0
0.5
1.0
1.5
2.0
Total problemsscore
Within‐familyproblems
Outside‐familyproblems
Mean
score
Pre Post
24
Figure 7. Mean number of friends (n=87), visits to friends’ (n=77), and friends’ to visit (n=81) for kids
(aged 7‐11) in the Champs Camp Program (* indicates statistically significant differences, p<.05).
Figure 8. Mean number of friends (n=28), visits to friends’ (n=33), and friends’ to visit (n=34) for
adolescents (aged 12‐17) in the Champs Camp Program (* indicates statistically significant
differences, p<.05).
Figure 9. Mean coping scale scores for adolescents (aged 12‐17) in the PATS Program (n=10) (*
indicates statistically significant differences, p<.05).
0
2
4
6
8
10
12
Number friends Visits to friends Friends to visit
Mean
numbner
Pre Post
0
2
4
6
8
10
12
Number friends Visits to friends Friends to visit
Mean
numbner
Pre Post
0.00
0.50
1.00
1.50
2.00
Total copingscore
Problem‐basedcoping
Emotion‐basedcoping
Social‐supportbased coping
Mean
score
Pre Post
25
Discussion
TazKidsClub:KidsFindings from the present evaluation showed that Kids (aged 7‐11) who attended the club
program showed significant increases from pre‐ to post‐intervention on the Coping scale (in
particular social support‐seeking coping), the Connections scale (in particular within‐family
connections) and the Problems scale (in particular within‐family problems). Kids also showed a
significant increase in the number of friends reported at post‐ relative to pre‐intervention.
These findings suggest an improvement in children’s wellbeing following participation in Taz
Kids Club and are relatively consistent with the findings of an evaluation of a similar COPMI program
(CHAMPS) conducted in Victoria (Goodyear, Cuff, Maybery, & Reupert, 2009). For example,
Goodyear et al (2009) found increases in social‐support seeking coping and within‐family
connections but no changes overall for emotion‐focused coping, problem‐focussed coping or
outside‐family connections (Goodyear, et al., 2009). There was also a significant increase in self‐
esteem, a construct which was not measured for the Taz Kids Club evaluation. However, in contrast
to the findings of Goodyear et al. (2009) increased rather than decreased scores were found on the
problems scale in the present study. One explanation for these opposing results is that the program
may have sensitised participants or increased their awareness of problems or difficulties within the
home.
Although there were no changes on the emotion focussed coping sub‐scale, the coping
strategies assessed by this scale may not necessarily be adaptive ways of coping as they largely focus
on dealing with problems using different forms of avoidance. While avoidance may be an adaptive
short‐term coping mechanism, it may be an ineffective long‐term coping mechanism (Connor‐Smith,
Compas, Wadsworth, Thomsen, & Saltzman, 2000; Dumon & Provost, 1999). In addition, in a
previous evaluation of a COPMI intervention in Canada (The Kids in Control program) a reduction
was observed on this scale from pre‐to‐post intervention possibly reflecting a shift toward other
more adaptive approaches for handling emotions (Richter, 2006, unpublished Masters thesis).
Given that there was an increase in the number of friends reported by children, it is
interesting that there were no improvements in terms of outside‐family connections. However, this
finding is consistent with that of Goodyear et al. (2009) and other evaluations of peer support
programs which have used other measures of ‘outside’ social connections (Fraser & Pakenham,
2008; Hargreaves et al., 2005). In addition, the scale used in the present study was modified such
that all but two of the outside‐family connection questions were removed. As such there were
probably insufficient items available to measure this construct.
TazKidsClub:AdolescentsFor adolescents who completed the Taz kids program, there were no statistically significant
changes on any measure from pre‐ to post‐intervention. It should be noted that the sample size for
adolescents was smaller relative to kids and this may have impacted on the results. However, given
that the Taz Kids club program was developed for children aged 7‐11 years, it is also possible that
the material is not necessarily appropriate for engaging the older children who take part. This was an
issue previously identified by Anglicare and has resulted in the PATS program being implemented for
adolescents who are referred to the Taz Kids program. Given the findings of the present evaluation it
is likely that adolescents are likely to benefit more from inclusion in the PATS rather than the Taz
Kids Club program.
26
PATSprogramAs there were only two PATS programs conducted since its implementation in 2012, the
sample size (n=10) was not adequate to fully evaluate the outcomes of the program and further
research with larger sample sizes is required. It is also possible that evaluation of the PATS program
could be improved by adopting outcome measures which are more appropriate for this age group.
For example, due to the higher literacy level and cognitive ability of adolescents, there is a potential
to use slightly longer scales which measure more complex psychological constructs.
ChampsCamps.For the Champs Camp program, despite a relatively large sample sizes for Kids (aged 7‐11
years), there were no significant changes on the Coping, Connections, or Problems scales from pre‐
to post‐intervention for either Kids (aged 7‐11) or adolescents (aged 12‐17). It is likely that this is due
to the limitations of the evaluation methodology. In particular, as identified by key personnel
involved in the program, there is limited or no opportunity for many of these measures to change
over the 2‐3 day time period of the camps. For example, as children have no contact with their
family during this time, it is not possible for within‐family connections or problems to change. In
addition, the Connections and Problems scales asks children to reflect on the ‘last few weeks’ and
the Coping scale responses are dependent on the child’s ability to put any new skills learnt during
the program into practice. Hence it is unrealistic to expect these measures to change over a 2‐3 day
period. Although there were no statistically significant changes on these outcome measures it should
be noted that Anglicare have conducted qualitative research which indicates that these Camps have
made a positive difference to children and their families. Such evidence can be provided by Anglicare
on request.
In a previous evaluation of a similar COPMI Camp program (Goodyear, et al., 2009), changes
were found on the Coping, Connections and Problems sub‐scales. However, this camp program was
4 days in length, delivered the same content as the after school program, and the post‐intervention
was completed four weeks after the camp. In contrast, the Taz Kids Champs Camp does not
necessarily cover the same content as the Taz Kids Clubs and the post‐intervention questionnaire is
completed on the last day of the camp. In light of these limitations, in future evaluations of the
Champs Camp program, it would be of benefit for post‐intervention assessments to be completed
four weeks following the camp. Alternatively, outcome measures could be adopted which correlate
with resilience but are modifiable within the timeframe of the program (e.g., self‐esteem, self‐
efficacy, emotional well‐being, optimism, positive affect).
27
DataLimitationsThe data available for analysis was a small subset (41%) of the total potential data between
2011 and 2012. As such the generalisabilty of the findings to all participants is limited, and the
results should be interpreted with caution. Where there was either pre‐ or post‐intervention data
missing for a child, this could have been due to children non‐attendance at particular sessions or due
to issues with data collation and coding including human error. In addition, where there was both
pre‐and post‐intervention data available, missing data within each of the sub‐scales limited the
number of data points available per analysis. Missing data was particularly evident for the
Connections and Problems scales (where just over two‐thirds of the data was available), and for
questions 10‐12 which ask children to indicate how many friends they have and how many times
they have played with friends in the last month. For questions 10‐12, it is possible that children do
not feel comfortable answering these questions (an issue that was reiterated in qualitative feedback
from staff) or that they found it difficult to estimate exact numbers in answer to these questions.
Missing data on the Connections and Problems scales could also be due to children not
understanding how to answer if the question does not seem relevant to them (e.g., if they do not
have a brother/sister/grandparent, or they have not seen the particular person within the timeframe
specified). The original scale included instructions on how to answer under these circumstances but
these instructions have been omitted from the current scale.
There are several other factors which could potentially confound the interpretation of
results. Firstly, it is unclear how many children attended multiple Camps and/or Clubs, thus it is
difficult to establish the impact of multiple program attendance on the outcome measures. In their
evaluation of the CHAMPS program, Goodyear et al. (2009) report that that there was no significant
effect of multiple program attendance. However, in their final analysis they only included data for
the first intervention that each child attended (e.g., camp or club). This allowed the authors to
establish true independence of data and therefore use analysis of variance (ANOVA) which allows
the examination of interactions between multiple variables (e.g., whether there were differences in
outcomes for males relative to females). This approach was not possible in the present evaluation as
independence could not be established from the data provided.
Another factor which could potentially confound the present findings are variations in
program delivery which may have occurred over the two year period (e.g., differences in program
delivery between the North/North‐West and South, changes to modules included in the program,
staffing changes etc.).
Another methodological issue is the absence of a control group. The inclusion of a control
group is a common experimental design which allows researchers to confirm that changes on
outcome measures are due to the intervention rather than due to other factors such as the passage
of time or developmental trajectories. While fully controlled investigations are not always practical
in field‐based evaluation research, it may be a consideration if an evaluation was designed for
potential peer‐reviewed publication.
28
Summary/ConclusionsThe present evaluation revealed some issues with missing data which limits the
generalisability of the results. Reasons for missing data included children’s non‐attendance at the
relevant sessions, data collation and coding procedures, as well as the children’s willingness to
respond or ability to understand some of the questions. Sample size was a greater issue for
adolescents (aged 11‐17) relative to children (aged 7‐11) due to the smaller number of adolescents
who completed the programs.
Despite data limitations, kids (aged 7‐11) who completed the Taz Kids Club program showed
significant increases on the Coping scale (particularly social support‐seeking) and the Connections
scale (particularly within the family) from pre‐ to post‐intervention, and an increase in the number of
friends reported. This suggests that the Taz Kids Club program has the potential to produce positive
outcomes that are consistent with the goals of the program for this age group. An unexpected
finding was the increase in problems reported by kids in the program, a result which may be due to
increased sensitisation or awareness of problems within the family as a result of the intervention.
In contrast, there were no changes on outcome measures for adolescents which may have
implications for the suitability of the Taz Kids Club program and the associated outcome measures
for this age group. Maximising the use of the PATS program for the adolescent age group and further
consideration of the suitability of existing outcome measures is recommended.
With regard to the Champs Camp program, there was no evidence for any changes on
outcome measures from pre‐ to post‐intervention. Given the short timeframe of the program (2‐3
days), adopting a longer post‐intervention timeframe or adoption of more appropriate measures
which are consistent with the short‐term goals of the Camp program is recommended. However,
given previous experience of low return rates in relation to follow‐up surveys among this population,
the latter option may be more appropriate. It should also be noted that qualitative data collected by
Anglicare indicates that the Champs Camp program has made a positive difference to children and
families who have participated.
Further consideration of evaluation outcome measures and associated recommendations
can be found in the following literature review and in the executive summary of this report.
29
Part2–AreviewofOutcomeMeasuresrelevantfortheEvaluationofCOPMIPrograms
QualitativefeedbackonexistingTazKidsCluboutcomemeasuresQualitative interviews were conducted with key personnel involved in the development and delivery
of the Taz kids clubs and Champs camps programs. Based on this, the following feedback regarding
the existing outcome measures and evaluation protocol was compiled:
Evaluation time frames
o A major concern was the time frame between the pre‐and post‐intervention
questionnaires for the Champs Camps. Currently children complete the pre‐
intervention questionnaire on Day 1 of the camp and the post‐intervention
questionnaire on Day 3 of the camp. Given that some questions are about
connections/problems within the family (or connections in which the child is asked
about going to a friend’s place to play), it is unrealistic to expect that these
measures could change within this three day time‐frame when they are away from
home. It was also suggested that the response rate would be very low if the post‐
intervention questionnaires were sent out for completion at a later date.
Coping sub‐scale (questions 1‐6)
o There were no specific comments in relation to this sub‐scale
Connections sub‐scale (questions 7‐16) & Problems sub‐scale (questions 17‐23)
o There was concern that some children did not like to answer questions in relation to
connections with friends (questions 7‐9) and that often these questions were left
blank on evaluation forms. It was perceived that these questions can be
distressing/intrusive for those children who do not have any friends or who are not
able to have friends to over to play etc.
o It was also suggested that some children tend to exaggerate when asked how many
friends they have (e.g., 1000) which may affect the reliability of the data.
o In relation to questions around ‘time spent with family’, it was suggested that these
questions may not be appropriate for children who are currently not in their
parents’ care. Provision for answering in relation to carers/guardians was put
forward as a possible suggestion. A related issue was that questions around
connections with brothers/sisters are not relevant to those who do not have
siblings.
o The term ‘play’ was thought to be unacceptable for children of adolescent age.
Perhaps ‘hanging out’ or ‘chilling’ would be more appropriate terms for this age
group.
General comments
o Some children tend to answer ‘sometimes’ to every question rather than thinking
carefully about their response (i.e. response bias)
o The length of the questionnaire was considered to be an issue and it was suggested
that the children generally do not enjoy filling it out.
o It was indicated that missing data can occur due to children arriving early/leaving
late or due to starting the program at a later date than other children.
30
Literaturereview:COPMIevaluationsandoutcomemeasures
IntroductionAccording to population estimates, between 21‐23% of children in Australia have at least
one parent with a mental illness (excluding substance misuse‐related mental illness) (Maybery,
Reupert, Patrick, Goodyear, & Crase, 2009b). Children of parents with a mental illness (COPMI) have
an elevated risk of developing mental health problems themselves (Hosman, Van Doesum, & Van
Santvoort, 2009). The associated risk factors include parent’s difficulty in fulfilling a parenting role
due to illness/treatments, family conflict, social isolation, emotional distance, low self‐esteem and
low participation in education and social life (Falkov, 2004; Handley, Farrell, Josephs, Hanke, &
Hazelton, 2001; Hinshaw, 2005; Hosman, et al., 2009; Maybery, Ling, Szakacs, & Reupert, 2005;
Pakenham, Bursnall, Chiu, Cannon, & Okochi, 2006; Tebes, Kaufman, Adnopoz, & Racusin, 2001).
A resilience framework has guided the development of programs aimed at supporting
COPMI (Australian Infant, Child, Adolescent and Family Mental Health Association 2001). These
programs aim to mitigate the risks associated with living with parental mental illness by enhancing
protective factors, such as resilience, understanding of mental illness, connectedness, coping skills
and self‐esteem (Reupert & Maybery, 2009). Many programs use a combination of standardised and
non‐standardised evaluation measures (i.e. designed specifically for the program) to assess whether
the program is achieving the program aims.
The following review will outline the risk and protective factors which are most relevant for
children of parents with a mental illness. This will be followed by a review of evaluation research in
this area with a focus on the evaluation methodology and outcome measures that are utilised.
Finally, a review of relevant outcome measures will be undertaken.
RiskFactorsCOPMI are faced with a range of issues not usually faced by other children their age because
they live with the symptoms, behaviours, and expressions of their parent’s mental illness (Falkov,
2004). Parental mental illness can impact on the entire family and increase family conflict or affect
familial relationships. For instance, Tebes et al. (2001) found that parental mental illness increases a
child’s exposure to family distress and conflict, thereby elevating the child’s risk of developing
behavioural and emotional problems. For example, when compared to a normative sample, COPMI
show significantly more externalising and internalising behavioural problems on the Child Behaviour
Checklist (CBCL) (Tebes, et al., 2001). Children may exhibit symptoms of their parents mental illness
due to behavioural modelling observed within the home (e.g., externalising behaviour,
overprotective behaviour, substance use) (Hosman, et al., 2009), or adopt maladaptive coping
strategies (e.g., withdrawing, avoiding and distancing) that may put them at risk for later
maladjustment (Maybery, et al., 2005).
Another source of distress faced by COPMI is separation from their parent, both physically
and/or emotionally. Children may be forced to take on additional caring responsibilities for their
parent and/or siblings (Hargreaves, et al., 2005). The care burden on COPMI (especially in single
parent situations) may greatly affect their participation in education and social life (Pakenham, Chiu,
Bursnall, & Cannon, 2007). Additionally, parent‐child interactions may be comprised which
contribute to attachment bonds, security and emotional nurturance essential for the child’s
emotional development and appropriate socialisation (Maybery, et al., 2005).
Children’s lack of knowledge about mental illness and the stigmatisation of mental illness is
also a risk factor. Children may experience difficulty disclosing to others that they have a parent with
a mental illness and avoid the topic because of confusion or embarrassment (Hinshaw, 2005). Stigma
surrounding mental illness may create a sense of isolation for children (Hinshaw, 2005) and they may
31
experience low self‐esteem which prevents them from forming close bonds with their peers and
puts them at a greater risk of experiencing interpersonal problems (Lee, Draper, & Lee, 2001).
Additionally, children’s lack of knowledge about mental illness may cause them to blame themselves
for their parent’s illness and present difficulties for reducing the effects of stigma of mental illness
from their peers (Handley, et al., 2001).
All of the above factors may contribute to an elevated risk of these children developing
mental health problems. The risk of developing mental disorder for children whose parent has a
mental illness ranges from 41%‐77% (Hosman, et al., 2009). Maybery, et al. (2006a) administered the
Strengths and Difficulties Questionnaire (SDQ) to parents of children with and without a mental
illness. The SDQ is a measure of psychological attributes in children commonly used to screen for
psychiatric illness. The results indicated that the children of parents without a mental illness had
approximately 17% of children in the combined borderline and clinical ranges, compared to 57% of
COPMI. This means that COPMI were almost 3.5 times more likely to experience symptoms of a
mental illness than their peers in the community.
While there are a number of risk factors, adverse conditions and stressors that COPMI must
cope with, not all children will experience difficulties or negative outcomes as a result of their
parent’s health status. Some children do not experience any more or worse negative outcomes than
children from the general population (Harvey & Delfabbro, 2004). The level of risk varies depending
on a range of environmental and individual factors, including genetic inheritance, the age of the
child, the nature of the mental illness, family relationships, and the involvement of other adults in
the child’s life (Hosman, et al., 2009; Maybery, Reupert, Patrick, Goodyear, & Crase, 2006b). One
important factor which mediates the level of risk experienced by children is resilience.
Resilience&ProtectiveFactorsResilience is the ability to bounce back from adverse events without negative outcomes
(Rutter, 1999). Closely related to the concept of resilience are a number of protective factors which
have a moderating effect on risks and buffer children against negative outcomes. Resilience in
COPMI is developed through enhancing protective factors, such as, access to information and
education about mental illness, improving problem solving and coping skills, identifying feelings and
regulating emotions, developing and maintaining strong sibling bonds and friendships, being
involved in community or school activities, and increasing self‐esteem (Riebschleger, Tableman,
Rudder, Onaga, & Whalen, 2009). Protective factors serve as targets in peer support programs for
COPMI. The most commonly targeted factors include self‐esteem, knowledge of mental illness,
coping skills and social connectedness (Reupert & Maybery, 2009).
Self‐esteemSelf‐esteem is the extent to which a person believes themself to be competent, successful or
worthy (Butler & Gasson, 2005). Low self‐esteem in COPMI has the potential to be a risk factor,
however high self‐esteem is a protective factor. High self‐esteem appears to have a ‘ripple’ effect on
many aspects of children’s well‐being. For instance, having positive views and beliefs about one’s
self increases the likelihood that children will try new things, make new friends, and participate in
activities (Rutter, 1999). Rutter (1999) suggests that high self‐esteem underlies the operation of all
protective factors as it buffers against distress, reduces negative chain reactions, and permits
positive chain reactions. One important aspect of enhancing self‐esteem is to increase children’s
knowledge and understanding of mental illness.
32
KnowledgeofmentalillnessAn accurate knowledge about mental illness is a significant distinguishing feature of resilient
children (Handley, et al., 2001). Knowledge of mental illness includes information about the causes,
symptoms and treatments of mental illness in a language that is appropriate for the developmental
age of the child (Falkov, 2004). Such knowledge allows the children to understand their parents,
empowers them to reduce stigmatisation, and fosters better communication and relationships
within their family (Hargreaves, et al., 2005). Enhanced knowledge of mental illness may result in
reduced anxiety, confusion, and feelings of guilt blame and isolation in children (Reupert & Maybery,
2010).
Socialconnectedness Lee, et al. (2001) use the term connectedness to refer to an individual’s various caring and
supportive relationships. Positive connections with others are a significant moderator on the effects
parental mental illness. COPMI have reported that positive connections and having someone to talk
to are an important coping resource to them (Fudge & Mason, 2004), and positive connections have
also been related to better adjustment outcomes (e.g., life satisfaction, positive affect, benefit
finding) (Pakenham, et al., 2007). A secure and stable attachment bond with at least one parent is an
important coping mechanism for young people (Fudge & Mason, 2004), but meaningful social and
emotional connections may also be made from within (e.g., siblings) and outside (e.g., peers,
teachers) the family (Maybery, et al., 2006a).
CopingskillsCoping can be defined as constantly changing cognitive and behavioural efforts to manage
specific external and/or internal demands that are appraised as taxing or exceeding the resources of
the person (Lazarus & Folkman, 1984). The use of effective coping mechanisms is an important
protective factor that contributes to resilience. Effective coping skills are associated with better
adjustment outcomes, such as low internalising and externalising problems (Connor‐Smith, et al.,
2000), lower distress, higher satisfaction with life and positive affect and may also contribute to the
development of other protective factors (Pakenham, et al., 2007). The effectiveness of a coping
strategy depends on its ability to manage the stressor successfully and a coping strategy should
change depending on the situation. Effective coping mechanisms are particularly important for
children living with a mentally ill parent as they must cope with the symptoms, behaviours and
expressions of their parents’ mental illness (Falkov, 2004).
ReviewofpreviousCOPMIprogramevaluationsStrengths‐based peer support programs for COPMI have been established in the United
States (Orel, Groves, & Shannon, 2003; Riebschleger, et al., 2009), United Kingdom (Grant, Repper, &
Nolan, 2008), Canada (Pitman & Matthey, 2004; Richter, 2006), the Netherlands (Van Doesum &
Hosman, 2009), and Australia (Fraser & Pakenham, 2008; Goodyear, et al., 2009; Hargreaves, Bond,
O'Brien, Forer, & Davies, 2008; Hargreaves, et al., 2005; Hayman, 2009; Morson, Best, de Bondt,
Jessop, & Meddick, 2009; Pitman & Matthey, 2004). While all of these programs have the
overarching aim of improving resilience in children, the program aims and outcome measures vary.
Fraser, James, Anderson, Lloyd, and Judd (2006) highlight the importance of identifying the linkages
between program aims, program content, and outcome measures to effectively evaluate programs.
Some of the programs for COPMI, and their evaluations will be reviewed below, with a particular
focus on the outcome measures used and the major findings.
33
PositiveConnections,USAThe Positive Connections program run in Toledo, Ohio is for children aged 8‐13 who have a
parent with a mental illness (Orel, et al., 2003). The primary focus of the program is to enhance
children’s ability to understand and cope with their parent’s mental illness by providing education,
support and mentoring. Children attended 5 weekly sessions of psycho‐education followed by 5
weekly peer support sessions and a minimum of 6 months one‐to‐one mentoring with a caring adult.
At pre‐ and post‐intervention, children (n=11) were given the Self‐esteem Index (SEI) (Brown &
Alexander, 1991) to measure changes in the perception of self, and the Family Assessment Measure
(FAM) (Skinner, Steinhauer, & Santa‐Barbara, 1995) to measure perceived changes in family
functioning by the children and their parents (including task accomplishment, role performance,
communication, affective expression, involvement, control and values and norms). At post‐
intervention participants showed improvements in all areas of the SEI (total, familial acceptance,
academic competence, peer popularity, personal security) and several improvements in the FAM
(parent rated improved role performance, children rated improved communication, both rated
improved affective expression, control and values and norms). These results suggest that overall the
program was effective in improving the quality of function in the children’s lives.
YES,USAA Youth Education and Support (YES) pilot program was run in the USA for children aged 11‐
16 (Riebschleger, et al., 2009). YES involves 6 two‐hour activity focused psycho‐education sessions
with each session focusing on a specific theme. The aim of YES is to strengthen protective factors
including increased access to information about mental illness and increased coping skills. Children
completed the 27‐item Knowledge of Psychiatric Illness and Recovery Test (KPIRT) (developed for
the program) and the 54‐item Adolescent Coping Orientation to Problem Experiences (A‐COPE)
(Patterson & McCubbin, 1987). At post intervention the children (n=17) showed significant
improvements (within the 90% confidence interval) in all sub‐scales of the KPIRT (prevalence of
mental illness, causes, functioning of people mental illness, stigma and rehabilitation) and
improvements in two of the thirteen coping sub‐scales, namely, avoiding problems and relaxing.
KAP,AustraliaA program run in Australia called the Koping Adolescent Group Program (KAP), is for 12‐18
year olds whose parent has a mental illness and involves three 6‐hour group sessions (Fraser &
Pakenham, 2008). The aim of KAP is to improve adjustment outcomes for children by modifying risk
factors such as social isolation and inadequate mental health literacy, and strengthen protective
factors, such as coping skills and intact peer relationships. Fraser and Pakenham (2008) recently
evaluated the effectiveness of the KAP program by examining changes in a number of outcome
variables are pre‐, immediately post‐, and 8 weeks post‐intervention in both a treatment (n=27) and
control (n=17) group.
The groups were compared on three groups of outcome variables. One group of variables
were the intervention targets and included measures of mental health literacy (developed for the
program), connectedness as measured by the 20‐item Social Connectedness Scale (Lee, et al., 2001)
and coping strategies as measured by the 57‐item Responses to Stress Questionnaire‐Family Stress
Version (Connor‐Smith, et al., 2000). Another group of variables reflected adjustment outcomes and
included measures of depressive symptomology as measured by the 10‐item Children’s Depression
Inventory‐Short Form (Kovacs, 1992), life satisfaction as measured by the 5‐item Satisfaction with
Life Scale (Diener, Emmons, Larsen, & Griffin, 1985) and strengths and difficulties as measured by
the 25‐item Strengths and Difficulties Questionnaire (Goodman, Meltzer, & Bailey, 1998). The last
34
variable assessed was caregiving experiences as measured by the Young Caregiver and Parents
Inventory (YCOPI) (Pakenham, et al., 2006).
While there were no statistically significant differences between the treatment and control groups, the treatment group did show significant improvements in mental health literacy, depression and life satisfaction immediately post‐intervention which were maintained at 8 weeks post‐treatment. On the responses to Stress scale there was a marginally significant effect of time for total scores, with significant decreases found for disengagement and involuntary disengagement coping from pre‐treatment to follow up and from post‐treatment assessment to follow up, and use of involuntary engagement coping decreased from post‐treatment to follow up. While this broad range of outcome measures allows for a thorough evaluation of the program’s effectiveness, the time it takes to accurately complete these questionnaires must also be taken into consideration.
SMILES,CanadaandAustraliaThe SMILES program has been run in Canada and Australia for children aged between 8‐18
years and is conducted over 3 consecutive days for six hours each day during school holidays (Pitman
& Matthey, 2004). It provides psycho‐education about mental illness and life skills training.
Outcomes of three SMILES programs (two in Australia and one in Canada), with 25 children in total
aged 5‐15 years (a 5 year old sibling was included) were measured using two questionnaires
developed for the program (knowledge of mental illness measure and a life skills measure)
completed at day 1 and day 3. Participants rated their confidence in relation to a knowledge
question (Knowledge of Mental Illness Measure), and their ability to use various life skills considered
beneficial for coping (Life Skills Measure). Self‐report data showed improvements in 8 of the 9
knowledge of mental illness items and all 10 self‐rated life skills items. While the outcome measures
do not reflect the children’s ability to actually use and practice their knowledge and life skills learnt
during the program, given the program length the children’s perceived confidence/ability was used
as a proxy measure.
KidsInControl,CanadaThe Kids In Control (KIC) program is an 8‐week psycho‐education group for children aged 8‐
13 years which is run in British Columbia, Canada (Richter, 2006 unpublished Masters thesis). The
program focuses on outcomes associated with enhancing resilience including, self‐esteem, coping
strategies and knowledge of mental illness. Scores on the school form version (for ages 8‐15yrs), of
the 58‐item Coopersmith Self‐Esteem Inventory (CSEI) (Coopersmith, 1981), the 9‐item Kids Coping
Scale (KCS) (Maybery, et al., 2009a), and the 20‐item Kids Knowledge Scale (KKS) (developed for the
program) were compared at pre‐intervention, post‐intervention and 8 weeks post‐intervention.
Participants general self‐esteem, and knowledge of mental illness improved significantly from pre‐ to
post‐intervention but these were not maintained at 8 weeks post‐intervention. There was a
significant decline in total coping and emotion‐focussed coping over the course of the program, but
no significant changes in problem‐focussed or social support focussed coping. At 8 weeks post‐
intervention there were no significant changes in participant’s use of coping strategies. Richter
(2006) suggested that the decline in the use of coping strategies could be positive or negative
depending on how adaptive the coping strategy is. For example, it can be argued that the coping
strategies assessed by the emotion focused sub‐scale may not necessarily be adaptive due to the
focus on avoidance strategies which may be an ineffective coping mechanism in the long‐term
(Connor‐Smith, et al., 2000).
Additionally the scores of program participants (n=16) were compared to a control group of
future program participants (n=17). Program participants did not differ significantly in terms of
coping strategies, however there were trends toward higher self‐esteem and knowledge of mental
illness among program participants compared to future participants. Richter (2006) noted that the
35
small sample size limited the ability to utilise inferential analysis in evaluation program outcomes
and recommended that a qualitative or mixed methods study may be beneficial to establish the
effectiveness of the program.
PATS,AustraliaAnother program run in Australia called Paying Attention to Self (PATS) was developed in
1997 as a response to an identified lack of support and resources for adolescents (aged 12‐18 year)
whose parents have a mental illness (Hargreaves, et al., 2005). PATS aims to provide young people
with the opportunity to share their experiences and be supported by others in similar situations. The
activities in PATS aim to increase knowledge of mental illness, improve help‐seeking and coping
behaviours and improve their sense of connection to peers, family and community. Each PATS group
comprises 6‐8 adolescents who meet weekly for 8 weeks. Activities include discussions, games, role‐
plays, art therapy, guest speakers and social outings with the overall focus being on healthy thinking
and coping strategies. The topics addressed in PATS include: Understanding their parents illness,
improving their relationships with their parent, communication and problem solving skills, dealing
with the stigma of mental illness, and developing strategies to cope with their parents condition and
to stay mentally healthy themselves.
Hargreaves et al. (2005) report on an evaluation of the PATS program. Participants (n=64)
completed a range of measures, including those addressing the specific program aims and other
additional measures at pre‐, post‐, 6 months post‐ and 12 months post‐intervention. The specific
program aims were assessed by measures of mental health literacy (developed for the program),
stigma (developed for the program), burden of caregiving (Montgomery Borgatta Caregiver Burden
Scale; Montgomery, Borgatta, & Borgatta, 2000), perceived social support (Multidimensional Scale
of Perceived Social Support; Zimet, Dahlem, Zimet, & Farley, 1988) and social problem solving skills
(Social Problem Solving Inventory Revised Short and Modified Self Report Coping Scale; Chang &
D’Zurilla, 1996). Participants also completed measures assessing depressive symptoms (Short Mood
and Feelings Questionnaire; Angold, 1995), positive emotional wellbeing (sub‐scale of a
psychological distress and well‐being measure Viet & Ware, 1983), risk of homelessness
(Chamberlain & MacKenzie, 1998) and substance use (developed for the program).
At post‐intervention, there was a significant reduction in depressive symptoms, risk of
homelessness and experience of stigma which was maintained at 12 months post‐intervention.
Participants reported significantly higher emotional wellbeing post‐intervention; however, this was
not maintained at 12 months post‐intervention. Participants were also asked to give feedback on the
program and comment on what they perceived were the best outcomes from participating in PATS.
Some of the responses included that PATS allowed them to learn about and understand their
parent’s mental illness, reduce stigma, increase confidence in seeking help, develop coping
strategies, and accept and deal with feelings.
Similarly to the KAP program, the broad range of outcome measures must be balanced with
consideration for the time it takes to complete these measures. This is especially true in the PATS
program where a large portion of the first and final sessions is taken up filling out the
questionnaires. Furthermore, the shorter amount of time it takes to complete the questionnaires,
the more likely it is that participants will return accurate questionnaires at follow up periods. Some
measures used in this evaluation (e.g., burden of caring, risk of homelessness, depressive symptoms,
positive emotional wellbeing and substance use) have some link to assessing the overall program
aim of enhancing participants well‐being but they do not address the specific aims. The measures of
mental health literacy, stigma, social support and social problem solving skills have clear links to the
program aims and content.
36
CHAMPS,AustraliaThe Children and Mentally ill ParentS (CHAMPS) program is for children aged 8‐12 years and
is offered as an after school program (2 hour sessions weekly over one school term or fortnightly
over two school terms) or a school holiday 2‐3 day camp program. The outcome measures used in
the CHAMPS program are similar to those used in the current evaluation of the Taz Kids Club and
Champs Camps programs. Goodyear, et al. (2009) recently evaluated a pilot CHAMPS program
conducted in Victoria, Australai. The wellbeing of participants (n=69) was evaluated at the beginning
of the program and four weeks after program completion on outcome measures including a
modified version of the Rosenberg‐Simmons Self‐Esteem Scale for adolescents (RSSES) (Rosenberg,
1979), Kids Coping Scale (Maybery, et al., 2009a) and Kids Connections and Kids Problems Scale
(Maybery, et al., 2006a). The Kids Connections/Problems scale asks respondents questions regarding
their key positive relationships (Connections) and key negative relationships (Problems). The 11
items can form a total connections/problems score, or be broken down into a sub‐scale for ‘within
family’ (i.e., mum, dad, brother, sister, grandparent) and ‘outside family’ (i.e., teacher, friend)
connections/problems. At post intervention participants attending the school holiday and after
school program showed significant improvements in self‐esteem, seeking social support coping,
connections within the family, and a reduction in total, within family and outside family problems.
Participants in the school holiday program also showed significant improvements in problem focused
coping compared to those in the after school program.
ReviewofOutcomeMeasuresrelevanttoCOPMIInterventionsThe following section provides a review of measures available to measure constructs that are
relevant to the evaluation of COPMI interventions. Further information on specific scales can be
found in Table 6 and examples of scales can be found in appendices.
CopingStrategiesCoping can be defined as constantly changing cognitive and behavioural efforts to manage
specific external and/or internal demands that are appraised as taxing or exceeding the resources of
the person (Lazarus & Folkman, 1984). While there are many different ways to classify how children
cope, Maybery, Steer, Reupert & Goodyear (2009) suggest that problem focused, emotion focused
and seeking social support coping represent common coping mechanisms for children. Problem
focused coping refers to efforts directed toward the stressor to change the situation and may
involve appraising a situation and taking appropriate action by using cognitive or practical problem
solving skills (e.g., brainstorming, planning or goal setting). Emotion focused coping refers to
attempts to alter emotional reactions to the stressful situation and involves recognising and altering
an affective reaction (e.g., avoiding problems or issues). Social support seeking coping is considered
a form of coping as help seeking and thinking about what others might do in the circumstances can
also help to reduce the stressor (Fudge & Mason, 2004; Garber & Little, 1999).The Kids coping scale
was constructed to measure these three forms of coping (Maybery, et al., 2009a) (see Appendix A).
The brevity of the KCS is a particular strength especially in relation to assessing younger
children. However, the benefits of having a short scale must be balanced with its ability to validly
measure the construct of interest. Among a large sample (n=834) of children aged 7‐13, principal
component analysis of the items did not support the hypothesised three factor structure, but a two
factor structure for the emotional and problem focussed coping measures was supported (Maybery,
et al., 2009a). Thus social support seeking did not form a factor independently of the problem and
emotion focussed items. Although the SSS scale did not form an independent factor, this does not
mean that this is an irrelevant coping mechanism for children, particularly given the focus on
encouraging help seeking in the Taz Kids Club program and the positive results of the present
37
evaluation. The KCS was shown to have low to moderate levels of internal consistency for the PFC
subscale (r=0.58) and the EFC subscale (r=0.30) (Maybery, et al., 2009a). The low internal
consistency of the PFC and EFC sub‐scales is a concern and among other reasons may be due to the
small number of items per sub‐scale (Maybery, et al., 2009a). The PFC scale demonstrated
reasonable construct validity with significant positive correlations found with measures of self‐
esteem and connections, and negative correlations found with adjustment problems. The construct
validity of the EFC was less clear with low negative correlations observed with self‐esteem and
connections. The KCS was developed for children aged between 8‐12 years and the brevity of the
scale makes it a particularly useful for younger children. However, given the below average
psychometrics of this scale it may not be so appropriate for adolescents who have the reading level
and cognitive ability to complete slightly longer and more psychometrically valid scales. Also, coping
styles may progress as children age with younger children using narrower range of coping strategies
than adolescents (Donaldson, Prinstein, Danovsky, & Spirito, 2000).
The Kids In Control program (Richter, 2006), the CHAMPS program (Maybery, et al., 2009a)
and the Taz Kids Club programs have all used the KCS to assess changes in coping strategies over
time as a result of COPMI interventions. The results of the Kids In Control evaluation (Richter, 2006)
showed a decline in the use of Total coping scores and EFC coping scores over the course of the
program, while the CHAMPS program (Maybery, et al., 2009a) and Kids (aged 7‐11) in the present
evaluation of the Taz Kids Club program showed improvements in SSS coping, with no changes on
the PFC and EFC scales. It has been suggested that the coping strategies assessed by the EFC sub‐
scale are not necessarily adaptive ways of coping for children due to the focus on avoidance
(Connor‐Smith, et al., 2000; Dumon & Provost, 1999). In this sense, it may actually be beneficial to
observe a decrease or no change in this sub‐scale as it may reflect a shift toward other approaches
for handling emotions. Other research suggests a distinction between avoidance and distraction as
coping mechanisms (Ayers, Sandler, West, & Roosa, 1996; Connor‐Smith, et al., 2000), but the EFC
scale includes items relating to these strategies within the same scale. Another criticism of the KCS is
that the problem vs. emotion focussed distinction is too simplistic (Connor‐Smith, et al., 2000), and a
four factor model of coping (active, distraction, avoidant and support seeking) is thought to better
capture the range of coping strategies used by children (Ayers, et al., 1996).
Alternative measures that may be useful for the assessment of coping strategies in
adolescents are discussed below and can be found in Appendix B. The Adolescent Coping Scale‐Short
form consists of 19 items that measure adolescents coping strategies (Frydenberg & Lewis, 1996).
This scale is appropriate for adolescents aged 12‐18 years and has three subscales – productive
coping, non‐productive coping and reference to others – which encompass 18 different coping
strategies. However, the scale is only available for purchase (https://shop.acer.edu.au/acer‐
shop/group/HU2) and must be administered to professionals with accredited training in psychology,
health sciences, counselling, special education, medicine and other specialist areas.
The Children’s Coping Strategies Checklist (Ayers, Sandler, West, & Roosa, 1990; Ayers, et al.,
1996; Sandler, Tein, & West, 1994) is a self‐report inventory in which children aged between 9‐15
years can describe their coping efforts. It consists of 44‐items which assess 11 dimensions of coping ‐
cognitive decision making, direct problem solving, seeking understanding, positive cognitive
restructuring, expressing feelings, physical release of emotions, distracting actions, avoidant actions,
cognitive avoidance, problem focused support, emotion focused support which can be collapsed into
four sub‐scales of active coping, avoidance, distraction and support (Sandler, et al., 1994).
The Kidcope scale (Spirito, Stark, & Williams, 1988) has two versions for younger (5‐12 years)
and older (13‐16) children which assess 10 cognitive and behavioural coping strategies. Over a brief
3‐7 day period test‐retest reliabilities are moderate to high (r=.41‐.83), but are unacceptably low
38
over a 10 week period (r=.15‐.43). The scale has demonstrated concurrent validity (.33‐.77) with
conceptually similar scales, such as the Coping Strategies Inventory (Tobin, Holroyd, & Reynolds,
1984). One of the criticisms of the Kidcope scale is that its internal reliability and factor structure is
difficult to establish because single items are used to represent some constructs (Maybery, et al.,
2009a) and another criticism is that the scale is not based on a theoretical model (Connor‐Smith, et
al., 2000). Thus the brevity of the Kidcope may be considered as a both a strength and weakness of
the scale. Research using this scale to examine the characteristics of coping strategies of children
(Donaldson, et al., 2000) has shown that children show a similar pattern of coping responses in
response to different stressors (school, family, sibling, peers). Wishful thinking, problem solving, and
emotional regulation were the most common strategies used by children, but children reported
using a wider range of coping strategies in later adolescence (15‐18 years) relative to early (9‐11
years), middle (12‐14 years) adolescence.
The 57‐item Responses to Stress Questionnaire (RSQ) (Connor‐Smith, et al., 2000) is based
on a multidimensional model of responses to stress (Compas, Connor‐Smith, Saltzman, Thomsen, &
Wadsworth, 2001) and includes volitional and involuntary responses to specific domains of stress
and includes factors of primary control engagement coping (problem solving, emotional regulation,
emotional expression), secondary control engagement coping (acceptance, distraction, positive
thinking, cognitive restructuring), disengagement coping (avoidance, denial, wishful thinking)
involuntary engagement (rumination, intrusive thoughts, physiological arousal, emotional arousal,
impulsive action), and involuntary disengagement ( emotional numbing, cognitive interference,
inaction and escape). The factor structures identified for this scale did not support the proposed
distinction between problem and emotion focussed coping and thus the authors argued that this
dichotomy is too simplistic. In a previous evaluation of KAP COPMI program in Australia (Fraser &
Pakenham, 2008), there was evidence for decreases in disengagement, involuntary disengagement,
and involuntary engagement coping following the intervention. The strengths of this questionnaire
are that it is based on a theoretical model, considers both voluntary and involuntary responses to
stress, distinguishes between distraction and avoidance coping (unlike the Kids coping scale where
these are combined in the emotion focused sub‐scale) and has adequate psychometric properties;
however, the length of the questionnaire is a weakness in terms of providing a brief measure of
coping skills.
Considering the pre‐ and post‐evaluations of the Taz Kids Club Camp program are conducted
over a short period of time, it may be beneficial to adopt an alternate measure of coping for the
Champs Camp. In an evaluation of the 3‐day SMILES program (Pitman & Matthey, 2004), a life skills
measure (see Appendix B) was used in which participants completed 10‐items about their perceived
ability to use various life skills considered beneficial for coping. Asking participants of their perceived
ability to implement skills still requires them to reflect on what they have learnt over the course of
the program, however, does not require a reflective period where they are able to practice and
implement the skills learnt during the program. Interestingly, children were not blind to their ratings
from Day 1 when they completed ratings at Day 3. This may represent a methodological issue, and it
may be better for children to post intervention ratings without looking at pre‐intervention ratings.
ConnectionsandProblemsThe Kids Connections and Kids Problems Scales used in the evaluation of the Taz Kids Club
program is a non‐standardised tool developed for children aged between 8‐12 years (Maybery, et
al., 2006a) (See Appendix A). The CHAMPS program (Maybery, et al., 2006a) and the Taz Kids Club
programs (present evaluation) have both used this measure to assess the key ‘within family’ and
‘outside family’ connections and relationships problems. However, there were a number of
variations made to the scale in the Taz Kids Club evaluation which may affect its reliability/validity. In
39
the original scale there were 10‐items regarding kids connections and 11‐items regarding kids
problems (Maybery, Reupert & Goodyear, 2005 cited in Maybery, Steer et al., 2009) while 7‐items
per scale were used in the Taz Kids Club programs. For the CHAMPS program responses were on a 4‐
point Likert scale (of increasingly bigger smiley faces for the connections scales, and increasingly
bigger frowning faces for the problems scale). In contrast, for the Taz Kids program, responses were
on 3‐point Likert scale with written responses ranging from ‘Never’ to ‘A lot’. Both programs also
included a set of 3 questions regarding friendships which required a written response but which did
not contribute to the connections score (Maybery, et al., 2006a), however, the evaluation of the
CHAMPS program does not report the results of these items (Goodyear, et al., 2009). The scales can
be combined into sub‐scales corresponding to ‘within‐family’ and ‘outside‐family’ connections.
However, given the questions omitted form the Taz Kids Club evaluation, fewer items are used to
calculate outside family connections/problems scales. One limitation of the Connections scale is that
it may fail to capture the full range and scope of positive social relationships. While good times spent
with others is one indication of positive social relationships, other indicators such as, trust in others,
ability to talk openly to others, feeling close to others, and being able to depend on others also
indicate positive social connections.
As the Kids Connections and Kids Problems Scales were designed for children aged between
8‐12 years it may be beneficial to adopt a different scale to measure connectedness among
adolescent participants. The Multidimensional Scale of Perceived Social Support (MSPSS) (Zimet, et
al., 1988) may be an appropriate measure (see Appendix C). The MSPSS is a brief 12‐item measure of
social support that includes three subscales measuring social support from family, friends and
significant others as well as yielding a total score. This measure was used in an evaluation of the
PATS program and small increases were found on the friends social support sub‐scale but not the
family or significant others sub‐scales (Hargreaves, et al., 2005). The Child and Adolescent Social
Support Scale (CASSS) (Malecki & Elliott, 1999) is a 40‐item scale which examines children’s
perceived social support from four sources: parent, teacher, classmates, friends and school (see
Appendix C). The CASSS has two versions for children aged between 8‐12 and 13‐18 years. Fraser
and Pakenham (2008) used the 20 Social Connectedness scale (Lee, et al., 2001) in an evaluation of
the KAP COPMI program but found no significant changes over time.
MentalhealthliteracyA measure of mental health literacy clearly links to the Taz Kids club program aims and
content and would be a good addition to the evaluation outcome measures (see Appendix D for
examples of mental health literacy measures). One of the aims of the program is to increase
children’s knowledge and understanding of mental illness. This is achieved throughout the sessions
as participants learn about the myths and facts of mental illness, and the symptoms, treatments and
stigma of mental illness. The COPMI programs described above developed measures of mental
health literacy specifically for the program. The SMILES (Pitman & Matthey, 2004) and Kids In
Control (Richter, 2006) measures of mental health literacy ask respondents general questions about
mental illness (e.g., what is it, what are its causes, as well as specific questions about common types
of mental illness such as depression, schizophrenia and bi polar disorder). The age of the respondent
may also determine the type of measure used. Children may benefit from multiple choice or
true/false response options, whereas adolescents may be asked more in‐depth questions about
mental illness and a variety of response formats, such as open‐ended questions or answers requiring
specific examples could be used. The KAP program for adolescents developed a measure of mental
health literacy that asks respondents questions about their knowledge of mental illness and
awareness of their parent’s mental illness, whereas the PATS program constructed questions
regarding participant’s ability to access mental health services for themselves and their parents (see
40
Appendix D). Mental health literacy should be assessed with an instrument that reflects the
emphasised elements of the intervention. As such it may be possible to develop a tailored
questionnaire for the Taz Kids program.
Self‐esteem Rutter (1999) suggests that high self‐esteem underlies the operation of all protective factors
as it buffers against distress, reduces negative chain reactions and permits positive chain reactions.
There are a number of short scales with good psychometric properties that are designed for use with
children (see Appendix E for examples).
The Rosenberg Self‐Esteem Scale (SES) (Rosenberg, 1979) is a 10‐item scale which measures
an individual’s global self‐worth by asking respondents how much they agree with positive and
negative statements about themselves. The SEI is also available in short form consisting of 6 items
for children less than 11 years of age.
The Coopersmith Self‐esteem Inventory‐ Short form (Coopersmith, 1981) is a 25 item scale
assessing children’s (aged 8‐15 years) global self‐esteem which is available for purchase. Although
the short form does not allow individual sources of self‐esteem to be differentiated, there is
evidence that with a modified version of the scale (19‐items) there are three clear subscales that
correspond to personal self‐esteem, and self‐esteem derived from parents and peers (Hills, Francis,
& Jennings, 2011). The disadvantages of this scale are its length and that it must be purchased from
the publisher.
The Self‐esteem sub‐scale of the Weinberger Adjustment Inventory (Weinberger &
Schwartz, 1990) measures an individual’s perception of his or her value. It is a brief 7‐item subscale
in which respondent’s (aged 11+years) rate how true positive and negative statements are of them
on a 5‐point Likert scale.
In relation to the Taz Kids Club Camp program, the State Self‐esteem Scale (Heatherton &
Polivy, 1991) may be useful as it sensitive to temporary fluctuations in self‐esteem. It is comprised of
three subscales – performance self‐esteem, social self‐esteem and appearance self‐esteem. While
the ‘appearance’ subscale may not be directly relevant to the Champs Camp program an adapted
version of this scale could potentially be useful for assessing the benefits of the Camp program on
children’s self‐esteem.
ResilienceEnhancing resilience in children of parents with a mental illness is the overarching aim of
many intervention programs. Protective factors, such as self‐esteem, mental health literacy, coping
skills and social connectedness may be specific contributing factors to resilience and often resilience
scales measure these constructs. While a full review of the resilience scales available is beyond the
scope of the present review, some examples of scales which may be relevant to the COPMI program
are given below and comprehensive reviews of other available resilience scales are available
elsewhere (Ahern, Kiehl, Sole, & Byers, 2006; Windle, Bennett, & Noyes, 2011) . Another scale of
interest is the Social‐Emotional Assets and Resilience Scales (SEARS) which is available for purchase
here.
The Resilience Scale of the California Healthy Kids Student Survey (Sun & Stewart, 2007) (See
Appendix F) measures perceptions of individual characteristics and protective resources from family,
peers, school and community. It consists of 34‐items which cover many aspects of resilience
including, communication and cooperation, self‐esteem, empathy, problem solving, goals and
aspirations, family connection, school connections, community connection, autonomy experience,
pro‐social peers, meaningful participation in community activity and peer support. In particular, the
three‐item self‐esteem scale may be useful for outcome evaluations of the Camp program or as a
41
short self‐esteem measure for the Club program. However, further research is required to determine
whether the sub‐scale is suitable for administration by itself.
The Resilience Scale for Children and Adolescents (Prince‐Embury, 2007) (See Appendix F) is
based on the assumption that resilience reflects the degree to which a child experiences a sense of
mastery (MAS), sense of relatedness (REL) and emotional reactivity (REA). The scale consists of 64
items in total however the scales can be administered independently and are comprised of 20‐24
items each. The MAS consists of three subscales: Optimism (positive attitude about life in general),
Self‐efficacy (sense of competence), and Adaptability (including asking others for assistance); the REL
consists of four subscales: Trust, Perceived access to Social Support, Comfort, and Tolerance; and the
REA consists of three subscales: Sensitivity, Recovery, and Impairment. The MAS and REL scales
could be useful in future evaluations of the Taz Kids club and camp programs. In particular,
constructs such as self‐mastery and optimism may be relevant to the goals and aims of the Camp
program and may also be amenable to change over a short period of time. In addition, research has
shown negative associations between psychological symptoms and the Sense of Mastery, and Sense
of Relatedness scale scores (Prince‐Embury, 2008). A disadvantage of this scale is that it must be
purchased from Pearson.
MeasuresofaffectGiven the length of the camp program and the difficulties this creates for observing reliable
changes in the existing outcome measures, other constructs which are likely to change over a short
period of time may be assessed instead (e.g., self‐esteem, self‐efficacy, emotional well‐being,
optimism, and positive affect). Given that one aim of the Champs Camps is to provide children with
respite and ‘time out’ from the stressors of their daily life it may be useful to measure changes in
affect over the course of the camp. A very simple measure which could be administered to younger
children is the Faces Scale (Holder & Coleman, 2008) (see Appendix F). The Faces Scale comprises a
seven‐item Likert scale using a progression of faces from very happy to very sad to address the
questions ‘How happy are you most of the time?’. This scale could be easily modified to included
other mood states and varying time frames (e.g. instead of ‘most of the time’, right now, in the past
few days). For adolescents, the Positive and Negative Affect Schedule (PANAS) (Watson & Tellegen,
1988) (See Appendix F) may be more appropriate. It is comprised of 10‐items describing positive
affect and 10 items describing negative affect and respondents rate on a 5‐point scale the extent to
which they had experienced each mood state during a specified time frame (e.g., right now, today,
last few days, during the last week, few weeks). In the context of the camp, a time frame of ‘the past
few days’ could be used at both pre‐ and post‐intervention. While a comprehensive review of other
potential measures is beyond the scope of the present evaluation, there is the potential for a
Masters of Counselling student within the School of Psychology to conduct a research project
around this in 2013.
42
Table 6. A summary of outcome measures relevant to COPMI interventions
Name Purpose No. items Age range
Response format Subscales Psychometrics
Coping Measures
Adolescent Coping Scale‐ Short form (Frydenberg & Lewis, 1996); 2nd edition (ACER, 2011) – available for purchase for qualified professionals
A self‐report inventory used to measure adolescents coping strategies
18 items 12‐18 5‐point Likert (1= doesn’t apply/don’t do it, 5=used a great deal)
Productive coping, non‐productive coping and reference to others (encompass 18 different coping strategies)
Alpha coefficients for styles range from .65‐.79 with a median of .67
Children’s coping strategies checklist (Ayers et al, 1990; 1996)
A self‐report inventory in which children describe their coping efforts
44 items 9‐15 4‐point Likert (1=never 4=most of the time).
4 Factors: Active coping, avoidance, distraction, support seeking
Internal consistency reliabilities (alphas) of .46‐.89 and test‐retest reliabilities of 0.49‐0.80
Kidcope (Spirito, Stark & Williams, 1988)
Checklist designed to assess the frequency and perceived helpfulness of 10 cognitive and behavioural coping strategies
15 items (younger) 10 items (older)
5‐12 (Younger) 13‐16 (Older)
Frequency scale: Younger: yes/no Older: 5‐point Likert (0=not at all to 3=almost all the time). Perceived helpfulness: 3‐point Likert (1=not at all – 3=a lot)
Social withdrawal, distraction, wishful thinking, cognitive restructuring, social support, problem‐solving, self‐criticism, emotional regulation, resignation, blaming others.
Moderate‐high validity with coefficients ranging from .33‐.77 when items were correlated with the Coping Strategies Inventory (Tobin, 1984)
Life Skills Measure (Pitman & Matthey, 2004)
Designed to assess children’s perceived ability to use various life skills
10 items 8‐16 10‐point Likert (1=i find it very hard to do, to 10= i find it really easy to do)
Non‐standardised (constructed for program)
Responses to Stress Questionnaire (RSQ) (Connor‐smith et al., 2000)
Designed to measure adolescents responses to stress based on a multidimensional model of responses to stress (see Compas et al., 2001)
57 items adolescents
4‐point Likert (1=not at all to 4=a lot); questions asked in relation to stress domains of social stress, economic strain, family conflict, and
Secondary control, engagement coping, disengagement coping, involuntary engagement, involuntary disengagement (with 19 domains in total each consisting of three items)
Factor structure verified using confirmatory analysis, adequate to excellent internal consistency and test‐retest reliability (factors:.69‐.81, scales: .49‐76), concurrent validity with COPE (Carver et al 1989),
43
pain physiological reactivity and parents reports
Connectedness/Social Support
Multidimensional Scale of Perceived Social Support (Zimet, 1988)
Self‐report measure of social support
12 items 12‐18 5‐point Likert (1= very strongly disagree, 5= to very strongly agree
Family, friends, significant others
High internal consistency was demonstrated, and factor analysis confirmed the three subscale structures of the MSPSS: Correlations with a family caring scale supported the discriminant validity of the Family subscale
Child and Adolescent Social Support Scale (CASSS) (Malecki & Elliott, 1999)
Multidimensional scale measuring perceived social support from parents, teachers, close friends and peers rated on two dimension: availability and importance
40 items Two versions. (8‐12 years) and (13‐18 years)
Availability (6‐point Likert 1= never, 6= always) and importance (3 point Likert, 1= not important and 3=very important)
Parents, teachers, peers , friends and total frequency and total importance scores.
Internal consistency (Cronbachs alpha): total frequency =.96 ; subscales range from.92‐.95, total importance score= .96 and subscales .99‐.93.
Social Connectedness scale (Lee et al. 2001) permission required from author
Based on psychoanalytic self‐psychology theory to measure social connectedness
20 items Not specifically designed for use with children
level of agreement rated on 6 point scale (1=strongly disagree, 6=strongly agree).
Internal reliability:0.92; Correlates negatively with measures of loneliness, social avoidance, social distress, social discomfort and dysfunctional interpersonal behaviours, and positively with measures of collective self‐esteem and independent self‐construal
Mental Health Literacy
Knowledge of Mental Illness Measure (Pitman & Matthey, 2004)
Measures children’s knowledge of mental illness reflecting what they learn in the SMILES program
9 items 8‐16 10‐point scale from (1=I know nothing at all to 10=I know everything there is to know.)
Non‐standardised
Kids Knowledge Scale (Richter, 2006,
Assess’ participants knowledge and
20 items 8‐12 True/False Non‐standardised
44
unpublished Masters thesis)
understanding of the content of the Kids In Control program
Mental Health Literacy (Fraser & Pakenham, 2008)
Designed to assess adolescents knowledge of mental illness and awareness of parents mental illness in the KAP program
5 knowledge items & 4 Awareness items
8‐16 Knowledge: 4 open ended items, 1 checklist item, Awareness =4 yes/no items
Knowledge of mental illness,awareness of parents mental illness
Non‐standardised
Knowledge of Mental Illness (Hargreaves et al., 2005)
Measures ability to access mental health services and awareness of parents mental illness
5 items 12‐18 Yes/No/Not Applicable
Non‐standardised
Self‐esteem
Name Purpose No. items Age range
Response format Subscales Psychometrics
Rosenberg Self‐Esteem Scale (Rosenberg, 1979)
Measures global self‐worth by measuring positive and negative feelings about the self
10‐items (11+ years) or 6‐item (<11 years)
11+ 4‐point Likert scale (0=strongly agree 3=strongly disagree)
Internal consistency: 0.75‐.92; test‐retest: .85 and .88 over 2 weeks. Significant correlations with Coopersmith SEI and depression and anxiety. High scores are positively correlated with resilience
Self‐esteem Inventory –Short Form (Coopersmith, 1981) available for purchase here
A measure of global self‐esteem for children
25 items 8‐15 like me/ unlike me Internal consistency from .75‐.95, test retest ranging from .88 (5 weeks) .64 (3 yrs)
Self‐Esteem‐Weinberger Adjustment Inventory (Weinberger & Schwartz, 1990)
Measures an individual’s perception of his or her value
7 items 11+ 5‐point Likert (1=False, 5=True)
Internal consistency:.55 to .72
State Self‐Esteem Scale (Heatherton & Polivy, 1991)
Measures temporary changes in self‐evaluation
20 items 12‐18 5 point Likert (1=not at all, 5=extremely)
Performance self‐esteem, social self‐esteem and appearance self‐esteem.
Internal consistency (alpha=.92), separable from mood, CFA has confirmed
45
three factors
Resilience
Name Purpose No. items Age range
Response format Subscales Psychometrics
Resilience Scale ‐California Healthy Kids Student Survey‐ (Sun & Stewart, 2007)
Assess perceptions of individual characteristics, protective resources from family, peer, school and community
34 items 8‐13 5‐point Likert (1= never, 5=all the time)
Communication and cooperation, self‐esteem, empathy, problem solving, goals and aspirations, family connection, school connections, community connection, autonomy experience, pro‐social peers, meaningful participation in community activity and peer support
Confirmatory Factor Analysis indicated a goodness of fit
Resilience Scale for Kids and Adolescents (Prince‐Embury, 2007, 2008, 2010) – available for purchase from Pearson
Designed to systematically identify and quantify core personal qualities of resiliency in youth expressed in their own words and about their own experiences
64 items 9‐18 5‐point Likert (0= never, 4= almost always )(grade 3 reading level)
Sense of mastery (optimism, self‐efficacy and adaptability), Sense of Relatedness (trust, perceived social support, comfort and tolerance) and Emotional Reactivity (20 sensitivity, recovery and impairment
In normative sample of 9‐18 years old (US) the coefficients total scores on the subscales were all ≥.85, Test–retest reliabilities for these three scales was all ≥.70
Measures of affect
Faces Scale (Holder &Coleman, 2008)
Used to determine happiness in children
1‐item Young children
7‐point smiley faces This measure has been used
successfully
to assess children’s
happiness (Holder &
Coleman, 2008)
The Positive and Negative Affect Schedule for Children (PANAS‐C) (Watson, Clark & Tellegen,1988)
Designed to measure two primary dimensions of mood – positive and negative affect with varying time frame instructions
10 items Older children
5‐point Likert (1=very slightly or not at all, 5=extremely)
Positive affect, negative affect High internal consistency
(PA:.86‐.90 NA:.84‐.87) and
largely uncorrelated (‐.12 to
‐22) for a variety of time
frames
46
ReferencesAhern, N. R., Kiehl, E. M., Sole, M. L., & Byers, J. (2006). A review of instruments measuring
resilience. [Comparative Study
Review]. Issues Compr Pediatr Nurs, 29(2), 103‐125. doi: 10.1080/01460860600677643 Angold, A., Costello, EJ., Messer, EC., Pickles, A., Silver, D. & Winder, F. (1995). Development of a
short questionnaire for use in epidemiological studies of depression in children and adolescents. International Journal of Methods in Psychiatric Research, 5, 237‐249.
Australian Infant, C., Adolescent and Family Mental Health Association. (2001). Children of Parents Affected by a Mental Illness Scoping Project Report. . Stepney, South Australia: Mental Health and Special Programs Branch, Department of Health and Aged Care.
Ayers, T. S., Sandler, I. N., West, S. G., & Roosa, M. W. (1990). Assessment of children's coping behaviors: testing alternative models of children's coping. Paper presented at the American Psychological Association, Boston, MA.
Ayers, T. S., Sandler, I. N., West, S. G., & Roosa, M. W. (1996). A dispositional and situational assessment of chislren's coping: testing alternative models of coping. Journal of Personality, 64(4), 923‐958.
Brown, L., & Alexander, A. (1991). Self‐esteem index examiner's manual. Texas: Pro‐Ed Inc. Butler, R., & Gasson, S. (2005). Self‐esteem/self concept scales for children and adolescents: A
review. Child and Adolescent Mental Health, 10, 190‐201. Chamberlain, C., & MacKenzie, D. (1998). Youth homelessness: Early intervention and prevention.
Sydney, Australia: Australian Centre for Equity through Education. Chang, E. C., & D’Zurilla, T. J. (1996). Relations between problem orientation and optimism,
pessimism and trait affectivity: A construct validation study. Behaviour, Research and Therapy, 34, 185‐194.
Compas, B. E., Connor‐Smith, J. K., Saltzman, H., Thomsen, A. H., & Wadsworth, M. E. (2001). Coping with stress during childhood and adolescence: Problems, progress, and potential in theory and research. . Psychological Bulletin, 127, 87–127. doi: 10.1037//0033‐2909.127.1.87
Connor‐Smith, J. K., Compas, B. E., Wadsworth, M. E., Thomsen, A. H., & Saltzman, H. (2000). Responses to stress in adolescence: Measurement of coping and involuntary stress responses. Journal of Consulting and Clinical Psychology, 68(6), 976‐992. doi: 10.1037//0022‐006x.68.6.976
Coopersmith, S. (1981). Coopersmith inventory‐school form. California: Consulting Psychologists Press.
Diener, E., Emmons, R., Larsen, J., & Griffin, S. (1985). The Satisfaction With Life Scale. J Pers Assess, 49, 71‐75.
Donaldson, D., Prinstein, M., Danovsky, M., & Spirito, A. (2000). Patterns of children’s coping with life stress: Implications for clinicians. American Journal of Orthopsychiatry, 70, 351‐359.
Dumon, M., & Provost, M. A. (1999). Resilience in adolescents: Protective role of social support, coping strategies, self‐esteem, and social activiies on experiences of stress and depression Journal of Youth and Adolescence, 28(3), 343‐363.
Falkov, A. (2004). Talking with children whose parents experience mental illness. In V. Cowling (Ed.), Children of parents with mental illness 2: Personal and clinical perspectives Melbourne, Australia: Australian Council of Education Research.
Fraser, C., James, E. L., Anderson, K., Lloyd, D., & Judd, F. (2006). Intervention Programs for Children of Parents with a Mental Illness: A Critical Review. International Journal of Mental Health Promotion, 8(1), 9‐20. doi: 10.1080/14623730.2006.9721897
Fraser, E., & Pakenham, K. I. (2008). Evaluation of a resilience‐based intervention for children of parents with mental illness. [Controlled Clinical Trial Research Support, Non‐U.S. Gov't]. Aust N Z J Psychiatry, 42(12), 1041‐1050. doi: 10.1080/00048670802512065
47
Frydenberg, E., & Lewis, R. (1996). A Replication Study of the Structure of the Adolescent Coping Scale: Multiple Forms and Applications of a Self‐Report Inventory in a Counselling and Research Context. European Journal of Psychological Assessment, 12, 224‐235.
Fudge, E., & Mason, P. (2004). Consulting with young people about service guidelines relating to parental mental illness. Australian e‐Journal for the Advancement of Mental Health, 3.
Garber, J., & Little, S. (1999). Predictors of Competence among Offspring of Depressed Mothers. Journal of Adolescent Research, 14(1), 44‐71. doi: 10.1177/0743558499141004
Goodman, R., Meltzer, H., & Bailey, V. (1998). The strengths and difficulties questionnaire: A pilot study on the validity of the self‐report version. European Child & Adolescent Psychiatry, 7, 125‐130.
Goodyear, M., Cuff, R., Maybery, D. J., & Reupert, A. (2009). CHAMPS: A peer support program for children of parents with a mental illness. Australian e‐Journal for the Advancement of Mental Health, 8, 296‐304.
Grant, G., Repper, J., & Nolan, M. (2008). Young people supporting parents with mental health problems: experiences of assessment and support. [Research Support, Non‐U.S. Gov't]. Health Soc Care Community, 16(3), 271‐281. doi: 10.1111/j.1365‐2524.2008.00766.x
Handley, C., Farrell, G. A., Josephs, A., Hanke, A., & Hazelton, M. (2001). The Tasmanian children's project: The needs of children with a parent/carer with a mental illness. Australian and New Zealand Journal of Mental Health Nursing, 10, 221‐228.
Handley, C., & Josephs, A. (2002). Taz Kids Club Training Manual: A supportive and educative group for children aged 7‐12 years who have parents/relatives living with a mental illness: Program funded by Australian Commonwealth Reform and Incentive Funding.
Hargreaves, J., Bond, L., O'Brien, L., Forer, D., & Davies, L. (2008). The PATS peer support program. Youth Studies Australia, 27, 43‐51.
Hargreaves, J., O'Brien, M., Bond, L., Forer, D., Basile, A., & Davies, L. (2005). Paying Attention to Self (PATS): An evaluation of the PATS program for young people who have a parent with a mental illness. Melbourne: Centre for Adolescent Health.
Harvey, J., & Delfabbro, P. H. (2004). Psychological resilience in disadvantaged youth: A critical overview. . Australian Psychologist, 39, 3‐13. doi: 10.1080/00050060410001660281
Hayman, F. M. (2009). Kids with confidence: a program for adolescents living in families affected by mental illness. Aust J Rural Health, 17(5), 268‐272. doi: 10.1111/j.1440‐1584.2009.01090.x
Heatherton, T. F., & Polivy, J. (1991). Development and validation of a scale for measuring state self‐esteem. Journal of Personality and Social Psychology, 60, 895‐910.
Hills, P., Francis, L., & Jennings, P. (2011). The school short‐form Coopersmith Self‐Esteem Inventory: Revised and improved. . Canadian Journal of School Psychology, 26, 62‐71.
Hinshaw, S. P. (2005). The stigmatization of mental illness in children and parents: Developmental issues, family concerns and research needs. . Journal of Child Psychology and Psychiatry, 46, 714‐734. doi: 10.1111/j.1469‐7610.2005.01456.x
Holder, M. D., & Coleman, B. (2008). The contribution of temperament, popularity, and physical appearance to childrens happiness. Journal of Happiness Study, 9, 279‐302. doi: 10.1007/s10902‐007‐9052‐7
Hosman, C. M. H., Van Doesum, K. T. M., & Van Santvoort, F. (2009). Prevention of emotional problems and psychiatric risks in children of parents with a mental illness in the Netherlands: I. The scientific basis to a comprehensive approach. Australian e‐Journal for the Advancement of Mental Health (AeJAMH), 8(3), 250‐263.
Kovacs, M. (1992). Children’s Depression Inventory manual. New York: Multi‐Health Systems. Lazarus, R. S., & Folkman, S. (1984). Stress, appraisal and coping. New York: Springer Publishing. Lee, R. M., Draper, M., & Lee, S. (2001). Social Connectedness, Dysfunctional Interpersonal
Behaviors, and Psychological Distress: Testing a Mediator Model. Journal of Counseling Psychology, 48, 310‐318.
Malecki, C. K., & Elliott, S. N. (1999). Adolescents ratings of percieved social support and its importance: Validation of the student social support scale. Psychology in the Schools, 36, 473‐483.
48
Maybery, D., Steer, S., Reupert, A., & Goodyear, M. (2009a). The kids coping scale. Stress and Health, 25(1), 31‐40. doi: 10.1002/smi.1228
Maybery, D. J., Ling, L., Szakacs, E., & Reupert, A. (2005). Children of a parent with a mental illness: Perspectives on need. Australian e‐Journal for the Advancement of Mental Health, 4.
Maybery, D. J., Reupert, A., & Goodyear, M. (2006a). Evaluation of a model of best practice for families who have parent with a mental illness. Wagga Wagga, NSW: Charles Sturt University.
Maybery, D. J., Reupert, A., Patrick, K., Goodyear, M., & Crase, L. (2006b). VicHealth Research Report on Children at risk in families affected by parental mental illness. Melbourne, Vicotoria: Victorian Health Promotion Foundation.
Maybery, D. J., Reupert, A. E., Patrick, K., Goodyear, M., & Crase, L. (2009b). Prevalence of parental mental illness in Australian families. Psychiatric Bulletin, 33(1), 22‐26. doi: 10.1192/pb.bp.107.018861
Montgomery, R. J. V., Borgatta, E. F., & Borgatta, M. L. (2000). Societal and Family Change in the Burden of Care. In Who Should Care for the Elderly? An East‐West Value Divide. In W. Liu & H. Kendig (Eds.), (pp. 27‐54). Singapore: The National University of Singapore Press.
Morson, S., Best, D., de Bondt, N., Jessop, M., & Meddick, T. (2009). The Koping Program: A Decade's Commitment to Enhancing Service Capacity for Children of Parents with a Mental Illness. Australian e‐Journal for the Advancement of Mental Health, 8, 286‐295.
Orel, N. A., Groves, P. A., & Shannon, L. (2003). Positive Connections: A programme for children who have a parent with a mental illness. Child and Family Social Work, 8, 113‐122.
Pakenham, K. I., Bursnall, S., Chiu, J., Cannon, T., & Okochi, M. (2006). The Psychosocial Impact of Caregiving on Young People Who Have a Parent With an Illness or Disability: Comparisons Between Young Caregivers and Noncaregivers. Rehabilitation Psychology, 51, 113‐126. doi: 10.1037/0090‐5550.51.2.113
Pakenham, K. I., Chiu, J., Bursnall, S., & Cannon, T. (2007). Relations between social support, appraisal and coping and both positive and negative outcomes in young carers. [Research Support, Non‐U.S. Gov't]. J Health Psychol, 12(1), 89‐102. doi: 10.1177/1359105307071743
Patterson, J. M., & McCubbin, H. I. (1987). Adolescent coping style and behaviors: Conceptualisation and measurement. Journal of Adolescence, 10, 163‐186.
Pitman, E., & Matthey, S. (2004). The SMILES Program: A group program for children with mentally ill parents or siblings. American Journal of Orthopsychiatry, 74, 383‐388.
Prince‐Embury, S. (2007). Resiliency scales for children and adolescents: Profiles of personal strengths. San Antonio, TX: Harcourt Assessments.
Prince‐Embury, S. (2008). The resiliency scales for children and adolescents, psychological symptoms, and clinical status in adolescents. Canadian Journal of School Psychology, 23, 41‐55.
Reupert, A. E., & Maybery, D. (2010). "Knowledge is power": educating children about their parent's mental illness. [Research Support, Non‐U.S. Gov't]. Soc Work Health Care, 49(7), 630‐646. doi: 10.1080/00981380903364791
Reupert, A. E., & Maybery, D. J. (2009). A "snapshot" of Australian programs to support children and adolescents whose parents have a mental illness. Psychiatric Rehabilitation Journal, 33, 125‐132.
Richter, G. A. (2006). Fostering Resilience: Evaluating the effectiveness of Kids in Control. Master of Arts Thesis, Trinity Western University, Langley, BC.
Riebschleger, J., Tableman, B., Rudder, D., Onaga, E., & Whalen, P. (2009). Early outcomes of a pilot psychoeducation group intervention for children of a parent with a psychiatric illness. [Research Support, Non‐U.S. Gov't]. Psychiatr Rehabil J, 33(2), 133‐141. doi: 10.2975/33.2.2009.133.141
Rosenberg, M. (1979). Conceiving the Self. New York: Basic Books. Rutter, M. (1999). Resilience concepts and findings: implications for family therapy. Journal of Family
Therapy, 21, 119‐144.
49
Sandler, I. N., Tein, J., & West, S. G. (1994). Coping, stress and the psychological symptoms of children of divorce: A cross‐sectional and longitudinal study. Child Development, 65, 1744‐1763.
Skinner, H., Steinhauer, P., & Santa‐Barbara, J. (1995). Family Assessment Measure (FAM). Washington DC: Psychological Assessment Resources, Inc.
Spirito, A., Stark, L. J., & Williams, C. (1988). Development of a brief coping checklist for use with paediatric populations. Journal of Paediatric Psychology, 13, 555‐574.
Sun, J., & Stewart, D. (2007). Development of population based resilience measures in the primary school setting. Health Education, 45, 575‐599.
Tebes, J. K., Kaufman, J. S., Adnopoz, J., & Racusin, G. (2001). Resilience and Family Psychosocial Processes Among Children of Parents with Serious Mental Disorders. Journal of Child and Family Studies,, 10, 115‐136.
Tobin, D. L., Holroyd, K. A., & Reynolds, R. V. C. (1984). Manual for the Coping Strategies Inventory. Unpublished manuscript, Ohio University.
Van Doesum, K. T. M., & Hosman, C. M. H. (2009). Prevention of emotional problems and psychiatric risks in children of parents with a mental illness in the Netherlands: 2 intervetiosn. Australian e‐Journal for the Advancement of Mental Health, 8.
Viet, C. T., & Ware, J. E. (1983). The Structure of Psychological Distress and Well‐Being in General Populations. Journal of Consulting and Clinical Psychology, 51, 730‐742.
Watson, D. C., L.A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS Scales. Journal of Personality and Social Psychology, 54, 1063‐1070.
Weinberger, D. A., & Schwartz, G. E. (1990). Distress and restraint as superordinate dimensions of adjustment: Atypological perspective. Journal of Personality and Social Psychology, 58, 381‐417.
Windle, G., Bennett, K. M., & Noyes, J. (2011). A methodological review of resilience measurment scales. Health and Quality of Life Outcomes, 9.
Zimet, G. D., Dahlem, N. W., Zimet, S. G., & Farley, G. K. (1988). The multidimensional scale of perceived social support. Journal of Personality Assessment, 52(1), 30‐41.
50
AppendixA–KidsProblems,ConnectionsandCopingScaleNote: question numbers have been added to the original questionnaire to allow reference to specific
questions for the purposes of this report
Name: ____________ Age ____
When you have a problem or something goes wrong, circle how often you;
10. Some people have lots of friends, others have a few. Think about your friends
now. How many friends do you have? _____
11. In the last month, how many times have you gone to a friend’s house to play?
_____
12. In the last month, how many times have you had a friend over to play? _____
1. Try to think of different ways to
solve the problem
2. Don’t want to think about the
problem
3. You think about what others might
do
4. You try your best to make things
better
5. You avoid the problem or where it
happened
6. You ask someone to help
7. You try hard to fix the problem
8. You do things to stop thinking
about it
9. If it is your fault, you say sorry
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
51
We want to know how often you had good and bad times with other people in
the last few weeks. For the people shown below, circle how often you did the
following;
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
Never Sometimes A lot
13. Time spent with your Mum
14. Time spent with your Dad
15. Good time spent with your Brothers
16. Good time spent with your Sisters
17. Time spent with a special Grandparent
or other relative
18. Time spent with your friend
19. Another grown up? Please write their
name here _____________
20. You had problems with your Mum
21. You had problems with your Dad
22. You had problems with your Brothers
23. You had problems with your Sisters
24. You had problems with a Grandparent
(or other relative)
25. You had problems with a friend
26. You had problems with another person?
Please write their name here ____________
52
AppendixB–CopingMeasures
Adolescent Coping Scale‐ Short Form (Frydenberg & Lewis, 1996)
Rated on a 5‐point Likert scale from (1= doesn’t apply/don’t do it, 5=used a great deal)
+ item 19 which asks respondents to write down any things they do to cope other than those things
described in the preceding 18 items.
53
Children’s Coping Strategies Checklist (Ayers, Sandler, West & Roosa, 1990)
Sometimes kids have problems or feel up‐set about things. When this happens they may do different things to solve the
problem or to make themselves feel better. For each item below, choose the answer that best de‐scribes how often you
do this to solve your problems or make yourself feel better. There are no right or wrong answers, just indicate how often
you usually do each thing." Each item is rated on a 4‐point Likert scale from 1 (never) to 4 (most of the time).
54
55
KidCope (Spirito, Stark & Williams, 1988) (Child form)
56
KidCope (Adolescent form)
57
Life Skills Measure (Pitman & Matthey, 2004)
58
AppendixC–ConnectednessMeasures
Multidimensional Scale of Perceived Social Support (Zimet, 1988)
59
Child and Adolescent Social Support Scale (CASSS) (Malecki & Elliott, 1999)
Example of items from Malecki & Demaray (2002) ‐ (level 1=8‐12 years, level 2=13‐18 years)
Frequency (6‐point Likert 1= never, 6= always) and importance (3 point Likert, 1= not important and 3=very
important)
60
Social Connectedness scale (Lee et al., 2001)
6‐point scale from 1=strongly agree to 6=strongly disagree
61
AppendixD–MentalHealthLiteracyMeasures
Knowledge of Mental Illness Measure (Pitman & Matthey, 2004)
62
63
Kids Knowledge Scale (Richter, 2006)
64
65
Knowledge of Mental Illness (Hargreaves et al., 2005)
66
AppendixE–Self‐esteemmeasuresRosenberg Self‐Esteem Inventory (Rosenberg, 1979)
67
Coopersmith Self Esteem Inventory (Coopersmith, 1981)
68
69
Self Esteem‐ Weinberger Adjustment Inventory (Weinberger & Schwartz, 1990)
70
State Self‐Esteem Scale (Heatherton & Policy, 1991)
Rated on 5 point Likert scale (1=not at all, 5=extremely)
71
AppendixF–ResiliencemeasuresThe Resilience Scale of the California Healthy Kids Student Survey (Sun & Stewart, 2007)
72
73
Resilience Scale for Children and Adolescents (Prince‐Embury, 2007)
e.g. of items
74
AppendixG–Affectmeasures
Faces Scale (Holder & Coleman, 2008)
Positive and Negative Affect Schedule (Watson, Clark & Tellegen, 1988)