program evaluation for kids on the block

7/31/2019 Program Evaluation for Kids on the Block

1/58

1

Evaluation Proposal

Program: Kids on the Block

Prepared by: Anna Gosselin, Christine Liu, Samriti Mishra, Marie-France

Par, & Hetal Patel

School of Public Health & Health Systems

University of Waterloo

Prepared for: Dr. Anita Myers

The Independent Living Centre of Waterloo Region


2/58

2

Executive Summary

Background: The Kids on the Block (KOTB) program is an educational puppet show

presented in schools and other community venues aimed at raising childrens awareness

of persons with disabilities. This program is a subset of the Independent Living Centre of

Waterloo Regions community outreach initiative.

Purpose: The purpose of this evaluation is to gain an in-depth understanding of the

KOTB program so that recommendations can be made on their current practices and

suggestions can be provided to enhance program delivery. Through observation of the

program and staff interviews, an assessment can be made on their record-keeping such

that it can be useful to identify the impact of their program. Concurrently, an evaluation

study will help KOTB meet the needs of the program, as well as the needs of its

stakeholders.

Recommendations: The evaluation team made the following key recommendations: a)

the program should analyze their previously collected data for a snapshot of their current

circumstances; b) collect more data and analyze it via graphs, tables and trend charts; c)

centralize their records to one location; d) update their teacher feedback form; and e)

enhance their volunteer and client recruitment strategies.

Methods: Background information for this evaluation plan was collected through an

interview with a staff member, and review of pertinent literature. The evaluation plan

consists of compiling and reviewing existing records, obtaining feedback from teachers


3/58

3

through focus groups, and gauging the value of the program through in-class interviews

with the participating children.

Final Notes: As a last remark, the evaluation team recognizes the resource constrictions

that KOTB may face. A detailed evaluation will allow KOTB to see what steps they can

take next to improve their program; this will provide KOTB the rationale and tools to

request further funding from stakeholders. Consequently, KOTB will be able to provide

their service to the community in the most effective way possible, while still fulfilling

their objectives to raise awareness of persons with disabilities.


4/58

4

Table of Contents

1. Introduction & Overview 1

2. Program Description 2

3. Current Record-Keeping Practices 34. Program Goals & Objectives 4

Logic Model 6

5. Evaluation Readiness 6

6. Literature Review 7

7. Recommendations 11

Routine or Ongoing Data Collection 11

Periodic Evaluation Studies 13

8. Proposed Evaluation Plan 15

Project Outline or Design 15Sampling Procedures 17

Data Collection Procedures & Tools 18

Timeline & Resources 21

9. Next Steps 23

10. Summary & Conclusions 24

11. References 25

12. Appendices 27

Appendix A: Kids on the block interview 27

Appendix B: Kids on the Block Stakeholder Relationships 29

Appendix C

30

Appendix C1: Kids on the Block Logic Model 30

Appendix C2: Current Teacher Feedback Form 31

Appendix C3: Improved Teacher Feedback Form 32

Appendix D 34

Appendix D1: Operational Definitions 34

Appendix D2: Literature Review tables 35

Appendix E: Evaluation design 37

Appendix F: Focus Group Recruitment Materials

38

Appendix F1: Teacher Recruitment Letter 38

Appendix F2: Participating Childrens Recruitment Letter

39

Appendix F3: Non-Participating Childrens Recruitment Letter

39Appendix F4: Parental Information Letter and Consent Form 40

Appendix G: Focus Group Materials 42


5/58

5

Appendix G1: Consent Forms for Educator/Teacher

42

Appendix G2: Focus Group Questionnaire

44

Appendix G3: Focus Group Protocol 46

Appendix G4: Focus Group Script 47Appendix H: In-Class Discussion Materials 48

Appendix H1: Questions for Ages 6-8 years 48

Appendix H2: Group Protocol 49

Appendix I: Staff Interview Script 50

Appendix J1: Consent from School Board for Evaluation

52

Appendix J2: Consent from Principle for Evaluation 53


6/58

6

1. Introduction & Overview

The purpose of this project is to gain an in-depth understanding of the Kids on the

Block program through observation along with staff interviews to facilitate

recommendations on its current record-keeping practices. Concurrently, an evaluation

study will be designed that meets the information needs of the program and its

stakeholders. The team member had no direct affiliation with the Kids on the Block,

however, Marie France Par has previously worked as an attendant for the Independent

Living Centre.

Information for this project was gained through first-hand observation of the

program at Sunshine Montessori School in Kitchener, ON, by two members of the

evaluation team (M.F.P., H.P., others couldnt attend due to scheduling conflicts). In

addition, three members of the evaluation team (M.F.P., A.G., S.M.) conducted a 90-

minute, structured interview with the ILCWR Community Support Service Director and

the Kids on the Block Coordinator at ILCWRs head office. The interview script was

developed via a preliminary understanding of the program via the programs website,

which has been transcribed (C.L., Appendix A). The evaluation team was provided with

two pamphlets at the end of the interview: one aimed at recruiting volunteers and the

other provided to potential host schools. The 2010-2011 annual report created by ILCWR

was also consulted. Both materials were used throughout to help synthesize materials for

evaluation.

This report aims to provide a detailed background description of the Kids on the

Block program, the programs current practices and the literature review on the topic of

disability awareness in children. This will provide rationale for the recommendations for


7/58

7

the programs ongoing data collection as well as the proposed evaluation plan (concurrent

process and outcome evaluation).

2. Program Description

Mennonite Central Committee founded the Independent Living Centre of Waterloo

Region (ILCWR) in 1982 under the direction of Henry Enns, a social worker and

wheelchair user and a pioneer in combatting social stereotypes towards persons with

disabilities. The organizations initial mandate consisted of assisted living services, which

quickly expanded to attendant services, outreach programs and community support

services. The Kids on the Block program began in 1983, as part of community support

services under the Public Awareness and Education branch of ILCWR (ILCWR, 2012).

The Kids on the Block (KOTB) program is an educational puppet show presented

in schools and other community venues aimed at raising childrens awareness of persons

with disabilities. The program makes use of a Japanese style of puppetry called Bunraku.

This authentic style of Japanese puppet theater is supposed to be childlike and realistic,

representing children aged 11 or 12 years old (Pappas, 2012). The show gives children

the unique opportunity to ask questions about living with disabilities. The performance

also celebrates differences and emphasizes inclusivity and friendship (ILCWR, 2012).

ILCWR purchases puppets and the rights to the show scripts from an American

organization, who initially established the concept of Kids on the Block.

Two staff members and a volunteer base of puppeteers operate the program

currently. Puppet shows are booked by schools and other community groups who make

arrangements with the puppeteers to choose a date for the show. While ILCWR has made

some advertising efforts in the past (brochures, mail-outs), knowledge of the program is


8/58

8

spread mainly through word of mouth and referrals. Kids on the Block is partially funded

by outside community groups such as the Region of Waterloo and United Way. While

the shows are offered for free, the program accepts any donations from the host

organizations. The stakeholder relationships are depicted in Appendix B.

3. Current Record-Keeping Practices

Currently, KOTB routinely collects basic information about each booking including

the contact information, facility, presentation content, date, time and attendance.

Additionally, data on volunteers, donations received and client satisfaction (although

brief) were recorded. In terms of client satisfaction, teachers at the host school are given

feedback forms to fill out and return by mail (Appendix C2).

KOTB collects data both before and after each show. To collect this information,

the program coordinator requests administrative information prior to the presentation.

Client satisfaction surveys are given out to hosts after each puppet presentation. All of

this information is stored electronically in a spreadsheet format and updated on a monthly

basis.

We identified improvements that can be made to the programs current record

keeping practices, especially with regards to the satisfaction survey. Better guidance will

be provided by the use of comprehensive questions. This will allow Kids on the Block to

obtain information that is relevant to the improvement or maintenance of the program.

Furthermore, the information from these surveys is not collectively summarized at a

centralized location (e.g. one computer system or location dedicated to Kids on the

Block). The Penpal program used to collect questions from children after the show is not

currently analyzed. Further, data from educators that dont fill out the feedback form and


9/58

9

students that dont ask questions arent accounted for. As a result of limited resources for

the program, regular outcome information is hard to both capture and maintain.

The lack of a centralized database for information hinders qualitative analysis that

can be done to reveal changes in client satisfaction or program needs. Without taking into

account the Penpal program, any shifts in awareness of the community wont be evident.

The proposed evaluation program will allow for adequate use of the abovementioned

information.

4. Program Goals & Objectives

The programs main goal is to contribute to a barrier-free environment for persons

with disabilities by raising childrens awareness using a variety of relevant topics. This

goal is portrayed on the Independent Living Centres website, as an overarching objective

or a broad mission statement for the Kids on the Block program. Currently, the program

does not have any goals and objectives officially composed. As such, the evaluation team

has put together a list to reflect the programs goals and objectives as closely as possible.

The list, found in Table 1, was generated through information retrieved from the

interview, program website, and observation.

The goals and objectives need to be even more specific to be able to determine

how they will be fulfilled. Table 1 is a good starting point for breaking down some of the

general objectives. The goals and objectives in Table 2 are explicit statements that are

classified into categories that include broad mission statements (BM), process or service

delivery objectives (P), outputs objectives (O), and client-oriented outcomes (C) (Myers,

1999). These specific goals and objectives are formulated to also reflect some of the

current needs of the program and will be discussed in greater detail later in this report.


10/58

10

Table 1. Program objectives

Broad Specific

To encourage discussion To encourage children to talk about their concerns and

feelings regarding those with disabilities

To encourage children to ask questions

To educate To educate children on both specific disabilities as well as

broader themesTo educate children and individuals on appropriate

terminology

To provide resources To offer resources for educators and children on

disabilities

To increase awareness To encourage community engagement by involving other

stakeholder groups

Table 2. Classification of goals and objectives

Goals and Objectives Classification

To promote a barrier-free environment BM

To increase awareness about disabilities BM

To develop a more comprehensive client satisfaction survey P

To hire more staff or recruit more volunteers P

To consolidate or summarize data currently collected P

To attract more clients and achieve a greater response rate P

To provide information and resources for students and the

community

P

To obtain more referrals P

To achieve a high satisfaction rating from program users O

To increase response rates in the pen pal program O

To increase openness and discussion of the topic presented O


11/58

11

To increase childrens knowledge about disabilities and itsassociated terminology

C

To increase self-esteem and ability to embrace individuality C

Logic Model

The Kids on the Block program doesnt have a formal logic model. Appendix C1

displays Kids on the Block as one of the activities of ILCWR in a logic model created by

the evaluation team. The model was created using the pamphlets provided by the program

as well as information collected from the ILCWR website and KOTB interview.

5. Evaluation Readiness

As mentioned earlier, the KOTB program has scarce resources to conduct an

evaluation. The program currently only has one part-time staff member and one full-time

staff member who is on maternity leave and is being temporarily substituted. This

program also has a very limited budget as it is a not-for-profit organization that depends

mainly on donations and funding from external organizations like the United Way (see

Appendix B for a complete list). The KOTB is accountable to the United Way and the

Region of Waterloo along with the ILCWR and the regional government. The program is

occasionally requested to provide data to these funders on the use of the program in order

to receive their funding. The Kids on the Block program is also reported on in ILCWRs

annual report.

There has been no prior evaluation conducted in this program. The staff in charge

of KOTB were very receptive to having an evaluation and are interested in obtaining


12/58

12

feedback about their program. Specifically, they are interested in knowing whether

children who see their show display increased knowledge and a positive attitude towards

disabled persons. We speculate that the key stakeholders will be interested in knowing

the breakdown of program costs to understand the specific impact of their donations.

They might also be interested in knowing if there are more resource requirements for the

program to optimally meet the prescribed goal objectives. Finally, the stakeholders would

like to know about any attitude shifts in the Waterloo community, as well as any future

plans the program may have to improve operations and delivery.

6. Literature Review

PubMed was used as the primary database, along with some initial searches on

Google Scholar. A news article was also obtained by doing a quick search on Google.

The search terms included disability, awareness, program evaluation, and child*.

This yielded a total of 16 articles, which was then subjected to the exclusion criteria of

topic relevance. This was applied to the article headings and abstracts, resulting in three

articles. Two additional articles were sought out from the reference section of these

articles for background information. Grey literature was retrieved for examination by

typing the words kids on the block disability awareness under recent news articles on

Google.

The following is a summary of the findings upon condensing the literature review.

It includes background information on adults and childrens attitudes towards persons

with disabilities in addition to examples of evaluation studies performed on disability

awareness programs similar to Kids on the Block. The section wraps up with the


13/58

13

implications of these studies on the proposed design of our program evaluation. Please

refer to Appendix D1 for all operational definitions of theories and instruments

underlined.

Attitudes towards disabilities

For a childs social and academic development, social acceptance and a true sense

of belonging are important. However, it has been found that despite a plethora of

opportune moments to interact with children with disabilities, most able-bodied children

do not take advantage of these opportunities. These barriers are overcome only by

encouragement and support towards positive interactions by authorities. Furthermore,

persuasive theory hypothesizes that for attitude shifts, repetitive information and positive

attitudes of authority figures play important roles (Tavares, 2011).

Inclusive schooling also requires institutions to train teachers to adapt to

heterogeneity in their classrooms. Research in psychology has revealed that increased

social contact between members of groups reduces negative attitudes substantially. Both

the number and quality of interactions affect the impact of the event. Further, it has been

found that the effect of interaction on perception is higher than an educational course on

disability awareness (Hein et al., 2011). Sharma et al. (2006) administered a survey to a

group of 1060 teachers in training to assess the impact of an inclusive education;

Australia, Canada, Hong Kong, and Singapore were the main areas survey distribution.


14/58

14

Results indicated that teachers with previous interactions with disabled persons had a

higher positive outlook towards inclusive education than those without.

The program usually caters to children from 8 to 11 years of age, which is when

imagination grows in leaps and bounds. Psychologists have seen an abundant amount of

evidence for symbolic thinking during these ages as well. Hence, the KOTB program can

provide an ideal environment for increasing childrens exposure to disabled persons by

making use of symbolic thinking via puppetry (Peskin & Wells-Jopling, 2012).

Guiding research

Fox and colleagues (2010) investigated childrens concepts of mental illness using

the framework of the nave theory approach. This school of thought maintains that

children develop concepts about their surrounding by creating simple theories that are

constantly challenged as they are exposed to new information. This study used semi-

structured focused interviews with children ages 5 to 11. It was found that younger

children tended to rely on their knowledge of common physical illnesses when thinking

about mental illnesses while older children demonstrated differences in their thinking

about mental and physical illnesses (Fox et al., 2010).

Hein and colleagues (2011) developed a new instrument, the Implicit Assessment

Test, and compared its finding with self-report measures. The researchers also asked

participants about the extent and quality of their contact with persons with disabilities. It

was found that participants overt attitudes were improved through increased quality of

contact with persons with disabilities. However, participants implicit, or subconscious,

attitudes remained unchanged (Hein et al., 2011).

Prior evaluation studies


15/58

15

While no formal evaluations of KOTB programs were encountered, similar programs

have undergone evaluations that provide useful insights for the development of an

appropriate evaluation plan.

Kids Are Kids is a program aimed at children in grades 1 through 8 which focuses on

encouraging and guiding kids to actively include their peers with disabilities. It consists

of a 45-minute presentation made by a consultant to depict a child with disabilities

experiencing difficult peer relations. An evaluation conducted by Tavares (2011) used

two measurement instruments: the Chedoke McMaster Attitudes Towards Children with

Handicaps scale (CATCH) and a Social Interaction Questionnaire. It was found that the

program had a positive impact on childrens attitudes towards peers with disabilities.

Students with disabilities for whom the program was held also reported increased social

inclusion (Tavares, 2011).

Ison and colleagues (2010) evaluated Just Like You, an in-class workshop for grade

5 students in Sydney, Australia. This program consists two 90-minute information

sessions. Following the program, the childrens attitudes were quantitatively assessed

using a questionnaire and qualitatively assessed through small focus groups. The

evaluation team found a significant short-term improvement in attitude, knowledge and

acceptance of people with disabilities (Ison et al., 2010).

Summary and implications

The background information gained from the literature review indicate that school-

aged children need guidance and encouragement in order to actively include their peers

with disabilities. Additionally, exposure to individuals with disabilities provides a

valuable opportunity for children to ask questions about living with these challenges (Fox


16/58

16

et al., 2010; Tavares, 2011). This clearly demonstrates the value of awareness programs

such as KOTB. In fact, using puppets provides non-threatening exposure to children with

disabilities while working with their symbolic thinking. These evaluations demonstrate

various tools and methods used to assess changes in attitudes in children. Questionnaire-

based instruments and focus groups aimed at obtaining childrens opinions were effective

in gathering information about these programs (Ison et al., 2010; Tavares, 2011). The

details of the aforementioned studies have been summarized in tables (Appendix D2).

7. Recommendations

Routine or Ongoing Data Collection

There are a number of limitations with regards to theprograms routine data

collection and record-keeping practices. These are outlined in Table 3, along with some

important recommendations.

Table.3. Limitations of the current record-keeping practices and recommended

actions

Limitations Recommendations for each Limitation

The information collected is not

collectively summarized

Make use of charts, graphs and link various

data to determine associations and trends

There is a lack of staff and volunteers Hire more staff or recruit more volunteers

by providing incentives and enhancing

recruitment strategies (promotions)

Limited internal reporting requirements Come up with standard and timely protocols

for reporting of program progress

Lack of formal external reporting

requirements

A periodic report should be compiled for allrelevant stakeholders to increase chances of

funding

Evaluation of program effectiveness is

minimal

Create a more comprehensive questionnaire

that includes more relevant measures

Client satisfaction questionnaires have low Implement an online survey system that is


17/58

17

response rates anonymous

Program is mainly operating on a referral

basis

Work on strengthening their advertising

strategies through the use of various media,specifically recruitment website along with

email bookings

Obstacle to determine consistency in

presentation delivery

Have a debriefing after each presentation

The information currently collected is very basic and mostly administrative. The

use of spreadsheets is a great tool to organize data like contact information, attendance

and donation amounts. It can further enhance the utility of their data if they were to add

additional information. For example, location information can be augmented by making

additional notes (i.e. capacity limitations) to identify possible barriers to effective

presentation delivery. Throughout the year, the program can periodically assess how

these limitationswhether it is space restrictions or accessibility issuesare associated

with client satisfaction. These minor adjustments in record-keeping will greatly improve

the KOTBs ability to control for any factors that may affect the impact of their program.

In this way, they are collecting more information relevant to program delivery and

operations so that analysis can be done to reach meaningful conclusions.

It would be advantageous for KOTB to enhance their online presence and utilize

the electronic networking opportunities available. As outlined in Table 3 above, it is

recommended that the program make use of a personal website. In conjunction with

consolidating their regularly collected booking information, a website platform can help

in creating a database of contact information of those interested. This database can be


18/58

18

used for sending out advertisements and/or a periodic community newsletter. Further, by

implementing increased use of online resources, the teacher feedback form could be

administered in an electronic format. Teachers would receive a follow-up email after the

show, enabling them to fill out the form. This approach would greatly increase the

number of responses. Refer to Appendix C2 and C3 for the current and improved

feedback forms.

There are some challenges with regards to the security of the information collected,

especially since most of it will be stored on a computer system. Passwords and

encryptions may be used for the documents and files. If any of the evaluation information

were to be used in the future, for example to advertise the success of the program, then

informed consent would be added with the feedback questionnaires. Consent will also

need to be obtained so that the clients are aware of what the data collected is being used

for. The evaluation team suggests that anonymity be used in the survey to gain more

accurate responses.

Periodic Evaluation Studies

Table 4. Rationale for the choice of evaluation plan

Evaluation Plan Priority Rationale

Needs Assessment No The program has already been implemented.

Periodic Needs Perhaps

Later

This may be required after outcome evaluation

to determine if the target groups need to beupdated, and to recognize any specific needs.

Implementation No The program has already been implemented.

Formative No Interest has already been established; program

has already been implemented.


19/58

19

Process Yes Record-keeping practices and data collection

need improvement to enable an adequate

outcome evaluation.

Outcome Yes The program wishes to know whether their

program creates an impact/is effective. This

will be done concurrently with Process.

Cost Analysis Perhaps

Later

Since the program does have budget

constraints, it would be beneficial to determineif the new implementations are efficient and if

their investments are paying off.

It is known that the program wishes to determine the impact of their program on the

children and the community. Consequently, it is important for them to identify which of

their program objectives are being fulfilled and which need more attention. Throughout

the literature, it is already known that awareness programs such as Kids on the Block can

have an important impact on childrens attitudes towards persons with disabilities.

However, it is difficult to actually measure changes in awareness. Thus, in order to carry

out an appropriate outcome evaluation, record-keeping practices and data collection

protocols need to be established. If there are no protocols in place for collecting data and

analyzing them, it may be difficult to conduct an outcome evaluation or analyze the

results for effectiveness. This is why the evaluation team has proposed to conduct both

process evaluation and outcome evaluation as stated in Table 4 above. The process

evaluation will enhance the programs ability to be prepared for an outcome evaluation.

The findings of the evaluations will lead to the following desired outcomes if executed

thoroughly and carefully:

The program will be able to:

Monitor the effectiveness or impact of their program

Determine trends over time

Make decisions about their staffing requirements


20/58

20

Make decisions on their advertising strategies

Determine what topics are popular among their clients to alter delivery strategies

Provide internal and external stakeholders with information on the effectiveness

of their operations

Receive more funding from stakeholders to enhance their program

Assist with the fulfillment of program objectives if data is collectively

summarized

We omitted a readability analysis, as KOTB is not delivered via use of print

materials and the pamphlets provided are too short to analyze.

8. Proposed Evaluation Plan

Project Outline or Design

This evaluation will help identify if the program is seen as effective and will

highlight any other barriers Kids on the Block program may face.

Additionally, this process evaluation is designed to address the following questions:

What are the strengths and weaknesses of the current procedures used by the

program?

Where is the demand of the program?

What age groups does the program reach-out to?

Are current record-keeping procedures efficient and collect sufficient

information?

Is the advertisement of the program adequate?

Are allocated resources to the program adequate?

The following is the rationale along with a proposed design for addressing the

abovementioned questions. An analysis of the consolidated spreadsheet (that contains the

programs administrative information) will shed light on which communities have been


21/58

21

exposed to the show, the age group of the audience and times of the year when the show

is most often used. Simple trend graphs would be suitable to display such information.

Any disparities between KOTBs goals and objectives will be noted to identify any

barriers to reaching their target audiences. This process will help indicate the types of

information that should be collected on a regular basis, as well as provide baseline

information for comparison post the evaluation process.

Second, focus groups will be conducted with teachers who have seen one of the

KOTB presentations. These focus groups will help identify if the program is seen as

effective, how these individuals found out about the program, why they decided to bring

in the play and if they believe the topics are appropriate.

Finally, in-class interviews lead by research staff with children directly following

the puppet show will provide insight into what the audience takes away from the

presentation. This will be compared with control focus groups with children who have

not yet seen the show. Both of these interviews will be held immediately after the show.

These discussions will consist of structured group interviews aimed at determining the

effect of the program on childrens attitudes towards persons with disabilities. The

control focus groups will also provide a baseline for current attitude towards disabilities.

The schematic for the proposed design can be found in Appendix E.

The following section discusses the strengths and limitations of the proposed

evaluation plan. The analysis of the currents records being administrative in nature, have

low burden on the participants, are comprehensive and do not require high level of

expertise. On the other hand, satisfaction surveys, focus groups and the in-class

interviews, which will provide improved qualitative data, require substantially more


22/58

22

funds, time, and expertise. These strategies also require heavy involvements by the

participants. Obtaining informed consent is critical for both the teacher focus groups and

the in-class interviews with the children. In the latter case, consent would have to be

provided by parents and extra care would have to be taken with the children. Someone

who has ample expertise in conducting discussions with children should be appointed to

lead the discussions. All in all, the analysis of current records, the teacher focus group

and the in-class interviews will help build a comprehensive understanding of the current

procedures as well as provide ideas to improve these procedures.

Sampling Procedures

In the first month of the process, starting in July, an analysis of the existing records

will be completed. The objective for this month is to consolidate all previous information

into a centralized database on a spreadsheet. This new data set will additionally include

information on previous use, media advertisements, contacts, and questionnaire responses

from current client feedback forms. In the next three months, the majority of the

evaluation will be carried out (i.e. August, September and October). Evaluators will come

in and train the KOTB staff to assist with information gathering and informed consent

procedures. After puppet presentations, KOTB will provide the teachers with a copy of

parental consent forms (Appendix F4) to be sent home to any willing teachers

homeroom students. Further, teachers from the non-participating classes in the same age

group (controlling for age) will be sent an information letter as well as teachers and

childrens consent forms (Appendix F3).

There will be four to six teacher focus groups, depending on when data saturation is

reached. Each focus group will have between six to eight teachers. Letters will be sent to


23/58

23

teachers who have previously seen the play asking them to participate in these focus

groups. Consent will be acquired at the beginning of the sessions (Appendix G). All

information gathered during the session is confidential and will be kept in locked

cabinets, preferably at the ILCWR head office. All audiotapes will be used to code the

sessions, but once the coding is complete they will be destroyed. Follow-up telephone

interviews (for which a consent form is provided in Appendix G1) will be conducted with

a few participants to ensure that the conclusions of the session are consistent with their

recollection. Finally, at the end of the session a questionnaire will be given to participants

(Appendix G2).

Four to six in-class focused interviews will be held at the school receiving the Kids

on the Block show. The number of children involved will depend on the size of the

classes and how many parents agree to their childs participation. Prior to the show,

letters will be sent to students informing them of the show and requesting parents

consent for their childs participation in an in-class discussion lead by a member of the

evaluation team. The information letter and consent form can be found in Appendix F.

Data Collection Procedures & Tools

A. Centralized Data Collecting System

Current records and data from the previous three years will be put into a centralized

database. Previous data include contact information, facility, donations and audience

numbers, volunteer numbers, presentation topics, and booking repeats. New data

collected will be placed in the centralized system to include previous data, in addition to

specific contact information (school email, telephone), the composition of the audience, ,

new or returning users, program advertisement, satisfaction questionnaire responses, and


24/58

24

Penpal program responses. Furthermore, information like how bookings are made (email

or telephone), topic/script request and number of volunteers will also be collected. This

allows for an organized collection of all records, and easy analysis of seasonal or yearly

trend that can be presented to United Way annually. This can also reveal information on

scheduling trends for volunteers and employees, and thus provide a better overview of

when the need is greatest for volunteer recruitment. Mileage spent on travelling to

presentation destinations can show current expenditure on travel costs, while satisfaction

survey information can provide indications of what advertising means would be most cost

efficient and result in a greater reach. The overarching rationale is to have all of the

records in one location. This ensures organization of data, allows for analysis of trends,

and can reveal if current practices are effective in promotion and maintenance of the

program. In order for information to be directly collected and transferred to a central

database, mobile data collection is recommended to gather data immediately after

presentations. This will allow for immediate collection of data with regards to numbers of

children present and any comments the teachers/educators may have. A codebook for

each data file should be created for the provision of data entry and interpretation. This

will be developed by the hired facilitators and will also depend on the software program

used.

B. Interviews with staff members

Interviews will be conducted with current KOTB staff members to gain insight

into daily procedures. This will allow program operators to express interests, concerns or

questions they would like answered in the process evaluation. It is important to note the


25/58

25

opinion of the program staff to evaluate current operations of the program, and of the

work setting. Please refer to Appendix Ifor interview questions and topics.

C. Focus group

Focus groups will collect information from educators that have attended the

presentation to collect information on the effectiveness of current advertising strategies,

registration and client recruitment efforts. Information about current program opinions

will also be collected and suggestions on improvements to the program can be made by

educators. Please refer to Appendix G for focus group script, procedure and recruitment

letter.

D. In-class discussions

Group discussions will be held with children, led by members of the research team

who have just participated in a Kids on the Block show. With the assistance of their

teacher, children will split up into groups of 4 to 6 and discuss pre-determined questions

lead by a member of the research team (Appendix H). The researchers will be sensitive

to any children who appear uncomfortable with the discussion. Provided permission from

Waterloo Region School board (Appendix J), principal consent (Appendix J) and parental

consent is obtained (Appendix F), the discussion groups will be conducted and the

childrens answers will be digitally recorded with an audio-recorder. Interview script will

depend upon age group of children watching the show and groups will run for 30 minutes

as to avoid interfering with class schedules. Different age groups will have questions

tailored to their comprehension level.

E. Feedback survey


26/58

26

Survey for program user (teachers, parent, camp supervisor) will be sent out via a

follow-up email shortly after presentation. The feedback survey will be available online.

Information on the survey is only made available to KOTB staff to maintain

confidentiality. For program users without internet access, a pre-addressed pre-stamped

envelope will be given after the presentation to collect feedback information regarding

the presentation. Survey will cover aspects of satisfaction with the play,

booking/registration, time availability for booking of presentations, program advertising

and overall view of the program. The online survey can be taken on surveymonkey.com

or with the use of the FluidSurveys program. The online survey will ensure instant

transfer of survey information to evaluators to input into the centralized data collection

and convenience for users. Please refer to appendix C3 for the feedback survey.

Timeline & Resources

July1st

to November 30th

, 2012 is the proposed timeline for the process evaluation.

The summer months and first semester of school is covered to reach two main

demographics including summer programs (camps, community programs) and

elementary schools. The deadline for the process evaluation release will be before school

resumes in the following year so that stakeholders may have this information for the new

upcoming year.

Table 5. Evaluation plan projected timeline

Projected Date Action

July 1 - November 30 2012 Process evaluation will take place.

July 1stAugust 1st Data entry of previous records dating back 3 years andcollection of incoming data from previous use, media


27/58

27

advertisements, contacts, questionnaire responses and

student letters. Consolidation of previous feedback and

preliminary analysis to guide the evaluation.

August 1 2012 Book rooms for staff interviews and teachers/educators

focus groups

August 10 - August 20 2012 Conduct interview with KOTB staff to assess currentprogram practices. Training by the evaluators to help

dispense forms to the teachers of the school. Handling of

documents

August 1 - August 25 2012 Focus group participation requests will be distributed to

the educators at the end of the play. In-class discussion

permission documents will be sent to parents/guardians of

children who have seen the play. Permission forms willalso be sent to Waterloo Board of Education for approval

of evaluation project in schools, to principals ofparticipating schools and also to teachers that may host thediscussion groups. Teachers of non-participating classes

from the same age groups will also be sent a background

information questionnaire and consent forms.

August 1November 302012

Process evaluation report writing

September 15October 302012

Focus groups will be conducted. At least one group will

be conducted each week, giving a greater amount offlexibility for educators. Follow up telephone interview to

confirm consistency of finding during focus group will be

conducted after two to four weeks of the focus group.

In class discussions with children will also take place afterpresentations. Permission forms from parents/guardians

should be collected a week prior to actual presentation.

November 1-November 30

2012

Synthesis of all data collected. Deadline for release of

process evaluation to Kids on the Block program staff

November 30th, 2012.

External consultants (facilitators) and assistants will need to be hired for the focus

groups, in-class discussion groups and to perform functions like data collection, data

entry, coding and staff interviews . Data analysis will also be done with external


28/58

28

consultant to avoid any bias and conflicting interest. Audio recordings for focus groups

and interviews will be sent to transcribing services to reduce time needed for information

collection. Bookings for rooms large enough to seat 14 people will be needed for focus

groups. This could be conducted at the schools where presentations take place, or in the

Region of Waterloo School Board training classrooms for ease of access. For focus

groups, at least 10 chairs, one large round table are needed for focus groups consisting of

6 to 8 teachers, 1 facilitator and 1 assistant. Interviews with current KOTB staff can be

performed in the KOTB facility to save transportation efforts from staff. An audio

recorder with noise cancellation and microphone is needed to record the proceeding

interview, focus groups and in-class discussions. In-class group discussions with children

will be conducted with assistance of 3-4 facilitators per class depending on class size (one

facilitator per 4-6 children). Location for in-class discussions will be homerooms of

children to ensure comfort level of participants. The NVivo 9 software will be used to

qualitatively analyze information collected on interviews and focus groups. For

centralized data collection, a Microsoft Excel program can collect and organize

information in a user-friendly manner. Surveymonkey or FluidSurveys online survey

program will be used for collection of survey data from educators. FluidSurveys has an

advantage of SPSS data export, a more flexible survey design, along with additional

features that make the survey collection and the distribution process more convenient.

Incentives such as food and refreshments, nametags and pens will be purchased before

each focus group. Scholars choice gift certificates of $25.00 will be given to all

educators who attend the focus group for added incentive.

9. Next Steps


29/58

29

Before the suggested changes and evaluation activities can begin, the Independent

Living Centre and Kids on the Block will need to seek funding from current or new

donors. This evaluation plan requires more staff and resources, and will not likely fit in

the programs regular operating budget.

It is also recommended that the program staff meet with the key stakeholders in

order to seek their feedback and support for this endeavour.

10. Summary & Conclusions

The Kids on the Block program is a valuable experience to expand childrens

knowledge about the lives of persons with disabilities. By raising awareness and

emphasizing the value of embracing our differences, this program fosters a community

inclusive of all persons.

This evaluation plan also has limitations. Firstly, it requires funding and resources

that are not available to the program at the present time. In addition, the evaluation is

not able to gather information from teachers and schools that chose not to host Kids on

the Block. Future evaluations may be able to elucidate this along with other outcome

measures not addressed in this project.


30/58

30

11. References

Education for kids | ILCWR Retrieved 3/27/2012, 2012, from

http://www.ilcwr.org/education-kids

Fox, C., Buchanan-Barrow, E., & Barrett, M. (2010). Children's conceptions of mental

illness: A nave theory approach. British Journal of Developmental Psychology,

28(3), 603-625.

Hazzard, A. P., & Baker, B. L. (1982). Enhancing children's attitudes toward disabled

peers using a multi-media intervention. Journal of Applied Developmental

Psychology, 3(3), 247-262.

Hein, S., Grumm, M., & Fingerle, M. (2011). Is contact with people with disabilities a

guarantee for positive implicit and explicit attitudes? European Journal of Special

Needs Education, 26(4), 509-522.

Hutzler, Y., Fliess-Douer, O., Avraham, A., Reiter, S., & Talmor, R. (2007). Effects ofshort-term awareness interventions on children's attitudes toward peers with a

disability. International Journal of Rehabilitation Research, 30(2), 159-161.

Ison, N., McIntyre, S., Rothery, S., Smithers-Sheedy, H., Goldsmith, S., Parsonage, S., &

Foy, L. (2010). "Just like you": A disability awareness programme for children that
http://www.ilcwr.org/education-kidshttp://www.ilcwr.org/education-kidshttp://www.ilcwr.org/education-kids


31/58

31

enhanced knowledge, attitudes and acceptance: Pilot study findings. Developmental

Neurorehabilitation, 13(5), 360-368.

Obrusnikova, I., Dillon, S. R., Block, M. E., & Davis, T. D. (2012). Validation of the

children's beliefs and intentions to play with peers with disabilities in middle school

physical education scale. Journal of Developmental and Physical Disabilities,

24(1), 35-51.

Peskin, J., & Wells-Jopling, R. (2012). Fostering symbolic interpretation during

adolescence. Journal of Applied Developmental Psychology, 33(1), 13-23.

Tavares, W. (2011). An evaluation of the kids are kids disability awareness program:

Increasing social inclusion among children with physical disabilities. Journal ofSocial Work in Disability and Rehabilitation, 10(1), 25-35.

Independent Living Centre of Waterloo Region. (2012). Our history. Retrieved from

Enhanced knowledge, attitudes and acceptance: Pilot study findings.Developmental

Neurorehabilitation, 13(5), 360-368.

Myers, A.M. (1999). Program evaluation for exercise leaders. Windsor, ON: Human

Kinetics.


32/58

32

12. Appendices

Appendix A: Kids on the Block Interview

Interview for Kids on the Block from January 30th

, 2012

What is the Kids on the Block Program? A troop of disabled and non-disabled puppets thateducate children on disabilities. Targeted towards: community events, schools, churches, camps

When are how was the program started? Established in 1982 in Waterloo Region

Has program changed at all? Originally from the United States but few changes have been

made as the scripts are copyright

Are there any program differences between Canada and United States?

The United States expanded the program to include more topics

Waterloo Region only has scripts to cover disability and diseases (15 scripts).

Must follow scripts completely which has caused some problems as Canada and the US

use different terminology, especially when referring to the disabled e.g. Retarded is not a

word used in Canada

Who is the pioneer of the program? A teacher named Aiello

Where is the program available?The program is available worldwide and in multiple languages

What are the criteria to offer the program?

Request goes into program

Performance dates depends on availability of volunteers

Free program but accepts donations (cannot charge for program, often do not receive

donation)

How are topics chosen for presentations?Certain presentations are tailored for certain age

groups; however, specific topic requests can be made

Do you receive funding from other organizations?o Funding: independent living Canada, United Way, region of waterloo( 2 grants: one for

kids on the block, one for community support services), donations

What do presentations include?o General introduction to disability

o Discuss feelings and experiences of normal kids and kids with disability

o Children can ask puppets question

o 45 minute puppet show

How many shows are conducted annually?~71 shows to over 2000 children

Capacity and setting for each show? 30 students is ideal but will accommodate any size and

classroom styles venues are best to allow interaction between the puppets and students

Is there anything other than the show that is given?o Students receive coloring books and have the opportunity to send a letter to a puppet

asking a question (the puppets send replies)

What are the challenges associated with the shows?o Maneuvering the puppets, learning scripts and learning about the disabilities

o Limited staff (mainly run by volunteers)

o Limited availability of volunteers

Is there a waitlist to see shows? No, usually a range of dates are given and are most oftenaccommodated

Has the program received a lot of media attention? Yes; however, copyright issues is a

concernHow is the program promoted?Word of mouth, mail outs, pamphlets

How does the community react to the shows? Very informative and entertaining


33/58

33

Who is the target audience? Children from the age of 8-11; however, most ages accommodated

(from preschool children to grades 7-8)

How many volunteers and staff are there currently in Waterloo?4-5 volunteers and a part-

time staff member

How are volunteers recruited? Volunteers are interviewed to become puppeteers

Requirements: speak clearly, physically able to hold up and maneuver puppets, schedules, trials

of puppets in practicesHow to train volunteers?

o Train individuals on how to lip synch, holding the body, arm movements and learn script

o Send puppets home with volunteers

o Bring volunteers along in presentations

o Start with small roles before going on to more difficult roles

What are the current record keeping practices?o Spreadsheet with information on shows (includes Contact name, email, date, address of

school, topics, age of kids, times, volunteer/employee names doing the show, number of

people reached at show)

o An evaluation questionnaire is also given to teachers

Who is program accountable to?

o United way report, apply to region of waterloo yearly for fundingo Part of Independent Living Centre report

Has there ever been an evaluation before? No

What would you like to know from an evaluation?o Are we making a difference?

o Is there a shift in attitude towards disabilities?


34/58

34

Appendix B: Kids on the Block Stakeholder Relationships


35/58

35

Appendix C

Appendix C1Kids on the Block Logic Model

C


36/58

36

Appendix C2 - Current Teacher Feedback Form

C


37/58

37

Appendix C3 - Improved Feedback Form

Thank you for completing in our feedback form. Please answer the following by selecting

the answer that best applies. Please do not write your name anywhere on the

questionnaire. Please do not feel obligated to answer all questions.

Presentation date: _____________ Topics covered: ____________________________

1. Was this the first time you have seen a Kids on the Block play?

Yes No

If no, how many other plays have you seen? __________

2. How did you first hear of the Kids on the Block program?

Television Newspaper

Radio

Internet

Word of Mouth Other Please Specify _________________

3. Was the play on time/schedule? Yes No

Comment:_______________________________________________________________

________________________________________________________________________

4. Do you feel the play was well rehearsed?

Yes

No

Comment:_______________________________________________________________________________________________________________________________________

5. In your opinion, do you believe the topic was appropriate for the group of children?

Very Appropriate

Appropriate Not Appropriate

Completely Inappropriate

Comment:_______________________________________________________________________________________________________________________________________

6. In your opinion, do you believe the information in the play was up-to-date?

Yes

No

Comment:_______________________________________________________________

________________________________________________________________________

7. In your opinion, do you believe the information in the play was realistic?

C


38/58

38

Very Realistic

Realistic Not Realistic

Comment:_______________________________________________________________

________________________________________________________________________

8. In your opinion, do you believe these shows are effective at encouraging positive

attitudes towards disabilities?

Very Effective Effective

Not Effective

Completely IneffectiveComment:_______________________________________________________________

________________________________________________________________________

9. Will you be able to use the information provided in the play to further encourage

positive attitudes towards disabilities? Yes

NoComment:_______________________________________________________________

________________________________________________________________________

10. Overall, were you satisfied with the play?

Very Satisfied

Satisfied

Not Satisfied Completely Dissatisfied

Comment:_______________________________________________________________

________________________________________________________________________

11. Would you recommend the play to other colleagues?

Yes

NoComment:_______________________________________________________________

________________________________________________________________________

12. If you have any additional comments or suggestions please feel free to provide them

in this section.

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________


39/58

39

Appendix D

Appendix D1Operational Definitions

The nave theory approach to childrens cognitive development postulates that

childrens understanding process makes use of construction of nave or intuitivetransient theories, which changes with integration of new information with a more

refined theory over time (Fox et al., 2010).

Leventhals five component model involves identity, cause, consequence,

curability and timeline (Fox et al., 2010).

Chedoke McMaster attitudes towards children with handicaps scale (CATCH)

was developed to evaluate the effectiveness of Programs designed to promote

positive attitudes in children towards their disabled peers (Tavares, 2012).

The Social Interaction Questionnaire was developed in conjunction with key

stakeholders for the Kids are kids program and a neuropsychologist and had very

simple yes-no questions to understand the childrens social life (Tavares, 2012).

D


40/58

40

Appendix D2 - Literature Review Tables

Kids are Kids

Author(s), year Tavares, 2011

Objective of study Determine the effect of the program on the attitudes ofchildren towards their peers with physical disabilities

Sample Intervention group n = 40

Comparison group n = 11

Intervention Measure affect, behavior and cognition via questionnaire

administration in a quasi-experimental group design

Tools Chedoke McMaster Attitudes Towards Children with

Handicaps Scale (CATCH)

Social Interaction Questionnaire

Summary of findings Program had a positive impact on childrens attitudes andincreased social inclusion

Just Like You

Author(s), year Ison et al., 2010

Objective of study Evaluate short term effectiveness of a brief disability

awareness package for fifth grade students

Sample n = 147 students from 9-11 years

Intervention Pre-post mixed methods evaluation

Tools Customized questionnaire for pilot programme, and focus

groups with the children

Summary of findings Significant improvement in attitude, knowledge and

acceptance of people with disabilities in the short-term

D


41/58

41

Childrens conception of mental illness

Author(s), year Fox et al., 2010

Objective of study Investigate childrens conceptions of mental illness, in verysimple terms

Sample n = 36 5 -11 year olds

Intervention Mixed factorial research design

Tools Focus groups structured around Leventals five componentmodel and individual interview and task participation

Summary of findings Younger children tended to rely on their knowledge ofcommon physical illnesses when thinking about mental

illnesses. Older children demonstrated differences in their

thinking about mental and physical illnesses.

Is contact with people with disabilities a guarantee for positive implicit and explicit

attitudes?

Author(s), year Hein et al., 2011

Objective of study Development of Implicit Assessment Test, assessing

explicit attitudes

Sample n = 47 undergraduate students without disabilities (11

males, 36 females). 20-32 years old

Intervention Qualitative experiment/quasi-experiment

Tools IAT Scores via indirect measurement and

Multidimensional Attitudes towards Disabled Persons

Scale (MAS)

Summary of findings While explicit attitudes were related to amount and quality

of contact with the disabled, implicit attitudes didnt


42/58

42

Appendix E: Evaluation design:


43/58

43

Appendix F: Focus Group Materials

F1: Teacher Recruitment Letter

Dear Teachers,

The Independent Living Centre of Waterloo Region along with the University of

Waterloo is looking for teachers to participate in discussion groups to help evaluate the

current processes of the Kids on the Block program. We are looking for teachers who

have previously seen a play put on by the Kids on the Block.

The Kids on the Block program needs your help to assist in identifying their strengths and

weaknesses in regards to their current procedures. These focus groups will concentrate on

the marketing of the program, the available topics, your thoughts on the program as well

as student reactions to the program.

The discussion groups will be held during the month of October 2012. They will be heldat the Independent Living Centre head office. Multiple evenings are available and

refreshments will be served

For more information or to sign-up for a discussion group please contact the research

team at (519) 888-4567

Thank you in advance,

The research team

F


44/58

44

F2: Participating Childrens Recruitment LetterDear Teacher,

The Independent Living Centre of Waterloo Region along with the University ofWaterloo is looking for teachers to participate in discussion groups to help evaluate the

current processes of the Kids on the Block program. We are looking for teachers whos

home classes have previously seen a play put on by the Kids on the Block.

The Kids on the Block program needs your help to assist in identifying their strengths andweaknesses in regards to their current procedures. These focus groups will concentrate on

attitudes towards disability and any thoughts on how interactions are carried out in a

naturalistic setting. Parental consent forms are also attached for your convenience.

The discussion groups will be held during October 15th

to November 15th

, 2012. They

will be held in your home classroom with the assistance of facilitators.


team at (519) 888-4567


The research team

F3: Non-participating Childrens Recruitment Letter

Dear Teacher,

The Independent Living Centre of Waterloo Region along with the University ofWaterloo is looking for teachers to participate in discussion groups to help evaluate the

current processes of the Kids on the Block program. We are looking for teachers whoshome classes havent previously seen a play put on by the Kids on the Block.

The Kids on the Block program needs your help to assist in identifying their strengths and

weaknesses in regards to their current procedures. These focus groups will concentrate on

attitudes towards disability and any thoughts on how interactions are carried out in a

naturalistic setting.


to November 15th

, 2012. They

will be held in your home classroom with the assistance of facilitators.


team at (519) 888-4567


The research team

F

F


45/58

45

F4: Parental Information Letter & Consent Form

Information Letter for Parents/Guardians

Dear Parents/Guardians,

The Independent Living Centre of Waterloo Region along with the University ofWaterloo is looking for students to participate in discussion groups to help evaluate the

Kids on the Block programs impact on the audience members. We are looking forstudents who will be watching a Kids on the Block puppet show at school.

The Kids on the Block program needs your childs opinion and feedback on the contentsshown and impact on their perception of people with disabilities. These discussion groups

will concentrate on student reactions to the program, and any changed perceptions on

disabilities.


to November 15th

, 2012. They

will be held in your childs classroom with the assistance of their teacher and facilitators.

For more information please contact the research team at (519) 888-4567


The research team

Parental Consent form for Group Discussion

I understand that I am agreeing for my child to take part in a 30 minute discussion groupwith 4-6 other students to explore the Kids on the Block puppet show regarding people

with disabilities. My child may choose when, and if, to make comments during the

discussion. My childs participation is completely voluntary. The discussion will beassisted by their teacher and a facilitator.

All contact information will be protected under strict guidelines. The information fromthe discussion and general descriptions collected will be summarized to help an

evaluation of the Kids on the Block Program.

Consent Form

You have explained the purpose of this project to my satisfaction, and I have had the

opportunity to ask questions. I will receive a copy of this consent form. If I have any

questions or concerns arising from my Childs participation, I should feel free to contact

our facilitator at (519) 888-4567.

Childs name ( please print)________

Parent or Guardian name ( please print) ________________

Parent or Guardian signature_______________

Date _________

F


46/58

46

Parental Consent form for audiotaping

We need the permission of all participants to audiotape the discussion session. The

purpose for audiotaping is so we do not miss valuable comments people make. Therecordings allow us to analyze the data more completely and we may cite specific

comments made in a process evaluation report. Individual comments will not be

identified. We will keep the tapes secure and destroy them following the analysis. Do you

have any questions?

Consent Form

You have explained the reasons for audiotaping the discussion session to my satisfaction.I understand everyone in the group has to agree before audio taping will take place. By

signing below, I give consent for my childs comments foraudio taping.

Childs name (please print)________

Parent or Guardian name ( please print) ________________

Parent or Guardian signature____________

Date ___________


47/58

47

Appendix G: Focus group materials

Appendix G1: Consent Forms for Educators/Teachers

Consent for discussion

I understand that I am agreeing to take part in a 1-hour focus group discussion with 6-8

other educators to explore the Kids on the Block presentation. I may choose when, and if,to make comments during the discussion. My participation is completely voluntary.

All contact information will be protected under strict guidelines. The information fromthe discussion and general descriptions collected will be summarized to help the

evaluation of the Kids on the Block Program. Do you have any questions?

You have explained the purpose of this project to my satisfaction, and I have had the

opportunity to ask questions. I will receive a copy of this consent form. If I have anyquestions or concerns arising from my participation, I should feel free to contact (519)

888-4567.

_____________________________

NAME (please print)

_____________________________ ______________________________

SIGNATURE DATE

Consent form for audiotaping

Permission is required of all participants to audiotape the discussion session. The purpose

of audiotaping is to ensure no valuable comments are missed. The recordings will allowgreater analysis of the data and give a more complete data. Direct citations may be used

in the final report. Individual comments will not be identified. The tapes will be kept in a

secure location and destroyed after use. Do you have any questions?

Consent Form

You have explained the reasons for audiotaping the discussion session to my satisfaction.

I understand everyone in the group has to agree before audiotaping will take place. By

signing below, I give consent to audiotaping.

_____________________________

NAME (please print)

_____________________________ ______________________________

SIGNATURE DATE

G


48/58

48

Consent for Follow-Up Interview

Thank you for taking part in the discussion group today. The information you provided

was very helpful. When the data will be analyzed, major themes and issues from each

group will be identified and summarized across groups.

To ensure that we have accurately captured the primary issues that emerged from eachsession, we would like to contact a few participants in each group two to four weeks from

now. We will conduct a 15-minute follow-up by telephone to: (1) verify the findings, see

if participants agree with the major points we felt emerged from the discussion; and (2)

see whether participants have any further thoughts to add.

Giving your permission allows us to contact you by phone two to four weeks from now.

If we randomly select your consent form, one of the facilitators you have met today will

contact you. If you still agree at that time, we will set up a convenient time for the 15-

minute conversation. We may not contact you at all, and we certainly would only contactyou once. If you do not wish to participate or participation is inconvenient, no further

calls will take place. We will keep these forms secure and destroy them once we havecontacted you or we determine we have a sufficient number for respondents. We will notgive your name and number to anyone or use it for any purpose apart from this project.

Do you have any questions?

You have explained the purpose of this follow-up to my satisfaction, and I have had the

opportunity to ask questions. By signing below, I give my permission for one of the

facilitators I have met today to call me at the number below to arrange a 15-minutefollow-up by phone. I understand that I may not agree at that time to the interview.

Should I decline, you will make no further contact and will destroy this form.

_____________________________ _____________________________

NAME (please [print) PHONE NUMBER

_____________________________ ______________________________SIGNATURE DATE

_____________________________

WITNESS NAME (please print)

_____________________________ ______________________________

WITNESS SIGNATURE DATE


49/58


50/58

50

7. In your opinion, do you believe the information in the play(s) was realistic? Very Realistic

Realistic

Not Realistic

Comment:________________________________________________________________________________________________________________________

8. In your opinion, do you believe these shows are effective at encouraging positiveattitudes towards disabilities?

Very Effective

Effective Not Effective

Completely Ineffective

Comment:_______________________________________________________________

_________________________________________________________

9. Will you be able to use the information provided in the play(s) to further encourage

positive attitudes towards disabilities? Yes

No

Comment:________________________________________________________________________________________________________________________

10. Overall, were you satisfied with the play(s)?

Very Satisfied Satisfied

Not Satisfied

Completely Dissatisfied

Comment:________________________________________________________________________________________________________________________

11. Would you recommend the play to other colleagues? Yes

No

Comment:________________________________________________________________________________________________________________________

12. If you have any additional comments or suggestions please feel free to provide them

in this section.________________________________________________________________________

________________________________________________________________________

________________________________________________________________________________________________________________________________________________

________________________________________________________________________

Thank you for completing this questionnaire.


51/58

51

G3: Focus Group Protocol

Phase Specifics

Preparation 1.

Confirmation calls to educators 48 hours ahead ofscheduled focus group session2. Make sure all required consent form and documents are

made

3. Ensure refreshments and name tags are purchased, andcharge voice recorder ( or ensure batteries are full)

Set-up 1. Arrive 30 minutes ahead of scheduled session and set uproom, refreshments, name tags, tape recorder, consent

forms and any other materials

Welcoming 1. Facilitator and assistant meet and greet participants

2. Allow 15-20 minutes for conversation and refreshments

3.

Give everyone a name tag

Beginning the

session ( everyone is

seated)

1. Facilitator makes formal introductions, explains focus

group purpose, and explain roles of facilitator and

assistant2. Consent forms for participation and audiotaping are

handed out

3. Assistant records seating plan, and turns on audio-

recorder when consent forms are signed

Setting the ground

rules

1. Facilitator establishes group rules of focus group sessionfor participants. Any questions regarding rules may be

addressed at this time.

Administering

background

form(optional)

1. Facilitator gives participants focus group questionnaire to

complete. Provide pens and pencils as necessary.

Following the script 1. Facilitator begins with icebreaker question, followed by

transitional questions, key questions, and summary

questions, being aware to name every speaker at thebeginning of his or her comment for the assistant

recording..

Closing and getting

follow-up consent

1. Facilitator summarizes main points of discussion and

allows for additions from participants2. Follow-up telephone consent forms are explained and

handed out3. Facilitator thanks participants and gives out Scholars

Choice gift cards

G


52/58

52

G4: Focus Group Script

Sample Educators

Purpose of evaluation project To assess current presentation, registration,scheduling and contact methods that Kids on the

Block utilizes

Icebreaker

Round-robin questions,

everyone answers

Tell us what you love most about working with

children.

Introductory question

To introduce the topic

How did you hear about Kids on the Block?

Transition question

To move the conversation to

key questions

What was the reason that you wanted your studentsto attend the presentation?

Key questions

In-depth examination

What aspects of the program do you enjoy the

most?

Are the topic contents appropriate for children

attending? If so, any improvements? If not, explain

which sections are unsatisfactory.

To which extent does the puppet show capturerealistic aspect of interacting with someone that has

a disability? Is the show effective at encouraging a

positive attitude towards disabilities?

Would you invite the presenters back next year?

Would you recommend this program to othereducators? If so what aspect of the program aided

your decision?

What do you dislike about the current program?

This could be regarding the registration process,scheduling, the presentation or contact methods.

Any recommendations for improvement?

What was the feedback (if any), from children that

attended the presentation?

Summary question The main points raised today are.. Do you

G


53/58

53

agree/disagree? Is there anything you would like to

add?

Appendix H: In-Class discussion Materials

Appendix H1: Questions for Ages 6-8 Years

H1: In-Class Interview

Questions for age 6-8 years

Sample

Educators

Purpose of evaluation project To assess current understanding and perception of

children regarding Kids on the Block puppet show.

Icebreaker

Round-robin questions,

everyone answers

Who is your favorite puppet?

Introductory question

To introduce the topic

What did you like most about the puppet show?

Transition question

To move the conversation to

key questions

Do you know of anyone with a disability?

Key questions

In-depth examination

From watching the puppet show, what do you think

the word disability means?

What did you think before the show about people

with __________ (disability, depends upon topic

coverage of presentation)?

After watching the show, what do you think aboutpeople with __________ (disability, depends upon

topic coverage of presentation)?

What did you like the least about the puppet show?

If you could ask the puppets questions, what would

they be? ( give children an opportunity to write to

puppets via the Kids on the Block Penpal program)

Summary question The main points raised today are.. Do youagree/disagree? Is there anything you would like toadd?

H


54/58

54

H2: In-Class Discussion Group Protocol

Phase Specifics

Preparation 1. Confirmation calls to home room teachers that are

facilitating discussion groups are made 48 hours ahead of

scheduled Kids on the Block presentation.2. Release discussion questions and protocol to teachers so

they may prepare in advance 2 weeks prior to scheduled

discussion group and allow questions from teachers.3. Make sure all required consent form and documents are

given out to parents 2 weeks before, signed and returned

by parent/guardian a week before the presentation.4. Name tags are purchased, and charge voice recorder ( or

ensure batteries are full)

Set-up 1. Arrive 30 minutes ahead of scheduled session to sign intoschool as a guest visitor and set up room, name tags, tape

recorder, and any other materials.Welcoming 1. Facilitator and assistant meet and greet participants

2. Give everyone a name tag

Beginning the

session ( everyone is

seated)

1. Facilitator makes formal introductions, explains focusgroup purpose, and explain roles of facilitator and

assistant to children

2. Assistant records seating plan, and turns on audio-recorder when facilitator begins

Setting the ground

rules

1. Facilitator establishes group rules of focus group session

for participants. Any questions regarding rules may be

addressed at this time.Following the script 1. Facilitator begins with icebreaker question, with

icebreaker question, followed by transitional questions,

key questions, and summary questions, being aware to

name every speaker at the beginning of his or hercomment for the assistant recording.

Closing and getting

follow-up consent

2. Facilitator summarizes main points of discussion and

allows for additions from participants

3. Facilitator thanks participants and gives participants

option to write a letter to the Penpal program

Appendix I: Staff Interview Script

About The Show

H


55/58

55

To whom do you offer the program?

Are there strict criteria to be able to qualify for a show?

What is included with the presentation?

What topics are covered?

How long does a puppet show (on average)?

What materials are needed for the puppet show?

How do staff/volunteer travel to the show?

How often do you perform?

During what time of year are you busiest?

What is the capacity of the program?

Is there a limit in students? Is there a limit in how many shows you can perform?

Is there a waitlist for shows?

How is the show promoted?

Do you receive feedback from your plays? (eg. Satisfaction survey)

If so, how do you receive this feedback

What is done with the information received from feedback forms?

Procedures at the Office

How is the program funded?

How often are donations received?

Are these donations noted in a database solely for donations received for the Kidson the Black program?

What information is collected for each show?

How is this information collected?

Who collects this information?

Where is the information stored?

For how long is this information kept?

Do you have a list of contacts to promote the show?

If so, where is this information found?

How often is the list updated

To whom is the Kids on the Block program accountable?

Funders?


56/58


57/58

57

To Whom It May Concern:

The Independent Living Centre of Waterloo Region (ILCWR) along with the

University of Waterloo wishes to conduct a program evaluation on the Kids on theBlock. This community outreach program is a part of the ILCWRs objective to increaseawareness and acceptance of persons with disabilities. Particular, the Kids on the Block

Program offers puppet presentations aimed at educating children about physical and

learning disabilities.

The program has been in operation, however, to assess the impact of theirperformances on childrens attitudes, we require consent from the Waterloo RegionDistrict School Board. The evaluation team is hoping to include educators and children

who have seen the puppet presentations in the evaluation of their program. The duration

of the evaluation is five

program evaluation for kids on the block

Documents