rising sun evaluation report

Rising Sun Energy Center California

Youth Energy Services Evaluation

Report

July 2014

Prepared For Rising Sun Energy Center

Prepared By Learning for Action

Learning for Action enhances the impact and sustainability of social sector organizations through highly customized research, strategy development, and evaluation services.

About Learning for Action

Established in 2000, and with offices in San Francisco and Seattle, Learning for Action (LFA) provides highly customized research, strategy, and evaluation services that enhance the impact and sustainability of social sector organizations across the U.S. and beyond. LFA’s technical expertise and community-based experience ensure that the insights and information we deliver to nonprofits, foundations, and public agencies can be put directly into action. In the consulting process, we build organizational capacity, not dependence. We engage deeply with organizations as partners, facilitating processes to draw on strengths, while also providing expert guidance. Our high-quality services are accessible to the full spectrum of social sector organizations, from grassroots community-based efforts to large-scale national and international foundations and initiatives.

Learning for Action 170 Capp Street Suite C San Francisco, CA 94110

(415) 392-2850 www.learningforaction.com

About Rising Sun

Rising Sun Energy Center (Rising Sun) got its start in Santa Cruz in 1994 as a renewable energy education center. Today, it is a leading nonprofit green workforce development, environmental education, and residential energy efficiency services organization serving communities in Alameda, Contra Costa, Marin, and Solano Counties. Headquartered in Berkeley, Rising Sun’s mission is to empower individuals to achieve environmental and economic sustainability for themselves and their communities.

Rising Sun Energy Center 2998 San Pablo Avenue

Berkeley, CA 94702 (510) 665-1501

http://www.risingsunenergy.org

http://www.learningforaction.com/

Table of Contents

Executive Summary .............................................................................................. 1

About CYES 3

About LIFT 3

How does it work? 3

CYES Participants 5

About the Alumni 7

How CYES Supports Gains in Knowledge and Skills ........................................... 8

What Youth Say about Increases in Green Knowledge 8

What Youth Say about Increases in General Workplace Skills 9

How CYES Supports Youth in their Education and Employment Goals ............. 12

Educational and Employment Plans and Outcomes 12

Influence of CYES on Post-program Plans and Outcomes 15

How Learning Knowledge and Skills at CYES Supported Alumni in the Job Market 16

Feedback from Participants on the CYES Program ........................................... 18

Feedback on Training 18

Feedback about Rising Sun Staff 19

The Influence of the LIFT Program ..................................................................... 21

Increases in Green Knowledge, General Workplace Skills, and Leadership

Competencies 21

How Learning General Workplace Skills Supported LIFT Alumni in the Job Market 23

Post-Program Decisions for LIFT Participants 24

Methodological Appendix ................................................................................... 25

Data Collection 25

Analysis 25

Strengths of the Methods 26

Limitations the Methods 26

“It gave me some insight as to how some small changes could still have a big impact on our environmental footprint. CYES also gave me a nice experience for a first job.”

“It was a great summer job to have. I learned a lot and got a lot of experience. I'm naturally not inclined to reach out and talk to strangers, but this job really pushed me out of my comfort zone. I

now feel like I have the confidence to talk with strangers. I look forward to using this new confidence in the future.”

“The CYES program allowed me to promote a sustainable lifestyle to residents in my local community at no cost at all. I enjoyed the aspect of being able to help others through basic green

methods. This program influenced my decision to major and minor in the sustainable field.”

“Team members were all very friendly and easy to work with. It's a great program that helps the community out.”

“I thought it was really fun and really looked forward to work every morning. My favorite part was talking to the clients who were really excited about the program!”

“I've gained knowledge about energy. Lots of tricks and tips that have helped me and my family. Also, I met a ton of awesome people… [CYES is a] great program that gives youth experience so

that we can go out and thrive.”

“I was proud to say I worked with Rising Sun and I'm very proud of the work we've done this summer.”

“I love this program. It is a great way to give back to the community and our environment.”

“I have educated my friends, family, and roommates on water and energy conservation since leaving CYES. Sharing this information with them was important…by telling them of these easy methods, they can thus tell others as well so awareness is continuously spreading through our

community.”

“[What I learned through CYES] is important to me because everybody needs to know what will help them and our environment, because if we don’t take care of our environment, we won’t exist.

It’s valuable information that everybody needs to know.”

“CYES allowed me to become a better team player.”

“I think this program is a really good idea, actually does help 95% of the people we visit, and it's the best summer job available to most young people.”

Rising Sun Energy Center | Learning for Action | July 2014 1

Executive Summary

The mission of Rising Sun Energy Center (Rising Sun) is to “empower individuals to achieve environmental and economic sustainability for themselves and their communities.” Central to its work are green workforce development programs for youth, in which youth offer residential energy efficiency services to households in Alameda, Contra Costa, Marin, and Solano counties. There are two youth workforce programs, California Youth Energy Services (CYES), and Leaders in Field Training (LIFT).

California Youth Energy Services. CYES, established in 2000, is a summer workforce development and green energy program for youth. Participants receive training, coaching, and mentorship to support the development of technical and professional skills, as well as opportunities to apply those skills in the field. Working as Energy Specialists, youth conduct Green House Calls for residents of their local communities. At these house calls youth inspect homes, provide information on energy efficiency, and install equipment designed to increase the water and energy efficiency of the home.

Leaders in Field Training. Since 2009, Rising Sun has offered the LIFT program to continuing CYES participants. As part of LIFT, youth have the opportunity to gain additional advanced training and to further develop their leadership skills. In addition to six full days of pre-program training, LIFT participants attend weekly training during the summer on topics such as supervision, project planning, facilitation, and leadership. LIFT participants serve as assistant managers at their CYES sites and as peer mentors to CYES Energy Specialists, leading weekly professional development workshops.

In 2014, Rising Sun engaged Learning for Action (LFA), an independent evaluation company, to evaluate their CYES and LIFT programs. The evaluation addressed the following questions:

What are the school and career plans of program participants, and what school and career goals do CYES and LIFT participants pursue after they leave the program?

To what extent does program participation influence their plans and the choices they make after leaving the program?

To what extent do CYES and LIFT result in gains in green industry-related knowledge and general workplace skills?

How does growth in skills and knowledge support former participants in the job market? What feedback do participants provide about the CYES program – in particular, training and staff

support? To answer these questions, LFA used three data sources:

A pre/post survey conducted at the beginning and end of the CYES program in the summer of 2013,

An alumni survey sent to youth who had participated in the CYES or LIFT program in the summers of 2011, 2012, and 2013, and

A focus group with program alumni. The findings of the evaluation indicate that the programs are very successful:

CYES and LIFT participants report growth in green industry-related knowledge, as well as in general workplace skills, and they attribute this growth to the CYES program.

Most CYES participants attend school and/or work after they complete the program, with those of college age attending college.

Significant percentages of CYES and LIFT participants report that program participation influenced their decisions about what to do after leaving the program, including going to college, staying in high

Where to go for Additional Information on Methods

Please see the Methodological Appendix at the end of this report for detailed information on the methods, including their strengths and limitations.


school, and pursuing a green or socially-related major (although these percentages are lower for alumni than they are for youth at the end of the program).

A majority of LIFT alumni enter leadership positions after leaving the program – and all those who do so say that LIFT influenced their decisions to pursue these roles.

CYES and LIFT participants credit the skills they learned in the program with helping them to find a job or to perform well in a job.

CYES participants provide positive feedback about the program – they appreciate the quality and quantity of trainings, and give high ratings to the program staff.


Program Overview1

About CYES

Since 2000, Rising Sun Energy Center has run CYES, a summer youth employment and community efficiency retrofit program in Northern California’s San Francisco Bay Area. CYES hires young people (15-22 years of age) and trains them to become Energy Specialists, serving their communities with no-cost Green House Calls. Energy Specialists install free energy and water saving devices, and provide personalized recommendations and education for further savings in homes. CYES provides services to all community members regardless of income. However, it was designed to serve hard-to-reach residents including renters, non-English speaking households, and low-moderate income households. It provides youth with opportunities for training and meaningful employment, which are often not adequately available to them. CYES youth receive employability skills training, paid summer employment, and the foundation for a green career. In its 14 years of operation, CYES has trained over 1,000 youth and served over 23,000 Bay Area households.

About LIFT

The LIFT program creates additional professional and leadership development opportunities for former CYES Energy Specialists. LIFT provides CYES alumni with the opportunity to return to the CYES program in a position of increased responsibility, while also developing their leadership skills in a learning environment. Serving as assistant managers, LIFT Energy Specialists learn how to supervise staff, facilitate trainings, and attend to administrative duties, while serving as positive role models for their peers and sharing their indispensable prior program knowledge and experience. LIFT members participate in ongoing weekly training and professional development modules and then, in turn, share their expertise with their peers during weekly workshops.

How does it work?

Each summer, Rising Sun sets up CYES satellite offices in partner cities. Throughout the summer program, CYES Energy Specialists meet each morning at their office and then depart in teams of two to provide Green House Calls to community residents. Residents can contact Rising Sun to schedule an appointment, or if calling in the off-season, to get on the appointment waitlist. Over the past 14 years, Rising Sun has worked to optimize the CYES model and found the following features essential: An outreach campaign is launched 3-6 months in advance of our program start date Appointments are scheduled to maintain professionalism and positive community image Satellite offices are staffed with two Managers and 8-12 youth Energy Specialists 50% of Energy Specialists are over the age of 18 so a legal adult is in the home at all times

What is a Green House Call?

The Green House Call provides renters and homeowners with a no-cost energy and water conservation assessment, education, and mini-retrofit. Renters can receive CYES services because the equipment installed does not permanently alter the infrastructure of a home. In other words, renters do not need special permission from their landlord in order to receive the service - in fact, on average 50% of CYES customers are renters.

1 This section offers additional contextual information about the CYES and LIFT programs, provided by Rising Sun. It is intended to give readers a better sense of the training and activities that program participants rated through the CYES Post-Program and Alumni Surveys.


A CYES Green House Call consists of the following items, all at no cost:

A walkthrough of the house with the client, looking for energy saving opportunities Replacing all incandescent bulbs with energy-saving compact fluorescent and LED versions Replacing inefficient water fixtures with efficient-flow faucet aerators and showerheads Inspecting all toilets for efficiency and leaks Exchanging energy-wasting halogen floor lamps for efficient fluorescent versions Checking the attic insulation of the home Installing a retractable clothesline, powerstrip, or LED nightlight Installing water heater pipe insulation Removing small electronic waste items for recycling Providing energy and water conservation education Emailing the client a report after the appointment with customized recommendations and additional

resources

In addition, Energy Specialists leave a client feedback card to solicit client comments and referrals, a reusable shopping bag, local utility rebate information, and nearby recycling and hazmat information for proper disposal of CFLs and other hazardous materials.

How does Rising Sun recruit and train youth?

The typical CYES summer season is organized into seven weeks: one week of intensive training and six weeks of field work. Youth are recruited locally through various channels. Methods of recruitment include schools, community-based youth organizations, community colleges and universities, word of mouth and other traditional methods of recruitment. Youth applicants go through a rigorous group interview process which assesses the following:

Critical thinking and decision-making skills Ability to work in pairs and teams Presentation skills Commitment to the community and/or environment Desire to transform and gain new skills Rising Sun Energy Center employs a youth employment training model. This model provides young people with job experience in a productive context where they receive coaching and mentoring. Professional skills are the most difficult skills to teach and often do not translate well in a classroom setting. However, in a work setting these skills translate directly. The CYES youth employment training model:

Builds on young people’s strengths and places them in situations where they are given a great deal of responsibility

Empowers youth to act as competent employees, where they can earn their own money and therefore take ownership of their lives

Implements code of conduct policies that mirror standard workplace practices


About the Participants

CYES Participants2 Who are the youth that participate in CYES? This section provides a snapshot of CYES participants based on information they shared at the beginning of the 2013 summer program regarding their demographics, households, educational status, and employment experience.

Demographics

While the CYES program hires youth ages 15 to 22, most survey respondents were between 16 and 20 years old (Exhibit 1). Participants were evenly split between male and female (Exhibit 2). While the program historically attracted more male participants, the gender balance in 2013 reflects an intentional effort on the part of Rising Sun to recruit more female participants.3

Asian/Asian American youth and African/African American youth each represented a quarter of CYES participants, while Euro American/Caucasian youth and Latino/Hispanic youth each represented a fifth of CYES participants (Exhibit 3). The remaining youth identified as mixed (1%) or “other” (8%). Forty-one percent of youth report speaking a language other than English at home (Exhibit 4). These languages include Spanish, Mandarin, Khmu, Tagalog, Gujarati, Tamil, and Telegu.

2 While each CYES group is different, Rising Sun believes that the 2013 CYES participants are generally representative of youth who participate in the program. 3 No participants identified as transgender.

Exhibit 2: Gender

Exhibit 1: Age

Exhibit 4: Speaking a language

other than English at home

Exhibit 3: Ethnicity

Male

50%

Female

50%

n=76

8%

15% 15%

17%

15% 15%

9%

5%

3%

0%

25%

15 16 17 18 19 20 21 22 23

n=76Age of Program Participants

No

59%

Yes

41%

n=76

8%

1%

18%

20%

25%

28%

0% 50%

Other

Mixed

Latino/Hispanic

Euro American/Caucasian

African/African American

Asian/Asian American

n=76


Household and Family

Most participants reported living in a two parent household, while nearly a quarter reported living in a single parent household (Exhibit 5). Most participants (61%) do not have one or more parent with a Bachelor’s degree or higher (Exhibit 6). However, almost all participants (91%) reported that one or more parent has a high school diploma or GED.

Educational Status

The vast majority of CYES participants reported that they were students (Exhibit 7). Of these, 42% attended four-year college programs, 22% attended two-year college programs, and 36% were still in high school (Exhibit 8). Of the 12% of CYES participants who were out of school, two-thirds had earned a high school diploma, and a fifth had earned an Associate’s degree (Exhibit 9).

Exhibit 5: Type of household

Exhibit 6: One or more parents has a

Bachelor’s degree or higher

Exhibit 7: Youths’ student

status at the start of CYES

Exhibit 9: Educational level achieved for youth

who are out of school

Exhibit 8: Educational level for youth

currently in school

72%

23%

6%

n=71

SingleParent

Household

Other

TwoParent

Household

No

61%

Yes

39%

n=76

Student

88%

Out of School

12%

n=76

11%

67%

22%

0%0%

100%

Some High School High SchoolDiploma

Associate'sDegree

Bachelor'sDegree

n=9

36%

22%

42%

0%

50%

High School 2 Year College 4 Year College

n=67


New Employment Experiences Provided by CYES

CYES was a first job for almost half of respondents (Exhibit 10). Looking only at the participants who had previously held a job, for three-quarters of them, CYES was their first green job (Exhibit 11).

About the Alumni

CYES and LIFT alumni who completed the Alumni Survey shared their age and current residence. Of the 81 alumni respondents, ages ranged from 16 to 25 years old, with the majority (67%) falling between 18 and 22 years old. Most alumni (67%) currently live in the Bay Area, followed by Northern California, not including the Bay Area (18%), Southern California (9%), and outside California (6%). The alumni who completed the survey worked in CYES and LIFT in 2011, 2012, and 2013. Exhibit 12 below summarizes the number of participants who worked in each program during those years.

Exhibit 10: Percent of youth for whom

CYES is their first job

Exhibit 11: For those who have previously

held a job, the percent of youth for whom

CYES is their first green job

Exhibit 12: Years worked in CYES and LIFT

Program Participants Who Worked in 2011

Participants Who Worked in 2012

Participants Who Worked in 2013

CYES 23 26 37

LIFT 5 8 7

Yes

46%

No

54%

n=74

Yes

74%

No

26%

n=39


How CYES Supports Gains in Knowledge and Skills

As a workforce development program, CYES aims to support the growth of career-related knowledge and skills. By participating in the program, youth are expected to learn about a variety of issues and topics that we group under the term “green knowledge:” energy efficiency, climate change, etc. By imparting green knowledge, the program experience should help to set youth up for success in green economy jobs. The program also seeks to impart more general skills that are valuable in the workplace: communications, leadership, teamwork, etc. CYES has very positive findings: youth report increases in knowledge and skills, and they credit these increases to their participation in the program.

Key Findings:

What youth say about increases in green knowledge: o At the end of the program, a majority of

participants say that CYES contributed to growth in a variety of green knowledge areas.

o Follow-up data shows even larger majorities of alumni who say that CYES helped build their green knowledge.

What youth say about increases in general workplace skills: o At the end of the program, a majority of

participants say that CYES contributed to growth in several categories of workplace skills.

o Follow-up data shows even larger majorities of alumni who say that participation in CYES helped to build their workplace skills.

What Youth Say about Increases in Green Knowledge

At the end of CYES, most participants expressed that because of CYES, their knowledge had grown on a variety of green issues and topics. Survey results from alumni are also impressive: while we might expect knowledge to fade over time, past program participants also reported a growth in knowledge as a result of program participation.

84% of CYES participants “agree” or “strongly agree” that they attribute their improved skills to their participation in the CYES program.

A Note on Data Used and Terminology

This report uses data from two surveys: a pre/post survey conducted at the beginning and end of the CYES program in the summer of 2013, and an alumni survey sent to youth who had participated in the CYES or LIFT program in the summers of 2011, 2012, and 2013. We use the pre/post survey to report post-program findings (what participants report at the end of a program). We use the alumni survey to report follow-up findings (what participants report at follow-up period after they have left the program).

I didn’t know anything when I started. [My knowledge] increased a whole lot. Everything they taught, it stayed with me because it was so important and I know it’s needed.

CYES Alum


Knowledge Increases Reported at the End of the Program

A primary goal of CYES is to increase participants’ knowledge of a variety of environmental and green industry-related issues. At the end of the program, participants reported that CYES had had a great impact in these areas. Large majorities found that the program considerably increased their knowledge of water conservation (81%) and energy efficiency (80%). Almost two-thirds of participants also reported increases in knowledge about climate change (64%) and building science (60%).

Knowledge Increases Reported by CYES Alumni

It is clear that immediately following CYES, participants are enthusiastic about the knowledge they gained. But are these perceptions similar for those who have been out of the program for one or more years? One to three years after participation in the program, most CYES alumni expressed that CYES had increased their environmental and green economy knowledge in several ways. In fact, majorities of youth provided the highest possible rating, reporting that CYES had helped build their knowledge to a great extent” in Green House Call procedures (78%), water conservation (63%), and energy efficiency (63%). Exhibit 14 shows the percentages of alumni who provided the two highest ratings; this is why the percentages shown are even higher.

What Youth Say about Increases in General Workplace Skills

The pattern we see with growth in green knowledge holds for general workplace skills: majorities of participants say at the end of the program that CYES has contributed to a growth in the skills – and even larger majorities credit CYES with building their skills.

4Survey item on a 5-point scale (1=Not increased at all; 2=Increased very little; 3=Increased somewhat; 4=Increased; and 5=Increased significantly). 5 Survey item on a 5-point scale (1=Not at all; 3=Some extent; and 5=Great extent).

Exhibit 13: Post-program: Increases in green knowledge4

Exhibit 14: Follow-up: Increases in green knowledge5

I have gained more effective communication skills and problem solving. I am more able to think on the spot when it comes to unexpected questions from clients.

CYES Participant

81%

80%

64%

60%

0% 100%

Water Conservation

Energy Efficiency

Climate Change

Building Science

n=80

Percent reporting "increased" or "increased significantly"

96%

92%

89%

0% 100%

Knowledge of Green House CallProcedures

(i.e., home auditing and installations)

Water Conservation

Energy Efficiency

n=73

Percent reporting 4 or 5 on a 5-point scale (5=Great extent)


Increases in General Workplace Skills Reported at the End of the Program

A goal of CYES is to increase general workplace skills among participants. Here again, CYES participants greatly benefited from the program: even though many youth rated their general workplace skills as above average upon entering the program, nearly all reported an increase in skill following CYES. Most participants reported substantial increases in field-based and administrative skills (79%), soft skills (75%), and professionalism (67%).

Among the broad range of skills that youth strengthened during CYES, customer service stood out as the highest rated, with 91% of participants reporting a substantial increase in that skill. Teamwork also stood out as a prominent theme: at the end of CYES, the majority of youth felt that their ability to work effectively with a team had improved substantially (79%), along with related skills such as adaptability (75%) and positive attitude (71%). CYES participants attributed this growth to multiple aspects of the program, including the practice of rotating work partners which exposed youth to many different personalities and work styles, and the positive attitudes modeled by the staff who mentored them throughout the program. Participants expressed that CYES had not only strengthened their interpersonal skills, but had also prepared them for future employment, providing them with valuable experience that they believed would be applicable to the jobs they planned to pursue.

6 Survey item on a 5-point scale (1=Not increased at all; 2=Increased very little; 3=Increased somewhat; 4=Increased; and 5=Increased significantly).

Exhibit 15: Post-program: Increases in general workplace skills6

*Includes customer service, outreach/marketing, presentation skills, and administrative skills. **Includes teamwork, adaptability, and leadership. ***Includes positive attitude, punctuality, professional language, and professional appearance.

[CYES] showed me a lot about what it means to work hard and have a good work ethic.

CYES Participant

CYES exposed me to a plethora of personalities, which helped develop my patience and professionalism in interacting with people.

CYES Participant

67%

75%

79%

0% 100%

Professionalism***

Soft Skills**

Field-Based andAdministrative Skills*

n=80

Percent reporting "increased" or "increased significantly"


Increases in General Workplace Skills Reported by CYES Alumni

Alumni indicted that CYES had helped them to develop a broad range of general workplace skills, including soft skills (81%), field-based and administrative skills (75%) and skills related to professionalism (74%). Among field-based and administrative skills, customer service skills were again an important growth area: 90% of alumni reported substantial increase in skills, and 62% provided the highest rating possible. Particularly striking are the findings on presentation skills (one of the “field-based and administrative skills”): 71% of alumni gave the two highest ratings when reporting the extent of their growth in this skill area. In the focus group and open-ended survey responses, a few youth shared that the CYES program had helped with their confidence, and that after the program, they found themselves better able to speak up and present information clearly to others. In terms of soft skills, the majority of alumni also felt that CYES had strengthened their abilities to work in teams and cooperate with others (82%), which some alumni attributed to the opportunities CYES provided for them to work with many different peers throughout the program. Following CYES, alumni found these skills extremely helpful in seeking and obtaining employment.

7 Survey item on a 5-point scale (1=Not at all; 3=Some extent; and 5=Great extent).

Exhibit 16: Follow-up: Increases in general workplace skills7

*Includes teamwork/cooperation and leadership. **Includes customer service, problem-solving/decision making, presentation skills, and time management. ***Includes communication.

74%

75%

81%

0% 100%

Professionalism***

Field-Based andAdministrative Skills**

Soft Skills*

n=73

Percent reporting 4 or 5 on a5-point scale (5=Great extent)


How CYES Supports Youth in their Education and

Employment Goals

Another goal of the CYES program is to support youth as they pursue their educational and career aspirations. And more specifically, CYES aims to encourage youth to enter fields of study that will position them for a career in the green industry, and also to support youth in finding jobs in the green industry (for those in the job market). The youth who participate in CYES tend to do well in terms of being on a path to strong educational and occupational attainment – and they attribute some of this success to participation in CYES. Alumni also say that the skills they learned in CYES helped them in the job market.

Key Findings:

At the end of the program, all youth plan to stay in (or return to) school, to work, or to pursue school and work at the same time.

Eighty percent of the alumni reported that after the program they had begun or returned to college, and most of the remaining 20% continued high school.

Forty percent of CYES youth reported at the end of the program that CYES had played a role in changing their minds about their plans for the future.

Of the alumni who worked after CYES, large majorities of them credited what they learned at CYES with helping them to land a job or with doing well on the job (80% or more for the different skills; 71% or more for the different areas of knowledge).

Educational and Employment Plans and Outcomes

Following the 2013 CYES program, nearly all participants (97%) planned to begin or return to school, while nearly one third (29%) planned to find jobs. As illustrated in Exhibit 17, the majority planned to go to school without working at the same time, while about a quarter planned to pursue work and school. Just a few participants (3%) planned only to work. The sections below delve into the specifics of the educational and employment plans, as well as what alumni say about their educational and career pursuits in the years since they participated in the program.

CYES Participants’ Educational Plans

Among CYES participants with school-related plans (see Exhibit 18), a third were returning to high school, 40% planned to begin or return to a 4-year college , and a third planned to begin or return to community college (33%). None of the 2013 CYES participants had plans to attend or return to vocational school. More than half of participants starting or continuing at a 4-year college had plans to pursue a green or socially-related major (see Exhibit 19).

Exhibit 17: Post-program: Participants’ education

and employment plans following CYES

3%

26%

71%

0% 100%

Work Only

School and Work

School Only

n=80


Especially exciting is the finding that of the youth with plans to pursue a green or socially-related major, all of them reported that CYES had influenced their choice of major (with 73% identifying CYES as having “strongly influenced” their decision). This finding highlights the ability of the CYES program to guide youth toward educational programs that can lead to careers focused on social good.

CYES Alumni Educational Outcomes

CYES alumni show strong educational attainment outcomes. Following their involvement in Rising Sun programs, most alumni began or returned to college (80%), and over three quarters of these attended four-year programs (77%), with the remaining 23% attending two-year associate’s degree programs. Of those who did not attend college after the program (20% of the alumni), 16% are accounted for by the youth who attended high school (and thus were not old enough for college). Another 1% attended a vocational or job training program, and 3% answered “other” (results not shown). Among the alumni who attended college, most selected majors that were environmentally, socially, or business-related (72%). Of these, environmental majors were the most popular by a slight margin (Exhibit 20).

8 Analysis includes participants beginning or returning to a 4-year college program. 9 Sample excludes 4 respondents who selected multiple options.

Of the youth planning to pursue a green or socially-related major, 100% said that CYES had influenced their decisions.

Exhibit 19: Participants’8 plans to pursue a

green or socially-related major

Exhibit 18: Post-program educational plans

Exhibit 20: Follow-up: CYES alumni college majors9

Yes

53%

No

47%

n=32

0%

0%

12%

12%

21%

28%

33%

0% 50%

Return to vocational school

Begin vocational school

Begin community college

Begin 4-year college

Return to community college

Return to 4-year college

Return to high school

n=78

28%

23%21%

28%

An environmental field (e.g.,biology, environmental science,environmental engineering, etc.)

A socially related field (e.g., socialwelfare, sociology, etc.)

A business related field (e.g.,business, marketing,communications, etc.)

None of the above

n=53


CYES Participants’ Employment Plans

Of the 29% of participants with work-related plans after CYES, the vast majority (83%) planned to search for another job (Exhibit 21). Some of the youth who planned to search for another job were also among the 35% of youth who planned to begin new jobs, which may suggest that while these youth had found employment, they were still looking for additional work, or different work. For example, one participant felt prepared to work for PG&E, and anticipated obtaining such a position in one to two years. As illustrated in Exhibit 22 (below), more than two thirds of CYES participants (71%) planned to pursue jobs in the green economy.

CYES Alumni Employment Outcomes

By looking at the alumni survey results, we can get a sense of how youth do in the job market after participating in CYES programs. Just over half of alumni searched for and were able to find employment following CYES (Exhibit 23).10 More than a quarter (28%) did not search for a job, which most likely relates to the fact that most youth exiting CYES planned to pursue school only (71%, as shown previously in Exhibit 17).11 Of the 33 youth who were able to find employment, only 21% were employed in green economy jobs. And of the 14 youth who searched for jobs but either did not apply or did not find work, 21% had searched for jobs in the green economy. At the end of the 2013 program, CYES participants expressed plans to pursue green jobs (71%).12 The results from alumni, however, tell a different story. When asked about their career goals, only 37% of alumni expressed an interest in a career in the green industry (33% said they did not want a green career, and 30% said they were not sure of future plans). Some of this difference may relate to the availability of green jobs: over time, as alumni are unable to find green jobs, they may develop alternate career paths. Considering the age range of CYES participants (15-25), it is also expected that career preferences may change as they

10 Due to prior skip logic in the survey programming, only 65 respondents (80%) received this question. 11 Based on 2013 CYES post-program survey data. 12 Based on 2013 CYES post-program survey data.

Exhibit 22: Post-program: CYES participants’

plans to work in the green industry

Exhibit 21: Post-program: Employment plans

Exhibit 23: Alumni employment following CYES

Yes

71%

No

29%

n=80

9%

9%

35%

83%

0% 100%

Begin a training program

Begin an apprenticeship

Begin a new job

Search for another job

n=23

51%

28%

14%

8%

Searched for ajob and got a job

Did not searchfor a job

Searched for ajob and did notget a job

Searched for ajob but did notapply

n=65


progress through high school and college.13

Influence of CYES on Post-program Plans and Outcomes

To what extent did participation in CYES change youths’ plans following the program? For most participants (60%), CYES did not change their plans (Exhibit 24). However, this finding may not be surprising considering that over half of youth in the program were 18 years old or younger (55%) and had likely made decisions to continue high school, begin college, or continue college prior to beginning the CYES summer program. With this in mind, the fact that 40% of participants did change their plans based on CYES is actually impressive. The ways in which CYES changed these participants’ plans are illustrated in Exhibit 25 (below). Furthermore, CYES participants’ comments suggest that even for youth who did not change their plans, involvement in CYES had reinforced their commitment to the green economy and environmental conservation.

13 In addition, some of the difference may be an artifact of the different ways that the questions were asked on the two surveys. On the post-program survey, there were only two response options, and “I am not certain of my future plans” was not included as a third response option.

Exhibit 25: Post-program: How participants’ plans

changed as a result of CYES

Exhibit 24: Post-program: Percentage of

youth who changed future plans

because of CYES

15%

8%

12%

15%

15%

19%

23%

58%

0% 75%

Other

Attend a vocationalor technical training program

Attend college

Focus on a socially related field in college(e.g., social welfare, sociology)

Focus on a business related field in college(e.g., business, marketing, communications)

Attend a green-related training program(e.g., GETS, Conservation Corps,

RichmondBUILD)

Focus on an environmental field in college(e.g., biology, environmental science,

environmental engineering, etc.)

Search for another job in the green field

n=26

I am now planning to:

Yes

40%

No

60%

n=65


Exhibit 26 shows how alumni see the influence of CYES on a variety of decisions. Among those who decided to get a job, 62% said that CYES influenced their decisions. Of those who attended college, almost one third said that CYES influenced their decision. This is a dramatic finding, given how critical college is for future life chances. The finding about the influence of CYES on the decision to continue high school might seem even more dramatic (since 42% reported that CYES had an influence), but the total number of youth answering this question is quite small (12), so it is risky to generalize from this group. Finally, only about one fifth said that CYES had influenced their decision on what to major in. This finding provides a strong contrast to the post-program survey results, in which participants planning to pursue a green or socially-related major all said that CYES had influenced their decisions. This difference between those who are just finishing the program and those who have been out of the program for a few years may simply reflect the fact that with the passage of time, participants have had many new experiences that have influenced their decisions. One participant explained that although he had already been planning to attend school following CYES, his experience in CYES had influenced him to change his major, because he had seen the impact of his Green House Call work on his family and others in his community, inspiring him to work toward a career in energy efficiency. Another youth explained that while she had learned a great deal from CYES, she did not feel it was responsible for her decisions following the program. Again, given the focus of the CYES program, it is likely that many youth who engage in the program are already interested in pursuing education and careers in environmentally, socially, or business related fields.

How Learning Knowledge and Skills at CYES Supported Alumni in the Job

Market

In assessing the ways that CYES can support youth in their school and career goals, this last section explores the extent to which alumni have found those knowledge and skills to help them in the job market. To what extent did learning “green knowledge” and general workplace skills help alumni to either get hired for a job, or do well in a job?


Exhibit 26: Follow-up: Influence of CYES on

post-program decisions14

[CYES] helped me realize that this is the career path I want to take because of how much I love outreaching, presenting, and working with community members in regards to the green movement.

CYES Participant

62%

42%

31%

21%

0% 100%

Search for or get a job (n=45)

Continue high school (n=12)

Attend college (n=52)

Pursue an environmentaly, socially, orbusiness related major (n=39)



The Job Market Utility of the Green Knowledge Participants Learn in CYES

Many CYES alumni indicated that green knowledge (related to energy efficiency, water conservation, and how to conduct Green House Calls) had helped them to land or perform well at their jobs. This finding is slightly unexpected given that only 21% of employed alumni reported that they work in the green economy. This suggests that even when alumni are not working in green jobs, they are still putting to good use the knowledge they gained through CYES. This green knowledge may be transferable to other types of jobs – or perhaps learning about these topics helps to set up CYES alumni for success as it provides them with the tools to quickly learn new types of knowledge specific to a new job position.

The Job Market Utility of the General Workplace Skills Participants Learn in CYES

Most alumni who had found employment in a field where CYES job skills were applicable15 reported that the skills they learned in CYES had either helped them to land jobs, or to perform well in their jobs. These general workplace competencies included professional and soft skills, such as leadership, communication, and teamwork, as well as “field-based skills” such as time management, fundraising, and customer service. One participant shared that CYES taught her how to work effectively with supervisors. She explained that before the program, she reacted negatively to authority and did not like being told what to do. However, with the supportive approach of her managers in CYES, she was able to accept direction and understand that her supervisors were supporting her growth and development as an employee. This shift allowed her to bring a more positive attitude to her work, and improved her working relationships with others.

15 Analysis excludes alumni who were unemployed or who found jobs for which the skills taught in CYES were not applicable

Exhibit 27: Usefulness of CYES knowledge in

obtaining and succeeding in employment

Exhibit 28: Usefulness of CYES skills in obtaining

and succeeding in employment

*Includes communication. **Includes teamwork/cooperation and leadership. ***Includes customer service, presentation skills, problem-solving/decision making, and time management. Analysis excludes alumni who were unemployed or who found jobs for which the skills taught in CYES were not applicable

Being able to communicate and my positive attitude actually helped me move up the ladder of success in the job that I still have now. It was great. I moved up really fast in the company. I can thank CYES and all they did for me and the growing process. They helped me grow up pretty much.

CYES Alum

77%

76%

71%

0% 100%

Energy Efficiency Knowledge (n=39)

Water Conservation Knowledge (n=38)

Knowledge of GreenHouse Call Procedures (n=34)

Percent reporting that their knowledge helped them land or

perform well at their jobs

85%

84%

82%

0% 100%

Professionalism*

Soft Skills**

Field-Based andAdministrative Skills***

n=66

Percent reporting that their skills helped them land or perform well at

their jobs


Feedback from Participants on the CYES Program

It is clear from the findings in the previous sections that most participants and alumni gained significant skills and knowledge from their involvement in CYES. What aspects of the program contributed to this success? We can gain some insight into this question by listening to the feedback that CYES participants gave on the training and on CYES staff, as well as to their thoughts on what they enjoyed most about the program.

Key Findings

CYES participants report that: o The trainings are of high quality, o The amount of training offered is generally just right, and o The staff they work with (Site Managers, Outreach Managers, and Regional Managers) provide

excellent support.

Feedback on Training

CYES participants receive substantial training during their first week, as well as professional development workshops throughout the program. The topics covered include: Technical aspects of field work Client-facing skills General Introductions and closings Green House Call procedures Program paperwork and

procedures Energy efficiency and water

information Hardware installation

Presentation skills Customer service Outreach and client

recruitment General professional

development

All-staff CYES training on the first day

CYES policies

Quality of Training

The survey asked participants to rate the training quality on a 5-point scale, from “poor (didn’t train me enough for my job)” to “excellent (felt very prepared for my job).” Nearly all of the CYES participants felt that the training they received through CYES prepared them to do their jobs, with the majority rating trainings as “above average” or “excellent” (Exhibit 29).

16 Survey item on a 5-point scale (1=Poor; 2=Below average; 3=Average; 4=Above average; and 5=Excellent).

Exhibit 29: CYES participants who felt that the

trainings prepared them well for their jobs16

I thought all of the exercises were necessary for team building and were great!

CYES Participant

76%

77%

85%

0% 100%

General Trainings

Client-Facing Skills

Technical Aspects of Field Work

n=80 Percent reporting "above average" or "excellent"


Quantity of Training CYES participants were also largely satisfied with how much training they received. On average, more than three quarters of participants considered the amount of training they received to be appropriate. The survey gave the option for respondents to say that there was not enough training for particular training topics. Some CYES participants did give the feedback that the amount of training was insufficient for: Hardware installation (23%), Outreach and client recruitment (15%),

and Professional development workshops

(10%). A few participants expressed through open-ended survey comments that they would have appreciated more training, even though they had rated the quantity of their training as appropriate in most or all areas. Focus group participants echoed this feedback. This desire for additional training indicates that youth found their training valuable.

Feedback about Rising Sun Staff

In evaluating their Site Managers and Site Outreach Managers, nearly all CYES participants rated the support they received as “above average” or “excellent.” Participants indicated that they felt comfortable approaching staff with their concerns or needs, that the expectations of their Site Managers and Site Outreach Managers were clear, and that they had received the support they needed to do their jobs well. More than two thirds of participants again rated their overall site management and site outreach management as “excellent,” the highest rating. Several CYES participants were enthusiastic about their experiences with their Site Managers, emphasizing characteristics such as approachability, problem-solving, and positive attitude.

17 Survey item on a 5-point scale (1=Significantly too little; 2=Too little; 3=Appropriate amount; 4=Too much; and 5=Significantly too much). 18 Survey item on a 5-point scale (1=Poor; 2=Below average; 3=Average; 4=Above average; and 5=Excellent).

Exhibit 30: CYES participants who felt that they

received the right amount of training17

I thought our managers were great at training us. They gave us the exact amount of info we were going to need.

CYES Participant

Exhibit 31: CYES participants rating

management “above average” or “excellent”18

I felt very comfortable to talk about anything to my site manager. He was very approachable and open to hearing new suggestions and ideas.

CYES Participant

76%

78%

80%

0% 100%

General Trainings

Client-Facing Skills

Technical Aspects of Field Work

n=80 Percent reporting an "appropriate amount"

89%

91%

0% 100%

Site Outreach Management

Site Management

n=80

Percent reporting "above average" or "excellent"


Similarly, youth greatly appreciated the positivity, support, and communication skills of their Site Outreach Managers. Multiple youth recognized the hard work and success of their Site Outreach Managers in securing Green House Call appointments with clients.

While youth generally had less direct contact with their Regional Managers than they did with Site Managers and Site Outreach Managers, the majority felt supported in doing their jobs well (90%), and felt comfortable approaching their Regional Manager with concerns, needs, or feedback (88%).

In addition to their praise for individual Rising Sun staff members, participants also greatly appreciated the site team’s ability to work together; 86% of participants rated their overall site teamwork as “above average” or higher (results not shown).

19 Survey item on a 5-point scale (1=Strongly disagree; 2=Disagree; 3=Neither agree nor disagree; 4=Agree; and 5=Strongly agree).

[My Site Outreach Manager] was always pumped up and full of energy. She is a positive person, which helped me stay positive in difficult or uncomfortable situations.

CYES Participant

Exhibit 32: CYES participants who felt that their

regional managers were supportive19

Every exchange I've had with [my Regional Manager] has been pleasant. He was always professional and always knew what needed to be done.

CYES Participant

88%

90%

0% 100%

Felt comfortable approachingmy Regional Manager with

concerns, needs, or feedback

Provided the support anddirection I needed to do my job

well

n=80Percent reporting "agree" or

"strongly agree"


The Influence of the LIFT Program

Like CYES, Rising Sun’s Leaders in Field Training (LIFT) program provides participants with the opportunity to develop a broad range of skills. Rising Sun’s evaluation focuses largely on the influence of CYES on its participants, but there are also a few items on the alumni survey that ask former LIFT participants to share the ways that the LIFT program – in particular – affected them. There is also a set of items on the CYES post-program survey asking CYES participants to rate the LIFT participants they worked with along several dimensions. This section uses those data to explore LIFT participant outcomes and skill-building.

Key Findings:

Large majorities of LIFT alumni say that the program contributed to growth in a variety of green knowledge areas.

Large majorities of LIFT alumni say that the program contributed to growth in several categories of workplace skills.

Nearly all LIFT participants report that the program helped them build a variety of specific leadership competencies.

CYES participants gave overwhelmingly positive ratings to the LIFTs they worked with as part of CYES.

LIFT alumni reported that the skills they learned in the program had supported them in getting hired into jobs, or in doing well on the job.

Three quarters of LIFT alumni entered leadership positions after they left the program, and all of the alumni who did so say that their experience in LIFT contributed to their pursuit of these roles.

Increases in Green Knowledge, General Workplace Skills, and

Leadership Competencies

Increases in Green Knowledge

The majority of LIFT alumni felt that participation in LIFT had substantially increased their knowledge in water conservation (72%), energy efficiency (71%), and climate change (71%). Among these, more than half selected the highest rating possible for each knowledge area, indicating that they had built their knowledge to a “great extent.”


Exhibit 33: Increases in green knowledge20

72%

71%

71%

0% 100%

Water Conservation Knowledge

Energy Efficiency Knowledge

Climate Change Knowledge

n=14Percent reporting 4 or 5 on a5-point scale (5=Great extent)


Increases in General Workplace Skills

LIFT alumni indicated that the program had helped them considerably to develop a broad range of skills. Exhibit 34 illustrates that through their participation in LIFT, the majority of LIFT alumni experienced a substantial increase22 in soft skills (94%), such as teamwork/cooperation, and adaptability. Most LIFT alumni also substantially increased skills associated with professionalism (93%), such as positive attitude and communication, as well as field-based and administrative skills (90%), such as customer service and presentation skills. Among field-based and administrative skills, problem solving/ decision making skills represented an important growth area for LIFT alumni: 100% of LIFT alumni reported a substantial increase in this area.

Increases in Leadership Competencies

In addition to the interpersonal and leadership skills promoted for CYES and LIFT alike, the distinct focus on leadership in the LIFT program allowed participants to gain a deeper understanding of the qualities of a leader, and to develop those characteristics themselves. For this reason, the Alumni Survey asked LIFT alumni to complete a more comprehensive “leadership inventory,” including specific leadership competencies such as the ability to motivate others, supporting team members to cooperate with one another, etc. Nearly all LIFT alumni reported that LIFT had helped them to build leadership competencies. These results bolster the case that the LIFT program contributes to leadership development for youth.

21 Survey item on a 5-point scale (1=Not at all; 3=Some extent; and 5=Great extent). 22 Based on ratings of 4 or 5 on a 5-point scale. 23 Survey item on a 5-point scale (1=Not at all; 3=Some extent; and 5=Great extent).

Exhibit 34: Increases in general workplace skills21

*Includes leadership, initiative, teamwork/cooperation, and adaptability. **Includes positive attitude and communication. ***Includes customer service, problem-solving/decision-making, presentation skills, project planning, facilitation, mentoring, delegation, time management, administrative skills, outreach/marketing, and fundraising.

Exhibit 35: Development of leadership competencies23

94%

93%

90%

0% 100%

Soft Skills*

Professionalism**

Field-Based andAdministrative Skills***


92%

92%

92%

92%

92%

92%

85%

77%

0% 100%

Motivating and inspiring others

Stepping up and being willing torepresent the needs or interests of your

team

Communicating interests of your groupto superiors or other

Making sure team members cooperateand work with each other on common

goals

Leading by example

Acting confidently

Encouraging communication amongteam members

Separating yourself from rest of groupand not just going with general

consensus



Learning Skills through Practice

As part of their program, LIFT participants support Energy Specialists by providing weekly professional development workshops and by serving as peer mentors. In this way, the LIFT program allows participants to gain skills through the direct experience of taking on leadership roles. To get a sense of how LIFT participants perform in these roles, we can turn to another data source: the CYES participants. In their post-program surveys, CYES participants rated the performance of the LIFTs they worked with. (Note that these data refer to a set of LIFT participants that is different from the LIFT participants who completed the alumni survey.) When asked to rate the LIFT participants they worked with, CYES participants were overwhelmingly positive (Exhibit 36). CYES participants felt that the LIFT participants they worked with had supported them by helping to ensure the smooth operation of their site, consistently demonstrating professionalism, and serving as good teachers, trainers, peer mentors, and/or role models.

How Learning General Workplace Skills Supported LIFT Alumni in the Job

Market

To what extent have the skills LIFT participants learned helped them to find jobs or to do well in them? Of the LIFT alumni who were employed at the time of the Alumni Survey in positions where LIFT job skills were applicable25 most reported that skills they learned through LIFT had in fact helped them to obtain employment or to perform well in their jobs. The vast majority of LIFT alumni felt that the skills they had learned through LIFT had helped them to obtain or perform well in a job, including skills related to professionalism (91%), soft skills (85%) and field-based and administrative skills (84%).

24 Survey item on a 5-point scale (1=Strongly disagree; 2=Disagree; 3=Neither agree nor disagree; 4=Agree; and 5=Strongly agree). 25 Analysis excludes alumni who were unemployed, or who found jobs for which the skills taught in LIFT were not applicable.

Exhibit 36: Perceptions of LIFT participants’ work

quality through the eyes of CYES participants24

Our LIFT was easy to get along with. Very positive, cheerful attitude. Always smiling and joking. She was serious when needed. Gave us great advice and tips for life. Great person to have added to our team.

CYES Participant

Exhibit 37: How learning general workplace skills

has helped LIFT alumni in the job market

*Includes communication and positive attitude. **Includes leadership, teamwork/cooperation, adaptability, and initiative. ***Includes delegation, mentoring, problem-solving/decision making, time management, fundraising, administrative skills, presentation skills, outreach/marketing, facilitation, project planning, and customer service. Analysis excludes alumni who were unemployed or who found jobs for which the skills taught in LIFT were not applicable.

85%

90%

91%

93%

94%

0% 100%

Demonstrated professionalismat all times

Was a good peer mentor

Was a good role model

Helped my site operatesmoothly

Was a good teacher/trainer

n=80Percent reporting "agree" or

"strongly agree"

84%

85%

91%

0% 100%

Field-based andadministrative skills***

Soft skills**

Professionalism*

n=11

Percent reporting that their skills helped them land or perform well at

their jobs


Post-Program Decisions for LIFT Participants

This section reports on the extent to which LIFT played a role in participants’ decisions following the program. It also provides data on the extent to which LIFT alumni took on leadership roles in their lives after leaving the program.

Influence of LIFT on Post-Program Decisions

For LIFT alumni who pursued employment, all reported that LIFT had influenced their decision to at least some extent and two thirds (67%) considered LIFT to have had a strong influence on that decision. As was the case with CYES, LIFT alumni who pursued education immediately after the program were less influenced by LIFT. However, two thirds reported that LIFT had had at least a slight influence on their decision to pursue college (66%), and 22% were strongly influenced by their LIFT experience. More than half of alumni who selected environmentally, socially, or business-related majors reported that LIFT had influenced this decision to some extent, with 43% reporting a strong influence.

Leadership Positions after LIFT

Three quarters of LIFT alumni (75%) reported that they have had experience holding leadership roles after leaving the LIFT program. All of these alumni (who report that they took on leadership roles) attribute these roles at least in part to their participation in LIFT, and 44% identify LIFT as a strong influence on their decision to pursue leadership roles (results not shown). Although the youth who apply for LIFT positions are likely to already have some leadership aspirations, these findings indicate that participation in the LIFT program helps to support youths’ aspirations and encourage youth to continue to identify ways that they can become leaders in their community.


Exhibit 38: Influence of LIFT on post-program decisions26

Exhibit 39: LIFT participants

taking on a leadership role after

leaving the program

67%

43%

22%

0% 100%

Search for or get a job (n=9)

Pursue an environmentaly, socially, orbusiness related major (n=7)

Attend college (n=9)


Yes

75%

No

25%

n=12


Methodological Appendix

This evaluation used quantitative and qualitative methods, including surveys collected pre- and post-program from CYES participants (at the beginning and end of the 2013 summer program), as well as a follow-up survey completed by CYES and LIFT alumni who participated in the 2011, 2012, and 2013 programs. To collect richer, narrative information from youth, LFA conducted an alumni focus group in 2014.

Data Collection

CYES Pre- and Post-program Surveys

In 2013, Rising Sun surveyed 85 youth who participated in CYES. Of these youth, 76 completed a pre-program survey at the start of CYES, which included demographic data, home life and educational status, and baseline information about knowledge and skills related to CYES. At the end of the program, 80 youth completed a post-program survey, which included updated ratings of knowledge and skills, as well as feedback on various aspects of CYES.

Alumni Survey

In 2014, Rising Sun reached out to CYES and LIFT alumni to learn about the lasting effects of program participation. A total of 81 CYES alumni completed the survey, including 79 who served as Energy Specialists, and 18 who participated in the LIFT program. (Seventy-nine and 18 add to more than 81 because 16 youth who completed the survey had participated in both programs.) Because only two of the 81 youth had participated only in LIFT, the report attributes findings from this survey to “CYES alumni.”

Alumni Focus Group

To gain additional insight about CYES’s impact on participants, LFA conducted a focus group of CYES alumni in spring of 2014. Although 10 youth committed to attend, only three youth participated in the focus group. LFA supplemented the data collected from these three youth by asking additional invitees to submit written responses to the focus group questions; two alumni submitted responses. A significant portion of interested youth could not attend the focus group because they no longer live in the San Francisco Bay Area.

Analysis

LFA ran descriptive statistics on the surveys: frequency distributions and means. As a result of reflection on the preliminary findings with the client, the client and the evaluation team made the joint decision to “collapse” some of the more detailed survey items to create aggregated indices (e.g. several skills were aggregated into one index called “soft skills”). For the final reporting, we re-analyzed the data to show – instead of the full frequency distributions – the percent of respondents who answered either a 4 or 5 on a 5-point scale. To analyze the focus group and open-ended survey results, LFA used content analysis and triangulated data to extract key themes and findings.


Strengths of the Methods

The surveys were developed in consultation with a broad range of Rising Sun program staff. Working with the staff helps to ensure that the data will be useful for program planners and implementers.

The surveys were developed explicitly to measure the key outcomes that are central to Rising Sun’s program objectives.

Using qualitative methods to supplement surveys means that the analysis can take advantage of the richer qualitative data in order to contextualize and validate the survey data.

Limitations of the Methods

The research design was “non-experimental,” meaning that there was no comparison or control group. This is common: using an experimental or quasi-experimental design is resource-intensive and often infeasible. However, without a comparison or control group, we cannot – technically – draw causal inferences about program participation. To identify the effects that the program had on participants, survey items directly asked respondents to reflect on the extent to which they themselves attributed outcomes to program participation. While these data are certainly highly suggestive, we cannot draw causal conclusions from the data in the absence of a comparison.

There were several technical survey errors: (1) skip logic unintentionally excluded certain participants from responding to items that may have been relevant to them; (2) survey items that should have been “check one” were coded as “check all that apply,” resulting in respondents sometimes checking multiple options when the answer options are logically mutually exclusive; and (3) data for some survey items such as “overall satisfaction” were unavailable.

LIFT was measured directly only through Alumni survey. In addition, many survey questions did not differentiate between CYES and LIFT, making it difficult to capture the impact of the LIFT program alone.

The Alumni survey included only a few open-ended fields, meaning that it is not possible for LFA to contextualize the Alumni quantitative data in the same way that the CYES pre- and post-surveys results are contextualized with open-ended responses.

The focus group was small, which means that the evaluators may have missed some important diversity of opinion.

rising sun evaluation report

Documents