Download - Intro to the cqi process 2015 4-8
Intro to Continuous Quality Improvement –Using the PSAYDN Tool
Today’s Objectives • Explain how CQI supports DHS youth outcomes• Use CQI as a way to tell your story• Explain how to use the PSAYDN self-assessment for CQI• Develop an implementation strategy for the PSYADN self-assessment
Today’s Agenda
• Why? – CQI Rationale
• With What? – Using the PSAYDN Tool
• Who + How? Strategy for Implementing the Tool
• When + How? Submitting the Assessment
• Q & A about CQI
Why CQI?
It Works!Here’s How…
CQI is an on-going process ofassessing the quality of workand making improvements
Continuous Quality Improvement (CQI)
Helps tell the story of your work to multiple stakeholders about achieving desired youth impact
Continuous Quality Improvement (CQI)
Uses feedback tools(e.g. parent surveys, debriefing forms)
Uses supports to make informed improvements
(e.g. staff meetings, coaching, professional development)
Continuous Quality Improvement (CQI)
Designed so improvements can be made throughout the process.
Continuous Quality Improvement (CQI)
Picture of Program Quality
Caregiver/Youth Assessment
External Assessment (Site Visit)
Internal Assessment (PSAYDN/APAS)
Getting a Complete Picture
But Why?
Outcomes.
DHS Mission Statement
The Philadelphia Department of Human Services mission is to provide and promote safety, permanency, and well-being for children and youth at risk of abuse, neglect and delinquency.
DHS OST Project FY 2015 Vision
Create a more intentional connection between the needsof all children and youth, particularly those served by DHS
and the DHS OST Project as an effective prevention strategy and critical community-based resource.
FY 15 Youth Impact
Youth Focus
Program Quality Focus
Evaluation Framework (Desired Youth Impact)
Youth who participate in DHS-funded OST services are less likely to be active within DHS;
and
Youth who were formerly active within DHS are less likely to re-enter the DHS system of services.
GOAL AREA OUTCOMES INDICATORS
Academic success
1. Increased Engagement in Learning
2. Increased Engagement School3. Aspires to Educational
Excellence
1. Improved Quality of Homework Time2. Increased Opportunities for Problem
Solving & Critical Thinking 3. Awareness of High School Choices
Develop positive life skills
1. Improved Life Skills2. Improved Relationships
1. Goal Setting 2. Personal Accountability3. Work with Others to Accomplish Goals4. Improved Interactions with Peers 5. Improved Interactions with Adults
Ready for college and/or career
1. Prepared for Higher Education and Employment
1. Awareness of and interest in Careers
DHS OST Goals, Outcomes & Indicators
GOAL AREAS OUTCOMES INDICATORS
Academic success
1. Increased Engagement in Learning
2. Increased Engagement School
3. Aspires to Educational Excellence
1. Improved Quality of Homework Time2. Increased Opportunities for Problem
Solving & Critical Thinking 3. Awareness of High School Choices
Develop positive life skills
1. Improved Life Skills2. Improved Relationships
1. Goal Setting 2. Personal Accountability3. Work with Others to Accomplish Goals4. Improved Interactions with Peers 5. Improved Interactions with Adults
Ready for college and/or career
1. Prepared for Higher Education and Employment
1. Awareness of and interest in Careers
DHS OST Goals, Outcomes & IndicatorsElementary School Priorities
Red = Primary Blue = Secondary
GOAL AREA OUTCOMES INDICATORS
Academic success
1. Increased Engagement in Learning
2. Increased Engagement School
3. Aspires to Educational Excellence
1. Improved Quality of Homework Time2. Increased Opportunities for Problem
Solving & Critical Thinking 3. Awareness of High School Choices
Develop positive life skills
1. Improved Life Skills2. Improved Relationships
1. Goal Setting 2. Personal Accountability3. Work with Others to Accomplish Goals4. Improved Interactions with Peers 5. Improved Interactions with Adults
Ready for college and/or career
1. Prepared for Higher Education and Employment
1. Awareness of and interest in Careers
DHS OST Goals, Outcomes & IndicatorsMiddle School Priorities
Red = Primary Blue = Secondary
GOAL AREA OUTCOMES INDICATORS
Academic success
1. Increased Engagement in Learning
2. Increased Engagement School3. Aspires to Educational
Excellence
1. Improved Quality of Homework Time2. Increased Opportunities for Problem Solving
& Critical Thinking 3. Awareness of High School Choices
Develop positive life skills
1. Improved Life Skills2. Improved Relationships
1. Goal Setting 2. Personal Accountability3. Work with Others to Accomplish Goals4. Improved Interactions with Peers 5. Improved Interactions with Adults
Ready for college and/or career
1. Prepared for Higher Education and Employment
1. Awareness of and interest in Careers
DHS OST Goals, Outcomes & IndicatorsHigh School Priorities
Red = Primary Blue = Secondary
CQI and You
• Program quality• Empowering site-level staff
to lead improvements• WILL NOT determine FY
2016 funding
CQI and You Data gathered will help:• PHMC and your program work
together to improve quality• Determine whether youth
experiences in your program result in desired DHS youth outcomes
• DHS make decisions about system-wide supports
The CQI Process - 2014Collect data,
Reassess
Provider self-assessment
Data collection
and analysis Goal
setting w/ Provider, Program Specialist
Program Improvements
Site Visit
ProgramImprovements Training on
CQI and Tools
Adapted from the National Institute on Out-of-School-Time
The CQI Process - 2015Collect data,
Reassess
Program Specialist Reviews
Goal setting w/
Provider, Program Specialist
Program Supports and Improvement
Site Visit
Program Supports and Improvement Provider Goal
Setting In Reapplication
Adapted from the National Institute on Out-of-School-Time
Picture of Program Quality
Caregiver/Youth Assessment
External Assessment (Site Visit)
Internal Assessment (PSAYDN/APAS)
Getting a Complete Picture
Here’s How.
PSAYDN Self-Assessment
Section A – Structure & Management• Fundamental to the implementation of a good program
• Will likely require management support to complete• Executive Director• Human Resources• Finance
• Useful for larger strategic planning discussion within the program and the agency
• Program Specialist will provide minimal coaching in this area
Section B – Positive Connections• Helps us examine the relationships between staff, youth,
and caregivers within our programs
• Research and experience tell us that relationships are key to successful youth development work
• Also assessment connections to community partners
Section C – Safety & Health
• Just like sound management practices, safety is critical to success in other areas
• Focuses on emergency policies, facilities, and activities that promote healthy lifestyles
Section D - Activities• Assesses the extent to which activities are:
• Thoughtfully-designed• Age-appropriate• Youth-driven and responsive to youth interests• Foster learning and positive youth development• Varied
• Also examines the planning process for activities
Addendum for Elementary Programs
• The quality of homework time is a priority indicator for DHS elementary programs.
• PSAYDN Self-Assessment does not include a homework time.
• The addendum is a chance to assess how well your program is providing homework support.
• Can be identified as an area for improvement.
Additional Questions for ALL Programs• #46 - Rank the PSAYDN categories in order of one to five based on
the areas your program could use the most additional support.
• #47 - What additional supports would your program like from PHMC/DHS to assist in the area from Q. #46 ranked as a number one (most important).
• #48 - How did program staff feel about the CQI process in FY2015?
• #49 - Why does your program believe the CQI process worked? Select all that apply.
• (If you select ‘No’ to #49) #50 - Why does your program believe the CQI process did not work well? Select all that apply.
Additional Questions for ALL Programs• #51 - What communication process did your program use to share
CQI information with your Executive Director and other stakeholders?
• #52 - What was the first goal you set for FY2015?
• #53 - Did your program achieve the first goal set for FY2015?
• #54 - Provide an explanation for why your program did or did not achieve the first goal.
• #55 - What was the second goal you set for FY2015?
• #56 - Did your program achieve the second goal set for FY2015?
• #57 - Provide an explanation for why your program did or did not achieve the second goal.
Additional Questions for ALL Programs• #58 - What was the third goal you set for FY2015? If you only set
two goals, write N/A below.
• #59 - Did your program achieve the third goal set for FY2015?
• #60 - Provide an explanation for why your program did or did not achieve the third goal.
• For each section, we will ask about your program’s rationale for the score selected for that section’s questions. Be prepared with typed text that you can insert into the online survey.
Scenarios
A.1.3 – Guiding Questions
• Do staff have access to on-going professional development opportunities such trainings and conferences?
• What are the yearly expectations for professional development, in terms of amount and/or content, by staff’s level of responsibility in your program?
• What types of on-site support do staff have to continue building skills?
Scenario #1: A.1.3Agency ABC: Staff members feel well supported.
They are invited to attend paid workshops offered by professional development providers, as well as
occasional on-site in-service trainings. All staff participate in PD in some form, but agency ABC does not monitor requirements for annual PD.
Meetings are held once per month and focus on program planning and policy updates. Staff receive annual performance reviews in writing with some
supervision is provided as needed.
Scenario #1: A.1.31 Only some staff attend professional development
Professional development topics not informed by any assessment of needs
3 Staff participate in on-going professional development to address the unique
characteristics of youth and families and the content focus of program
Staff receive only informal supervision or feedback related to their performance
Staff meetings held at least monthly to address program issues
5 Staff receive written, performance-based, assessments of their work, identify areas for
needed skill-building, and plan for professional development based on identified
priorities
Professional development opportunities are available in multiple formats (on- and off-
site) including workshops, observation, coaching, and peer-to-peer learning
Staff receive regularly-scheduled one-on-one supervision and feedback
Staff meetings held more than once a month to address program issues and support
staff skill-building
B.1.1 – Guiding Questions
• In what ways do staff support positive communication and conflict resolution with youth and co-workers?
Scenario #2: B.1.1Agency DEF: The type of communication happening varies a great deal. Six of eight members of the team get along very
well and have positive communication with the youth. Among these staff, conflicts with co-workers are resolved
quickly. These staff also use the organization’s reward/consequence structure to guide youth behavior.
There are two staff members that don’t gel with the rest of the team. They are quick to snap at the youth and at the other staff members. The kids have started to call them
Oscar #1 and Oscar #2 (the grouch). There are noticeably more behavior problems (put downs, teasing, squabbling, etc.) among the youth these two staff members supervise.
Scenario #1: B.1.11 Staff either ignore youth conflicts, or impose negative consequences without
taking the time to understand the cause
3 Staff relate to all participants in positive ways Program establishes, maintains and communicates code of conduct to participants,
staff, and their families Program applies rewards and consequences for participant behavior appropriately and
consistently Staff communicate with respect and are role models of positive relationships
5 Staff relate to all participants in positive ways and intentionally build individual relationships
Staff approach negative behavior calmly, acknowledging youth’s feelings while stopping hurtful behavior
Staff facilitate constructive conflict resolution when needed
C.1.2 – Guiding Questions
• Does your program have written policies to ensure equal access and safe participation for all youth?
• How do staff promote and maintain a safe physical and emotional environment for all participating youth?
Scenario #3: C.1.2Agency GHI: Prides itself on creating a safe and inclusive
space for youth. Last year, their 70 year-old building added a wheelchair ramp and is now ADA compliant.
Program activities focus on celebrating differences and there is virtually no bullying as a result. Several youth
have special needs and the Site Director met with each caregiver to understand how to best accommodate their
youth’s needs. Six months ago, agency leadership and father of one youth decided the program wasn’t suited
to meet his son’s needs, who had a severe autism diagnosis. The staff and family worked together to find
another program that would work better.
Scenario #3: C.1.21 Program space or offerings are not accessible to all youth
3 Program space and operations meet all local, state and federal regulations regarding accessibility (e.g. ADA)
Program develops and manages effective arrival and dismissal procedures and plans for safe travel home
Participant files include emergency contact information, medical release forms, allergy and health information
No one in the program feels threatened or unsafe because of who they are
5 • Program materials and policies emphasize commitment to serving all youth in the community
• Staff make reasonable accommodations to serve youth with learning differences or special needs
• Staff consistently intervene when youth are adults make hurtful or negative comments
• The emotional climate of the program is positive, characterized by mutually respectful, supportive interactions among staff and youth
D.1.1 – Guiding Questions
• Describe your planning process - how do you determine overall goals and daily activities (e.g., using your logic model, documented program design)?
Scenario #4: D.1.1Agency KLM: Youth are always busy. The staff plan their activities the evening before, using a couple of activity idea books the Site Director bought online.
Examples of activities include friendship bracelets, a soda bottle terrarium, and multiplication jeopardy.
When asked how activities are chosen, the staff say, “Whatever sounds fun.” When the staff don’t have
the materials they need for a particular activity, they have some board games, jump ropes, and a kickball.
Youth of all ages (K-5) participate in activities.
Scenario #4: D.1.11 Activities are not intentionally planned according to program mission,
developmental or learning objectives
3 Activities reflect the mission of the program and promote the physical, mental, and/or social-emotional development of all participants
All activities planned according to the age and skill level of the participants and identify targeted skills that participants will develop during the program cycle
5 Comprehensive written program design and activity plans define learning objectives that build on each other throughout the program cycle and show how activities connect to objectives
Program emphasizes the connections between learning and real life
What’s Your Process?
The Who and How
Self-Assessment Comfort Test
High comfort level with self-assessments • Complete with team and submit online.
Some experience with self-assessments• Program Specialist support for questions as needed.• Work with team to complete and submit online.
Completely new to self-assessments • Share online training with key individuals.• Regular email/conversation with Program Specialist.• Work with team to complete and submit online.
PSAYDN Discussion Guide
IMPORTANTWhen entering the survey online• Complete it in one sitting – It does not save progress.
Have your answers ahead of time• Have your scores for each item that applies• Have the rationale for each section and additional items
answered in advance.
Due c. May 15th• Do not wait until the last minute – plan it on a calendar.
SCORE YOUR PROGRAM ACCURATELY. DHS IS NOT USING THIS TO JUDGE YOUR PROGRAM. THEY USE IT TO
DETERMINE SUPPORTS.
Minimal Investment- CQI Training
- Manager/SD completes form independently- Enter data
Moderate Investment
- CQI Training- Management team
completes form with some staff input
- Enter data
Maximum Investment- CQI Training
- Exec. Staff input for Part A- Line staff input for Parts B-D
- Time to discuss- Align with site visit assessment to
determine evidence of quality - Enter data
How much time will it take?
Tips for Getting Started: Program Directors
• Make an appointment with administrative (managers, directors) staff to ask questions.
• Let your team know that this is coming.
• Consider involving other stakeholders (families, funders, activities partnerships)
• Try to collect “evidence” (sample parent/family survey, newsletters, student artwork, rubrics, etc.) that support your program’s efforts.
Tips for Getting Started: Working With Administration (EDs, Managers)
• Make yourself available on their schedule.
• Set up a brief phone meeting before starting.
• Review the self-assessment before meeting.
• Make suggestions for potential improvement project and program changes.
Submitting the Self-Assessment
EXAMPLE PSAYDNSELF-ASSESSMENT
Timeframe
Goal Setting Reapplication
May - June
PHMC Analysis of
Data May-June
Complete Tool and
Submit Data
May 15
Review Tool & Determine
Support Needed
Apr - May
Intro to CQI Process
Early April
Tools available online
http://afterschoolinphilly.weebly.com
Wallace Foundation: Strategies for Quality After School Programs and Activities
NIOST: Understanding Basic Standard for A Quality Out-of-School Time Program
Quiz Question #1
What does the abbreviation CQI stand for?
A. Continuous Quality ImprovementB. Content Quality ImprovementC. City Quality Instrument
Quiz Question #2
How many sections are on thePSAYDN self-assessment?
A. ThreeB. FourC. Five if you include the HW addendumD. SixE. Seven
Quiz Question #3
Who should ideally be involved in the self-assessment process?
A. Executive Staff, Finance & Human ResourcesB. Program Directors/ManagersC. Site DirectorsD. Direct Line StaffE. All of the Above
Quiz Question #4
Where can I find self-assessment submission instructions?
A. The Scope of WorkB. PCAPSC. PHMC Website D. LearnUpon