community connections allamakee community school district february 23, 2012
TRANSCRIPT
Community Connections
Allamakee Community School District
February 23, 2012
Data to base decisions upon to inform program to be valid
Data from qualitative and quantitative perspectives
Data from various sources to inform program of reliability
External evaluation provides insight different from your own perceptions
Provides basis for investment of others in future sustainability
To Communicate with stakeholders
To Build collaboration towards common goals
Why invest in evaluation?
Review and communicate PPIC’s Numbers of Students Attending School-Day Attendance Characteristics of Students Attending Grades, Test Scores, Teacher Surveys Contributions of Partners Range of Programming Activities Characteristics of Staff and Volunteers
Build upon PPIC’s Data Experts in goal areas of your grant Basis of sustainability in future
Data-based Decision-making
Data-based Decision-making
Build upon PPIC’s DataQuantitative Datao Olweus Bullying Survey Resultso Disciplinary Referralso Collaboration Rating Scaleo Dynamic Indicators of Basic Early Literacy Skills (DIBELS)
Qualitative Datao Collaboration Rating Scaleso Consensus Building Program Priorities in 10 groups with 68
participantso Goal Status Report: Based on Interviewso Afterschool Clubs connected to the Core Curriculum
Demonstrate an understanding of written language and the relationship of letters and words to the sounds of speech. Demonstrate an understanding of the concepts of print.
o Hold a book correctly.o Identify the front/back cover of the book.o Identify the top/bottom of the book.o Read left to right with return sweep.
Club Leader would record number of students participating in club and anticipated skills to be mastered.
Club Leader would record number of students mastering each skill at end of the club.
Evaluator compiles all data and reports back skill mastery percentages.
Example: K - 2 Reading
“It is very valuable (evaluation information) in that it gives us data to help to drive programming. It also gives us specific data that is very specific to our students, staff, parents and community members.”
ACSD Superintendent states:
Outcomes and Process Review goals with Third Party or External Evaluators Ask and discuss questions about how data will be collected related
to attaining the outcomes desired Ask and discuss questions about the process of collaboration and
review of data previously collected and what needs to be collected next for program modifications
Communicate and build relationships with evaluators, administrators, community members, students, businesses, agencies, senior citizens, organizations, and parents involving all types of stakeholders
Use Data to Build Program
Paper and Pencil Surveys Search Survey Iowa Youth Survey Program Specific Surveys
Focus Groups
Interviews Formal Informal conversations
Reports Monthly Reports from Staffo Numbers of Daily Attendance (Quantitative)o Success Stories and Challenges (Qualitative)
Updates at Staff Meeting Informal Conversations
Variety of Methods to Collect Data
Review Data in various settings with staff and stakeholders
Compare Data with previous data collected 13 years with 5 survey results for 6th, 8th, 10th, and 12th grade
students
Publish Data in numerous media venues Posters in Business Coffee Break Rooms Data Book for legislators and stakeholders Table Tents Labels on Popcorn Bags Charts Power Point Presentations
Use Data to make decisions
If missing data then make plans to collect additional data
Data Analysis
Search Institute Survey Instrument: Developmental Assets: A Profile of Your Youth
Iowa Youth Survey
Recent Data Analysis
Average Number of Forty Assets
EXTERNAL ASSETSEXTERNAL ASSETS SupportSupport EmpowermentEmpowerment Boundaries and Boundaries and
ExpectationsExpectations Constructive Use of Constructive Use of
TimeTime
INTERNAL ASSETSINTERNAL ASSETS Commitment to Commitment to
LearningLearning Positive ValuesPositive Values Social CompetenciesSocial Competencies Positive IdentityPositive Identity
Positive Youth Development Asset Positive Youth Development Asset Building Model developed by Search Building Model developed by Search
InstituteInstitute
Constructive Use of Time
0
10
20
30
40
50
60
70
80
90
100
CreativeActivities
Youth Programs ReligiousCommunity
Time at Home
19982002200720092011
Commitment to Learning
0
10
20
30
40
50
60
70
80
90
100
AchievementMotivation
SchoolEngagement
Homework Bonding toSchool
Reading forPleasure
19982002200720092011
Questions?
Surveys • specific to program goal(s)
Interviews• open-ended questions • analyze patterns in answers
Focus Groups• trained facilitator• structured format• “Do you have any other information to tell?”• transcript of answers verbatim
Qualitative Data Methods
To gather information for goals not easily supported through quantifiable methods
Sample scenario: All partners in the community including parents and youth have strong relationships working toward common goals.
Answers from structured questions asked in interviews (1) site coordinators/staff, (2) parents, (3) students
Excerpts from monthly written staff reports.
Qualitative Data – Why bother?
Sample scenario: Families in the school district become more supportive of student success.
Design statements representing a hierarchy of change: Pre-contemplation, Contemplation, Preparation, Action, Maintenance
Replies to structured statements asked in interviews:
(1) site coordinators/staff, (2) parents of students served in program, (3) parents of students not served
Measuring attitudinal change
Measuring Collaboration with PartnersBelief that collaboration adds strength to program effectiveness and ability to influence change.
Rubric: Levels of Collaboration. Adapted from Borden, Lynne and Daniel Perkins. 1998. “Evaluating Your Collaborative Effort.” Program Evaluation Newsletter (Volume 1: Number 5). Center for the Study and Prevention of Violence. Used with permission. Jennifer E. Cross, [email protected]
1
32
Level Purpose Structure Process
Networking 1
Communicate for a common understanding
Clearinghouse for information
Create a base of support
Non-hierarchical Loose/flexible link Roles loosely defined Community action is
primary link among members
Low key leadership Minimal decision
making Little conflict Informal communication
Alliance
2
Match needs and provide information
Limit duplication of services Ensure tasks are done Increase cooperation
Central body of people as communication hub
Semi-formal links Roles somewhat defined Links are advisory Group leverages/raises
money
Facilitative leaders Complex decision
making Some conflict Formal communication
within the central group
Partnership
3
Coordinate and share resources to address common issues
Merge resource base to create something new
Central body of people consists of decision makers
Formalized links Defined roles Groups develops new
resources and joint budget
Autonomous leadership but focus is on issues
Group decision making in central and subgroup
Communication is frequent and clear
Coalition
4
Share ideas and be willing to pull resources from existing systems
Develop commitment for a minimum of three years
All members involved in decision making
Roles and time defined Links formal with written
agreement Group develops new
resources and joint budget
Shared leadership Decisions making
formal with all members Communication is
common and prioritized
Collaboration
5
Accomplish shared vision and impact benchmarks
Build interdependent system to address issues and opportunities
Consensus used in shared decisions making
Roles, time and evaluation formalized
Links are formal and written in work assignments
Leadership high, trust level high, productivity high
Ideas and decisions equally shared
Highly developed communication
Our/My Group Name __________________________________________
Group Selected for this Ranking _________________________________
Key words/phrases that describe current relationship________________________________________________________________________________________________________________________
Current Level of Collaboration
1 2 3 4 5
CCOLLABORATIVE RANKING CARD
Key words/phrases that describe desired relationship________________________________________________________________________________________________________________________
Desired Level of Collaboration
1 2 3 4 5
Data are organized in two-way tables where each agency/entity rates the level of collaboration with each of the other agencies/entities. The rows correspond to the groups of raters, the columns to the (same) groups of ratees.
Three C x C tables (where C = # agencies or entities):
1.Current ratings (level of collaboration actually occurring)
2.Desired ratings (level of collaboration rater would like to have with ratee)
3.Difference between desired and current ratings
Summary statistics: Mean current rating, mean desired rating, and mean of the differences between desired and current ratings (SDs also computed)
Desired outcome: Mean of the differences should decrease over time.
Example: In the Community Connections in 2010, the mean current rating was 2.54, mean desired rating was 3.47, and mean of differences was about 0.9. This was based on C = 27 agencies/entities rating each other’s level of collaboration.
See Excel file attached for hypothetical data with 4 raters/ratees.
Collaboration Statistics & Tables
Questions?
Ask State Department Employees whom they have worked with in evaluation in the area where you want to develop more evidence within your program
Ask other grantees for names of people they know or have worked with in a specific area of interest
Invite National, State, and Universities, that may have experts teaching or leading initiatives in the area you want to pursue in greater depth
Conversations with experts at conferences about a direction you would like your program to grow
Engage in conversations that explore ways that you may be collecting additional information
How to start? Gather information
Review your program goals, choose one that you would like to collect data for outcome or process
Review methods or instrument to collecting data that is a good fit
Consult with an expert and discuss a timeline and method
Collect the data
Have the evaluator develop a report based on the data
Include the evaluator in staff, advisory board, partnership and school meetings in person or teleconference
Share the results with others in various media forms and locations
Project Director and Evaluator share insights in how to make program modifications and staff development
Next Steps
Start with most valuable information
Incorporate data-based decision-making
Use results
Begin again with next most valuable information
Incorporate interviews, focus groups, and surveys into the methods used to collect data
Build Relationships of Trust leading to Validity and Reliability
Include all types of stakeholders on an equal basis
Work together in the evaluation process to build collaboration
How to Build a Comprehensive Plan
Barbara Winters
Community Connections
Allamakee Community School District
107 6th St. N. W.
Waukon, IA 52172
Phone: 563-568-4013
Contact Information
I look forward to working together! Enjoy the Journey!!!
Thanks, Barb
Jan Mitchell
Jim Veale