moving toward evidence-based practice: innovative ideas for the csbg network
DESCRIPTION
Moving Toward Evidence-Based Practice: Innovative Ideas for the CSBG Network. NASCSP September 2012. Barbara Mooney, Ed. D. Karen Walker, Ph.D. National ROMA Training and Certification Project . Performance? Evidence? . - PowerPoint PPT PresentationTRANSCRIPT
Moving Toward Evidence-Based Practice:
Innovative Ideas for the CSBG Network
NASCSPSeptember 2012
Barbara Mooney, Ed. D. Karen Walker, Ph.D.National ROMA Training and Certification Project
We know we have a groundwork in the CSBG program for Performance Measurement and Performance Management, but how do we build on this foundation to move forward?
Performance? Evidence?
We need strategies for moving the CSBG Network toward greater adoption of evidence-based practices.
Workshop Objectives
Definitions and examples of Evidence-based, evidence-informed
Participants will understand the regional and national factors pushing toward new practice.
Participants will consider how the core activities in IM 49 support universal implementation of high quality practices
Participants will learn about examples of using evidence to evaluate and improve programs
Participants will brainstorm “next steps” for action to move us toward more evidence based practices
What Does it Mean to be “Evidence-based”?
What Do People Mean by Evidence-based?
Multiple terms are in use: Evidence-based practice Evidence-based program Evidence-informed program
…How do we make sense of these terms?
What do we mean by Evidence-based programs and practices?
Evidence-based programs: Interventions that have been found to have
positive effects or impacts on targeted outcomes (Nurse Family Partnership, Parent Management Training)
Evidence-based practices: Intervention strategies or program practices
that have been found to improve outcomes for children and youth (role-modeling, positive reinforcement, avoiding grouping children with problem behaviors together)
Evidence-informed Program
Intervention that has been designed with research evidence in mind. Some of the program’s strategies may be evidence-based practices, they may be informed by the basic research, or both.
Evidence-informed program may also refer to a program that uses performance information to monitor and guide its progress.
Time Out for a brief description of what we mean by Randomized-Controlled Impact and Quasi-Experimental Evaluations
Randomized-Controlled Impact Evaluation Provides most certainty about program
effectiveness, but must be well-designed and well-implemented
Study participants randomly assigned to control or intervention group (think lottery).
Compares the difference in outcomes between the intervention and control group.
Quasi-Experimental Study
Like randomized-controlled impact study (RCT) a quasi-experimental study includes a comparison between people who received the intervention and those who did not.
Unlike an RCT, the comparison is not based on random assignment.
As a result, researchers are less certain that the comparison is a good one.
What Can You Do to Become Evidence-Based or Evidence-Informed?
Adopt evidence-based programs and monitor their implementation to ensure that they are being replicated correctly
Assess “home-grown” programs to determine whether or not they include evidence-based or evidence-informed practices
Set the program on a path toward rigorous evaluation
…These strategies are not mutually exclusive
Collect and Use Data on Performance & Outcome Measures
Conduct Needs
AssessmentIdentify
Population
Becoming an Evidence-Based Program*
Targeting
Conduct a Quasi-Experimental Outcomes Evaluation
Conduct an Implementation Evaluation
Select Intervention, Develop Logic
Model & Identify Indicators
Implement Program Conduct Ongoing
Performance Management
Conduct a Randomized-Controlled Impact Evaluation
Figure adapted from: Performance Management and Evaluation: What’s the Difference by Karen E Walker and Kristin Anderson Moore. Child Trends
Federal, State and Local Efforts and Policies
Widespread move toward evidence-based programs and practice and performance management Federal funding tied to evidence State initiatives and funds for
intermediaries Local efforts include both public and
non-profit efforts Administration’s guidance from OMB to
agencies about encouraging evidence-based efforts.
Evidence-Based Practice – Approaches to prevention or treatments that are validated by some form of documented scientific evidence. These could be findings established through scientific research, such ascontrolled clinical studies or other comparable and rigorous methods.
Evidence-Informed Practice – Approaches that use the best available research and practice knowledge to guide program design and implementation within context. This informed practice allows for innovation and incorporates the lessons learned from the existing research literature.
Definitions from OCS:
14
Use Data to Show Results and Improve Performance
• “Success should be judged by results, and data is a powerful tool to determine results. We can’t ignore facts. We can’t ignore data.”
President Barack Obama July 24, 2009
• “The test of a performance management system is whether it’s actually used….Federal managers and employees at all levels must use performance goals and measures to set priorities, monitor progress, and diagnose problems.”
Chief Performance Officer Jeffrey ZientsOctober 29, 2009
New Guidance:GPRA Modernization Act 2011
Shift emphasis from production of annual reports to collecting and using performance information more frequently throughout the year to set priorities and make decisions that improve results
Establish common Federal Government performance indicators with Quarterly Targets and Quarterly Milestones
The plan to target CSBG resources
“Building on the President's budget request to target CSBG resources to high-performing, innovative agencies, ACF plans to work with Congress and a wide variety of organizational stakeholders to establish a set of core Federal standards that States will be required to use to assess whether an eligible entity is meeting a high standard of service delivery.
These standards of service delivery will include a focus on compliance with basic requirements of the CSBG Act, and will also go beyond basic compliance to focus on high-quality performance across the CSBG Network.
Core performance standards will focus on organizational management issues, as well as strategic process for performance management.” (OCS T/TA overview, 2012)
The CSBG National T/TA Strategy is designed to:
Measure and document performance in accordance with CSBG reporting requirements.
Assure performance measurement and use of performance data at all levels within the CSBG program (local, State, and Federal).
Mitigate risks associated with the use of CSBG grant funds. Promote exemplary practices and innovative programming that
stimulates the creation and sharing of information and knowledge in the CSBG T/TA Network.
Measure and document performance in accordance with CSBG reporting requirements.
Work with States to help ensure CSBG-eligible entity boards know, understand, and meet their fiduciary responsibilities to CSBG (i.e., participation in the development, planning, implementation, and evaluation of CSBG-funded activities and services).
Ensure the management and delivery of CSBG-funded services creates and strengthens sustainable economic resources in communities
ROMA Next Generation and Organizational Standards
Two targeted T/TA Centers of Excellence embody OCS focus for coming years
Organizational Standards
The CSBG T/TA Organizational Standards Center of Excellence (COE) will coordinate the development and dissemination of a core set of standards with input from the State CSBG Lead Agencies as appropriate, regional and national partners, including the CSBG T/TA Regional Performance and Innovation Consortia (RPIC).
The central mission of this effort will be to ensure that all CSBG-eligible entities have the capacity to provide high-quality services to low-income individuals and communities, and are able to meet high-quality organizational performance standards.
The movement toward continued accountability, risk mitigation, and performance plays a key role in developing organizational standards toward improved performance management and the capacity to thrive within the CSBG Network.
ROMA Next Generation
Progress has been made in recent years in the quality and breadth of State and local performance reporting systems through ROMA and the NPI framework… (which) provides a foundation for additional targeting of resources based on outcomes and effectiveness of services.
However, practices of data collection and reporting within the ROMA system vary from State to State; while some States‘ reporting systems reflect the full ROMA cycle, other States only report NPI data.
Such inconsistencies within the ROMA system minimize the capacity of ROMA to be an integrated and comprehensive effort that lends to accurate and reliable performance management and accountability within the CSBG model.
Key goals of the ROMA Next Generation COE will be:
To review and strengthen the current performance measures and performance indicators with input from experienced CSBG organizations, as well as performance management experts from outside the CSBG Network;
To work with State CSBG agencies to communicate consistent standards for performance management throughout the ROMA cycle, including community needs assessment, program planning, and performance measurement;
To assure a coordinated, national strategy for nationwide implementation of an updated performance management system;
To assure that a performance management system can be used, as necessary, for comparison of the effectiveness of local entity use of CSBG resources.
We are keeping the basics:
Six National Goals ROMA Cycle (core activities for
eligible entities) Core activities for state offices Checklist for Monitoring (IM102) Risk Mitigation (IM 112) Key elements of the Logic Model
Six National Goals
1. Low-income people become more self-sufficient.
2. The conditions in which low-income people live are improved.
3. Low-income people own a stake in their community.
4. Partnerships among supporters and providers of services to low-income people are achieved.
5. Agencies increase their capacity to achieve results.
6. Low-income people, especially vulnerable populations, achieve their potential by strengthening family and other supportive systems.
24
IM 49 Core Activities for State Offices
25
1. The state has developed, in coordination with eligible entities and the State CAA association, a State-wide vision statement that speaks to the goals and purposes of community action within the State and that supports the 6 national ROMA goals. The state is encouraged to participate in, and contribute to, broader State anti-poverty/community development initiatives with outcome measures and goals compatible with ROMA;
2. The agency has trained all its eligible entities (staff and boards) in outcome-based management, and that 80% of the entities use ROMA concepts to guide needs assessment, agency mission review, activity planning, resource allocations, service delivery, measuring and reporting results;
Core Activities for State Offices (cont)
26
3. 80%of the plans and program reports received from eligible entities in the State describe plans to achieve projected outcomes, and evaluate results based on measurable improvements of condition(s) among clients and/or communities served;
4. The agency submits complete, accurate, and timely annual reports to OCS on the measured performance of the State and the eligible entities in the State as required by Section 678E of Public Law 105-285, the Community Services Block Grant Reauthorization Act of 1998.
ROMA Logic ModelIdentified Problem,
Need, Situation
Service or Activity (Output)
Identify the timeframe.
Identify the # of clients served or the # of units
offered.
Outcome (General statement of results expected)
Outcome Indicator Projected # and % of
clients who will achieve each outcome.
Actual Results
Actual # and % of clients who achieve
each outcome.
Measurement Tool
Data Source, Collection Procedure,
Personnel
Frequency of Data Collection and
Reporting
(1) Planning (2) Intervention (3) Benefit (4) Benefit (5) Benefit (6) Accountability (7) Accountability (8) Accountability Short Term
Short Term
Short Term
Intermediate Term Intermediate Term
Intermediate Term
Long Term
Long Term
Long Term
CommunityFederal, State and Local Policies PopulationCulture (norms and behaviors) Funding
OrganizationMission & Goals Leadership ResourcesPolicies Program Monitoring Staff assessments
ProgramActivitiesStaff roles and skillsTrainingQuality and FidelityParticipationOutcomes
28
ExamplePerformance Management In Action
After-school tutoring program assisting at-risk youth – attempting to improve their math and reading performance.
Collect the following data: Demographics and risk/protective factors Program attendance Amount of service (hours and topics of
tutoring) Report card grades (first / last grading
period) 29
Example (cont.)Performance Management in Action
Finding #1: Halfway through program, program attendance is low among youth from Neighborhood A. Action: Reach out to teachers in schools in
Neighborhood A Action: Talk with youth in Neighborhood A
Finding #2: Math grades increased, but reading grades did not change. Action: Investigate. Finding: Tutors unfamiliar with reading curriculum Action: Train tutors in reading curriculum
30
Executive leadership To set the tone, to implement policies
Staff expertise To identify measures, to collect data, and to analyze
data Staff time
To collect, enter, analyze, share, and discuss data Technology
Forms (paper or electronic), spreadsheet software, management information systems/commercial software
Consider Resources for Performance Management
31
Performance management policy outlining: Staff roles and responsibilities related to
performance management Description of data collection and reporting
methods How often regular meetings should take place
to review and discuss findings Performance appraisal process which
evaluates staff on how well they carry out performance management roles and responsibilities
Create Policies for Performance Management
32
Are we reaching our intended population with the intended services?
Are participants getting the amount of program time we want them to get?
Are participants engaged in the process? Are participants’ outcomes changed? Do our staff carry out activities as intended? Are we providing the training our staff need?
If the answers are no, organizations need to determine why and what needs to change
Example Questions in Performance Management
34
An example about how sharing performance management data on outcomes can increase staff buy-in, inspire youth, and create a learning culture
Video Clip from “Saving Philanthropy”
http://www.savingphilanthropy.org/preview-footage/
Implementation will take resources
Time and attention to develop a plan of action
Staff time and support (travel, meeting costs, provision of materials, etc.) At the state level, for local EEs
Follow up to evaluate actions and assess new needs/resources
T/TA from National ROMA Trainers
The demand for training and technical assistance exceeded the available resources in the ROMA Trainer network to address the needs.
Two specific areas were identified: The certified trainers do not have any resources to bring local CAAs
together as they begin to implement the full range of ROMA activities. Implementation support is critical to working out solutions to local challenges and establishing a consistent use of terms, concepts and processes.
NCRTs do not have the time or the expertise to provide individual on-site technical assistance to local CAAs as they apply the principles to agency systems.
We are working to increase the number of available NCRTs in partnership with the RPICs and State Offices
State CSBG Discretionary Dollars
NASCSP Resources on discretionary funds: Annual Reports and appendix tables Issue Brief:
State Discretionary Spending
Distribution of FY 2011 CSBG Discretionary Funds by Purpose
CSBG National Resource Center & Clearinghouse
Resources online Evidence-based discussions Forums Lists of consultants
Child Trends’ Resources
Using Data in Multi-Agency Collaborations to Ensure Accountability and Improve Performance. http://www.childtrends.org/Files//Child_Trends-2012_02_23_FR_UsingData.pdf
Performance Management and Evaluation: What’s the Difference? http://www.childtrends.org/Files//Child_Trends-2011_01_19_RB_PerformMgmt.pdf
at www.childtrends.org
Leap of Reason: Managing to Outcomes in an Era of Scarcity, available at leapofreason.org
What’s next?
How can you identify and focus resources so our network can move toward evidence based practices?
What specific steps do you need to take to begin to identify what works and how to collect the evidence?
What do you need to consider?
Steps toward improving practice
IMPROVING REPLICATION AND “SCALE-UP”
The history of replication and scale-up efforts indicates the difficulty associated with these efforts.
As President Bill Clinton frequently declared, “You can find virtually every problem in our country solved by somebody somewhere in an astonishingly effective fashion….The challenge for us is to figure out how to replicate that.”
The day before his first inauguration, Clinton told the nation’s governors that his number one disappointment as governor had been that it was so hard to “take something that works to the next level.” He said he “could never figure out a way to make the exception the rule, and that is our enduring problem in America in public life.”
For more discussion:
Barbara MooneyProject Director
National ROMA Training and Certification Project243 E. High St.Waynesburg, PA
Karen Walker Senior Research Scientist and Senior Program
Area Co-Director
Child Trends 4301 Connecticut Ave NW
Suite 350Washington, DC 20008