united states department of labor employment & training administration evaluating taaccct...
TRANSCRIPT
United States Department of LaborEmployment & Training Administration
EVALUATING TAACCCT
Kristen MilsteadRegion 2 TAACCCT Roundtable
July 29-31, 2014
2
Topics
• Overview of the National Evaluation
• Managing Your Third-Party Evaluation
• Group Activity
3
Overview of the National TAACCCT
Evaluation
National Evaluation
Round 1
$500 million
49 grants
Round 2
$500 million
79 grants
Round 3
$474.5 million
57 grants
Third-party evaluator required
4
5
National Evaluation: Objective• To develop and implement a mixed-method
evaluation design including an outcome analysis, a formal implementation analysis, and a systematic review of evaluation findings to better understand the TAACCCT grants program
National Evaluation: Research Questions
• Innovations in service delivery and/or system reform? Resulting in capacity building?
• How were models implemented? Can they be replicated?
• Areas for further research?
6
National Evaluation: Activities
• Implementation Analysis– Survey of all colleges– Site visits to selected grantees
• Outcomes Analysis
• Synthesis of third-party evaluation findings
• Support to third-party evaluators
7
8
National Evaluation:Working with Third-Party Evaluators
• House evaluation resources on SharePoint site
• Hold virtual roundtables provide peer-learning opportunities for smaller groups of third-party evaluators
• Provide semi-annual webinars on evaluation topics
• Hold evaluation workshops as a part of national DOL conferences for grantees and third-party evaluators (if feasible)
9
National Evaluation: Reports
• Interim Reports on TAACCCT Evaluation Findings– Round 1 grants (2015)– Round 2 grants (2016)– Round 3 grants (incorporated into Final report)
• Final Report (August 2017)– Analyze implementation and outcome data
from all rounds implementation data and systematic review of third-party evaluation findings
11
Managing Your Third-Party Evaluation
12
Managing Third-Party Evaluations:Most Important Thing!
Ensure the most rigorous, independent evaluation
possible, with both an implementation and
impact/outcomes analysis
13
Managing Third-Party Evaluations:Expectations
• All SGA requirements are met (research questions, outcomes, reports, data, etc.)
• An implementation analysis that answers key research questions
• A participant impact/outcomes analysis that is of sufficient rigor (establishing causal inferences, if at all possible)
14
Managing Third-Party Evaluations:Deliverables
• Evaluation Plan– Round 2: Evaluation Plan
• All grants have been approved to implement evaluation plan
– Round 3: Summary and Detailed Evaluation Plan• All grants can work on evaluation plan, and must rely on the
expertise of the third-party evaluator• Some grants have been approved to implement evaluation
plan• Feedback on detailed evaluation plans• Some grants will be required to submit a revised plan
• Interim Report
• Final Report
15
Managing Third-Party Evaluations:Tips for Implementation Analysis
• Research Questions: Consider additional implementation research questions that are tailored to your TAACCCT-funded activities in addition to the required research questions from the SGA
• Logic Model: Be sure to develop a logic model/theory of change with your third-party evaluator to ensure there is a common understanding of the TAACCCT-funded activities and the intended outcomes
• Capacity Measures: Work with your third-party evaluator to develop measures of how capacity was built at your institution(s) with your TAACCCT funds (e.g., additional class seats/programs, permanent hires, new technology)
16
TAACCCT Conceptual FrameworkCurrent
Workforce NeedsTAACCCT Activities
Capacity-Building Outputs
Long-Term Outcomes
• TAA-eligible and other workers - Need training for new or upgraded skills
•Education and Training Program Development
•TAACCCT Core Elements• Evidence-based
design
• Stacked and latticed credentials
• Online and technology-enabled learning
• Transferability and articulation
• Strategic alignment
• Student/Participant Education Outcomes
•Enrollment
•Persistence
•Completion
•Credentials
• Establishing Innovative Education and Training Strategies
•Expand student services
•Train and hire staff
•Upgrade supplies/equipment
•Build/improve technology infrastructure
•Add new facilities/renovate existing facilities
•Develop articulation agreements
•Build/improve evaluation capacity
•Develop partnerships with employers, other institutions, and the community
• Improved student/participant employment and earnings outcomes
• Skilled workforce to meet employer demand
• Well-connected, efficient systems in community/region
• Increased employer involvement
• Effective training programs at community colleges with capacity to serve students
Contextual Factors: Economic Conditions and Policy/Budget Climate
• Systems - Need to be better connected and more integrated
• Community Colleges - Need to improve capacity to train students for high-demand occupations
• Employers – Need for skilled workers
• Transformation of training for adult workers
17
Managing Third-Party Evaluations:Tips for Implementation Analysis
• Key Informants: Help your third-party evaluator identify key informants/ respondents (e.g., program staff, students, partners) and inform them of the evaluation purpose and activities
• Check-Ins: Build in timely check-ins with your third-party evaluator to understand whether the TAACCCT-funded activities are being implemented as designed (i.e., fidelity to the model)
• Program Changes/Improvements: If changes to the original model have occurred, work with your third-party evaluator to make sure that the planned impact/outcomes analysis is not affected by these changes.
18
Managing Third-Party Evaluations:Tips for Impact/Outcomes Analysis
• Methods: Be educated consumers of the methods selected by your third-party evaluator to assess the impact of your TAACCCT-funded activities on participants and the level of evidence of effectiveness the evaluation is likely to produce for DOL and the field– Weak evidence: outcomes only study (with no
comparison group or a non-matched comparison group)– Moderate evidence: quasi-/non-experimental study (with
matched comparison group)– Strong evidence: experimental design (with random
assignment)
19
Managing Third-Party Evaluations:Tips for Impact/Outcomes Analysis
• Comparison Group(s): Assist your third-party evaluator in identifying appropriate student comparison groups for your TAACCCT-funded activities – Could be historical cohorts or concurrent cohorts within
the same institution or other institutions and in a similar program (by occupation, for example) but not receiving the intervention
– Appropriate = having characteristics, including motivation and ability, that are similar but for the intervention and having sufficient numbers to conduct the impact analysis
20
Managing Third-Party Evaluations:Tips for Impact/Outcomes Analysis
• Data: Serve as an intermediary with departments within your institution and with other institutions or government agencies to help your third-party evaluator access needed administrative data (often better than survey data, esp. for employment outcomes)– Help them develop relationships with your IR office to
access student records– Develop data sharing agreements with the state labor
department to access unemployment insurance wage records
– Connect them with other institutions that may be able to provide student comparison group data
21
Group Exercise – Addressing Evaluation Issues
22
Group Exercise• Create a group of 4-6 people, with people who are
not from your institution• Read and discuss how your group might address
the evaluation issue provided for 10 minutes• Assign a group member to report out to the
audience about your potential solutions and/or strategies to develop solutions
23
Discussing the Issue• Recognize that there are probably no perfect
solutions and you may want to come up with more than one solution, depending on what might work for your institutional circumstances.
• Think creatively about who you might reach out to – within your institution, employer partners, other institutions, other contacts in government – to help you problem-solve.
24
Evaluation Issue #1• Your third-party evaluator provides formative
feedback from their early evaluation work indicating that the students being recruited into the TAACCCT programs are different from what was originally planned. Now, the comparison groups the evaluator wanted to use do not seem appropriate, jeopardizing the impact analysis. How do you help them identify alternatives and/or reconsider your recruitment efforts?
25
Evaluation Issue #2• Your third-party evaluator plans to conduct a
student survey but they are concerned about ensuring a high response rate, especially for students that may no longer be at the institution. How can you help them find and encourage the students to complete the survey? What things could you and your staff do to help early in the project ensure your evaluator can reach these students? Are there non-monetary incentives that could be offered?
26
Evaluation Issue #3• Staff at the Institutional Research Office are
concerned about your evaluator using individual-level student data. How do you address the privacy concerns? Are there others at your institution or at the state level that could address this issue? Are there things you could do early in the project to obtain informed consent for the research?
27
Contact Information
• Questions about the evaluation: • [email protected] (copy your FPO)
• To obtain access to the Sharepoint website:
28
THANK YOU