findings from the national evaluation of child welfare training grants: lessons and implications...
TRANSCRIPT
Findings from the National Evaluation Of Child Welfare Training Grants:
Lessons and Implications
Mary Elizabeth Collins, MSW, Ph.D.Maryann Amodeo, MSW, Ph.D.Cassandra Clay, MSW, EdM.
Boston University School of Social Work
May 2007
Grant #9OCT0124 funded by U.S. DHHS, Children’s Bureau
National Evaluation of Child Welfare Training Grants
Funded by the Children’s Bureau (Oct 2003- Sep 2006)
Large project: multiple data collection tasks and several research questions
Overall goal: Greater understanding of federally-funded CWT and how to use information from the evaluation to improve our training strategies.
National Evaluation of Child Welfare Training: Four Main Components
Case studies of 9 previously federally funded training projects – Independent Living
Comprehensive review of child welfare training literature
Survey of faculty of (mostly) schools of social work
Survey of state (and some county) child welfare training administrators
Update on Four Components
Multiple case study of IL projects – DONE, report available
Review of child welfare training literature – DONE, report available
Phone interview of state child welfare training directors – 90% done, response rate=48/51, great data!
Mail survey of faculty – 60% done, N=97; response rate=54%
Conceptual Model CONTEXTUAL FACTORS TRAINING PROJECT
ACTIVITIESINDIVIDUAL PROJECTOUTCOMES
CLUSTER OUTCOMES LONG TERMOUTCOMES
State Agency Nature of collaboration Level of personnel
involved Organizational
constraints Training policies Project match with
agency need Children and Family
Service Reviews
Children’s Bureau RFP Grantees’ meeting
Grantees’ Organization Level of support Experience with CWT Seniority of project staff
State-Level Issues Politics Funding Regulations and laws County, Tribe or
community issues
Other (media, role oftechnology)
Develop Materials Curricular
development skills
Deliver Training Trainer skill Number of trainees Cost Effectiveness
Involve Youth Planning training Delivering Training Other
Evaluation of Project Process Outcome
Disseminate Findingsand Materials Presentations Articles/reports Curricula/tapes
Collaboration Development
Other Project Activities
Worker Skills Knowledge Attitude Behavior
Supervisor Skills Knowledge Attitude Behavior
Training of Trainers Knowledge Attitude Behavior
Agency Culture Youth focused
Impact on Youth Skill development Empowerment
KnowledgeDevelopment(evaluation)
Knowledge Sharing(dissemination) Institutionalization of
curricula
Collaboration Institutionalization of
collaboration
Other Outcomes
Literature Review Survey of Schools of
Social Work Survey of Child Welfare
Administrators
Improvement in AgencyPractice
Improvement in School ofSocial Work Youth-focused curricula
Improvements in the ChildWelfare Field (privateagencies, ancillaryagencies, child advocacyorganizations)
Improvement in thePositive YouthDevelopment Field
Other Outcomes
Improvement in Child,Youth, and Family Well-being
Recommendations from Case Study Report: Children’s Bureau, Future Grantees, Public Agencies
Youth/Consumer Involvemente.g., Public agencies have been making progress in emphasizing more partnership with services users; training initiatives and strategies are another mechanism by which agencies can engage consumers in their work.
Evaluatione.g., Public child welfare agencies need to cooperate with the evaluation task. This will include allowing evaluators to conduct follow-up with trainees from the state agency. Examination of the transfer of learning to the agency setting and mastering skills in the work setting is sorely needed. This will nearly always require follow-up in the practice setting which will include data collection involving interviews, observations, case record reviews, and other methods. Additionally, access to comparison groups may be needed.
Collaboratione.g., Grantees should aim to establish collaborations with organizations rather than individuals to provide more stability to the collaborations. Collaborators should be chosen in part to facilitate long term institutionalization. Thus, at the start of projects, active advocacy should occur to secure organizational collaborators who will share the work and responsibility for outcomes
Recommendations from Case Study Report: Children’s Bureau, Future Grantees, Public Agencies
Institutionalizatione.g. The Children’s Bureau should aim to fund the type of training projects that are of critical need to public child welfare agencies. If the funding priorities of the Children’s Bureau are not aligned with the needs of the field, public child welfare agencies have no reason to engage in long-term institutionalization of training programs. After projects are completed, the Children’s Bureau is the only entity with appropriate infrastructure to keep the products that have been developed at the forefront of child welfare practice.
Knowledge Developmente.g., Grantees should recognize that their projects are opportunities to develop learning about the field of child welfare training that can and should be shared with wider constituencies. Although “lessons learned” are typically requested as a part of project reporting, the content is generally thin and lessons are not shared. Grantees should think conceptually about the core lesson of their project and disseminate the contributions via conference presentations and journal articles. These should be less focused on promoting projects and more focused on linking project innovations to the wider field of child welfare training.
Findings from the Literature Review Adult Learning Theory
e.g., Organizational realities that are barriers to training based on these theories need to be confronted head-on. These barriers include time and space for sophisticated training; professional orientation toward workers that recognizes their expertise; and development of a learning culture in which organizational learning and development is the norm. Without addressing these barriers, training programs will have limited impact.
Training Implementatione.g., The field should move toward the development of training systems rather than training courses. Training systems infuse training with an organizational context and have greater potential for viewing training holistically, with connection to achievement of organizational outcomes. This will add to better conceptualization of the purposes of training and the anticipated linkage of training content and training activities with expected outcomes
Findings from the Literature Review Agency/University Partnerships
e.g., As with other forms of training, better evaluation is needed to assess the results of partnership efforts. Most of the literature emphasizes the benefits of partnerships, but there are limitations. Sometimes these are discussed as logistical problems or the melding of academic and practice cultures. Partnerships may limit the voice of schools of social work to critique child welfare practice and policy, and may cause schools to shift toward vocational education rather than professional education.
Evaluation of Traininge.g., The methodology for evaluating outcomes related to knowledge and attitude change are fairly well developed and easily implemented. The measurement of skill development is more complicated and requires more methodological sophistication. Measures such as case plans and action plans should be more fully developed for use in training evaluation. Also observational methods and measures should receive attention so that the field can begin to measure the interaction of worker and client.
Survey of State and County Child Welfare Training Administrators -- Content
Main challenges agency faces in providing effective CWT Experiences with federally-funded CWT
Description Strengths/Weaknesses Impact
Involvement with IV-E partnerships Description Strengths/Weaknesses Impact
State/county funded training programs
Survey of State and County Child Welfare Training Administrators -- Content
Administrative supports for training Impact of CFSRs on trainingExisting mechanisms to institutionalize trainingRelevant historical factors that have influenced CWTFuture initiatives in CWTTraining evaluationWhat could federal government do to help states and
counties deliver effective CWT?What should be federal priorities (topics) of CWT in the
next few years?
Survey of State and County Child Welfare Training Administrators: Training Evaluation
Many (n=18) refer to “levels” of evaluationLevel 1/satisfaction, n= 8; “Would like to do more”Level 2/pre-post change, n=13 with some moving
toward more complex evaluation for some training.Additional evaluation steps e.g., longitudinal follow-up,
n=20Most developed training systems, n=4
Survey of State and County Child Welfare Training Administrators: Training Evaluation
“The success of training is first measured when trainees take a test. … followed by a field based assessment piece, which is left up to various supervisors. The supervisor’s evaluation includes an evaluation form, document review and observation of workers in various case management activities. Also an online tracking system, which tells them who passes competencies and other information.”
Survey of State and County Child Welfare Training Administrators: Training Evaluation
“Evaluation method varies based on the kind of program. Some Level 1 (happy sheets) that are standardized. Available electronically—when a course is offered, evaluation information is put in immediately, can access it. Use pre/post knowledge test for basic training and all specialty programs. Occasional attempts at level 3 & 4 for very special situations; it’s expensive and hard. Also have a system to do field evaluations – conduct an on-site field review, either scheduled randomly or a project manager requests it.”
Survey of State and County Child Welfare Training Administrators: Training Evaluation
Utilization:
Generic statement about reviewing results (n=9)
Primarily to review courses and trainers (n=23)
With information system to tract participation, link with retention, overall quality assurance (n=12)
Survey of State and County Child Welfare Training Administrators: Training Evaluation
The information is compiled and examined quarterly. It is then utilized for the following: 1. to look at trainers; 2. are workers getting what they want? (i.e., if they need to change curriculum); 3. how is the environment? (are environmental changes needed). Future plans include a practice reform initiative with [the University] which will have its own formal evaluation and they are working towards developing a model that will include behavioral anchors for new workers.
Survey of Colleges/Universities: Relationship with Child Welfare Agency
N=97
Your Relationship Your School’s Relationship
Quality 3.46 3.16Amount 3.22 2.97Mutuality/Reciprocity 3.09 2.83Productivity 3.13 2.82Desire for Future Collaboration 3.73 3.44
Scale: 1 (low) – 4 (high)
Survey of Colleges/Universities: Perceived Impact of Federally-Funded Projects (N=30)
Moderate/Strong
n percentRecruitment of new CW staff 15 50%Development of current CW staff 26 87%CW worker knowledge 30 100%CW worker attitude 23 85%CW worker skills 27 90%CW supervisor knowledge 22 76%CW supervisor attitude 16 59%CW supervisor skill 18 69%Macro level CW policy/practice 11 47%
Triangulation: Common Findings
Reluctance of state agency to fully support training and its evaluation
Potential and pitfalls of partnerships between agencies and universities
“Training” knowledge dominated by curriculum development and training delivery; expansion needed in evaluation, organizations, and systems
Training knowledge dispersed in variety of fields and ranges from highly micro (e.g., instrumentation) to macro (e.g., policy implementation)
Discussion Topics
Relative roles of faculty, state/county training agencies, and the federal government in advancing the state of training.
Enhancing the knowledge base of child welfare training.Addressing conceptual and methodological challenges
to conducting training.Assessing what is known and unknown in child welfare
training.Challenges of linking training outcomes to performance
outcomes.
For more information, contact:
Mary Elizabeth Collins, Ph.D.Boston University School of Social Work264 Bay State RoadBoston, MA [email protected]