onc’s workforce program final evaluation report

Click here to load reader

Download ONC’s Workforce Program Final Evaluation Report

Post on 22-Feb-2016




1 download

Embed Size (px)


ONC’s Workforce Program Final Evaluation Report. February 2014. Today’s Discussion . Executive Summary Summary of key findings University-Based Training Program Community College Consortia Program Curriculum Development Program HITPro Competency Exam - PowerPoint PPT Presentation


ONCs Workforce Program Final Evaluation Report

ONCs Workforce ProgramFinal Evaluation ReportFebruary 20141Todays Discussion Executive SummarySummary of key findingsUniversity-Based Training ProgramCommunity College Consortia ProgramCurriculum Development ProgramHITPro Competency ExamCross Cutting Findings and Lessons Learned

22BackgroundONC funded NORC at the University of Chicago to conduct a program evaluation of the four workforce programsContract period of performance: March 2010 through December 201333Research Questions & Data CollectionData Collection EffortsKEY RESEARCH QUESTIONS44Executive SummaryWorkforce Program Overview and Policy ContextOverview of Evaluation MethodsUniversity-Based Training ProgramCommunity College Consortia ProgramCurriculum Development CentersHIT Pro Competency ExamKey Lessons Learned and Cross Cutting FindingsStructure of Final Evaluation Report5Executive Summary66

**Data is from the third cohort only. The first two cohorts asked students whether they were employed in health IT. The third cohort had a follow-up question asking students that answered no to health IT employment, whether they had health IT responsibilities. The measure above reflects students employed in health IT or had health IT related responsibilities. 8

1-99100-199200-399400-9991,000+StudentsCommunity CollegeUniversityStudents Trained for Health IT EmploymentSummary of Key FindingsUniversity-Based Training99UBT Students: Perceptions of Program1089%93%

Students Willingness to Recommend ProgramProgram SatisfactionProgram provided a solid foundation, but students often experienced difficulties transitioning into the health IT fieldEmphasized importance of hands-on experience with EHRsAppreciated opportunities for group work citing soft skills development and exposure to classmates diverse backgrounds as an added benefit.

Students Trained by UBT Employment in Health IT Among UBT Students12

Close to two-thirds of students were employed in health IT or health IT related responsibilities 6 months after program completionEmployed in health IT Working in health IT with the same employer as prior to the programWhat Employers Are Seeking in EmployeesExperience in at least oneif not severalhealth-care setting(s)Hands-on experiences and exposure to various clinical systemsPrograms varied in ability to incorporate internships/practica into curriculum Many distance-based programs identified extensive administrative challenges with coordinating internships across statesEmployers believe these experiences help students by: Expanding and linking clinical and technical backgroundsGiving them an understanding of the needs of clinical staffTeaching decision-making, problem-solving, and soft skills in real-world situations.13Employers Views of the Workforce ProgramMost employers were not familiar with the ONC programEmployers generally thought the training roles aligned well with their employment needs. HoweverJob titles did not always correspond with training roles, making it challenging for students to know what positions to apply forand for some employers to grasp applicants skillsMany employers ideally wanted employees who could cover multiple rolesPrograms need to be nimble and update curricula in real-time to reflect ongoing changes in the industry14Summary of Key FindingsCommunity College Consortia Program1515CCC Students: Perceptions of Program1666%84%

Students Willingness to Recommend ProgramProgram SatisfactionTwo-thirds of students took courses exclusively online (popular for many, but some desired more opportunities for in-person and hands-on training)Preferred for-credit programs Appreciated instructors w/ real-world health IT experienceThose with health care backgrounds found IT course material challenging; those with IT backgrounds found the job market more challenging Many not sufficiently prepared for the difficulty or workload of the courses Posed challenges in light of the six-month timelineEmployers also skeptical about a six-month, non-credit program without a certification

16Key Site Visit Findings: Implementation & Program DesignFlexibility afforded grantees critical to their ability to launch the programs.Several colleges altered the structure of the roles.Collaboration with consortium leads and other member colleges varies across the regions. The majority of faculty members are employed in health IT. Some schools have found students insufficiently prepared for the difficulty of the courses and/or the workload. Students backgrounds have affected their experiences in the classroom as well as their ability to find jobs after the fact.Students referred to two main areas for improvement in the program:additional opportunities for hands-on experience, and a workload that is more appropriate for the length of the program.

17Students Trained by CCC *Only the Cohort 3 survey contained a question for students who were not employed in health IT, which asked if they had any health IT responsibilities. Thus, in addition to the 28 percent of Cohort 3 students who reported employment in the field of health IT, an additional 40 percent reported having health IT responsibilities, increasing the share working in the health IT space. Cohorts 1 and 2 may have had a similar proportion of students with health IT responsibilities had the question been asked of them as well.Percent of CCC Students Employed in Health ITKey Site Visit Findings: Employment Summary of Key FindingsCurriculum Development Centers2121Key Findings: Curriculum Development CentersIn general, instructors and students appreciated the materials and found them comprehensiveInstructors and students noted numerous typos in the first release as well as points of overlap, but this improved in subsequent releasesDevelopers felt the materials should have been developed prior to the start of the program as opposed to in parallel with implementationDevelopers noted that the short development timeline limited collaborationDevelopers struggled to create materials appropriate for the types of students who ended up enrolling in the CCC programs; many wished for more communication with both the CCCs and the HIT Pro Exam grantees22Key Site Visit Findings: Curriculum MaterialsResults from Faculty SurveyNORC invited all 648 CCC instructors to participate in the survey. Survey was in the field from 9/22/2011-1/3/2012460 instructors responded (80% response rate).Summary of Key FindingsCompetency Exam2525Competency Exam: Key FindingsIn 2012, the number of exams administered was lower than expectedDevelopers attributed this to:Colleges not placing an emphasis on the exam The fact that the exam was not a graduation requirementLack of advertising, the absence of a credentialEmployers lack of awareness of the exam Employers remained largely unaware of the exam and what it reflected Providing all exams free of charge and allowing exam takers to sit for more than one exam led to a large increase in the number of exams delivered at the end of grant period26Cross-Cutting Findings2727Cross-Cutting FindingsImportance of communication and clarity of purpose at the outsetRapid implementation posed challenges for structured communication channelsFor example, the Developers felt more communication with the CCCs and the HIT Pro Exam Developer would have helped them better target the materials

ONCs decision to allow grantees flexibility was a great assetThe community colleges and universities were afforded significant latitude in structuring their curricula to meet their needs, capacities, and programmatic prioritiesCCC and UBT students, instructors, and administrators appreciated the opportunity to use online learning platforms

Schools efforts to forge connections with the employer community were of paramount importance to graduates' satisfaction and employment prospectsMany employers were unaware of both the training programs; however, once they learned about them, they felt confident the training could fill gaps in the workforceReaching out to industry experts and employers was an effective means of enriching training, setting up internship programs, and introducing the program to employersHands-on experience helped prepare students for employment; schools with well-developed employer partnerships were better able to support students in this regard

28Matt SwainProgram AnalystONCmatthew.swain@hhs.gov

Kristina LowellPrincipal InvestigatorNORC at the University of Chicagolowell-kristina@norc.org For additional information, please contact:29

View more