Download - Case Study #9
Case Study #9Jackie Adams: Evaluating a Federally Funded Faculty Training Program
Melody Murphy AIL 606The University of Alabama
Part One: Creating an Evaluation Plan• Jackie Adams has a instructional
design position with a large university for a federally funded project.– Her main responsibilities were to work
with technical subject matter experts in the development, delivery, and evaluation of in-service faculty education.
• The name of the project Jackie was working on was the Advanced Manufacturing Technology Education (AMTE).
Part One: Continued• The goal of the AMTE project was one
of many projects funded by the Advanced Technology Education (ATE) program. – Its goal was to improve science and
engineering technician education at the undergraduate and secondary school levels.
– Also, it focused on curriculum development, instructional materials development, teacher/faculty enhancement, and/or student recruitment.
Main Goal• Its main goal was to focus on teacher
enhancement as a means of advancing technology education.– The rationale behind AMTE was to provide
educators with state-of-the-art knowledge in the technical disciplines, so that their students would in turn benefit from their teachings.
• Over a three-year grant, the AMTE project was to provide faculty development to 100 science, math, and engineering technology educators each year.
Progress• Jackie had been working on the
project for about four months when Ray DeMilo (her boss) asked her to develop an evaluation plan for the project.
• In order for them to receive funding for the project an evaluation plan was going to be required to assess the performance of their project.
Jackie’s Plan for the Evaluation Plan1. Give it to her boss.2. Send it to the funding agency.3. File it with the university grants
office.4. Use it to conduct the evaluation.
Jackie’s Planning Evaluation Notes1. Why Evaluate?2. Evaluate What?3. How should the evaluation be
conducted?4. Who should be involved?5. When should it be done?
Preliminary Analysis Questionsfor Part One – Question #1
Based on Jackie’s questions, what does she need to do to complete the evaluation.
First of all Jackie decided that on why the program must be evaluated. It should be evaluated for the ongoing funding of the program and to measure whether or not learning is taking place in the workshops
Preliminary Analysis Questionsfor Part One – Question #2
How can Jackie determine whether short- and long- term outcomes are being achieved? What
evaluation data does she need to collect? Short-term• Jackie can conclude by receiving the
surveys from the faculty about how they are using what they have learned in the workshops to conclude that by what they have learned they can use in their classrooms.
Long-term• Jackie can conclude from the surveys that
if the faculty can continue or use the information now that they can use then the faculty can use it for long range planning in their classes.
Preliminary Analysis Questionsfor Part One – Question #3
How should Jackie design the instruments for this evaluation?
Based on the material taught, she wants to know what the learners have gained from the workshops in order to report to the team that is reviewing her project in order to keep the grant money.
Part Two: Conducting a Meta-Evaluation
Conducting the Evaluation• The funding agency sent out a six-
member team to review the project’s progress and impact.
• The team would report the success of the project to key administrators at the university and the funding agency.– The team included four post-secondary
educators with a background in science, engineering, and technology and two engineering practitioners.
Team Review• During the two-day meeting, Hank
spent the entire first day with Jackie.• He reviewed the progress and impact
of the project.• At the end of their meeting, Hank felt
that confident that he had enough information to start the review.
Preliminary Analysis QuestionsQuestion #1Part A: Questions Related to the Evaluation Plan
What are the main evaluationquestions Jackie appears to be addressing? Why – why evaluate the project?What – what to evaluate in the project? What should
be measured?How – How should it be evaluated?Who – Who should be evaluated?When – When should it be evaluated? Periodically?
Preliminary Analysis QuestionsQuestion #2
What are the strengths and weaknesses of the evaluation design for answering the questions?Strengths – the evaluation plan answers why
evaluate the objectives (reason for evaluation), purpose, when it should be done which is periodically.
Weaknesses – How it should be done. It says documentation but it does not specify the
documentation used or the learners in Jackie’s evaluation plan.
Preliminary Analysis QuestionsQuestion #3
Given the goals stated in the ATME proposal, are these questions sufficient?Yes, it does give the basic why, what, how, who, and
when.
Preliminary Analysis QuestionsQuestion #1Part B: Questions Related to Instrumentation
How appropriate are the instrumentsfor the goals of the evaluation?I question the validity. This is a lot of Jackie’s opinion on the evaluation plan. I would probably recommend incorporating interviews and surveys to the evaluation plan.
Preliminary Analysis QuestionsQuestion #2
Are the evaluation questions answered in enough detail, or are other instruments needed?• I felt like there could have been more detail, for
example, when stated “determine how disciplined and effective the organization’s operations are.” How will these be determined?
Preliminary Analysis QuestionsQuestion #3
What, if anything, would you change about the instruments and why?
I would add a survey to be given to the faculty/staff. For reinforcement for what is stated in the evaluation plan.
What are the implications for instructional design practice in this case? • The instructional design of high-tech
areas using computers.– For example computer numerical control,
programmable logic controllers, robotics, electromechanical controls, lasers, solid modeling, and rapid prototyping in the workshops conducted by Jackie.
As an instructional designer, what did I learn? • In order to keep your
program or project funded it is important to regularly evaluate it.– Ask questions such as:
– Why, what, how, who, and when.
My Questions1. How important do you think
evaluation plans are to instructional design projects?
2. What types of instruments would you recommend for Jackie to use in helping with the evaluation of her project?