What Makes Good Evaluation? - Purdue Agriculture Evaluations.pdf · What Makes Good Evaluation? ... knowledge about garden ... 挀栀愀渀最攀搀屲 Pre/ and Periodic observation
29
What Makes Good Evaluation? Developed by Susan Barkman , 4-H Department, Purdue University Gearing Up to Report Program Impact
Developed by Susan Barkman , 4-H Department, Purdue University
Gearing Up to Report Program Impact
Presenter
Presentation Notes
This morning you heard about IRB and the process that program evaluation must go through before using with your clientele. This afternoon’s session is designed to help you GEAR UP to design good evaluation tools that will yield valuable impact data. I can’t make you an expert in 45 minutes, but this will serve as a great start. I would encourage you to sign up for any of the evaluation staff development programs that CES offers: FOR APRIL 12 & 19 MEETINGS ONLY - Because of the ice and snow storm on March 28, the Using Qualitative Methods to Measure Program Impact Training has been rescheduled for April 30 at the Marion County Office 10 – 3:30 pm. Just let me know if you are interested in attending. ALL OTHER SESSIONS: We will probably be holding two different sessions next year on designing both quantitative and qualitative instruments. So watch the Staff Development offerings.
“The systematic collection and analysis of information to determine the worth of a curriculum, program or activity.”
Alkin 1990
Presenter
Presentation Notes
Evaluation means different things to different people. Many definitions have been been developed, the one most appropriate for use within the Extension is: “The systematic collection and analysis of information to determine the worth of a curriculum, program or activity.” ( Alkin 1990) The word systematic means the evaluation must be planned as in integral part of the design or planning process. It is not just an “event” that occurs at the end.
Worth can be a measure of:
The quality or satisfaction level leading to program improvement.
The effectiveness as a means to demonstrate the impact of the program or that the targeted outcomes have been achieved.
Presenter
Presentation Notes
Worth can be a measure of: The quality or satisfaction level leading to program improvement 2.The effectiveness as a means to demonstrate the impact of the program or that the targeted outcomes have been achieved Both kinds of “worth” are important. Let’s talk about evaluation leading to program improvement first.
Program Improvement Evaluation
What did you like best (least) about the program?
How would you improve the program?
Presenter
Presentation Notes
This is the end-of-program evaluation we have done in extension for years. We ask questions like: What did you like best about the program? What did you like least about the program? What was the most valuable part of the program? How would you improve the program? OR
Program Improvement Evaluation
Please rate the following sessions?
Poor Fair Good Excellent
Opening Session 1 2 3 4
Applying Pesticides 1 2 3 4
Planning Your Budget 1 2 3 4
Gardening with Kids 1 2 3 4
Closing Session 1 2 3 4
Presenter
Presentation Notes
You might have a list of the various sessions or components of your program and ask participants to rate them each on a 4 or 5 pt. scale. The data collected from this type of evaluation is primarily used to help you plan and implement the program again. This information is NOT used in SAM or impact reports. As was mentioned earlier in the day, this kind of evaluation (program improvement) DOES NOT have to go through an IRB review process. So, let’s take a look at the kind of evaluation that can be used to demonstrate the impact of the program.
Program Impact Evaluation
Has the program achieved it’s targeted outcomes?
How have the participants changed as a result of this program?
Presenter
Presentation Notes
What Programs Should Be Evaluated? Are participants involved in at least 2 or
more hours of educational programming?
Does the program have clearly stated outcome(s)?
Do your stakeholders care about whether this outcome is achieved?
Is the program addressing an issue that the community cares about?
Presenter
Presentation Notes
Targeted Outcome
The “change in participants” you would expect to see as a result of participating in the program.
what are the participants going to:know, feel or do different
after the program
Presenter
Presentation Notes
Awareness Created
Knowledge Gained
Attitudes Changed
Skills Developed
Aspirations Sparked
Behavior Changed
Practice Adopted
Decisions Made
Policies Changed or Adopted
Social Action Initiated
Social Impact
Economic Impact
Civic Impact
Environmental Impact
LEARNING ACTION IMPACT
Targeted Outcomes
Presenter
Presentation Notes
Farmers adopt new management practices that reduce risk to water quality
(ACTION OUTCOME – practice adopted)
Water quality improved
(IMPACT OUTCOME – environmental)
Presenter
Presentation Notes
Teen mothers ate healthier and did not consume alcohol, tobacco and other drugs
(ACTION OUTCOME – behavior changed)
Babies live
IMPACT OUTCOME social
Babies have to stay less days in hospital
IMPACT OUTCOME economic
Presenter
Presentation Notes
You Be the Judge?
Master Gardeners will increase their knowledge about garden management.
Increase the number of schools that offer school enrichment programs.
Adults will develop a more positive attitude toward foods that contain products derived from biotechnology.
Presenter
Presentation Notes
Ok -- let’s practice a little. I’m going to give you a statement and I want you to: Identify if it is an clearly written outcome What level it is And how you might measure if the outcome has been achieved (INTERACTIVE DISCUSSION) Master Gardeners will increase their knowledge about garden management. yes (but you must clearly know what garden management knowledge you want the to know in order to measure it. LEARNING – knowledge gain Pre/post test (if you only do a post test your can not say they gained the knowledge from your program ) Increase the number of schools that offer school enrichment programs. No – this is what you are going to do as an educator Adults will develop a more positive attitude toward foods that contain products derived form biotechnology. Yes LEARNING – attitude changed pre/post attitudinal survey
You Be the Judge?
Adults will adopt practices that will decrease the amount of drift when applying pesticides.
The educator will create an awareness and knowledge of gang related symbols.
Farmers will improve their farm financial management decisions.
Presenter
Presentation Notes
(INTERACTIVE DISCUSSION) Adults will adopt practices that will decrease the amount of drift when applying pesticides. yes (but you must clearly identify what those practices area: drift reducing agents, drift reducing nozzles, not spraying on a windy day) ACTION – practices adopted A post self-report questionnaire( sent out after next growing season) or use a end-of-program survey to measure intentions to adopt practice (if you did this you would be to reword the outcome) The educator will create an awareness and knowledge of gang related symbols. No – this is what you are going to do as an educator. Farmers will improve their farm financial management decisions. Yes ACTION – decisions made; you may also be and ECONOMIC IMPACT Self-report survey or farm records
You Be the Judge?
Parents will use positive parenting techniques.
Youth will improve their classroom behavior.
Increase involvement of citizens in community affairs.
Presenter
Presentation Notes
(INTERACTIVE DISCUSSION) Parents will use positive parenting techniques. No – Reword to” Parents will increase their positive parenting skills.” LEARNING – skills developed A pre/post self-report questionnaire (P.E.P.P.E.R) Youth will improve their classroom behavior. Yes ACTION – behavior changed Pre/ and Periodic observation checklist Increase involvement of citizens in community affairs. Yes ACTION – decisions made; you may also be and ECONOMIC IMPACT Pre/post survey, possibly at post only depending on how worded
Evaluation Methodology(when and how you are going to administer)
Post-Test only
Pre and Post-Test
Presenter
Presentation Notes
Before you begin to actually design your evaluation instrument, you need to determine when and how you are going to administer your evaluation This is called the evaluation methodology. The four most common are described below. Most extension field staff would use one of these methodologies. Post-Test only. Since you are only collecting data once, you can NOT calculate the change that occurred in participants. You can only state that participants have a certain level of knowledge, they can perform a skill or act a certain way. This is not a very rigorous method, but remember schools do collect most classroom data this way. Pre and Post-Test - The evaluation is conducted both before and after the program is completed. The two scores are then compared indicating whether the program changed participants’ knowledge, attitudes, skills and/or behavior. This is more rigorous than post -test only. However, remember that the change in scores could be influenced by other factors (other programs the participants attended, the media, etc.). You could strengthen the rigor of this methodology by adding testing a similar comparison group (a group that takes the pre/post during the same period, but does not participate in the program.
Evaluation Methodology(when and how you are going to administer)
Post/Post Test
Post-Retro Pre
Presenter
Presentation Notes
Post/Post-Test - This is similar to the pre/post-test methodology except that the evaluation is conducted immediately following the completion of the program and then again after a period of time (i.e. 30 days, 6 months). The two scores are then compared indicating whether the program changed participants.’ knowledge, attitudes, skills and/or behavior. This design helps eliminates the problem of having participants over estimate their skills and ability, which sometimes occurs with a pre-test. Then once they participate in the program they realize that they didn’t know or have the skills they thought they did. Post-Retro Pre* - The evaluation is conducted with participants at the end of a program. The instrument is design to have the participants rate their knowledge, skills, or attitudes at the present time and how they would have rated themselves before the program. Like the pre/post test, the two scores can be compared to indicating whether the program changed participants’ knowledge, attitudes or skills. This method is easy to use, but often participants tell you want they think you want to hear - they have changed by this program, therefore does not have high rigor. SPECIAL NOTE*: This method should not be used with anyone under age 15. Developmentally, youth have a hard time distinguishing between what their knowledge, skills and attitude were before with those they now have.
Measuring Gains in Knowledge
• True-False
• Matching
• Multiple-Choice
• Completion
Presenter
Presentation Notes
Let’s take a look at how we can design good evaluation instruments that help us measure whether out programs are achieving their targeted outcomes. A pre/post test is the most common way to measure knowledge. Basically there are four different kinds of questions you can use: True-False - easy to write and score; tends to promote memorization and encourage a high degree of guessing, so you need a large number of questions to gain a true picture of what the participants knows Matching – easy to write and core ideal for measuring associations between facts; tends to emphasize memorization Multiple-choice – considerable versatility in measuring outcomes; designed to make participants discriminate among options that vary in degree of correctness thus a higher level of cognitive thinking than T/F questions; time consuming to write and have to be carefully written Completion – relatively easy to write, but sometimes hard to score because there can be more than one defensible answer; restriction of an answer to a few words t measure the recall of specific facts,. Names, places and events as opposed to more complex items
Measuring Attitudinal Change
BELIEF
Genetically-modified foods are inferior to foods from plants developed using conventional methods.
FEELING
I like to do science experiments.
Presenter
Presentation Notes
A pre/post survey is the most common way to measure a change in attitude. Basically there are three different kinds of questions you can use: Cognitive items – express beliefs about the attitudinal object Genetically-modified foods are inferior to foods from plants developed using conventional methods. Affective items – express a feeling toward an attitudinal object I like to do science experiments.
Measuring Attitudinal Change
BEHAVIOR INTENTION or PREFERENCE
Would – I would serve genetically-modified foods to my family.
Should – The government should ban production of genetically-modified foods.
Presenter
Presentation Notes
Conative items – express behavioral intention or behavioral preference with regard to the object. Would items express a behavioral intention to do something. Should items express a behavioral preference for a social action. Would – I would serve genetically-modified foods to my family. Should – The government should ban the production of genetically-modified foods.
Measuring Skills and Behavior Changes
Pre/post or post/retro pre questionnaire
Observe the participants
Presenter
Presentation Notes
Skills and behaviors are usually measure in one of two ways. You can use: a pre/post questionnaire or post/retro pre questionnaire actually observe the participant performing the skills or doing the behavior.
Measuring Skills Developed
How comfortable do you feel about your ability to lead a small group activity.Very uncomfortableSomewhat uncomfortableSomewhat comfortableVery comfortable
Presenter
Presentation Notes
Here’s and example of a pre/post questionnaire where the questions are designed to assess the participants perceived degree of comfort in performing the skill. How comfortable do you feel about your ability to lead a small group activity. Very uncomfortable Somewhat uncomfortable Somewhat comfortable Very comfortable
Measuring Skills Developed
Please circle the letter that best describes your ability to do the following things:
1. Finding someone’s email address on the web.2. Using HTML to design a simple web page.3. Adding motion and sound to your web page.
Cannot Much help Maybe Little help Can Do It needed needed Do It
Presenter
Presentation Notes
Measuring Skills Developed
Please rate your skills after the training and what you thought they were before the training
BEFORE AFTERTRAINING TRAINING
None A lot None A lot
Ability to design a 1 2 3 4 5 1 2 3 4 5quantitative instrument
Ability to conduct a pilot test 1 2 3 4 5 1 2 3 4 5
Presenter
Presentation Notes
Measuring BehaviorThe following statements describe how you might communicate with others in your everyday life. Place an “X” in the box that best describe how often you have done what is described over the past month.
Never Rarely Sometimes Often AlwaysI find it easy to get my point across.
I use body language to help get my message across.
I use my tone of voice to reinforce what I am trying to say.
Presenter
Presentation Notes
When trying to measure behavior, you either can either use a self-report questionnaire or observe them doing it. Here is an example of one of the Four-Fold Youth Development Instrument to measure communication skills. In this instruments the youth indicated the frequency of doing each of the subscale behaviors. I find it easy to get my point across. I use body language to help get my message across. I use my tone of voice to reinforce what I am trying to say.
Measuring Behavior Changes
Behavior 1st Rating 2nd Rating
1. Stays with task until completed.
2. Participates in class discussion.
3. Cooperates on tasks.
4. Follows directions.
1 = never, 2 = occasionally, 3 = 50% of time, 4 = frequently, 5= never
Presenter
Presentation Notes
The most common way to measure behavior change is to observe it. Here’s an example of an observation checklist. Each participant is observed near the beginning of the program and periodically. Then you make a comparison. Stays with task until completed. Participates in class discussion. Cooperates on tasks. Follows directions. You indicate how often you observed the behavior. NOTE: You can also use an observation checklist to measure whether participants in your program can perform a skill.
Measuring Intentions to ActDo you plan to soil test for phosphorus?
Yes No
As a result of this program, how willing are you to soil test for phosphorus this coming season?
Not willing
Reluctant
Somewhat willing
Quite willing
Very willing
Presenter
Presentation Notes
Sometimes, time and money prevent us from actually measuring behavior. In those cases, you might want to measure their intentions. Psychology suggests that knowing participants’ intentions is a good predictor of behavior. You can sure a post survey to measure intentions. When trying to measure intention don’t ask participants if they plan to do something you recommended in the program, for example in a new nutrient management program: Do you plan to soil test for phosphorus? Such a question sounds threatening. It puts participants on the spot. Also , many participants are kind an don’t want to admit they don’t plan to something you just tried to convince them to do. Thus your data will very likely be skewed. Instead try asking this question: As a result of this program, how willing are you to soil test for phosphorous this coming season. In this question, people will find it easier to tell the truth.
Open-ended Questions Advantage – reveals what is on the
participant’s mind, instead of what the evaluator thinks is on their mind
Disadvantage – takes time to analyze data and participants don’t always take time to fill out
Not very effective with youth
Presenter
Presentation Notes
Open-ended questions on surveys or questionnaires allow the participants to give answers in their own way. The major advantage of the open-ended question is that it reveals what is in the participant’s mind, as opposed to what the evaluator suspects is in participants mind. Sometimes, the responses provide quotable material. The responses to open-ended questions are often difficult to compare and interpret. Questions regarding participants’ needs, expectations, knowledge, attitudes, skill level, behavior, experiences during the program and any results of those experiences may be asked. NOTE: Since it takes longer to answer open-ended questions, it is important not to use more than one – three on an evaluation instrument. In most cases, open-ended ended questions are very effective with youth (especially younger youth) because they don’t take the time to answer them. Youth tend to skip these kind of questions.
Open-ended Questions In the next 3 months, what is one thing you
will do as a result of participating in this training?
Name one or more ways you learned to economize your finances?
What is one positive ways you can express your emotions?
Describe any changes in your child’s behavior that has occurred since he/she has been participating in the ____ program.
Presenter
Presentation Notes
Here’s four examples of good open-ended questions: In the next 3 months, what is one thing you will do as a result of participating in this training? Name one or more ways you learned to economize your finances? What is one positive ways you can express your emotions? Describe any changes in your child’s behavior that has occurred since he/she has been participating in the ____ program
Hopefully this brief session
has helped you to
GEAR UP
to design good evaluation toolsthat will yield valuable impact data.
For more info or help in designing evaluation instruments contact Susan Barkman,
4-H Department, Purdue University
Presenter
Presentation Notes
Hopefully this brief session has help you to GEAR UP to design good evaluation tools that will yield valuable impact data. Don’t forget about the Evaluation Staff Development Training that are offered. You’ll have more opportunities for hands-on work at designing your own instruments, Also remember that the field guide will be up on the CES intranet. They will provide some easy to follow steps. Lastly, if you need help design an evaluation instruments, please feel free to contact Susan Barkman in the 4-H Department at Purdue. She’ll be glad to sit down and work with you through the process. Just call and set up an appointment to meet the next time you are on campus.
