This toolkit was originally developed by PC Burkina Faso. The original has been revised by OPATS for wide use and to include global health indicators. Please revise to align with your programming as needed.

Volunteer and Counterpart HandbookMeasuring Our Results

Table of ContentsPurpose of the Handbook

Developing a way to measure resultsStep One: Assess what you are doing by topic or issue areaStep Two: Select the key results you and your counterparts agree to measureStep Three: Decide how to measure your key resultsStep Four: Develop tools to measure your key resultsStep Five: Practice using the measurement toolsStep Six: Improve the tools based on how well they worked in practiceStep Seven: Pilot test the revised tools againStep Eight: Discuss the results of the pilot test and improve tools some moreStep Nine: Develop an action plan to measure the key results of your workStep Ten: Carry out your action plan

Coordinating and supervising a survey effortCompiling and summarizing what you learned from your survey/measurement effortSharing findings with your communitySharing findings with Peace CorpsUsing your findings to improve your workUsing your findings to strengthen funding proposals

AppendixAppendix A: Detailed facilitator’s agenda for 5 day ‘measuring success’ workshopAppendix B: GlossaryAppendix C: Malaria surveyAppendix D: HIV/AIDS survey

1

PurposeThe purpose of this draft handbook is to assist Peace Corps Volunteers, their counterparts and members of their communities to better measure the results of their efforts. It is based on a 5 day workshop conducted in April of 2007 with Volunteers and counterparts active in community health promotion. As a result, this first draft of the handbook contains training session ideas as well as questionnaires and data collection tools designed and piloted by the participants of that workshop. Please continue to add to this handbook to make it more useful to Volunteers working throughout Burkina Faso. Developing a way to measure resultsStep One: Assess what you are doing by topic or issue areaVolunteers and their counterparts should work together in pairs to list their main health education or other activities in the past year.

What types of health education or other activities have you undertaken in the past year? What was the focus of the activity--what messages? How many times did you do the activity? How many people did you reach through the activity?

To see the breadth of topic areas (e.g., AIDS, malaria, hygiene, etc.) that are being covered by Volunteers and their counterparts, return to the large group and have each Volunteer/Counterpart list their activities by topic area. Take each topic area in turn. Record activities by topic area on flip charts like the one below:

PaludismeActivités

e.g. talk to mothers at the clinic about the importance of sleeping under nets

Step Two: Select the key results you and your counterparts agree to measureAs a group, discuss what outcomes you hope to achieve. Chose one topic area (e.g., AIDS) to do together as a group discussion, asking the following questions:

What type of change are we trying to encourage? (improved knowledge, attitude, behavior)

Who is the target population? (pregnant women, youth, men, etc.)

Have two Volunteer/Counterpart pairs ask each other about the kind of results they want from their work in health education or other areas. Participants then briefly present the results of their interviews to the group. Record desired results/outcomes on flipcharts like the one below:

PaludismeActivités

e.g. talk to mothers at the clinic about the importance of sleeping under nets

Résultats Désirése.g. more women and children will sleep under nets

2

Facilitator then takes a few of the desired results for one health topic or issue area and looks at them carefully, asking these questions to the group:

What kind of outcome is this—a change in knowledge? Skills? Attitude? Behavior? Condition?

Is this outcome a realistic result of the activity? Is this outcome something we all agree is very important? Is this outcome something that more than one of us is hoping to achieve? Should we call this outcome a key outcome? How will we know when this outcome has been achieved?

Step Three: Decide how to measure your key resultsLook at each desired result and discuss how you would measure whether or not it is being accomplished. If it is a change in knowledge or awareness, you will need to ask people questions about their awareness and knowledge. If it is a change in behavior, discuss how you can collect this data, i.e. through observation, or simply reported practices via questions to target population. Examples of observation include checking on presence of soap where family members wash their hands; checking to see the latrine or how drinking water is being stored in the household. Examples of reported behavior include asking individuals about the actions they took during an illness in the weeks preceding the survey; or child feeding in the 24 hours, or week prior to the survey. Health providers can also be interviewed as they can speak to trends in care-seeking behavior, and provide data showing use of health services. If the change is knowledge, you will need to discuss these questions to decide on how you will go about measuring the change in knowledge: Who will you need to interview/ask? Where is the best place to do this? When is the best time to do this? How should you do these interviews/questions? What question(s) will you ask them to assess their knowledge/awareness?

Step Four: Develop tools to measure your key resultsIn a big group, take one key outcome and address the following questions:1. What will tell you that people have increased their knowledge or changed their attitudes or

behavior? What questions should you ask them?

3

PaludismeRésultats Clefs

Par example : P eople will recognize the symptoms of malariaWhat information do we need? The level of knowledge throughout the village about

the symptoms of malaria : fever cycles every 24 hours, headache, chills/convulsions

What question(s) should we ask ? “How do you know when someone has malaria ? What

are the signs ?”Who do we ask ?Where do we ask them ?When do we ask them ?

Then have people work in small groups to begin drafting other questions related to assessing other changes in knowledge.In drafting these questions, bear in mind that in many ways “less is more.” This means that you should only ask a question you are very sure will yield useful information. The goal should be to ask as few questions as you can to get the information you absolutely need. This will make the overall effort much more efficient for you and for the people you need to interview, and it will make managing all the information you get much easier.

Step Five: Practice using the measurement toolsOnce you have a draft of the questionnaire/survey instrument, it will be useful to practice using it before you head out to interview people for real. To practice it, have two volunteers—one interviewer and one respondent—agree to model the interviews. The respondent should pretend to be someone in a village, sitting at home, not expecting to be approached by someone asking for an interview. The interviewer should approach the villager, greet her appropriately, explain why s/he is there, and ask permission to do an interview.

The facilitator of the workshop should stop the role play at this time. Ask the large group to reflect on how well the interviewer approached the beginning of this interview :

Q : What did you think about how the interviewer introduced himself ?Q : Was the purpose of the interview clearly explained ?Q : Did the interviewer ask permission to do the interview ?Q : Did the interviewer set an appropriate tone for the interview/put the respondent at ease ?Q : What could the interviewer do better next time ?

After this discussion, the interview continues, question by question. The facilitator should wait for the question to be asked and answered, and for the interviewer to try to record the response, then stop it and ask the group to reflect again.

Q : How clear was the question ?Q : How clear was the response ?Q : Could the interviewer have done anything to prompt a clearer or more complete response ?Q : Is this question in the right place ? Should it be asked earlier or later in the interview ?

4

Continue this process until all the questions have been asked and answered.

Step Six: Improve the tools based on how well they worked in practiceMake revisions to the questions based on the lessons from the previous role play exercise. The objective is to refine the questions to make them clear, and to put them in an appropriate order. If there is time suggest that the group include any instructions to the interviewer on the questionnaire. For example, if the answer is no, skip to question 5.

Step Seven: Pilot test the revised tools againIn a large group setting, discuss the purpose of the pilot test, if you can do one.

What is the purpose of your pilot test? What are you testing, exactly ?

Allow for some group discussion of this, and emphasize the following points about what the pilot test is testing: The clarify of the questions. Do you have to rephrase or restate the question too much ? Is it

clear the first time you ask it ? How can you make it clearer ? The usefulness of each question. Is it necessary to ask all the questions you have ? Can you

eliminate any questions ? The length of the interview. How long does each interview take, on average ? The availability of people to respond at that time. Are enough people available at the time

of day/month/year for this survey ? The clarity of responses to your questions. If more than one person is listening to the

interview, do they each agree on how the respondent answered each question ? The availability of other data/information. If some of the information you are looking for

would come from the records of a health clinic or school, for example, is the specific information you would need available to you in a form you can easily use ?

How easy is it to accurately record responses on your data collection form(s) ?

Small group planning of pilot test Spend some time discussing these questions in your group and coming up with a plan for the pilot test tomorrow. How many interviews are you going to attempt to do ? Where are you going to go when you get there ? Who is going to do the interviews ? If translation is necessary, who is going to do the translating ?

Materials prep for pilot testTo type up and copy final drafts of the survey/questionnaires as well as any additional data collection tools, get at least one volunteer from each group to do the administrative work of preparing materials for their small group members to use tomorrow. The rest of the group can either break early for the day or meet on other issues.

Implementing the pilot test

5

Small groups go out, each with a facilitator, to the village to test tools. Each group should try to conduct 5 to 10 interviews. Observers give feedback after each interview. Each group should also look for data as necessary at the health clinic, boutiques, and through observations.

Groups should use the following questions as a guide to evaluate their experience during the field test: How did you determine which way to go in the village? How many people/households did you interview? How did you choose the interviewees? How did the introductory part go? (did you put the interviewees at ease? did you ask for their

informed consent? did you explain confidentiality? etc.) Were the questions understood by the interviewees? If not, why not? Is the order of the questions logical? If not, why not? Is the length of the survey appropriate? If not, why not? Can you eliminate any questions ? Do you need to add any questions ? Is the data recording instrument appropriate ? If not, why not? Did you each record the same responses to each question ? If not, why is there disagreement

or confusion about how people responded to your questions ? Did you find relevant data at the health center? If not, what did you do? Did you encounter any other difficulties? What are your recommendations for improving the process?

Step Eight: Discuss the results of the pilot test and improve tools some moreSmall groups meet together first to review how their pilot test went, and to prepare theirsummary for a debrief to the larger group. In your debrief to the larger group, focus yourremarks on : How many interviews you did. The average length of each interview Which questions seemed to cause confusion and how you recommend changing those If members of your group recorded responses differently, why did that occur and how are

you going to fix that ? Other challenges or lessons learned from the pilot

Step Nine: Develop an action plan to measure the key results of your workEach Volunteer and counterpart should put together an action plan that lays out how they are going to start measuring results when they get back to their sites. Emphasize who is going to do what, by when.

Step Ten: Carry out your action plan

Coordinating and supervising a survey effortBe sure to get approval from every village you intend to do a survey or collect data. Follow your training as Peace Corps Volunteers in how to approach a village, who to ask for permission, how

6

to explain what you are doing and what you hope to learn from this effort, and how you intend to share results with them.

It may be necessary to find interpreters or guides to assist your research efforts.

Compiling and summarizing what you learned from your survey/measurement effortFacilitator leads the group through a review of the data collected by the groups. Note differences between men and women for each survey and what the data suggest about priority areas for health education. What do you do with data? (Analysis) If evidence shows that behavior isn’t changing, it could be because: the message is wrong, a key piece of the message is missing, the treatment isn’t intense enough, the wrong people are targeted.

Use a simple clear approach to summarizing the results of your research, along these lines:# hommes :

# femmes :

# enfants :

Nombre Total d'Individus qui acceptent l'interview

Réponses correctes# hommes :

# femmes :

# enfants :

Nombre de personnes qui connaissent les

symptomes du paludisme

Sharing findings with your communityAnytime you do research in a community, be sure to share what you find with them as soon as you have summarized them. Present your findings in as clear and simple a way as you can. Avoid using research terms. Present what you think are the most interesting findings and ask what they think about them. Ask if anything you found is surprising

Sharing findings with Peace Corps

7

Using your findings to improve your work

Using your findings to strengthen funding proposals

8

Appendix A:Detailed facilitator’s agenda for 5 day ‘measuring success’ workshop English/French

Peace Corps/Burkina Faso: ‘Measuring Success’ WorkshopOuagadougou, April 23-27, 2007

Participants: 11 Peace Corps Volunteers and 10 counterparts

Facilitators: Dr. Claude Millogo – PC/BF APCD Health; Jerry Black – PC/W Evaluation Specialist; Kathy Jacquart – PC/W HIV/AIDS Programming and Training Advisor; Joachim Ouedraogo– CNLS Representative, Dr. Joanne Capper – PC/W Evaluation Specialist

Workshop Purpose: To train and equip workshop participants to measure the outcomes of their health education interventions

Workshop Outcomes:o Draft handbook for staff, Volunteers and counterparts on how to measure outcomes of

health education and disease prevention activities (e.g. malaria, HIV/AIDS, tuberculosis, polio, nutrition, guinea worm, other?)

o Set of simple measures/instruments that can be used by literate or semi-literate health educators/promoters/community members.

Workshop Format: Interactive presentations, small group work, field practice.

Materials Needed: Flipcharts and markers; tape; post-it note pads; paper, pencils, pens, markers.

Monday, April 238:30-8:45 Welcome and Purpose of the Workshop Marily Knieriemen, PC/CD and Dr. Millogo open the workshop.

8:45-9:15 Introductions/Icebreaker/volunteers for energizers and daily evaluations

9:15-9:45 Overview of Workshop Objectives, Agenda, Expectations and NormsWalk through what we hope to achieve each day, and what the approach will be each day. Ask participants what they hope to get out of the workshop; record answers on flipchart. Note how/where the proposed agenda is going to address participant expectations. If some expectations are not part of the agenda, propose modification to agenda, or explain why expectations can’t be met through this workshop. Reinforce these ideas: This workshop is building on the work they are already doing by developing a simple

approach for measuring the results of their health education and promotion activities. Workshop participants will create this simple evaluation approach, as well as some draft

measurement tools (surveys, data collection forms, etc) together to ensure that the approach is relevant and appropriate.

Over the next 5 days we will design a practical approach to evaluation that will help Volunteers with their counterparts learn what is working well as well as where, with whom,

9

when and how to focus their health promotion messages and activities to have a more powerful effect in their communities.

Norms for the next 5 days include: Speak slowly and loudly in French Do not interrupt anyone who is speaking Turn off all cell phones Be on time Raise your hand before you ask a question or make a point If any word or expression is not clear to you, ask for clarification. We will keep a running list

of definitions on a flipchart on the wall.

9:45-10:00 What are our health education activities?Volunteers and their counterparts work together in pairs to list their main health education activities in the past year.

What types of health education activities have you undertaken in the past year? What was the focus of the activity--what messages? How many times did you do the activity? How many people did you reach through the activity?

10:00-10:15 Coffee break

10:15-10:45 What are our health education activities? To see the breadth of topic areas (e.g., AIDS, malaria, hygiene, etc.) that are being covered by Volunteers and their counterparts, return to the large group and have each Volunteer/Counterpart list their activities by topic area. Take each topic area in turn. Record activities by topic area on flip charts like the one below:

PaludismeActivités

e.g. talk to mothers at the clinic about the importance of sleeping under nets

10:45-11:15 What are we trying to accomplish through our work? As a group, discuss what outcomes they are hoping to achieve through their activities. Chose one topic area (e.g., AIDS) to do together, asking the following questions:

What type of change are you trying to encourage? (improved knowledge, attitude, behavior)

Who is the target population? (pregnant women, youth, men, etc.)

11:15-12:30 Small group work and presentations on what we’re trying to accomplishHave two Volunteer/Counterpart pairs ask each other about the kind of results they want from their work in health education. Participants then briefly present the results of their interviews to the group. Record desired results/outcomes on flipcharts like the one below:

10

PaludismeActivités

e.g. talk to mothers at the clinic about the importance of sleeping under nets

Résultats Désirése.g. more women and children will sleep under nets

12:30-1:30 Lunch

1:30-2:00 Looking more carefully at a few outcomesFacilitator takes 3 to 4 of the intended outcomes for one health topic and looks at them carefully, asking these questions to the group:

What kind of outcome is this—a change in knowledge? Skills? Attitude? Behavior? Condition?

Is this outcome a realistic result of the activity? Is this outcome something you all agree is very important? Is this outcome something that more than one of you is hoping to achieve? Should we call this outcome a key outcome? How will you know when this outcome has been achieved?

2:00-3:00 Participant analysis and discussion of outcomesBreak group into small groups, giving each group a health topic area. Ask each small group to go through a similar exercise of analyzing the intended outcomes on their flipcharts. Assign someone to record the group’s decisions about the outcomes. This person will present back to the larger group your small group’s recommendations. Be prepared to explain which outcomes you believe the group should consider the key, or core ones.

Co-facilitators should sit with each small group and make sure they understand what to do and get through this exercise, prepared to talk about their decisions.

On flipchartSmall Group Work: Outcome Analysis Questions

What kind of outcome is it? (knowledge, skill, attitude, behavior, condition)?

Is the outcome or result possible and realistic? How many of us are working to achieve this outcome? How important is this outcome to all of us? Do we agree that this outcome is a key outcome we all want to

measure? How will we know when this outcome has been achieved?

3:00-3:15 Break

3:15-4:45 Small group presentations and discussionEach small group gets 10 minutes to present their discussion and proposal to the group, and about 5 minutes for group discussion. The goal of this discussion is to arrive at a group agreement about the key desired results (outcomes) for all their health work. These key

11

outcomes will then become the focus for the development, testing and refining of related measures.

Facilitator keeps each group to the time limit, and pulls out the key questions for the larger group to consider, which are:

Do we all agree that this/these are the key outcomes we’re hoping to achieve in this health area?

PaludismeRésultats Clefs

Par example   : people will recognize the symptoms of malaria people will understand how malaria is transmitted people will understand how to prevent malaria more women and children will sleep under nets families will take their children to the clinic at first

signs of malaria pregnant women will take malaria prevention

medication at pre-natal visits

4:45-4:55 Recap of the day and preview of Day 2Summarize the highlights of the day.

Tomorrow we will explore the question: “How do I know when my work has been successful?”

4:55-5:00 Day 1 evaluationTake a few minutes to write your thoughts about what worked well today as well as what could be improved; put your post-it notes on the flipchart. Can we get a Volunteer or two to agree to summarize this for us by tomorrow morning?

Day 1 evaluationWhat worked well What could be improved

Thanks for good work today.

12

Tuesday, April 248:00-8:15 Introduction and recap of Day 1Energizer.

Recap Day 1. Ask volunteers to present summary of participant feedback from Day 1. Ask group if there are issues or questions about yesterday.

Tell the group that today we are going to discuss why it’s important to provide convincing evidence of what you have accomplished. We are then going to design some draft data collection instruments together. These are the tools you will use to measure your successes and provide convincing data. The idea here is that we will design these tools together and all agree to use them, and to encourage other Volunteers or counterparts we know to also use them. The more people use them, the more powerful a sense of your outcomes you will start to get. Also it should help you to have more informed and productive conversations about where the priority health education needs are in Burkina Faso and how Peace Corps can best meet them.

Ask for volunteers to lead the afternoon energizer and evaluation for Day 2.

8:15-9:00 Accomplishments to dateQ: Before we go further, who wants to share with us a result that they have seen from their work?

Q: How have you measured your successes so far? What approaches have you taken to figure out how successful your work has been?

If no one mentions anything, be prepared to cite the following outcomes from Claude’s FY06 PSR last year, namely:

PCVs observe more pre-natal visits to health clinics. PCV(s) noted that the outcome of theatre groups work in malaria education included

increasing knowledge of malaria and increasing use of mosquito nets. Who did this? How was the study done?

Collaboration between COGES, clinics and theatre groups has improved: What does this collaboration look like? Is this something PCVs are actively working on? How do you know collaboration is happening the way you want it to?

Self-sufficiency of theatre groups has improved in some cases: How is self-sufficiency measured? Who does this exercise?

PCV(s) found increases in community members’ knowledge after theatre groups did their shows. Was this a pre-post test with the crowd? Whose knowledge increased?

More use of water pumps (Guinea worm): What does this statement ‘more use of water pumps’ mean—where is this happening? How much more use?

Acknowledge that this group and their colleagues are achieving a lot in Burkina Faso, and that what we’re trying to do in this workshop is figure out a better way to identify the most important accomplishments and successes. We made some progress here yesterday. We have agreement on the key outcomes you all are trying to achieve.

13

9:00-10:00 The value of good measures The next question is: How will we know when our work has succeeded?

Look at the outcome about use of mosquito nets. Some Peace Corps Volunteers did a study of the results of the work of theatre groups in Burkina Faso and found that more people understand how to prevent malaria, more people bought mosquito nets, and more people are using mosquito nets.

Q. Why is it important to have a good way to measure the effectiveness of your activities?

Allow a few minutes for group to respond to this question. If no one responds, say something like: The value to you of having a good way to measure the effectiveness of things like the theatre group’s malaria messages is that: you will know what messages are really working; you will know how to improve your malaria prevention work so that it is more effective; it can motivate you to do more; you will be able to convince other people that your work is successful – to get funding, to get

support from community leaders, etc.

Q: Which statement is more convincing? (write statements on flip chart)A. As a result of the theatre group’s work in malaria education, more people are using

mosquito nets.B. Before the theatre group’s work in malaria education in the village of Linkinkele, 10

people in the village slept under mosquito nets, including just 2 women and 3 children. Two months after the theatre group’s education activities and performances in Linkinkele, 100 villagers sleep under mosquito nets, including 35 women and 40 children under 5.

Reread them. Allow some discussion.

Q: Why is statement B more convincing to you than the first one?

Q: What would you have to do to be able to make a statement like B?Allow participants to respond. Know how many people used mosquito nets before the theatre group’s performance about

malaria prevention; then know how many people use mosquito nets two months after the performance.

Q: How would you figure out how many people sleep under a net at the beginning? (we call this getting a ‘baseline’)

Allow participants to respond.There are two ways to figure this out. The first thing to find out is if there is already an existing source of data about the use of mosquito nets in those villages.

14

Q: Where might you find data like this? (e.g., Demographic and Health Survey, CSPS, NGOs?)

If there aren’t data available to you, then you should consider doing a survey of the villages you know the theatre groups are going to, before they go there. So it’s important to know how to do a survey.

In doing a survey, plan carefully what questions you are going to ask, who to interview, where to interview, when to interview, and practice how you do the interview.

What questions? Crafting good questions is a skill. We will spend a lot of time in this workshop crafting (drafting and refining) the questions you will ask to make sure they are good ones. We’ll turn to this later today and tomorrow.

Who ?Paludisme: Enquete de Bas (Malaria baseline survey)

Resultat desire : people recognize the symptoms of malaria

Who to interview ? Men, women, youth (in and out of school), children Q : If you want to assess a village’s understanding of malaria, who should you interview ?Record participant responses on the flip.

A : You should ask a range of people in the village, including men, women, students, children, and older adults. It would be a good idea to ask as many people as you could with the time and resources you have available for the task.

Where ?Paludisme: Enquete de Bas (Malaria baseline survey)

Resultat desire : people recognize the symptoms of malaria

Where to interview ? People who live in each part of the villageQ : If you want to assess a village’s understanding of malaria, where should you go to do these interviews ?Record participant responses on the flip.

A : You would want to try and interview people throughout the village so that you ask people who live near the health clinic as well as people who live far from it. You would not get enough information about the level of malaria knowledge throughout the village just by asking people who live in the same neighborhood.

15

When ?Paludisme: Enquete de Bas (Malaria baseline survey)

Resultat desire : people recognize the symptoms of malaria

When to interview ? Before and after our education activities Before, during and after the rainy season Before the harvest season begins Just before bedtimeQ : If you want to assess a village’s understanding of malaria, when should you do these interviews ?Record participant responses on the flip.

A : There are several ‘whens’ to keep in mind here : Before and after your malaria education activities. Certain months of the year, or certain seasons, may be better than others. You may decide it’s

important to ask people before the rainy season when malaria is on their minds, or before the harvest season when they may be too busy for you.

Time of day may matter. You should also consider the best time of day for the survey. If, for example, you want to ask about the use of mosquito nets, and you want to ask to see their nets to verify that they are attached, you might want to try and ask people at the end of the day when the nets are more likely to be attached.

How ?Paludisme: Enquete de Bas (Malaria baseline survey)

Resultat desire : people recognize the symptoms of malaria

How to interview ? Practice (pilot) the questions before actual research begins Explain the purpose of the interview and how you will use

the information Explain that the results will be confidential Ask for permission to do the interview before asking any

other questions Make the person feel at ease. Find a comfortable private

setting for the interview, if possible Speak slowly and clearly Listen and don’t interrupt Don’t ask questions in a way that suggests the ‘right’

answer Stay neutral throughout the interview Prompt for more complete answers in a neutral way, « do

you have anything more to add to that ? » or « is there anything else you want to say ? »

Q : If you want to assess a village’s understanding of malaria, how should you do these interviews ?

16

Record participant responses on the flip.

A : Interviewing is a skill, and you will get better at it through practice. In conducting an interview, be attentive : to how you introduce the purpose of the interview to the atmosphere you create for the interview to how you ask each question to how you listen to each response to how efficiently and accurately you record each response In asking and listening, stay neutral. It’s critical that you do not ask questions in a leading or

suggestive manner. The tone of your voice, the way you move your eyes or body can often suggest the kind of response you want. Many people will respond to subtle verbal and physical cues by providing the response that they think you want to hear, rather than their opinion or understanding. When this happens the interviewer has corrupted the integrity of the interview and essentially rendered the results meaningless. To avoid wasting your own time or anyone else’s, it is very important to stay as neutral as you can.

10:00-10:15 Coffee break

10:15-10 :45 Developing measurement instrumentsIn a big group, take one key outcome and address the following questions:

2. What will tell you that people have increased their knowledge or changed their attitudes or behavior ? What questions should you ask them ?

PaludismeRésultats Clefs

Par example : P eople will recognize the symptoms of malariaWhat information do we need? The level of knowledge throughout the village about

the symptoms of malaria : fever cycles every 24 hours, headache, chills/convulsions

What question(s) should we ask ? How do you know when someone has malaria ? What

are the signs ?

Who do we ask ?

Where do we ask them ?

When do we ask them ?

10 :45-12 :00 Small group work to begin drafting measurement toolsDivide into 4 small groups: Malaria A and B, AIDS A and B. Each small group asks the What, Who, Where, When questions for each outcome and records their answers.

17

12:00-1:00 Lunch

1:00-2:00 Small group reportsEach small group (Paludisme A, Paludisme B, SIDA A, SIDA B) reports how they answered the What, How, Who, Where, When questions.

2:00-3:00 First draft of Survey or questionnaireSmall groups return to complete the first draft of their survey/measurement tool

3:00-4:00 Practice/Role play of surveyTwo volunteers—one interviewer and one respondent—agree to model the interviews. The respondent should pretend to be someone in a village, sitting at home, not expecting to be approached by someone asking for an interview. The interviewer should approach the villager, greet her appropriately, explain why s/he is there, and ask permission to do an interview.

The facilitator of the workshop should stop the role play at this time. Ask the large group to reflect on how well the interviewer approached the beginning of this interview :

Q : What did you think about how the interviewer introduced himself ?Q : Was the purpose of the interview clearly explained ?Q : Did the interviewer ask permission to do the interview ?Q : Did the interviewer set an appropriate tone for the interview/put the respondent at ease ?Q : What could the interviewer do better next time ?

After this discussion, the interview continues, question by question. The facilitator should wait for the question to be asked and answered, and for the interviewer to try to record the response, then stop it and ask the group to reflect again.

Q : How clear was the question ?Q : How clear was the response ?Q : Could the interviewer have done anything to prompt a clearer or more complete response ?Q : Is this question in the right place ? Should it be asked earlier or later in the interview ?

Continue this process until all the questions have been asked and answered.

4:00-4:45 Revise questions based on role playEach of the four groups makes revisions to their questions based on the lessons from the previous role play exercise. The objective is to refine the questions to make them clear, and to put them in an appropriate order. If there is time suggest that the group include any instructions to the interviewer on the questionnaire. For example, if the answer is no, skip to question 5.

4:45-5 :00 Day 2 evaluation

What worked well What could be improved

18

Wednesday, April 258:00-8:15 IntroductionEnergizer. Recap Day 2. Ask group if there are issues or questions about yesterday.

8:15 - 9:00 DHS data exercise Look at one question from the 2003 Burkina Faso DHS. Have the group guess the results. Show the real results. Ask what the results tell them. Use a table like the one below.

Percentage of people who know that you can not become infected with HIV by eating with someone with AIDSTotal Men Women Rural Urban No

educationPrimary Education

Secondary Education

Rich Poor

M M M M M M MW W W W W W W

9:00 – 9:30 Why do we measure our results?Brainstorm with the group the reasons for measuring our results. Reasons include: to see if activities worked to organize future activities to monitor if people are understanding the messages of an activity to change our work strategy to motivate ourselves to mobilize resources to mobilize support from the community

9:30 – 10 :00 Recording responses in an interviewLook at how well the questionnaire is formatted to allow the interviweer to efficiently and accurately record the responses to his/her questions.

If the question is open-ended, include a list of likely responses below the question, with blank space next to each likely response where you can put a check mark. This is going to work better than trying to write down each response. In testing or piloting the questionnaire, one of the things you’re trying to do is come up with the best list of likely (potential ‘correct’ and ‘incorrect’ responses) to a question.

For example:Comment est-ce qu’on peut attraper le SIDA? Ne pas lire ces réponses possibles. Notez-les seulement si l’enquêter les dit.Correctes Incorrectes________Sang _________Moustique________Mère-Enfant _________Poussière________Relations sexuelles _________Manger avec les gens infectés

_________Salive_________Utiliser la même toilette_________Etre proche/dormir à coté_________Embraser une personne infectée

19

10:30 - 12:00 More practice and role playing of interviews Give the groups time, if needed, to make additional revisions to their questionnaires. Then have someone from each small group interview someone from another small group, with the objective of modeling complete interviews without interruption, start to finish. Each member of the group should be trying to record the responses, independently. Then have the group reflect on these questions : How well did the interviewer introduce himself/herself ? How clear was the purpose of the interview ? Did the interviewer ask for ‘informed consent’ before proceeding with the interview

questions ? Did the interviewer set an appropriate, respectful tone for the interview ? How long did the interview take ? Was each question necessary ? Can you eliminate any questions ? Do you need to add any questions ? Are the questions in the right order ? What was the response to question 1 ? Question 2 ? etc. Does everyone agree what the

reponse was ? Why did you record different response, if you did ?

Each group should model their survey/questionnaire in this fashion and agree on any additional revisions to their survey/questionnaire.

12:00 – 1:00 Lunch

1:00-1:45 Pilot testing the survey/questionnaire and sampling What is the purpose of your pilot test? What are you testing, exactly ?

Allow for some group discussion of this. Record on flip and emphasize the following points as well :

The clarify of the questions. Do you have to rephrase or restate the question too much ? Is it clear the first time you ask it ? How can you make it clearer ?

The usefulness of each question. Is it necessary to ask all the questions you have ? Can you eliminate any questions ?

The length of the interview. How long does each interview take, on average ? The availability of people to respond at that time. Are enough people available at the time

of day/month/year for this survey ? The clarity of responses to your questions. If more than one person is listening to the

interview, do they each agree on how the respondent answered each question ? The availability of other data/information. If some of the information you are looking for

would come from the records of a health clinic or school, for example, is the specific information you would need available to you in a form you can easily use ?

How easy is it to accurately record responses on your data collection form(s) ?

20

1:45 – 2:00 Small group planning of pilot test Spend some time discussing these questions in your group and coming up with a plan for the pilot test tomorrow. How many interviews are you going to attempt to do ? Where are you going to go when you get there ? Who is going to do the interviews ? If translation is necessary, who is going to do the translating ?

2 :00-2 :15 Day 3 evaluation

What worked well What could be improved

2 :15-5 :00 Materials prep for pilot testTo type up and copy final drafts of the survey/questionnaires as well as any additional data collection tools, get at least one volunteer from each group to do the administrative work of preparing materials for their small group members to use tomorrow. The rest of the group can either break early for the day or meet on other issues.

21

Thursday, April 268:00-1:00 Pilot test of data collection tools in villageSmall groups go out, each with a facilitator, to the village to test tools. Each group should try to conduct 5 to 10 interviews. Observers give feedback after each interview. Each group should also look for data as necessary at the health clinic, boutiques, and through observations.

Groups should use the following questions as a guide to evaluate their experience during the field test: How did you determine which way to go in the village? How many people/households did you interview? How did you choose the interviewees? How did the introductory part go? (did you put the interviewees at ease? did you ask for their

informed consent? did you explain confidentiality? etc.) Were the questions understood by the interviewees? If not, why not? Is the order of the questions logical? If not, why not? Is the length of the survey appropriate? If not, why not? Can you eliminate any questions ? Do you need to add any questions ? Is the data recording instrument appropriate ? If not, why not? Did you each record the same responses to each question ? If not, why is there disagreement

or confusion about how people responded to your questions ? Did you find relevant data at the health center? If not, what did you do? Did you encounter any other difficulties? What are your recommendations for improving the process?

1:00-2:30 Return to Ouagadougou/lunch

2:30-3:00 Debrief of pilot testSmall groups meet together first to review how their pilot test went, and to prepare theirsummary for a debrief to the larger group. In your debrief to the larger group, focus yourremarks on : How many interviews you did. The average length of each interview Which questions seemed to cause confusion and how you recommend changing those If members of your group recorded responses differently, why did that occur and how are

you going to fix that ? Other challenges or lessons learned from the pilot

3:00-4:00 Finalizing questionnairesForm two groups by theme (i.e., AIDS and malaria) to create final draft questionnaires. This will be the version that all Volunteers and counterparts agree to use.

4 :00-4 :15 Day 4 evaluation

What worked well What could be improved

22

Friday, April 278:00-8:15 IntroductionEnergizer. Recap of Day 4.

8:15-8:45 Review of final survey/questionnairesSomeone from each group (SIDA1/2 and Palu ½) presents the final version of their tool.Group approves or suggestions final, minor changes.

8:45-10 :00 Summarizing and using data from the pilotFacilitator leads the group through a review of the data collected by the groups. Note differences between men and women for each survey and what the data suggest about priority areas for health education. What do you do with data? (Analysis) If evidence shows that behavior isn’t changing, it could be because: the message is wrong, a key piece of the message is missing, the treatment isn’t intense enough, the wrong people are targeted.

# hommes :

# femmes :

# enfants :

Nombre Total d'Individus qui acceptent l'interview

Réponses correctes# hommes :

# femmes :

# enfants :

Nombre de personnes qui connaissent les

symptomes du paludisme

Summarize all the results, by key result, of the pilot test and prepare a few summary statements to share back with the village where the pilot took place.

10:00-10:15 Coffee Break

23

10 :15-11 :00 ‘Measuring Success’ HandbookLead group through a discussion of the contents of a handbook/guide for measuring outcomes. Brainstorm issues to be addressed before collecting data, during data collection and after data collection.

Each of these on a separate flip (before, during, after).Before Data Collection Who will collect data? How will these people be trained? motivated? What resources will be needed for data collection? What are other steps to follow to prepare for data collection?

During Data Collection What should happen when data collectors arrive in a community? What community resources may be called upon to facilitate the data collection?

After Data Collection How will results be analyzed? Who will analyze the data? What will we do with the results? With whom will results be shared?

11:00-11:45 Next steps and individual action planningDr. Claude outlines her expectations about getting data from this effort, and how she plans to use it. The point is to clarify for the group: When you would like to receive data for each of the health areas/topics. How you would like to receive this data—from whom? What you plan to do with all this data. How you plan to share results back with the group. Other issues/questions related to expectations and use of this data

Each Volunteer and counterpart should put together an action plan that lays out how they are going to start measuring results when they get back to their sites. Emphasize who is going to do what, by when.

11:45-12 :20 Final workshop evaluation and closing ceremony

12:20-1:30 Lunch

24

Atelier du Corps de la Paix/Burkina Faso :Mesurer Les Succès

Ouagadougou, le 23 au 27 avril 2007

Le but de l’atelier : Améliorer nos capacités d’évaluer les résultats dans le domaine de la promotion de la santé

Pendant l’atelier nous allons :

- élaborer des outils d’évaluation- tester les outils - développer un guide pour l’utilisation de ces outils

Agenda

Journée 1- Présentation et discussion de des activités dans le domaine de la promotion de la santé.

Pendant les 12 derniers mois, quelles sont les activités que vous avez faites ? o le thème de l’activité (le paludisme, le VIH/SIDA, le PF, etc.)o le type d’activité (le théâtre, les visites à domicile, les causeries) o la fréquence de l’activité (une fois par mois, chaque semaine, tous les 6 mois)o le nombre de personnes touchées (le nombre de participants)

- Présentation et discussion des résultats désirés de ces activités. Suite à vos activités quels sont les résultats que vous avez souhaités ?

o Chez qui avez-vous voulu effectuer des changements ? (les femmes enceintes, les hommes, les jeunes, etc.)

o Quel type de changement ? (des connaissances, des attitudes, des comportements)

- Sélectionner les résultats clés que nous allons mesurer. Analyser les résultats désirés :o C’est quel type de résultat ? (connaissance, attitude, comportement)o Est-ce que nos activités peuvent produire ces résultats ?o Est-ce que le résultat est parmi les messages clés du Ministère de la Santé ? du

CNLS ?o Comment peut-on savoir si on a atteint nos résultats?

Journée 2- Développement des outils d’évaluation

o Comment savez-vous que vous avez atteints vos résultats désirés ? Quelle information vous l’indiquerait ?

o Qui pourrait donner cette information ?o Où collecteriez-vous cette information ?o Quand collecteriez-vous cette information

Journée 3

25

- Finalisation du développement des outils

- Préparation pour l’essai sur le terrain

Points à Considérer pour l’EnquêteAvant de commencer l’entretien, saluez votre interlocuteur/interlocutrice, présentez vous et décrivez l’objet de votre visite. Dites-lui que vous avez besoin de lui poser quelques questions, s’il vous donne la permission. Rassurer le (la) de la confidentialité et de l’anonymat de l’entretien. Retirez-vous avec lui dans un endroit calme et discret. Rappelez-lui encore que sa participation est volontaire, qu’il (ou elle) n’est pas tenu de répondre à une question qu’il (elle) juge trop intime mais que vous souhaitez dans la mesure du possible qu’il (ou elle) réponde sans se gêner. Enfin, informez le (la) que sa participation s’inscrit dans le cadre d’un vaste projet qui vise à améliorer le bien-être et la santé des populations que cela ne peut se faire sans informations sur leur niveau de connaissance, leurs attitudes, comportements et besoins sur les questions de santé.

- Apprentissage de comment utiliser les outils

Journée 4

- Test des outils sur le terrain

Utiliser les questions suivantes comme guide pour évaluer le pratique sur le terrain :1. Quelle direction avez-vous suivi dans le village ? 2. Combien de personnes /ménages avez-vous intervuvé ? (au moins 5 à 10 personnes)3. Comment est-ce que vous les avez choisis ?4. Est-ce que la phase introductive de l’enquête est correcte ? (mis la personne à l’aise,

demandé le consentement informé, expliqué la confidentialité, etc.)5. Est-ce que les questions sont claires ? Si non, pourquoi ?6. Est-ce que l’ordre des questions est logique ? Si non, pourquoi ?7. Est-ce que la longueur du questionnaire est appropriée ? Si non, pourquoi ?8. Est-ce que le formulaire pour enregistrer les réponses est approprié ? Si non, pourquoi ?9. Est-ce que vous avez trouvé les données souhaitées au CSPS ? Si non, qu’est-ce que vous

avez fait ?10. Difficultés particulières ?11. Recommandations ?

- Discussion des leçons apprises de l’expérience sur le terrain

- Amélioration des outils

Journée 5- Discussion de l’analyse et l’utilisation des données collectées - Révision du guide d’utilisation des outils - Elaboration des prochaines étapes

26

Appendix B: Glossary

English FrenchResults/outcomes Les résultatsData Les donnéesMeasures/tools/instruments Les outilsSurvey L’enquêteRespondent L’enquêterInterviewer L’enquêteurBaseline data Les données de baseWorkshop L’atelierTo measure MesurerKnowledge La connaissanceAttitude L’attitudeSkill La capacitéBehavior Le comportementSample L’échantillonSampling L’échantillonnageProcess La méthodologie

27

Appendix C   : Malaria Survey

Questionnaire sur le PaludismeVillage _________________________________________

Enquêteur(s)____________________________________

Date: __________________________________________

Le Nombre Total d’Individus qui acceptent l’interview

Hommes Femmes Enfants

Notes:

28

Résultat Désiré #1: Les personnes connaissent les Symptômes du Paludisme

Hommes Femmes EnfantsLe nombre de personnes qui connaissent les symptomes du paludisme

Quels sont les symptômes du Paludisme?Réponses Correctes:

Symptômes Hommes Femmes EnfantsLa fièvreLe coma

La fatigueLes

vomissements/la nausée

Les frissons/tremble

mentsMaux de tête

La douleur musculaire

Maux de ventreLe vertigeManque d’appétit

29

Autres:

Réponses Non Correctes:

Hommes Femmes EnfantsUn oiseau a attrapé l’enfant

Autres:

Résultat Désiré #2: Les personnes connaissent la voie de transmission du paludisme

Hommes Femmes EnfantsLe nombre de personnes qui savent comment on attrape le paludisme

Comment est-ce qu’on attrape le paludisme?Réponses Correctes:

Hommes Femmes EnfantsMoustiquePiqûre de moustique

Réponses Non Correctes:

Hommes Femmes EnfantsLes mouches

30

La première pluieLes mangues

Le mais frais

L’huile

L’eau sale

Nourriture (sale)Etre Sale

Manger Grave (beaucoup)Autres:

Résultat Désiré #3: Les personnes connaissent les moyens de prévention du paludisme

Hommes Femmes EnfantsLe nombre de personnes qui savent comment éviter le paludisme

Comment est-ce quelqu’un peut se protéger contre le paludisme?

Réponses Correctes:

Symptômes Hommes Femmes EnfantsUtiliser/Dormir

sous moustiquaire

imprégnéeUtiliser/Dormir

sous moustiquaire

31

Eliminer l’eau stagnante

Eviter l’eau salePorter les

manches longuesPrendre les

medicaments (les femmes

enceintes)Eviter les

moustiquesUtiliser les

spiralsPomper

InsecticideAutres:

Responses Non Correctes:

Hommes Femmes EnfantsPorter les amulettesEviter les mouchesEviter les pluieEviter les manguesMédicament Traditionnel/MarcheAutres:

Résultat Désiré #4: Les femmes enceintes et les enfants moins de 5 ans dorment sous moustiquaires

(imprégnées)Ménages

Le nombre de ménages ou on dit que les femmes enceintes

32

et les enfants moins de 5 ans dorment sous moustiquaire imprégnée

Est-ce que vous avez une moustiquaire (imprégnée)?

Hommes Femmes EnfantsOuiNon

Si non - Pourquoi?

Hommes Femmes EnfantsPas d’argentCe n’est pas disponible

Autres: Si oui - Est-ce que je peux voir vos moustiquaires?

Utilisation de MoustiquaireMoustiquaire AttachéeMoustiquaire non attachée

Apres avoir vu les moustiquaires imprégnées - Qui dorment sous les moustiquaires?

Hommes Femmes EnfantsPèreMèreEnfants (moins de 5 ans)Enfants (plus de 5 ans)

33

Résultat Désiré #5: Les femmes enceintes prennent la chimio prophylaxie contre le paludisme

pendant la CPN (question pour les femmes seulement)Oui Non

Le nombre de femmes enceintes qui prennent le chimio prophylaxie contre le paludisme pendant la CPN

Aux Femmes - Est-ce que vous avez un enfant moins de 1 an?

Si “oui” - Avez-vous suivi des CPN?

Si “oui” - Est-ce que vous avez pris la chimio prophylaxie contre le paludisme aux CPN pendant votre dernière grossesse?

FemmesOui

Non

Aux Femmes Enceintes - Est-ce que vous faites les CPN?

Si “oui” - Au CSPS, avez-vous pris la chimio prophylaxie contre le paludisme?

Femmes EnceintesOui

Non

Où avez-vous appris cette information sur le paludisme ?Hommes Femmes Enfants

Performance de Teatre

34

CSPS/InfirmierSensibilisationUn Ami

Autres :

35

Appendix D   : Enquête sur le VIH/SIDA

Village/Quartier:___________________

Nombre de personnes enquêtées: hommes femmesAge : 10-14 ___________ __________ 15-20 ___________ __________

21-24 ___________ __________ 25+ ___________ __________

1. (Tout le monde) Qu’est ce que c’est que le SIDA? -(Si nécessaire, reformuler : Avez-vous déjà entendu parler du SIDA ?)

2. (Tout le monde) Comment est-ce qu’on peut attraper le SIDA? Ne pas lire ces réponses possibles. Notez-les seulement si l’enquêter les dit.

Correctes Incorrectes________Sang _________Moustique________Mère-Enfant _________Poussière________Relations sexuelles _________Manger avec les gens infectés

_________Salive_________Utiliser la même toilette_________Etre proche/dormir à coté_________Embraser une personne infectée_________Par les vêtements_________Autres moyens_________Je ne sais pas

3. (Tout le monde) Comment est-ce qu’on peut éviter le SIDA?Ne pas lire ces réponses possibles. Notez-les seulement si l’enquêter les dit.

Correctes Incorrectes_______Utiliser les condoms ___________Utiliser les condoms chaque fois_______Etre fidèle ___________Dormir sous une moustiquaire_______Eviter les objets ___________Utilisant le même condom plusieurs foissouillés par le sang ___________Exciser les filles_______S’abstenir ___________Utilisant plusieurs condoms à la fois

___________Ne s’approcher pas aux personnes infectées___________Ne pas manger avec personnes infectées___________Autres moyens___________Je ne sais pas

4. (Tout le monde) Est-ce que vous pensez que la mère VIH+ peut accoucher un enfant VIH-?

_________Oui ____________Non __________Je ne sais pas

36

5. (Tout le monde) A quel moment est-ce que la mère peut donner le VIH à son bébé? Ne pas lire ces réponses possibles. Notez-les seulement si l’enquêter les dit.

Correctes Incorrectes_______pendant la grossesse _______Jamais_______pendant l’accouchement _______autres moments_______en allaitant le bébé _______Je ne sais pas

6. (Tout le monde) Est-ce que vous avez entendu parler d’un programme à l’hôpital pour protéger un bébé contre le SIDA ?

_______________Oui ________________Non __________Pas sure

7. (Tout le monde) Pensez-vous qu’il y a des hommes ici au village qui portent les capots?_______________Oui ________________Non __________Pas sure

8. (Tout le monde) Pensez-vous qu’il est important d’utiliser les capots pendant les rapports sexuels avec les partenaires en dehors du mariage ?

_______________Oui ________________Non __________Pas sure8.a. Si oui…Pourquoi?

8.b. Si non…Pourquoi pas?

9. (Tout le monde) Est-ce qu’il y a des gens avec le SIDA dans ce village, ou les villages voisins?_____________Oui ____________Non __________Pas sure

9.a. Si non, pourquoi les hommes portent les condoms ?

10. (Tout le monde) Est-ce que tu pourrais saluer quelqu’un qui a le SIDA?_____________Oui ____________Non __________Pas sure

11. (Tout le monde) Est-ce que tu pourrais manger avec quelqu’un qui a le SIDA?_____________Oui ____________Non __________Pas sure

12. (Hommes et Femmes) Si quelqu’un dans votre village avait le SIDA, comment est-ce que on traiterait cette personne?

Pas de stigma Stigma________comme n’importe qui _______on la chasserait________on la soutiendrait _______on la mettrait à l’écart________autre _______autre

13. (Elèves) Accepteriez-vous qu’un élève VIH+ fasse la même classe que vous ? _____________Oui ____________Non __________pas sure

14. Où est-ce que vous avez appris l’information sur le SIDA?_________au pièce de théâtre _________à l’école _______au CSPS_________des amis _________à la radio _______autre

37