who archives - heading 1archives.who.int/prduc2004/materials/trainers/pretesting... · web viewto...

PRETESTING EDUCATIONAL MATERIALS

TRAINER’S NOTES

PURPOSE AND CONTENT

Pretesting is an essential step in the development of educational programme interventions. This step serves two main purposes. It tests draft materials for recognition, acceptability, familiarity and relevance among the target audience. It also opens a dialogue with the individuals and groups for which the material is being developed.

This module will introduce the concepts involved in pretesting materials for visuals, text, and broadcast programmes. It will also teach participants how to plan and carry out pretesting interviews with individual and group members of the target audience. The module also describes the reasons pretesting should always be included in educational projects and the social and economic implications of not pretesting.

OBJECTIVES

At the end of the session, the participants will be able to:

1. See the importance of pretesting in order to improve educational materials, as well as to obtain information and perspectives on other project issues.

2. Understand when, where and with how many people pretesting should be done.

3. Conduct pretesting interviews on a visual and a leaflet text with both individuals and a group; assess the quality of the information obtained.

4. Understand the skills needed for effective pretesting interviews; the advantages and disadvantages of different types of interviews

5. Convince decision-makers that pretesting should be included in the project’s budget.

PREPARATION

1. Collect any materials people have brought with them, and information they have on the pretesting and use of these materials. Hang the materials on the wall. If someone has brought material in more than one version, and comments from the target audience leading to the changes in subsequent version(s), be sure to use this in the teaching.

1

2. Prepare (write some open questions - be sure to cover recognition, acceptability and familiarity/ relevance) and practise the demonstration interviews with a colleague (the visual and the leaflet).

3. Bring enough copies of the materials for the exercises.

2

Pretesting Trainer’s Notes

CORE LIBRARY

PATH (1996) Developing Health and Family Planning Print Materials for Low-Literate Audiences. Revised ed. (Available free of charge to developing country organizations or individuals from PATH: 4 Nickerson Street, Seattle, Washington, 98109-1699 USA).

Haaland A. Pretesting communication materials. UNICEF, Rangoon (out of print) (single Xerox copies available from department of Essential Drugs and Medicines Policy/Policy, Access and Rational Use, World Health Organization, 20 avenue Appia, ch-1211 Geneva 27, Switzerland).

VISUAL AIDS

1. Session title 2. Overview of session objectives 3. Pretesting assesses4. Mother with child on lap, sponging his fever: Assessing perception of

picture5. Why should you pretest?6. Uganda Red Cross Drug Use poster7. Version 1 and 2 of Nepali poster of mother feeding her child8. Litrosol preparation: Small detail makes big difference9. Main influences on quality of information10. When should you pretest11. Healthy baby poster Nepal12. With how many do you pretest?13. Individuals or groups14. Interview Guidelines - getting started15. Interview Guidelines - essential methods16. Interview Guidelines - what to avoid17. Interview Guidelines – methods to test pictures18. Interview Guidelines - methods to test text19. Interview Guidelines - methods to test radio programmes20. Summary

ORGANISATION AND KEY POINTS OF THE SESSION

Introduction and objectives - 10–15 minutes

A. What is pretesting - 15-20 minutes

B. Why should we pretest - 35-30 minutes

C. When to pretest - 5 minutes

3


Break

D. Planning a pretest - 45 minutes

Activity 1 - 30 minutes

E. Conducting a Pretest - 45 minutes

Activity 2 – 30 minutes

Introduction and objectives (10-15 minutes)

Give participants an overview of objectives of the session (OH 2)

Ask how many have participated in a pretest, and ask some to briefly describe their experience.

Participants will also have their own ideas and opinions about pretesting, whether or not they have participated in the actual practice. Establishing this “experience and opinion base” will engage people in the subject from the beginning, and help the trainer to make the explanations and discussions more directly relevant and stimulating.By doing this, the trainer is also following the main criteria for pretesting: to assess the recognition of the idea (do participants already understand what pretesting is, and why it should be used), the acceptability of the idea (do participants believe pretesting is a good idea), and the familiarity and relevance to them (do they think pretesting is something they, or their department, should be doing).

A. WHAT IS PRETESTING? (15-20 MINUTES)

Ask what you can test for when pretesting IEC materials. Use the poster from the Uganda Red Cross (URC) on rational drug use, which you have for the demonstration (OH 6). Agree that you can test for:

Recognition Acceptability FamiliarityDiscuss what this means.(OH 3 and 4)

Demonstrate a good pretesting interview on the URC poster (OH 3) in front of the group, asking questions relating to recognition, acceptability and familiarity. The demonstration will be the common framework for discussion during the session. Ask for comments and questions.

4


Agree that pretesting means to test out IEC materials before they are printed or produced. Getting feedback from the users is an opportunity to improve the materials - NOT a criticism of the people who have developed the materials (and presumably done their best)!

B. WHY SHOULD WE PRETEST? (25-30 MINUTES)

Ask participants to discuss in pairs (for 5 minutes) why we should pretest IEC materials, and what a project could gain for itself by pretesting.In plenary, list on the flipchart the reasons for doing pretesting. Add reasons from Session Notes, if these are not mentioned. Discuss briefly the possible implications, including cost, of not pretesting.Refer back to the initial demonstration of the URC poster. What could have been the cost implications for URC of sending out thousands of these posters?Relate pretesting to behaviour change, which is the aim of many public health projects. (OH5)

Research has shown that people are more likely to consider change when: they can identify with the people portrayed in the visuals, text and

radio programmes being shown to or discussed with them; the new practice shown is acceptable, accessible and affordable to

them, and the problem or situation described is familiar, and relevant to their

lives.

Pretesting can help establish if these criteria are met, and thus whether the material is a good basis for influencing behaviour change.

Ask if anybody has an example of an IEC material which fills these criteria and has been evaluated. Use the example below to illustrate this point (if no example from the participants):

5


Urban woman poster

The aim of this teaching poster (OH 7) was to motivate Nepali mothers (more than 90% of whom live in rural areas) to give their children supplementary food from about 6 months of age.

Ask participants what they think could be wrong with Version 1.

Mothers said this woman is a city woman who has nothing to do with them. Her hairstyle was very modern, her blouse was not the type village women wear. The baby is also wearing city clothes, and being fed with a spoon.

Ask participants how effective they think this poster would have been to encourage village mothers to feed their children extra foods, and why.

(Comment from the photographer: Version 1 WAS a city woman - because the male photographer did not think it was important to go to the “dirty villages” when you could get a nice, clean woman in the city....)

Rural woman poster In the second version of the picture (right on OH), more than 80% of the rural women identified the mother as “one of us” - she wears village clothes, is neither rich nor poor, and feeds her child the way village mothers do - with their hand. This “model” was recognizable from a social, cultural and economic point of view.Ask participants for comments and questions on this example.

6


Convincing managers about the need for pretesting Ask participants to discuss in pairs (for five minutes), and come up with arguments to convince their managers to give them time and resources for pretesting in their project.

Ask for a volunteer to play out his/her arguments in front of the group, using either the discussion partner or the trainer as the “manager”.

Agree that: Most decision-makers are sensitive to the idea of wasting money

and resources, and are tempted by quick ways of preventing this. “Packaging” the convincing arguments in the right way is a very

successful strategy. A good strategy is to show him or her an example of two (or more)

versions of the same materials, improved based on comments from the target audience (preferably local).

Explain to the manager that pretesting is an accepted method in projects developing IEC materials for public health.

Discuss the financial, health and institutional image implications of sending out materials which will not be understood or accepted.

Use the example of the two versions of Litrosol (OH 8) to illustrate the point.

C. WHEN TO PRETEST (5 MINUTES)

Explain that the best time to pretest is early in the process of development of the materials. (OH 10)Ask participants why this is so, and discuss.

Agree that: Materials are in a rough shape, and people you test them with can

immediately see that this is a rough draft, and will believe that you really want their comments

Project and pretesters have not invested too much energy in the materials yet, and are more open to change.

(One hour till here)

BREAK

7


D. PLANNING A PRETEST (45 MINUTES)

Explain: Pretesting is a creative process throughout the life of a project. To get the best results, pretesting must:

be done systematically have clear goals be flexible be monitored well.

D1. How many people should you test with? (5-10 minutes)

Ask participants for suggestions.

Agree that the golden rule of pretesting is: Keep on testing until you have defined the problem, or seen the trend

in the responses, and you have an idea about what the next version of the text or the visual should look like.

Agree on some main principles:Often, you will get the main impression of what the problems are within the first five to ten interviews. This is especially true early in the testing process.We recommend testing with around 20 people. Sometimes, if you get many very different responses, you may have to test with up to 30 people.If materials are to be used for several different communities, select at least 2-3 different sites for the testing. The rules above apply within communities, and between communities For a final version, 70% of your respondents should recognise the material correctly.

D2. How long does pretesting take? (10 minutes)

Present the following rough overview of pretesting time requirements for the team of two, if the audience is still within one hour’s drive:

A simple poster with one idea - 10-20 people: 1 day. A set of three teaching posters, with some text - 10 people: 2 days. A simple leaflet (A-4 size, printed one side, 4-5 main messages): 10

people, 1 day. A 10 minutes’ educational radio programme: 10 people and 1-2

groups: 2 days.

Point out: The main variables that influence the time are travel distance, number (and experience) of people in the pretesting team, number of materials to be tested, complexity and sensitivity of the materials. Also, it depends on how many versions you need to get it “right”.

8


Example: Testing the Healthy Baby poster in Nepal (OH 11)

NOTE: if you do not have much time or resources, it is better to pretest once, than not to pretest at all!

D3. Preparations for fieldwork (5 minutes)

Ask participants to pretend they are going to the field to pretest a set of materials the next day. What do they need to prepare, and how?

Get suggestions from the group.

D4. Review what you want to know (15 minutes, might have to skip group work here and reduce to 5 minutes).

Review the need to assess recognition, acceptability and familiarity. Use a visual as an example. Divide the group in three, and ask participants to discuss in pairs within the group about what they would ask to assess recognition (assign this to one of the three groups), acceptability (group 2) and familiarity (group 3).

Write suggestions from the three groups on flipcharts, and discuss. For group 3, be sure to discuss familiarity with the people and style, as well as with the idea portrayed.

Discuss what is the difference between asking these questions about a visual and asking about a text or a radio programme.

D5. Testing with individuals or with groups? (5 minutes)

Explain: Pretesting can be done with individuals or groups, depending on the purpose of the test. Ask for suggestions of when to use individual and when to use group interviews.

Agree that:

Testing with individuals can be used to define the problem with a picture or a text. Always start with individual testing.

Testing with groups can be used to identify solutions to problems in a picture or in a text. Use groups of 6-8 people.

When testing with groups, it is usually better to work with the gender groups separately.

(OHs 12 and 13)

9


D6. Who should do the pretesting? (5-10 minutes)

For a discussion about different kinds of people who can do pretesting, and the advantages and disadvantages of the various professionals, please refer participants to the Session Notes.

Ask if anybody has worked with an artist in the field. What were the experiences - positive and negative?

Agree that: Working with an artist in the field can be a good way of improving

sketches on the spot, and get suggestions and feedback from people, thus finding the “right” solution faster.

Artists need to learn how to work in the field, and they need a number of skills. Many of these skills can be developed through experience - if the artist is talented and interested. Such skills include a number of abilities

E. CONDUCTING A PRETEST (45-50 MINUTES)

E1. Arriving in the communityStress the need to introduce and demonstrate the idea of pretesting to the community leader(s) before starting your work.

E2. What to ask (15-20 minutes)Ask participants what they think are the most important attributes and skills for a pretester. List suggestions on newsprint. Emphasize “Attitudes”, “Interpersonal communication skills”, “asking good questions” and “active listening”.

Demonstrate introduction to a pretest interview (e.g. to test a brochure). Be sure to explain:

the purpose of the interview; that you are testing the text - not the intelligence of the person; that the person is the expert on the subject, that is why you are

asking him/her; invite the person to ask questions.

(OH 9)Ask participants for comments on what the demonstrator did, and how he/she did it. What was it about the introduction that convinced/motivated the respondent to be interviewed?

10


Agree that attitude and interpersonal communication skills are essential features throughout the pretesting interview, and especially in the introduction. Discuss why.If you manage to do the introduction well, gain the confidence of your respondent and motivate him/her well, the quality of your information will be good, and the rest of the task relatively easy.

Stress that this is the point where most people who are new to the method, fail. And that if you fail here, the quality of your information will be poor at best, useless at worst.

Asking good questions (10-15 minutes)Ask participants to pay attention to the kind of questions you ask in the following demonstrations (about a picture of your choice), and the quality of information you obtain.

Demonstrate closed questions: First, ask closed/leading questions. Instruct your “respondent”/co-worker to answer mainly with “yes” or “no”.

Ask participants: What kind of questions were asked? What characterised these questions? Write some of them on the flipchart. Discuss the quality of information obtained. Does the interviewer really know what the respondent thinks or feels about the picture?

Demonstrate open questions and probing: Do the same interview again, asking open questions, and probing.

NB: DON’T ASK WHY-QUESTIONS!

Ask participants to characterise the questions asked this time. Write some of them down.Discuss the quality of the information obtained.

Stress/agree: Why it is important to ask open questions and probing questions in pretesting. Agree why it is important to avoid asking closed or leading questions.Agree why it is important to be careful about asking “Why”-questions.

Listening activelyAsk participants to define “active listening”. How do you know a person is listening actively? List suggestions, and also point out the cultural differences: For example, having frequent eye contact may be acceptable in one culture, but in another it may be a challenging insult.

11


Ask if anyone can describe what it feels like to have somebody listening actively to your opinions.

Discuss/agree: To have someone listen actively to you is very stimulating and inspiring, and it motivates the person to express his or her honest opinion. Active listening improves the quality of the information you obtain in an interview, and it also affects positively the motivation of your respondent. Active listening can be learnt, through discussion, practice and feedback. (OH 15)E3. Pretesting methods for different materials (5 minutes)Summarise the differences between testing a picture, a text and a radio programme. Ask for comments, and discuss.

(OH 17, OH 18, OH 19)

E4. During the interview (10-15 minutes)

Summarize guidelines for the pretesting interview by presenting (OH 20).

Ask participants to summarise what you should NOT do as a pretester. Note on flipchart and use the (OH 16) to supplement points.

E5. Dealing with disturbancesSummarise main strategies for dealing with disturbances, and ask for comments.

E6. Recording: (5 minutes)Explain: Pretesters should always work in pairs. The interviewer asks better questions, probes deeper and listens better when s/he does not have to write down the answers as well. There is a much better flow over the interview when the interviewer can concentrate fully on the respondent.To record the results of your pretesting, you can either use a form, or develop a question guide where some main “Core” questions are formulated and asked for each picture to all respondents, and then followed up in different ways with probing. Experienced pretesters usually prefer not to use a form, as they feel it is restricting, and does not encourage probing. However, “new” pretesters often feel safer with a form to guide their work.

In this course, we will use a pretesting form.

Agree that it is important for the recorder to write out the answers fully, and why.

12


Section E: 45-50 minutes

13


ACTIVITY 1 (30 MINUTES)

Testing a poster (Uganda Red Cross, on drug use), using a recording form.

This exercise, plus discussion, takes 30 minutes.

Target audience: Rural men and women, relatively poor, with no or low education.

Poster theme: Do not share your drugs with other family members

Practice: Introduction, asking questions and listening attentively (interviewer), recording and observation (recorder), and ability to “step into someone else’s shoes, and ways of thinking” (respondent).

Work in groups of three: One interviewer, one recorder and one respondent.

Instructions to the team:The interviewer should motivate the respondent to participate, and then ask questions from the form. He/she should probe when necessary.The recorder should use the checklist for the introduction (below) to note what the interviewer did to motivate the respondent, and how he/she did it. Then, note answers on the form. The recorder should also note what the interviewer and the recorder are doing well, and where they could improve their methods. He/she should take note of WHAT is being done as well as HOW it is being done.The respondent should try to put him/herself in the shoes of one of the target audience, and respond as much as possible according to the idea of such a person. Be a bit sceptical, and let the interviewer work hard to convince you to participate, and to answer the questions.

Familiarise yourself with the recording form. Discuss possible probes you might use.Conduct one interview. If there is time, repeat the exercise, and change roles.

Discuss briefly in the small team afterwards: How was the introduction? Were the main points covered well?

What was the result? Did the interviewer ask good questions? Did he/she probe? Did the interviewer listen attentively? How did you notice he/she

was, or wasn’t? What does the interviewer need to improve? How well did the recorder do his/her job?

14


Use the checklist below to see if you have commented on the most important points.

Sum up what you did well, and points that are difficult to practice. Bring some of the main learning points to the plenary.

In plenary, the teams will give some examples of main things they learnt from this exercise. For example: Something they do very well, or a skill they really need to practise more, or insights they gained. Discuss briefly.

Checklist for the interview Interviewer keeps full attention on asking questions, probing, and

maintaining good communication. Leave the writing to the recorder. Keep neutral but friendly expression throughout. Give neutral but encouraging feedback. Listen carefully and attentively.

What to avoid when pretesting Telling people (or showing non-verbally) that they are wrong. Contradicting people. Arguing with a point. Teaching. Talking with your recorder during the interview. Talking with other community members during the interview,

except to give information/request them not to "help". Asking challenging questions, like "Did you understand"? Asking leading questions (which can be answered by Yes or No). Testing too many pictures with one person. Five to ten is usually

maximum. Testing too much text at a time. Using expressions like "show me", or "point to" as they tend to be

directive or instructional.

15


ACTIVITY 2 (30 MINUTES)

Testing a leaflet

This is a repetition of Activity 1, where the material is a text/leaflet, using a pretesting form.

Try to probe on the answers you get. To prepare for this, two participants will sit together and go through the form, agreeing what probes could be asked for the different questions. Use the aims of the text as a guide.

Target audience: Semi-urban men and women, mostly literate (primary school, some secondary)

When discussing the exercise, one group should be invited to demonstrate pretesting of their leaflet/printed material. Ask the other participants for comments and suggestions.

Work in groups of three: One interviewer, one recorder and one respondent. Take some time to go through the text and prepare possible probes. Use the same checklist for the introduction as for the first activity.

Trainers move among the groups and observe, and give direct feedback during the group’s feedback session, where appropriate.

16

who archives - heading 1archives.who.int/prduc2004/materials/trainers/pretesting... · web viewto...

Documents