pretesting communication for family planning programs
TRANSCRIPT
ILI.-
kk
PRETESTING COMMUNICATIONFOR FAMILY PLANNINGPROGRAMS
Iqbal Gureshl and D. Lawrence KincaidA PROFESSIONAL DEVELOPMENT MODULE
1. It starts with specific objectives that tell the participants what theycan expect to learn or be able to do when they complete the module.
2. The participants are able to do self-testing to match their existingknowledge and skills with these objectives, and to make assessment
of their own progress.
3. The participants are given the opportunity to look at real examplesand cases.
4. The participants are able to apply their newly acquired knowledge bydoing exercises.
There are four main features of the module that are equally important foran effective learning experience: the readings, the exercises, the self-tests, and
the examples of pretesting found in the Appendix. Some of the material in themodule may not have enough depth to acquire a full knowledge, while other partsmay seem too overloaded to some participants. However, the most learning willresult if the "learning by doing" approach is followed as strictly as possible.Therefore, a heavy emphasis should be given to the exercises of the module.Also, a close review and examination of the examples and cases given in theAppendix should be considered as critical for a maximum learning experience.For this purpose a sufficient time will have to be given by the manager to theseaspects of the module.
Earlier versions of the module were conducted at the East-West Commu-nication Institute in five days, but the participants had to work either alone or
together during the evenings as well as during the daytime. Although all sub-groups were able to successfully complete one simulated pretest in the field,everyone would have liked to have had more time for field work. This revised,final version of the module has much more content to cover than earlier versions,so it is advisable to spend at least two weeks with the module, or at least use thetwo extra days on the weekend for the participants field exercise. If the partic-ipants produce their own communications to pretest then even more time willhave to be allowed. The schedule that follows assumes that the participants will
practice pretesting with communications that have already been developed andare probably already used on a wide-scale basis. Participants who have difficultyreading English will also have to spend more time on the module.
On the following page is a time schedule that was found to be successfulwith middle level family planning communication specialists from Asia, theUnited States, Latin America, and Africa. New managers are advised to designtheir own schedule to fit their particular situation and the level of their partici-pants. Thus, our schedule should be used merely as an example for new manag-ers, not as a schedule for their own use.
246
1:30 p.m.
SUNDAY
9:00 a. m.
WEDNESDAY THURSDAY
9J Designing jj Problems
Pretest Proce- and Constraintsdures and in PretestingTechniques
10 -------- -Field
FieldField Work Work and/orReport Pre-sentation
12t^R!Jepoxt
20 —Pretesting
Field Work Communication:
Preparation Special Problems
FRIDAYi7 ReportPresentations
Report
Presentations
LUNCII
11
Field Work
LUNCH
151 DataAnalysis andReport Pre-
sentation
LUNCII
SummaryDiscussion
EVENING LOLTAN i. EVENING
Field Work ReportPreparation
MODULAR PROFESSIONAL DEVELOPMENT SCHEDULEPretesting Family Planning Communication
November 10 - 14, 1975
10:30 a. m.
3:00 p. m.
MONDAY TUESDAY
Introduction 5J Sampling
and Overview Exercises andReading: Unit I Discussion
"BasicPrinciples"
IReading
Reading Period: "Pre-
Period testing Meth-ods, " Unit IIIAppendix Cases
LUNCH LUNCH
3 Group 7J Group
Discussion of Discussion:Basic Princi- t' Pretestingples Methods"
4, Reading': 8 I Field WorkUnit II Preparation:"Sampling Pre- Selection of Corntest Respon- communications,dents" objectives and
respondents
EVENING EVENINGDesigning Pre-test Proceduresand Techniques
PRETESTING COMMUNICATION FOR FAMILY PLANNING PROGRAMS
by
Iqbal Qureshi
and
D. Lawrence Kincaid
PROFESSIONAL DEVELOPMENT MODULES
A series of learning modules for professional and administrative
staff working in development communication programs.
• John Middleton, General Editor
Oc 1977East-West Center
East-West Communication InstituteHonolulu, Hawaii
This module is designed to be taken alone, but it would be moreeffective if it was taken along with two other modules, FundamentalHuman Communication , by D. Lawrence Kincaid with Wilbur Schramm,
and Planning Communication for Family Planning , by John Middleton and
Yvonne Hsu Lin. This is important for two reasons. First, there areseveral references made to the contents and concepts of these modules.Second, these modules will help emphasize that pretesting is an integralpart of the total communication process--planning as well as productionand distribution.
The recommended sequence for taking these modules is
1. Fundamental Human Communication
2. Planning Communication for Family Planning
3. Pretestin g Communication for Family Planning Programs
Although several examples of pretesting have been included in the maintext as well as in the Appendix, it is strongly recommended that the managergather as many examples as possible from local resources. These examplesmay not give complete procedures and methods of pretesting, but they willshow the participant what happened before the communication was created inthe shape of a finished product. These examples can provide important pointsfor discussion of various aspects of pretesting in relation to the local situation .
PRETESTING EXERCISES: PRACTICE IN THE FIELD
Nothing is more instructive than having the participants take communi-cation that has already been developed (or develop one themselves), design themethod and procedures for pretesting it, conduct the pretest themselves, andthen report the findings to the other participants. If there is any artificialfeeling about doing this in the beginning, it disappears very quickly. As soonas the participants are faced with a real problem, and real pretest respondents,the "exercise" becomes a real communication pretest.
Such a field exercise can be done throughout the discussion of the module
contents--by doing each part as it comes up in the module--or after the texthas been read and thoroughly discussed. If done after finishing the text, thenthe content of the text can be reviewed after the fieldwork, when the results ofthe pretest exercise are being discussed and evaluated.
To give the module manager a better idea of how such an exercise mightwork, we have included two recent examples contributed by participants in themost recent professional development program at EWCI. Javed Sajjad Ahmad
248
OTHER TITLES NOW A VAILABLEIN THE MODULAR PROFESSIONAL DEVELOPMENT PROGRAM
Modules in revised form :
Fundamental Human Communication D. Lawrence Kincaid with WilburSchramm January 1975 ( T, C)
Planning Communication for Family Planning John Middleton withYvonne Hsu Lin January 1975 ( T, M, W)
People and Population James R. Echols January 1975 ( T, M, W)
Helping People Learn: A Module for Trainers Francine J. Hickersonand John Middleton January 1975 ( T, M, E)
Organizational Communication and Coordination in Family Planning ProgramsGeorge Beal and John Middleton January 1975 ( T, M, W)
Strangers and Changers: Consulting Roles in Social Development.Ellwood B. Carter, Jr. June 1976 (T, M, D)
The Role of Communication in Programs That Extend beyond Family PlanningRobert P. Worrall and Oliver D. Finnigan III December 1976 ( T, E)
Clinic Education Sanford Danziger, Victor Valbuena, and Jan Brittain-La Brie
(T)
Modules in prototype form :
Using the Media for Family Planning Mary-jane Snyder and Margaret White
with Merry Lee San Luis
Communication and Education for Rural Development David Kline andRobert P. Worrall with Syed A. Bahim
Using Information for Problem Solving Sumiye Konoshima, David Radel,Generoso Gil, and Elizabeth Buck
KEY: C (Case Study) T (Text) M (Manager's Guide)D (Databook) W (Workbook) E (Exercise Book)
ii
and Raheel Akbav Javed, both from Pakistan, created their own poster to pre-test with some of the Pakistani students studying at the University of Hawaii.Deborah Yoho pretested the first version of a pamphlet that she helped developfor use in the family planning program in central South Carolina. After tryingthe pretest for practice at EWCI she returned to her position at PlannedParenthood and repeated the pretest with a more appropriate group of respon-dents from the intended audience. The first and the final version are attachedfor inspection, along with the report from her field exercise. These twoexamples should give the potential manager a better idea of what can be doneand the wide range of field simulations that are possible from a creative, hardworking group of module participants.
249
IMPROVING IMAGE OF POPULATION PLANNING
WORKERS IN PAKISTAN
Report by;
Javed Sajjad Ahmad
Raheel Akbav Javed
November 14, 1975
East-West Communication Institute
East-West Center
1777 East-West Road
Honolulu, Hawaii 96822
251
EDITOR'S INTRODUCTION
These materials are part of a series of development modules designedand produced at the East-West Communication Institute to assist professionalsworking in population and family planning information, education, and commu-nication (IEC) programs in sharpening their professional skills. A wide rangeof expertise--drawn from IEC programs in Asia and the United States, fromuniversities, and from the Communication Institute staff--has been brought tobear on the development of the materials. Acknowledgement of authorship isgiven in each module. The project has been supported with a grant from theU. S. Agency for International Development.
We began this project with the major goal of producing professionalinstructional materials which, in addition to serving as the core of populationIEC professional development programs at the Institute, could be adapted andused in a variety of training and development settings. To this end we haveattempted to make each module as complete and self-sufficient as possible.The modules are self-instructional to lessen the burden on teaching and train-ing staffs, and to facilitate their use on an individual basis. We have built themodules around real life cases, problems, examples and data, and have soughtat all times to strike a balance between principles and techniques for practicalapplication.
A basic premise of our work with the modular materials is that they willbe constantly revised. As we use the materials in Honolulu, and as cooperatinginstitutions use them in other institutional settings in Asia, Africa, Latin Amer-ica, and the United States, we receive feedback which helps us refine and im-prove the modules. We are especially grateful to the 40 participants from Asia,Africa, Latin America, and the United States in the First Modular Program ofProfessional Development in Population and Family Planning IEC who helped usconduct the first full field test of the materials in Honolulu in the spring of 1974.Their critical review and commentary has been a rich source of ideas for im-provement. We owe a similar debt of gratitude to the numerous IEC expertsaround the world who reviewed and criticized the materials. A special vote ofthanks is due the Planned Parenthood Federation of Korea, which has generouslyshared with us the results of their project to review, revise, and adapt modulesfor their own use.
Recognizing the need for continual improvement of the modules, we arenonetheless sharing them in this "second revised form. " We encourage non-profit education and training institutions to use the materials, revising, adapting,translating and tailoring them to meet their needs. We would be grateful forfeedback on the nature and results of such efforts.
v
We intend to continue developing existing modular materials and will beadding modules as the need arises. Institutions interested in obtaining copiesof the modules and audiovisual support materials are encouraged to write to theCommunication Institute for more details.
The conceptualization and coordination of the project has been the workof the Task Group for Modular Professional Development. Without the creativityand hard work of these people, there would have been no modular materials:Ronny Adhikarya, George Beal, Jerry Brown, Ellwood B. Carter, Sanford
Danziger, James R. Echols, 0. D. Finnigan, Francine J. Nickerson, Ying YingHsu, D. Lawrence Kincaid, David Kline, Sumiye Konoshima, Jan LaBrie,Iqbal Qureshi, David Radel, Syed Rahim, Merry Lee San Luis, John Shklov,Mary-Jane Snyder, Victor Valbuena, Hichul Whang, Margaret White, and RobertP. Worrall.
This project was supported by the Office of Population, U. S. Agencyfor International Development, AID/csd-1059. Special thanks are due to Dr.Wilbur Schramm, whose guidance has been essential, and to Dr. Robert P.Worrall, who, as Assistant Director, gave this project the support neededto transform an idea into reality.
John Middleton
Honolulu, 1977
vi
BACKGROUND
In 1965, when the Family Planning Program was launched nationwide inPakistan, only "Dais" were employed to contact face-to-face village women forfamily planning. However, "Dais" who were generally illiterate and untrainedmid-wives of a sort, did little to spread the message effectively. In 1970, onan experimental basis, in Sialkot District of Pakistan, a new strategy of "fieldstructure" was adopted by the national leadership of the FP Program. Underthis strategy, which is now commonly known as Continuous Motivation Scheme( CMS), men and women, preferably married and with a high school educationwere employed. For every 10-15, 000 population (urban and rural), a teamconsisting of one man and woman called Family Planning Workers (FPW) wastrained and appointed under the supervision of FP officers.
The primary functions of these FP workers were
1. To map all the areas in their jurisdiction.
2. To register all married couples unless they were beyond reproduc-
tive age or were not likely to reproduce.
3. Inform and motivate those who were registered.
4. Carry and supply condoms, foams, jellies, pills, and informational
materials to the acceptors.
5. Refer to nearest clinic/hospital those wanting I. U. D. , vasectomyor tubectomy.
In 1972, when an evaluation of the CMS was carried out, it showedalmost a 20 percent increase in the number of acceptors against an average of5 percent in the rest of the country. Consequently, the CMS was graduallyspread to other districts and by 1973, almost the whole country was coveredincluding only very thinly populated areas. At a later stage, the designation ofthe FPW was changed to Population Planning Workers (PPW). Henceforth, wewill refer to them accordingly.
Subsequent evaluations did throw some light on the effectiveness of thisapproach. But one thing which stands out in these observations is the relativelypoor image of the PPW in the country, and unfortunately, not very much seemsto have been done to raise the status and image of the estimated 15-20, 000PPWs. Although it may be true that the workers themselves were responsiblefor their poor image, it is also not unlikely that they suffered all the projectedhostility towards family planning, failure of its methods and weaknesses of theorganizational structure as well.
253
TABLE OF CONTENTS
FOREWORD iii
EDITOR'S INTRODUCTION v
OVERVIEW I
UNIT I: THE BASIC PRINCIPLES OF PRETESTING 3
Objectives 4
Introduction 5
Pretesting as a Process 13
Deciding What Needs to Be Pretested 26
Pretesting at Various Stages of Development 27
Pretesting the Whole or Its Parts? 32
Various Effects of Communication 34
Selecting a Place for Pretesting 40
Summary 41
Review Questions for Unit I 43
Answers to the Review Questions for Unit I 47
UNIT II: SELECTING RESPONDENTS FOR PRETESTING 51
Objectives 52
Introduction 53
Obtaining a Representative Sample 54
Types of Sampling 59
1. Accidental Sampling 59
2. Probability Sampling 59
vii
Our assumption is that one important cause of the ineffectivenss of thePPWs is their poor reputation, untrustworthiness, and the fact that people donot regard them as a source of help, information, and sympathy. And for thisreason, it will not be only additional training, effective supervision, and orga-
nizational support that will increase PPWs effectiveness; it is also necessaryto raise their status, morale, and image to make a difference.
Although we suggest an adequate campaign to achieve this goal, we alsofeel that such a campaign should only accompany a retraining of the PPWs as
well as their seniors and a genuine shift in attitudes of supervisors towardsPPWs who are the foundation and basic limbs of the program at the people level.
PRETESTING COMMUNICATIONS FOR FAMILY PLANNING
In the Modular Program offered by the East-West Communication Insti-tute of the East-West Center, Hawaii, it just so happened that we decided totake the module on pretesting. As a requirement of the module and for the pur-pose of gaining actual pretesting experience, we decided to create a poster onthe theme of improving the image of the PPW, and pretest it with a small panelof Pakistani students who were studying at the various institutes of the East-West Center and staying in the same dormitory. A rough sketch of the posterwas prepared and a questionnaire was developed using attitude rating scales,projective questions, and structured as well as unstructured questions.
POSTER
A poster with a rough sketch was developed hurriedly. Its message was
"Family Planning Workers are Your Friends.For the betterment of your family, consult them.Family Planning Association of Pakistan"
Beside the written message, the normal national symbol of the FP Programand a line drawing of a young looking male were shown.
The main goal of the poster was to improve the image of the male work-er in the eyes of male viewers in Pakistan by showing him as an honest andfriendly looking young man, (because most male workers are young) wearingShelwar Kamelz, the dress of the average man in the country.
SAMPLE
The poster was intended for Pakistani males. For the purpose of
254
3. Quota Sampling 72
Sample Size 80
Summary 82
UNIT III: METHODS AND TECHNIQUES FOR PRETESTINGCOMMUNICATIONS 83
Objectives 84
Introduction 85
Methods for Pretesting Communications 86
The Panel Method 86
The Experimental Method 88
The Survey Method 93
Techniques for Pretesting Communication 101
Summary 118
Review Questions for Unit III 120
Answers to the Review Questions for Unit III 123
UNIT IV: CONSTRAINTS AND PROBLEMS 127
Objectives 128
Introduction 129
The Problem of Coordination 129
Problems in the Field 132
Problems Related to Production 133
Overcoming a Lack of Confidence 134
.\ PPKNDLX: EXAMPLES OF PRETESTING FAMILY PLANNINGCOMMUNICATION 137
introduction 139
viii
pretesting we obtained the latest list of Pakistanis at the East-West Center.The list had 32 names in all, of whom we excluded the names of females, our-selves, and one man whose whereabouts were unknown. Therefore, the finaluniverse consisted of 23 persons. Although the panel method does not requirea random sample, we selected eight names for the panel using the table of ran-dom numbers given in the module text in order to practice drawing this type ofsample.
INTERVIEWING
All interviews were conducted on the evening of November 12 and themorning of November 13, 1975 in the dormitory and lounge area of EWC.
A questionnaire with two parts was given to the respondents to be com-pleted by them. After the respondent completed part one (for example, pageone), he was shown the poster to be pretested. The poster was kept within hisview until he completed the second part of the questionnaire. Hardly any needarose to explain any of the questions. No probing was done. However, onerespondent strongly felt that attribute "responsible" should have been added tothe rating scale in question 4, on Page 1. One of the other respondents felt
the attribute "dedication" should have been added. In general, the questionnaireseemed to be understood by all respondents.
Seven interviews were completed by the time limit already set for thispurpose. The eighth respondent could not be contacted within this period.
MAIN FINDINGS
BEFORE SEEING THE POSTER
1. Six out of seven respondents had almost the correct notion aboutthe job of the PPWs, for example, to inform and educate peopleabout family planning and contraception.
2. Three respondents had hardly ever seen a PPW in Pakistan, while
the other three had never seen one.
3. Only one respondent had known a PPW personally for the last twoyears.
4. On five-point rating scales, on the average, respondents had posi-tive attitudes towards PPWs, for example, on attributes like com-
petence, friendliness, helpfulness, and honesty. However, they
255
Example 1: Pretesting Comic Booklets (Philippines) 141
Example 2: Printed Materials (U.S.) 149
Example 3: Themes, Slogans, and Messages (Iran) 155
Example 4: Pamphlets and Leaflets (Philippines) 161
Example 5: Pictures and Pamphlets (Pakistan) 179
Example 6: Fliers, Pamphlets, TV Playlets, and Slides(Taiwan) 193
Example 7: Flipcharts and Filmstrips (U. S.) 199
Example 8: Radio, Television, and Film (U.S.) 213
Example 9: A National Symbol for Family Planning(Sri Lanka) 227
Example 10: A Questionnaire for Improving the Qualityof Instructional Communication (U, S.) 237
MODULE MANAGER'S GUIDE 243
Pretesting Exercises: Practice in the Field 248
"Improving Image of Population Planning Workersin Pakistan" 251
"Pretesting a Pamphlet about Condoms" 261
Additional Readings 273
ix
Competent
Friendly
Helpful
Villager
Honest
Trustworthy
Gentle
Happy
Good
Clear
Acceptable
Trustworthy
Honest
Careful
rated the PPWs lazy, as compared to hard working. They alsothought that they were rural rather than urban, which is a positiveattribute from our point of view (see Figure 1).
Figure 1. Comparison of Attitudes towardFamily Planning Worker ( FPW)
RESPONDENT'S FPW IMAGERESPONDENT'S PERCEPTION OF THE POSTER FPW IMAGE -------------
Lazy 2.4 . Hard working
Incompetent 3.0 *'a 3.3
Unfriendly 3.1 4 3.7
Harmful 3.1 3.7
City dweller 3.3 4 3.6
Dishonest 3.3 3.7
Not trustworthy 3.9 {
Rude
Sad 2.3 r'"
PERCEIVED POSTER ATTITUDE -•-•-•-•-•-
Bad 3.8 a
Vague ,y 4.3
Unacceptable 3.1 ^'•
PERCEIVED FPW IMAGE ...........••.
Not trustworthy 1.9.
Dishonest i 2.4
Careless 2.1
1 2 3 4 5Rating Scale
256
OVER VIEW
All communication that is produced for a family planning campaign iseventually "tested" if it is ever used in the field. The results of this ultimatetest often remain unknown, however, except for rather unsystematic feedback,such as an informal report that the posters never got out of the warehouse in oneprovince, or a phone call or letter from someone pleased with the last series ofradio messages. Occasionally, clinic personnel may ask clients where theyheard about family planning, or what made them come to the clinic. A certainpercentage may mention your radio messages; others may recall a particularposter. The majority, however, are likely to say that their friends told them.Many cannot rember exactly where they heard about family planning, or whatinfluenced them the most to come to the clinic. It was probably a combinationof sources and messages over a period of time that finally influenced them tocome to the clinic.
This situation makes it extremely difficult to determine what one particu-lar communication (a series of posters or a radio broadcast, for example) hascontributed to the total communication effort. The feedback that is received mayonly be about especially good communication products, or more likely, espe-
cially bad ones which someone feels are seriously hurting the campaign. Butin between these two extremes is a large number of products about whose effectwe know very little.
How can we discover the effects of our communication for family plan-ning programs? The large-scale knowledge, attitude, and practice (KAP) sur-vey of a national sample may eventually give us some indication about changesin the audience's level of knowledge, attitude, and behavior regarding family
planning. But there is usually no way to find out how much each specific com-munication contributed to these overall changes. Furthermore, many othersources of communication, as well as other factors, have also contributed tothese changes. In fact, it is quite possible for a particular communication suchas a radio spot-announcement to have a negative effect--to confuse the audienceor create negative attitudes--while the overall communication program haspositive effects. In other words, the improvements which are shown in theresults of a KAP survey might have been even greater if some of these negativecommunication products had been eliminated or changed before they were usedon a large scale.
The problem with full-scale evaluation methods like KAP surveys, theanalysis of clinic acceptor data, fertility, and so forth, is that:
1. The results become available much too late to make useful changes
in our communications.
1
AFTER SEEING THE POSTER
1. Only three respondents were close to the designer's intent, in theirthinking about the poster. They said that the poster showed a devotedPPW worker providing family planning guidance and a friendly PPW.Others thought the poster was meant to be an advertisement of FP.One thought the PPW shown on the poster looked like a salesman orInsurance Agent.
2. In general, respondents rated the poster positively on a five-pointscale. They rated it good, (Aug. Score 3. 8), clear (Aug. Score 4. 3)and acceptable (Aug. Score 3. 1).
3. Only one respondent said that the purpose of the poster was to tellpeople workers were friendly and helpful. Another respondent saidthe purpose was to draw people closer to PPWs. Four regarded itas advertisement for family planning and one said it was difficultto conclude.
4. Interestingly, all respondents felt that in Pakistan, common peoplehad negative attitudes towards PPWs. On our five-point rating scale,
the average scores on trustworthiness, honesty, and helpfulness ofthe workers were below 2. 5. If we consider their responses as pro-jections of their own feelings, respondents did not hold high opinionsof PPWs themselves.
5. Majority of respondents thought that when the poster is shown acrossthe country, the image of PPW will improve. One respondentthought the image would deteriorate.
6. Only three respondents said their own opinion improved after seeing
the poster. Two reported no difference and one respondent said hisopinion of PPWs deteriorated after seeing the poster.
7. Any other remarks were classified into four categories as follows:
LanguageNo specific changes suggested although some said local and/orsimple languages should be used. One respondent suggested that we
replace the existing second line on the poster with the following:
Do you want your family to be prosperous? If yes, you should havefewer children. You should consult PPW and profit by it. Theseworkers are your and nations servants.
257
2. The results are difficult to connect, or to relate, to individual
communication products.
The obvious conclusion is: Why wait? Why remain ignorant of this kindof effects that each specific communication will have when there are wars to
find out for sure ?
There is much that can be done to test and to improve communicationbefore it is widely used in our programs, and before large amounts of moneyand time are spent for communications that have little, if not harmful, effecton our audiences. By developing a careful program for systematically pre-testing the messages that we produce we can avoid unforeseen mistakes andi mprove our communication before it meets the ultimate test of the full-scalecampaign. In fact, it is the nature of the human communication process andour family planning programs that makes pretesting so important: The resultsfrom our pretesting may be the only ones we ever see.
OBJECTIVES
When you are finished with this module, you should be able to do thefollowing things:
1. Explain the rationale and basic principles for pretesting communication.
2. Design appropriate procedures for sampling respondents for pretesting
communication.
3. Design the procedures and techniques for pretesting the effects of com-munication at various stages of development.
4. Analyze, interpret, and utilize the results of pretesting to improvethe effectiveness of communication.
5. Describe the problems which are often encountered in pretesting andrecommend various ways of overcoming them.
Several different methods will be used throughout this module to accom-plish these objectives. In addition to reading this module text, we will participatein group discussions of the role of communication in the family planning programs,and how pretesting can be utilized to improve our communication. Each participantwill share his or her own experience about the current use, underuse, and misuse
of pretesting. We will review and analyze selected examples of pretesting whichhave actually been conducted on family planning communication. And finally,each participant will have the opportunity to conduct a simulated pretest of actualfamily planning communication.
2
Figure
1. A smile on the face of the worker. (3)
2. Good looking figure on the left and insignia at the beginning. (1)
3. Female worker being consulted by a women. (1)
4. Should be shown walking. (1)
LayoutShould be improved. (2)
Color1. Picture in black and white and figure in green color. (1)
2. Word "friendly" should be in a different color from rest ofscript. (1)
3. Last line should be smaller and in different color. (1)4. Color does not matter. (1)
Only two respondents who thought the poster was unacceptable to themgave reasons. One said that the poster is only in one language and the figureis not proper. The other said that a female worker should also be shown sothat the emphasis is on the couple.
For further probes about the appearance of PPW in the poster, a five-point rating scale was used on eleven attributes, such as sad or happy, unfriend-ly or friendly, rude or gentle, etc.
happy sad
The average score was above three for all indices (showing generallypositive attitudes) except that he looked relatively unfriendly and sad. Obviously,the last attribute requires changes in the art work.
When compared with their attitudes towards PPW prior to seeing theposter, it seems obvious that their attitude became more positive on points ofhelpfulness and honesty but changes towards being neutral regarding friendlinessof the PPW. in other words, the poster created a slightly negative feelingtowards the PPW, but still within the positive side.
CHARACTERISTICS OF THE RESPONDENTS
All respondents were relatively young being in the age group of 25-34years. Professions were not clearly stated. All of them had postgraduateeducations. Six were married with the average number of 1.6 children.
258
UNIT I
THE BASIC PRINCIPLES OF PRETESTING
MAJOR CONCLUSIONS
Considering the findings of this survey, the following conclusions seemwarranted regarding improvements of the poster:
1. Worker should be shown happy or smiling.
2. Posters with both male and female figures should be prepared.
3. Figures should preferably show worker in action (walking, etc.).
4. Posters should be prepared in the language of the area where it isto be shown.
5. Name of the organization issuing the poster should be in smallertype.
According to the pretest results, this poster with slight improvements assuggested above can be displayed in Pakistan with success. As the findingsindicate, three out of seven respondents have already mentioned improvementin their attitudes towards PPW after seeing the poster. However, rememberingthat our sample was atypical (for example, highly educated, middle class) it
will be necessary to pretest it again with less educated people back in Pakistan,especially after making the necessary improvements.
259
OBJECTIVES
After completing this unit you will be able to:
1. Explain why pretesting is important for developingmore effective communication.
2. Describe the difference between evaluation, evaluativeresearch, and pretesting.
3. List and describe the basic steps for developing andpretesting effective communication.
4. Write goals for communications with which you arefamiliar, according to three criteria which make themclear and practical for pretesting.
5. Explain what needs to be considered when deciding whatto pretest at various stages of communicationdevelopment.
6. Describe seven general effects of communication whichmay be pretested.
7. Explain the advantages and disadvantages of pretestingin a variety of places and settings.
4
INTRODUCTION
In recent years a great amount of money has been invested in familyplanning communication activities all over the world. Everyone is interested inknowing what kind of results these resources and manpower are producing. Someexperts in the field believe that a great deal of resources have been wasted in thisarea, perhaps more than in other development programs. Most family planningprograms, however, are still relatively young and deal with especially difficultkinds of change in human behavior.
Nevertheless, the failure to achieve widespread acceptance and practiceof family planning is very often attributed to ineffective communication. Herelies the great challenge for professional family planning communicators.
WHY DOES COMMUNICATION SOMETIMES FAIL ?
Before reading any further, list the most important reasons why youthink family planning communication sometimes fails to accomplishits objectives:
1.
2.
3.
4.
5.
Some of the reasons you have listed probably emphasize the practical prob-lems that cause family planning communication programs to fail. For the moment,let us not concern ourselves with the bureaucratic and administrative problems ofdelivering family planning services, except where they affect our communicationprograms. These are not within the scope of this module. We will concentrate onbetter professional methods to make our communication more effective.
5
PRETESTING A PAMPHLET ABOUT CONDOMS
Report by:
Deborah W. Yoho
November 14, 1975
East-West Communication Institute
East-West Center
1777 East-West Road
Honolulu, Hawaii 96822
261
If we look very closely at the causes of communication failure, oneproblem seems to stand out above all the others: This is the lack of informa-tion on the part of communicators about their audiences, and how their audiencesreact or respond to the messages that are produced. Communicators often designmessages on the basis of their own professional expertise, technical judgments,intuitions, and so forth. In doing this, they sometimes forget about the audience.This may be a practical approach, especially if the audience is unknown or notvery clearly defined. Sometimes this might work, but the chance for failure isalso high. Regardless of the outcome, however, no one is able to say whatworked or why, nor what failed, and why it failed. Professional progress isseverely handicapped under these circumstances.
We have learned from communication research that meaning is not inthe messages we create, but in the people who interpret them. Peopleselectively attend to messages, selectively perceive and recall their content,and interpret them according to their own past experiences, their own likesand dislikes, and the values that they share with their friends and relatives.Thus, to make our messages more effective, we have to know as much aspossible about the characteristics and reactions of our audiences. Or simply,we have to know our audience and how they will respond to our messages.
We should ask ourselves, "How often and how much do we keep ourintended audience in mind while developing family planning messages?" Many
times we design our messages for someone else. We use our own professionalor artistic standards of quality because we are concerned about the reactionsand comments of our superiors and colleagues where we work. How will theyjudge our work? What will they say about it? Naturally, their reactions areimportant. Certain standards of excellence must be met. Their opinions areimportant. But we have to understand that this kind of approach by itself willnot help our superiors, nor ourselves as professionals, if our work has littleeffect or unintended effect on the audience for the family planning program.
To improve our effectiveness as communicators we must go to ourintended audiences in the field. It is not enough to know what our messages
mean to us, or to our colleagues. We must know what they mean to theaudience. What do they see? How do they feel? Do they understand? Do theybelieve it? Agree with it? What will they do about it?
COMMUNICATION PRETESTING AS EVA LUATIVE RESEARCH
No two words can create as much "fear" and misunderstanding amongadministrators and IEC production staff as "evaluation" and "research."Communication pretesting is a part of the evaluative research process, so itmay be useful to clarify what these terms mean. Simply stated, evaluationattempts to answer the question, Is it any good ? For example: Is the family
0
planning poster beautiful? Does it meet our standards of professional quality?Is the idea a good one? These kinds of questions are important is one wants todo good work, and be rewarded by one's professional colleagues. *
Evaluative research , on the other hand, attempts to answer an entirely
different question: " What is it good for ?" When communication is produced itis necessary for each product to do more than just be appreciated by theaudience. Each product must contribute in some way to the stated objectivesof the program. The purpose of evaluative research is to find out what and how
much a given activity or product contributes to these objectives. If the statedobjective of a poster is to help a certain segment of the male audience learn theadvantages of a vasectomy, it is quite possible that an ugly poster (ugly to us,at least) might lead to more learning than a beautiful one.
This takes us to our second term, "research. In evaluative researchwe do not just assume that a certain effect will occur; we try to find out forsure. What does it mean to do research? Research is nothing more, nor less,than careful and systematic investigation of a problem or question about what ishappening in the world --about what is and about what is related to what. Whathappens, for example, if one uses bright red letters on the poster for theadvantages of a vasectomy? Does the passerby pay more attention to it? Doeshe remember this part of the message more easily? In other words, what is
the relation between the color used for part of a poster's message and its effecton the passerby's attention and learning? The thing that makes this researchuseful is that you actually find out what effect your messages have on yourintended audience. You investigate such questions carefully and systematicallywith reliable methods of research. Most importantly, the results of this kindof investigation do not replace our own professional judgment based upon ourprevious experience and expertise. These results are used to improve ourjudgment and the decisions we must make in our daily work.
To get information about the effects of our communication upon theopinions and reactions of the audience we can do evaluative research at twodifferent stages in our program: during the operation of the program, or at theend of a specific communication program or activity. + At the end of a
* For information on evaluation, see John Middleton and Yvonne Hsu Un'smodule Planning Communication for Family Planning (East-West Communicationinstitute, 1975) or E. M. Rogers and R. Agarwala-Rogers, eds.., EvaluationResearch on Family Planning Communication (Paris: UNESCO TechnicalReport, 1975).
The first type of research is often referred to as "formative" evaluation, thelatter, as "sumrnative" evaluation.
7
SUMMARY FOR THE ORAL REPORT
BACKGROUND
Purpose of the pretest to revise and improve a brochure about condomsdesigned for distribution primarily to women requesting and receiving birthcontrol information (in a clinic). Some men also receive the leaflet uponrequest or the woman shows it to them. More than fifty percent of the clinic'sclients are nineteen years old or less. Over half of the clients have had somecollege education. And over sixty percent are unmarried.
METHOD
1. Accidental sample: Chose four men and eight women at Universityof Hawaii Manoa campus who happened to be either in or near thelibrary and willing to give ten minutes of time to the research.Selected by interviewer. Criteria for selection: Appeared to havesome time to spend.
2. Interview: Person read the pamphlet, commenting both during and
after reading. Interviewer subjectively drew out the person's com-ments and wrote them down.
3. Questionnaire: Person then filled out questionnaire. Interviewerrecorded comments as they appeared.
RESULTS (see attached reports)
1. Perception: Generally positive (see chart). Most positive percep-tion was that it was informative. Most negative was that it wasunattractive. Author hoped the wording would be funny and/orentertaining. Audience generally did not respond accordingly.
2. Attitudinal change: Inconclusive except that seven out of twelvesaid they would show a copy to a friend. Reinforces informativeusefulness.
3. Recall of information: Much confusion about whether more expen-sive condoms are better. Some (two) unclear about space at end---a significant point.
4. Additional comments: See attached report.
263
communication campaign, for example, we can conduct surveys to find out theincrease or decrease in the knowledge or attitudes of our audiences, which mayhave resulted from our communication program. We can analyze the recordsand reports that have been kept which tell us how many new family planningacceptors have been recruited as a result of our program. We can analyze themost recent national census data to determine what has happened to the fertilityrate over the last ten years while the program has been in operation. This kindof evaluative research is widely used and it provides valuable information aboutthe overall impact of the family plam7ing program, both its communication andits service delivery components.
The limitations of research done after a program of activity is completedhave been mentioned already. There is no way at this stage to take correctivemeasures, and most likely the program will have been in operation quite a longtime before this kind of research is completed. The results are delayed and areuseful only for planning and operating future family planning programs andactivities. In other words, this kind of evaluative research is done after aprogram or important part of a program is finished, and the goal is to increasethe effectiveness of the whole program in the future.
Evaluative research on the communication component may also beconducted during the time the program is being developed and run. Doing the
research beforehand or during a program allows us to avoid serious programmistakes and to eliminate many unnecessary activities. Pretesting communication
products is one of the most important kinds of research at this stage, because
it provides information about the audience's opinions and reactions beforeundertaking final production and widescale use. It may provide useful leadsfor improving the material or the program. Pretesting is done before or duringa program, and the goal is to improve quality of the communications to be usedin the program.
Let's stop here for a moment and look at the following description of howa family planning poster was developed in Kenya. *
Barbara Yount, ed. , IEC Newsletter , no. 15. (East-Vest CommunicationInstitute, 1974), p. .5.
8
RECOMMENDATIONS
1. The general tone is inoffensive, easy to understand though notparticularly funny. Leave it unchanged.
2. Clarify the points of the one-half inch space and expensive condoms.
3. Make it more colorful and attractive (illustrations).
4. Consider combining with information on foam.
5. Research more regarding the opening paragraph.
6. Re-phrase the part about the man "coming
FINDINGS
SUBJECTIVE-QUALITATIVE COMMENTS MADE
1. Written in an easy manner.
2. Learned something I did not know before.
3. Devote equal time and space in brochure to discuss spermicides.
4. Most people really want this information. It will automatically holdyour attention.
5. Add more color; improve appearance; add illustrations.
6. 1 would expect men to react differently to the questions compared to
women.
7. Most men pretty well know this stuff.
8. I am glad the interviewer is female. (Respondent female.)
9. What do you do if it does break?
10. Introduction unnecessary and negative.
11. Looks like a passout sheet at a student demonstration.
12. Title at the top would be useful.
13. Omit the phrase "to come. " Embarrassing to me. (Respondent male.)
14. The introduction is catchy and funny.
264
THE BIRTH OF A POSTER: FROM IDEA TO PRODUCT
The idea of the poster "Why Carry More Burdens" wasconceived by S. Raheem Sheikh of IPPF's African Regional Office
in Kenya while he was reading a book. In that book, entitledLife After Birth , author Stanley Johnson says that a young motherin the developing world is carrying two burdens--a baby in herbelly and another on her back. The words triggered off a visualimage in Sheikh's mind, and he commissioned an artist to make"poster-roughs," pictured on the next page.
To enhance its visual impact, Sheikh wanted the poster tobe without the clutter of any background images, such as huts orvillages, which would detract from the main subject of the picture.In short, he wanted the poster to be clean, to avoid any unnecessaryand confusing visuals, and to have typographical impact by means
of a short and precise message expressed in a bold and cleartypeface. The stark boldness of the poster was enhanced stillfurther by using only two colors, black and brown on a whitebackground.
In order to save both the time and money involved inextensive pretesting and market research, Sheikh evaluated hisposter with people in his office, with patients at a family planningclinic in Nairobi, and with a small sampling at a rural clinic out-side Nairobi. The questions asked were very general, such aswhether the poster conveyed the message to the people without awritten slogan and whether it made any difference when the head-line and sub-head were added. The general impression was thatthe message was clear and simple. Most of the comments were:that the legs of the woman were not right, that her eyes and nosedid not look African, or that the cloth around the child was tied atthe wrong place. The necessary corrections were then carriedout by the artist to the satisfaction of the rural audiences.
The poster was then presented to a meeting of the Informationand Education Officers from the Africa Region with its differentoverlays. One showed the woman with five children and the otherwith seven children. Opinion was also asked as to whether a small
patch on the skirt would be useful or not; this was rejected, and infact the officers said that the woman should wear shoes or sandals.In the end, it was recommended that the poster picture just a
9
Please indicate:
1. age
2. sex
Average: 21.4 yearsRange: 17-26 years
Male: 4
Female: 8
3. nationality
American: 6
Japanese: 2Am. Chinese: 1
Chinese-Hawaiian: 1
Chinese-Japanese: 1Chinese-Italian: 1
7. interesting
8. unattractive
9. funny
10. tasteful
11. confusing
5 4 3 2 1
1 2 3 4 5
5 4 3 2 1
v_5 4 3 2 1
VI
1 2 3 4 5
12. uninformative1 2 3 4 5
13. entertaining r/5 4 3 2 1
STATISTICAL RESULTS OF THE OPINION QUESTIONNAIRE
Your answers to this questionnaire will be used to develop better mate-rials to inform others about family planning. YOUR ANSWERS ARE COM-PLETELY CONFIDENTIAL. WHEN YOU HAVE FINISHED, PLEASE FOLDTHE QUESTIONNAIRE IN HALF AND RETURN TO ME.
Please check any of the followingthat apply to you:
4. 9 undergraduate student
3 graduate student
5. 2 married
10 unmarried
6. 11 resident of Hawaii
1 othe r
How would you describe the leaflet you have just read? Please check the blankmost closely corresponding to your opinion.
(SEE Figure 1)
Mean
dull (4.0)
attractive (2.4)
serious (3.0)
tasteless (3.3)
easy to (4.0)
understand
informative (4.3)
boring (3.0)
265
CARRY
1 ^^
CARRY
..
n
!.
1 2 3
T1 II 1.1r
DEVELOPMENT OFA FAMILY PLANNING
POSTER FOR KENYA
5
4
10
2.2 18. I would not rely onthe use of condomsfor effective birth
control.
response: 9/12
2.8 19. I would ask for morecopies of this leafletto give to others.
response. 10/12
2.6 20. There is nothing
unusual about awoman carryingcondoms in herpurse.
response: 11/12
2.7 21. I would recommendthe use of condomsto a friend.
response: 10/12
2.45 22. Most people are notembarrassed to buycondoms in a drug-store.
response: 11/12
14. embarrassing — — / not (3.5)1 2 3 4 5 embarrassing
15. happy _ ELIMINATED _ sad5 -4 3 2 1
16. offensive _ 3 _ inoffensive (3.6)1 2 3 4 5
17. colorful _ ,/ colorless (3.1)
5 4 3 2 1
Indicate whether you agree or disagree with the following statements by check-ing the box closest to your feelings:
Mean Score
4 3 2 1
stronglyagree agree disagree
stronglydisagree
4 3 2
1 6 3
7 4
1 5 4
1 4 5 1
266
pregnant woman with a baby on her back and with a child looking upto her. Also, the message on the bottom of the poster was changedfrom upper case to upper and lower case typefaces.
The poster was tested again in accordance with the firstprocedure outlined above. However, it was not until a widerdistribution had been achieved that some women in the countrysidestarted talking about the "exotic shoes this woman was wearing."They did not identify her as one of them or as a woman who shouldfeel the burdens of an unspaced family. Thus, to avoid unnecessarydiversion of the main message, the next printing will show thewoman wearing simple sandals.
Initial reports from Kenya indicate that the poster is success-ful in urban and rural areas--even among those who cannot read theEnglish text. Furthermore, during just one week requests for theposter in its current version have come from places as far apart asTehran and Kuala Lumpur.
For more information about or a copy of the poster, write to:Mr. Raheem Sheikh, Regional Information and Education Officer,IPP F--Africa Region, P.O. Box 30234, Nairobi, Kenya.
11
Please write the letter "T" next to the statements that you think are true, andan "F" next to any statement you think is false.
CORRECT-INCORRECT-NO ANSWER
23. 10-0-2 It is necessary to leave a space at the end of the condom whenit is put on. TRUE
24. 9-2-1 Condoms may be left in place after the man has lost his
erection. FALSE
25. 4-7-1 The more expensive condoms are reliable. TRUE
26. 10-1-1 A wallet is a good place to store condoms. FALSE
27. 11-1-0 Condoms can be lubricated with Vaseline. FALSE
28. 12-0-0 Condoms can be used more than once. FALSE
29. 11-0-1 Condoms and birth control foam may be used at the sametime. TRUE
30. Is there anything else that you would recommend or suggest in order toimprove this leaflet?
267
The Kenya poster for "spacing" is a good example of the process that onegoes through to develop new communications for family planning programs. It isvery useful to think of the development of a new design as a process with several
steps rather than something that can be finished in just one attempt.
Look closely at the first version of the poster, then quickly switch to thefinal version. Notice what a big difference that seemingly small changes canmake in the overall impact of the poster. The final version is much more clear,and more simple. Compared to the first attempt, it seems to focus our attentionbetter on the main point. The other children in the earlier versions distract usfrom the mother's immediate problem with spacing. Several changes in thedesign were made before this clarity was achieved, using different kinds ofprocedures and techniques for pretesting in the process.
First came the general idea for the poster. We can assume, however,that before this, spacing births had already been identified as an importantproblem for a certain audience in the family planning program. In the secondstep, the idea was transformed into an initial image or design which could bedrawn and tested. In the third step, the initial design was pretested with otherpeople in the office and with patients in a rural and urban family planning clinic.In the fourth step, the poster was changed according to the recommendationsfrom the pretest. In the fifth step, it was presented to a group of experts fromthe African region. They were the ones who suggested that it be simplified byeliminating all but one of the children on the ground. Unfortunately, they werealso the ones who suggested adding the inappropriate shoes. After these changeswere made (step six), the revised poster was pretested again "in accordancewith the first procedure outlines above" (step seven). Only after the wider dis -tribution with the intended audience (step eight) was it discovered that the "exoticshoes" were causing a serious distraction from the main message about spacing.Fbrtunately, this last change was a small one, and could be made before thesecond printing of the poster.
This is an important lesson for us: The groups that we select for ourpretests may be good for some purposes but not for others. Problems arisewhen we try to "get by" with just one type of group, especially if we do notinclude respondents who adequately represent our intended audience. The ex-perts above, for example, had very good ideas about the overall compositionand design, but should not have been expected to know how local audiences wouldreact to the bare feet or to the shoes. Apparently the clinic patients chosen forthe pretest were not representative enough to catch the problem with the shoes.To answer that kind of question, we need to take our new communications directlyto members of our intended audience. In the case above, this was not possibleuntil the widespread distribution of the first printed version (with the shoes).
12
4.
4.3
Figure 1. Respondents' Average Perceptions of the Leaflet
POSITIVE NEGA TI VE
4.interesting
attractive
funny
tasteful
easy to understand
informative
entertaining
not embarrassing
inoffensive
colorful
dull
2.4 unattractive
serious
tasteless
confusing
uninformative
boring
embarrassing
offensive
colorless
5 4 3 2 1Rating Scale
268
We will have more to say about selecting pretest respondents who arerepresentative of the intended audience in Unit II. Before we turn to the prob-lems of sampling, however, let us continue with the idea that pretesting proceeds
through several steps along with the development of better communications. Inthe next section we will examine a model of the pretesting process, and discusseach step in greater detail.
PRETESTING AS A PROCESS
Planning communication, creating specific communications, and conduct-ing pretests of communications are all part of the more general process of humancommunication. To make this process more clear and to organize our workbetter, let us follow the usual practice of breaking up this process into a seriesof steps to develop more effective communication. On the following page, adiagram of the pretesting process is presented.
All of the six steps given in the diagram are important for the effectiveuse of pretesting, but depending upon the objectives of our pretesting, some ofthese steps may be skipped or treated more lightly while others may requiremore time and attention. Following these steps, however, will make pretestingeasier to do. It will also make the results of pretesting easier to understand andapply to the rest of our work. Now let us discuss these six steps in more detailand look at some examples from each one.
STEP 1: COMMUNICATION PLANNING
It is useless to pretest communication without some idea about whichaudiences it is designed for, the kind of changes that it is expected to produce,
the planned goals of the communication, the overall strategy with which it is tobe used, and the communication approach used for its design. Before anypretesting is done every effort should be made to study the planning which wasdone before the communication was produced. In some cases it may be possibleto get a written plan. Then you can find out where the specific communicationto be pretested fits into the overall plan. Ideally, of course, those of us who dothe pretesting will have participated in this planning process. If no written planexists, and the pretesting team did not participate in the planning process, then itis necessary to talk to those who did the planning, or produced the communication.
We cannot discuss the whole planning process in this module. A specialmodule, Planning Communication for Family Planning (1975), has been prepared
13
FIRSTVERSIONOF THEPRETEST:
MAKING LOVE WITH A RAINCOAT ON?......The condom (or rubber) isthe only method of birth control known that is used by the man.It is also the only method that helps to protect both the manand the woman from V.D.
A few years ago, most men and teenagers, too, carriedcondoms in their wallets to flash around as a sign of theirmanliness! Today, some men don't like condoms because theythink it cuts down on pleasure, like "making love with a rain-coat on ' . This is not necessarily true. If your man has neverused a condom, perhaps he will at least try it. Properly used,the condom is very effective. If the woman puts on the condomas part of the loveplay , this will help end any awkwardness ata romantic moment.
MOST COMDOMS COME WITH NO INSTRUCTIONS1.....But for the condomto work, it must be used correctly. It is a very good methodif used right.
AND IT ISN'T HARD .........The condom must be rolled on to theman's penis after it has become hard and erect, but beforethe first time the penis enters the woman's vagina.
AT A MOST UNCOMFORTABLE MOMENT .........Don't roll the condomon all the way. Leave a space of about ½' at the end, so thatwhen the man comes, there is someplace for the liquid to go.Otherwise, the condom might break at a most uncomfortable moment.
DON'T GREASE!.......... Do not lubricate the condom with Vase-line or anything.else. Such stuff might eat away at the rubber.If you want lubrication, use a condom that comes that wayalready.
BEFORE HE COMES DOWN ......... .After the man comes, he shouldremove his penis before it becomes soft again. As he does so,he or the woman should hold on to the top end of the condom se
it won't slip off and spill.Do not use a condom more than once.
1̂^ ribber For extra protection: Condoms ares
1 " even more effective when used at thesame time that a woman uses contra-ceptive foam. The foam will also
SpE{m naturally and safely add lubrication.
"BUT IT'S TOO TIGHT!"....If the man complains that the condom is too
the CONDOM tight, try another brand. Condomscan be purchased in any drug storeor in a discount store. Don't he
embarrassed to ask for them. The druggist sells many each day.There are many brand names, including 4xxxx (Forex), Trojans,Sheik, Lambskin, etc. They cost between 70C and $2.00 for apackage of three. Condoms are also cheaper by the dozen.Generally, the higher- priced ones are more reliable. Don'tbuy condoms from a vending machine, and don't keep them in awarm place such as the glove compartment or i;t a wallet. Awoman may wish to carry some at all times to be prepared".
REMEMBER
When you go on the pill, you must use someother method of birth control, such as condoms,for the first month that you are taking thepill. This gives your body time to adjust tothe pill, which will not become 10O/ effectiveuntil this month has passed. To be protectedduring this month, condoms or foam must alsobe used. A couple may wish to use both.
rti^ td,rer bq eko*ce
A PROCESS MODEL OF COMMUNICATION PRETESTING
STEP
Accept forWide-Scale
orSelective
Use
/
CommunicationPlanning
i. Audience Analysisb. Goal Setting
c. Designing Strategy\d. etc.
STEP VI'*- ~ \ /STEP
Reporting & rejection VDesi nin
Implementing change g tigCommunication
theResults
STEP V\ /STEP Ill
Analyzing & DesigningInterpreting Methods for
the PretestingResults
STEP
Conducting
thePretest
M O
o m R ^
C E m ° _ 0 y m trO^?^Eor#H;m m a E o
m C m O^0 m r C V R r .L-.
7;
r m? 7 m
DE c m - — E
C L% m V T4 VS. 7 m ld m t9 25
U^6 m U 3 .L.. m .fie
mE c m L
° Vm m
y t a C
0 0 0 E r n m` c
E g o vU E E m L ca c o^ Y¢nwc°'o5r' C i^ O 0 W 4
Q CD i L
m
°-
m2r
o C ° am I cEE` L° ^n
m r gy
m° omnEo°o
_ t m C f,T ° C
;^E!'n ^n^^L #U 7
FINAL VERSION FOR USE IN CENTRAL SOUT H CAROLINA
W y's c n m
m a c m° c^°' W S^o^'
E
u E 3m._ L_-O i
y m p =L
^C 7..o m C 9 mfl }' C O O O m^nOt!°Eo^;mm O m m m m m O m n
L E E mm - E n_ v
_ L ^
8 C L m ^
p O iE ], C r m mE `1 O ` b m_ ^C C
m m c n> = g E C
r a c
O p O ^_ n m O ^ U 3 E
m m c^ nm E^^- o__
3 0
m^ V^ A^^ m O d m
T^ ucnvcc.t^ cEm^C m = ^ U m V 0 C m Em o m m_ o P E
d E u m E m 3 °n ° m°
0 O Om m m O y
% m ^p ' a c
m m m m n m
E M n` E
E L m m m
L ' L ° C m L3 y G L O 3 m E
° m=
2: c v r E
@
m m ^C mbS_CFe° T0E eEmaO t m _ K n p m O
L d
m 6 >^'p 3 E E as c
` c 2 mn
M 0
O m aH .L. U C!) ^p L U U L.. U Q m '^
270
by John Middleton and Yvonne Hsu Lin for this purpose. Anyone who isunfamiliar with audience analysis, goal setting, strategy design, and so forth,should read this module. Meanwhile, we will present a brief review of the firstthree phases of their communication planning model, since these phases are themost important for pretesting. If systematic planning is not done, or if nowritten plans are available, then we can use these phases as a guideline forasking questions about the communication which we have been asked to pretest.
(a) Choosing and Analyzing Audiences --The way that people interpret our
communication, the meaning which they give it , depends upon their own ways ofthinking and concepts that are determined by their past experiences in life andtheir expectations for the future. * Before designing communication and pre-testing it, we need to know as much as possible about the audience for which it isintended, why they were chosen, and what kind of prior experiences they mighthave had. In general, some questions we should ask about the audience are
• How well educated is this specific audience?
• What percentage are literate?
• What language(s) or dialect(s) do they speak?
• Is it for men, women, or both? Of what ages?
• What is their most common religion?
• What kind of work do they do? What is their social status?
• What kinds of social groups are most important to them?
• Are they married? How many children do they have already?How many do they want? Why?
• What percentage already know about family planning? Have practiced
family planning? Have discontinued family planning? Why?
• What are the most important values of this audience? Why do theywant children? Why would they be willing to have less children?
• What channels of communication does this audience enjoy the most?Which channels do they use for news ? For advice about familymatters?
* For a more complete discussion of the basic principles of human communicationsee D. Lawrence Kincaid with Wilbur Schramm, Fundamental Human
Communication: A Professional Development Module (Honolulu: East-WestCommunication Institute, 1975).
15
m C u.- m
m`mis8°c^=03^eE mo
U C 0 ._ m N 15 y]
vp
.t '► ° IS c° 0 E m v E E C c " v
EDm N oem' esm c oa^ c
00 c Y 151515
a m m
c u y E> c E A m u nQ m o oem m o D_ E mS,] 9 n W L b S D a o F '^ Q
D xm a p 0_ ° n i. sxc omov AOmo r^iA^^arfiEo^m^oa°i^°9^m-pc 0
15 m D m 0 m O m mo O- o CC
Cp - 0 00
E o f m 6 ro [R' m n Yo m • C 0
m$ e`e D C moo, m E d m m m
L m o m m o E m 5) t E E
— U C£ ul m_ m H C_ J
m „ E ®m m 9c00C E
L # Om 6 c
m Q ` OD U p
m- m00150.
-
m° a.°m L^sc a o L C<c0.15m»m3
m Ea ^om ? Q C
0Ev2 G
am
i m
m C
o O m' L 2c y 9 n O
o^ ` L^ m^^^ E> m nm O O# m y mn m
n n m ^, CDn n t
15 °E E' = m0 o m m —Iu r _
o 0 E C D m m
00 m° o O C t» o v LLL u m n 3 u^ m- 3 u m
m mL C ` } m #
a m 15 c 2. o m - 15
el vi L 'L" YI m 7 O C E E
L mV
C ib m O m S v
C m E O O r
- m Pm E L Q+m - nom D
c C O C
E p m m_ o; c m yO
E L
-C L m U N
CJ U m O
m m oo —E mcno°1mc v Y —_^ E_m 'naC.L
° m r c n ' O= m E E> m= o c D m - U `c m R m o x
3nWaE5 a. ^a°_°am
Planned Parenthood of
Central South Carolina
2719 Middleburg DriveSuite 202
Columbia, S. C. 29204256-4908
0 Copyright 1976 Planned Parenthood of Central South Carolina. Used by
permission.
271
• What channels will be used for the communication we are planningto pretest?
The answers that we get to these kinds of questions may be incomplete,and there are many other relevant questions that we could also ask. Nevertheless,the more information that we can obtain about our specific audiences, the betterwe will be able to design our communication and to pretest it later. One of the
best sources of information about our audiences comes from the results of
previous pretes ting of communication with them, from fieldworkers who knowthem, etc.
Many marketing and advertising firms keep regular files on the resultsfrom previous research that they have done. Over a period of time, it is notunusual to discover that we are being asked to design pretests for communicationswhich are very similar to ones we have already tested in the past. Sometimes thecommunications are different, but they are designed for the same audience forwhich earlier pretests have been conducted. If good records are kept of thisearlier pretesting and audience characteristics, the problems and work requiredfor good pretesting should become easier.
(b) Setting Communication Goals --Once the specific audience is known, weneed to know what kind of change is expected from our communication. We alsoneed to know how much change is expected to occur, how the change is to bemeasured , and the time period during which the change is expected to occur.The goals which our communication is designed to produce should be clearlyrelated to the overall goals of the family planning program. if we phrase ourquestions about goals in terms of these four criteria--the type, amount, measure,and time period of change expected--then we should be able to avoid much of theconfusion and variety of meaning which people have for words like "objectives,""targets," "purposes," and 'goals."
(c) Designing Communication Strategy--Once the audience and goals havebeen established, the next task is to choose a strategy based upon how communica-
tion can be used to achieve these goals. A strategy is "a particular combinationof resources, based on a communication model or models, and used within aparticular administrative framework to achieve goals.'' In general, the basiccomponents of a communication strategy are the approaches to change and themessages, channels , and sources of communication. The change approach isespecially important for pretesting since we must have some idea whether thecommunication is primarily
• For transmitting information to make our audience aware ofsomething new.
• For instructing our audience to learn something new.
16
• For persuading our audience to a ccept new values, or new means ofsatisfying their existing values.
• For creating a dialogue among members of our audience to improvetheir mutual understanding of some topic or issue.
The type of communication approach which is used to designcommunication is closely related to the kind of goals that are expected to beachieved. When we conduct our pretest, for example, we need to know whetherthe communication is designed to inform the audience about a new family planningclinic, or to instruct them how to get to the clinic and use its services; about theavailability of different contraceptives, or how to use each one; how to use thecontraceptives, or how they each work to prevent pregnancy. Is the
communication designed to persuade them to come to the clinic, to choose onecontraceptive method over another one, to accept less children for themselves,or for "people in general?" Will the audience know that the communication isdesigned to persuade them, or instruct them? Pretesting is conducted better ifwe know the specific goals and the communication approaches used to achievethose goals.
17
ADDITIONAL READINGS
There is hardly any consolidated literature on pretesting that coversboth theory and practice. The information is scarce and scattered. It isfound in books on a variety of subjects like instructional technology, education,training, advertising, marketing, mass media, and the social sciences.Unfortunately, in almost all such books what one finds is a very light treatmentof the subject of pretesting. In most cases they are the descriptions and pre-sentations of haphazardly organized principles of pretesting. However one canfind more systematic information in books and publications on marketing andadvertising. Managers are encouraged to consult them.
It is even harder to find good material on the pretesting of family plan-ning material. There is one which should be available to many managers:Pre-testing Communications: A Manual of Procedures , by Fred W. Reed (Univer-
sity of Chicago, Community and Family Study Center, 1973). We encouragemanagers to use this manual along with this module, since it presents manyuseful pretesting techniques not covered in the module.
A short bibliography of publications containing information on pretestingis given on the following pages. The managers may find them useful for moreinformation on the subject of pretesting if they can be obtained from the localresources. it is needless to emphasize, however, that all the literature onfamily planning communication and research should be collected from local aswell as international resources. This collection can provide a great deal ofinformation related to family planning communication pretesting that may befound under subheads of a variety of topics. The examples of these resourceswould be the newsletters, periodicals, and reports published by various familyplanning organizations, such as Studies in Family Planning by PopulationCouncil of New York, the World Neighbor Newsletter, the East-West Communi-cation Institute IEC Newsletter, and so forth.
273
AN EXAMPLE OF COMMUNICATION PLANNING
FOR PRETESTING
for just one of many possible audiences
Choosing the Audience : Married couples age 30-40 in rural villageswith population less than 1,000.
Analyzing the Audience: Approximately 85 percent of this audienceis employed on small farms in agricultural production. Theirincome is less than the national average, and many do not owntheir own land. Over 60 percent of the men, but only 25 percentof the women, have completed primary school. Buddhism is themost common religion; the percentage who practice it is unknown.
Almost 80 percent of both the men and women can describethe idea of family planning, including spacing, but only 25 percentof the women can explain exactly how to obtain and use oral pills.Only 10 percent of the men have heard of a vasectomy, and half ofthese confuse the operation with castration. Most of these couples
know that more children survive today than in the past and theyrecognize that many young people must leave their village to findwork elsewhere, but very few understand the relationship between
their economic situation, the population and its growth rate, andthe effect of family size on future populations. On the average,most couples still want at least four children. A recent surveyshows that only 12 percent of the couples in this age group practice
family planning, primarily for purposes of spacing.
There is a strong preference for sons. The main reasonsstated for having children are (1) economic assistance for thefamily, (2) continuity of the family name, and (3) care for theparents in old age. Only a fourth of the village wives in this agegroup are engaged in any kind of economic activity outside of thehome, mostly with helping to cultivate the fields. The economicburden on the family is the main reason for not wanting more thanfour children.
Goals for This Audience: The main purpose in communicating withthis audience is to improve their understanding of the population
problem and to increase their knowledge and methods to prevent
18
ADDITIONAL READING LIST
PERIODICALS
Achenbaum, A., R. Haley, and R. Catty. "On-air vs. In-home Testing ofCommercials." Journal of Advertising Research 7 (December 1967): 15-19.
Pros and cons for each method; approach and analysis. In-home test-ing yields more reliable results.
Brown, N. and R. Catty. "Rough vs. Finished Commercials in Telpex Tests."Journal of Advertising Research 7 (December 1967): 21-27.
Finds that rough (preliminary) forms of commercials produce similarresponses to finished forms with the advantages of early evaluation.
Chien, R. "Testing Prescription Drug Promotions." Journal of AdvertisingResearch 4 (September 1964): 9-11.
Use of random subjects in given areas, grouped and then exposed toidentical stimuli to isolate net effects of promotion.
Crass, II., L. Winters, and W. Wallace. "A Behavioral Pretest of Print Ad-vertising." Journal of Advertising Research 11 (October 1971): 11-14.
Measuring amount of ad effectiveness on behavioral change. Pretest
of two commercial product ads on sample from target audience.Measuring amount of attention apid and learning score.
Kominik, N. "Pretesting Ads Is Essential." Advertising Age 36 (October 18,1965): 92.
O'Neill, Harry W. "Pretesting Advertising with the Differential AttitudesTechnique." Journal of Marketing 27 (January 1963): 20-24.
Ostberg, Henry. "The Measurement of Non-Verbal Communication in CopyTesting." Proceedings of the Thirteenth Annual Conference of the A dver -tising Research Foundation, New York, 1967, p. 27.
Advantages of non-verbal communication in pretesting ads.
Stevens, B. and J. Axelrod. "Three Ways to Improve Ad Pretests." Journalof Advertising Research 1 (December 1961): 33-60.
Van De Dandt, Udolpho. "Pretesting with Competition." Journal of AdvertisingResearch 9 (September 1969): 17-19.
274
pregnancy. The specific goals to be achieved by the next KAP (knowl-edge, attitude, practice) survey in three years are as follows:
a. Over 50 percent of this audience will be able to explainthe negative effects that a national average size familyof four children (rather than two children) will have onthe economic condition of their country and the standardof living in that village .
b. Eighty percent of the women will be able to explainwhere to obtain oral pills and how to use them correctly.
c. Over 80 percent of the men will be able to identify thepurpose of a vasectomy, at least 50 percent will be ableto explain how it differs from castration, and 50 percentwill be able to explain where to get one, how long it takes,its immediate aftereffects.
Communication Strategy for Radio; Since almost 75 percent of thisaudience listens to radio, radio spots, radio dramas, and radiogroup discussions will be used in an effort to instruct women abouthow to use oral pills and where to obtain them in their vicinity, and
to inform men about the availability of vasectomies, and to instructthem on where to obtain them and how they work to prevent pregnancy.Dramas will be designed to introduce discussions of vasectomies innon-threatening, sometimes humorous ways, but within situationsand settings which will be similar to the real-life situ^ tions of theintended audience. Radio discussions will take place betweenagricultural experts who work in villages, village farmers, anduniversity economists. Farmers will be encouraged to describetheir current situations and problems, and their own expectations forthe future. The economists will attempt to describe the nationaleconomic picture in terms that the farmer will understand. Thenthey will describe different economic scenarios, or projections,which will occur in ten years according to changes in the populationgrowth rate, changes in agricultural and industrial productivity,and so forth. The local agricultural experts will be responsible forinsuring that both the farmers and the economists explain themselvesin terms that can be understood by each other, then they will try toexplain how changes in the national economic situation usually
effect a typical village or farm household. Each group--farmers,
experts, and economists--will be asked to summarize at the end
of each program what they have learned from listening to the other
19
Outline of print ad pretest method using comparative and competitive de-designs with constant sum scales and Guttman scales.
Wheatley, John. "Assessing TV Pretest Audiences." Journal of AdvertisingResearch 11 (February 1971): 21-25.
Effectiveness of commercial can be evaluated without special consid-eration as to whether respondents are users or non-users of product.Finds potentially significant audience attributes affecting response insex, education, and age.
BOOKS
Ball, Samuel and Gerry A. Bogatz. The First Year of Sesame Street: An Eval-uation. New York: Teachers College Press, 1970.
Dunn, S. Watson. International Handbook of Advertising. New York: McGraw-Hill Co. , 1964.
Chapter 22: "Testing the Effectiveness of Advertising in Foreign Mar-kets. "
Lucas, D. B. and S. H. Britt. Advertising Psychology and Research. New York:McGraw-Hill Co., 1950.
Chapter 15: "Considerations of Pretesting 'Opinion Ratings'."
Reed, Fred W. Pre-testing Communications: A Manual of Procedures .Chicago: Communication Laboratory, Community and Family Study Center,
University of Chicago, 1974.
Robinson, E. Communication and Public Relations . Ohio: Morrill Books Co.,1966.
Chapter 10: "Content Analysis--Content Response Code Used in Pre-testing Positive/Negative Reactions of a Sample to a Communication."
Schramm, Wilbur, ed. Quality in Instructional Television . Honolulu: Univer-sity Press of Hawaii, 1973.
Trenaman, J. H. Communication and Comprehension . London: Longmans,Green, and Co., 1967.
Investigation of effective communication of educative material and anassessment of factors making for such communication--especially withreference to broadcasting.
Udry, J. Richard. The Media and Family Planning. Cambridge, Mass.:
275
groups. No direct effort will be made to focus the discussion onfamily planning until the second half of the twenty-week series.
Pretesting: As these radio programs are produced, they must eachbe pretested to establish their contribution to the stated goalsbefore they are used on a wide scale in each region.
20
Ballinger Publishing Co.; Carolina Population Center, University of NorthCarolina at Chapel Hill, 1974.
Wolfe, Charles H. Modern Radio Advertising . New York: Printer's Ink Pub-lishing Company, 1949.
Chapter 11: "How to Pretest Your Programs"; Chapter 39: "How toPretest Commercials."
REPORTS AND PAPERS
Research, Development of Materials and Training Programs for 1973-1974.Quezon City, Philippines: Institute of Mass Communication, University ofthe Philippines, n. d.
Pre-testing and Field Evaluation of Communication Methods . Bangkok: Researchand Evaluation Unit, Development Support Communication Service, n. d.
RESEARCH METHODS
Beckstrom, C. H. and G. D. Hursh. Survey Research . Evanston, Illinois:Northwestern University Press, 1963.
Seltiz, C., M. Jahoda, M. Deutsch, and S. W. Cook. Research Methods in SocialRelations . New York: Holt, Rinehart & Winston, 1959.
Hursh, G. D. Survey Research Methods in Developing Countries. Delhi;Longmans--Orient Press, 1974.
276
AN EXERCISE ON SETTING COMMUNICATION GOALS
Briefly describe a specific communication (or set of communi-cations that you have already used in your own program, and then in-dicate the audience for which it was intended. If you know of certaincommunications that are now being planned or designed for your pro-gram, it would be even better to use one of them for this exercise.After describing it, state in general what the communication was (is)expected to accomplish (its general purpose).
Description:
Intended Audience:
General Purpose:
Now list one or more speci fic goals that the communication(s)was designed to achieve according to the three main criteria for goalsetting presented in this module.
21
THE EAST-WEST CENTER is a national educational institution established in Hawaii bythe U.S. Congress in 1960 to "promote better relations and understanding between theUnited States and the nations of Asia and the Pacific through cooperative study, trainingand research."
Each year the East-West Center brings together more than 1,500 men and women from themany nations and cultures of these regions. They work and study together while exchangingideas and experiences in cooperative programs seeking solutions to important problems ofmutual concern to East and West. For each participant from the United States in Centerprograms, two participants are sought from the more than 60 countries and territories inAsia and the Pacific area.
Five institutes with international, interdisciplinary academic and professional staffs conductthe East-West Center's problem-oriented programs. East-West areas on which Center pro-grams are focused include communication across national barriers, culture and languagelearning, food systems, population dynamics, and technological adaptation in developmentalprocesses aimed at improving the quality of life. Each year the Center awards a limitednumber of Open Grants for graduate degree education and innovative research by SeniorFellows in areas not encompassed by institute programs.
The Center is directed by an international Board of Governors of a public, non-profit edu-cational corporation—known as the "Center for Cultural and Technical Interchange BetweenEast and West, Inc."—created by the Hawaii State Legislature in 1975. The United StatesCongress provides basic funding for Center programs and for the variety of scholarships,fellowships, internships and other awards. Because of the cooperative nature of Center pro-grams, financial support and cost-sharing arrangements are also provided by Asian andPacific governments, regional agencies, private enterprise and foundations. The Center issituated on land adjacent to and provided by the University of Hawaii, which conductsclasses and grants degrees for degree-seeking East-West Center students who also areinvolved in the Center's problem-oriented programs.
THE EAST-WEST COMMUNICATION INSTITUTE concentrates on the use of communica-tion in economic and social development and in the sharing of knowledge across culturalbarriers. The Institute awards scholarships for graduate study in communication and relateddisciplines, primarily at the University of Hawaii; conducts a variety of professional develop-ment projects for communication workers in specialized fields of economic and social devel-opment; invites Fellows and visiting scholars to the Center for study and research in commu-nication and to help design projects; offers Jefferson Fellowships for Asian, Pacific, andU.S. journalists for a semester at the Center and the University of Hawaii; conducts andassists in designing and carrying out research; arranges conferences and seminars relating tosignificant topics in communication; conducts a world-wide Inventory-Analysis of support,services and country program needs in communication programs; assembles relevant com-munication materials with emphasis on Asian and Pacific material and makes these availablefor students, scholars, and practitioners at the Center and elsewhere; and publishes papers,reports, newsletters, and other materials emanating from the above activities.
GOAL NO. 1
Type and Amount Type of Time Period
of Change Expected Measurement for Change
GOAL NO. 2
GOAL NO. 3
22
STEP II: DESIGNING THE COMMUNICATION
Design is one of the first steps toward implementing the communicationthat has been planned. Since it is impossible to predict exactly which kind ofmessages will have the most effect on the intended audiences, designing is stilla very creative and artistic process. It is often useful to design two or morealternate versions of a given message, varying those aspects or elements aboutwhose effect we are most uncertain. Pretesting with the intended audience canthen be done to determine which versions contribute most to our planned goals.
STEP III: DESIGNING THE METHODS FOR PRETESTING
As you will see in Unit III, there are a number of methods and techniquesthat may be used for pretesting. To decide what particular method suits yourparticular situation requires again a careful consideration of several factors, suchas the goals, the type of communication, the resources available, the channelsused, the stage of completion, the time factor, the validity and reliability of themethod, the availability of a representative sample of respondents, and so forth.
STEP IV: CONDUCTING THE PRETEST
Once we have decided upon the methods to be used for pretesting, we areready to plan and conduct the pretest of the communication that has been developedso far. You should assign duties to various individuals and provide them withenough information about the objectives of the pretest and the procedures to be
followed. We will discuss this in detail in the unit on the techniques of pretesting,because each technique requires different kinds of procedures.
STEP V: ANALYZING AND IN TERPRETING THE RESULTS
Although this is a specialized task, it can be made more simple bychoosing research techniques and collecting data in such a way that they will beeasy to analyze. However, in most cases, analysis of pretest results can be donewithout advanced statistical training or use of a computer, since the sample sizeis usually small and the data is uncomplicated. Most of the data can be analyzedby hand tabulation, and in other cases, simple calculating machines will be
sufficient. Interpreting the results of our analysis is very important. It requiresskill and clear knowledge about the problem and the information that is collectedfrom the pretesting. Any personal bias or misinterpretation of the results maylead to conclusions which do more harm than good to the development of thecommunication.
23
STEP VI: REPORTING AND IMPLEMENTING THE RESULTS
A clear report that answers the main questions of the pretest is importantfor many reasons. The report should be made so that the production and creativestaff can understand and use the information that has been obtained through pre-testing to improve their work. Frequently, not enough attention is given to thisaspect of the pretesting. In the final report the recommendations should beclearly and precisely written. They should not only be written in the languagethat is understood and used by the production staff, but the recommendationsshould be presented so that they make it easy for the production and creativestaff to use them. Using unfamiliar research terminology and giving themunnecessary statistics may discourage some production staff from using thereport's recommendations.
One useful approach is to reverse the usual order when the report of thepretest results are given:
1. Introduction to the problem and brief overview of the report
2. Conclusion and recommendations
3. Body of report (including research methods, tables, and detailed
results)4. Supporting material and appendices
This approach gives quick and easy access to the implications of the
pretest for the production staff, followed by more detailed supportive evidence.
This order of presentation should be used for the oral presentation of theresults as well as the written report. It is always a good idea to hold a meeting
in which the pretesting team and production staff are present to discuss theresults. The limitations and problems with conducting the pretest, and thedifficulty in making the recommended changes should be openly and thoroughlydiscussed by everyone involved. This is a good way to overcome some of theobstacles and sources of resistance to using the results and recommendationsfrom pretesting. The production staff may offer their own interpretation of theresults, and arrive at more creative recommendations for improving theircommunications. The results of a pretest should be considered as preliminaryuntil all those involved in their use have had an opportunity to make their owncontributions.
Naturally, there are many different kinds of recommendations which mayresult from pretesting communication. In general, the recommendations will beone of the following;
(a) Accept It
The pretesting results may show that the communication contributes
effectively to the goals which have been planned for its intended audience. In
24
this case we can go ahead and use the communication and proceed with wide-scale production and distribution as planned.
(b) Change ItPretesting results may suggest that changes are required to make the
material more effective. We may want to test the communication again after ithas been revised. In many cases, however, it is not necessary to pretest,especially if the suggestions made by the pretesting team are precise and explicitenough to the production staff, if the changes are minor, and if time and budgetdo not permit. However, if substantial changes are made, a second pretest shouldbe conducted.
(c) Use It SelectivelyWe may find out that the communication we have pretested is only effective
in certain situations, and only for certain selective audiences. This is valuableinformation since the communication can still be used, but only for a morelimited audience than had been planned. A poster for a general audience, forexample, may turn out to be effective only for urban women when they are visiting
health clinics. In this case, a smaller number could be printed and distributedonly to urban health clinics with specific instructions about how they should be used.
(d) Reject ItFinally, our pretesting may suggest that the communication does not
adequately contribute to our stated goals, or that it has different kinds of effectsthat are not intended and that may even harm our program. There is not muchdifference between rejecting a communication and recommending such large changesthat it is no longer the same communication. If the basic idea for a communicationis inappropriate for the selected audience, then the whole communication shouldbe rejected. If it is discovered that a radio spot, for example, will simply notreach the right audience or allow for enough air time to achieve its goals, thena different channel or different approach would have to be tried.
To reject something into which a lot of work and time has already beeninvested is difficult to do. It is usually tempting to make a few more changes andtest it one more time. The cost of trying to improve a communication should becompared with the advantages and costs of starting over again with something new.We can usually avoid getting stuck with expensive productions that cannot bemade to work effectively by pretesting at various stages in its development. By
getting feedback early in the production, we can avoid making big mistakes inour work.
Pretesting is part of a much more general activity: developing effectivecommunication. Rejecting communication that does not live up to our expectationsis a positive outcome if it has taught us something important about how tocommunicate better with our intended audiences. Rejecting something that has
25
cost us much time and effort is well worth the investment if it helps make our
future work more creative and more effective than what we have done in the past.This is why pretesting is so much more than "evaluation" of someone's work.Pretesting should be considered as part of that work, as part of the overall processfor developing artistic and effective communication. Those who pretest have theresponsibility to insure the development of effective communication processeseven when some of our work must be changed or rejected.
All six steps in the pretesting process are important for improving thequality of our communications. This model is not meant to suggest that our workalways goes from step one to step six, nor that we would only be working at onestep at a time. We may have a whole set of communications which are each indifferent stages of design and development. It is possible, therefore, to be doingwork at all six stages, but for several different communications. Somecommunications may go through several cycles of these six steps before they arefinished and accepted for wide-scale use. On the other hand, if someone bringsyou a finished product to be tested then you will mainly be responsible for thelast steps in the cycle. Some steps may not be done very thoroughly, as whencommunications are produced without much analysis of the audience or planning.This will make the other steps much more difficult to do, of course. Having anidea of the whole process should make it easier to make up for steps that havebeen skipped, or not done thoroughly enough to conduct the pretest properly andto develop the communication.
DECIDING WHAT NEEDS TO BE PRETESTED
Deciding what communication needs to be pretested, at what stage, bywhom, and for what effects is very complicated simply because of the greatdiversity and wide range of communication media which are available. Somecommunication staffs may have more experience with some media than others,
and more knowledge of some audience's reactions than others. There is no generalrule which can be followed by everyone, but the basic principle which underliespretesting is applicable: If you are not sure how a certain communication willeffect a specific audience, then find out before going any further . The more priorinformation you have--the more certain you are about your communication--thenthe less need there is to pretest. The more pretesting experience a communicationstaff has, of course, the more confident they will be about new communications.Very new ideas, or communication about especially sensitive or easily misunder-stood topics, should be tested as they are developed.
It should help to maintain a broad idea of what pretesting is: finding outwhat happens when we share information with someone else. Naturally, thisbecomes more complicated when we consider that communication can be testedat almost any stage of development--from its initial idea all the way through to the
26
finished product. Furthermore, we can test the communication as a whole , or bybreaking it up into several of its parts (e. g., by testing reactions to a whole radiospot at once, or by testing its music, the script, the announcer's voice, or keyphrases separately). And finally, we may test many different kinds of effectsthat communication can have, from simple attention and interest through under-standing, agreement, and other changes in behavior.
PRETESTING AT VARIOUS STAGES OF DEVELOPMENT
When someone creates a new idea for communicating about population,family planning, or other development problems, he often tests out that ideabefore he actually produces a poster, a radio spot, a pamphlet, and so forth. InKorea, for example, someone came up with the idea for a "no pregnancy year" in1974. The idea seemed somewhat strange at first, so the idea was well testedbefore anyone went ahead and produced any communication based on it.
First, members of the family planning staff discussed the idea and whatthe implications would be for a campaign based on such a theme. Then variousgovernment officials and community leaders were contacted to get their reaction.
What looked like a negative theme on the surface turned out to have manyadvantages. It aroused interest and stimulated discussion among people. It ledeasily to the concept of spacing births. It had special relevance to couples whowere not planning to do anything to avoid pregnancy, and stimulated them toconsider postponing it for a year or so. And finally, it led nicely into discussionsabout how to have a "no pregnancy year. " This kind of test can yield many otherideas for specific communications derived from the general idea.
Such an idea can also be tested with women who are visiting clinics, withfieldworkers who come in for their regular meetings, and so forth, long before
any specific communication is produced for it. Then it can be tested withpartially completed posters, magazine layouts, radio spots, etc. , or on alter-native versions of communication for the same medium. And finally, when thefinal version is completed for a certain medium it may be tested again, perhapson a regional basis, before being widely distributed.
Sometimes an idea cannot be adequately tested until at least a partiallycompleted version has been produced. To test an idea may require the firstdraft of a script or the preliminary sketches or photos for a poster. Pretestingat the partially completed stage of development can help us to improve an idea forfurther development before very much time and money is spent upon production.It should always be kept in mind, however, that the partially completed commun-ication is not the final version, and that the effects of the final polished productmay be quite different from its roughly-cut forerunner.
Pretesting alternative versions of the same communication has several
27
advantages. It usually makes it easier for our pretest respondents to give usinformation, and ;t can save time and money on our field work. If respondentscan look at four different versions of an icjea for a poster, for example, they
are able to give us comparative information. For example, "poster A is themost pleasing to look at, followed by poster C, then B." Or, "poster C clearlygives me the idea that spacing is better for the mother's health and the children'sdevelopment, but this idea is not as obvious in the other three. " Alternativeversions allow the respondent to say how, and to what degree, one version seemsbetter than another. Furthermore, since it is often too costly to test onlyone version of an idea, then return again with another, or with a new revision,it is less expensive to wait until several versions are ready in more than onemedium, such as poster, pamphlets, and radio spots. Your research team canthen be sent out to the intended audiences.
If a particular communication has been pretested throughout its variousstages of development, then by the time it has reached its finished form , weshould have a good idea how it will affect the audience. Even then, a finishedversion can still be tested on a pilot basis (less than full distribution) to seehow it affects different kinds of audiences who have previously not been tested.And finally, it can be tested to learn better ways to use the communication withdifferent kinds of audiences.
A family planning pamphlet, for example, may be developed and pretestedat various stages for mothers who are visiting public health clinics. A quick testdoor-to-door, in a market area, or in schools, may show that the same pamphletis also very effective, and even requested by younger women, newlyweds, orinterested husbands. Rather than spending the money to develop a differentpamphlet for each of these audiences, it may be possible to save money by pro-ducing a greater quantity of the clinic pamphlet with only a special insert or anew cover.
If a finished version of a communication has not been pretested at earlierstages of development, then everything which has been said thus far is applicable;Find out as much as possible about how it works with which audiences beforeunnecessary expenses are made to produce it in large quantities. If it costsalmost as much to produce a small quantity for pretesting as for the fibal dis-tribution, then it is still worthwhile to pretest it on a pilot basis with a limitedaudience to insure that its effects are not harmful.
On the following page are examples of family planning posters used inIran. The first set (numbered 1 to 4) shows one poster that was pretested atvarious stages of development. A complete description of the pretesting pro-cess used in Iran may be found in Appendix Example 3. The second set ofposters was chosen for its aesthetic qualities.
28
IIIi to
POSTER DEVELOPMENT IN IRAN
•
H N ♦ N•
r • w 4
fr /
1 .. ibflelAw»Wnw..,:
2 4
Other Finished Posters from Iran Chosen for Aesthetic Qualities
FXFRCTSF
1. Describe the two communication products you have most recentlydeveloped or worked with. Indicate type (e. g. , pamphlet), distri-bution (e.g., all civil servants), and any other important charac-teristics.
Product 1:
Product 2:
2. Were these products pretested? If so, indicate the stage andmethod of pretesting you used and the go to question 3. If not,
go to question 4.
Product 1:
Product 2:
30
3. How were the results of pretesting used?
Product 1:
Product 2:
4. Did you encounter any difficulties In using the materials? Didthey achieve their planned goals? How do you know?
Product 1:
Product 2:
31
PRETESTING THE WHOLE OR ITS PARTS?
Some types of communication are difficult to pretest until the whole prod-uct is more or Iess finished. It is difficult to test the parts of a film, for example,until a more or less completely edited version is produced. After pretesting, itmay be necessary to do some additional editing of the film. Individual still framesfrom some of the key footage for the film can be tested for their impact on mem-bers of the intended audience, and various types of music could be tested beforethe filming is completed, but the results of such testing are difficult to interpretbecause reactions are obtained out of the context_ in which it is to be shownlater . The full context can make a big difference in how information isinterpreted.
With this limitation in mind, however, some pretesting of the parts of aproduct is feasible and can save a lot of time later. If, for example, you un-knowingly select music for a film which is too light or too amusing for certaingroups in your intended audience, it is much better to eliminate it before all thework has been done to integrate it into the film. It is much easier to transport,operate, and administer audio tape recordings of several music selections thanto move the equipment for a complete film. The script and audio portions of avideotape may also be tested separately from the video portion. Then the audioand video can be integrated and tested as a whole.
When deciding whether or not to test the incomplete parts of a product,the time and cost have to be considered in relation to what is already known aboutthe effects of each part. If the producer feels confident about how his audience
will react to certain kinds of music, colors, dramatic sequences, photographs,key words or themes, and so forth, then it would probably be more advisable to
wait until the whole product (or several completed versions of the product) isfinished before testing it with the intended audience.
The cost of this kind of pretesting can be reduced over time if it is donewith careful forethought. Rather than spending the time and money to test threekinds of music for use of one radio program, it might be less expensive over thelong run to test ten or more possible types of music at one time and then save theresults of this test for future reference and for use in other products. This sameprocedure would also be feasible for photographs, preliminary drawings, colorpreferences, and so forth. Some I.E. C. production staffs have accumulated a lotof experience with their audiences in this manner. But where there is frequentturnover of production staff (and hence, many new, inexperienced productionstaff), it is useful to maintain a record of previous pretest results. In this
manner, the experience gained over the years can be saved and used by futurestaff.
Even when complete versions of a communication are ready for pretesting
32
the question of what to test is still relevant. We still have the choice of pretestingthe overall effects of the whole product on members of the audience, or theseperate effects of various parts or aspects of it. The overall impact of thecommunication as a whole is obviously the most important. After the audiencehas seen a poster, listened to a radio program, or watched a film, how well doits members understand what is intended? Do they agree with it? Do theybelieve it? Are they more likely to talk about it with their spouse, or to seekadditional information, or go to a clinic? The overall effects of communicationare the most important, but sometimes various parts of our communication canmake a significant difference in the overall effects.
We should not forget the main objective of pretesting: to improve theeffectiveness of our communication . By changing some of the minor elementsor components of our products we may be able to obtain the attention and interestof more people. By changing a few difficult words or phrases, we might make itpossible for many more people to understand the whole thing better. Sometimesthese minor changes may have dramatic effects on the overall impact of the wholeproduct. Sometimes such changes contribute very little to the overall effect. Forexample, if you are concerned about the way an artist dressed certain figures ina poster, then, when you test the poster, ask the members of the audiencespecific questions about how the people in the poster are dressed.
In general, some very serious mistakes can be avoided this way. Forexample, the poster from Kenya, which we looked at earlier was improved by
merely changing the kind of shoes that the mother was shown wearing. Some
members of the pretest audience were greatly distracted because no woman intheir area ever wore such exotic shoes. This required only a very simple changein the poster's design. Was it worth the cost and trouble? The answer depends
upon how many people are distracted and whether you want them to remember andtalk to others about the main family planning message or the funny shoes.
THE FUNNEL APPROACH
Asking questions tbout specific parts of a communication, at the beginningof a pretest, can sometimes distract the respondents or bias their reaction to thecommunication as a whole. The best approach is to obtain the audience's generalreaction to the whole communication first, then find out the effect of specific partsof the product. In interviewing this is sometimes referred to as the funnel_ approach ,because you begin with a few general questions about the audience's reactions to thewhole product, then gradually narrow down to more specific questions. Do youlike it? What does it mean to you? Do you agree with it? Then the interviewerprobes for reaction to specific parts of the product: What do you like best aboutit, and why? What is the size of a "small family" to you? Which parts do youagree with most? Disagree with most? Why? Does the woman in the posterlook like someone from this village? How old does she look? What makes her
33
look young or old? This funneling approach allows the interviewer to get therespondent's general reaction before he is influenced by the interviewer'squestions about specific elements of the communication. It keeps the interviewer'sown bias at a minimum, because it first lets the respondent himself say whichparts of the message are responsible for his reaction. After the respondent hasbeen given the opportunity to respond freely to the communication, then the
interviewer can ask more specific questions about elements the respondent hasnot mentioned.
The woman's shoes in the poster is a good example. The interviewerwould want to find out the respondent's overall reaction to the poster first, thenher reaction to specific elements which he thinks are important. This is the onlyway to find out if the respondent will freely or voluntarily mention that the shoesare never worn by anyone in her village. Once she has been given the opportunityto mention the shoes herself, then the interviewer can specifically probe for herreaction to the shoes: Is the woman dressed like most women in your village?How is she different? What kind of women wear shoes like hers? Did you noticeher shoes when you first looked at the poster? What would most people in yourvillage think when they see these shoes? And if necessary, how would changethe shoes so they look more familiar to people "around here"?
THE VARIOUS EFFECTS OF COMMUNICATION
The number and type of effects that communication is expected to have on
the audience may be quite extensive, or limited to just a few. A nonverbal symbollike the red triangle of India, for example, might be tested in a number ofdifferent situations just to see (1) if it can attract attention, (2) if the color redor triangle have any unintended or negative connotations, (3) if those who see it
are reminded of family planning. These may be the only objectives of the symbol,and therefore the only effects on the audience that need to be tested. By contrast,a fifteen minute radio drama may be designed to accomplish these same threeobjectives, plus many other objectives: (1) to interest men in a vasectomy as anew contraceptive method, (2) to teach the advantages and ease of vasectomies,(3) to foster a sense of responsibility in men for contraception, (4) to persuademen that vasectomy has been accepted by many men just like them, (5) topersuade men over 30 that vasectomy is relevant for them, (6) to get some mento visit a clinic for more information or to obtain the operation, and so forth.
Naturally the length and amount of time spent on pretesting will vary,depending upon the planned goals of the communication and its level of complexity.In order to interpret the results of the pretest it is useful to have some knowledgeabout the pretest respondents' prior level of achievement of the desired goals. Inthe previous example, how many men over 30 in our pretest audience already canlist the most important advantages of a vasectomy? Once this initial level is known
34
then the effects of the communication can be established: After one exposure tothe radio drama, twice as many respondents were able to name at least twoadvantages of vasectomy. Or, after one exposure the proportion of respondentswho could name two advantages increased from 25 percent to 50 percent.
How much change is reasonable to expect? Stating the objectives inconcrete terms requires some previous knowledge of how much effect a similartype of communication has had with a certain audience after one exposure. Anearlier pretest may have shown that a similar radio drama had successfullyincreased the number of women between 30 and 45 who could remember twoadvantages of the I.U.D. from 25 percent to 60 percent. So, setting the targetfor your radio drama on vasectomies at 50 percent may be a reasonable estimateof what can be achieved. To be realistic, goals should be derived from prior
experience.
By setting a minimum standard of achievement, it becomes easier to
interpret the results of the pretest. If a taped version of the radio drama isplayed for 50-100 men over 30 (say, 10 small work groups of 5-10 men in afactory), and only 20 percent are able to recall two advantages, then work willhave to be done on the program to place more emphasis on the advantages in thetape, to make the advantages clearer, and to make them easier to understand andto recall. If your pretest shows that over 60 percent would recall two or moreadvantages then you may want to turn your attention to improving other aspectsof the product. If you are close, but under your expected objective (say, 45 per-cent) then you at least have some objective standard to use when deciding whetheror not to change the product. If many of the other objectives of the product areachieved well, than you may decide that 45 percent recall is good enough to go
ahead and use the product.
Another reason to maintain records of all pretesting results is so that theproduction and research staff can develop more realistic estimates of how mucheffect various kinds of message-channel combinations can have. This will
eliminate a lot of unnecessary disagreement among the staff and focus attention
on product development.
SPECIFIC EFFECTS
The following list of effects may be useful for a number of different typesof communication which are developed for family planning. The specific goals ofa communication product should be determined by the overall communicationstrategy that is planned and the various communication approaches to human changethat are used to design the product (e. g., information transfer, instruction,
persuasion, or dialogue).
Attention Does the message compete well for the attention of the intended
35
audience? Is it capable of maintaining the audience'sattention long enough for effective communication
to occur?
Interest and This is closely related to attention and to maintaining
Enjoyment attention. Is the message tedious or stimulating? Isthere enough immediate reward during communication tokeep the audience interested? Do they enjoy it? Howentertaining is it? Do they like it? Would they like tosee (or hear) it again?
Understanding The message may be entertaining, but can the audienceunderstand the main points you are trying to get them tolearn? Are any parts of it misinterpreted? Is it clear,logical, and at the right pace and language level for theaudience to comprehend it?
Believing Once they understand it well enough, do they believe whatit says? How credible is the source? Is the sourcecompetent and trustworthy enough to base my own decisionson what he says? Is the information consistent with theaudience's own experience? Inconsistent?
Agreement If the audience believes what is said, do they personally
accept or reject it ? Do they agree or disagree? Does itapply to them? To their life situation? Is the recom-mended course of action for them personally, or onlyfor other people who are not like them ?
Recall After exposure to the message, is the audience able torecall what was in it? Is the content presented so that itis easy to remember later? Are important things forgot-ten while minor things are recalled? Do trivial thingsinterfere with remembering the important points?
Behavior Does the audience do anything after exposure to the message?If applicable, do they do what the communication recom-mends? Do they ask for further information? Do they
intend to mention it to their spouse, or to friends? Do theyactually do this later? Do they plan to take any action? Dothey actually go to the clinic mentioned in the message?
36
ONE LAST QUESTION: WHY?
Since the purpose of pretesting is to improve our communication, it isnever enough just to find out the degree of achievement of these or other kindsof effects. It is necessary to find out wh members of the audience react the waythey do. If they do not understand the message, find out what parts they do under-stand and which they do not. Then ask them y some parts are difficult to under-stand, and how the basic idea could be made clearer to them. Try to achieve aclearer message during the pretest, and then note down what made it easier tounder stand.
Many pretest respondents are very willing to give you their opinion why aproduct is boring, difficult to understand, or unacceptable to them or to theircommunity. If is is embarassing for them to say which parts they cannot under-stand, ask them indirectly to indicate which parts would be the most difficult forthem to explain to a neighbor. The key questions for pretesting are y and howcould it be improved. The researcher needs to be able to tell the I.E. C. produc-tion staff y certain respondents reacted a certain way and which parts of theproduct may be responsible. This requires that the interviewers who conductthe pretest be very skilled at achieving rapport with pretest respondents and atasking questions to probe for more information to their initial reactions.
37
EXERCISE: ESTIMATING THE EFFECTS OF COMMUNICATION
Refer to the list of "effects" for pretesting. Choose a commu-nication product that you have worked with in the past year (or areworking on now) and describe the product in the space provided below.Then, describe the audience for the product. Finally, for each of theseven effects noted in the module, write a pretesting question relatedto the product and the audience.
Example
Type of Communication: vasectomy Pamphlet
Intended Audience: military forces
Effect Attention: From a sample, what percentage selected
Questions:the vasectomy pamphlet to read from among five pam-
phlets displayed on a shelf in their living quarters.
Recall : From a sample of soldiers, what percentage
could recall the time required to have a vasectomy?
The time for recuperation and immediate effects?
Behavior : What age group of soldiers and officers
asked for more information about vasectomies or
where t o g et one? How many or what perce n tage asked?
Your Example
Type of Communication:
Intended Audience:
38
Effect Attention:
Questions
Interest:
Understanding:
Believing:
Agreement:
Recall:
Behavior:
39
SELECTING A PLACE IGOR PRETESTING
The kind of setting that is chosen to conduct the pretest in is often a greatsource of controversy. Should you invite respondents into your studio to listento a new radio program or watch a new television spot? Is it better to have ruralresidents come into a regional office to see a new slide-tape show, or should yougo out to their village and show it? Should you interview people in their home oroutside? Under natural conditions or artificial conditions? Many people rejectpretesting of any kind simply because the conditions under which the test is con-ducted are so unnatural, so unlike the "real" conditions under which they willeventually experience the communication.
Testing in a central place, like a laboratory, studio, or office, has theadvantages of speed, lower cost, and a higher degree of control over other fac-tors that may influence the audience's reaction. A billboard on a busy street hasnot only several billboards around it; there are usually many other activities go-ing on in the street: traffic, people walking by, etc. , that may distract the respon-dent or influence his reaction. The research team can be more sure that eachrespondent actually saw, read, or heard the communication being tested if theydo it in their offices. They can control the way the audience is exposed to theproduct. The disadvantages should be obvious. Studio or laboratory settings arenot at all like the setting in which the population will be exposed to the product.In the real setting, the communication may have to compete with many distractions.
It may not be necessary to control completely the way the audience is ex-posed to the communication, If the normal conditions surrounding an outdoormeeting in a rural village interfere with the comprehension or attention to a new
slide-tape show, you certainly would like to find out during the pretest. Testingin the field, or in the respondent's own home, does not completely overcome thedisadvantages of central testing in a laboratory. The test situation itself is notnormal. The presence of an interviewer, or research team, may affect reactionsto a product. Testing a film at a regular village meeting where films are oftenshown approaches a normal situation, however. Observing the passerby as heor she reads a new poster on a corner wall where posters are usually found iseven closer to the normal course of events, especially if the observer is notnoticed by the passerby until he or she is interviewed afterwards.
Regardless of how close you approach normal conditions, however, a pre-test always represents a less than perfect means of inferring something aboutthe respondent's reaction to your new product. There are very few direct waysof measuring responses to communication, so the results are always a mixtureof his or her reactions to your communication and the test situation. In general,better information can be obtained in situations that are as typical of the normalsetting as posible, and where the methods of testing are familiar to the respon-dents. Nevertheless, if control over the pretest situation, time, and cost areimportant factors for the kind of pretesting that must be done, then a less thanperfect setting can still provide useful information about the communication.
40
SUM24IARY
In an ongoing family planning program or a large-scalecommunications campaign it is often impossible to learn the effectthat each separate communication is having on our intended audience.Feedback often arrives too late to make any changes in the designof the communication. Pretesting each communication as it is beingdeveloped and before it is used on a wide scale in combination withmany other communications is a practical way to overcome thisproblem. Pretesting is a means to find out the kinds of effectseach communication will have on a representative sample of ourintended audience.
Meanings are in people, not in messages, so the only way tobe sure how effective our communication will be is to go to thefield and see how specific audiences react to them.
Evaluation tells us if our work is good; evaluative research
tells us what it is good for: what and how much it contributes tospecifi c goals that have been set . Research is a careful and system-atic investigation about what happens in the world when, for example,someone sees or hears communication about family planning.
Pretesting is that part of evaluative research which takesplace before communications are used on a large scale in combina-tion with many other kinds of communication. It is part of theprocess of designing and developing more effective communication,not a separate research activity by itself.
The steps of communication planning which are most importantfor pretesting are the selection and analysis of audiences, goalsetting, and designing communication strategy. Some of the initialproblems of pretesting will be resolved if the communications havebeen thoroughly planned and clear, specific goals have been set. Tomake goals as clear as possible, we should indicate the kind and
amount of change that is expected, how that change can be measured,and the time period within which the change is expected to occur.
Communication can be pretested at various stages ofdevelopment and completion, in parts or as a whole, with alternative
versions, and separately or mixed with other kinds of communication.
41
Pretesting in natural settings like a health clinic, a factory, avillage meeting hall, or a respondent's home, reduces the amount ofbias or influence on the respondent's reaction due to the pretestingitself, but with some sacrifice to the amount of control over theconditions and manner in which the respondents are exposed to thecommunications. More control over the resting situation canusually be gained by pretesting in a centrally located studio oroffice.
The purpose of pretesting is to develop more effectivecommunications. The results may indicate that a particularcommunication is effective enough for widespread use, that furtherdevelopment and improvement are required, or that it can be usedeffectively only in more limited, selective situations. Any ofthese outcomes will give us valuable information about our audiencesand the effects of various techniques and processes of communicationwhich may be used in the future when it is necessary to design newkinds of communication.
42
REVIEW QUESTIONS FOR UNIT I
I. Which important obstacle to designing effective communication can pretesting
help us to overcome?
2. What are two of the main reasons for pretesting communication?
a.
b.
3. What are the main differences between evaluation, evaluative research, andpretesting?
a. Evaluation:
b. Evaluative Research:
c. Pretesting:
4. What are the six basic steps for developing and pretesting more effectivecommunication?
a.
b.
C.
d.
43
e.
f.
5. The first phase of communication planning is choosing and analyzing theaudience. What are the second and third phases of communication planning?
6. What are the four basic criteria for writing communication goals which makethem useful for pretesting and development of effective communication?
a.
b.
C.
d.
7. When reporting the results of pretesting to others, what is the recommended
order of presentation?
8. What is the main advantage of pretesting the reactions to separate parts of a
communication, such as the color, size, drawings or photos, and verbalmessages of a poster?
44
9. What are the disadvantages of pretesting reactions to various parts of acommunication as opposed to testing the effects of the communication as
a whole?
10. Describe how the technique called "funneling" achieves the advantages oftesting the parts and the communication as a whole?
11. List the seven specific effects of communication which may be pretested forvarious lands of communications. (if you cannot recall all of them review
pages 35-36.)
a. d. f.
b. e. g.
C.
12. What are the main advantages of conducting pretests in the studio where theyare produced (or testing laboratory), or in a regional field office?
45
13. What are the main advantages of conducting pretesting in family planningclinics, in village settings, or in the homes of members of the intendedaudience?
14. Describe one way of pretesting several alternative designs for a poster without
the respondents from the intended audience being aware that they are
participating in a test.
46
ANSWERS TO THE REVIEW QUESTIONS FOR UNIT I
1. The most difficult obstacle to designing effective communication is the lackof knowledge about our intended audience, and how our communication mightaffect them. Pretesting is one way to overcome this problem.
2. a. to develop more effective communication before it is used on a widescale
b. to avoid spending time and money on communication which contributeslittle to our intended goals, or which may even harm our overallprogram of communication
3. a. evaluation is concerned with the basic question: Is it any good? It isoften a subjective, qualitative judgment.
b. evaluative research is concerned with the more specific question: Whatis it good for? What is the impact, or the effects of our communication?How much does it contribute to our communication goals? As research,it consists of careful and systematic investigation of the effects ofcommunication on its intended audience.
c. pretesting is that part of evaluative research which measures the impactor effectiveness of s ecific communication products, such as a videotape,a film, a poster, a leaflet, a radio program, etc., before they are usedon a wide scale. Evaluative research is also concerned with the overallimpact of a whole program of communication after a given time period,and with feedback about the management and day-to-day process of thewhole program.
4. The six steps for developing and pretesting communication area. communication planningb. designing the communicationc. deciding on the methods for pretestingd. conducting the preteste. analyzing and interpreting the resultsf. reporting and implementing the results
5. The second and third phases of communication planning are settingcommunication goals and designing communication strategy .
6. The four basic criteria for setting useful goals area. clearly stating the kind of change that is expectedb. specifying how much change is expected to occur
c. stating how the change can be measuredd. specifying the time period during which the change is expected to occur
47
7. In order to make the results of pretesting easier to understand and implement,it is recommended that the conclusions and recommendations be given in thebeginning rather than at the end:a. introduction to the problem and brief overview of the reportb. conclusion and recommendationsC. the main body of the reportd. supporting material and appendices
8. Sometimes a specific part of a communication, such as the size of a posteror the photograph that is used, can have a significant impact on the effective-ness of the communication as a whole. A dramatic color or a strikingphotograph may increase the amount of attention and interest by the audience.A key phrase may confuse some people and decrease the overall understandingof the message. Unless information is obtained about the important ap its ofa communication, they will be ignored when efforts are made to improve itlater.
9. Limiting our pretests to just the parts is likely to leave us unable to "see theforest because of the trees." We may develop a lot of good parts--the rightcolor, the right size, a clear slogan, a beautiful photo, etc. --but the partsmay not have much effect when they are taken as a whole. Eventually, weneed to find out how our audience reacts to everything. Does the photographmerely take the audience's attention away from the verbal message? Afterexposure to the whole communication, what is remembered most clearly?Does the audience know how the picture and verbal message are related?
10. "Funneling" is a simple way to organize our questions about the communicationwe are pretesting so that we can get the respondent's general reaction to thewhole thing before he is influenced or biased by our questions about specific
parts of it. Questions begin with the reaction to the whole communication,then gradually narrow down (like a funnel) to more and more specific aspectsof the communication's impact on the audience.
11. Seven specific effects which communication may have on the audience area. attention
b. interest and enjoymentc. understandingd. believing
e. agreement
f. recallg. behavior
12. The main advantage of pretesting in a centrally located studio or office is
that it allows us to control the situation in which the audience responds to our
communication. We can eliminate outside interference or competition from
48
other sources of communication. We can control the order and pace inwhich information is shared, and we can ask our audience to do things (likeuse special equipment) which cannot be done very practically in the field.
13. The big shortcoming of studio or laboratory testing is the unnatural orartificial atmosphere in which communication takes place. Field tests,especially if they are conducted in settings and conditions very close to thosein which the communication will ultimately be used, overcome this problemand reduce the bias or influence of the testing itself on the audience's reactionto our communication.
14. One of the key advantages of posters is their ability to reach people inplaces where they frequently walk or visit. To be effective, however, theyoften have to compete with other posters and other kinds of announcements.The effectiveness with which a poster can attract attention and invite apasserby to stop and read further can be tested by alternating a set of postersdeveloped for the same purpose, and then observing how people react whenthey walk by them. Do they stop and read further? Do they talk to peoplenext to them about it? What do they say? Answers to all of these questionscan be obtained without the respondent knowing you are conducting a test untilafter he or she moves on and is interviewed by your co-investigator.
49
UNIT II
SELECTING RESPONDENTS FOR PRETESTING
OBJECTNES
After completing this unit you will be able to:
1. Explain the importance of pretesting a representativesample of the intended audience.
2. Define the basic terms used to discuss and to designsampling procedures.
3. Describe the basic differences between accidental,probability, and quota sampling methods.
4. Use a table of random numbers to select a probabilitysample for pretesting communications.
5. Use information (percentages or distributions) about anaudience to calculate and to design a quota sample forcommunications.
52
I NTRODUCTION
All people have their own unique, personal experiences that influence (or
bias) their reactions to communication, and hence the feedback they will giveus about particular communications. This is the reason it is so important to
get more than just one person's response. This is also why it is important toget the responses of people who are representative of the intended audience forthe communication. One of the main purposes of pretesting is to find out how thepersonal experiences and biases of the audience will affect the impact of ourcommunications.
The problem of who to select to pretest a communication product is rifewith disagreement and misunderstanding. Much of the misunderstanding resultsfrom confusion regarding the purpose of pretesting. If the purpose is to judgethe aesthetic quality or professional standard of the product, then there areusually competent judges available in your own office. But if the purpose is toassess the impact or effects upon a specific audience, then the only way to obtainthat information is by getting the responses of members of that particular groupof people.
The basic question is, Which group of people should we select for thepretest? It is important to remember that: (1) anyone can give us some usefulinformation about our communications, (2) only people with special qualificationscan give us professional opinions, and (most importantly), (3) only members of ourintended audience can give us valid information about what effects the communicationwill have on them . Therefore, it is useful to test our ideas and communicationproducts with our professional colleagues and other staff members. Their experienceand insight should give us many valuable suggestions and help us to prevent mistakesthat have been made in the past.
Valuable information can also be obtained from staff members who work inthe field, such as family planning clinic personnel, doctors, nurses, field workers,agricultural extension workers, and so forth. Many of these people are expertsin family planning, of course, but their main value for pretesting is based on theirexperience in the field. Many of them have had much experience communicatingwith members of your intended audience. This personal experience with the audienceis what makes their opinions useful. Their advice may be useful, both at theconceptual stage and later when alternative products have been completed.
As with all information that is obtained from pretesting, the characteristicsof the source of information (the respondents) have to be taken into account. Forexample, field workers or clinic personnel, because of the nature of their job, maythink that some messages are more important than others, that certain words shouldbe used, and that objectives related to their job should be given more emphasis.Field workers may think they know our intended audience well, but in actuality
53
PRETEST OF SELECTED FIGURES FOR USE AS A
NATIONAL SYMBOL FOR FAMILY PLANNING
Questionnaire No.(for office use only)
Your (Respondent's) Name:
Address:
Occupation:
Whether Married?: Yes: No: Age:
Whether the spouse is employed?: Yes: No:
If so, the occupation:
Place of work of the spouse:
Your educational qualifications:
The last grade you passed in school?:
Whether you read newspapers: Yes: No:I. Regularly: 2. Seldom:
Your status at home?:( Whether the chief occupant, housewife, son, daughter, or relation)
At the conclusion of the interview, the interviewer may include therespondent in any one of the categories mentioned below on the basis of theinterviewer's observations reached by the exchange of views with him or herand the interviewer's personal estimation with regard to the respondent's socio-economic status.
Group: A: B: C:
234
Interviewed at: Religion Nationality
Home: Buddhist: _ Sinhala:Work place: Hindu: Tamil:Any other place: Muslim: Moor/Malay:If so, state Catholic: Burgher/
the place: Protestant: Eurasian:Other:
1. What do you understand by this symbol?
A:B:C:D:E:F:G:H:
2. Is this symbol suitable or unsuitable as a symbol for a national project?[A through HI
3. i. Among these symbols which symbols express love and affection?
[A through H]
ii. Among these symbols, which symbols express discipline?[A through HI
iii. Among these symbols, which symbols express brotherhood?[A through H]
iv. Among these symbols, which symbols express justice?[A through H]
v. Among these symbols, which symbols express hostility?[A through H]
vi. Among these symbols, which symbols express suspicion?
[A through H]
vii. Among these symbols, which symbols express religious faith?[A through H]
4. (a). Which one of the symbols do you like best? (In order of mention 1, 2, 3)[A through 11]
235
(b). Explain in your own words why you like this symbol best?
5. Out of these symbols, which are the symbols that you dislike most?
(In order of mention 1, 2, 3) [A through H]
6. Which colour do you like best?
Red: Yellow: Blue: Violet: Brown:
7. Out of the symbols I have shown you, which symbol can you draw by memory?[A through H]
8. Please draw it here:
9. Please look at all the symbols. Among these symbols, which symbol ismost suitable for family planning?
[A th rough H]
10. Which symbol is not suitable to be used in family planning activities?
[A through HJ
Date of the interview:
Signature of Interviewer
236
Example 10: A Questionnaire for Improving the Quality
of Instructional Communication
Source: Adapted from "Guidelines for Improving Instruction" inD. Lawrence Kincaid with Wilbur Schramm, Fu ndamentalHuman Communication , Professional Development Module( Honolulu: East-West Communication Institute, 1975),pp. 152-154.
237
A QUESTIONNAIRE FOR IMPROVINGTHE QUALITY OF INSTRUCTIONAL COMMUNICATION
Tinctrii ti nn G•
You will be viewing segments from two educational television programs:(1) Sesame Street, designed for pre-school children, and (2) Electric Company,designed for primary school children. Each of these programs should providegood examples of the factors which make learning more effective. These fac-tors are listed below for your use while you are viewing these videotapes. Readthrough the following questions before we begin viewing the programs, theneither during the program or immediately afterwards, fill out your answers tothe questions.
1. Are the learning objectives made clear?
YesNo
If yes, then write down what you think the learning objectives are:
How are these learning objectives made clear to the viewer?
2. Does the program get the attention of the viewer?
YesNo
If yes, what is done during the program to get the viewers attention?
3. How often is the content to be learned repeated ?
Once or twiceSeveral timesMany timesToo many times
239
Is the content to be learned repeated with variation? Is it done over againa different way?
YesNo
4. Does the presentation of the content to be learned proceed step by step at theappropriate rate or pace for the viewer? Do you think the pace is . . .
Too slow
Just the right pace (or speed)Too fast
5. Does the viewer have the opportunity to practice what he is to learn? In
other words, does anything stimulate the viewer to do something while hewatches ?
YesNo
If yes, what would the viewer be stimulated to do? What kind of activity?
What method or technique is used to stimulate this viewer activity?
What rewards the viewer for a correct response ?
6. Does the viewer have the opportunity to see another person on the screen(adult or child?) doing what the viewers are supposed to learn? In otherwords, is the viewer presented with an example of the correct response so
that he can imitate it at home ?
YesNo
240
How is this done?
7. Does the program begin at the learner's present level of ability , knowledge,and accustomed ways of learning?
YesNoNo way to tell
If yes, how can you tell from the program that this is done?
241
MODULE MANAGER'S GUIDE
PRETESTING COMMUNICATION FOR FAMILY PLANNING PROGRAMShas been designed to be self-instructional material. It is especially aimed atfamily planning communication production and/or research professionals. It mayalso serve a useful purpose for administrators and practitioners who are engagedin I.E.C. (information, education, and communication) and training work for anykind of program that involves the use of mass communication and materials for
programs of planned change.
For professional development, the module requires some working expe-rience in communication to grasp the contents of the module thoroughly. Themodule can be conducted in small groups of 5 to 15 or large groups of 20 to 30people with the guidance of a manger, or resource person, who has either con-ducted similar training before, who who has actually carried out pretesting. Itcan also be used effectively by individuals who have had practical experience inthe production or research of mass communications. Ideally, the group shouldconsist of the participants who represent both areas of specialization (produc-tion and research in mass communication).
A fairly good knowledge of the English language is required to take partin the module. Although there are very few technical terms and little profes-sional jargon used, some of the terms will be new and will require thoroughexplanation. Whenever possible these explanations have been provided eitheras footnotes or through examples and definitions. The past experience has indi-cated that the majority of the participants who took this module--people fromAsian countries, the United States, Latin America, and Africa--found the lan-guage quite easy to understand. The concept of pretesting and many of the metho-dological procedures suggested in the module were reported to be new by previ-
ous participants, but most participants were able to understand them, especiallyafter they had a chance to apply them in practice. Similarly, the organizationof the module was found to be quite logical and interesting.
It is important for the manager, leader, or resource person who conductsthe pretesting module to know the basic concepts and process of "modular" profes-sional development training. A module is a set of self-contained, self-instruc-tional learning materials that enables the participant to read and understand thecontents at his or her own individual ace. It also provides an opportunity to theparticipants to apply the concepts that they have learned from the module by doingthe exercises and self-tests in the module as they proceed. Also, a module isbetter than the conventional way of taking notes in the sense that it becomes apermanent record of the learning experience for later reference. Furthermore,the module is designed and bound so that it is possible to add more material to it,to take material out of it and, more importantly, to adapt it to suit a variety ofsituations.
In general, a module should have the following characteristics:
245
they may have spent more time with only certain types of audience members(perhaps older, better educated, higher status men and women who already
practice family planning).
A field worker may not be familiar with, or favor, the use of a particularslang word (or drawing) on a family planning poster, but because of the audience'sprevious experience, it may be the best word (or drawing) to communicate the
intended meaning to them.
OBTAINING A REPRESENTATIVE SAMPLE OF THE AUDIENCE
There is an old rule of sampling that if all members of a population arethe same, then you only need to sample one person to find our what all of them arelike. If the members of a high school are all of the same sex, for example, thenyou only need to see one student to find out whether the school is only for boys orgirls. In some countries today, the answer to the question, "Have you ever heardof family planning?" is so common among such a large proportion of the
population, that you have to sample only a very small number of people to findout what most people would say.
For most of the questions that we have, and for most audiences we areinterested in, there are enough differences (variations) among people that it isnecessary to sample many people to get answers that tell us adequately how thepopulation in general would respond. Suppose we want to test the reaction of a
certain audience to a new family planning theme, for example, "family planningis a moral obligation. " If there is a lot of difference (variability) in religionamong members of this population then we certainly would not be satisfied withjust asking people of only one religion about the slogan. People with differentreligious backgrounds and experience--Christian, Moslem, Buddhist, Hindu,and so forth--are likely to react differently to such a slogan.
Furthermore, if there is approximately the same proportion of membersof each religion in the whole population, then we would not want to include manymembers of one religion in the sample but just a few members of the other
religions. In this case the sample would not be very similar to the whole audiencefor which the communication is intended. The responses of the sample to ourcommunication---the slogan--would be biased in favor of just one of the religious
groups in the population. The results are not a very good indication of how therest of the population would react to our communication.
If one religious group (or one sex, one age group, one region, etc.) isover-represented in our sample, then we might get a more favorable reaction toour slogan than we will get later from the whole population. If our sample isbiased by the over-representation of another religious group, then we may be wrongthe other way; we may think that most of the population disapproves of our sloganwhen only a small portion of the population actually disapproves.
54
The purpose of all sampling methods is to overcome this problem, to
obtain a sample of responses that reflects or represents the responses of the whole
population as closely as possible. In general, the more different lands of peoplethere are in the intended audience, the greater the variation in the responses toour communication, and the more people we will have to sample for pretesting.We would have to select a larger sample in order to include members who representeach of the various (different) groups in the population.
This implies that pretesting communications for national distribution orbroadcast in a country like India, with many cultural regions and language groups,will be much more expensive and time-consuming than in a smaller, more similarpopulation such as that of Korea. These are national extremes, of course. ForIndia, this may mean that most communications cannot be developed for nationaluse, but must be designed (or adapted) separately for use in each district orregion. On the other hand, the greater similarity among the people of Korea
does not necessarily eliminate the need for pretesting messages on a variety of
subgroups within its population: differences in income, education, occupation,position in the life cycle, sex, local dialect and historical experiences will stillinfluence message perception, interpretation, and behavioral impact. Thisdifference in variability among audiences just means that the cost and difficultyof testing communication for national use will be less in Korea than in India.
DEFINING THE LANGUAGE OF SAMPLING
Before we describe the ways in which respondents for pretesting may beselected, we will clarify some of the terms which will be used. It is not uncommonfor people to use terms like audience, population, census, sample, and so forth,without paying too much attention to their specific, technical meanings. Sometimesthese terms are used interchangeably as if there were not much difference intheir meaning. Learning these terms now will not only help us design ourprocedures for pretesting, but also help us understand the research that othershave done.
1. A population is an aggregate, set, or collectivity of all of the cases thatmeet certain specific characteristics. If we use "people" and "who livein Thailand" as our specified characteristics, then we would define apopulation of all of the people who live within the boundaries of Thailand.Population is a general term which refers to all of the members of acertain set of objects which have common characteristics.
A population does not have to consist of people. We can definea population as all of the official, government health and fami-ly planning clinics within Thailand. if we are concerned abouttesting the quality of the oral pills used in Thailand, we maywant to define our population of interest as all of the oral con-traceptive pills imported or produced by Thailand. The
55
population is defined according to what group of objects we want to study.
We can even treat time as our population of interest. Suppose there area certain number of places in a city where we can place posters ormobile vans with information about family planning. Which places wouldprovide the most exposure to the greatest number of people? Becauseit might be too time consuming to study all of these places at every hourof the day, we might want to define our population further as "all five-minute (or fifteen-minute) time-segments at all of these locations from6:00 a. m. until 9:00 p. m. , seven days of the week.
"Why would we want to define our population of interest according totime --segments?" If it is impossible or too expensive to study thepeople who pass by all of these places during all of the time required,then we might select just some of the time-segments at each location.Selection of just some of the time-segments from the whole populationwould give us a sample of time-segments. If we studied all of thesetime segments we would be conducting a census of the total population(time-segments).
2. A census of a population is a count of all of the members (or elements)of a population to determine their characteristics from informationobtained from each member. Since it is often unfeasible and tooexpensive to obtain information from all members of a defined
population, it is usually necessary to choose only a sample of memberswho are representative of the whole population.
3. A sample is a part or subset of all of the members (elements or objects)of a specified population. We usually select a sample of a population inorder to estimate (or to infer ) from the characteristics of the samplewhat the characteristics of the whole population are. If we pretest aradio tape with sample of mothers who come to a clinic, we expect thatthe mothers who actually get selected (the sample) are similar on theaverage to all of the mothers (the population) who come to the clinic.Then we would go one step further and infer that the responses of thesample to our radio tape would be similar on the average to theresponses we would get if all of the mothers who attend the clinic hadheard it and responded.
The example of the clinic shows the importance of sampling forpretesting and other kinds of research. If we stop and think about aclinic for a minute, we would realize that it is impossible to study allof the visitors because the total set is never complete or final. Thereis endless flow of people through a clinic over time, so that the wholepopulation continues to change as time goes by. The people who cometo the clinic may be very similar on the average as time goes by, but
56
it is easy to imagine how their characteristics might change.
The average age of typical visitors, for example, may increase or
decrease as time goes by. The average years of education may increase
as time goes by. Furthermore, communication may be designed tochange the type of people who come to a clinic. Special radio messagesmay be produced to persuade younger women with fewer children (or even
no children) to visit the clinic, or to encourage more men to visit theclinic. In this case, we would want the average characteristics of the
population who uses the clinic to change over time. The communicationwhich is designed for this clinic should be appropriate for this changingpopulation. Pretesting helps us insure that the communication matches
the audience which uses the clinic.
The kind of population whose members continue to change (or leave asnew ones join) over time is an endless population. In sampling theorythis kind of population is often called an "infinite" population because
its members change indefinitely. A population with a fixed, orrelatively fixed, set of members over a given time period would becalled a "finite" population. For sampling purposes, it is usuallypractical to treat a population whose members and characteristicschange relatively slowly, such as the population of Thailand, as a finitepopulation. We mention the distinction between infinite and finitepopulations because it is important to remember that change is goingon, and that sometimes we use communication to direct or speed up this
change.
The population of Thailand today is not the same as the population ofThailand ten or twenty years ago. It has more people of a younger agebracket, with better education than it used to have. The population of"couples who practice, or need information about family planning" isalso different, in part because of the family planning program and thecommunication that was developed for that program. Pretesting, ofcourse, is one of the ways to make sure that our message designs arenot lagging too far behind actual changes in the population, and changesin the members' life experiences.
4. Sampling is a procedure that is used to select some of the members of apopulation in order to estimate the characteristics of all of the membersof the same population. When we pretest communication with a samplefrom a population, we hope that the responses will be similar to theresponses we would get if we could study the whole population. For thisreason, it is important to use procedures for selecting a sample whosemembers are as representative as possible of the members of the wholepopulation. When we present the results from the study of a sample,we should always be ready to describe the procedures that we used to
57
select the sample from the population, and give some idea of howsimilar our sample is to the whole population.
5. The audience that is chosen for communication is often treated as if itwere the same way as the "population" (defined above). Certaincharacteristics are used--such as all males between the ages of 30and 55 who live in cities--in order to define an audience for which com-munication is designed. For purposes of pretesting, we would considerall members of this special audience to be members of our population ofinterest. To test our communications, we would want to select a sampleof this audience or population.
At this point the difference between audience and population is important.Continuing with the same example, not all of the urban men from age30 to 55 would ever actually be exposed to our communication. Some ofthe members of this population probably cannot be reached by the mediawhich are usually available for our communication. If we only plan touse radio, for example, it is possible that only 30-40 percent of thispopulation of men would ever hear our communication over a period oftime. The eventual audience would be different from the population wehad defined.
Is there any difference between the members of this population wholisten to radio and those who do not? The obvious difference is thatone group has access to radios, and the others may not. But this isvery likely to be related to other important differences, including thekind of jobs and income they have, their educational level, differences
in their style of life, and past experiences. All of these differences caneffect the way they respond to our communication, and to the pretestsof our communication.
When we pretest communication on a sample from a population that wehave defined we should always ask them whether or not they would everbecome members of the actual audience. Ask them a few questions abouttheir media habits, if they have ever seen family planning posters ormobile vans, have ever been to clinics, have ever talked to their friendsabout family planning, and so forth. Find out which members of thesample are most likely to become exposed to the communication we arepretesting, then check to see how their characteristics and responses tothe communication that is pretested differs from that of the other membersof the sample.
If funds and time for pretesting are limited, then we might want to screenmembers of our sample before going ahead with the actual pretest. Thosewho are not likely to be reached by our communication could be eliminatedfrom the pretest. It makes little sense to pretest and redesign ourcommunication so that it is more effective for people who are never
58
going to be exposed to it. During the screening, however, we shouldtry to find out why they are not exposed to certain media, how thatcould be changed, and which kinds of media or interpersonal com-
munication they are exposed to that could be used for other familyplanning communication. This problem suggests that one of the outcomesof pretesting might be to change the media that we were intending to use,as well as to redesign the communications. When we go to the field to
conduct our pretests we should take the opportunity to learn as much aspossible about our audience and the media they use, as well as theirresponses to our communications.
TYPES OF SAMPLING
1. Accidental sampling is done by simply choosing the first cases that one
finds in the population and then continuing until the sample reaches thedesired size. An interviewer, for example, may take the first 30people he meets on a street-corner who are willing to read a pamphletand be interviewed. Or, if this is too much trouble, he might just ask
his friends who fit the characteristics of the population. A teachermight ask members of his own class to respond to a new family planningflip-chart he has developed, hoping that the results would generalize toall students in the nation's secondary schools. A mobile van teammight get responses to a new film from volunteers in the next threevillages on their itinerary.
The main problem with accidental sampling is that all sorts of knownand unknown biases will affect the results. People who easily volunteerto read a pamphlet on the street are obviously very different from thosewho refuse (perhaps better educated, more out-going, better communica-tors). Those who refuse may actually be more like our intended audience.The members of a teacher's own class may have already learned how tounderstand many of their teacher's unique ways of communicating, andthey certainly will have learned how to please the teacher. Besides theproblem of having only volunteers, the mobile van team may be restrict-ing their results to only one geographical area, and thus to one type ofaudience. The most obvious fault with accidental sampling is thatcertain members of the population will never have a chance of beingselected, and therefore can never be represented in our pretest results.
2. Probability sampling is done so that no one's personal preference deter-mines who is selected. And most importantly, every member of thepopulation has an equal (or known) chance of being selected in the sample.To achieve an equal chance, members of the population are randomlyselected. Random selection cannot be done according to the "accidental"
59
choices of an individual. Some mechanical method must be employedwhere no human choice is possible. Human choice always involves ajudgment of some kind, and often the reasons (bias) are unknown.
The most efficient, scientific method of selecting a random sample isby using a list or table of random numbers such as may be found in mosttextbooks of statistics. These are sets of numbers created by a computerwhich, after careful examination have shown no evidence of any systematicorder. Within any set of these numbers, each digit from 0 to 9 is justas likely to occur as any other number. Thus, we can say that eachnumber has an equal probability of occurring on the list and of beingencountered.
How can a list of random numbers be used to select a sample of membersfrom our population of interest? First, we must assign a number to every memberof our population. This can be done by simply numbering them all in order fromthe first to the last one (1, 2, 3...n). Sometimes the members of our populationalready have some kind of number, where no two members have the same number.Seats in a movie theater, for example, sometimes have numbers which can beassigned to whoever sits in them during a film. A mobile van team visiting arural village could easily hand out to each audience member a family planningpamphlet which has been numbered in advance. Then they could announce thatsmall prizes would be given after the showing of a film to a random selection offifteen to twenty people who have the chosen numbers on their pamphlets. Thewinners could then be asked to stay after the film for a short interview about theirreactions to the film... and to claim their prizes. Once every member has his ownunique number, we can use a table of random numbers to select a sample whosenumbers correspond to the numbers of the population members.
To do this we would first enter the table of random numbers not bychoosing the first number, but rather by blindly placing our pencil on the pageand beginning with the nearest number. In most pretesting situations we would
want either a two-digit number (01 to 99) or a three-digit number (001 to 999).Then moving down the list of random numbers, we would simply take all of thenumbers of this size (two or three digits), and thus the members of ourpopulation who have the same numbers. We would continue through the table ofrandom numbers until we have obtained enough numbers to complete the size of thesample that we need from our population. The numbers in the table are randomand have an equal chance of occurring, so the members selected from ourpopulation who have these same numbers will also have been selected randomly.Every member of our population to whom we have assigned a number has an equalchance of being selected because the numbers being drawn from the table ofrandom numbers all occur with the same probability.
Random probability sampling is often too expensive and time consumingfor purposes of pretesting. Sometimes it is impossible to get a complete list of
60
all the members of a population beforehand. This should not discourage us fromusing probability sampling where it is feasible. It may be possible to select arandom sample of postpartum patients in a hospital, for example. It is relatively
easy to have hospital personnel number all new patients up to say 300, as they
enter over a three-month period. A random sample of 30-40 numbers between 1and 300 can then be selected from a table of random numbers, and when the womenwho are assigned these numbers are in the hospital they can be interviewed. Thismay give us a wider range of patients over a longer period of time than if you justtook the first 30 to appear.
This would also be useful if there were not enough personnel to interviewall of the women in the hospital before some had left. If it is impossible tointerview all of them, then the situation calls for a choice, a selection of a sample ,to be interviewed. Why not number them all, and select a random sample with a
table of random numbers?
By simply numbering the 50-100 members of a village who come to viewa new film in the order in which they sit, a staff member can select 10-15 membersfrom a table of random numbers during the film, then ask those who are selected tostay after the film to be interviewed. If this procedure is followed in ten villages
then the team will soon have the responses of 100-150 viewers chosen randomlyfrom the audiences in these ten villages. This sample would represent thepopulation of viewers in these ten villages.
How well these ten villages represent all of the villages who willeventually see the film depends upon how the villages were sampled. Where it isimpossible to get lists of people, it is often possible to get lists of villages, towns,districts, blocks, neighborhoods, or cities. These lists can be numbered and usedfor random sampling purposes for a number of years until changes require using anew list.
Suppose we want to test a slide-tape show designed for rural audiences.If we have ten townships to choose from, for example, we may want to eliminatetwo because they are too close to urban areas, and three more isolated townships,because we are sure the show will never be used there. This leaves 5 townships.Let's assume we want to sample at least 20 villages, and that each township hasapproximately the same number of villages, say 16 villages per township (16 x 5townships= 80 villages). If each township office maintains a list of villages whichcan be numbered, then it is quite easy to select 20 out of the 80 villages by usinga table of random numbers. If a mobile van team regularly operates in these fivetownships, then their route can be altered so that they can go to the 20 randomlyselected villages to conduct the pretests.
Specific sampling procedures for each village can then be followed. Theteam which shows the film may be able to observe the whole audience's reactionduring the show, assess their level of interest with a question and answer session
61
afterwards, and then interview 10-15 randomly selected viewers, either individ-ually or as a group. By using a random selection procedure to choose the 10-15viewers for the individual interviews, we should be able to get results which are
more representative of the whole audience, and probably more reliable. If we askfor volunteers, for example, we are likely to get the usual spokesmen for thevillage who often have higher status and more education than the rest of thevillagers. And past experience may have taught them exactly what to say to pleasethe interviewers.
We do not always have to use accidental sampling procedures just be-cause it appears to be the easiest. For very little additional work or cost it isoften possible to use probability sampling and thus eliminate a lot of unnecessarybias from the pretesting results. Various combinations of sampling proceduresand pretesting techniques can often be used depending on the nature of the audience,the type of communication, the time allowed, the cost and skill of the pretestingteam, and so forth. Once a system for using probability sampling has been set upit becomes easier to use with experience.
62
AN EXERCISE FOR PROBABILITY SAMPLING
To get an idea of how to use a table of random numbers, let'sactually select the four villages from each of the five townships.Using the excerpt from the following table of random numbers,proceed thus:
(1) Enter the table by blindly placing your finger or a
pencil anywhere on the table without looking.
(2) Since our villages are numbered from about 01-80 (or5 x 16 villages) take the closest two-digit number (00-99)that your pencil falls on. These first two digits will bethe number of your first village if they match one of the
villages' numbers. if the first number does not match(is higher than 80, the highest village number), thencontinue to the next step.
(3) After writing down the first number on the followingpage, move down the table to the next two-digit numberin the column. This is the second village numberselected for our sample. Write this number in theappropriate space on the following page.
(4) Continue down the table, or go to the next column, untilyou have selected 20 two-digit numbers correspond-ing to the twenty villages sampled out of eighty.
(5) If you come to a number like 92 which is above 80, thenthis number may be skipped because it is outside ourrange, and therefore does not correspond to any of ourvillages.
(6) List all 20 of your two-digit numbers and indicate thename, location, and township of each village selected.
NOTE: By randomly selecting 20 villages out of 80, you may by chance
get more villages from one township than others. If you wantfour from each township , then enter the table five times and
select a separate probability sample of 4 villages from each ofthe five townships.
63
(7) An extra sample for substitution : In some situations itis useful to have an extra sample from which to drawsubstitutes for cases which cannot be studied. In thepresent exercise, for example, one or more of the 20villages in the original sample that you have selectedmay not be willing to cooperate with the pretest. Anothervillage may turn out to be too difficult to reach becauseof difficult road conditions, poor weather, and so forth.One of the sampled villages may be experiencing an un-expected calamity, or an internal political problem thatcreate inappropriate conditions under which to pretestnew family planning communications.
Since the unexpected often does happen, it is usually agood idea to continue selecting numbers (and theircorresponding villages) from the table of random numbersuntil you have an extra sample which can be used whensuch problems occur. Substitutes should not be used fromthis extra sample because they look better than others inthe original sample, because they are closer and easier totravel to, because someone on the pretest team Imows thevillage leaders, and so forth. The extra sample should be
used only when reasonable efforts have been made to usethe original sample. The extra sample should be used inthe order in which members are originally selected fromthe table of random numbers. As far as possible, alwaysfollow the general rule: Allow no human judgment toinfluence the selection, and insure an equal probability forall members to be selected.
64
USING A TABLE OF RANDOM NUMBERS
On the next two pages we have reproduced a set of randomnumbers for you to use to draw a sample of your own for this exercise.
Enter the table blindly and record the first 20 two-digit numbers.you encounter in the table which are within the range of the 80villages from which you wish to sample (range = 01 to 80). Thenumbers you select may be recorded on the page after the table ofrandom numbers.
Source : The Rand Corporation, A Million Random Digits with 100, 000Normal Deviates . (Glencoe, Illinois: The Free Press, 1955.)Q 1955 by the Rand Corporation. Used by permission.
65
12651 61646 11769 75109 86996 97669 25757 32535 07122 7676381769 74436 02630 72310 45049 18029 07469 42341 98173 7926036737 98863 77240 76251 00654 64688 09343 70278 67331 9872982861 54371 76610 94934 72748 44124 05610 53750 95938 0148521325 15732 24127 37431 09723 63529 73977 95218 96074 42138
74146 47887 62463 23045 41490 07954 22597 60012 98866 9095990759 64410 54179 66075 61051 75385 51378 08360 95946 9554755683 98078 02238 91540 21219 17720 87817 41705 95785 1256379686 17969 76061 83748 55920 83612 41540 86492 06447 6056870333 00201 86201 69716 78185 62154 77930 67663 29529 75116
14042 53536 07779 04157 41172 36473 42123 43929 50533 3343759911 08256 06596 48416 69770 68797 56080 14223 59199 3016262368 62623 62742 14891 39247 52242 98832 69533 91174 5797957529 97751 54976 48957 74599 08759 78494 52785 68526 6461815469 90574 78033 66885 13936 42117 71831 22961 94225 31816
18625 23674 530 32827 81647 80820 00420 63555 74489 8014174626 68394 88592 70745 23701 45630 65891 58220 35442 6041411119 16519 27384 90199 79210 76965 99546 30323 31664 2284541101 17336 48951 53674 17880 45260 08575 49321 36191 1709532123 91576 84221 78902 82010 30847 62329 63898 23268 74283
26091 68409 69704 82267 14751 13151 93115 01437 56945 8966167680 79790 48462 59278 44185 29616 76531 19589 83139 2845415184 19260 14073 07026 25264 08388 27182 22557 61501 6748158010 45039 57181 10238 36874 28546 37444 80824 63981 3994256425 53996 86245 32623 78858 08143 60377 42925 42815 11159
82630 84066 13592 60642 17904 99718 63432 88642 37858 2513114927 40909 23900 48761 44860 92467 31742 87142 03607 3205923740 22505 07489 85986 74420 21744 97711 36648 35620 979,1932990 97446 03711 63824 07953 85965 87089 11687 92414 6725705310 24058 91946 78437 34365 82469 12430 84754 19354 72745
21839 39937 27534 88913 49055 19218 47712 67677 51889 7092608833 42549 93981 94051 28382 83725 72643 64233 97252 1713358336 11139 47479 00931 91560 95372 97642 33856 54825 5568062032 91144 75478 47431 52726 30289 42411 91886 51818 7829245171 30557 53116 04118 58301 24375 65609 85810 18620 49198
91611 62656 60128 35609 63698 78356 50682 22505 01692 3629155472 63819 86314 49174 93582 73604 78614 78849 23096 7282518573 09729 74091 53994 10970 86557 65661 41854 26037 5329660866 02955 90288 82136 83644 94455 06560 78029 98768 7129645043 55608 82767 60890 74646 79485 13619 98866 40857 19415
17831 09737 79473 75945 28394 79334 70577 38048 03607 0693240137 03981 07585 18128 11178 32601 27994 05641 22600 8606477776 31343 14576 97706 16039 47517 43300 59080 80392 6318969605 44104 40103 95635 05635 81673 68657 09559 23510 9587519916 52934 26499 09821 87331 80993 61299 36979 73599 35055
02606 58552 07678 56619 65325 30705 99582 53390 46357 1324465183 73160 87131 35530 47946 09854 18080 02321 05809 0489810740 98914 44916 11322 89717 88I89 30143 52687 19420 6006198642 89822 71691 51573 83666 61642 46683 33761 47542 2355160139 25601 93663 25547 02654 94829 48672 28736 84994 13073
66
99601 51471 65762 95234 77402 20188 18849 13919 33596 87CC725257 33617 00655 11574 47550 38116 23244 51005 16264 722736G280 56404 91348 85343 30667 46904 42425 45099 58463 4869521544 16607 59016 26219 60428 48467 26833 35153 64738 4591447718 78264 26855 17858 38396 68330 91469 96411 66433 79718
61302 14994 30389 78942 46196 28117 19894 58106 76366 2983870816 66830 89859 04680 11549 34163 99851 64754 30226 0953243320 41106 28176 94665 31622 88452 69750 77548 66310 2359031113 54441 77616 78977 97764 21658 42293 16096 70724 5506322696 10093 91060 10747 16264 73623 34329 93638 33602 48534
31233 53364 67098 85674 63744 50411 19826 23572 02841 9256539168 73348 79469 73044 714C7 74514 79768 32901 39000 2267756241 03598 66677 22159 80929 69312 29143 87230 30877 2385048690 40746 13957 95733 84913 08361 53559 67588 50666 4138589633 97516 54609 73318 12957 89932 21938 34869 00691 77241
93877 62102 66636 55867 924).9 03881 95413 40654 38506 7824673949 69166 77245 89795 04384 53923 31444 41555 75766 1359770860 44944 46881 30390 56910 74263 97360 70507 24079 3964213601 50427 42059 91206 03480 31180 84892 16022 95166 7442319044 41385 78385 83251 99352 39493 62516 56078 39129 28776
20260 84785 27316 08705 67860 06892 08835 23570 98996 6862945442 95461 32214 51281 54593 91758 42151 19638 29640 2822891025 31825 46374 85062 53154 13729 51277 22360 35946 6953571551 73305 71533 82520 58649 66882 62731 88648 03547 1189699093 75072 39307 78334 50001 13849 99488 38293 01443 80620
80149 54187 32644 00232 81067 57848 79389 82826 06426 8474635199 16009 69814 60275 10908 52193 16002 12014 27608 5761306888 77507 14802 29621 32274 40015 56728 12687 48951 5966256493 51337 99588 22502 72306 37880 99346 77619 63034 0934283581 99353 94901 06872 32339 95240 11542 51796 25861 77485
47556 43388 68872 57938 41865 60560 38774 09447 97019 7751775123 85725 95533 21770 31215 37123 53873 09753 38104 3444241454 98644 31366 92865 95690 24449 22680 46319 01078 1184976358 61463 83596 11395 26703 89051 16667 42369 23497 4850295294 40334 87223 65615 06592 85625 77340 24729 21342 64111
37004 97392 54517 96093 61969 15548 27777 85607 22501 9284148593 00413 80814 89520 19498 21795 21426 46789 41950 4321576317 63969 30354 57446 06656 85245 86715 66893 48876 6817135458 44736 74185 25C62 39358 96634 21854 74382 36058 3972602340 09602 93024 60237 52050 18315 60075 75244 46991 78737
17315 70525 57884 86772 24035 09697 33803 99992 24529 2639615973 29212 69833 27638 09388 29418 02872 85196 21381 3883913657 44807 89449 28615 90867 11932 07120 59073 98040 7597850980 78043 36212 85699 17275 47473 98954 56577 35232 2096041717 25633 38792 51649 10653 88024 14628 50360 95161 55748
77638 92514 14073 63818 78394 41353 11863 21433 47409 3374419274 58884 86895 46992 41353 90328 96211 32408 30022 8567996990 15406 38232 62274 78496 82433 51260 89515 11742 7869270562 73804 77738 30026 22116 67311 80734 36520 00742 0829731605 29308 02748 62615 97160 03 .109 98644 03994 271.76 89803
67
List of NumbersFrom Table of Random Numbers
Selected NumbersFbr Each Village Village Names and Locations
Primary Sample
1.
2.
3-
4.
5.
6.
7.
68
21.
22.
23.
24.
25.
18.
19.
20.
Selected NumbersFbr Each Village Village Names and Locations
Extra Sample for Substitution
69
SAMPLE LOSS AND SUBSTITUTION
In the exercise above, there was room for an extra sample in case some
of the members selected originally could not be studied. If there are too many
members missing from the original sample, won't this destroy the advantages ofusing random procedures in the first place? Technically speaking, the equal
probability of the sample is slightly affected each time one of the sampled memberscannot be studied because that member actually had no chance, or had zeroprobability of being selected and taken into account in the results of the study.
If all of the isolated, hard-to-reach villages have to be eliminated fromthe sample, for example, then our pretest results will not give us any idea howpeople in these kinds of villages would react to our communications. This is
important only if the reason for elimination, or the characteris tics of the members
eliminated , would make a difference in the pretest results. Isolated villages andtheir inhabitants are often different from those located closer to major roads and
lines of communication. Income, education, even the local dialect may be some-what different. The problem is especially serious when we remember thatisolated villages are often the ones in greatest need of family planning informationand services, and the most difficult ones for whom to design effective communica-tions. The same applies to certain areas, or certain kinds of people in urbancenters.
For many obvious, practical reasons these difficulties sometimes cannotbe overcome. The most important way to handle this limitation is to make carefulnotes describing the members of the sample which are excluded from the study.Then take this information into consideration when the results of the pretest arebeing analyzed. Knowing the differences between the members of our sample whoare studied and those who are not at least allows us to estimate how thesedifferences would influence the effectiveness of our communications. Randomprobability sampling is a procedure for selecting a representative sample ofpretest respondents, and it should not be used to justify ignorance where informa-tion is readily available.
As the size of our random sample becomes smaller the effect of such lossesbecomes more important. The possibility of randomly selecting a sample which isextreme, or very different from the rest of the population, also becomes greater.A small sample which you know is extremely different from the rest of thepopulation (and is obviously unrepresentative) should be eliminated and a new sampleshould be selected. In the exercise above, if 4 villages are eliminated from asample of 20 villages, the effect of their loss on our overall pretest results willbe greater than if they are lost from a sample of 50 villages. If these four villagesare from the same area, then we certainly would want to select another set of fourvillages from that area, again using random selection procedures.
71
For pretesting purposes, sample sizes are often small because of theshortage of time and resources. Since improvement is more important than thefinal evaluation of the communications, it is better to get a lot of informationfrom a few representative people, than a little information from a lot of peoplein the population. These requirements and difficulties with small randomsamples often force us to use more purposive methods of sampling in manysituations. For example, we may have to choose just one village from each offive districts which (1) we think is typical of most villages, and (2) which iseasily accessible by jeep. Nevertheless, even though this may be necessary onone level (districts or villages), probability sampling procedures can still beused at a lower level (such as selecting audience members to view a film). Theadvantages of having a "typical" village can be lost if only the better educatedvillage leaders volunteer to be interviewed after a film. We should attempt toobtain the most representative sample at each stage, and at every point it isnecessary to choose members of the population defined.
3. Quota sampling is a procedure for sampling which attempts to over-come the obvious bias of accidental sampling and the cost and dif-ficulties that sometimes arise with probability sampling. Quotasampling is done by purposely including different types of membersof the population and by making sure that each of these types appearsin the sample in approximately the same proportion that they do inthe population. For example, if a recent census of the population
showed that 20 percent are Christians, 35 percent are Buddhist, and45 percent do not belong to any organized religion, then we wouldattempt to pretest our communications with a sample which includesapproximately the same proportion (20 percent, 35 percent, and 45percent) of each preference that exists in the population. If a totalsample size of 50 is desired, then the pretest team, or interviewers,are instructed to include 10 Christians, 18 Buddhists, and 22 personswith no religious preference.
For example, let's assume that we want to select a sample of thirtypeople in a certain city, and that we have the following information about themfrom a recent census (or survey).
72
Primary or
less
Above Primary
ColumnTotals:
80% x 16 = 1:3 80% x 14 = 11
20%x1.6=3 20%x14- 3
16 14
24
6
30
CITY A
Male-- 55%Female--45 %
100%
55% x 30 16. 5 or 16 men45% x 30 = 13. 5 or 14 women
total: 30
Primary school or less--80%.Above primary school-- 20 %%
100%;
80%, x 30 - 24 primary or less20% x 30 - 6 above primary
total: 30
Sample Quotas :
Education Sex Row
Totals:Male Female
73
In this example, the pretesting team would be instructed to find a sampleof men and women in City A according to the following uq otas :
Find 13 men with a primary education or less,3 men with education above the primary level,
11 women with primary education or less, and3 women with education above the primary level.
When these people are found among the population and have responded to
our pretest, the sample will have the same proportions of sex and educational
level as the population from which it was selected.
This may seem like the ideal way to sample for pretesting, but quotasampling also has some shortcomings. The problem is that people differ inmany ways which may affect their responses to communication about familyplanning.
Geographic location, occupation, income level, education, age, sex,marriage status, parity, religion, and so forth might be used to establish quotasfor sampling. The more characteristics that are required, the more difficultit will be for the pretest team to fill their quotas. It may be easy to test a posterin every geographical region (say, district) in which it will be used later, andrelatively easy to get reactions from an equal number of men and women. Mak-ing sure a certain proportion (corresponding to census data) of these men andwomen have certain levels of education and income becomes a more difficulttask, especially if it is hard for the researchers to determine their respondents'level of education and income. By adding further restrictions, like age andnumber of children, the quotas (number for each type) become extremely dif-ficult for researchers to obtain without an expensive and time-consuming search.The cost may soon exceed that of probability sampling unless only two to three
requirements are made for the quotas.
Once people are found to fill a given quota (for example, high/low incomemothers from age 30-45 in a certain township), it is then up to the interviewer
to select the respondents to include in the sample, so accidental sampling pro-cedures must be used. Often the most willing and accessible people with the rightcharacteristics are selected for each quota. Other people who also meet thequota's specification, but who differ in other important ways from those chosen,may be accidentally or systematically excluded from the sample.
Probability sampling does not have this problem. Since all members(and hence, all characteristics, life experiences, and so forth) have an equalchance of being selected, the probability of each of these important character-istics (of members) being included in the sample is equal to their proportion inthe population. If 10 percent of eligible women in a population have had an abor-tion, for example, and every eligible women has an equal chance of being selected
74
for the sample, then you are likely by chance to obtain a sample in which approxi-
mately 10 percent have had an abortion. Only random variation due to the proce-dures used in probability sampling would make the proportion in the sample dif-
ferent from the proportion in the whole population. Since all characteristicshave a chance of being included in the sample, we do not have to know beforehand(from census or survey data) what proportion of the population has a certain char-
acteristic, like (say) abortion. Quota sampling requires previous knowledge ofthe population's characteristics. To base a quota sample on religion, for example,we would have to know beforehand which religions exist and their relative size inthe population.
With probability sampling all characteristics have a chance of being includedin the sample in approximately the same proportion that they exist in the population.Within the range of chance variation, we can safely say that a probability samplerepresents all of the important characteristics of the population. With quota sam-pling we can only say that our sample is representative of the population in termsof the two or three characteristics that are used to make up the quotas.
Quota sampling gets us a more representative sample than accidental sam-pling, and it is preferred where probability sampling is impossible or too expen-
sive. The results are often quite good, especially where the characteristics used(location, occupation, education, age, etc.) are the most important differences inthe audience which would affect the response to the pretest. The results are moreuseful because they spread the coverage of research teams into areas or types ofpeople they might otherwise avoid if they are allowed to choose their own respon-
dents. A quota sample, for example, would reflect the responses of young peoplefrom 14-18 years of age better than a probability sample of high school students,especially if a large proportion of this population drops out of high school to work.
75
AN EXERCISE FOR QUOTA SAMPLING
Problem: Select a representative quota sample of fifty rural womenfrom age 25 to 44 for a pretest, using the following infor-mation about the whole population based on census data.
Population Characteristics :
(a) Age distribution of age
rural women--0-1415-1920-2425-2930-34
35-3940-4444+
proportion (in percentage)
44%10
767 17 =26% only
613
100%
(b) Education-- 15% no school or primary schoolincomplete
60%% primary school complete25 % middle school or more complete
100%
(c) Literacy rate-- 78%, with most illiterates over age 50.
(d) Family planning practice rate-- 23%
(e) Marriage status-- 85% married15 % never married or widowed
100%
Note: up to age 24 only 50 %I are married
by age 30 only 1% have nevermarried
Instructions: On the following page calculate the number of respon-dents that the pretest team should select (total=50)with the most important characteristics listed above.
76
Do Your Calculations Here
77
Age
25-2930-3435-39
40-44
Educational Level
High Medium Low Totals
11141411
12 50
L23 % x S= 2 (23% x 30 =) 7 231 x 12= 2
27% x 8= 2 (27% x 30 = 8 (27% x 12= ) 4
27% x 8 = 2 (27%x 30-) 8 (27% x 12 = ) 4
23% . x 8- 2 27%x30= 7 23%x 12=)2Total: 8 30
ANSWER SHEET
1. Ignore literacy since almost all of the women between ages 25-44to be selected are literate (most illiterates are over age 50).
2. Age distribution : Find:25-29 6%/26% = 23% x 50 people = 11.5 or 11 women30-34 7%/26%% = 27% x 50 people = 13. 5 or 14 women35-39 7%/26% = 27% x 50 people = 13. 5 or 14 women40-44 6%/26 % = 23%x 50 people = 11.5 or 11 women
Total: 100% Total: 50 women
Note : Since the .5 persons have to be rounded off in the in-structions to the pretest team (or else 52 women se-lected), it was decided to have greater representationof the middle age brackets rather than of the youngeror older women.
3. Education: ;Find:HIGH: 15% x 50 people = 7. 5 or 8 women with middle
school or higherMEDIUM: 60% x 50 people = 30 women who have completed
primary schoolLOW: 25% x 50 people = 12.5 or 12 women who have no
school or incomplete primary
school
50 women
4. Education Evenly Distributed Across All Ages
5. Family planning: 23% x 50 people = 11. 5 or 12 women who arecurrently practicing familyplanning should be included.
78
SAMPLE SIZE
The size of the sample should be governed by:
1. the purpose of the test,2. the type of product that is tested,
3. the amount of variability (differences in the population), and finally,4. the amount of resources and time available.
If the purpose is to get a quick check on possible mistakes in wording,comprehension of specific phrases, credibility, and so forth, of a new radioprogram or poster that is being developed, then a relatively small sample of15 to 30 respondents representative of the intended audience may be adequate.
The results of a pretest should be checked as they are received in orderto discover extreme trends as soon as possible. If, for example, fifteen out ofthe first twenty respondents of a pretest cannot understand the content of a newpamphlet, then it is probably not necessary to continue the test with the rest of
the sample, especially if the first few respondents point out an obvious problemwith the pamphlet which should be corrected. The most important objective in
pretesting is to find out why the first fifteen respondents out of twenty cannotunderstand our communication, not to "prove" statistically that 75 percent of asample of 100 respondents cannot understand it.
How can we be sure that a problem with the content of the pamphlet made
it difficult for 15 out of 20 people to understand it? The problem might just bewith the 15 people who could not understand rather than the communication. Thenext 80 respondents may understand it without any trouble, in which case only15 percent of a sample of 100 could not understand its content. How can we bemore sure with just a small sample of twenty respondents?
We would be more confident that a serious mistake had been made in thepamphlet (rather than the sample) if a similar negative response (like misunder-standing) is obtained from very diverse , or very different kinds of membersfrom our intended audience. Using this same example, if the fifteen respondentswho could not understand the pamphlet represented different ages, differentlevels of education, different regional or ethnic subgroups, and so forth, thenwe would be more confident that the lack of understanding was due to a problem
with our pamphlet, and not due to some unusual characteristic of the sample.
On the other hand, if we notice that most of these fifteen respondents arefrom the same ethnic group, or they all speak the same dialect, or none of themare literate, then we should not automatically conclude that the problem of under-standing is due to the design of the pamphlet. The problem might be due to theunusually high proportion (15/20 = 75 percent) of this kind of respondent com-pared to the whole intended audience, where, let us say, there are only 20 percentof this kind of person.
80
To find out if the problem is just due to the small, distorted nature ofour sample, we would want to continue our pretest with a larger number of
respondents (increase the sample size), making sure that we are getting respon-dents which represent many of the other subgroups or types of people in thewhole audience. If the larger sample does include a greater, more proportionalrepresentation of the audience (more educational levels, the other important
ethnic groups, dialects, and so forth) and the majority still cannot understand
the pamphlet, then we would be more confident that the problem is in the com-munication being pretested and not the nature or size of the sample.
This simple example shows again the connection between the nature ofthe intended audience and the size of the sample that is required to make a good
decision. For pretesting communications it is very important to select a sam-
ple which represents most of the important types of people who exist in ourintended audience. In order to accomplish this, we may have to select a largersample. If there are many different types or subgroups of people in the popula-tion which will become our audience, then we will have to select a larger sampleto include some members from all of them.
It is possible in both probability sampling and quota sampling to have toomany people in the sample who would make the sample's characteristics differfrom that of the population that it is supposed to represent. This kind of sam -pling bias affects our pretest results much more when the sample size is very
small.
The implications for quota sampling should be obvious: Choose the char-acteristics of the audience for the quota which are likely to have the greatest
influence on their responses to the communications to be pretested. Then makesure that the proportion of these characteristics in the sample is very close tothe proportion that exists in the population. Education, ethnic group, sex, andso forth, often have a strong bearing on how people interpret, understand, andbehave towards communications.
"How big a sample should I have?" is very often the first question that isasked about pretesting. It should probably be the last question. As we have triedto show, the size of the sample should be governed by the answers to other ques-tions: the purpose of the pretest, the type of communications to be tested, theamount of variability or differences in the characteristics of the intended audience,and the amount of time and resources available. We need a good enough sampleand a large enough sample to tell us how well our communications will work, andwhat needs to be done to make them work better.
81
ST'ZMTITARV
It is worth repeating what we said in the introduction of thismodule. We can usually learn something useful from anyone to whomwe show our new communications. People in our office, our profes-sional colleagues, people off the street, and so forth, may all have
some useful ideas about our work. But if we want to know how theintended audience would react to our communications, then we needto find a sample of people who have characteristics that are repre-sentative of that audience.
The greater the number of ethnic, cultural, socioeconomic,and geographical subgroups that there are in the intended audience,the more people we will have to include in our sample to get an ideaof how these different kinds of people would respond to our new com-munications.
A small sample is inadequate if it leaves out important typesof audience members. A large sample will produce misleading re-sults if it contains these various kinds of people in proportions (per-centages) that are very different from the audience as a whole.
Quota sampling is useful if it is sufficient to obtain represen-tative proportions for just a few main characteristics, such as sex,
educational level, age, ethnic group, and so forth. If it is importantto get a representative sample according to a larger number or all ofthe audience's characteristics, then larger samples should be selectedwith probability sampling procedures. If very little is known about thecharacteristics of the intended audience, or if learning more about theaudience is one of the important purposes of the study in addition topretesting, then it would be worhwhile to select a larger probabilitysample.
And finally, it should be remembered that it is often possibleto use a combination of quota and probability sampling in the samestudy, depending upon the level at which choice is necessary--fromgeographical/cultural regions, villages, factories, and schools tohouseholds, people, time-segments, and so forth.
82
UNIT III
METHODS AND TECHNIQUES FOR PRETESTING COMMUNICATIONS
OBJECTIVES
After completing this unit you will be able to:
1. Describe the characteristics of the three main methodsused for pretesting communications: the panel method,the experimental method, and the survey method.
2. Describe and diagram the five basic steps of the exper-imental method.
3. List the three kinds of evidence provided by the exper-imental method that the effects observed in a pretestwere actually caused by the communication beingtested.
4. Explain why a smaller than usual sample size is
acceptable when the survey method is used to pretestnew communications.
5. Select several different sample sizes from 20 to 1000respondents, and then identify and explain the range of
"possible error" for a proportion of 75 percent (or50 percent) based on pretest information from a sample
survey of each size that is selected.
6. Describe ten different techniques used in pretesting toobserve or measure the effects of, or the audience's
reaction to, new communications.
84
INTRODUCTION
The methods and techniques that have been developed for pretesting com-munications or borrowed from other kinds of communication research arenumerous. They vary in many ways depending on the type of communicationbeing tested. Most of the techniques used today are simple and easy to use.
It is both fortunate and unfortunate that so many techniques are available.It is fortunate because the research team has more flexibility and the opportunityto select the most appropriate method and techniques to fit their particular pre-testing situation. It is unfortunate because the methods and techniques are notappropriate for all purposes, and some of them can easily be misused.
The most important advice that we can contribute to this problem is donot rely on just one pretesting technique . When possible use a variety of pre -testing techniques for each pretesting situation so that the results and the con-clusions are based on more than just one type of information . This will becomeclearer when the methods and techniques are discussed below. What this means,for example, is that rather than relying on respondents' answers to a writtenquestionnaire (or rating form), we would also observe their immediate reactionto the communication, and engage them in an informal discussion about it, eitheralone or in a group. We would place more confidence in the results of three ormore different kinds of information than we would in the results from just onekind.
In this unit we will present methods and techniques that can be used in avariety of situations. Because there are so many different kinds of pretestingmethods and techniques, it is beyond the scope of this module to go into greatdetail for each one. To learn how to use these methods and techniques requiresthat they be used in actual practice. They are not the kind of things that can bedescribed in an easy "cookbook" manner. They should not be lifted out as theyare and applied to any pretesting situation. Each one needs to be adapted andredesigned by the pretest team to fit each particular pretesting problem.
To use these methods well requires experience over several differentpretesting situations. This is no reason, however, to avoid trying new methodsor techniques. Each time we conduct a pretest we should consider experimentingwith new ones to improve our results. This can be done without too much risk
if at the same time we also use some of the standard methods and techniques withwhich we already have a lot of experience and confidence.
This module will provide three kinds of experience that will help theparticipant learn how to use these methods and techniques. When possible, wewill give an example of each technique as it is being described. Secondly, wehave collected a set of examples where some of the methods and techniques haveactually been used to pretest communications for family planning programs.
85
This "experience of others" may be found in the appendix at the end of the moduletext. And thirdly, the module participants are encouraged to try for themselvesone or more of these methods and techniques in an exercise with actual commu-nications. Existing materials, such as family planning posters, pamphlets,radio programs, and so forth, should be readily available in most places wherethis module is being used. If unavailable, communications from other develop-ment programs or even commercial advertising may be used. These materialscan be borrowed and then pretested again by the module participant to practiceand to gain experience with some of the methods and techniques discussed in thisunit.
METHODS FOR PRETESTING COMMUNICATIONS
In the discussion above we have used two different terms: method andtechnique . We are using both terms to emphasize the difference between theoverall approach or general design for pretesting and the specific proceduresthat are used to collect information within each general design. We willdescribe three general methods, three systematic procedures of inquiry thatare most commonly used for pretesting communications: the panel method , the
survey method , and the experimental method. Within each of these general
methods, it is possible to use a variety of specific techniques to collect informa-tion: observation, group discussion, interviews, questionnaires, and so forth.
The distinction that we make between method and technique is somewhatarbitrary. In practice the two terms are often used interchangeably. We areemphasizing the difference to encourage pretesting teams to think about andchoose their general method for pretesting before considering and designing
techniques to collect information. Pretesting techniques should be designed tofit the basic method rather than the other way around.
THE PANEL METHOD
A panel is a group of persons selected to investigate a problem, or toevaluate a matter and give a judgment. Because of its nature it is often referredto as the "jury" method. Sometimes a group of experts are selected to serve onthe panel, but this is not necessary. In many legal systems, members of a juryare selected to decide the outcome of a trial because they are considered to bethe "peers" of the person on trial. A peer is a person who is equal, who has thesame rank or abilities. It is very useful and appropriate to apply the same termsand general method to pretesting communications.
The panel method is the quickest and easiest method to use. A variety ofpanel groups can be used for the same pretest. For instance, we can recruit apanel of family planning experts from the field or a panel of experts in media
86
production. The latter panel would be equivalent to selecting some of the peersof the staff members who produced the communications we want to pretest. We
would ask them to judge the quality of work done by their peers. As we empha-sized in the unit on sampling, however, we would also want to form one or morepanels of typical audience members. In a sense, an audience panel would consist
of the peers of our target audience. We would ask them to judge the communica-tions that have been produced for people who are very similar to themselves.
There is no established number of members for the panel method, noris it necessary for the members of a panel to meet together as a group, face-to-face. Generally, a panel would consist of five to twenty-five, and occasionally,
as many as fifty persons.
Depending upon the nature of the communications to be pretested, we maywant to select a variety of panels consisting of the most important subgroups inour intended audience. For example, we could invite a group of postpartummothers in a maternity hospital to volunteer for a panel. At the same time, wecould form other panels of unmarried women working in a factory, young men
serving in the army, and so forth. It is obviously less expensive and less timeconsuming to test our communications-with these kinds of intact groups whosemembers live or work in the same place.
Each panel member can be asked to judge our communications by himself,separated from other members of the panel, either in his own home or at a com-mon location such as an office or studio. One of the advantages of the panelmethod, however, is to offer the members a chance to listen to each other's
opinions, to discuss the communication together, and reach a consensus or group
decision. In other words, the panel method consists of one or more groups ofexperts or peers of the intended audience which examine our communications forus, evaluate them, and judge their quality and appropriateness for the intendedaudience.
The panel method is a widely used method for pretesting communications.It is the simplest design, the easiest and least expensive to conduct. Thesefactors alone may explain why it is such a common design. Because a smallernumber of persons are selected, it is usually possible to spend more time witheach panel member, or the whole panel as a group. This makes in-depth inter-viewing techniques, as well as extended group discussions more feasible. If itis more important to get a lot of information from each pretest respondent, thenthe panel method is a very useful approach.
The type of sampling that is used to select respondents is independent ofthe overall pretesting method. Nevertheless, the small size of most panelsmakes random probability sampling impractical and unnecessary. Thus, with thepanel method we will not have a set of respondents (panelists) who are as repre-sentative of the rest of the intended audience as with a larger sample survey
87
based on a random probability sample. As we said earlier, we can improvethe representiveness of our pretesting results by combining the basic ideabehind quota sampling with the panel method. We can usually set up a varietyof panels (or groups) that represent the most important subgroups in ourintended audience. Some examples of this are (1) skilled and unskilled lowincome, urban males 35-50 years of age; (2) middle to upper class, marriedwomen 30-45 years of age; (3) rural, male farmers with primary education orless, and so forth. Because people with similar characteristics are more likelyto associate with one another or work in the same place, intact groups of peo
-llee with the same characteristics can usually be found to form panels. We caneasily find the men in the first example above in an urban factory, the womenin the second example in urban social organizations like the Y. W. C. A. or theparent-teacher associations of schools, and the men in the third example in oneor more rural villages.
THE EXPERIMENTAL METHOD
An experiment is simply a test or trial, in our case, of new communica-tions that we are developing. More precisely, an experiment is an operationthat is conducted under controlled conditions in order to discover an unknowneffect . When we produce new communications we can never be sure what theireffects will be until we have made some kind of test or trial run on members ofthe intended audience.
The distinctive feature of the panel method is the formation of a group ofexperts or audience members to judge new communications for us. With theexperimental method we actually t the new communications with a group ofaudience members and observe or measure the effects . In practice, however,it is often possible to use both methods with the same group of respondents (orpanelists). First we can run them through a trial exposure to the communica-tion, then ask them to judge its quality and appropriateness, and even the effectsthat they would expect it to have on their friends and other audience members"just like them."
As part of modern science, the experimental method has become muchmore than just a trial operation. It is a test under controlled conditions withmeasured effects. The conditions for the experiment are controlled to obtain
evidence that the operation (exposure to a communication) actually does producethe effect that is observed or measured.
To demonstrate a causal effect we must have evidence
1. That the exposure to the communication and the effects that areobserved are associated; that is, where one occurs the other occurs.
88
2. That the observed effect did not occur before, but rather after expo-
sure to the communication.
3. That other factors did not cause the effects which are observed.
The experimental design allows us to obtain these three types of evidence.The general outline of an experiment is simple enough: An experimental group
is exposed to the communication while a control group is not exposed, then the
two groups are compared in terms of the observed effect. The experimentitself shows whether or not the communication and the observed effects areassociated or not. The pretest team knows which persons (the experimentalgroup) have been exposed to the communication. They measure the degree towhich the effects occurred in the experimental group and in the control group.
If the measured effect is greater or occurs more frequently among members ofthe experimental group than among members of the control group, then we haveevidence that the effect is indeed associated with exposure to the communication.
We still must rule out the possibility that the observed difference in effectexisted before the experimental exposure to the communication. To be sure ofthis, we have to set up the experimental group and control group in a way thatpermits us to assume that their members do not differ beforehand in ways thatwill influence the expected effects. For example, suppose we intend to measurethe level of understanding of the "national population problem" after exposure toa communication about the same topic. We would want to assign persons to ourexperimental and control groups in such a manner that we can assume that eachgroup's level of understanding is approximately the same before exposure to ourcommunication.
One way to accomplish this is to assi gn persons to each group in a mannerthat does not give those who already have a high level of understanding a greaterchance of getting into the experimental group rather than the control group. Fromour discussion of random sampling in Unit II of this module, we know that randomprobability sampling would allow us to make this assumption. By taking a num-bered list of everyone and then selecting half to serve in the experimental groupby means of a table of random numbers, we can assume that persons with a higher
understanding do not have any greater chance of being assigned to the experimentalgroup than those with lower understanding.
With random sampling all persons have the same, equal chance of beingassigned to either the experimental or the control group. If there is still a dif-ference in the level of understanding between the two groups before the experi-ment then we know it is due to chance only (or random sampling) and not anyother factors or sources of selection bias. The control group is just as likely(equally probable) to be higher in their prior level of understanding as the experi-mental group.
89
The second way to solve this problem is to measure each person's levelon the intended effect before the experiment as well as after the experiment. Inother words, rather than assume that persons randomly assigned to the controland experimental groups are at about the same level before the experiment, we
actually measure the level beforehand. By measuring each group's level before-hand we can assess exactly how similar they are before exposure to the new com-munication. Then after the experiment we can compare the change in the level
of effect between the experimental and the control group.
If we measure the level of effect (say understanding or attitudes towardsfamily planning) after they have already been assigned to the experimental andcontrol groups then we would go ahead and compare their change in level of ef-fect after the experiment unless their initial difference was extremely large. Ifwe measure each person's level of effect before assignment to a group, then wecan use their scores for the assignment process. Simply rank each person inorder of their scores on the effect that was measured, and then assign every otherone (for example, odd versus even numbered persons) to one group and the rest
to another group. Then randomly decide which group will serve as the experi-mental group. Example 4 in the Appendix of this module shows the questionnaireitems used to measure the communication effects of a pamphlet before and afterexposure in a field experiment conducted by the Institute of Mass Communicationsin the Philippines.
If the pretest team decides to obtain measures before and after exposureto communication in an experimental design, then great care must be taken inthe construction of the questions or procedures for measurement. Taking awritten test before exposure may add to the effect caused by the communication.By taking a test before exposure the respondent in the experimental group maybecome more sensitive or receptive to the content of the communication. If itis a test of comprehension or understanding, the respondent may be looking in
the communication for the answers to questions he could not answer in the testbeforehand. Both the experimental and the control group would have an oppor-tunity to learn something from the measurement (before test) itself, but only theexperimental group can search the content of the communication for possibleanswers to use later.
If we expect the effects of our communication to be strongly influencedby measurement before the experiment, then it would be better to drop the
measurement beforehand. Learning from the measurement itself can be con-trolled by constructing two different forms of a test to measure the same effect.Instead of constructing ten questions to measure the respondents' attitudes, forexample, we could construct a test of twenty related questions which measurethe same set of attitudes, and then use ten in the measurement beforehand, andthe other ten questions in the measurement after the experiment. This way therespondent is not able to use the information from the test beforehand to improvehis scores on the second test.
90
A third type of evidence is required to rule out other factors that may
possibly have caused the effects that are observed from the experiment. Char-acteristics of the persons in the experiment, such as level of education, occupa-
tion, age, prior exposure to communication, and so forth, that were determinedin the past can have an enduring effect on their present behavior, includingresponses to new communications. Most of these factors can be controlled bymeans of the random assignment of persons to the experimental and controlgroups as described above, and by checking these background characteristics
beforehand to make sure that the two groups of respondents do not differ too
greatly.
Events other than exposure to communication during the experiment mayalso influence the effects that are observed. If different times or different daysare used for the experimental and control groups, then we have to watch for theeffects of outside events which may affect one group but not the other. Anational leader, for example, may make the newspaper headlines with unexpectedsupport for legalized abortion, or perhaps public support for families like hisown with seven children. Such an event could affect one group's attitudes towardsfamily planning but not the other.
The way that instructions are read or given to the experimental and thecontrol group should be controlled so that they are as similar as possible. Some-times the person who conducts the experiment influences the respondents' answersto questions, or interest and attention to the communication. Make sure he treats
both groups the same way, or better yet, don't let him know which group is theexperimental or control group. If a thirty minute film about population is beingshown to the experimental group, it may be better to have the control group see afilm too (say a neutral film about current events or rice production) rather thanhave them get impatient waiting, or spend the time talking to one another. Insummary, the conditions before and during the experiment should be controlledas well as possible to rule out the possible effects of factors other than exposureto the experimental communication.
The main characteristic of the experimental method is simple enough: Theactual trial or test of new communications under controlled conditions with somekind of observation or measurement of the effects. As the discussion above sug-gests, however, obtaining the required degree of control and getting an unbiasedmeasure of the effects is not as simple as it looks at first. The flow chart on thenext page summarizes the five basic steps in the experimental method.
The flow chart shows the basic steps of the experimental design. Thereare several variations of this design that are useful for pretesting communica-tions. We have already mentioned the advantages and problems of obtaining mea-surements before the experimental exposure to the communication as well asafterwards. It is also worthwhile to set up more than one experimental group.
91,
4. 5.
Measurementor observationof communica-tion effects
Assignment bya table of ran-dom numbers
Comparisons are made of
the experimental and control
groups' measured effectsand the differences arereported+
The population or Selection of aintended audience sample of audi-of the communica- _1 ence memberstion being developed for the experi-
ment*
FLOW CHART SUMMARIZING THE EXPERIMENTAL METHOD
1. 2. 3.
Experimentalgroup to be ex-posed to the newcommunication
Control group(1) not exposedor (2) exposed
to other com-munication
Measurement
or observationof communica-
tion effects
*The preferred method of sampling from the audience is random probability sampling, but if this is not feasible, thena good quota sample should be selected. Random probability methods should always be used to assign members tothe experimental and control groups.
+Large differences in effects are easy to interpret and report. If the difference is relatively small, then statisticaltests can be done to make sure the difference is significantly larger than what might be expected to occur by chancefactors alone, such as the way respondents are randomly assigned to each group.
Once the funds and time have been spent to sample respondents and
assign them to groups for the experiment, it is often more productive to pretestseveral versions of the communication that is being developed. If alternativeversions of a new communication have been developed, it is a simple matter torandomly assign persons to two, three, or more experimental groups in addi-tion to the control group. In this situation, each experimental group would beexposed to an alternative version of the communication. This would allow thepretest team to compare the level of effects in different experimental "treat-ments" as well as the level of effects in the control group. *
THE SURVEY METHOD
In general, a survey is simply a collection of information about a topic
or area of study. This is the way that the term is most commonly used. Whensomeone ways that they want to "survey the situation" before making any deci-sions, they are merely talking about the collection of some information "outthere" about their problem. We survey a small number of people from anaudience in order to learn something about the whole audience. Social scien-tists use the term more specifically to mean a study of a group of units orobjects (such as people), for the purpose of drawing conclusions about the largerpopulation that this group comes from. In these terms, the survey method isvery similar to the concept of sampling that we discussed in the second unit ofthis module.
In the social sciences, the term survey and sample survey are oftenused interchangeably. A sample survey is the collection and analysis of statis-tical data on some aspect of a group to draw conclusions about the larger aggre-gate from which the group is selected (sampled).
The logic of the survey method can be found in our unit on sampling pre-test respondents. We want to test or expose new communications to a sampleof people who are as representative of our intended audience as possible. Forpretesting purposes, we would want to "survey" the responses to our new com-munication, and then use the answers we obtain from our sample of the audienceto draw conclusions about the audience as a whole. Because this is the mainpurpose of the survey method, sampling methods that select the most represen-tative samples are preferred. Hence, most discussion of sample surveymethods assume some form of random probability sampling will be used to
* For a complete discussion of variations on the experimental design, seechapter six of E.A. Suchman's Evaluative Research (New York: RussellSage, 1967).
93
select a relatively large group for study. Most researchers who use surveyresearch methods to draw conclusions about national populations prefer to studysamples that range from a minimum of around 400 to 600 people up to 10, 000people.
Because sample surveys are usually used to study groups that number inthe hundreds or thousands, many researchers rule out too quickly the possibility
of using sample survey methods for pretesting new communications. With onlyenough time and money to study groups of twenty to fifty people, it hardly seemsnecessary, let alone practical, to draw a random probability sample from theintended audience. It is much more practical to use the panel method for pre-testing, selecting either a small group of expert judges, or else a set of panels"similar" to the audience by means of quota sampling procedures.
The size of the sample, however, does not by itself define a study as a
survey. This is going about the problem backwards. The purpose of the study
should determine the method to be used. If the survey method is chosen, then
the purpose of the survey should be used to determine the proper sampling pro-cedures and sample size. When our purpose is to pretest a new communication,it is possible to use a much smaller probability sample.
In probability sampling the size of the sample determines how much errorthere will be--how far wrong you will be- -in drawing conclusions about the whole
population from information (statistics) obtained from that sample. A largersample will allow you to make more precise estimates with less error about the
whole audience.
This idea can be seen very clearly in the following table. It shows therange of possible error that could exist for a sample proportion of 50 percentobtained by means of a simple probability sample survey with sample sizesranging from 20 to 9, 604.
Table 1 shows exactly how much we can reduce the osp sible range of
errors of a proportion based on a probability sample by increasing the size ofthe sample (surveying more people). Suppose, for example, we have pretesteda new pamphlet with a sample survey of 60 members of our intended audience.If our sample findings show that 50 percent of those interviewed reject thepamphlet or fail to understand the main elements of the pamphlet, then we wouldconclude that the proportion in the whole audience from which our sample wasselected would also be around 50 percent.
The population's proportion will not be exactly the same as our sampleproportion because of differences due to our use of probability sampling (that is,the random selection of only 60 people from the audience for our sample). Inmost samples (95 out of 100 samples) of this size, however, we can be sure thatthe actual proportion for the whole audience is either plus or minus 12. 8 percent
94
Table 1. The Range of Possible Error from SimpleProbability Sampling at Various Sample
Sizes n
Sample Proportion = . 50
Population's
Sample Lower Higher Range of
Size (n) Limit Limit Tolerated Error
(percentage)
20 . 275 . 725 +22.525 . 300 .700 ±20.0
30 .318 . 682 + 18.2
40 . 343 . 657 ±15.7
60 .372 . 628 ±12.880 . 390 . 610 + 11.5
100 . 402 . 598 ± 9.9200 . 431 . 569 + 7.0400 . 451 . 549 + 4.9
1000 . 469 . 531 t 3.12000 . 480 . 520 ± 2.0
9604 . 490 .510 + 1.0
* Where the confidence limit is .95; for 95 samples out of 100 the proportion(%) in the whole population should fall within the range of error shown in thetable for each sample size.
from our sample finding of 50 percent. In other words, we can be confident thatthe proportion of the whole audience who would reject or fail to understand ourpamphlet would be somewhere between 37. 2 percent and 62. 8 percent. This isshown in the table above for a sample of 60. With a small probability sample of60 persons we have to be willing to tolerate a possible error of 12.8 percent oneither side of a sample proportion, which in this example is 50 percent. Is thisrange of possible error too big for purposes of pretesting new communications?
Many survey researchers would automatically say that a possible errorof + 12.8 percent is much too large an error to tolerate. They would not say itin these terms, however. They would be more likely to reject a sample surveyof only 60 people because it is just too small to draw reliable conclusions.Nevertheless, we should never say that a sample size is too small or too large
95
in general . The appropriate size of any sample depends upon the specific pur-pose of the survey. Once again, in our case the question should be, Is a possibleerror of +12. 8 percent too big for purposes of pretesting new communications ?If we were considering an even smaller sample of say 30 people, we would askwhether +18. 2 percent is too big for our pretesting problem.
Since the amount of money available for pretesting new communicationsis usually limited, it is not very practical to consider selecting large samples.In most situations, this means that the smallest amount of error we could hopefor would still be around ±10 percent (or ±9.9 percent for a sample of 100). Tosay whether or not this much error is too big requires that we review the og alsthat we originally expected our communications to achieve.
In Unit I of this module, we recommended that pretesting should betreated as a part of the overall process of planning and designing communicationfor planned change. Along with the specific goals and objectives for which eachnew communication is designed, there should also be some criterion or standardof goal achievement for the intended audience. These are often stated in termsof the proportion of audience members for whom the effect is expected to beachieved: "Seventy-five percent of those exposed to the pamphlet will pass a testwhich demonstrates that they understand its basic concepts.' This particulargoal can also be stated in a negative manner: 'Only 25 percent of those exposedwill fail to pass a test of basic understanding of its contents."
The range of error from a sample proportion is greatest when the pro-portion is in the middle, at 50 percent. The error is somewhat smaller thecloser the proportion is to 0. 0 percent or 100 percent. A proportion of 75 per-
cent, for example, obtained from a probability sample of 60 people would havea range of error from -13 percent to +10 percent. Thus, we could be sure that
the actual proportion in the audience that this sample was selected from would
be between 62 percent and 85 percent. In other words, if we find that 75 per-cent of a sample survey of 60 people pass a test showing understanding of ourpamphlet's contents, then we could conclude with confidence that somewherebetween 62 percent to 85 percent of the whole audience would probably pass thesame test.
The main purpose of pretesting is to decide whether a new communica-tion is effective enough with a pretest group to go ahead and use it with the whole
audience. In the example above, we said that we achieved our stated goat of 75percent with the sample selected for the survey; only 25 percent failed the testabout the pamphlet's contents. Although the failure range for the whole audienceis from 15 percent to 38 percent, we can still be sure that at least 62 percentwould pass the test in the intended audience. Sixty-two percent is the worst out-come that is likely, and the outcome is more likely to fall closer to the samplefinding of 75 percent than to the maximum or minimum for the range of error.
96
If we are willing to tolerate a possible success rate of only 62 percentof the audience, then we can go ahead and conclude that the new communicationwould be effective enough for our original purposes. If this potential rate ofsuccess is too low, then we would have to improve the communication until weachieved a success rate higher than 75 percent in another sample survey of 60,or else pretest it with a much larger sample to reduce the possible range oferror until it is acceptable.
Table 2. The Range of Possible Error from SimpleProbability Sampling at Various SampleSizes (n)*
Sample Proportion = . 75
Population's
Sample Lower Higher Range of
Size (n) Limit Limit Tolerated Error(percentage)
20 . 50 . 91 . 4125 .54 . 90 . 36
30 . 56 .89 . 3340 .58 . 87 . 2960 . 62 . 85 . 23
80 . 64 . 84 . 20
100 . 65 . 83 . 18200 . 68 . 81 . 13400 . 71 .79 . 08
1000 . 73 .78 . 05
* Where the confidence limit is .95; for 95 out of 100 samples the proportion (%)in the whole population should fall within the range of error shown in the tablefor each sample size.
There are two very important things to consider at this point. First,even for a sample of 400 the range of possible error for a proportion of . 50 isstill +4. 9 percent. A survey result of . 75 obtained from a sample of 400 has arange of error that would make the proportion in the audience fall between 71percent to 79 percent. Is it worth the time and resources to reduce the rangeof possible error of the population's proportion from 62 percent to 85 percent
97
(n=60) down to between 71 percent to 79 percent (n=400) when the sample propor-
tion falls at 75 percent?
Remember, we do not really care if the audience proportion is actuallyhigher, say 85 percent rather than 75 percent. If the error from the sample ison the positive side, the communication we are pretesting is not only good enoughaccording to our original goal, it is much better. Therefore, we are really onlyconcerned about half the range of error, error which indicates that the proportionin the audience is really much worse than in our sample. Increasing the samplesize from 60 to 400 only raised the lower limit of error from 62 percent up to70 percent. An improvement of only 8 percent is probably not worth the extracost: If 62 percent is the worst likely proportion of success when the goal is 75percent, then it is probably good enough. Notice that the range of error arounda sample proportion of . 50 when the size of the sample is as high as 1, 000 isstill +3. 1 percent.
This same kind of logic can be used to consider a survey with a smallersample size, or a much larger sample size. A sample of 30, for example,would have a potential range of error for a proportion of . 75 from . 56 to . 89 inthe audience. This would mean that in the worst likely case we could expect,56 percent of the audience would pass our pretest. Once again, this is theworst likely outcome with the whole audience; the outcome will probably be muchbetter than 56 percent. And the error in our small sample may also be on thepositive side: Our new communication may be effective with 89 percent in theaudience compared to 75 percent in our sample of 30. In other words, the pro-portion in the whole audience has a much greater probability of being close tothe sample proportion than it does of being out near the edges or limits in thepotential range of error.
A more complete discussion of small sample surveys may be found in arecent article by K. Finsterbusch entitled "Demonstrating the Value of MiniSurveys in Social Research" (Sociological Methods & Research , vol. 5, no. 1,1976, pp. 117-136). After several tests of various sample sizes he recommendsthat:
. . . a mini survey of 20 respondents is better than nothing, but evenwhen the budget is tight, an additional 20 interviews should be obtainedbecause they add so much to the accuracy of the results. Interviewing40 respondents is sufficiently inexpensive for the tightest budgets andsufficiently accurate for many descriptive studies (p. 129).
A small sample survey is a quick and inexpensive way to check on thejudgments of a group of experts who are selected for the panel method of pre-
testing communication. Experts' judgments may appear to be very reasonableand consistent with our own opinions, but they are frequently very wide of themark. The error in their judgments may be much greater than the error that
98
exists in a small sample survey. If the budget allows it, every effort should bemade to compare the results from a panel of experts with the results from asmall sample survey of the intended audience.
CAN ANY ERROR BE TOLERATED ?
Can communication specialists and pretesting teams tolerate any errorwhen they have to decide whether or not to use new communication with thewhole audience? In most situations the answer is yes. In fact, we can toleratequite a lot of error compared to other kinds of researchers who use the samplesurvey method.
The reason is very obvious when we reconsider our up rpose for develop-ing new communications in the first place. Look back at the audience analysisthat was done before we designed the communication. What proportion of our
intended audience already understands the content we have planned for the newcommunication (the pamphlet)? If we lain with a situation where only 15 per-cent of the intended audience know what we want to communicate beforehand,then we would be pleased to see that proportion increase to 56 percent to 89 per-cent among those who are exposed to our new communication.
In most situations we do not plan new communications for the whole pop-ulation of a country, but rather a subpopulation, or target audience, that needsour information the most. Furthermore, we usually do not design new commu-nication "to gain more approval for family planning contraceptives" among sub-groups in our population when 80 percent to 90 percent already approve. In theplanning process we should spend a lot of time narrowing down our intendedaudience to those who have a much lower rate of approval, perhaps because theyhave never received the right kind of information about family planning contracep-tives. Therefore, when the success rate in the intended audience (proportion ofapproval or understanding) is very low to begin with, we can tolerate what seemsto be a wide range of error around an outcome we get from pretesting with asample survey. Increasing the approval of vasectomies, for example, from10 percent of our intended audience to a likely minimum of 56 percent amongthose exposed to our new communication about vasectomies would be considereda success by almost anyone.
This is a rather long way to say that the purpose for designing new com-munications and the purpose for pretesting should determine what method of pre-testing we use, and what size sample we think is appropriate for a pretest sur-vey. The norms or standards for acceptable sample survey sizes used by otherkinds of researchers should not discourage us from using sample surveys forpretesting purposes. When demographers, for example, are trying to obtain anestimate of the population's growth rate they cannot tolerate very much errorfrom the growth rate they find in the sample that they have surveyed. If the
99
growth rate in the population is around 3 percent, they cannot use a sample
estimate that may have a range of error as great as 5 percent on either side of3 percent. This kind of finding would be worthless for their purposes. Even1. 0 percent error is too large. This is why they must work with very largesamples or complete census data. If a demographic estimate is off by even asmall percent, the difference in the estimated size of the future population wouldbe quite large, and lead to all kinds of mistakes in national policy and decisionmaking about the future.
When communication is being designed to increase approval for familyplanning or to increase the proportion of the population who understands theimportance of the population problem, we can often afford quite a range of erroraround a finding from a survey pretest without the risk of any serious conse-quences later. If somewhat fewer people approve than we expected, at least noone will be seriously hurt as a result. We may not be getting as much benefitor success as expected from the money we have spent to develop and producethe new communication, but the amount of money is still very small comparedto, for example, the amount of money that might be invested to build a nationaleducational system to meet the future needs of the children which are predictedfrom a demographic survey (or census) but who are never born.
There are times, however, when the negative consequences from errorin our communication does have serious consequences. When we are communi-cating about how to use oral pills safely, or about the possible dangers that mayresult from abortions or vasectomies, then the negative consequences could bevery serious indeed. We can joke about the man who misunderstands a lecture
about how to use condoms, and who tells us later that his wife got pregnant eventhough he had the condom on his finger "just like you showed me." It Is not ajoke to him or his family. It is not a joke for women who develop serious sideeffects from oral pills or the I. U. D. because the instructions were so poorlycommunicated that they did not use them correctly.
It should be clear by now: The purpose of our communication determineshow much error we can afford to accept. If there is a possibility of injury oreven death from a poorly designed communication, then we may not want to toler-ate any error . We may insist that the message be so good that anyone (100 per-cent) can understand it. In most cases 0 percent error in our communication isnot necessary, and in fact quite a bit of error can be tolerated. Otherwise wewould probably not communicate very often: Error is an inherent part of infor-mation and the communication process in general. For a better understandingof the nature of error and uncertainty in the communication process see D. L.Kincaid and W. Schramm's module, Fundamental Human Communication , espe-cially the discussion of the convergence model of communication.
100
TECHNIQUES FOR PRETESTING COMMUNICATIONS
Once the general method of pretesting is selected--panel, experiment,or survey--the next step is to design the techniques that will be used to collectinformation from the pretest. In most pretesting situations we would use morethan one technique so that we can compare the information obtained in differentways. To pretest a poster, for example, we might organize an experimental
test of several different versions of the poster, and then use a combination ofdirect observation, a questionnaire, and unstructured interviewing techniquesto obtain information about the effectiveness of each version.
1. Observation of typical audience members as they view or listen toour new communication is a simple and often efficient way to gather useful in-formation. Several versions of a poster, for example, can easily be placed inactual locations where people frequently pass by. Trained observers may bestationed nearby to observe and record what happens when people walk past.How many stop to read the poster? How long do those who stop actually spendlooking at the poster? What are their reactions--thoughtful, amused, angry,puzzled, or simply bored ? Do people appear to talk to other viewers about theposter? What do they say?
One of the advantages of this kind of observation is that the observersmay remain hidden, or at least appear unassociated with the poster. This allowsfor a very natural pretest of the communication under actual field conditionswithout the interference of an interviewer. If desirable, however, one observermay interview every fifth to tenth viewer afterwards to gather more informationwhile others continue recording observations. If the interview is done out ofsight of the poster, it would be good to ask the viewer what he saw and how muchhe remembers about the poster. Does he understand it? Agree with the mes-sage? Does he plan to do anything about it? What?
Observation in clinic settings may also be convenient. Audience reac-tions to slide-tape shows, lectures, flipchart demonstrations, films, and soforth, may also be obtained through simple observation. If several differenttrials, or groups, are shown the product, then the observers should have anobservation sheet that instructs them to look for certain things and that leavesspace for them to record their observations easily. This will increase thereliability of their observations and allow you to compare results from oneobservation with another. Space should also be provided for observers to re-cord 'other" things, or unusual reactions that seem important to report.
If is fairly easy to design similar types of observation sheets for otherkinds of observation or to record more detailed observations. For recording
101
Person no.
Passed byGlancedReadDiscussed
Person no.
Passed byGlancedReadDiscussed
OBSERVATION SHEET
1. Content: poster no. 11
2. Place and Time Interval: Clinic B, 9-10 a.m. (i hr.), 10/22/76
3. How many people passed by the poster or sat in the room where it waslocated ?
4. How many glanced at it?
5. How many stopped to read the poster?
6. How many appeared to discuss the poster with someone near them?
1 2.3 4 5 6 7 8 9 10 11 12 13 14
3 r 3 3 3 r 3 r 3 r i r 3 3
15 16 17 18 19 20 21 .. _n Totals
r r 3 r 3 r 3 21
3 r 3 r 3 3 3 15
3 3 3 3 83 3 5
7. Percent of those who passed by during the hour
(a) who glanced at the poster = 15/21 = 71%(b) who read the poster = 8/21 = 38%(G) who discussed the poster = 5/21 = 24%
8. Most frequent comments about the poster:
102
frequencies people like to use slash marks that can be grouped into units of five:,f ?5W /// = 13. In some countries It is common to count with marks that form
the four sides of a square, and hence group the frequencies into units of four:[] [] U = 11. Whichever technique is used should be tested beforehand to makesure the observers can use it without too much trouble or too many mistakes.
2. Unstructured interviews may provide many unexpected but valuableresponses from members of our expected audience. Instead of seeking answersto specific questions, the respondent is shown the communication and then
encouraged to talk freely about it. The interviewer stimulates discussion withoccasional probes for additional information about certain things that are men-tioned. As the interview progresses, however, the interviewer lets the respon-dent determine which features of the product are worthy of comment or elabora-tion. Such interviewing is time consuming and requires very competent inter-viewers capable of creating good rapport with respondents. The completeanswers are difficult to record by hand, but a tape recording can be made forplayback later. The results are often difficult to interpret and to compareacross respondents, especially if they all talk about different aspects of the com-munication. A few in-depth interviews with very typical audience members canprovide a lot of insight for the production staff. They may want to read a verba-
tim transcript of the interview or even listen to a tape recording of it. Theresults from a few in-depth interviews are also useful for designing question-naires or interview schedules with more specific questions to use with a largernumber of respondents later.
3. Group discussion is often used on the assumption that people talkmore freely in a group of peers. Furthermore, they can react not only to theproduct, but also to the comments of others. Group security can encouragesome to reveal more private, personal opinions. The opposite can also resultdepending on the nature of the group, the discussion leaders comments, and soforth. Some group members may dominate the discussion and inhibit freeresponses from more quiet members.
One of the main advantages of group discussion is that it allows us tocollect the responses of a lot of people at the same time. Thus, much informa-tion can be collected very quickly. The utility of the information depends uponthe skill the discussion leader has in keeping the group on the relevant aspectsof the product. Recording so much information fast enough is often a problemwhen tape recorders are not used. An efficient solution would be to have oneperson take notes, then summarize the main conclusions of the group beforeleaving in order to give them a chance to correct any misinterpretations, or toadd further comments. As with unstructured interviews, unless discussionguides are followed, it may be difficult to compare the results of one group withothers.
103
4. Questionnaires : Writing a set of good questions is not easy. If wedo not ask the questions in the right way we will not get the kind of answers thatwe want. If the question does not have the same meaning for the respondent asit does for us, the questionnaire will not serve our purpose. Often questionshave no meaning at all for respondents. Hence it has always been suggestedthat the questionnaire must be pretested, maybe several times, before it isfinally administered.
There are four important aspects in developing a questionnaire that areworth considering.
a. Subject Matter : We must be very, very clear about what we wantto know from our respondents before writing the questionnaire. Dowe want to know facts like the age or sex of the respondents, or dowe want to know their opinions and attitudes toward the communica-tion? Or do we want to know how much knowledge they have obtainedfrom our communication? For example, asking "How many childrenwould you like to have?" is asking for an attitude or opinion, asopposed to asking "How many children do you have?" which asks fora fact. Similarly, asking, "What contraceptive methods are avail-able in your area?" is a question about their level of knowledge.Asking about behavior is difficult, as people do not always behavethe way they say they do. In many cases we may ask "Do you prac-
tice family planning?" and you get the answer "Yes, I do." Thisanswer may be merely out of courtesy and politeness. But, it isdifficult to be sure if your respondent really uses a family planningcontraceptive. By formulating our questions carefully and testingthem beforehand, reliable information can be gathered with thistechnique.
b. Structuring Questions : Questions can be structured or unstructured.Structured questions force the respondent to give one of severalpossible alternatives, such as "yes" or "no." In the case of unstruc-tured questions (sometimes called open-ended questions), the respon-dent is free to answer the way that he or she wants. Both kinds ofquestions have advantages and disadvantages. In structured ques-tions, the researcher knows in advance what kind of answers he or sheexpects from the respondents, and thus puts restrictions on the re-spondents' answers. Structured questions makes it easier for theresearcher to record answers and draw conclusions later:
Structured Do you currently use some method of family plan-Question: ning contraception?
noyes
104
Structured What contraceptive method do you currently use?Question: oral pills
loop or I. Ti. D.condomsmale/female sterilization
others
Unstructured What do you think are the main benefits from prac-Question: ticing family planning contraception?
Un structured What would you tell a friend of yours about theQuestion: family planning method that you use?
c. Question Wording: It is not only important what is said in the ques-tionnaire. It also is as important how the questions are formulatedand written. Only good questions can be expected to get good an-swers. A good question is the one that is understood and perceivedby the respondent as intended. Therefore, pretesting the question-
naire (maybe more than once) is a useful way of improving the ques-tionnaire.
It is very useful to use only those words which are generally used bythe respondents in their daily lives. Avoiding ambiguous and longwords in your questions make the questionnaire more useful.
d. Qiestlon Sequence: This refers to the order in which you ask thequestions. The sequence of questions is so important that all theanswers will depend upon the sequence in which they are asked.Fbr example, asking your respondent "Is family planning 'good' or'bad' ?" and then asking "Do you practice family planning?" canresult in different answers than asking the same questions in reverseorder. A general rule in sequence of questions is start with unstruc-tured and general questions and then gradually work up to the struc-tured and specific details of the subject. This is sometimes calledthe "funnel" approach to questions. Start with questions dealing withoverall reactions of the respondents, and then ask questions whichdeal with the various components of the product specifically. First,general questions, such as "What do you understand from this pos-ter?" or "Do you like this poster?" are asked. Then, ask more andspecific questions, depending upon the objectives of the pretest.These specific questions may deal with the colors in the poster, levelof language, size of the text in the poster or even the people, thefeatures of the people, the background in the illustration, and so forth.
105
Difficult Level
1. Do you think that a highfertility rate is not goodfor your family?
2. Which method would you 2.prefer to limit permanentlyyour family's size?
a) Vasectomyb) Tubectomy
AN EXAMP LE OF QUESTION WORDING
Easy Level
1. Do you think that too manychildren in your family wouldaffect your ability to provideenough food and clothing?
There are two permanentmethods of family planning(or birth control), one formen and another for women.They both require a smalloperation. Which one wouldyou prefer?For men For women
3. Do you like the color schemein this "mock up" for a pos-ter?
4. How do you like the sloganin this radio spot ?
5. Do you like the jingle in thisradio feature?
3. Do you like the colors in therough sketch for a poster?
4. How do you like the mes-sage in this radio program?
5. Do you like the song in thisradio program?
6. Which artists and characters 6. Which voices and people inappealed to you most in this this radio program did youradio drama? like most?
7. Which episode in this movie 7. Which part of the story inwas appealing to you? this movie did you like most?
106
AN EXAMPLE OF QUESTION SEQUENCING
1. What is your overall opinion about this poster?
2. What is the most important message in the poster?
3. What does this message mean to you?
4. Do you agree with the message in the poster?yesno
5. Which words in the message are difficult to understand?
6. Have you ever seen a style of writing like this before?yesno
7. Is the writing style easy to read from a distance?
yesno
107
5. Recall Tests : This method is used in two general ways. The two ways
are (a) dummy magazine and (b) portifolio. In dummy magazine tests, the sam-ple audience is given a dummy magazine with standard editorial content, butcommunication messages that are to be pretested are inserted along with otherstandard messages. These magazines are distributed to the sample audience.Audiences are asked to read them as they read ordinary magazines. Immedi-ately after reading, or after a couple of days, the interviewer returns again andthe respondents are asked to recall the message. The question may deal onlywith the interest created by the message. Fbr example, the interviewer willask what messages in that magazine the respondent can recall. Or, going fur-ther, he may ask what the various messages were trying to communicate. Theanswers can be further probed to find out the impact of the message in terms ofthe objectives set for the pretesting.
In portfolio tests, special portfolios or folders are prepared containing dif-ferent versions of the message being pretested. Different portfolios are givento different groups or audiences. They are asked to look through them and readwhatever interests them. After this, the folders are taken back from the respond-
ents and the interviewers start asking questions already prepared in accordancewith the objectives set for the pretesting of the particular message, which maybe dealing with perception, interest, clarity, credibility, and so forth.
Aided and unaided questions : In the above techniques two kinds of ques-tions are used. These are aided and unaided. In the case of unaided questions,the respondents are not given any cues on the message being pretested; ratherthey are asked straightforward questions, such as "What messages did you seeor read in the magazine/portfolio?" This kind of questioning makes the respon-dent quite uneasy because he has to recall completely on his own the answer tothe question. He is likely, to such cases, to be unwilling to answer the questionand may, just out of courtesy and politeness give us very irrelevant answers.
Obviously this kind of answer will not give us reliable information that can heused to improve our material. Because of these difficulties, unaided questionsare not a very popular kind of recall test. However, under certain circumstancesand for certain objectives, this may be a useful way of questioning. An exampleof such a case would be a test to determine if the message has received any atten-tion in the audience.
In the case of aided questions, there are numerous ways of getting answers
that will give us useful information. The interviewers can provide some cues tothe respondents to assist them in recalling what they saw in the magazine or
folder. Such questions can be asked in this way: "Have you seen any messagesin the mafazine/folder that deal with having a happier life?" "What do you remem-
ber about this message?" In some cases, the respondents are given a part ofthe message being pretested on a sheet of paper (for example, one word from aslogan, or just the sound effects of an audio message) that provide them withcues to recall the message. Then further probing questions are asked.
108
6. Awareness and Knowledge Tests : Awareness and knowledge about
the contents of our message among our audience is an important step before themessage can result in a concrete action. Therefore it is very necessary tomeasure these aspects of our messages before using them on a mass scale.
Pretesting can help us learn the level of awareness and knowledge of what weare trying to communicate with our audience. These tests are administered thesame way as those in the recall tests except that the questionnaires are devel-oped to test the awareness and knowledge of the target audience.
The respondents are exposed to the messages being pretested and then askedvarious questions such as "Have you heard that too many children create economicburdens for the family?" "Where can you get help if you decide to practice familyplanning?" "Is there a field worker who comes here every month to distribute con-traceptives?" These questions can be asked as open-ended, multichoice, fill-in-the-blank questions.
The information thus obtained about the level of awareness and knowledge ofour audience will help us improve our communication material and devise new mes-sages because we will know more about the audience in terms of where they stand onawareness and knowledge about specific aspects of family planning.
7. Indirect Questioning : In the previously discussed pretesting techniques,the respondent is asked directly to reveal how he or she perceives or evaluates aparticular message. Sometimes certain respondents may try to please the investi-gators by giving them the answers that they think are expected by the investigators.Others may attempt to give the reactions they think an expert in art or theaterwould give rather than their own responses. And finally, some respondents may beafraid to reveal their own private feelings, especially if they are related to verycentral, sensitive areas as sex, children, and family planning sometimes are.
To overcome some of the disadvantages, it is useful to use indirect techniquesalong with the more direct approaches described above. Rather than directly askingpeople to reveal their own reactions, the investigator might ask them how they thinkother people in their village or neighborhood would react to a product. Since theymust use their own judgments to a certain extent to estimate how others "like them"would react to a message, their answers may be used to infer something about theirown responses.
A projective technique is one that presents the respondent with an ambig-uous stimulus that requires her or him to use her/his own feelings, past experi-ences or expectations, motivations, and thoughts to interpret it. The informationthat is obtained may tell us much about the stimulus; the respondent's own thoughts,feelings, and values; and finally, the kind of thoughts, feelings, and values that thestimulus is capable of eliciting in a variety of people. Analysis of rht results ofprojective techniques is often very subjective and inferences should be madewith much caution. The assessment of the personalities of the respondentsshould only be attempted by a psychologist who is well trained in the analysis
109
AN EXAMPLE OF KNOWLEDGEAND AWARENESS TESTING
1. What are the main economic burdens that results from havingtoo many children in your family?
2. What kind of government service has been launched as a nation-wide family planning program to give services to those who
decide to practice family planning?
3. Can you name the most important modern methods of contracep-tives available?
Which ones are the cheapest and easiest to use?
4. How can you get family planning services in your areas?
5. The current population growth rate for our country is 34 perthousand. Do you know what goal our government has set in
order to reduce this growth rate? How many ? per
thousand?
110
should only be attempted by a psychologist who is well trained in the analysis of
projective tests. Nevertheless, analysis and careful inference can provide val-
uable information about certain kinds of communication products.
Projective techniques are very useful for assessing the nature of draw-ings and photographs that are designed to communicate certain kinds of feelingsand associations to complement a verbal message. It might also provide valu-able insight to stop a radio drama, a television program or a film at a pointwhere the main characters are about to make an important decision, or take anew course of action, and ask the audience to describe what these charactersare thinking and feeling, and what they think they will do next. Their answerscan then be used to assess how well these characters are developed, how wellthe story themes have been understood up to that point, how much membersof the audience identify with the main characters, and finally, what membersof the audience would think, feel, or do in the situation that is described.
The instructions for projective tests are very simple. Each respondentis asked to make up a story about a drawing or photo, or to finish a story devel-oped in a radio program. After seeing the picture he is asked to describe whatis going on in the scene, what the people are doing and feeling, what led up tothis scene, and what will happen next, or what will be the outcome. Used withthe familiar posters of the happy small family and unhappy large family, thesetechniques would at least give you some idea whether they really appear happyand sad to the respondent, and whether the respondent identifies with the char-acters to the extent he can empathize how they might be thinking and feeling.Problems often arise with respondents who just cannot make up verbal storiesabout pictures, and with stories that are so ambiguous that they cannot be inter-
preted.
8. Ranking Techniques : When more than two messages are to be testedto find out which one is the best to accomplish the objectives, it is desirable tohave the respondents select and grade the messages. There are two techniquesused in this case. One is the order-of-merit technique and the other is thepaired comparison technique. These are discussed separately.
a. Order of :Merit : In this method the respondents are given, or ex-posed to, a number of messages or materials and asked to arrangethem in rank order according to various criteria such as whichversion is (a) liked best, (b) most interesting, (c) most convincingabout family planning, (d) most noticeable or attention-getting,(e) most likely to persuade someone to go to the clinic, and so forth,depending upon the objectives of the product. For example, if thereare five different messages and ten respondents, the rank ordersheet can be made up as follows and the ranking for each messagecan be determined:
111
Rank Order Results for "Interest"
Respondent Number: Messages Ranked From 1 to 5 :
A B C D E
1.2.3.4.5.6.7.8.9.
10.
2 1 4 3 5
3 2 5 4 1
2 3 4 5 1
1 3 5 4 2
3 2 5 4 12 4 3 5 1
1 3 4 5 13 2 4 5 12 3 5 4 1
2 4 3 5 1
Total 21 27 42 44 13
Average Rank 2.1 2.7 4.2 4.4 1.3Fbr EachMessage
Overall Ranking 2 3 4 5 1Based on Averages
In this case, message E has been ranked as number one and mes-sage D has been ranked as number five. In other words message Eis judged as most interesting and message D, the least interesting.Message A is the second most interesting one.
b. Paired Comparison: This method consists of comparing each mes-sage with every other message. At one time only two messages areexposed to the respondents and they are asked to give their opinionas to which one is better than the other. Then another pair is simi-larly tested and the procedure is repeated until each message hasbeen compared with all other messages. All the winners are noteddown in each case. The number of comparisons made will be equalto n n-1 where n is the number of messages. The following table
will explain how the best message can be sorted out through thismethod. Let us suppose there are five messages which we want tocompare. For example A, B, C, D, and E.
112
A E
E
C DC E
D E
Number oftimes as
winner:A: 1B: 3C: 2
D: 0E: 4
A BA C.
BCAEBBECEE
B CB DB
A B C D E Winner Score
In the table above, B is the winner when compared with A; C is the
winner compared with A; A is the winner compared with E; B is the
winner compared with C; B is the winner compared with D; E is the
winner compared with B; C is the winner compared with E; and Eis the winner compared with D. Then the scores for messages A,B, C, D, and E are added up as winners. From the scores listed,it is apparent that E has scored the maximum and D has scored none
while B is second in scoring and so on. In the case of tie scores forfirst place It may be necessary to eliminate the items with the low-est scores and then repeat the paired comparisons to find which isthe best.
Different standards or criteria can be used in the paired comparisontechnique just like they were with order-of-merit. When the respon-dent is given a pair of posters (two versions), for example, he canbe asked to judge which one is the most interesting, which one is themost "artistic," which one is more likely to persuade someone ofthe value of a small family, and so forth. The technique is flexible.Its use depends upon the purpose of the researcher and his ingenuityin adapting It to his pretesting objectives.
9. Testing Attitudes or Opinions : While opinions are just statements,an attitude is a stronger indicator of preference (or no preference) about a thingor issue. It Is believed that a positive attitude will lead to a positive behavior,while a negative attitude will lead to negative behavior or no behavior. Thusthe measurement of attitude provides a more reliable indication to behaviorchange. Some measuring scales have been extensively used and found to givequite reliable measures of attitudes.
113
a. Scaled-response questions may be used to measure a variety ofattitudinal responses to a communication product. After seeing aslide-tape presentation on vasectomies, for example,' -you mightwant to ask each viewer questions like the following:..
1. How important do you think it is for a husband to take responsi-bility for preventing his wife from becoming pregnant after theyhave the number of children they desire?
very important (4)important (3)unimportant (2)very unimportant (1)
2. How good do you think vasectomy would be for a person like youafter you have had the number of children you desire?
very good (4)good (3)not very good (2)not good at all (1)
3. How credible was the main speaker in this presentation to you?
very credible (4)credible (3)not very credible (2)not credible at all (1)
The type of questions asked and the scaled-respoi}jses used dependsupon the objectives of the message to be tested. If two differentversions of the product (for example, slide-tape show and pamphlet)are used, then the numbers next to each response can be used tocompute an average (or percent who answer each response) for eachtest group and the results can be compared.
b. Rating scales are often used for pretesting because they are generallya quick way to collect a variety of information about responses to amessage. The basic procedure is to expose a person or group to aproduct, then ask the following type of question:
How would you describe the film (poster, flipchart, etc.) that youhave just seen?
114
bad /_7 good
(1) (2) (3) (4) (5) (6) (7)
interesting _ _ _ / / dull(7) (6) (5) (4) (3) (2) (1)
unimportant 7 / important
(1) (2) (3) (4) (5) (6) (7)
acceptable / 1 unacceptable
to me (7) (6) (5) (4) (3) (2) (1) to me
confusing (1) (2) (3) (4) (5) (6) (7) easy tounderstand
Any number of opposite adjectives may be used depending upon theobjectives of the product and the object (characteristic of the prod-uct) to be described. As with scaled-responses, the numbersassigned to each response can be used to arrange the percentage ofresponses given, or the average of the group tested for each scale.Note that the end of the scale with the more osp itive adjective is
assigned the highest number, and that the negative-positive end ofeach scale is reversed as you move down the list in order to control
for the effect of their order upon the respondents. These numberswould not appear on the respondent's copy; they are used only by
the researcher to calculate average scores for all the respondents.
Also note that the middle response (numbered as "4") is indicatedby a box. The box clearly marks the midpoint so that all respon-dents know which one is neither "good" nor "bad," but in between.When the box is left out some respondents may accidentally treat"3" or "5" as the midpoint of the scale.
c. Agreement scales are similar to the above scales, but they ask forthe respondent to indicate how much he agrees or disagrees with asample of attitudinal statements:
1. The 1. U. D. is a safe method of contraception for most women.
strongly agreeagreecannot decidedisagreestrongly disagree
115
2. By having fewer children, I will be able to provide a better life
for my whole family.
strongly agreeagreecannot decidedisagreestrongly disagree
3. Daughters are just as valuable for my family as sons.
strongly agreeagreecannot decidedisagreestrongly disagree
10. Measures of Behavior or Action: The ultimate objectives of somecommunication programs is to persuade the audience to take some action.Ultimately, family planning communicators want eligible couples to start prac-ticing family planning. This is a question of behavior change, and it is extreme-ly difficult to measure whether the message which we are trying to pretest willconvince the audience to take any action. This can be measured by the atten-dance at the clinic, the increase in demand for contraceptives, or for moreinformation. But let us not forget that this kind of overt behavior change maybe the result of many other factors besides our particular message. However,some useful feedback based on actual behavior can be obtained during or imme-
diately after pretesting your messages. This can be done in several ways.
a. If at the end of the exposure, the members of the audience have alively discussion about our message and ask for more information,we have a good behavioral indication of attention, interest, andintended action stimulated by the message.
b. If as a result of exposure to our message the members of the audi-ence ask for clinic appointments or for a field worker to visit, wehave further evidence of action stimulated by your message.
c. We can make judgmental observations of the members of the audi-ence while they are watching our message or listening to our pro-gram. From their facial expressions we can judge if the messageis creating a favorable or unfavorable impact.
d. Some of the audience may start leaving while we are showing themour film, or playing them a recorded program. Some may go tosleep. This also indicates a behavioral reaction to our messages.
116
e. We can question members of the audience after exposure to our com-munication about what they would do. Would they do as the messagerecommends them to do or not? The replies can give us some indi-cation of their intended behavior. Some of the audience may comeup and request contraceptives. We may not be sure whether this isthe result of our message or the person's prior intention towardsfamily planning, but, nevertheless the reaction gives some indica-
tion of public behavioral response to our message.
To be useful, of course, the observation of such audience behavior hasto be systematically and reliably recorded as it occurs. It is not enough if theresearch team merely recalls that the audience seemed interested and excited
during exposure to the product. Memory has a way of being very selective andsubjectively biased in such cases. It Is not too difficult to have someone recordthese kinds of observations on special forms. How many attended? What per-cent stayed for the whole presentation? How many fell asleep or engaged inirrelevant activity during the presentation? How many out of the total stayedafterwards to ask questions, get advice, or obtain information about availablecontraceptive services ? How well do their questions reflect that they havelearned something from the product they have seen? If these observations arecarefully recorded, it is possible to compare the results with those from othergroups of respondents or other types of products (perhaps a different version).
117
STTMMA RV
In this unit of the module we discussed the three mainmethods for pretesting communications--the panel method, theexperimental method, and the survey method. Then we brieflydescribed ten techniques which are frequently used to collectinformation about the quality and effectiveness of new commu-nications. Much more, of course, is required to conduct a pre-test. The pretesting team must design the pretest so that all ofthe pieces we have discussed fit together to accomplish the mainpurposes of the pretest. The goals of the communication to bepretested must be clearly specified. Sampling procedures mustbe developed to obtain the most appropriate group of respondents.The pretest method and the techniques for data collection must bedesigned so that a fair and adequate test of the new communica-tion can be conducted with the least amount of error or bias intro-duced by the y that the pretest is conducted.
It is beyond the scope of this module to demonstrate howsampling, methods, and techniques should be designed to pretesta specific type of communication. A pretest depends too muchon the specific situation for which the communication is developed.We have, however, collected several good examples of pretestingwhich were actually conducted for communication developed forfamily planning programs. These may be found in the Appendix
of the module. A great deal can be learned by carefully studyingeach one of these examples. Once this Is done, the best way tolearn how to conduct a pretest is simply to conduct one, eitherby oneself or with a small group of others who are also participa-ting in this module.
There is no perfect way to pretest communication, andany of the methods and techniques can produce misleading results,especially if they are not implemented properly. Relying uponjust one method or one technique over and over again can alsocause problems. This is why we have repeatedly emphasized theadvantages of using more than one method and several differenttechniques for collecting information. This allows us to look atthe performance and effectiveness of our new communications fromseveral different points of view and in more than one kind of pretest
118
situation. Making pretesting a regular part of the planning, design,evaluation and implementation of communication programs is thebest way to improve competence of the professional staff and thequality of the program's performance.
119
REVIEW QUESTIONS FOR UNIT III
1. What are the main differences between the panel method and the surveymethod used for pretesting communications?
2. In your own words, describe the 5 basic steps in the flow chart of theexperimental method.
a.
b.
C.
d.
e.
3. Fill in the missing words or phrases in the three sentences below thatdescribe the kinds of causal evidence provided by the experimental method.
120
a. The exposure to the communication and the effects that are observedare , that is, where one occurs
b. The observed effect did not occur , but ratherexposure to the communication.
c. Other factors did not
4. Why is it acceptable to use a small sample size when the survey method isused to pretest communication?
5. Use the tables in this unit to identify the range of tolerated error, the lowerlimit, and the higher limit for a sample survey size of 30 where it wasfound that . . . .
a. Seventy-five percent approved of the visual design of a family planning poster.
Sample Range of Lower Highersize tolerated limit limit
error
30
b. Fifty percent could not understand what the word "vasectomy" meantin the written message on the poster.
Sample Range of Lower Highersize tolerated limit limit
error
30
121
For either 'a' or 'b', explain in your own words the possible results thatwould be expected from the whole audience (population) from which therandom sample of 30 was selected.
6. List as many of the techniques of pretesting that you can remember, thenreturn to the module text to review the ones you cannot recall.
a. f.
b. g.
C. h.
d. I.
e. J.
122
ANSWERS TO THE REVIEW QUESTIONS FOR UNIT III
1. In the panel method, a small group of expert judges or "typical" audiencemembers are selected and asked to judge the quality and possible effect ofnew communications. Several different types of panels can be used, andthe size of each group is from 5 to 25 persons. Unlike the survey methodno attempt is made to get a representative or probability sample of theintended audience. Panels can be composed of intact groups from the audi-ence, such as a group of women who work together in the same urban fac-tory. Data is more often qualitative rather than quantitative.
In a survey, the emphasis is on the quantitative analysis (such as percen-tages of the reactions of a representative sample of audience members inorder to draw conclusions about the possible range of reactions of the wholeaudience. If probability sampling procedures are used for the survey, it ispossible to obtain a precise estimate of the range of error around a sampleproportion; with the panel method the amount of error is unlmown . Andfinally, in the survey method we accept the sample results as being approx-imately the same (plus or minus error) as how the whole audience wouldrespond to our communication, whereas in the panel method we are onlyasking the panelists to judge how the whole audience might respond.
2. Summary of the basic steps of the experimental method:
a. Define the intended audience for the communication that is beingdeveloped.
b. Select a sample of members from that audience for the experiment,then randomly assign half of them to the "experimental group" andhalf to the "control group."
c. Expose the experimental group to the new communication and either(1) do not expose the control group or else (2) expose the control groupto some other communication with different content.
d. Measure or observe the effects of the communication (or non-communi-cation) in each of the two groups.
e. Compare the effects observed in the experimental group with the effectsin the control group, and report the differences. If the differences are"large enough," then we can report that the new communication doesmake a difference when the audience is exposed to it.
3. Missing words or phrases:
a. The exposure to the communication and the effects that are observed
123
are associated, that is, where one occurs the other occurs.
b. The observed effect did not occur after, but rather before exposureto the communication.
c. Other factors did not cause the effects that are observed.
4. Small sample sizes are acceptable for pretesting communication becausea larger range of error around the sample is usually acceptable. Thesmaller the size of a probability sample of a survey the larger the rangeof possible (tolerated) error in a proportion that is found in that survey.In pretesting, we are often looking for large mistakes in design. A smallsample can usually help us identify major design problems. Also, thekind of goals usually set for communication can usually tolerate a consid-erable range of error without very serious consequences.
5. a. for 75 percent
Sample Range of Lower Highersize tolerated limit limit
error
30 . 33 . 56 . 89
b. for 50 percent
Sample Range of Lower Highersize tolerated limit limit
error
30 +18.2% . 318 . 682(or . 364)
In other words, 15 out of our random sample of 30 (or 50 percent) did notunderstand the word "vasectomy" s used in our poster. Approximately--but not exactly--50 percent of our intended audience would probably not beable to understand this word either. More precisely, since our finding of50 percent is based on a sample of only 30, the proportion which we wouldbe likely to find in the whole audience would be somewhere between 31.8percent to 68.2 percent, plus or minus 18.2 percent around our sample find-ing of 50 percent. So, we can be confident that as many as 68 percent mightnot understand "vasectomy, " and at best only 32 percent would not be ableto understand it. Since the poster is primarily about vasectomies, even ifonly 32 percent do not understand the word, the poster is not good enoughfor our purposes. We need to find a better word, or do a better job on theposter itself of teaching the reader what a "vasectomy" is, and what the
124
6.
word means. In the pretest of the revised version, we should try toachieve a level of understanding of at least 75 percent or better (using asample of 30) so that we can be more confident that most of the audiencewill understand what "vasectomy" means.
Pretesting techniques . . .
a. observation f. awareness of knowledge testsb. interviewing g. indirect questionsc. group discussion h. rating scalesd. questionnaires i. attitude testinge. recall tests j. behavioral measures
125
1100a AI
CONSTRAINTS AND PROBLEMS
OBJECTIVES
After completing this unit you will be able to:
1. Describe the different points of view toward pretestingthat are usually taken by those who produce new com-munications and by those who pretest them.
2. List the main functions performed by the productionstaff, the research staff, and the person who is re-sponsible for coordinating them.
3. Describe some of the potential obstacles and problemsof pretesting that are usually encountered
(a) in the field
(b) in production
(c) from a lack of confidence
4. Recommend ways to overcome the main obstacles andproblems of pretesting new communications.
128
INTRODUCTION
Even though pretesting is considered a valuable means of improving the
quality and effectiveness of all types of communication, the use of pretesting isnot as widespread as we would expect. There are several problems that haveprevented pretesting from being used more often. Some of these problems applyto other kinds of social science research. To conclude the module we would like
to discuss those problems that most commonly prevent pretesting from beingconducted, or prevent its results from being properly used.
THE PROBLEM OF COORDINATION
One of the most sensitive and crucial problems in pretesting is the lackof coordination and misunderstanding that can occur between those who conductthe pretesting and those who produce the communications. The research andproduction staffs often have quite different attitudes and approaches to the sametask--improving communications. They may sometimes agree that pretesting
can help improve the effectiveness of communication efforts, but a lack of under-standing and appreciation of each other's problems can become an obstacle tothe full use of pretesting results.
If the two staffs are separate, each may use language that is not under-stood by the other. Research has its own set of terms, or "professional jargon,"
that colleagues of other professional backgrounds find difficult to comprehend.This module has attempted to clarify many of the terms used in research. Eachstaff may also lack knowledge of the basic principles and skills of each other'sprofession, difficulties that make them unable to appreciate each other's prob-lems. This may ultimately have an adverse effect on the conduct or use ofpretesting.
Pretesting should always be considered a part of the overall communi-cation process, not a separate activity called "research," It is part of the plan-ning, design, and implementation of new communication. If the two staffs areseparate, both the production staff and the pretesting staff are responsible forthe use of pretest results. Both are responsible for the ultimate effectivenessof the communication program. The staff that produces new communicationsshould improve its efforts by making full and proper use of pretesting as anintegral part of its activities.
At the same time, let us recognize that communication specialists cannothe expected to be expert researchers who can conduct pretesting as efficientlyas professionally trained researchers. The roles of these two kinds of profes-sionals are quite distinct, and their differences should be taken into account bythose responsible for the coordination of both activities.
129
The role of the production or creative staff is to pose problems forresearchers, keeping in view the objective of maximizing their communicationefforts. The problems given to the research staff for investigation should beexplicit and precise so that the research people can easily understand and trans-late them into their own professional activity. For example, members of theproduction staff may consider a communication campaign to be very appropriatefor a particular audience. Nevertheless, they still may want to know if a cer-tain theme will be persuasive enough to achieve their particular goals. The roleof the research or pretesting staff would be, first, to restate the problem in itsown terms thus making it capable of being researched, and then to go to a rep-resentative sample of the audience to test the theme. As they go through thisprocess they need to keep the mutually agreed upon objectives in view. The roleof the pretesting staff is to help, to advise, and to guide the communication stafftoward the development of an effective theme. These two roles should not con-flict with each other. Rather, they should complement one another so that theshared objectives can be achieved. Let us not overlook the fact that, in mostcases, the communication staff has to make the ultimate decisions as far as thedesign of its communications is concerned.
In conclusion, if the objective is to maximize the output, both the skillsof research and the creativity of communication production should be combinedand used side by side. Achieving the overall goal of improving communicationshould take highest priority, and the work should be done with team spirit.
Some researchers concerned with the problem of underutilization ofresearch results have suggested that an intermediary function be added to organ-izations that must use research results as part of their operations. * Thesefunctionaries have been given such names as "linkers" or "middle men. " Therole of the "linker" is to provide a bridge between the two kinds of professionals,those who do the research and those who need to use research findings. We callthis function the coordination function , especially in the context of communicationpretesting.
The coordination function should be assigned to the chief or head of the
Information/Education/Communication (EEC) section. This person should havesufficient knowledge of both communication research and production. Linkersmay not be expert in both fields, but they should be able to understand andappreciate the difficulties, problems, purposes, methods, and attitudes of thetwo kinds of professionals involved in pretesting. Their role is to coordinate
the two activities rather than to involve themselves directly in each activity.
This structure might not be possible in some organizations responsible forfamily planning communication, but where problems exist between the produc-
*For a recent review of this problem see J. R. Echols, ed. , Making Population-
Family Planning Research Useful--The Communicator's Contribution (Honolulu!East-West Communication Institute, 1974).
130
RESEARCH STAFF
1. Stating the problem of the com-munication staff in terms capableof being researched.
2. Working with the communicationIEC CHIEF staff to develop objectives for
prete sting.1. Coordinating the two functions
and facilitating communication 3. Reviewing the information alreabetween the communication available and sharing it with theand research staff, production staff.
2. Making available the policyof the program.
3. Malting available funds andresources.
4. Giving information on theoverall goals and prioritiesof the program.
ROLES OF PERSONNEL IN VOLVED INCOMMUNICATION DEVELOPMENT AND PRETESTING
COMMUNICATIONPRODUCTION STAFF
1. Planning and clearly stating theobjective for developing commu-nications and defining the audience(involving the research staff ifpossible).
2. Creating new ideas.
3. Reviewing and using relevantinformation.
4. Requesting specific informationneeded for improvement from theresearch staff.
5. Designing and producing alter-native versions of messages forpretesting.
6. Using the information suppliedby research staff.
7. Developing and producing the finalversion of the communication.
4. Selecting the sample.
5. Developing appropriate methodsand techniques for pretesting.
6. Conducting the pretest.
7. Interpreting the results.
8. Reporting the results, andadvising and guiding the commu-nication staff to improve thecommunication that they are
developing.
tion and research staff, someone should be given direct responsibility forcoordination. On the following page we have listed some of the specific rolesof these three kinds of professionals.
PROBLEMS IN THE FIELD
Many times field problems are so acute that they may cause pretestingto fail. Such problems fall into many categories, ranging from the nonavaila-bility of suitable respondents to inadequate transportation facilities (especiallyfor rural areas) and lack of cooperation from local officials. One of the mostcommon problems is finding suitable respondents who are available at conveni-ent times. Not only should they be good representatives of your intended audi-ence, but they should also be prepared to cooperate with you and able to sparetime for you. Often, respondents may be very cooperative, but their reactionsand replies may not reflect their true feelings because in many cultures it ispolite to answer questions in a pleasing way, out of courtesy and hospitality.
Pretesting may involve placing demands on the respondents that are notcommon in other social science or marketing research and surveys. They maybe expected to perform the functions of experts--to evaluate and to react to thecommunication material being tested--in addition to giving their own reactions.Many respondents are not familiar with interviewing techniques, rating forms,and so forth. When pretesting family planning communication some respondentsmay become reluctant to talk because of the sensitive and personal nature of
the topic.
The local officials can either be helpful or an obstacle to pretesting. Insome places it is almost required to go through the local authorities to contactpeople living in that area. Sometimes it is necessary to obtain the permissionof the authorities prior to entering certain areas. In any case, developing a goodrelationship with the local authorities is important. This can be achieved througha variety of ways: by clearly explaining the purpose of your work in the area, bybehaving or even dressing in a way that will not offend them, by accepting anykind of offers of hospitalities that they very often extend as a part of ceremonialgestures, and by listening to their advice about how to work in their area.
These kinds of things may irritate the interviewers and they can requirea great deal of time, but they are often necessary and preferable to local resist-ance to the research. In fact, these very systems often can be beneficial to con-ducting pretesting. If the local leaders (both official and unofficial) are convincedthat you are not just a meddlesome outsider and not there to influence the peopleagainst their own values and belief systems, it will facilitate your work greatly.The interviewers must be patient and must respect local customs and ceremonies.They should not try to push, but should proceed in a manner acceptable to therespondents.
132
PROBLEMS RELATED TO PRODUCTION
The demands and problems of communication production are so complexthat plans to conduct pretesting can become uncertain. Fortunately, there aremany pretesting methods and techniques available, so it is usually possible tofind a way to conduct some pretesting, even in very difficult situations. As wementioned earlier, some compromises must be made depending on the type ofcommunications to be tested, the availability of resources, and time pressures.
As an illustration, let us consider the example of pretesting a shortmotion picture. One has to make many decisions before starting with the pre-testing. For example, one must decide what effects are to be tested, whatapproaches to take (whole vs. parts), and at what stage the pretesting should bedone so that the results will be the most useful. It would be costly and time-consuming in this case to test the completed film. What one can do in this caseis to take various elements of the film at different stages of development andtest these elements individually. For example, a test of the basic concept canbe done using simple, less expensive and less time-consuming techniques such
as a panel discussion. A test can be done on the story board, which does notinvolve very much expense, or it can be done at the first stage of rough rushprints.
Let us look at another example--a poster that is to be developed. Sup-pose the communication staff has decided on a poster using the slogan, "Rapidgrowth of population in the country is an obstacle to development programs."The illustration that has been chosen is an actual photograph of crowds ofpeople. In the background, industrial progress is to be depicted. Both theslogan and the illustration are chosen for pretesting. It is also decided that theposter should be tested across the country because it is intended for a nationalaudience, rural as well as urban. And it is also decided that the effect of thewhole poster will be tested, as well as specific parts of the poster. Supposethere are twenty-five provinces in the country and two hundred districts. Suf-ficient funds and personnel are made available, but the time available is onlyone month.
These decisions and plans all sound very good. But from the productionpoint of view, they may not be feasible at all. First of all, they will need morethan one copy of the poster and surely not as many as they plan to have finally(say 25, 000). To prepare one copy for each province would mean twenty-fivecopies. Here lies the main production problem. You cannot go to a printer andask for twenty-five copies because the costs will be prohibitive. What you cando is ask your artist to make twenty-five copies by hand. How much time thiswill require will depend upon the number of artists you can afford to engage.This will involve time and money. Will it be possible to prepare twenty-fivecopies in one week? The rest of the three weeks may be required by the researchstaff to carry out pretesting.
133
Then there is another important problem that cannot be overlooked. Thisis the technical problem. It is almost impossible for an artist (or several artists)to prepare twenty-five copies that will be exactly alike. If they are not exactlyalike, the results of the pretesting are very likely to be inconsistent and, there-fore, difficult to interpret. And thus, the pretesting results may contributenegatively instead of providing useful and reliable information to improve theposter.
The reproduction of pretested material is probably the most difficultproduction-related problem. For example, If as a result of pretesting, it isfound that a certain communication (color, picture, lay out, size, voices, etc.)is good and will achieve the objectives that have been set, the question stillremains as to whether it is possible to copy exactly the pretested material with-out the processes of reproduction affecting quality. The technical skills in anymedia are not so perfectly developed that they can ensure the exact reproductionof a particular material.
Under these circumstances the skill and experience of the production andcreative staff becomes very important. Although the members of this staff maynot be able to solve all these problems, they may have some alternative sug-gestions and may provide help to the research staff in deciding upon, planning,and implementing pretesting. Let us not forget or underestimate the problemsfaced by both kinds of professionals in carrying out effective and useful pre-testing.
OVERCOMING A LACK OF CONFIDENCE
In some organizations there is a general lack of confidence in pretesting.Personnel at all levels, whether directly or indirectly involved in the productionof communication materials, may have doubts about the capacity to conduct goodpretesting and, therefore, about the usefulness of the results. The peopleholding this attitude often consider pretesting a waste of money and human re-sources. This attitude toward pretesting is just the opposite of what is neededto conduct useful pretesting. Why this kind of attitude is so prevalent in someorganizations is easy enough to explain. The most important reasons for thiskind of attitude towards pretesting are (1) the results of pretesting are con-sidered unreliable and invalid, (2) the purposes and methods of pretesting aremisunderstood, and (3) many think their work will be "good enough" withoutpretesting. These reasons emerge from bad experiences (failures) in the past;irrelevancy of previous pretesting results for day-to-day problems; misuse ofpretesting techniques; misleading pretesting carried out under unnatural,artificial conditions; and perhaps most important, the lack of sufficient interestand enthusiasm on the part of professional communicators and researchers towork together to develop better, more practical methods.
One of the most frequent criticisms of pretesting is that it is irrelevantto real problems, that it does not solve any real problems faced in everyday
134
communication efforts. This attitude mostly exists in development programs,
like family planning and agricultural innovation. Business organizations (mar-keting and advertising) have a very different attitude towards pretesting. Theyconsider pretesting an integral part of their communication efforts. By pre-testing communication material they believe they can reduce risks and avoidcostly mistakes. That is the reason they invest a good deal of resources inpretesting.
Pretesting is not automatically relevant to the problems faced by thoseresponsible for developing effective communications. Pretesting must be maderelevant by those who do it. The techniques of pretesting do not automatically
yield useful information. The research staff must use those techniques creativelyto obtain useful information with them. Once the process of pretesting has beensystematically built into the production process, both production and researchstaff can devote more time to making pretest results more relevant to the designand production of communication material.
Very often, expectations about pretesting are unrealistic. In all kinds ofsocial science research (including marketing and advertising), the researchfindings can only provide information to facilitate decision making. Pretestingshould never become a substitute for responsible decision making. Researchmay bring to the notice of decision makers the existence of certain problems intheir activities. As with any other kind of social science research, pretestingwill not solve the problem by itself. It is a tool communicators can use formaking better decisions and thus improving the effectiveness of their efforts.
Pretesting cannot and does not solve all communication production prob-lems. It can give you new information about the intended audience that is im-portant for making good decisions at all levels of developing and producing com-munications. By using pretesting we can learn the reactions and estimate theattitudes of the intended audience before investing large sums of money inproducing and distributing new communications.
One way of increasing confidence in pretesting is to involve all peopleconcerned with pretesting at all stages. It may be at the stage of formulatingproblems for pretesting, setting objectives for pretesting, selecting pretestingtechniques, choosing suitable respondents or geographical areas, and so forth.More participation and free flow of communication among the people involveddirectly or indirectly in communication production can greatly increase theirconfidence in pretesting. This kind of involvement should increase everyone'smutual understanding of what the pretesting can and cannot realisticallyaccomplish. It should increase everyone's interest and personal "investment"in the results. Sharing the contents of this module should also correct many ofthe misunderstandings and doubts that some people have about pretesting.
As already has been pointed out, it is the production staff that must makethe decisions about the design of new communications. Final approval from the
135
top management of the organization is often required. These decisions, if basedon competent pretesting results as opposed to purely personal intuitions, oughtto be more realistic and practical, provide better direction, and reduce manyof the errors that have been made in the past. Combining the results of pre-testing with experienced, professional judgment is the way to develop effectiveeffective communication for programs of planned change like family planning.This approach to the problems of communication requires a climate of innova-tiveness and trust in our organizations, and an openness to change from theconventional way of doing things. In this kind of environment pretesting canbecome a useful tool for improving the effectiveness of communication.
136
EXAMPLES OF PRETESTING FAMILY PLANNING COMMUNICATIONS
INTRODUCTION
This Appendix presents several interesting examples of pretesting thatwere actually conducted for family planning programs over the last five years or
so. There are a great variety of approaches, methods, and techniques representedin these examples due to the wide variety of media and situations for which they
were used.
By presenting these examples we are not necessarily recommending thatthese methods and techniques should be used by others. The examples should beused as a source of new ideas for pretesting, and perhaps as "proof" that someoneelse has already faced a certain pretesting problem and has developed practicalprocedures and methods.
It is quite likely that those who originally used these methods and tech-niques would want to change and improve them the next time they use them, espe-cially where the conditions are different than the last time. We should take thesame constructive attitude towards these examples. If we see methods and tech-niques that we think would be appropriate for our own situation, then let's treatthe example as a good starting point from which to develop better methods andtechniques for ourselves. This is another way to say: Use the ideas found in theexamples from others' experience, but use them creatively .
139
Example 1: Pretesting Comic Booklets
Source: Demetrio M. Maglalang, Agricultural Approach to FamilyPlanning (Manila; The Communication Foundation for Asia,1976), pp. 51-58.
141
PRETESTING COMIC BOOKLETS *
A pretest should have the following basic objectives:
1. It should determine format and content acceptability.
2. It should determine content comprehension.
Notwithstanding the work that has gone into the preparation of the mate-rials, one must assume that there can still be possible slips and errors discern-ible only to the end-user. A pretest should be able to expose these slips anderrors.
PRETEST MATERIALS
The ideal material for the pretest is, of course, the finished product it-self, as this has all the features to be presented to the end-user or the target.
However, this procedure may be considered uneconomical, as majorrevisions may be required after the pretest.
A next best alternative is to pretest at a point where the material approx-imates the finished product and yet can still be revised without undue additionalexpense. This would be the blueprint stage of the comic books.
One must, however, recognize the limitation of the material. A blue-print lacks the color scheme of the finished product.
And this may be a crucial factor in a pretest for format acceptability.
Despite this limitation, the pretest can still probe into the other and,possibly, more significant aspects of the product: namely, illustration, messageand story. These, after all, constitute the essence of the agricultural approach.
PRETEST TIMETABLE
Pretest activities must be scheduled within the time frame of the entireproject.
* The authors used this same basic method to pretest flipcharts for theirproject.
142
Adequate time must be allotted to the preparatory work such as the prep-aration of discussion guides and questionnaires and the gathering of respondents.Sufficient time must also be assigned to the pretest proper and the analysis of
the results.
It must be kept in mind that pretest results have still to be used by thescriptwriter in revising his script and by the illustrator in revising his illustra-
tions.
Hence, as much as possible, scripting, illustration and pretesting mustbe done well in advance of distribution dates to prevent protracted time lagsbetween issues.
The pretesting of the 12 issues of the comic books of the media projectmodel was completed in four months. The rest of the project year was spent inpresswork, distribution and field use.
PRETEST PROCEDURE AND THE DISCUSSION METHOD
The pretest sample should be representative of the intended audience. Asthe target of family planning programs should include all those with decision-making potential, the pretest sample should include wives, husbands, and maleand female representatives from the youth sector, possibly four from each group.Resulting comments and suggestions would be more varied as well as more repre-sentative of the potential target.
While there may be other effective methods of pretesting, the staff of themedia project model find the discussion method the most effective for its purpose.
For one, it is much easier to elicit a consensus from members of a groupwho are interacting with one another than from those reacting to a structuredquestionnaire. The latter may generate as many opinions as there are individualsreacting.
Secondly, the discussion method is an activity already familiar to therural people. Here the respondents are not only "audiences" to whom the mes-sages are poured in a unilineal mode of communication (from source to mediumto audience). They themselves become participants in an activity of involvement,and co-producers, in the sense of their being jurors and evaluators.
Thirdly, the use of a structured questionnaire poses a difficulty of drawingout genuine responses from the rural people who normally find it difficult to putinto words or writing their comments and suggestions.
The discussion method, however, has its own disadvantages. This wouldinclude the over-participation of the more vocal and the more influential of the
143
group or the non-participation of the shy and the timid who tend to become merespectators or listeners. Or the discussants may bring in extraneous problemsand concerns, such as requests for information on how to obtain agriculturalloans and the like. Or the discussants may become too enthusiastic over oneaspect of the comic books at the expense of the other aspects.
All of these problems can, however, be resolved by a trained discussionleader, who should ensure that all the members of the group are given equalopportunity to voice their opinions and are steered effectively to the topics underdiscussion.
DESCRIPTION OF THE METHOD
The blueprints of each particular are first distributed several days inadvance to twelve prospective village respondents to enable them to read thecomic stories.
Each discussion group of twelve respondents is proportionally divided asfollows:
Mothers -- 4
Fathers -- 4
Youths -- 4
Total -- 12
On the day agreed upon for the group discussion meeting, a traineddiscussion leader meets with the twelve respondents and discusses the comicblueprints with them. He elicits from them comments and suggestions regardingthe acceptability of the illustrations, language and "adaptive approach" parallelsutilized in presenting family planning messages. The comments and suggestionsare recorded by a secretary and taped on a cassette recorder.
The group discussion session is usually held in the house of a villageofficial or an influential member of the community. For the convenience of thevillage respondents, the meetings are held in the early evening, lasting for anaverage of two hours.
DISCUSSION LEADERS
It is advisable to recruit discussion leaders from the staff of the fieldunit. Being known to most of the rural people in the areas covered by the agency,they can serve as an effective link between the research group and the respondentswho attend the discussion meetings.
144
The discussion leaders must show skill and technical know-how in thehandling of group discussions. The effectiveness of the sessions depends largelyon their competence. They may be assisted by other discussion leaders fromoutside who may be trained in a one-day preliminary workshop.
OUTLINE OF THE DISCUSSION PROCEDURE
The discussion procedure to be adopted in the comic pretest should followa basic format. There may, however, be certain variations from place to place,
from leader to leader or from audience to audience.
But whatever the format may be, the discussion leaders should alwaysattempt to stimulate an atmosphere of informality by establishing pleasant rap-port, fostering team spirit and encouraging spontaneous dialogue. Whenevernecessary, the discussion leader should resort to the art of careful "probing"as an effective vehicle for eliciting relevant remarks on the comic featurestaken up during the discussions.
The following discussion format may be employed by the discussionleader:
I. Preliminaries
A. Introductory Portion
1. Welcoming of the participants
2. Introduction of the visiting team to the group
3. Self-introduction of each participant to the group
B. Entertainment Portion (Optional)
1. Group or individual singing
2. Literary renditions by volunteers or by anyone popularly acclaimed
II. Group Discussion
A. Readership Check
1. Inquiry concerning participants' preparedness for the discussion
2. If necessary, a reading period of 15 to 20 minutes whetherindividually or by a team member reading to the group
145
3. Review for re-enforcement through actual dramatization of thestory by the participants
B. Discussion Proper
1. Discussion of format presentation
a. Eliciting remarks concerning the title
b. Eliciting remarks concerning the language
c. Eliciting remarks concerning the illustrations(Note: the color component is not touched on in the discussionas the materials under study are still in their blueprint stage).
2. Discussion of content presentation
a. Checking comprehension and eliciting remarks concerningstory
b. Eliciting from the participants the messages intended to beconveyed by the story
c. Checking appropriateness of agricultural parallels
d. A-ssessment of agricultural approach by participants
III. Summary
A. Final recapitulation
B. Overall assessment and/or endorsement of pretest material
IV. Conclusion
A. Refreshments
B. Acknowledgment and expression of gratitude to all who participated in thepretest
AN INNOVATIVE FEATURE OF THE METHOD
As can be seen from the outline, several motivational activities areencouraged, to break the ice, as it were. Participants join in group singing orare engaged in lively discussions of farm and home problems, jokes, etc.
146
The media project model introduced an innovative device which wasfound to be effective in arousing the interest and involvement of the participants.The device consisted of the dramatization of the stories by the participants them-selves. Aside from clearing the air of shyness, indifference, or even of hostility,this stirred the emotional involvement of the participants in the stories they weresupposed to evaluate. At the same time, it resolved the practical problem ofparticipants who did not have the time to read the stories assigned.
The device, as presented in the above discussion format, does not belongto the pretest proper, being a mere motivational activity. It has, however, thepotential of being an effective pretest instrument.
In the course of the drama sessions, the participants may be able to pin-point unrealistic situations, dialogues, characterization or whatever weaknessesthere are in message presentation.
AFTER THE PRETEST
Since the sessions are taped, it should be fairly easy to reconstruct themand subject them to staff evaluation and analysis.
The study being qualitative, the staff should be able to group together thepoints of consensus and to isolate individual comments to make the pretest resultsmore manageable and easier for implementation by the scriptwriter and illustrator.
While the results have to be taken seriously, the staff should keep in mindthat the pretest sample, though representative of the proposed target, does notnecessarily speak with absolute authority for the rest of the rural audience. Itcan happen that some observations or comments may reflect only a limited pointof view. The sample consists, after all, of only 12 respondents.
The pretest results of the media project model showed that the prospectiveaudience was, by and large, receptive to the original material. Critical com-ments were understandably more in the pretest of the first three issues than inthe pretest of the rest of the 12 issues. The subsequent blueprints received morefavorable endorsements as they were prepared following the lessons learned inthe pretest of the first three issues.
The results were, by and large, implemented except where technical andtime limitations did not permit it. But the latter exception happened only incases of minor comments.
147
Example 2: Printed Materials
Source: Fred W. Reed, Pre-testing Communications: A Manual ofProcedures (Chicago: Communication Laboratory, Commu-nity and Family Study Center, University of Chicago, 1974),pp. 41-43.
149
PRETESTING PRINTED MATERIALS (U.S.A.
The Content Response Code . The content response code method of pre-testing was developed by Carter (1955). This method of pretesting printed mate-rials permits an intensive analysis of texts and scripts. It does, however, requiremore prior preparation than some of the other pretest methods. The idea behindthe content response code method of pretesting is to allow the pretest subject toread the material at his or her own speed, to react freely to what is read, andthen to explain his or her reactions to the materials. The method incorporates acoding technique which allows the pretester to use simple summary statistics toanalyze the results.
Procedure . The first step in the procedure is to break the printed mate-rial into content segments. A content segment is a portion of the script whichcontains a single idea. Usually a content segment will be a single short paragraph;if the paragraphs are long, they may be broken into two or more content segments.Each content segment is then numbered sequentially through the text (see illustra-tion below). Once the script has been divided into content segments and the codingsheet (shown below) has been made, the pretester can proceed.
Once the pretester has found appropriate subjects, the procedure is asfollows:
"Hello, I am (pretester's name). I am working with a group ofpeople who are trying to produce a brochure (radio program, letter, orother message) which will be given to your friends and neighbors. Wehave had numerous disagreements about how the message should be puttogether and what it should say. We have agreed that since the brochurewill be given to some of the people whom you know best, you would be oneof the best people to judge whether it would be appropriate. I would appre-ciate it if you would help us to make some improvements in this brochure.I won't require much of your time and you might find the brochure interest-ing. (Allow the subject to agree or disagree to participate at this point. )
( Hand each of the respondents a pencil and the script.) "This is thetext of the brochure which we have prepared. I think that you will findsome things that you will like and some things that you don't like. I wouldlike you to read the text. When you find something you like, please makea mark like this, "+", in the left hand column. When you find somethingyou don't like, then make a mark like this, "-", in the left hand column nextto the part you don't like. When you make the 4 mark, I will understandthat you find that part of the text true, interesting, fair, honest, important,valuable, or appropriate. If you make the - mark, I will understand thatyou find that part of the text uninteresting, unfair, untrue, worthless, orinappropriate for your friends to read. (Give the respondents an example.
151
Read a portion of the text, make a + or -, and explain what that would
mean. )
NOW let me go through the directions again. Please make a -mark in the left hand column to indicate the parts you like and a - markin the left hand column to indicate the parts you don't like. You may
make as many marks in the left hand column as you like. Some people
make lots of marks while others make only a few. Don't worry aboutmaking too many or too few. Just read along at your own speed makingmarks when you like. A + mark means that you like a portion or find it
interesting, valuable, or true, while a - mark means just the opposite.
Please begin. "
As soon as each respondent reads and marks the material, the pretestercan take the script from them and code their responses in the code sheet shownbelow. One or more "-" marks in a content segment is entered in the codesheet as one mark. The same is true of "+}' marks. If there are both plus andminus marks in a segment, both should be entered in the code sheet.
Content Segment Code Sheet (example)
Content
Unit #
'°-" marks (number) "+" marks (number) % "a-" mark
1 /11/// 6 // 2 . 25
2 // 2 //////// 8 . 80
3 //// 4 //// 4 . 50
4 // 2 ////////// 10 . 83
5 ////// 6 ///// 5 . 46
6 /// 3 //////////// 12 . 80
Once the marks have been entered, the pretester should return the scriptto the respondent. The respondent should then be asked to make a brief explanatorystatement about each segment next to which a "-" mark has been placed. Eitherthe subject or the pretester can write the brief statements with their segmentnumbers. The statements should be kept with the script and later analyzed in theprestester's office.
Once the respondents have marked the scripts, the scripts have been codedinto the Content Segment Code Sheet, and the brief statements have been writtenand placed with their respective scripts, the pretester can return to the office toanalyze the responses.
152
Analysis . Content segments which have received less than 60 percent" p-" marks should be revised until a later pretest indicates that they have achievedthat criterion. The pretester and the production group with which he works maydiscover, however, that the segments which achieved better than 60 percent "i"marks can be further improved. The short comments which have been providedby the pretest subjects can assist in that effort.
This method of pretesting has yielded reliable results with samples of asfew as 20 subjects. The ease with which it can be administered, however, mayencourage the pretester to use larger and more diverse samples.
153
Example 3: Themes, Slogans, and Messages
Source: Robert W. Gillespie, A Manual on Evaluation of Family Plan -
ning Communications Programs (Tehran, Iran: The Popula-
tion Council, 1974), Appendix C.
155
PRETESTING CAMPAIGN THEME AND POSTER :
ISFAHAN IRAN) COMMUNICATIONS PROJECT
POSTERS:
Posters were pretested using a combination of paired comparison andrank order techniques. In addition to the usual criteria of preference, clarityand truthfulness, 116 women were asked how the posters could be improved andwhat captions could be added. Respondents were asked to rank posters accord-ing to their sutiability for various groups, such as those with many children.Three posters generated much favorable response. One showed a hand holdinga cycle of pills next to photos of three women. The second was one of severalposters with small family motif, all of which were popular. It was a photographof a happy and prosperous looking two-child family in a suburban context. Thethird poster showed a doctor talking to a woman. Respondents' reasons forchoosing a particular poster seemed mostly to relate to its physical appearancerather than its message. When asked why they liked their favorite poster mostof them said that it was beautiful, colorful, and best. As a result of this pre-test a composite poster [see the figures on page 29 of this module] was devel-oped containing all the elements liked by those who were interviewed. It showsa prosperous family, mother in chador (a long shawl that serves as an overgarment), entering a family planning clinic. . . . *
The poster has been further modified following feedback from the field:
The design of the new poster is much bolder and can be seenfrom a greater distance and, more importantly, stencils canbe made so that it can be replicated on walls; the wasted spaceused for feet and the bottom part of the clothes was eliminatedto provide more room for the visual design of the poster, anddue to emphasis on using paramedical personnel in the countrythe picture of a doctor has been changed to that of a nurse ornurse midwife. The nurse is now looking to the audience andasking "which number of children is best; two or three?Which method is best; the loop, pill or permanent method ?"The slogan has been changed so that couples can think abouttwo alternatives of small family sizes and three alternativesof contraceptives, including sterilization.
Excerpted from The Isfahan Communications Project, " by S. S. Lieberman,
Robert Gillespie, and M. Loghmani, Studies in Family Planning , vol. 4, no. 4
(April 1973):73-100. Published by The Population Council.
157
PRETEST FORM FOR MESSAGES :
Although communication programs can have several messages reinforcinga central theme, there should be only one slogan. The slogan can evolve from
testing various messages. Place all the slogans or messages on 4" x 1211
cards
in bold type. You will have to read the messages to illiterates. Read four mes-sages at a time and then note the preferred message. Messages used here arefor illustration purposes only. After you have selected out of each of the fourgroups the messages liked "least" or "most," you can then have the messagesranked by various criteria. To avoid bias in message order presentation, changethe order of showing the slogan with each subject.
For questions 10 through 15 you will need to get the frequency of responsesand assign codes to them. Once you have the coding instructions, you can thenprovide the proper codes. Remember that all columns should have one code, notmore than one.
1. I am going to read you a series of statements. I would like to know whichof the statements you like most and which you like least.
LEAST GROUP 1 MOST
1-1 For safe, reliable, inexpensive methods to stopchild birth, use the loop or pills.
1-2 To avoid another pregnancy, take a pill a day.1-3 The loop and pills are safe.1-4 For protection against pregnancy continue using
the pills.1-5 Your friends and neighbors are using the loop
and pills; why don't you?
LEAST GROUP 2 MOST
2-1 To have children only when wanted, visit thenearest health station.
2-2 Do you want to be pregnant now?2-3 The more mouths to feed from your wages, the
less schooling for each. The more children inthe home, the less space for each.
2-4 Children that are planned have better health,social status, more security, more food, betterschooling and jobs.
2-5 You can have children when wanted.
158
LEAST GROUP 3 MOST
3-1 Men, inform your wives of family planning.3-2 A planned family is a happy family.3-3 For more food, better housing and education
of your children, practice family planning.3-4 Family planning is safe and harmless.3-5 Family planning is for parents who want
children later.
LEAST GROUP 4 MOST
4-1 You do not need to have five or six childrenbecause you think that three or four may notsurvive today.
4-2 Two or three children can provide bettersupport in your old age than five or six.
4-3 Two children is better.4-4 For the prosperity of Iran, have only two
or three children.4-5 Two or three children is best.
2. Now, of the four you like most, which do you think is the best?
Group 1 2 3 4
Message 1 2 3 4
3. Of the four you liked the least, which is the worst?
Group 1 2 3 4
Message 1 2 3 4
4. Now I am going to put all the messages in front of you (or, will read allthe messages again). What one message explains most what a personlike yourself would most want to know before accepting a method offamily planning?
Group 1 2 3 4
Message 1 2 3 4 5
5. Is there a message you think is not truthful?
Yes (1) No (2)
159
If yes, which message?
Group 1 2 3 4
Message 1 2 3 4 5
6. Is there a message you did not understand?
Yes (1) No (2)
Group 1 2 3 4
Message 1 2 3 4 5
7. Do you think a family planning message should describe a specific familysize, such as 2 or 3 children?
(1) Yes (2) No (3) No answer or don't know
8. Do you think the message should describe the pills, IUD, or other
contraceptives ?
(1) Yes (2) No (3) No answer or don't know
9. What do you think is the most important aspect of a contraceptive ?
(1) Safety (5) None of these
(2) Revisable (6) All of these
(3) Reliable (7) No answer(4) Easy to use (8) Don't know
10. What are some of the reasons that couples have seven children?
11. What are some of the reasons that couples have two or three children?
12. Do you know of any reasons why women do not use the IUD?
13. Do you know of any reasons why women do not use the pills ?
14. Do you know of any reasons why men do not accept vasectomy?
15. Do you know of any reasons why women do not get tubectomies?
16. Place a code to identify the message pretest.
160
Example 4: Pamphlets and Leaflets
Source: UNESCO-UNFPA Family Planning Project, "A Design forPretesting Materials in a Clinic Setting," Appendix C--"High-
lights of the Fifth Pretest: Leaflets on the Pills and IUD, " and"The Use of a Field Experiment to Test a Family PlanningPamphlet" (Quezon City, Philippines: Institute of Mass Com-munication, University of the Philippines, n. d. ).
161
A. PAMPHLETS AND LEAFLETS IN CLINIC SETTINGS
A common design is needed for testing a pamphlet and a leaflet in aclinic setting. We are faced with two problems. First, we want to test thematerials on a sample of individuals who are representative of the populationwhich constitutes the target of the mass mailing campaign. Second, a methodis needed which will permit us to obtain almost immediate feedback concerningthe characteristics of our communications which concern us at any particulartime.
SAMPLE
The survey conducted at the beginning of this project had several find-ings . . . pertinent to the selection of a sample for pretesting: (1) Most of thefemales in the target area have positive attitudes towards family planning;(2) Over ten percent of the females knew of nowhere to obtain family planninginformation and advice; (3) Over ten percent of the females in the target areaare actively interested in obtaining family planning advice and information; and
(4) Approximately sixty percent of the females in the target area who do notwant a pregnancy within the next year are not contracepting at present.
It seems plausible to assume, for the time being, that the females whoused Planned Parenthood clinics share a substantial number of relevant charac-teristics with the above mentioned females in the target area. They also havepositive attitudes towards family planning, do not want pregnancies in the nearfuture, . . . are actively using or did actively seek family planning advice.
We have chosen Planned Parenthood clinic patients as the sample uponwhich to pretest the communications which we will eventually send to the targetarea.
METHOD
Pamphlet: The method used for pretesting the pamphlet has been sug-gested by Webb, Campbell, et al. , in their book, Unobtrusive Measures . Wehave chosen to supplement our "unobtrusive measure" with an "interpersonal
simulation. " Interpersonal simulation has been used successfully by Daryl Bernin difficult measurement situations. The reader will be able to identify at whichpoints we are using "interpersonal simulation."
Procedure: At the beginning of a clinic session, the experimenter willplace twenty-five copies each of three booklets on the pamphlet shelf in a PlannedParenthood clinic. The only difference between the booklets will be that theywill have three different covers on them. After waiting for two hours, the
163
experimenter will remove all three booklets from the pamphlet rack. E willthen approach a group of women waiting in the clinic and proceed as follows:
"Hello, I am . I am working with a group of people whoare trying to produce a booklet which explains different methods of contracep-tion or family planning. Most of the people in the group with which I am work-ing think that they know how people think and how they will react to a bookletsuch as this (holds out one of the pamphlets). It turns out though that we allhave different ideas about how best to put this booklet together. It seems to methat people like yourselves, who are using family planning services would havethe best idea of how your neighbors would regard this booklet. Would you mindtalking with me for a few minutes so that I can get your ideas about how a book-let like this should be put together?
After an introduction such as that above, the E will ask the followingquestion:
1. If you were producing a booklet on family planning such as this, and weregoing to send it to your neighbors, which of these covers would you put on it?Why is that?
2. Which of these covers do you think they would find most pleasing? Why?
3. Do you think that one of these covers would draw your neighbors attentionbetter than do the others? What is it about the cover that makes you saythat ?
164
4. A lot of people will not read a booklet like this if the cover annoys them orif they find something they don't like on the cover. Do you think that thereis anything about any of these covers that would annoy one of your neighbors?
The interviewer may notice that one of the members of the group has abooklet with a less favored cover on it. The interviewer may then ask whythe individual took that booklet rather than the one with the favorite cover.The idea here is that a person may take a booklet with an annoying coverjust because it is disgusting and she is going to take it home to show herfriends the kind of junk in the PP clinic, or it simply may have been theone nearest to her when she stood at the rack.
The interviewer may have noticed if any of the females have read the book-let. If they have, she may ask whether they found that it would be difficultfor one of their neighbors to read or if it was simple-minded and assumedthat they didn't know about birth control.
If none has read the booklet, the interviewer may ask: "Would you mindlooking through the booklet with me to tell me what you think of its content?"
Turning the pages only as fast as the audience can read the booklet, shecan ask about both difficult and/or annoying passages in the pamphlet (mark-ing in the booklet to keep notes).
5. Sometimes when people receive booklets like this in the mail, they throwthem away or put them on a shelf and don't read them. If you were to sendthis to one of your neighbors, do you think that they would read it or doyou think that they would throw it away?
165
MF..ASTTT F.S
1. We may assume that the pamphlet which is most frequently taken is the mostappropriate one.
2. Questions about the covers which are asked of the audience can both confirmand clarify the differences in pamphlet use by the clients.
3. The clients, if Bern's findings are valid, should be able to pinpoint difficultareas in the pamphlet.
There will be two repetitions of the above procedure. After the first interviewperiod, the experimenter must wait until the waiting room no longer containsany of the persons who were in it at the time of the first round. The pamphletscan then be put on the racks again. This time, however, the order of the place-ment of the pamphlets will be changed in an attempt to avoid the biases causedby differential placement.
B. PRETESTING LEAFLETS BY GROUP DISCUSSION
PRfRT PM
Different kinds of methods appeal differently to different people. Tofind out what appeal is most commonly associated with the pill, six leaflets(two each for health appeal, economic appeal and status appeal) were preparedby the Development of Materials Committee and pretested by the ResearchCommittee. These leaflets are to be used later in the Third Traveling Labora-tory Experiment.
OBJECTIVES
The objectives of the pretests were:
1. To find out the most preferred leaflet out of six alternatives
2. To find out why they preferred that particular leaflet
3. To find out the appropriateness of the size
4. To find out the respondent's perception of the different aspects of thecover such as the meaning it conveys, size of drawing, color used,content of the leaflet and lettering
5. To determine their perception of what the leaflet conveyed, how theleaflet was written and the words that they could not understand
166
6. To find out if the respondents recognize the appeals (health, economic,or status) used in the leaflets
METHOD
A total of 27 respondents for the pills and 57 for IUD were gathered forthe pretest. The areas covered were Legaspi, Tacloban, Dumaguete, Dagupan,Zamboanga, and Davao [1973].
The group discussion approach was used for the first time in conductingthe pretest. The respondents were shown the sample leaflets. Then they wereasked to choose what they consider the best leaflet. The leaflets were numberedto facilitate selection. The majority's choice (the leaflet with the highest vote)was the one used for discussion
A team leader, usually selected from the Research Team or from amongthe group members, guides the discussion. A discussion guide which coversthe different aspects of the materials has been prepared by the Research Commit-tee for the purpose.
The group discussion approach which we thought could be a better proce-dure than the Questionnaire approach was also found to have a lot of limitations:
a. Since the discussion considers only the majority's vote, the prefer-ence of some respondents for a leaflet other than the one chosen for thediscussion was left out automatically.
b. Even with the presence of a lively discussion leader, only a minority[of the] group participates in the exchange of comments and opinions.Majority of the group members feel [too] shy to even react to a commentgiven by another member.
c. The leaflets do not seem to meet the objective of trying to find outwhat appeal is most commonly associated with the pill because therespondent is made to read only the leaflet of his choice and thereforeexposes himself to only one kind of appeal.
d. Besides, the over-all majority's choice cannot really be said to be
the majority's since we found that in many of the places where the pre-tests were conducted, a different leaflet was chosen and discussed.And so there is very slight difference in the ranking scores.
FINDINGS
Based on these limitations, the findings of the pretests are:
167
Pill
1. The leaflet depicting a girl who seems to be dreaming of money and aluxurious life was the most preferred. Seven (7) out of 27 or 25.92 percentchose this type.
2. The main reason for the preference revolves around the idea of familyplanning being associated with higher income and better standard of living.
3. One respondent wants the leaflet to be bigger; another likes it longi-tudinal not lateral. The rest seem to all agree that the size is just right.
4. a. The illustrations suggested "happy family with earning just rightfor the needs"; "luxurious life just like what the girl is dreaming of"; "the pillwill make the girl's dream come true"; etc.
b. As to size of drawing in the cover, one respondent wanted the houseto be bigger. Another said make the whole body of the girl appear and not justher head. In addition, one respondent said there should be a car.
c. The colors used were not very appropriate, according to many of therespondents. The white coloring on the girl's face should be changed because itappears as if she's crying. The money should be colored just like the actualcolor of money. The shadow of the family in the background is not clear; itwould be better to put more color in it. The sun should be a little bit yellowishand reddish.
d. The most common notion was that the leaflets deal with "how to planthe family"; "how to use pills"; "effects of pills"; etc.
e. The lettering of the title should be a little bit bigger and should becolored red to be more attractive.
5. a. The first sentence or the opening line was familiar to the respond-ents.
b. The publication wants to convey that the parent's wishes for a betterlife will come true if they use pills.
c. Generally, the text was easy to understand. However, two respondentsdid not understand the line "pill makes your cycle regular. " The respondents whoare sisters later explained that both of them, even with the use of pills, haveirregular cycles of menstruation.
d. The style of presentation is good, according to majority of the respond-ents. Only one respondent gave negative comments about the layout of the text
168
and suggested that instead of putting subheads like "Ano and Pildoras ?"; or"Para sa tamang paggamit ng Pildoras . . . " put the statements in declarativeform like "Ang pills ang pinakamabisang paraan sa pagpaplano ng pamilya,etc.
6. None of the respondents who chose this leaflet recognized the appealused.
7. Majority of the respondents have not tried the pills or other methodsof family planning.
C. TESTING PAMPHLETS WITH A FIELD SURVEY EXPERIMENT
With certain kinds of communication products, such as a flier, pamphlet,newspaper, newspaper or magazine advertisements, it is relatively easy to con-duct a field experiment using basic survey techniques. The materials providedbelow were used to test two different versions of a family planning pamphlet usedin the Philippines.
The general procedures would be to:
1. Select a random sample of survey respondents from among yourdefined population.
2. Prepare a questionnaire which measures the respondent's attitude andlevel of knowledge related to the content of the product (pamphlet),and which obtains information about the respondent's general back-ground characteristics (age, sex, education, parity, occupation, etc.).
3. Prepare a packet of materials for the interviewers to use with eachrespondent whose home they visit, consisting of:
a. Instructions on how to locate each respondent to be interviewed.
b. The questionnaire to be administered to each respondent beforethey read the pamphlet.
c. An envelope which contains either version A or version B of thepamphlet to be tested.
d. The questionnaire to be administered to each respondent immediatelyafter they read the pamphlet.
4. Half of the packets will contain envelopes with version A of the pamphlet,and the other half will contain envelopes with version B. Since the
169
interviewers do not know which version each respondent will receivebeforehand, and since versions A and B are randomly placed in halfof the packets, you have the necessary conditions for an experimentalresearch design: two message treatments with respondents randomlyassigned to each treatment, and before and after measures of knowl-edge and attitude related to the message.
The results of the enclosed example are not available, but it is useful toanalyze the format of the questionnaire and the organization of the packet.
Administered Before Reading the Pamphlet
UP/TMC-UNESCO-UNFPA FAMILY PLANNINGCOMMUNICATION PROJECT
This is an experiment regarding your attitudes towards family planningand the pill. You may (1) strongly agree , (2) slightly agree , (3) be neutral ,(4) slightly disagree , (5) strongly disagree with each of the statements. Pleaseput (X) to the blank corresponding to your attitude. This is not an examinationso there is no correct or wrong answer. Hence, no need to ask your seatmatesabout your answer. Thank you.
1. There is a need to practice family planning if the couple wants tosave for the future.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
2. There is a need for plenty of children so that they will take care ofthe parents when they get old.
strongly agreeslightly agreeneutralslightly disagree
strongly disagree
3. The pill is the most effective contraceptive method in family planning.
strongly agreeslightly agreeneutral
170
slightly disagreestrongly disagree
4. There are still other contraceptive methods which are better thanthe pill.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
5. The side effects of the pill are only temporary.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
6. The side effects of the pill are permanent.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
7. A mother has nothing to worry about the side effects of the pill.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
8. The pill is bad for the health of the mother.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
9. The pill does not cause cancer.
171
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
10. The pill causes high blood pressure.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
11. Mothers should not hesitate to take the pill if it is prescribed by a
doctor.
strongly agreeslightly agree
neutralslightly disagreestrongly disagree
12. Mothers should be careful in taking the pill even if it is prescribed
by a doctor.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
Your characteristics:
13. Age
14. Sex: Male Female
15. Educational attainment
16. Civil status: Single Married Widow/widower
17. if married or widow/widower, how many living children do youhave ?
18. Your religion
172
19. Occupation
20. Dialect used at home
21. What information have you already heard about the pill?
22. Have you tried any contraceptive method for family planning?
Yes No
23. If yes, what contraceptive have you tried ? What methods are youusing at present?
Method Tried before Being used at present(check) (check)
PillIUDRhythmCondomDiaphram
Others:
24. If you have tried or you are using the pill at present, what can yousay about it?
173
VERSION A
ANG PILDORAS
Milyun-milyong kababaihan as buang mundo
ang gumagamit ng pildoras sa pagpaplano
ng kanilang pamilya.
Bakit hindi rin kayo gumamit ng pildoras?
*Ayon sa mga duktor, ito ang pinakamabisangparaan sa pagpaplano ng pamilya, kung ang
paggamit ay tama.
*Nagagawa nitong regular ang inyong regla.
*Nakakabawas ito ng mga sumpong na dala
ng pagdating ng regla.
*Ito'y hindi sagabal sa pagtatalik
ng mag-asawa.
*Napakadaling gamitin nito. Lunukin lamang
Huwag mabahala kung kayo'y makaramdam
ng mga sumusunod:
Kung kayo'y umiinom ng pildoras, karaniwan
lamang na makaramdam kayo ng ilang
pagbabago sa inyong kalusugan. Kung minsan
anumang sakit ng katawan ay inaakalang
dahil sa pildoras.
Kung makaranas kayo ng pagsakit ng ulo,
ito ay dulot ng hindi pagkatimbang ng
hormon sa inyong katawan. Ito'y nawawala
pagkaraan ng ilang linggo.
Kung nakararamdam ng pagduwal, ito'y
maaaring dahil rin sa sobrang hormon naestrohen sa inyong dugo. Nawawala ito
pagkaraan rin ng ilang linggo.
Walang ebidensiya na ang pildoras ay
nagdudulot ng kanser. Subalit ang mga
duktor ay nag-iingat maghatol ng pildorassa mga babaing dati'y may kanser.
Hindi totoo na ang sakit o mantsa sa balatay sanhi ng pildoras. Magkagayon man ito'y
tulad rin sang lumalahas sa mga nagdadalantao.
Iisa ang dahilan ng mga ito--maraminghormon na dumadaloy as dugo. Papalitan
ng duktor ang uri ng pildoras.
Walang tiyak na relasyon ang pildoras sa
pagtaas ng presyon. Ngunit may mga
babaing hindi matanggap ng katawan ang
dagdag na estrohen. Ito ang nagpapanatili
ng asin sa bob ng katawan na nagiging
sanhi ng pagtaas ng presyon. Dapat ring
palitan ng duktcr ang uri ng pildoras.
Hindi talagang nagkakadoble ang regla ng
babae. Nangyayari lamang ito kung sakaling
nakalimot kayong uminom ng pildoras sa
takdang araw kay nagkakaroon ng dagling
pagkabagabag na nauuwi sa regla. Ito'ylumilipas rin.
VERSION B
ANG PILDORAS
Milyun-milyong kababaihan sa buong mundo
ang gumagamit ng pildoras sa pagpaplano
ng kanilang pamilya.
Bakit hindi rin kayo gumamit ng pildoras?
*Ayon sa mga duktor, ito ang pinakamabisang
paraan sa pagpaplano ng pamilya kung ang
paggamit ay tama.
*Nagagawa nitong regular ang inyong regla.
*Nakakabawas ito ng mga sumpong na dala
ng pagdating ng regla.
*Ito'y hindi sagabal sa pagtatalik
ng mag-asawa.
*Napakadaling gamitin nito. Lunukin lamang
PARASATAMANGPAGGAMI TNGPI LDORAS:
*Kumunsulta sa Duktor.
*Reresetahan niya kayo ng pildoras na dapat
ninyong gamitin.
*Tuturuan niva kayo ng tamang paraan ng
paggamit rite.
Ang pildoras ay makukuha sa mga Family
Planning Clinic o mabibill as botika.
Para sa dagdag na kaalaman sa paggamit
ng pildoras, magsadya sa pinakamalapit na
Family Planning Center.
Administered After Reading the Pamphlet
UP/IMC . UNESCO-UNFPA FAMILY PLANNINGCOMMUNICATION PROJECT
This is an experiment regarding your attitudes towards family planning
and the pill. You may (1) strongly agree , (2) slightly agree , (3) be neutral ,(4) slightly disagree, (5) strongly disagree with each of the statement. Pleaseput (x) to the blank corresponding to your attitude. This is not an examinationso there is no correct or wrong answer. Hence, no need to ask your seatmatesabout your answer. Thank you.
1. There is a need to practice family planning if the couple wants tosave for the future.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
2. There is a need for plenty of children so that they will take care ofthe parents when they get old.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
3. The pill is the most effective contraceptive method in family planning.
strongly agreeslightly agree
neutralslightly disagreestrongly disagree
4. There are still other contraceptive methods which are better thanthe pill.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
175
5. The side effects of the pill are only temporary.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
6. The side effects of the pill are permanent.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
7. A mother has nothing to worry about the side effects of the pill.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
8. The pill is bad for the health of the mother.
strongly agreeslightly agree
neutralslightly disagreestrongly disagree
9. The pill does not cause cancer.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
10. The pill causes high blood pressure.
strongly agreeslightly agreeneutral
176
slightly disagreestrongly disagree
11. Mothers should not hesitate to take the pill if it is prescribed by a
doctor.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
12. Mothers should be careful in taking the pill even if it is prescribedby a doctor.
strongly agreeslightly agreeneutralslightly disagreestrongly disagree
177
Example 5: Pictures and Pamphlets
Source: Roland von Euler, "Mass Communication Experiment"(Stockholm, Sweden: Swedish International DevelopmentAuthority [SIDA], 1972), pp. 55-57; 64-48; Annex II, Part III.
179
PRETESTING PICTURES
The pictures to be used as the "signature" of the campaign, on posters,match-boxes, and pamphlet covers, were to be photographs, it was decided,rather than drawings, on the assumption that the family members in photographywould took more real and be more easily recognizable as belonging to theaudience. Colour should be used for attractiveness but, for the sake of cost,only two colours; the photographs should be in black and white and the back-ground--or the "sky"--in a bright colour, which might differ among the differ-ent pictures--the man and the woman with children, and the whole family--butbe the same wherever the picture was used. Both these things, the colourlesspeople and the brightly coloured sky, deviate from "reality. " But the pictureswere assumed to be more attractive this way than they would be if two colourswere used for the persons, leaving the background without colour of its own.And the gain in attraction was assumed to outbalance the loss in realism.
The question whether to use two or to use three or more colours couldbe taken as a question of quality as quantity, the frame of the total cost beingrather fixed. The assumption was that one or two more colours would notenhance comprehension or attraction to the extent that this would justify to godown significantly in the number of copies to be used, nor to drop some of theitems of the material.
In the discussions on these issues it was, of course, taken into considera-tion that the target people probably were not used to looking at pictures orto interpret-ing them. A colourless man under a bright red sky might look very confusingindeed to somebody who never saw a photograph.
In order to verify or falsify assumptions like these, at least tentatively,alternative designs could, conceivably, have been prepared and pretested forcomparison. Due to the lack of time this was not feasible, however. The onlything that could be done was to design the pictures according to those assumptionswhich seemed most reasonable and pretest whether they were likely to beaccepted.
These pictures were pretested separately. The questions used differedto some extent among the three areas. The pretest had to be done as soon as thepictures for a certain area were prepared. Uniformity in questions was notconsidered so essential as to justify a delay.
Colours were included in the pretest only in East Pakistan (Bangladesh).There was nothing in the pretest--not in the actual campaign for that matter--that indicated that the way colour was used caused irritation or confusion. Infact, almost all interviewees at the pretest said they liked the colours of the picture.
181
The picture of a woman with her two children--to be used on the small-size poster, on the cover of the pamphlet for the wives, etc. --was originallydesigned without much background detail, in accordance with a sketch by theCommunication Adviser. At the pretest people were asked, "In your opinion,does this lady look like a village woman or a city woman?" The majority,around 80 percent, felt she looked like a city woman. Then, as suggested inthe campaign programme, a rural background was added--a typical farm hut,some bamboo trees, etc. In order to accommodate these things the format waschanged from vertical to horizontal. At a second pretest a majority of 70 per-cent gave the answer that she was a rural woman. It is possible, to be sure,that more penetrating questions would have indicated doubts about her identityin spite of the undoubtedly rural background. Anyway, the picture was used inits new shape.
More elaborate series of questions were used for the West Pakistanimaterial to find out whether the pictures actually were portraying such nice,respectable, only-slightly-better-off-than-average people they were supposedto portray for easy and satisfying identification.
The following questions were used (in English translation):
1. What area do you think the people in the picture belong to?
2. Do they look like rural or urban people ?The husband?The wife ?The son?The daughter?
3. What do you think the ages are of the husband, the wife, the son,and the daughter?
4. Do you think this man is rich, well-to-do, average, or poor?
5. Does he look like a respectable man or an ill-reputed man or justa common man?
6. What do you think is his occupation?
7. Do you feel this man looks like being a leader in good deeds or baddeeds ?
8. Would you like to be friends with this man?1
182
1 For male interviewees only.
i 'J"r"d'r"ly
" C O r
First rough sketch
L^
Final version of the picture
9. Would you like your family to meet their family?
10. Does the wife look like a good housewife or a bad housewife or anordinary woman?
11. What kind of family do you think she belongs to?
12. Would you help her to find a house in this area?
13. Do you think she is a loving mother or a careless mother?
14. Do you think this woman looks like an intelligent woman or anunintelligent woman?
15. Do you think the pictures show that the children are well cared for?
16. Would you like to have this woman as your friend ?2
17. Do you feel it is good to use these pictures for posters? . . .
PRETESTING THE CONTENT OF THE PAMPHLETS
Besides finding out whether there were any offensive or else inadvertentdetails in the two motivational pamphlets, the pretest aimed at assessing to whatextent the pamphlet was likely to have an impact on the reader or listener. Asto this latter purpose, three possible approaches were considered.
The first one aimed at eliciting any changes in the perception of familyplanning following the reading of, or listening to the pamphlet. This might bedone in either of two ways. One would be to select a sample, reasonably similarto the prospective audience, question them on points where a change might beexpected, then make them read or listen to the pamphlet, and, finally subjectthem to the same questions again. The difference in the replies would indicatethe change. This method has the obvious draw-back of presuming that theimpact would be, at least approximately, the same in this kind of an interviewsituation as when a person, one way or another, is exposed to the pamphlet con-tents during the actual campaign. The initial series of questions is likely tofocus peoples attention on certain points in the messages that the pamphlet istrying to convey. The impact of the pamphlet is consequently likely to be over-rated by this method.
The other method would be to use two matching samples of people,
184
2 For female interviewees only.
subject one of them to the pamphlet, question both groups, and compare theanswers. This method carries less of a risk for a bias produced the interviewsituation. But the samples have to be larger to keep down the risk for chanceerrors of sampling.
Both methods meet with difficulties which have been discussed inChapter One of the present report, in connection with the research design forthe study as a whole, and need not be repeated here. For many reasons thisapproach to the pretesting problem could be abandonned.
The aim of the second approach is limited to assessing whether theaudience will be able to follow the story. A sample of people is subjected tothe pamphlet and then questioned about the various points of the story. Thisprocedure was followed in the urban areas. The questions were as follows:
1. Is he (the husband) a literate or illiterate person?
2. How many children do they (the couple) have ?
3. From where did he learn about family planning?
4. How many children did they have when they got to know about familyplanning?
5. Did he consult his wife (Did she consult her husband) before adoptingfamily planning or not?
6. What did his mother (her mother-in-law) suggest regarding thenumber of children?
7. From whom did he get advice and medicine for family planning?
8. Did he pay any price for the advice and medicine he received ?
9. What method did they adopt after trying several methods ?
10. What were their plans regarding the education of their children?
11. What did they decide regarding the birth of their third child?
12. Why did they postpone the birth of their third child?
13. Did their economic conditions improve or not after starting familyplanning ?
14. Which methods of family planning are mentioned in this pamphlet?
185
The answers were, on the whole, quite satisfying. Percentages ofcorrect answers, calculated for men and women separately for each of theareas, amounted to 90 percent or more in two out of three cases. Exceptionswere answers about contraceptives, obtained from women in the middle income
area.
The third approach intended to check whether the pamphlet is likely tocommunicate its main themes and not only the details of the story to the readerand the listener. This checking would be done by asking a sample of people avery general question like "Now that you have read the pamphlet (Now that thepamphlet has been read to you), tell as what ideas there are in this pamphlet."If a person has understood the themes of the story he or she should, we assumed,be able to mention the possibility of planning for what one wishes to have, aswell as the possibility of planning one's reproduction. As to the first of thesethemes, the person should be able to specify some of the wishes mentioned inthe story about, like increased farm operations, improved living conditions,education of children, health of children. As to the second theme, the possibil-ity of fertility control, the person should be able to mention such points as theaccess to advice and services, the idea of spacing childbirths, the idea of hav-ing the number of children desired by the parents rather than complete stoppingof reproduction, and the idea of a joint decision of husband and wife.
This approach was applied in pretesting the pamphlets for the ruralareas in East and West Pakistan. We felt that if a person mentioned two pointsunder each of the two themes, this would indicate a satisfactory communication.Against this expectation the findings were disappointing. In East Pakistan theaverage number of points made was 1. 1 for male respondents and 1.4 forfemale respondents. In West Pakistan the averages were 1. 1 for male and 1.5for female respondents. This result gave rise to some afterthought.
Probably the interviewers were not sufficiently trained to stimulate therespondents to keep on mentioning the points they remembered. It is also prob-able that they were insufficiently trained in recording all points mentioned.Although the purpose of the interview was explained to the repondents many ofthem may have found it difficult to understand why an educated governmentofficial from the city should come and ask about the contents of a pamphlet that
he himself had brought along and must be familiar with. Doubts in this respectmay very well have hampered them in interpreting the story.
Finally, the question presumed an ability on the side of the respondentsto abstract an idea from the story and to verbalize it without much time forpondering. In a campaign the situation would be different. As long as peoplehad been able to follow the story they would ponder . . . it, [be] repeatedlyreminded of it, gradually realize its implications, learn to verbalize theirinterpretation and their reaction to it in discussions in the family and withfriends and neighbours. It seemed to us reasonable to assume that the pamphlet
186
would communicate more effectively within the course of a campaign than thepretest results had indicated. With these considerations, and with the deadlinesimmediately ahead of us, we decided to keep the story as it was. In the Urduversion some changes in the vocabulary were made on the basis of observationsat the pretest as well as some grammatical changes to make the usage consistent.
Besides the general question discussed above a separate question wasincluded concerning the point in the story that the decision to adopt family plan-ning was taken jointly by husband and wife. The question was formulated: Whoin the family decided to practice family planning?" In the East Pakistani pre-test 48 percent of the men and 30 percent of the women reported that the decisionwas taken by both husband and wife, while in the West Pakistani pretest only 12percent of the men and 50 percent of the women replied in this fashion. Wecould not find any reasonable interpretation of these differences between menand women, going in different directions in the two provinces. The formulationof the question may have been inadequate--the husbands word is almost by defini-tion, the deciding word in the family, no matter whose wish it reflects. It shouldbe mentioned, furthermore, that two questions concerning central themes of thepamphlet were included in the urban pretest schedule mentioned above. Thesequestions were, however, formulated as questions about the respondent's ownopinion and, therefore, not suitable to elicit the communication achieved throughthe pamphlet. The respondent ma y have had the same opinion before being sub-jected to the pamphlet.
A different, more pertinent question, related to the reader's and listen-er's possibility to identify with the characters of the story, was included in theurban schedule. It read: "Is there any person in your area whose circumstancesare similar to the circumstances of this person in the story?" The question issomewhat complicated, it should be realized, and the meaning of "circumstances"not unambiguous. The response differs between men and women as well as betweenthe middle income area and the low income area. The latter difference is likelyto reflect the fact that the story was adapted to the middle income group ratherthan the low income group. . . . In the middle income area replies to the affirmativewere given by 91 percent of the male respondents and by 42 percent of the female
respondents. In the low income area such replies were given by 25 percent of themen and by none of the women.
SAMPT.TIVC;
The samples for pretest interviews should represent the prospectiveaudiences in the campaign areas. The samples should not be selected in thosevery villages where the campaigns were to be conducted but in similar communities.It was decided that a strict probability sample should not be used due to the short-age of time to our disposal; selecting and using a probability sample is a rather
187
time-consuming affair for the reason, among others, that the sampling frame,i.e. the prospective audience, would be troublesome to identify in advance.Experiments made previously when pretesting other communications materialhad suggested that it did not make too much of a difference whether such a sam-ple was used or not, at least not as far as the men were concerned.
Since the pamphlets were meant for both literate and illiterate peopleboth these categories should be included in the sample. Among the men approx-imately equal numbers of literates and illiterates should be included. Amongthe women it was realized, the literates would be few, anyway. The sampleswere to be selected among married men and women below 45 years of age.
In East Pakistan 25 men and 30 women were interviewed. In the WestPakistani rural area 33 men and 28 women were interviewed. Some of theliterate men were actually unmarried because too few married literates wereavailable in the village at the time of interviewing. In the urban areas 47 menand 50 women were interviewed. These were practically equally divided betweenthe middle-income and the low-income areas.
INTERVIEWING PROCEDURE
First a short introduction as to the purpose of the interview. Then, ifthe person was literate the pamphlet for men or that for women, respectively,was handed over. The interviewee was asked to read carefully each item andlook at the picture corresponding to that item. To illiterate respondents thepamphlet was read out slowly and the picture was shown for each item. Then
the actual interview started. The respondents were not allowed to look back inthe pamphlet to find the answers. The interviews were conducted by the per-manently employed, experienced interviewers of the Research Units.
188
PART TTT
INTERVIEWING SCHEDULE - PUBLICITY
1. Do you know that family planning officials are working in your area?
Yes No (Skip to Q. 2)
1. a) What are they doing?
Fixing postersDistributing pamphletsDistributing invitationsDistributing match boxesDistributing medicinesSelling medicinesArranging meetingsArranging puppet showsArranging film showsOperating clinicsCan't say exactlyOther (SPECIFY)
1. b) Anything more? (CHECK THE ABOVE CATEGORIES ORWRITE VERBATIM)
Conducted by Sweden Pakistan Family Welfare Project and West PakistanFamily Planning Board, Research Section.
189
2. Did anyone in your home get any pamphlet about family planning?
Yes No
2. a Which one of these pam- 3. Have you seen any of these
phlets ? (SHOW THE PAM- pamphlets ?
PH LE TS ONE BY ONE)
1. Male 1. Male
pamphlet pamphlet
2. Female 2. Female
pamphlet pamphlet
4. Contracep- 4. Contracep-
tive Live
pamphlet pamphletHave not
seen any one (Skip toQ. 10 and then ask
Q. 15)
4. Have you read any one of them?
1. Male pamphlet
2. Female pamphlet4. Contraceptive pamphlet0. Did not read any
IF RESPONDENT DID NOT READ ANY PAMPHLET ASK:
5. Has someone read any of these pamphlets for you?
1. Male pamphlet2. Female pamphlet4. Contraceptive pamphlet0. None of them have been read out (Skip to Q. 10 and
then ask Q. 15)
IF ANYONE HAS READ ANY OF THE PAMPHLETS TO THERESPONDENT, ASK:
6. Who read?
NAME: (write relation with respondent)
190
Male pamphlet
Female pamphlet
Contraceptive pamphlet
RECORD VERBATIM.
7. Would you please tell me what has been said in the pamphlet
Male pamphlet
Female pamphlet
Contraceptive pamphlet
Don't remember
IF RESPONDENT KNOWS ONLY ABOUT CONTRACEPTIVE PAMPHLET,SKIP TO Q. 13.
8. In this pamphlet one male/female told that he/she had adopted familyplanning, and mentioned some of its advantages. Do you remember thoseadvantages?
Better dwelling house/extension of the houseChildren's educationBuying of motor cycleBetter foodBetter standard of livingDon't rememberAny other
9. Do you remember who decided to adopt family planning in that family?
1. Husband2. Wife3. Both4. Anyone else (SPECIFY)
191
10. Do you believe that if a common man has less children he can educatethem properly or do you think that even if one has less children he cannot
educate them properly.
( RECORD VERBATIM)
11. Have you learnt anything from the booklets about benefits of family plan-ning which you did not know before?
Yes No Don't know
12.
What have you learnt from the booklet about benefits which youdid not know before?
Don't (know or remember)
13. Have you learnt anything from the pamphlet about the methods of familyplanning which you did not know before?
Yes No Don't know
14.
What have you learnt about the methods of family planning which youdid not know before?
Don't (know or remember)
192
Example 6: Fliers, Pamphlets, TV Playlets, and Slides
Source: T. H. Sun and G. Cernada, "General Procedure for Pre-testingand Evaluation of IEC Material" (Taichung, Taiwan: Commit-tee on Family Planning, n. d. ).
193
GENERAL PROCEDURE FOR PRE-TESTINGAND EVALUATION OF IEC MATERIAL
Committee on Family PlanningTaichung, Taiwan, Republic ofChina
A. Pre-testing of fliers and pamphlets:
1. Questionnaire; sample: about 20 to 60 cases.
2. Oral inquiries: What does the picture mean?Is the writing understandable?Is the whole thing appealing?Is the content (of booklets) interesting or useful?Do you have any suggestions to improve it?
3. Questions asked in various surveys (RAP, etc.).
B. Evaluation of TV playlets and slides:
1. Head count: Telephone calls during the program to check how manyare watching.Quiz and prize for viewers who send in answers.
2. Viewer's comments: Selection of written comments sent in by viewerspublished on one newspaper (prize: NT$100).
3. Request and reaction: Letters came to special post office box.Phone calls to special family planning tele-phone in Taipei.
4. Questions asked in survey: RAP IV.
5. Newspaper write ups or letters to the editor criticizing the program.
As a result of public opinion, pictures of condoms were taken off the airbecause parents complain [ofj small children asking them about it. For samereason, drawings of tubal ligation and vasectomy were removed from comic books.
Many request that the play should be in Taiwanese instead of Mandarinbut because of the government policy to promote Mandarin speaking, no change
has been made.
General reaction to the brighter color fliers produced in 1973 is "It's adelightful change from the usual serious looking educational material, verydecorative! "
195
Evaluation of Fliers: Do you want to know about the Loop?
Interviewer:Date:
Name: Age: Sex: Male ( ) Female ( )
Marital Status: Married ( ) No. of Children:boysgirls
Single ( )Widow(er) ( )
Education: ( ) Senior High/Vocational School grad.( ) Junior High grad.( ) Junior High ungrad.( ) Primary School grad.( ) Primary School ungrad.
1. Have you ever seen this flier? (Show the flier.)
( ) Yes ( ) No
2. What does it tell us ?
( ) Tells people to go get the loop.( ) Tells advantage of the loop.( 7 2 children are just right.( ) Plan birth.( ) Educational material available.( ) Health station & public hospital provide service.( 1 (
3. Is writing easy to understand?
( ) No
( 1 Yes( ) Some part difficult
4. Is the picture understandable?
( ) All Difficult( ) Part part: page pie.Other
196
5. Do you feel this flier is useful in helping you to understand FamilyPlanning?
( ) Yes/some( ) Possibly in the future( } No
( )
6. Do you know how the loop prevents pregnancy?
( ) Increase movement of fallopian tube to prevent conception.( ) Prevent conception.( ) Only know it will, but not how.
( ) No.
7. Side effects of the loop is long lasting or temporary?
( ) Long lasting( ) Temporary( ) Don't know
8. When to insert?
( ) After period.( ) 42 days after the child birth.( ) Right after miscarriage.( ) Don't know.
9. Do you know the place and price of loop insertion?
(1) ( ) Health station.( ) Family Planning clinic at public hospital.( ) Contracted doctors.( ) Don't know.
(2) ( ) NT$30.( } Other price or don't know.( ) Postpartum case or women with more than 3 children are free.
197
Example 7: Flipcharts and Filmstrips
Source: Office of International Health, U. S. Department of Health,Education, and Welfare (Washington, D.C., n. d. ).
199
INDIRECT EVALUATION OF FLIPCHARTS AND FILMSTRIPS
An inexpensive and useful pretesting method has been used by variousfamily planning programs, especially international agencies. In this methodthe field workers and/or clinic workers are used to provide feedback on thematerial to be developed. They are also requested to give recommendations toimprove the material. In our opinion this method is not, only inexpensive andpractical, this also provides the opportunity to involve the workers in pretest-ing, who are the ultimate consumers of the material. By involving the workersin pretesting the chances of the use of the communication material are enhancedgreatly and because they are the people who are directly in contact with theintended audience, their recommendations can be very valuable.
For various materials to be pretested, different kinds of questionnairesare developed. In many cases the workers are first given the test themselvesas an intended audience and then asked to test the material on the intendedaudience. Some times special instructions are prepared jointly by the produc-tion and research staff for these workers to reduce biased opinion and haveconsistency in conducting pretesting. In the following pages two examples ofthis method are given in the form of questionnaires to be used to test communi-cation material (flipcharts and filmstrip).
EVALUATION OF FAMILY PLANNING FLIPCHART MATERIALS (B)
Population /Family Planning Education Officers receiving the prototypeeducational materials (easel and three flipcharts) on Family Planning designedfor use in group and individual presentations, have been asked to kindly give
their comments, suggestions and evaluation of the materials for possible usein their programs.
This form is provided to assist in recording your comments and sugges-tions.
Country Date:
Name of AID Officer
Title:
National Family Planning Education Officer's comments: (Name, Title andOrganization of those responsible for evaluation.
Name of Officer Title Organization
201
Name of Officer Title Organization
Name of Officer Title Organization
I. Easel Evaluation
(Examples of items about which you may wish to comment: hinges; ringbinders; sides; weight; size; style)
What do you feel are the positive features of the easel?
What were the negative features of the easel?
What changes would you recommend to improve the easel?
II. Flipchart Evaluation
1. With what groups were the flipcharts reviewed? (community groups,i.e., Agriculture Extension; FP Clinic, MCH Clinic, Post Partumpatients; health professionals, i.e., physicians, nurses, healtheducators, others)
Flipchart I:
Flipchart II:
Flipchart III:
2. What were the favorable continents regarding each flipchart?(summarize comments)
202
Flipchart I:
Flipchart H:
Flipchart III:
3. What were the unfavorable comments regarding each flipchart?
Flipchart I:
Flipchart II:
Flipchart III:
4. Recommended changes in the Flipcharts:
a) What changes should be made in the message in order that itwill best serve your program?(For each change recommended in the narration, please giveFlipchart number, page number, and a brief explanation of thereason)
203
b) What changes would be necessary in the illustrations in order to
serve your educational program?(For each change recommended in the illustrations, please giveFlipchart number, page number, and a brief explanation of thereason)
c) What specific changes in the illustrations would be necessary inorder to depict population groups in your country closely enoughfor such a material to be acceptablePlease describe changes in illustrations that need to be madebefore the material would be acceptable:
d) Were there any ideas or facts presented in the Flipcharts thatcreated a special reaction (positive or negative) which you wouldcare to share with us? (This will serve as a guide for anyrevision. )
e) Do you have any other suggestions for changes in the Flipchart?(type of paper used, size, number of pages, size of print, etc. )
204
III. Recommendations
1. What further action in regard to the easel and/or flipcharts would you
recommend to AID/W?
2. What other support services are needed by the education component ofyour program with which AID/W may be of assistance?
Additional space for answers to questionnaire, further comments and suggestions_
If you would please complete and return the questionnaire within two weeksof receipt of your materials, it would be greatly appreciated, as this will expediteour review of suggestions and the initiation of further action on our part.
May we express our sincere appreciation to you for participating in thisreview and evaluation of materials.
205
FIELD TEST EVALUATION FORMfor the filmstrip FAMILY PLANNING AND THE LOOP
I. AUDIENCE DESCRIPTION
1. How many people viewed the filmstrip?
menwomen
2. What was the educational level of these people? (example: 3 illiterate,2 new literate adults, 5 primary school, 2 high school, 2 higher educa-tion. )
illiteratenew literate adultsprimary school levelhigh school levelhigher education
3. What work did the people in the audience do? (example: 10 farmers,3 housewives, 2 teachers, 1 extension worker, 5 shopkeepers.)
farmershousewivesteachersextension workersshopkeepersother (please tell other professions)
4. They were primarily from a (please mark box)
village, small town
medium sized town
L1 city
5. They hadq seen several visual presentations before
q seen very few visual presentations before
206
Li never seen a visual presentation before
6. The people (or person) who viewed the filmstrip had expressed interest
in, and some knowledge of family planning.
yes
Lk7. The people (or person) who viewed the filmstrip had tried some method
of contraception before.
yes
H no
8. The people (or person) who viewed the filmstrip already had some
knowledge of the LOOP.
yes
• 1:1 no
9. The viewers ranged in age from to
10. The average age of the viewing audience was
11. The number of children of the viewers ranged from to
12. The average number of children per member of the viewing audiencewas
13. The average age of the youngest child of members of the audiencewas
14. Most of the people were
able to obtain family planning services from a mobileclinic.
lived within 3 miles of a family planning facility.
207
q lived further than 3 miles from a family planningfacility.
II. FRAME BY FRAME ANALYSIS
1. Was the audience able to identify with the activities in frames 1 and 2?
Frame 1 Frame 2
F l yes ]yes
no q no
(If not able to identify, please explain why.)
2. What did the audience (you) feel was happening in frame number 3?
3. What did the audience (you) think the group of women in frame 5represented?
4. Were most of the viewers able to recognize one of the types of familyplanning centers in frame 6?
U. If they did not recognize one of the types of family planning centersshown, what type of family planning center would be recognized?
6. What did the audience (you) feel was happening in frame 7?
7. Did the viewers seem to like or dislike the idea of seeing their neighborsat the family planning center?
208
8. Did the audience (you) think the man and woman in frame number 8looked like doctors?
9. Was anyone in the audience offended by the poster drawings in frame 10?
10. Did the viewers understand the concept of looking inside the female
body?
11. Did the viewers understand what the uterus or "'baby box"' was, and didthey find the idea believable?
12. Did the audience understand where the LOOP is placed?
13. What did the audience (you) think was happening in frame number 15?
14. Did anyone in the audience ask (or know) what the red triangle shown inframes 6, 7, 8, 9, 10, 12 and 15 represented?
15. What did the audience (you) think was happening in frame number 17 ?
16. Did the audience seem comfortable discussing (or hearing about) sexualhappiness resulting from the LOOP?
17. Did the audience (you) think the man in frames number 18 and 19 lookedlike a doctor?
209
18. Was the comparison of the LOOP to a new pair of shoes in frame number20 understood?
19. Did the comparison with a pair of shoes help the audience to understandthe side effects often occurring when a loop is inserted?
20. What did the audience (you) feel was represented in frame number 21?
21. Did the audience (you) think the family shown in frame number 24 lookedhappy ?
M. OVERALL COMMENTS
1. What was the emotional response of the audience?
q Restless and talkative, bored, puzzled
q Quiet, fairly interested
Actively interested, apparently involved
Deeply interested, exchanging comments and arguments
2. Were the members of the audience able to identify with drawn figuresin the filmstrip?
3. The information in the filmstrip was
Li right for the audience
q too complicated for the audience
210
1-1 too simple for the audience
4. Did you have to re-arrange the frames or repeat frames to make theconcept presented understandable?
5. Were you, as an extension worker, able to understand and make thenecessary substitutions (in parentheses) for frames 3, 5, 8, 9, 13, 19and 21?
6. Do you like this method of script writing (to help make the presentationmore applicable to your own situation)?
7. Were you, as an extension worker, able to supply the necessary informa-tion for frames 18 and 23 ?
8. Were you able to furnish the more detailed instructions for showing thefilmstrip to women who had al ready decided to accept the LOOP?
9. Was the script clear enough for you to understand which informationshould be added--and where it should be added--to show the filmstrip towomen who had already decided to use the LOOP?
10. Were the LOOPS included with this filmstrip passed around after thepresentation?
11. Did these LOOPS help the audience to understand this contraceptivemethod ?
211
12. Did these LOOPS help alleviate any fear of this contraceptive method?
13. Were the viewers able to answer the questions which follow the scriptafter they viewed the filmstrip?
14. What other questions were asked by the viewing audience ?
15. Did the filmstrip and accompanying discussion encourage anyone in theaudience to seek further information about the LOOP?
16. Did the filmstrip and accompanying discussion motivate anyone in theaudience to accept the LOOP as a contraceptive method?
212
Example 8: Radio, Television, and Film
Source: Fred W. Reed, Pre-testing Communications: A Manual ofProcedures (Chicago: Communication Laboratory, Commu-
nity and Family Study Center, University of Chicago, 1974),pp. 44-49, 52-57.
213
A. Pretesting the Radio Spot as Part of Regular Programming .
Procedure . Using a high quality tape recorder, record approximately onehour of regular radio programming of the same time segment during which theproposed radio spots will be broadcast. After the radio programming has beenrecorded, include the spots to be pretested in the radio program using eithermanual splicing or electronic editing. The pretester will now have a one hourtape of regular radio programming which includes one or more spots which are
to be pretested. The spots may be included with or instead of the other spotswhich might have been included in the radio program. The pretester can nowtake the tape recorder with the one hour of altered tape to the pretesting site.The pretesting may be conducted in family planning clinics where clients oftenhave to wait for one or more hours to receive services or at some other sitewhere the audience might gather.
The pretester can announce that he or she is conducting a pretest or isconducting a study on the effectiveness of radio programming. The pretestaudience can then be asked to listen to the taped program. After the point onthe tape when the spot or spots have been played, the pretester can wait a fewminutes and then begin the formal questioning.
If the pretest audience is literate, pencils and paper can be passedaround to the subjects. If it is not, then the subjects should be interviewedindividually. The pretester can now announce that this was actually a study ofthe effectiveness of commercial or informational messages on the radio. Thesubjects should then be asked to recall as many of the commercial or informa-
tionai messages as they can. (Subjects sometimes will work harder at thistask if a small prize is offered to the one remembering the most announcements.)For each commercial or spot which the subjects remember, they should beasked: What do you remember about that message? This question serves twopurposes. First, the answer will identify the particular message which thesubjects are recalling. This is especially useful if more than one spot is beingpretested. Second, by asking the subject to recall as much as possible aboutthe spot, the pretester can establish whether the subject actually understood themessage and in the intended manner.
As noted above, this is the best pretest method for establishing theextent to which the spot has achieved its awareness goals. However, it onlyshould be used in combination with one of the other methods for pretesting radiospots.
B. Pretesting Isolated Radio Spots . Pretesting isolated radio spots isstraightforward. The spots are recorded on sound tape. After the pretestermeets with the pretest respondents and explains the purpose of the pretest,each spot is played several times for the respondents. This allows the respond-ents to understand clearly the spot and to ponder the various characteristics of
the message, i. e. , the script, the voices of the actors, the cadence with which
215
the script is read, and any accompanying music. If more than one spot is beingpretested, no more than three spots should be tested and compared at one time.It is difficult for subjects to remember and to compare more than three thirtysecond spots. After each spot has been played enough times for the respondentsto understand it thoroughly, either the pretester or the subjects should completeone of the following questionnaires.
Spot Evaluation Questionnaire 1
Name of Interviewer Spot number
1. What message was this spot trying to convey?
2. Is the message of this spot announcement clear ? How many of yourfriends would understand what the sponsor wants them to understand?
Very clear : almost everybody would understand . . . . . . . . . . . . 1A little confusing : many would not understand . . . . . . . . . . . . . 2Definitel y not clear : my friends would not understand . . . . . . . . . 3
3. What is your reaction to the announcement: How much do you like it?
Excellent: like it very much . . . . . . . . . . . . . . . . . . . . . . 1Acceptable: it's o. k. , but not really good . . . . . . . . . . . . . . . 2
Poor: not good enough to be used on the radio . . . . . . . . . . . . . 3Terrible: so bad it would upset the listeners . . . . . . . . . . . . . 4
IF "ACCEPTABLE" OR "EXCELLENT"a. What do you like about this announcement? What makes this
interesting or convincing? WRITE IN THE ANSWER
IF "ACCEPTABLE," "POOR," OR 'TERRIBLE"b. What do you dislike about this announcement? What is wrong
with it?
}. What do you think the announcement should be like? If you were askedto broadcast this message by radio, what changes would you make in thewords used and how would you say it?WRITE YOUR ANSWER
5. What is your opinion of the background music (or other sounds)?
216
Like very much : appropriate for the purpose . . . . . . . . . . . . • 1
Acceptable : could be better or more appropriate . . . . . . . . . . . 2
Poor : do not like or not really appropriate . . . . . . . . . . . . . 3Terrible: so bad that it would ruin even a perfect message . . . . . 4
IF "POOR" OR "TERRIBLE"What kind of music do you think would be better or more appropriate?
WRITE YOUR ANSWER
6. What is your opinion of the voices in the announcement?Like very much : appropriate for the purpose . . . . . . . . . . . . . 1Acceptable : could be better or more appropriate . . . . . . . . . . . 2Poor : do not like or not really appropriate . . . . . . . . . . . . . . 3Terrible : so bad that it would ruin even a perfect message . . . . 4
IF "POOR" OR "TERRIBLE"What is wrong with the voices used in this announcement?
What changes in the voices do you think would improve it?WRITE YOUR ANSWER
7. What do you think is the attitude of the actors toward the person who hearsthis on the radio? WRITE YOUR ANSWER.
8. Are there any words or ideas used in this announcement that would offendyour friends and neighbors ?
No : this message uses no offensive words or ideas . . . . . . . . . 1Yes : there are words in the message which are offensive . . . . . . 2
IF "YES"What words or ideas are in the message that would offend yourfriends and neighbors? WRITE YOUR ANSWER.
Spot Evaluation Questionnaire 2
Name of Interviewer Spot number
1. In your own words, please describe what went on and what was said inthe message.
217
2. Which of the following terms would you use to describe the elementsin this spot? (Check as many of the following words as you think apply.
Mn1e vnine
_ dominatinggood humored
_ aggressivedullkind
_ masculine
_ unimaginativeconsideratesexually attractive
_ sexually unattractiveeffeminate
^_ a good fatherforcefulintelligentconfident of himselfcondescending
_ someone I would like
someone I would notlike
Female voice_ dominating
good humoredaggressive
- dullkind
feminine_ unimaginative_ considerate
_ sexually attractivesexually unattractivemasculine
_ a good motherforcefulintelligent
_ confident of herself_ condescending_ someone I would like_ someone I would not
like
Music *_ disruptive
pleasantadds to the effect
- boring
T interestingattention drawing
_ inappropriate_ should be left out*other terms may beadded as necessary
3. In your personal opinion, what was the main message of the spot? Whatdid the message want the listener to do?
4. Are there any phrases or lines that you remember in particular? Whatare they ?
5. In your opinion, did this spot seem directed to:a. people wealthier than you?b. people who are like you?c. people poorer than you?
d. all kinds of people?Why do you think so?
6. In your opinion, was this spot made especially for:a. men?b. women?c. both men and women?d. men, women, and teenagers?
218
7. Is there anyone whom you think should not hear this spot?Yes No
IF YES:What kinds of people should not hear this spot?
8. Did the spot make any claims or statements with which you don't agree
or which you do not think are true?Yes No
IF YES:What false claims or statements were made in the spot?
9. If you were listening to the radio at a friend's home and this spot wasbroadcast, what do you think your friend would say about it?
10. In your opinion, what could have been done to make this a better spot?
Analysis. After this pretest (questionnaires 1 and 2) has been adminis-tered, the results can be tabulated rapidly in the pretester's office. Bothquestionnaires have yielded reliable results with samples as small as 20 individ-uals. Therefore, the results can be tabulated by hand on an extra questionnaire.
Used in combination with the pretest which is a part of regular programming,this is an excellent diagnostic device.
These questionnaires and methods may be used when several versions ofa spot are being compared. The comparative worth of each can be establishedeither by the use of the hand tabulations of the pretest results or by asking therespondents the following questions.
1. Which of these spots would you prefer to hear on the radio?a. Spot 1b. Spot 2
c. Spot 3
2. These spots have already been used in (name of nearby area) and one
was much more effective than the others. Which one do you thinkwas the most effective?a. Spot 1b. Spot 2
c. Spot 3
219
3. Which of these spots do you think would be the most effective inthis area?a. Spot 1b. Spot 2c. Spot 3
IF THE SPOT YOU CHOSE FOR YOUR AREA IS DIFFERENT FROMTHE ONE YOU CHOSE IN QUESTION 2, WHY DID YOU CHOOSE ADIFFERENT SPOT? WRITE YOUR ANSWER IN FULL.
A. Pretesting Television and Movie Messages as Part of RegularProgramming. As noted in the discussion of pretesting radio messages, pretest-ing persuasive messages as part of the regular programming most effectiveestablishes whether a message achieves its awareness goals. The protester mayrecord a regular movie or television program, including the spots to be pretestedas part of the program. The following questions to be asked of the pretest audiencecan establish the level of awareness.
Questionnaire for Establishing Awareness in RegularProgramming
You have just watched a regular program (or movie). As part of the program
there were several messages or persuasive announcements. The announcementsasked you to buy something or to do something. We would like to know howmany of these announcements you remember. Would you please write (or tellme, if the audience is illiterate) about the announcements which you remember.Describe what went on and what was said in each announcement.
First announcement remembered
2. In your own words, please describe what went on and what was said inthe commercial.
3. Which of the following terms would you use to describe the husband andthe wife in this spot? (Check as many of the following words as youthink apply.)
220
Musictasteful
_ distracting
too fasttoo slowpleasantattention gettingboringshould be left out
Husbanddominatinggood humoredagressivedullkind
_ masculine_ unimaginative_ considerate
dominated by the wife_ sexually attractive_ sexually unattractive_ effeminate
a good fatherforcefulslow
_ intelligent_ unintelligent_ confident of himself
condescending
Wifedominatinggood humoredagressivedullkindfeminineunimaginativeconsideratethe boss of the familysexually attractive
_ sexually unattractivetired
_ a good motherforcefulslowintelligentunintelligent
_ confident of herselfcondescending
4. In your opinion what was the main message of the film ?WRITE YOUR ANSWER IN FULL.
5. Are there any phrases or lines that you remember in particular?
WRITE YOUR ANSWER IN FULL.
6. From the spot did you get the impression thata. he dominates in the marriageb. she dominates in the marriagec. they make joint decisionsd. it was unclear who dominates in the marriagee. other impression of the marriage:
Why?
221
7. Did this spot seem to be directed toa. the upper classb. the middle class
c. the lower classd. any/all classesother:
Why do you think so?
8. In your opinion was this film directed
a. primarily to malesb. primarily to females
C. to both men and womend. to men, women, and teenagersother:
9. Did you consider that this spot was realistic; that is, that it showed a
couple as they would normally act?
What parts were realistic?
What parts were unrealistic?
10. Did the spot make any claims or statements with which you didn't agree
or didn't think were true?Yes No
IF YOU SAID YES:Which claims do you think are untrue:
11. In terms of technical quality, how would you rate this spot?Excellent Good Fair Poor
NarrationPhotographyQuality of soundCoherence between
audio and visual
222
12. if you were watching TV at a friend's house and this ad came on, whatdo you think your friend's reaction would be?
13. In your opinion, what could have been done to make this a better spot?
14. Can you think of any racial or ethnic group who would be offended bythis ad?Yes No
IF YOU SAID YES:List groups who would be offended.
B. Comparing the Potential Effectiveness of f TV and Moviel Spots.
The pretester often will be trying to compare two or more messages in orderto provide information about which should be developed further or which finallyshould be used. This manual has proposed pretest methods and questions forcomparing the effectiveness of persuasive messages. The following questionshave been developed for use with TV and movie spots. The innovative pretesterwill adapt one or more of them to other kinds of persuasive communications.
Questions for Comparing Spots
1. Which of the spots tells you most about the importance of family planning?
2. Which of the spots does a better job indicating the value of family planning?
3. Which of the spots would you prefer to watch?
4. Regardless of your previous answers, which of the spots would be morelikely to make you or your friends use family planning?
223
5. These spots have already been used extensively. One has been provenmore effective than the others. Which do you think was proven the most
effective ?
6. Which spot would make you more interested in using family planning?
7. Which spot made family planning seem most valuable?
8. Which spot suggests solutions to problems you have?
9. Which spot did a better job convincing you of the ability of familyplanning to solve one of your problems?
10. Which spot would stay in your mind best?
11. Which spot made you think of your home and your family?
12. Which spot do you think women would talk about most?
13. Which spot would more people stop to look at?
14. Which spot would people understand best?
15. Which message would people believe most?
224
16. Which of the messages creates the most positive impression aboutfamily planning?
These questions cover most of the areas on which one would wish tocompare messages. Usually the pretester will be able to use only 4 or 5 ofthe questions suggested for comparison. We would suggest that the mostimportant areas for comparison are indicated by questions 1, 5, 8, 12, and 15.
225
Example 9: A National Symbol for Family Planning
Source: Excerpted from H.M. Goonetilleke, National Symbol forFamil y Planning in Sri Lanka: Pretest Report (Colombo, SriLanka: Family Planning Communication Strategy Project,
Department of Information, 1974).
227
PRETEST OF A NATIONAL FAMILY PLANNING SYMBOL
OBJECTIVES
The specific ends that should be achieved by a national Family PlanningSymbol were not defined or agreed upon. In the discussions, however, a numberof objectives were suggested in terms of possible gains that could be derivedfrom such an emblem. These suggestions could, to some extent, be used toguide the selection of criteria for the pretest.
When used on publicity material, on packages of contraceptives, onsigns at clinics, centres, etc. , the emblem would, according to one of the
notions suggested, help the public identify all the different activities and suppliesas manifestations of one and the same idea, one and the same pursuit.
The symbol may serve, in other words, to make all these differentmanifestations re-inforce each other in the minds of the public adding up to thei mage of a forceful endeavour, sponsored by the Government as well as by civilorganizations and with strong international support, and thus enhancing thei mpact of total communication efforts.
According to a similar notion the symbol would also serve to make allthose who are involved in population/family planning activities but belonging todifferent organizations, voluntary as well as Governmental, more conscious of
working for a common cause, under a more concerted and powerful programmethan that of their own organization alone.
In addition to helping establish the identify of activities and workers, thesymbol may serve, according to a third notion, to convey specific ideas pertain-ing to the family planning concept, to the programme or to a family image to bepromoted. This may be done by depicting something that would symbolize thatidea or it may be achieved by first vesting, by means of publicity, such connota-tions in an in itself nondescriptive symbol and thereafter deploy the symbol tocommunicate the connoted idea, or it may be done, as a combination of theseways of communicating, by employing a design that already has a positive emo-tional charge and vest with it the message connotations one wants to transmit.
OBJECTIVES OF THE PRETEST:
1. Find out how people understand and interpret the pictures that had beenselected for pretest. In particular, to find out to what extent theseinterpretations have any relationship to family planning or have anyemotional charge relevant to family planning.
229
2. To assess the acceptability of the different symbols for the family plan-
ning programme.
3. To find out to what extent the symbols differ with respect to people's
ability to discern, retain and reproduce their different shapes.
4. To assess preferences with respect to colour . . . .
SAMPLE
A total of 100 persons were to be interviewed. Primary considerationsfor selecting the sample were that it should include different categories of peo-ple. The different races of Sri Lanka should be represented, both rural andurban people should be included, both men and women and people of differentsocial positions. On the other hand a truly representative sample could not beselected, since the interviewing had to be confined to Colombo and surroundingareas due to travel costs . . . .
This [sample] comes close to the proportion in the total populationaccording to the Census in 1971, Sinhalese 71. 9 percent, Tamils 20. " percent,Moors and Malays 7. 0 percent, and Burghers and Eurasians 0. 4 percent.
Out of the total respondents, 74 were married and 26 were unmarried.The racial and sex distribution of the sample is given below:
Nationality Male Female Total
Sinhalese 50 20 70Tamils 9 10 19Moors/Malays 4 3 7Burghers/Eurasian 3 1 4
Total 66 34 100
As far as sex is concerned almost twice as many men as women were
interviewed ((56 as against 34). An attempt was made to have about equal num-ber but this was found to be difficult due to the fact that only one out of the fourinterviewers was a female.
When a male interviewer called on a house, the husband if he was athome usually did not like to let him talk to his wife. In view of the fact thatthe interviewing had to be completed in a short time we had to accept that thesample had a far too low proportion of women in comparison with the population
230
Education Level
No educationPrimary and secondary
Passed SSC/GCE/(O. L.)and HSC/GCE (A. L.)
Higher education
of the country. This however, applies primarily to the Sinhalese part of the
sample, as can be seen from the table above.
As our lady interviewer was a Tamil, the number of male and femalerespondents was about equal among Tamils, Moors and Malays.
No serious attempt was made to select the sample in such a way thatdifferent educational levels would be correctly represented. In each areahouses were selected in the areas surrounding the centre of the village. Houseswere picked where people were expected to be more educated or more coopera-tive or more able to understand the question. In this way the sample becameheavily biased towards higher education, as shown by the following figures:
[ Pretest] Sample Nation
- 32.225 61.8
46 5.7
29 0.3100 100.0
* Survey of Sri Lanka's Consumer Finances 1973, Central Bank of Ceylon
When classified on the basis of occupation, the respondents fall into thethree groups, upper middle class, lower middle class and working class: . . .
Upper middle classLower middle classWorking class
Pretest Sample
(percentage)235423
100
Central Bank Sample(percentage)
7. 6
7.9
79. 5
95. 5
These figures confirm a heavy bias of the sample towards higher socialclass . . . .
c). Interviewing:
The interviews were carried out by five officers of the Family PlanningCommunications Project. Out of them four were male and one was female.
231
Each of the eight symbols was shown separately to the respondentas the questions were asked. Each answer was recorded by the interviewer.Answer to open-ended questions were taken down verbally.
Most of the respondents (68 percent) were interviewed at their homes,21 percent at their work places while some other portion 11 percent at theboutiques or some other non-specific places.
The interview took 15 - 30 minutes to complete. The field work wascompleted three week-ends in the month of July in 1974 . . . .
d). Processing ;As a first step all answers in Sinhalese and Tamil languages were
translated into English. All the answers were transfered to specially designedcompact processing cards. All answers to open-ended questions were coded.The processing cards were sorted manually and cards and code marks countedand tabulated . . . .
THE FINAL DECISION
Most of the respondents accepted symbols C, D, E, F, and G, represent-ing a formal unity. One section [of the questionnaire] certified its suitability.Some others were rejected as unsuitable. However, the symbol B gives theidea of a family in Sri Lanka and . . . represents some national aspirationwhich inspired 93 percent out of the hundred respondents to select it as suitablefor [a] national symbol.
232
Fee
MIL! PLANNSG
a1 IATfGY Pit—
r0
C
I
M
Pre -Tested Symbols
icy
C«
