Qualitative Evaluation Issues in Funded School Health Projects

Judith McLaughlin, Sandra L. Owen

n addition to logical positivitism, reductionist exper- I iments, broad-based surveys, operationalized defini- tions, and statistical analyses, school health research now includes phenomenology, naturalistic inquiry, description, and inductive analysis.’ Educational re- searchers acknowledge the contribution of qualitative approaches such as participant observation, ethnogra- phy, and in-depth interviewing. The increased number of journal articles addressing or employing qualitative methods, as well as the number of educational research courses that focus entirely on qualitative approaches, indicate the importance of this paradigm in educational research. Some funded school health projects include ethnographic inquiry as part of the research design.

Ethnography involves the study of people and their culture.’ Fieldworkers ask questions and participate in the culture, striving to understand it from the perspec- tive of the people who live in it, and to describe it critically from the trained perspective of their discipline. Data consist of people’s words and interpretations of events and activities. Presentation of the findings also employs description that is rich in quotations. Research typically occurs in the field, at sites where programs are conducted. Data are collected in the environment of those being evaluated. Analysis and design proceed inductively.

Not limited to starting from predefined goals or from an official description, qualitative evaluators as fieldworkers describe programs as they observe them occurring. The emphasis reflects process - how events happen, rather than whether or not a particular out- come was reached. There is a concern for meaning - how program participants see and understand what happened. People at all levels and positions provide data concerning what the program or curriculum meant to them. Administrators’ views receive no more or no less influence than staff views. Unanticipated, as well as anticipated, occurrences are described.

As a primary method for cultural anthropologists, fieldwork increasingly is a research method chosen by

Judith Mcl.oitghliti, PhD; atid Sandra L . Owen, BSN, MEd, F A S H A , Dept. of I k v l i h and Safety Erlircation, Stegetmti Hall, The Universitv nf Georgia, Athens, G A 30602. Pre.sented at the 6 I s i annual American School Health Association Cot1 vetitioti diiritig the Special Session on Research Method,c and I.wie.s, Denver. Colo.. Oct. 9, 1986. This article wm s i rh i i t l ed N o w t i h e r 26, 1986, and acceptetl,for puhlica- l ion Jat1itur.v 26, 1987.

social psychologists, sociologists, nurses, educators, and others who seek insight into a particular subculture, that of alcoholics, prisoners, the chronically ill, parents, teachers, and school-age children.’ Fieldwork is not confined by experimental design, rigid and predetermin- ed observation, or interview schedules, treatments, or external measuring instruments. Fieldworkers are the measuring instrument, and the method by which they collect data creates a natural human relationship with one or more members of the group being studied to learn their culture by interacting with them, talking with them, and observing and recording them.

The protocol for collecting ethnographic data for evaluation obtained through fieldwork has a unique set of issues to consider, including creating respectful rela- tionships with research participants, obtaining valid data from them, assuring privacy and confidentiality, interpreting data accurately, and disseminating data responsibly. When the evaluation receives support from a funding agency other difficulties such as degree of confidentiality, publication rights, and contractor- enforced limits may e m e ~ g e . ~ With funded projects, evaluators receive payment. Thus, evaluators are guided not only by responsibilities of the project, but by expectations of the contractor. Four such points are of particular concern.

Who owns fieldwork data? Ownership of data should be agreed upon before

evaluations begin. For federal contracts and most state contracts, data become the property of the government unless the contract states otherwise. For grants, how- ever, data become the property of the researcher. Such data may or may not be accessible through the Freedom of Information Act of 1976, which will be discussed in another section.

Contractors may feel the agency paid for the data, so they can claim it. Nevertheless, research ethics suggest study participants must be protected from those who make decisions about their future.’.” More importantly, what transpires between evaluators and study participants should be confidential. Participants should feel what they say to qualitative evaluators will not be attributed to them in reports or conversations. Otherwise, they may not be honest. Field notes contain the words of study participants that may be critical, pro- fane, or blasphemous. I f read in or out of context,

journal of School Health March 1987, Vol. 57, No. 3 119

grounds may exist for administrative retribution against the respondent or interviewee.

The notion that an ethnographic approach to evalu- ation is innocuous compared to a clinical survey approach should be dispelled. Survey data can be at least aggregated to protect individual identities. Both the contractor and the study participants should under- stand that the qualitative evaluator will remain impar- tial and will not act as an informant. Evaluators should think carefully before relinquishing information about specific individuals. Verbatim remarks in reports from which the identity of the respondent can be inferred should be avoided. Despite taking adequate precau- tions, Chambers’ suggests confidentiality may be diffi- cut to guarantee, but that every effort must be made to ensure and enforce confidentiality and anonymity.

What are the limits of confidentiality? Legal or ethical concerns related to federal funding

for evaluation include laws pertaining to the Freedom of Information Act of 1976. The act provides for indi- vidual access to information pertaining to self, and enables individuals and groups to obtain legal access to most nonclassified information collected with public funds.B Qualitative data from most governrnent-funded projects are subject to these provisions including field notes which, i f made public, would compromise prorn- ises of confidentiality and potentially place research subjects at risk. Some local, state, and federal govern- ment agencies including the U.S. Congress, the General Accounting Office (GAO), the CIA, and law enforce- ments agencies, also have access to sensitive informa- tion pertaining to individuals, including data collected by social scientist^.^ In a special issue of Social Prob- lems devoted to research ethics, Trend’O described a case study involving a large social experiment in housing sponsored by the U.S. Dept. of Housing and Urban Development (HUD). Experimenters tested the feasibil- ity of using direct cash payments to low-income house- holds by allowing them to choose their own housing on the open market, rather than being placed in housing projects. To evaluate the experiment, several methods of data collection were used by a research firm, includ- ing ethnographic interviews. Extensive information from individual households, including how the money was spent, was collected under written assurances of confidentiality. Trend comments,

“Looking back, I’d say we observers were naive. We blithely filled out our function logs according to a prearranged schedule and turned in copies of our field notes to the headquarters . . . I remember that one of the on-site observers had worked out some sort of crude coding system to hide identities from casual onlookers. For myself, I simply used people’s initials - more because it saved writing than anything else. Even i f we had kept our notes fo r ourselves as personal property we still would have been adjudged naive by any objective source; few of us questioned the guarantees of confidenti- ality we so forthrightly gave. We were unaware of the possibility of government agencies, other than HUD, being interested in data from the experi- ment, whether from field notes or not . . . . 1 ,

“Each of us lived at the site for approximately 18 months. We watched agency activities day in and day out. We worked with participants. Like good ethnographers we used the vacuum cleaner approach and gathered extensive data, more than was needed for a program evaluation.” The GAO conducted an audit of the $200 million

used in the experiment to determine if the data gathered by the evaluation contractor were accurate and reliable. This audit placed two principles in conflict. The Privacy Act of 1974, which ensures the individual’s right to privacy, was in conflict with the Federal Property and Administrative Services Act, which authorizes the GAO to examine records in an audit. After months of negotiation, a compromise allowed a GAO audit involv- ing a sample of participants who consented to be inter- viewed and to have their files examined.

The case contained a conflict of legitimate interests. Disagreement was intensified because the project was important from a policy standpoint. Confidentiality was promised to obtain reliable information. At the same time, the GAO sought to confirm the findings as valid and reliable. Consequently, Trend admonished funded researchers to read contracts carefully before they sign. If qualitative evaluators work for or as a private contractor, and the contract indicates the data are to be given to the government after the study, Trend suggests caution. Despite good intentions from the sponsoring agency, the amount of confidentiality that can be promised is limited. Trend also suggested evalua- tors may not wish to obtain certain data. I f data have not been collected, they cannot be released.

Can interpretations of qualitative data decide a program’s success or failure?

Qualitative methods constitute an evaluation method in their own right.ll.l* They provide rich, illustrative reports that focus on description. This focus heightens the persuasive power of evaluation results, especially when evaluators provide clear explanations of method - what and how data were collected (by and from whom), as well as how data was recorded, ana- lyzed, and summarized. Bogdan and Biklen* suggest contractors may expect a judgment from evaluators on a program’s success or failure based on qualitative evidence alone, in addition to how well a program’s goals are being met. They write:

“The contractor who insists that you answer the question, “Is the program working well?” has to be satisfied with the answer, “That depends on how you look at it ’’ . . . . There are several ways to avoid this conflict. First, the best cure ispreven- tion. Make it as clear as possible at the outset of your research that the focus of your work is description or documentation rather than judg- ments of success or failure. In other words, try to reiterate in your agreement that your goal is not to provide in formation on whether the program is good or bad. This position will not preclude evaluation of a program’s impact. ’’ Another suggestion to improve the power of the

evaluation judgment is sequential or simultaneous (tri- angulated) use of both quantitative and qualitative

120 journal of School Health March 1987, Vol. 57, No. 3

evaluation approaches.’ Qualitative techniques can be used initially to define, conceptualize, or operationalize variables, for formulating hypotheses to be tested later quantitatively, or to explain or understand results ob- tained from a quantitative instrument. Data triangula- tion is better ensured by a mix of quantitative and quali- tative data, which also counteracts shortcomings of each approach used alone.”

What happens to the report? In studies sponsored and funded by public agencies

or private industries, controlling the use of data is difficult for the evaluators.14 In some cases, reports are submitted to the contractor and never released or made public. Some federal agencies can refuse to allow field- workers the right to publish their work or to present findings at conferences. Punch” noted that sources that sponsor research often believe the knowledge gained is for their exclusive use and may resent its widespread publication. In other cases, publication of the re- analyzed data represents a contribution to knowledge in the social sciences.I4 Bogdan and Biklen’ advise qualita- tive evaluators not to relinquish too many publication rights, or at least to be aware some agencies will not give these rights. I f evaluators have plans for publication, negotiate with the contractor before commencing the evaluation.

Qualitative evaluators also must consider the nature and extent of possible harm or wrong to persons from publication of data related to their behavior. Vidich et all6 pointed out the importance of establishing a clear understanding between evaluators and study partici- pants about matters pertaining to publication and confi- dentiality, by stating:

“I t may well be, on occasion that, at an early stage of a project, there is a temptation to provide greater assurances concerning anonymity than are justvied in view of any use of data in published form. Sometimes this is done in order to “get in ’’ with the thought that, once in matters can be resolved later. I t seems to me that this is not appropriate and if, in a given situation, agree- ments cannot be arrived at which are satisfactory to the research and to those individuals within the situation which is to be studied, research oppor- tunities must be sought elsewhere. ’’

IMPLICATIONS FOR S C H O O L HEALTH RESEARCH

These issues emphasize the importance of contrac- tual relationships incurred while engaging in research, including school health program evaluation. Chambers9 suggests that social scientists performing fieldwork often resist formal contracts on the grounds that con- tracts might be detrimental to the informality existing among themselves, their sponsors, and their study sub- jects. However, the absence of a formal contract may not abrogate existing legal obligations. Contracts that specify clearly the obligations and responsibilities of sponsors, researchers, and study participants may be the

best mechanism to ensure protection to all parties involved.

Most domestic fieldwork is subject to regulation to protect human subjects, to protect the public’s right to know, and to protect the individual’s right not to be known. The regulations are both complementary and potentially conflicting. Evaluators must be aware of laws that govern their work and do their best to ensure confidentiality and anonymity.

The qualitative approach probably will continue to be part of evaluation designs in school health program research. However, school administrators and policy- makers accustomed to wading through statistical reports usually do not equate the qualitative approach with “legitimate” research. Ianni” noted:

“Most educational administrators still regard anthropological studies of schools as insightful, empathetic descriptions, which they do not trust because they are so Anderstandable and inevitably they send them on !o the social studies teacher for classroom use. ”

Nevertheless, as these studies increase, investigators who use ethnography in evaluation designs should know the laws that govern their work, should understand that grant monies necessitate careful planning and negotia- tion before the study begins, and that a legally enforce- able contract should outline these conditions. n

References I . Mullan PD, lverson D: Qualitative methods lor evaluation

research in health education programs. Hedlth Educ l982:l3(3):lI-I8. 2. Bogdan RC, Biklen SN: Qualitative Research for Edrrcation.

Boston, Allyn and Bacon, 1982. 3. Deloria V: Our new research society: Some warnings to Focial

scientists. Social f r o b 1985;27(3):265-271. 4 . Wax RH: Doing Fieldwork: Warnings and advice. Chicago,

University o f Chicago Press, 1971. 5 . Reynolds PD: Ethical Dilemmas in Social Science Research.

San Francisco, Jossey-Bass, 1979. 6 . Patton. MQ: Qualirarive Evaluation Methods. Beverly Hills,

Calif, Sage Publications, 1980. 7. Chambers E: Working for the man: The anthropologist in

policy-relevant research. Human Org 1977;36(3):258-267. 8. Reiss AJ: Governmental regulation of scientific inquiry: Some

paradoxical consequences, in Klockars CB, O’Connor FW (eds): Deviance and Decency. Beverly Hills, Calif, Sage Puhlications, 1979,

9 . Chambers E: Fieldwork and the law: New contexts for ethical decision making. Social f r o b 1985;27(3):330-341.

10. Trend MG: Applied social research and the government: Notec on the limits o f confidentiality. Social f r o b 1985;27(3):342-349.

I I . Goetz JP, LeCompte MD: Ethnography and Qualitative Design in Educational Research. Orlando, Fla, Academic Press, 1984.

12. Lofland J , Lofland LH: Analyzing Social Settings: A guide lo qualirative observation and analysis. Belmont Hills, Calif, Wadsworth, 1984.

13. Qualitative Evaluation of Health Promotion Programs. Baseline 1986; 1 ( 10): 1-5.

14. Johnson CG: Risks in the Publication o f Fieldwork, in Sieber JE (ed): The Ethics of Social Research: Fieldwork. regularion. and publicarion. New York, Springer-Verlag. 1982, pp 71-96.

IS. Punch M: The Politicsand Ethics of Fieldwork. Beverly Hills, Calif, Sage Publications, 1986.

16. Vidich A , Bensman J , Risky R , Ries RE, et al: Freedom and responsibility in research: Comments. Human Org 1958-1959; 17:2-6,

17. lanni F: Anthropology and educational research: A report on federal agency programs, policies, and issues, in Committee on Anthropology and Education (eds): Reporr and Working Papers. Washington, DC, National Academy o f Education. 1978, pp 427-488.

pp 61-98.

/ournal of School Health March 1987, Vol. 57, No. 3 121