Commentaries

Reliability and Validity Issues in School Ethnography and Qualitative Research

Judith Mc Laugh I i n

n recent years, more emphasis has been placed on I using qualitative data in school ethnographies and in evaluating school health and health education pro- grams. Qualitative research is referred to as case study research, field research, and anthropological research as well as ethnography. It is characterized by participant and nonparticipant observation, a focus on natural set- tings, use of participant constructs to structure the research, and investigator avoidance of manipulating study variables.’ Qualitative/ethnographic research differs from traditional quantitative reseach and its con- tributions to scientific programs are in such differences. These differences may involve the data collection that necessarily preceeds hypothesis formulation and revision or may focus on descriptive investigation and analysis.

By introducing into the research framework the sub- jective experiences of the participants and investigators, ethnography may provide a depth of understanding that is lacking in other approaches to investigation.* Though these approaches are most common in sociology and anthropology, they are used to some extent by all social science disciplines. These studies may be supplemental, in which they verify or supplement quantitative results or reveal participant satisfaction with a program,’ or they may clarify, refine, or validate a construct.‘ A paucity of such studies exists in school health education, though this situation is changing. The question is not whether qualitative techniques should be used, but to what extent, when, and how they should be used. According to Mullen and Iverson,’ in certain circum- stances qualitative methods may be the techniques of choice.

This commentary examines several validity and reli- ability issues specific to the qualitative method. These strategies are incorporated throughout the investigative process including the study design, data collection, data analysis, and presentation of findings. Qualitative methods have been criticized as subjective and soft.’J One factor contributing to this criticism is that many qualitative researchers work alone, perhaps without a standardized research protocol, and research tradition shows that human judgments are less accurate than Judith McLaughlin, PhD, Assistant Professor, &pt. of Health and Sqfety Education, University of Georgia, Athens, GA 3W02. Presented during the Research Council presentations, 59th Annual Convention of the Amencan School Health Association. Litfle Rock, Ark., October 1985.

actuarial Also, journal reports often emphasize the findings and description of a study and devote less attention to how the researcher obtained those findings; readers may not know how conclusions were generated from 500 pages of field notes. A part of this problem relates to the strictures imposed by journal-length manuscripts and the tradition of artful presentation of results.

However, the value of scientific research depends on the ability of the researcher to demonstrate credibility of findings, regardless of methods used for data collection and analysis. Addressing credibility problems in et hno- graphy and qualitative research requires different tech- niques than those used in experimental studies since the two approaches differ in formulation of research prob- lems, nature of research goals, and application of results. The questions in this paper address several credibility issues in qualitative and ethnographic re- search. Strategies are suggested to reduce the multiple sources of potential analytic bias that can weaken or invalidate results and to assure the basic quality of the data.

Are the findings representative of a more general phenomenon?B.10

Did the researcher overrely on information from nonrepresentative subjects? Are dramatic events present in the data that may not be representative of what really happened? Did the research rely on an emerging expla- nation based on a faulty sample of subjects and events? Strategies that address nonrepresentativeness include use of other readers of the data or additional researchers to determine if evidence had been accumulated gradual- ly from within, not drawn validly from without. A second strategy assumes nonrepresentativeness and extends the universe of study by increasing the number of cases, searching for negative or extreme cases, sorting cases systematically, and then filling in with more of the weakly sampled cases. Random sampling is recom- mended.

Are researcher effects present?I1-l3 When researchers study a phenomenon, both the re-

searcher and the subjects influence each other to a certain degree. The researcher may create behavior in participants such as switching to a stage persona for the benefit of the researcher, perhaps to conceal negative aspects and weaknesses they are reluctant to reveal. In other cases, the researchers are seen as outsiders. In

Journal of School Health May 1986, Vol. 56, No. 5 187

these circumstances, they may make biased inferences and observations. I‘ Alternatively, researchers may be deceived into blind acceptance of the subjects’ version of events. Strategies to address researcher effects in- clude: I

Remain at the site long enough to acquire a lower profile,

Use unobstrusive measures where possible, lnform subjects why you are there and be unequi-

vocal about what is being studied, how information is collected, and what is going to be done with the infor- mation,

Ask a subject to observe you for your influence on the subjects and vice versa.

Include subjects who have different points of view than the mainstream,

Use several data collection methods, Submit field notes to another researcher to detect

where and how bias is occurring, and Keep research questions firmly in mind, do not

exclude them in favor of something more alluring or dramatic. Generally, the more time to conduct the study, the greater the likelihood of identifying the most explanatory factors and the less danger of problems with representativeness and researcher effects.

Do the findings withstand different methods and sources of data collection?’ ’

Webbi6 described triangulation as support of a find- ing by showing that independent measures agree with the finding or at least do not contradict it. Different sources that might occur in a school health education program are the testimony of teachers using a program, testimony of teachers not using the program, testimony of pupils, observations of the program, observation of classrooms not using the program, and analysis of the program manual and materials. Strategies to utilize tri- angulation as a way to avoid bias include:

Using indices that are independent, and using several types and sources including histories, docu- ments, and scores,

Using different people with different roles, both mainstream and deviant,

Searching for new sources of data, new inform- ants, or comparable events, and

Triangulating with different researchers who are taking parallel measures or confirming them.

Are some data “Wronger” or “weaker” than other data?’’.‘

The qualitative researcher can employ this difference in verifying conclusions because stronger, more valid data strengthen emerging theories. Conclusions based on weak data should be discarded. Miles and Huber- man’ suggest several markers analysts can use to decide to give more weight to some data than other data (Figure 1). Another strategy involves keeping a log of data quality issues in the form of reflective or marginal remarks on the field notes. This strategy can determine if other methods are needed to validate the data.

Are rival explanations and exceptions or negative evidence present ?i8-20

It is tempting to disregard conflicting information, but careful scrutiny of such evidence can test or

strengthen a finding and can protect against self- selecting bias. Invest the time to consider other possibilities that might counter an emerging theory. Negative evidence and exceptions must be pursued actively and rival explanations that account for the phenomenon should be explored carefully.

Some authors suggest the search for rival explana- tions usually is more thorough in qualitative research then in survey research or laboratory studies.iJi They suggest field researchers should look for the most plaus- ible, empirically grounded explanation of events from among several alternative accounts. Discounting rival explanations too early produces systematic measure- ment error and a favorite theory may be fostered by selectively choosing supportive evidence. To address the issue of rival explanations, have a colleague not familiar with the study challenge emerging theories. Also, examine further any discrepant information, search for alternative cases the information builds, then analyze it further.

Is the relationship bet ween l wo variables spuri0us?22~~~

Before concluding factor A is related causally to factor B, consider the possibility that another factor may be causing both A and B to occur. Again, a knowl- edgeable but detached colleague can assist in the search for other variables that might cause or influence a cor- relation.

Can the finding be repli~ated.=~.‘~ Once a hypothesis emerges and is confirmed by

interviews, observations, and documents, there is the potential to place more logic and coherence into the meaning of the events than they warrant. Replication reduces this potential problem. A finding that can be reproduced, either by oneself or others, probably is dependable. Data from new subjects, new settings, and new events can confirm existing data. By testing the emerging hypothesis at another site, an if/then situation emerges. If certain findings exist at one site, then similar findings should exist at the second site. Emerging pat- terns at one site should be tested at other sites. Has feedback been obtained from particbants?‘J Bronfenbrenner2‘ classified feedback to informants

as a source of “phenomenological validity.” Stakez’ maintains that participants have the right to know what the researcher found. Local respondents can act as a panel of judges, evaluating singly and collectively the

Figure 1 Data Collectlon Clrcumstances that may Strengthen or

Weaken the auallty SbOW OJb WMk Om

Collected later or after

Observed or reported firsthand Observed behavior. activities Collected in official or

informal setting Volunteered to tieldworker Respondent is alone with

fieldworker

Miles MB. Huberman AM Oualirarive Data Analysis A sourcebwk ol new

Collected early during entry

Reports or statements Collecled in informal setting Prompted by lieldworker question Respondent in presence ol others

repeated contact Heard secondhand

or in group settings

188 Journal of Sehool Health May 1986, Vol. 56, No. 5

study findings. The findings must be described clearly and systematically and placed into the context of local experiences and perceptions. A confounding factor is that the participants will not always agree with the re- searcher or even with each other. This is known as the Rashomon effect.' However, feedback is invaluable and can provide more information about the site and the accuracy of theories or conclusions.

CONCLUSION Several benefits can be gained from carefully evalu-

ating critical components of a qualitative or ethno- graphic study. Ethnographic evidence can be compared to results of other methods such as surveys and experi- mental and quasiexperimental techniques. A systematic evaluative framework allows evaluators of proposals, journal and conference submissions, and dissertations to determine the quality of an ethnography. Likewise, scholars engaged in comparative analyses of ethno- graphic and qualitative evidence can determine the rela- tive merit of data quality and investigators involved in designing or reporting empirical studies can evaluate antecedent work in their areas. Finally, and most importantly, the researcher is assured that conclusions are derived from faithful and accurate renditions of participants' experiences and that they are grounded empirically in the most plausible and relevant explana- tions possible.0

References

I . Miles MB, Huberman AM: Qualirative Data Analysis: A sourcehook of new merhods. Beverly Hills, Calif, Sage Publications, 1984.

2. Goetz JP , Lecompte MD: Ethnography and Qualitative Design in Educational Research. Orlando, Fla, Academic Press, 1984.

3 . Christenson PD: An evaluation of quantitative and qualitative assessment techniques on the impact of the SERR curriculum. J Sch Healrh 1985;55(5):200-204.

4. Mullen PD. Reynolds R: The porcntial of grounded theory f o r health education research. Healrh Educ Mono 1978;6(3):280-294.

5 . Mullen PD, lvcrson D: Qualitativc methods l o r cvaluaiinp research in health education programs. Health Educ 1982;13(3): I 1-18.

6. Meehl P: Clinical versus stalistical prediction. J Exper Res Personality 1%5;63( 1):8I-97.

7. Oskamp S: Overconfidence in case-study judgments. J Counsel

8. Schatzman L, Strauss AL: Field Research: S/raiegie..v ,for a

9. Wolcott HF: Criteria for an ethnographic approach to research

10. Fuchs E: Teachers Talk: Views from inside city .schoolr. New

1 I . Magoon AJ: Constructionist approaches in educational

12. Kaplan D, Manners RA: Culture Theory. Englewood Cliffs,

13. Erickson F: What makes school ethnography ethnographic?

14. Yoors J: The Gypsies. New York, Simon and Schuster, 1967. I S . Glaser BG, Strauss AL: Discovery of Grounded Theory:

Slralegies for qualitarive research. Chicago, Aldine. 1967. 16. Webb EJ. Campbell DT, Schwariz RD, Sechrest L: Unobrru-

sive Measures. Chicago, Rand McNally, 1965. 17. Denzin NK: The Research Act: A theoretical inrroduction 10

sociological methods, 2nd ed. New York, MrGraw Hill, 1978. 18. McCutcheon G: On the interpretation of classroom observa-

tions. Educational Research 1981;10(5):5-10. 19. Mehan H: Learning Lessons: Social organication in rhe c'luss-

rooni. Cambridge, Mass, Harvard University Press, 1979. 20. Robinson WS: The logical structure of analytic inducrion. Am

Soc Rev 1951;16:812-818. 21. Filstead WJ: Qualitative methods: A needed perspective in

evaluation research, in Cook TD, Reichardt CS (eds): Qualitative and Quanritative Methods in Evaluation Research. Beverly Hills, Calif, Sage Publications, 1979. pp 33-48.

22. Scriven M: Evaluation perspectives and procedures, in Popham WJ (ed): Evaluation in Education: Current applicurions. Berkeley, Calif, McCutchan Co, 1974. pp 1-93.

23. Pelto PJ, Pelto GH: Anthropological Research: The srrucrure of inquiry. Cambridge, England, Cambridge University Press, 1978.

24. Campbell DT: "Degrees of Freedom" and the case study, in Cook TD, Reichardt CS (eds): Qualitative and Quanritarive Merhods in Evaluarion Research. Beverly Hills, Calif, Sage Publications. 1979.

25. Stake R, Easley J (eds): Case Studies in Science Education. Urbana, 111, Center for Instructional Research and Curriculum Evaluation, 1978.

26. Bronfenbrenner U: The experimental ecology of education. Teachers' College Record 1976;78(2): 157-1 78.

27. Stake R: Evaluating Educarional Progranis: The need and /he response. Washington. DC, OECD Publications Center. 1976.

PSYChOl I 965;29(3):261-265.

nalural sociology. Englewood Cliffs, NJ. Prentice Hall, 1973.

in schools. Human Organizarion 1975;34:111-127.

York. Doubleday, 1969.

research. Rev Educ Res 1977;47:65 1-693.

NJ, Prentice Hall, 1972.

Anthro Educ Q 1973;4(2):10-19.

pp 49-67.

School-based Clinics Conference The Support Center for School-based ClinicsKPO will hold its third annual

conference at the Radisson Hotel Denver, Oct. 5-7. The conference precedes the American School Health Association Convention, Oct. 8-1 I , at the Radisson. The Support Center assists current and emerging school-based clinics through training, technical assistance, policy analysis, advocacy, and publications. For conference information, contact: Support Center for School-based ClinicsKPO, Suite 1200. 1012 14th St., NW, Washington, DC 20005.

ASHA Study Committee Seeks Members Health professionals interested in serving as a member of the ASHA Study Committee

on Health Guidance in Sex Education should submit a resume and letter stating their interest to: Judy C. Drolet, PhD, Dept. of Health Education, Southern Illinois University, Carbondale, IL 62901. The Study Committee on Health Guidance in Sex Education is comprised of individuals from various school health disciplines who serve as a medium for discussion, promote research topics in sex education, and assist and advise ASHA on issues that contribute to favorable sexual health for the school-age population.

Journal of School Health May 1986, Vol. 56, No. 5 189