Chapter 11: Quasi-Experimental Designs

A. LEARNING OUTCOMES. After studying this chapter students should be able to: Describe how quasi-experiments differ from true experiments. Discuss the advantages and disadvantages of quasi-experimental designs. Describe the basic features of one-group posttest-only, one group pretest-posttest, and simple

interrupted time series-designs. Discuss the key threats to the internal validity of each of the three designs. Describe the features and threats to the internal validity of posttest only, pretest-posttest, and

time-series designs that have nonequivalent control groups. Explain why switching replication designs are generally considered to be strong quasi-

experimental designs. Explain how switching replications are incorporated into pretest-posttest and time-series

designs, and discuss their advantages and limitations. Discuss the benefits and risks of conducting program evaluations. Describe five types of program evaluation. Identify four phases of program diffusion and factors that contribute to successful diffusion.

B. KEYWORDSAttrition Process evaluationContamination Program diffusionEfficiency Assessment Program evaluationHistory Program theory and design assessmentInstrumentation Quasi-experimentMaturation Regression to the meanNeeds assessment SelectionOne-group posttest-only designOne-group pretest-posttest designOutcome evaluationPosttest-only design with a nonequivalent

control groupPretest-posttest design with a nonequivalent control group

Selection interactionsSimple interrupted time-series designSimple interrupted time-series design with a nonequivalent control groupSwitching replication designTesting

125

126 CHAPTER 11: Quasi-Experimental Designs

C. BRIEF CHAPTER OUTLINE

I. What Is a Quasi-Experiment?A. Quasi-Experiments And Experimental ControlB. Advantages and DisadvantagesC. Diagramming Quasi-Experimental Designs

II. Designs Without a Control GroupA. Basic Threats to Internal ValidityB. One-Group Posttest-Only DesignC. One-Group Pretest-Posttest DesignD. Simple Interrupted Time-Series Design

III. Designs with a Nonequivalent Control GroupA. Basic Threats to Internal ValidityB. Posttest-Only Design with a Nonequivalent Control GroupC. Pretest-Posttest Design with a Nonequivalent Control GroupD. Simple Interrupted Time-Series Design with a Nonequivalent Control Group

IV. Switching Replication DesignsA. Pretest-Posttest Design with Switching ReplicationB. Switching Replication with Treatment Removal

V. Program EvaluationA. Benefits and Constraints of Program EvaluationB. Types of Program EvaluationC. Program Diffusion

D. EXTENDED CHAPTER OUTLINE*Much of this summary is taken verbatim from the text.

IntroductionIn this chapter various quasi-experimental research methods are described. Quasi-experimental

designs are considered nonexperimental because the purported “cause” is either a subject variable, or the participants were assigned to treatment conditions in a nonrandom fashion.

Part I: What Is a Quasi-Experiment?A. Quasi-experiments and experimental control. Quasi-experiments often look identical to true

experiments but they differ from them in that there is less experimental control. Lack of control may occur from one or all of the following: a. not being able to control the independent variable (i.e., using a subject variable instead of a

true independent variable [IV]),

Quasi-Experimental Designs 127

b. not having control over factors associated with the dependent variable (DV), c. extraneous variables, and d. participant characteristics.

B. Advantages and disadvantages. Just because quasi-experiments do not have the degree of experimental control that experiments do does not mean they are not valuable. First, they enable scientists to examine research topics using both basic and applied approaches. An example of applied quasi-experimental research is examining social programs to improve the lives of human beings. The main disadvantage of quasi-experiments, on the other hand, is that they are vulnerable to threats to internal validity because of lower levels of experimental control.

C. Diagramming quasi-experimental designs. Campbell & Stanley (1966) recommend diagramming quasi-experiments alpha-numerically. X is used to indicate the treatment and O the observation. The order of Xs and Os indicate the temporal order of the design, and numerical subscripts are used to indicate specific observations when there are more than one.

Part II: Designs Without a Control GroupOne way in which quasi-experiments may lack experimental control is by not including a true control

group in its design. A. Basic threats to internal validity include: history, maturation, testing, instrumentation, regression to

the mean, and attrition. These threats were described in more detail in Chapter 10.B. In a one-group posttest-only design, a treatment occurs and afterward the dependent variable is

measured once (X, O1). For example, people may participate in a training program and then be asked whether they believed it was effective for them personally. Because there is no control group (or pretest measure) one cannot conclude that the training itself contributed to any change in behavior.

C. In a one-group pretest-posttest design a dependent variable is measured before and after treatment (O1 X O2). The addition of a pretest makes it possible to assess the amount of change in the dependent variable but it still does not enable one to draw valid conclusions about the treatment’s direct impact on the behavior. Possible alternative explanations such as history, maturation, testing, instrumentation, regression to the mean, and attrition might also be responsible for the change in behavior. Researchers also modify pretest-posttest designs to suit their goals by adding follow-up observations. The advantage of this is that it allows researchers to examine whether any immediate posttreatment changes persist over time. Second, an intervention may have delayed effects that would not be detected by using only one posttest.

D. In a Simple interrupted time-series design a dependent variable is repeatedly measured at periodic intervals before and after a treatment. The key is to have enough observations to establish both pretreatment and posttreatment trends in the dependent variable (O1 O2 O3 X O4 O5 X O6). Advantages of this design include an increase in the design’s internal validity, and that they also help determine whether maturation, testing, and regression are potential confounding variables.

Part III: Designs with a Nonequivalent Control GroupTreatment and control groups are nonequivalent when (1) it isn’t possible to randomly assign

participant conditions, or (2) random assignment is used in a way that cannot be assumed confidently to create equivalent groups at the start of a study.

128 CHAPTER 11: Quasi-Experimental Designs

Quasi-Experimental Designs 129

A. Basic threats to internal validity. Selection refers to situations in which, at the start of the study, participants in the various conditions already differ on a characteristic that can account for the results of the study. Selection becomes an important issue in any study that has two or more groups and is particularly important in quasi-experiment designs because the groups are nonequivalent by design. The issue of selection may interact with other threats to internal validity in the following ways:a. In selection x history participants in one group experience outside events that the other group

does not.b. In selection x maturation the two groups have different maturation rates.c. In selection x testing one group experiences testing effects that the other group does not.d. In selection x regression when one group is selected on the basis of a more extreme score than

another group it’s likely that the group’s posstest score will reflect regression to the mean.e. In selection x attrition the rate of attrition differs between the groups.

B. In a posttest-only design with a nonequivalent control group participants in one condition are exposed to a treatment, a nonequivalent group is not exposed to the treatment, and scores from both groups are obtained after the treatment ends: X O1

O2This design is the weakest quasi-experimental design with a control group. An example of a posttest-only design with a nonequivalent control group is from Wood et al., (1992). Wood and colleagues asked whether San Francisco residents suffered from more nightmares following a devastating earthquake in the city. To that end they measured nightmare data from San Francisco-area students and compared it to data collected from Arizona-area students who had not experienced an earthquake. In this example, nightmare data were not collected from either group pre-earthquake so it is unclear whether the two groups were equal at the outset of the study. Factors other than the earthquake are potential alternative explanations for the differences in nightmares that the authors observed.

C. In a pretest-posttest design with a nonequivalent control group, the experimental and control group are measured at pretest and posttest but only the experimental group receives the treatment between the tests. The advantages of this design include the ability to determine exactly how different the experimental and control groups were prior to treatment. Second, it helps one understand how much a dependent measure changes due to the treatment. Finally, pretest scores help identify whether there is regression to the mean and attrition in a study.

D. In a simple interrupted time-series design with a nonequivalent control group, a series of pretreatment and posttreatment scores are obtained from the experimental and control groups. The experimental group receives a treatment and the control group does not. This design also enables one to examine how different each group is prior to the introduction of the independent variable. In addition, this design has all of the advantages of other time-series designs.

Part IV: Switching Replication DesignsIn a switching replication design, one group receives a treatment and a nonequivalent group initially

does not receive the treatment but then is exposed (i.e., switched) to it at a later point in time. In the standard switching replication, the initial treatment group continues to receive the treatment throughout the study. Types of switching replication designs include:

130 CHAPTER 11: Quasi-Experimental Designs

A. Pretest-posttest design with switching replication. The most basic design is one in which there are two groups that are observed/measured three times. The first group is given the treatment between the first and second observation and the other group is given the treatment between the second and third observation. The advantage of this design is that it allows the researcher to replicate the findings in a single study. On the other hand, a lack of replication of results would provide information about threats to internal validity.

B. Switching replication with treatment removal. In this design the group that received the treatment first does not continue to receive the treatment throughout the study. This design enables the researcher to determine if behavior is dependent on the time at which treatment is administered, as well as whether the change in behavior purportedly caused by the treatment continues when it is removed.

Part V: Program EvaluationProgram evaluation involves the use of research methods to assess the need for, and the design,

implementation, and effectiveness of, a social intervention. Evaluation sponsors are those agencies or individuals who initiate a program evaluation, and stakeholders are others who also have a vested interest but who may not be directly involved in the research. The benefits of program evaluations are that they (1) help organizations determine whether there is a need for a program, (2) identify whether a program is effective, (3) help those involved with a program improve it, and (4) add to scientific knowledge. There are also many constraints associated with program evaluation. First, many program evaluations tend to be politically charged. Second, the researchers may not have developed the program and therefore be unable to determine what the objectives of the evaluation are. Finally, since the results may be used to make programmatic changes, it is important that the confidentiality of respondents be maintained.

A. Types of program evaluation. Program evaluations usually begin with the evaluation sponsors and key stakeholders identifying why the program evaluation has been requested. The program evaluators (researchers) then develop a plan to conduct the evaluation to suit the request. There are five components of a comprehensive program evaluation.a. In a needs assessment the need for a social program and the steps required to meet that need

is determined. b. A program theory and design assessment evaluates the rationale for why a program has been,

or will be, designed in a particular way. c. A process evaluation determines whether a program is being implemented as intended. This

type of program monitoring provides vital knowledge that determines the ability to draw valid conclusions about a program’s effectiveness.

d. Outcome evaluation assesses a program’s effectiveness. In doing so researchers design and execute a study that helps answer the question of whether the program produced its intended effects. In outcome evaluation contamination may occur. This is when knowledge, services, or other experiences intended for one group are unintentionally received by another group. When contamination occurs it may lead researchers to conclude that the treatment had no effect when it in fact did.

e. An efficiency assessment weighs the program’s benefits and effectiveness in relation to its costs, to determine whether it is an efficient method for addressing this problem.

Quasi-Experimental Designs 131

B. According to Durlak and DuPre (2008), program evaluation is a first step toward program diffusion, the implementation and maintaining of effective programs in other settings or in other groups. It includes four steps. In dissemination, target groups and organizations are informed of the study’s results. In adoption, a new group or organization agrees to implement the program. Implementation involves initiating the program in the new setting and executing it properly during a trial period. Finally, sustainability concerns the ability to maintain the program over time.

E. LECTURE AND CLASSROOM ENHANCEMENTS

PART I: What Is a Quasi-Experiment

A. Lecture/Discussion Topics The “almost experiment” experiment. When I talk to my class about quasi-experiments I introduce

these kinds of experiments as a research design that is “almost an experiment.” Many quasi-experiments look exactly like a true experiment in that there are at least two levels of an independent variable and at least one dependent variable. In its most basic form, a quasi-experiment has an independent variable with two levels and one dependent variable. The characteristic that distinguishes many quasi-experiments from true experiments is the independent variable. Oftentimes the purported “cause” of behavior is something the researcher cannot directly manipulate, such as age or gender. Although one can test to determine whether there are differences between age and gender on dependent measures, both age and gender are variables that exist naturally. Because the researcher cannot randomly assign participants to the “young” or “old” condition, she relinquishes some control. This decrease in control makes the experimenter unable to make causal statements about variables. The data may strongly suggest age affects a dependent measure, but because age is out of the experimenter’s control other variables could have contributed to the change in behavior.

Analyzing quasi-experiments. Even though quasi-experiments are different than true experiments, the design of each enables researchers to use the same statistics determine the study’s results. For example, to test whether two levels of one variable (X) produced different effects on behavior (Y) one would use a t-test for independent groups, regardless of whether X is a subject variable or independent variable. Likewise, if a scientist counted the frequency of children who signed up for tennis lessons or golf lessons based on gender or after they watched a film on tennis, a Chi Square test of independence would be performed. Emphasize to students that although the same statistical tests may be used for quasi-experiments and experiments, the inferences made from the statistical results differ. Causal statements may be made based on the t-statistic associated with an experiment, but cannot be made based on the t-statistic associated with a quasi-experiment.

132 CHAPTER 11: Quasi-Experimental Designs

B. Classroom Exercise Experiment or quasi-experiment? One way a researcher exerts experimental control in an

experiment is by directly manipulating a variable to determine whether it causes a change in another variable. Although the basic design of quasi-experiments is almost identical to that of true experiments, because variable X is not directly manipulated, but instead is measured as it naturally exists, they cannot make causal statements about the relation between X and Y. To help students distinguish between experiments and quasi-experiments provide them with the following list of references. Ask them to determine whether each is more likely a true experiment or quasi-experiment, based upon the title of the study, and to provide a rationale for their answer. Next, have students think of ways to turn the quasi-experimental studies into true experiments. (Note: This exercise is identical to an exercise presented in the instructor’s manual for Chapter 8.)

Lin, C., & Huang, M. (2011). Effects of trait coping styles and emotional display rules on emotional labor strategies. Acta Psychologica Sinica, 43(1), 6–-73. (quasi-experiment; trait coping style is a subject variable)

Ramchandani, P. G., O'Connor, T. G., Evans, J., Heron, J., Murray, L., & Stein, A. (2008). The effects of pre- and postnatal depression in fathers: A natural experiment comparing the effects of exposure to depression on offspring. Journal of Child Psychology and Psychiatry, 49(10), 1069–1078. doi:10.1111/j.1469-7610.2008.02000.x (quasi-experiment; fathers’ depression is a subject variable)

Snodin, N. S. (2013). The effects of blended learning with a CMS on the development of autonomous learning: A case study of different degrees of autonomy achieved by individual learners. Computers & Education, 61, 209–216. doi:10.1016/j.compedu.2012.10.004. (experiment; learning style can be directly manipulated)

Yang, Q., Wu, X., Zhou, X., Mead, N. L., Vohs, K. D., & Baumeister, R. F. (2013). Diverging effects of clean versus dirty money on attitudes, values, and interpersonal behavior. Journal of Personality and Social Psychology, 104(3), 473–489. doi:10.1037/a0030596 (experiment; cleanliness of money can be directly manipulated)

Quasi-experiments are just as valuable as true experiments. The text provides several advantages and disadvantages of quasi-experimental designs. Still, students tend to walk away from the quasi-experimental design believing it’s less important than a true experiment. To help them understand the importance of each research design, and how in some cases a quasi-experiment is the preferred method, have students generate their own list of pro’s and con’s for both experiments and quasi-experiments. Next, ask them to determine which research method is probably best to study the following IV–DV relationships. Also ask them to determine whether any of these research topics might be examined using either an experimental or quasi-experimental design, and if so, which would be preferred:

o Music – Memoryo Gender – Muscle masso Parental attachment style – Child independenceo Diet – Mental health

Quasi-Experimental Designs 133

C. Web Resources Here is a wonderful website that lists the various quasi-experimental designs as well as a link to

each design that summarizes and diagrams it. http://www.fammed.ouhsc.edu/tutor/qexpdes.htm

Research methods by dummies defines a quasi-experiment and also provides examples of various types of quasi-experiments. http://psych.csufresno.edu/psy144/Content/Design/Nonexperimental/quasi.html

The Writing Center at Colorado State University has a very detailed discussion on quasi-experimental research methods (as well as experimental methods). http://writing.colostate.edu/guides/guide.cfm?guideid=64

D. Additional References Comparisons between quasi-experiments and experiments.

Berk, R., Barnes, G., Ahlman, L., & Kurtz, E. (2010). When second best is good enough: a comparison between a true experiment and a regression discontinuity quasi-experiment. Journal of Experimental Criminology, 6(2), 191–208.

Green, J. (2010). Points of intersection between randomized experiments and quasi-experiments. The ANNALS of the American Academy of Political and Social Science, 628(1), 97–111.

PART II: Designs Without a Control Group

A. Lecture/Discussion Topics One-shot case study. Research methods often go by more than one name. Another name for the

research design in which there is only an experimental group that is measured after the administration of the treatment is called a “one-shot case study.”

Why would one conduct a study without a control group? Quasi-experiments are sometimes called “pre-experiments.” This suggests that they are studies that may be conducted as a first step toward conducting a true experiment. This is certainly true in some cases. For example, a group of elementary-school teachers may have been trained in a new technique to increase student participation. To determine its effectiveness they begin to measure how many times each student contributes to class discussions. If the teachers see a change in behavior they may then create a more controlled experiment to rule out other possible explanations for the behavior change.

B. Web Resource This experimental and quasi-experimental handout from Dartmouth’s Office of Institutional

Research describes a number of designs, including those with and without control groups. http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Fwww.dartmouth.edu%2F~oir%2Fdocs%2FTypes_of_Experimental_Designs_Handout.doc&ei=cLi4UYKWKY_L0gGd-IHgBQ&usg=AFQjCNE9W2ZgXBiGa5Nd_FXRK5sP5Sq-Tg&sig2=5l5_3eTgcnisbklTDj-W9Q&bvm=bv.47810305,d.dmQ

134 CHAPTER 11: Quasi-Experimental Designs

Quasi-Experimental Designs 135

Diagrams of experimental and quasi-experimental designs. This webpage defines and diagrams the one-group posttes- only design. By clicking “next” the user is directed to the next design. A number of experimental and nonexperimental research methods are provided. http://www.cvgs.k12.va.us:81/research/PAPER/defining/six.htm

Advantages and applications of time-series designs.o This document compares time-series designs to experimental designs and describes how

time-series designs are analyzed. http://www.gvglass.info/papers/tsx.pdfo http://www.aacn.nche.edu/membership/members-only/presentations/

2013/13cnlsummit/Bender.pdf

C. Additional References Studies using the one-group prettest-posttest design.

Case-Smith, J., Holland, T., Lane, A., & White, S. (2012). Effect of a coteaching handwriting program for first graders: One-group pretest–posttest design. American Journal of Occupational Therapy, 66(4), 396–405.

Verra, M., Angst, F., Staal, J. B., Brioschi, R., Lehmann, S., Aeschlimann, A., & de Bie, R. (2011). Differences in pain, function and coping in Multidimensional Pain Inventory subgroups of chronic back pain: A one-group pretest-posttest study. BMC Musculoskeletal Disorders, 12(1), 145.

PART III: Designs with a Nonequivalent Control Group

A. Lecture/Discussion Topics Is it really that bad to present your work as a true experiment when there is only a very, very, very

minute possibility that the treatment groups (control, experimental) are not equivalent? What makes designs with a nonequivalent control group a quasi-experiment is the difference, or potential for difference, between the two treatment conditions (control and experimental) at the outset of the study. Even if someone believes it is possible, but very unlikely, that the two groups differ, it’s important to present the study as a quasi-experimental design instead of a true experimental design. With good conscience one can’t say that it is an experiment when there is any possibility that the groups are not equivalent, no matter how minute that difference may be. What choice would your students make if given the option to conduct the study as a true experiment or as an unequal control group’s quasi-experiment? This question can be used to describe ethics in research.

136 CHAPTER 11: Quasi-Experimental Designs

B. Classroom Exercises Applying designs with nonequivalent control groups to student research designs. If your students

are conducting a research project ask them to examine whether their research question can be asked with a design in which there is a nonequivalent control group. (For example, if students plan to play music to one group and not the other and then measure their ability to recall items previously studied, they could do an experiment. However, they could take this design and modify it into a nonequivalent control quasi-experiment very easily by recruiting the music participants from the first section of a history class, and the no-music group from the second section of the same history class. Then make comparisons between the two sections to see if music had an effect on performance.) After students modify their design have them take a critical look at both designs—experimental and quasi-experimental— and determine the pros and cons of each. Based on this analysis, which design will they use and why?

C. Web Resources These resources summarize and explain nonequivalent group designs. They do a nice job defining

what nonequivalence means and what its implications are in research, as well as discussing the ability for nonequivalent designs to draw conclusions about relationships between variables. Each resource highlights the fact that the nonrandom assignment of participants to treatment conditions is the sole factor making this design quasi-experimental.

o http://www.socialresearchmethods.net/kb/quasnegd.php o http://facultyweb.berry.edu/nmiller/classinfo/323/2.4/quasi_nonequivalent.htm

D. Additional References Studies using Nonequivalent control groups.

Kang, Y. S., Choi, S. Y., & Ryu, E. (2009). The effectiveness of a stress coping program based on mindfulness meditation on the stress, anxiety, and depression experienced by nursing students in Korea. Nurse Education Today, 29(5), 538.

May, H. (2012). Nonequivalent comparison group designs. APA Handbook of Research Methods in Psychology, APA

PART IV: Switching Replication Designs

A. Lecture/Discussion Topics Ethics of switching replication designs. One of the biggest advantages of the switching replication

design is that, in clinical settings, it enables all participants to benefit from treatment while enabling the researcher to compare the effects of treatment to nontreatment. For example, if depressed patients enrolled in a clinical trial to examine the efficacy of a new depression medication, a traditional experiment would randomly assign participants to either the treatment or placebo-control group. If those in the treatment condition exhibited fewer depressive symptoms relative to the placebo-control group, the clinicians would conclude that the drug does attenuate depression. However, is it ethical to not treat the control group? By using a switching replication design all participants would receive the drug but at different times.

Quasi-Experimental Designs 137

B. Web Resource Switching replication design.

o http://changingminds.org/explanations/research/design/switching_replications.htm o http://fycs.ifas.ufl.edu/swisher/6800_12/M11_Strengthening_Quasi_Experiments.pdf

Nonexperimental designs flashcards. This quizlet.com link is to 48 flashcards about quasi-experimental research designs, including the switching replication design. Students may select “both” sides to see each flashcard’s question along with its answer, or leave it unchecked to see only the question. http://quizlet.com/20996811/non-experimental-research-design-flash-cards/

C. Additional References Studies using a switching replication design.

Burger, C., Schade, V., Lindner, C., Radlinger, L., & Elfering, A. (2012). Stochastic resonance training reduces musculoskeletal symptoms in metal manufacturing workers: A controlled preventive intervention study. WORK: A Journal of Prevention, Assessment and Rehabilitation, 42(2), 269–278.

Stoddard, H. A., & Piquette, C. A. (2010). A controlled study of improvements in student exam performance with the use of an audience response system during medical school lectures. Academic Medicine, 85(10), S37–S40

PART V: Program Evaluation

A. Lecture/Discussion Topics Program evaluation at your institution. To demonstrate how program evaluation is done in the real

world try to find one that is currently being conducted at your university. Where I work, graduating psychology seniors are required to take the Major Field Test in Psychology (MFTp). It is a comprehensive assessment of psychology distributed by Educational Testing Services (ETS), the same folks who distribute the GRE. We use the MFTp as a way to evaluate our psychology program. We believe that students who have completed their major field of study should have some basic knowledge of psychology. This test enables us to examine their knowledge of many key areas such as the areas of clinical, cognitive, developmental, and quantitative psychology. Although the students do not directly benefit from the test, the university uses the results to see what areas student are understanding well and what areas we might want to focus on more in the future.

Drug Abuse Resistance Education (D.A.R.E.). One well-known example of a program that was developed based on a needs assessment is the D.A.R.E. program. This program was developed in 1983, and its success, according to the D.A.R.E. website, is demonstrated through research (see dare.org). Many of your students most likely participated in the D.A.R.E. program while in elementary school. Use this opportunity to discuss their perception of its efficacy. You could also assign students to provide evidence to support its effectiveness as well as studies that suggest it is not an effective way at preventing drug use.

138 CHAPTER 11: Quasi-Experimental Designs

B. Classroom Exercise Develop a needs assessment. Have students work in groups to identify something that they feel

their neighborhood, school, or community is lacking. Next, have them create a needs assessment that they could use to provide evidence of that “something’s” need in the community. To that end, ask them to (1) clearly define the problem, (2) examine the objectives and standards that the evaluation sponsors and stakeholders want to achieve, and (3) state whether those objectives are currently being met. When they’ve completed that task, ask the groups to exchange needs assessments with one another. Do the “reviewers” feel that the “something” proposed is really needed based on the needs assessment? Why?

C. Web Resource Program evaluations at the state and federal Levels.

o http://www.cdc.gov/eval/framework/index.htm o http://www.epa.gov/evaluate/ o http://www.programevaluation.org/

D. Film Suggestion The Bully Project (thebullyproject.com) is a program whose goal is to end bullying. The

documentary Bully is a powerful film that demonstrates the pervasive problem of bullying in our schools. The Bully Project can be discussed as something that developed from a growing need to address the issue of bullying in the United States. Waitt, C., Hirsch, L., Lowen, C., & Foudy, S. (Producers). Hirsch, L. (Director). (2011). Bully. New

York, NY: The Weinstein Company.

E. Additional References Program evaluation resources.

West, S. L., & O’Neal, K. K. (2004). Project D.A.R.E. outcome effectiveness revisited. Journal Information, 94(6).

Astin, A. W., & Antonio, A. L. (2012). Assessment for Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Education. Rowman & Littlefield.

Mertens, D. M. (2009). Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods. SAGE Publications, Inc.