experimental research methods in language learning chapter 14 analyses of variance (anovas)
TRANSCRIPT
![Page 1: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/1.jpg)
Experimental Research Methods in Language
Learning
Chapter 14
Analyses of Variance (ANOVAs)
![Page 2: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/2.jpg)
Leading Questions
• If there are three experimental groups and you would like to determine which group is more effective in terms of learning improvement, what would you do?
• What do you know about ANOVA and ANCOVA?
• If you know something of these statistical tests, do you think they are difficult to learn to use? Why or why not?
![Page 3: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/3.jpg)
ANOVA Family
• Analysis of variance (ANOVA) allows researchers to examine mean differences in experimental research.
• This chapter presents three types of ANOVAs.
• One-way ANOVA
• ANCOVA
• Repeated-measures ANOVA
![Page 4: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/4.jpg)
The One-way Analysis of Variance (ANOVA)
• A one-way (or one-factor) (independent-measures) analysis of variance (ANOVA) is a parametric test used to determine whether there is a statistical significance between scores obtained by two or more groups in an experimental study.
• A one-way ANOVA can perform the same analysis as the independent-samples t-test.
![Page 5: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/5.jpg)
The One-way ANOVA
• A one-way ANOVA has several advantages over a t-test. Its major advantage is that it can compare more than two groups (e.g., three, four, five, and so on) in one single analysis.
• ANOVA can minimize the possibility of a Type I Error (i.e., rejecting the null hypothesis when it should not be rejected) more conservatively than a t-test.
• A one-way ANOVA considers errors arising from within-group differences when it analyses group differences.
![Page 6: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/6.jpg)
Post hoc Analysis
• Since a one-way ANOVA performs comparisons among three or more groups all at once, when there is a statistical significance, we will not know which groups differ.
• In order to identify where a group difference exists, we need to perform a statistical test known as a post hoc analysis.
• Examples of post hoc tests: Bonferrori, Scheffe, and Tukey.
• Each of these post hoc tests functions similarly to an independent-sample t-test.
![Page 7: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/7.jpg)
Examples of Studies
• Akakura 2012;
• Ahmadian 2012;
• Ahmadian & Tavakoli 2011;
• Morgan-Short & Bowden 2006;
• Sheen 2010;
• van Gelderen et al. 2010.
![Page 8: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/8.jpg)
Statistical Assumptions of the One-way ANOVA
The statistical assumptions for the one-way ANOVA are the same as those for the independent-samples t-tests:
•Type of Scale
•Random Sampling
•Normal Distribution
![Page 9: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/9.jpg)
Example of ANOVA Results
• Table 14.1.1 reports on descriptive statistics (e.g., means, SD, and standard error mean) between the three groups.
![Page 10: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/10.jpg)
Example of ANOVA Results
• Table 14.1.2 reports the Levene’s Test for Equality of Variance. Remember that it must not be statistically significant in order to make sure that both groups were relatively equal.
![Page 11: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/11.jpg)
Example of ANOVA Results
• Table 14.1.3 shows the ANOVA findings. We examine the F, df and Sig columns in this output. Sig will tell us whether there was a statistically significant difference between the three groups. We found that such a difference did exist (F(2, 59) = 7.97, p = 0.001).
![Page 12: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/12.jpg)
Example of ANOVA Results
• Table 14.1.4 presents the Bonferrori post hoc test outcome.
![Page 13: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/13.jpg)
Example of ANOVA Results
• It was found that there were statistically significant differences between the experimental methods 1 and 2 (p = 0.029), and the experimental method 1 and the control group (p = 0.001).
• However, the experimental method 2 and the control group did not differ statistically (p = 0.55).
![Page 14: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/14.jpg)
Example of ANOVA Results
• Cohen’s d effect size for these two pairs can be computed: http://www.cognitiveflexibility.org/effectsize/
• It was found that a Cohen’s d for the difference between the experimental groups 1 and 2 was 0.796 (≈ 0.80), whereas a Cohen’s d for the difference between the experimental group 1 and the control group was 1.35.
![Page 15: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/15.jpg)
Analysis of Covariance (ANCOVA)
• Analysis of covariance (ANCOVA) is designed to control a pre-existing difference between comparison groups before an experimental treatment.
• Examples of studies: Ammar (2008); Ammar & Spada (2006); Brantmeier (2005); Goo (2012); Lee & Kalyuga (2011); Lyddon (2011); Van Beuningen, De Jong, & Kuiken (2011).
![Page 16: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/16.jpg)
Statistical Assumptions of the ANCOVA
• The statistical assumptions of the ANCOVA are the same as those of the one-way ANOVA.
• These assumptions must be met before ANCOVA is implemented.
![Page 17: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/17.jpg)
Example of ANCOVA Results
• Table 14.2.1 reports on the descriptive statistics (including means, SD, and std error mean) between the two groups.
![Page 18: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/18.jpg)
Example of ANCOVA Results
• Table 14.2.2 presents the Levene’s Test for Equality of Variance. This test was non-significant (p = 0.26).
![Page 19: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/19.jpg)
Example of ANCOVA Results
• Table 14.2.3 presents the results of the Test of Between-subject Effect.
![Page 20: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/20.jpg)
Example of ANCOVA Results
• We examine the F, df and Sig columns in this output. Sig will tell us whether there was a statistically significant difference between the two groups.
• According to this table, it was found that the pre-existing test-anxiety trait was still the main factor affecting the posttest scores (F(1, 48) = 14.96, p < 0.05, partial eta squared (ηp
2) = 0.24).
• There was also a statistically significant difference in the group effect (F(1, 48) = 4.70, p < 0.05, partial eta squared (ηp
2) = 0.09).
![Page 21: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/21.jpg)
Comparison with a Univariate ANOVA
• Let’s compare a univariate ANOVA on the posttest without using the test-anxiety as a covariate.
![Page 22: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/22.jpg)
Comparison with a Univariate ANOVA
• We find the following main effect of the experimental treatment: F(1,48) = 10.48, p < 0.05, ηp
2 = 0.19. The effect size of the experimental treatment in the ANCOVA was reduced to half the size of that produced in a univariate ANOVA.
• ANCOVA therefore allows us to be more realistic in evaluating the effect of an experimental treatment.
![Page 23: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/23.jpg)
Repeated-measures Analysis of Variance (ANOVA)
• The repeated-measures ANOVA can serve the same function as the paired-samples t-test.
• However, the repeated-measures ANOVA can measure changes at more than two times points.
• The repeated-measures ANOVA has broad applications for experimental data analysis.
• It is useful for a pretest-, posttest-, and delayed-posttest design.
![Page 24: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/24.jpg)
Repeated-measures Analysis of Variance (ANOVA)
• The repeated-measures ANOVA takes sources of variation into account when evaluating whether there have been statistically significant changes across times points :
• variation associated with the average scores (i.e., mean scores) for each group each time;
• variation associated with the within-subject variance between Times 1, 2 and 3; and
• the interaction between the first and the second sources; Urdan 2005 ).
![Page 25: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/25.jpg)
Repeated-measures Analysis of Variance (ANOVA)
• Examples: Baralt & Gurzynski-Weiss 2011; Benati 2005; Folse 2006; Iwashita, McNamara, & Elder 2001; Kissling 2013; Moskovsky, Alrabai, Paolini, & Ratcheva 2012; Shintani 2011; Shintani et al. 2014; Strapp, Helmick, Tonkovich, & Bleakney 2011; Takimoto 2008; Tian & Macaro 2012).
• The repeated-measures ANOVA is one of the most commonly used statistical techniques in experimental research in language learning.
![Page 26: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/26.jpg)
Statistical Assumptions of the Repeated-measures ANOVA
• The statistical assumptions of the repeated-measures ANOVA are similar to those discussed previously in this chapter.
• There is one additional assumption that we need to consider. This assumption is known as the sphericity assumption.
• Sphericity refers to whether the variances of the differences between all possible pairs of comparison groups are equal.
• This assumption is assessed by using the Mauchly’ sphericity test.
![Page 27: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/27.jpg)
Example of Repeated-Measures ANOVA Results
![Page 28: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/28.jpg)
Example of Repeated-Measures ANOVA Results
• Table 14.3.2 is the Box’ Test of Equality of Covariance Matrices. This test must not be significant at 0.001 (p > 0.001). This statistic tells us that we have homogeneity of variance.
![Page 29: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/29.jpg)
Example of Repeated-Measures ANOVA Results
• Table 14.3.3 presents the multivariate tests.
![Page 30: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/30.jpg)
Example of Repeated-Measures ANOVA Results
• The multivariate tests determine whether there are significant group differences for a linear combination of the dependent variables (i.e., the reading tests).
• This table contains 4 tests (i.e., Pillai’s Trace, Wilks’ Lambda, Hotelling’s Trace, and Roy’s Largest Root).
• However, we only need to consider one test. Choose the Pillai’s Trace. This test must be statistically significant at 0.05.
![Page 31: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/31.jpg)
Example of Repeated-Measures ANOVA Results
• For both ‘time’ and ‘time and factor’, the Pillai’s Trace was significant (p = 0.00).
• This statistical significance means that we can now move on to examine the univariate/between-subjects effects that follow.
• If the Pillai’s Trace is not significant, we stop our analysis here because the rest of the outcome will not be meaningful because there are no group differences.
![Page 32: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/32.jpg)
Example of Repeated-Measures ANOVA Results
• Table 14.3.4 presents the analysis for Mauchly’s sphericity test. This test tells us whether our data violate the sphericity assumption.
• We need this test to be non-significant for this assumption to be met. In Table 14.3.4, we can see that this test was significant (p < 0.05).
• Examine an alternative assessment to the sphericity assumption. Choose the Hyunh-Feldt test, which was significant at 0.896.
![Page 33: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/33.jpg)
Example of Repeated-Measures ANOVA Results
![Page 34: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/34.jpg)
Example of Repeated-Measures ANOVA Results
• Table 14.3.5 shows the results for the test within-subject effect. Focus on Huynh-Feldt.
![Page 35: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/35.jpg)
Example of Repeated-Measures ANOVA Results
• Table 14.3.6 presents the Levene’ Test of Equality of Error Variances. For all the three reading tests, the homogeneity assumption was not violated (p > 0.05).
![Page 36: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/36.jpg)
Example of Repeated-Measures ANOVA Results
• Table 14.3.7 reports the Tests of Between-Subjects Effects, which will tell us whether there is a statistical group difference.
• According to this table, we found that the three groups did not significantly differ from one another (F(2, 57) = 2.719, p = 0.08, ηp
2 = 0.09).
![Page 37: Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)](https://reader036.vdocuments.mx/reader036/viewer/2022070403/56649f2c5503460f94c47ba8/html5/thumbnails/37.jpg)
Discussion
• What are the principles underlying an ANOVA? How do they differ from an independent-samples t-test?
• What kind of experimental situations do we need to use an ANCOVA?
• What are experimental research designs (discussed in Chapter 4) that a repeated-measures ANOVA is suitable for?
• What is a post hoc test? When do we need to use a post hoc test?