Validity of web-based placement testing outcomes:

Some recent findings

Hoi Suen

Professor of Educational Psychology

Pennsylvania State University

University Park, PA 16802-3108, U.S.A

Office: 814-865-2235

Fax: 814-863-1002

Email: HoiSuen@psu.edu

Website: http://suen.ed.psu.edu

Distance learning and assessment

• We deliver and facilitate instructional activities over the Web or other media.

• However, we require students to take high-stakes exams at designated centers where the tests are secured and where the identities of the examinees can be authenticated, and the examinees can be proctored and observed.

Unproctored web-based Exam Security and Authentication

• Without the physical presence of a human proctor, there is no known easy way of ensuring that the responses to exam questions in fact originated from the examinee; nor that the examinee did not obtain assistance from other people, from printed reference books, from his/her own computer hard drive or even from the web itself, while taking the exam.

Assumption regarding unproctored web-based placement testing

• Students should not and would not cheat when taking a placement test, because cheating in this situation would only hurt the student himself/herself. If a student did better due to cheating and were thus placed into a course more advanced than his/her capabilities, the student would be the one who would suffer.

We theorized that:

1. When given convenient opportunities to improve test scores, students will tend to take advantage of these opportunities whether these opportunities are appropriate or not and whether the end result will hurt the student or not. This is because students today have been brought up in a high-stakes testing environment and that they generally believe that it is important to score as high as possible in any exam. Therefore, even for a placement test, when administered through an unproctored web-based setting, students will cheat.

We theorized that:

2. As a result of cheating in these placement tests, a substantial portion of students will be placed into courses beyond their capabilities.

3. When confronted with a substantial portion of students being placed beyond their levels of abilities, instructors would change instructional method and/or contents to adjust to the students’ lower level of abilities; i.e., as an indirect result of unproctored web-based placement testing, instructors will tend to “water down” instruction.

Study to test the theory designed and executed by:

Dawn Zimmaro, Ph.D.Research Associate

University Testing Center

Penn State University

University Park, PA 16802

Email: dmz115@psu.edu

SAMPLE

• Penn State freshmen placement testing program• Focused on math testing: The mathematics placement exam

was a multiple-choice format. The test contained 72 five-response option items based on a set of 29 mathematical topics ranging from arithmetic of integers to inverse trigonometric functions.

• Matched samples: Matched samples of web-based and paper testers based on SAT Math and high school grade point average. A total of 1,010 exactly identical matches on these two variables were identified. The final sample included 1,010 Web testers and 1,010 paper testers with an average high school grade point average of 3.79 (s.d. 0.25) and an average SAT Math score of 628 (s.d. 66).

Major Hypotheses:Focus of this presentation

I. After controlling for differences in mathematics ability students who take an unproctored Web mathematics placement test will score higher on the placement test than students who take an equivalent proctored paper-and-pencil mathematics placement test.

II. After controlling for differences in mathematics ability, there will be an interaction between type of placement test and the type of math course in which the student enrolled with lower first exam scores in calculus courses for students placed based on Web test results.

(The testing of a 3rd “watered-down” hypothesis was inconclusive.)

Results: Math placement test scores of matched samples

TEST TYPE Mean Std. Deviation N

Paper testers 48.89 11.89 1010

Web testers 51.34 12.16 1010

Total 50.11 12.08 2020 t (2018) = 4.591, p < .001. The correlation between math placement test score and placement test type was found to be r = 0.102. The amount of variation in total math placement test score that can be attributed to the type of test the student took was found to be 1% (r2 = .010).

Results: First exam scores by course by placement test type

COURSE TYPE TEST TYPE Mean Std. Deviation N

Non-calculus Paper testers 73.43 16.37 234 Web testers 70.13 18.04 182 Total 71.98 17.18 416

Calculus courses Paper testers 67.77 17.47 202 Web testers 61.20 19.26 231 Total 64.27 18.72 433

Total Paper testers 70.81 17.10 436 Web testers 65.13 19.23 413 Total 68.05 18.38 849 

Results: First exam scores by course by placement test type

Estimated Marginal Means

of First Math Exam Score

COURSE TYPE

Calculus courseNon-calculus course

Estim

ate

d M

arg

ina

l M

ea

ns

76

74

72

70

68

66

64

62

60

TESTTYPE

paper test

web test

Results: First exam scores by course by placement test type

Tests of Between-Subjects Effects Dependent Variable: First Math Exam Score Source

SS df Mean Square F Sig. Model 18408.392 3 6136.131 19.346 .000 Intercept 3899194.845 1 3899194.845 12293.290 .000 COURSE TYPE 11163.866 1 11163.866 35.197 .000 TEST TYPE 5118.687 1 5118.687 16.138 .000 COURSE TYPE * TEST TYPE 562.125 1 562.125 1.772 .183Error 268017.723 845 317.181Total 4217644.000 849

*R Squared = .064 (Adjusted R Squared = .061)