Validation and ACCUPLACER®

ACCUPLACER National ConferenceSan Francisco, CA

June 29, 2012June 29, 2012

Mary-Margaret Kerns, Ph.D.Executive Directorecut e ectoHigher Education ResearchResearch and Development

Structure of presentation

• Validity

• Admitted Class Evaluation Service™ (ACES™)

• ACES ACCUPLACER Placement Validity StudyC S CCU C ace e t a d ty Study

2

Reliability is not enough

• If test scores are used to make inferences about an examinee’s ability, the test must be both y,reliable and valid. • Reliability is a prerequisite for validity and refers to the

ability of a test to measure a particular trait consistently.

• Tests can be highly reliable and still not be valid for a particular purposeparticular purpose.

• Determining the reliability of a test is an important first step but not the defining step in determining the validity p g p g yof a test.

3

What is test validity and test validation?

• A test that is perfectly appropriate and useful in one situation may be inappropriate or insufficientone situation may be inappropriate or insufficient in another.

For example a test valid for determining high school• For example, a test valid for determining high school graduation may be inappropriate for use in college placement.

• Test validity, or the validation of a test, explicitly means validating the use of a test in a specific g pcontext, e.g., placement into a course.• Therefore, it is important to study the test results in the , p y

setting in which they are used.4

Validity Evidence 5

Validity refers to the degree to which evidence and theory support the interpretations of test scores entailedtheory support the interpretations of test scores entailed by proposed uses of tests.

Standards of Educational and Psychological Testing

(AERA, APA, NCME, 1999)( , , , )

5

Validity Evidence

• Content Validity

C t t V lidit• Construct Validity

• Consequential Validity

• Criterion-Related Validity

6

Criterion-Related Validity

• Criterion-related validity looks at the relationship between a predictor and a criterion / outcome.

• A criterion-related validity study is completed by collecting the test score (predictor) information and information on grades (criterion / outcome) for the same students.

• E.g., ACCUPLACER Reading Comprehension scores and ENG100 grades

• E.g., ACCUPLACER Elementary Algebra scores and MATH100 grades

• Refers to the “power” or usefulness of test scores to predict future performance.

• E.g., future performance as academic success (or non-success) in college

7

ACCUPLACER TESTS COURSE GRADES

Arithmetic Elementary Reading Sentence WritePlacer ENG100 Math100

Algebra Comprehension Skills Grade

Student A 85 70 35 88 6 A B

Student B 92 100 108 99 7 C A

St de t C 65 52 24 30 1 D CStudent C 65 52 24 30 1 D C

Student D 90 73 65 68 3 C B

Student E 79 80 58 75 6 A B

Student F 90 95 87 99 8 A A

Student G 69 98 42 53 1 C C

Student H 85 102 91 95 4 C B

Student I 99 108 118 115 8 A A

Student J 75 68 115 100 7 B BStudent J 75 68 115 100 7 B B

8

E

F

I

ACCUPLACER READING COMPREHENSION SCORES and ENG100 GRADES

A

J

H

D

RAD

ES

B

CG

ENG

100

GR C

D

DH

B

C

20 30 40 50 60 70 80 90 100 110 120

READING COMPREHENSION SCORES

F

READING COMPREHENSION SCORES

9

10

11

12

13

E

F

I

ACCUPLACER Reading Comprehension Scores and ENG100 Grades

90

J

H

D

Succ

ess

75

H

G

Prob

abili

ty o

f S 60

45

DH

B

C

20 30 40 50 60 70 80 90 100 110 120

30

ACCUPLACER Reading Comprehension Scores

14

http://professionals.collegeboard.com/higher-ed/validity/aces

15

Admitted Class Evaluation Service (ACES)

• ACES offers two models of validity studies: • Admission

• PredictivePredictive• Placement

P di ti• Predictive

• Concurrent

16

Placement Validity Studies

• ACES is able to process predictive placement validity studies using scores from:validity studies using scores from: • SAT

• SAT Subject Tests

• ACCUPLACER Tests

17

Number of Predictors

• In some situations, one factor can predict a certain outcomecertain outcome.

• The more types of relevant information, the better you are able to predict the criterionyou are able to predict the criterion.

18

Designing a Predictive Validity Study

• You are predicting success on the criterion (end of course grade) using a predictor or(end-of-course grade) using a predictor, or predictors

The predictor information is gathered at some• The predictor information is gathered at some point before the student enters the course

It is important that the predictor is current and• It is important that the predictor is current and that little or no instructional intervention has occurred between the predictor and the start of pthe course

19

Designing a Predictive Validity Study

• The predictor/s and criterion need to be collected from all students at the same point in the coursefrom all students at the same point in the course sequence.

• In an effort to increase sample size some investigators• In an effort to increase sample size, some investigators include students who took the predictor, were then placed in a developmental course, and then take the college-level courselevel course.

• The intervention of the developmental course changes the relationship between the predictor and the college-level p p gcourse criterion, making the data incompatible.

20

Designing an ACES Predictive Validity Studyy

• Number of students in the study• A reasonably large sample is important to increase the

likelihood of a sample being similar to the population it is intended to representp

• Validity coefficients are extremely sensitive to chance fluctuations (a change in one score can make a large difference))

• The minimum number of student required for an ACES study depends on the number of predictors used

• 1 to 3 predictors requires a minimum of 30 students

• 4 predictors, a minimum of 40 students

5 predictors a minimum of 50 students• 5 predictors, a minimum of 50 students

21

Designing a Predictive Validity Study

• In order to effectively evaluate how well a predictor is working, ACES requires a range of at least 4 points onworking, ACES requires a range of at least 4 points on a scale for the criterion.

• Students with pass/fail grades should not be included in p gan ACES study.

22

Predictive Validity Study

• Statistical factors play a role in predictive validity t di (i d th it d f di tistudies (i.e. reduce the magnitude of predictive

validity coefficients)• Small sample sizes

• Measurement unreliability

• Restriction of range

23

http://professionals.collegeboard.com/higher-ed/validity/aces

24

Requesting a Placement Study

• You may analyze up to five separate courses in your Placement Validity Study.y y y

• For each course you will begin by specifying the tests and the predictors you are most interested i l tiin evaluating. • You may use a single predictor, or up to five predictors,

for a single course.for a single course.

25

26

ACES Data Submission

• Choose from a variety of common formats including:g• Excel • Access • SPSS • SAS • ASCII delimited format

• Upload the file directly from your PC to ACES p y y• ACES automatically encrypts the data during

transmission to protect confidentiality

27

Getting an ACES Study

• ACES notifies the institution by e-mail when the study is completedstudy is completed

• The document is password protected for confidentiality and is encrypted until downloaded y ypat the institution

• Results of ACES studies are confidential and only l d t th i di id l th t t d th t dreleased to the individual that requested the study

• Placement studies take up to 40 business days to process from time of successful data submissionprocess from time of successful data submission

28

ACES Validity Handbook(http://professionals.collegeboard.com/higher-ed/validity/aces/handbook)

• It is designed to serve as a general reference for validity and includes information about validity beyond what is specifically applicable to ACESis specifically applicable to ACES.

• It includes specific information about• It includes specific information about the types of validity studies and their design that are available through the ACES system and helps interpret y p pACES study results.

29

30

ACES Placement Validity Study

• The predictive validity study provides two probability tables for cut scores:probability tables for cut scores:• For a course grade of B or higher

F d f C hi h• For a course grade of C or higher

31

Section 1: The purpose of this report

• When requesting this report, you indicated that you wish to study placement in two courses.

• You indicated you plan to use scores from ACCUPLACER tests to predict success in English 100.

The specific predictor variables are: Reading Comprehension• The specific predictor variables are: Reading Comprehension and Sentence Skills.

• You indicated you plan to use scores from ACCUPLACER tests to predict success in Math 100.

• The specific predictor variables are: Elementary Algebra andCollege Level Math.g

32

Section 2: Your sample of students

33

Section 2: Your sample of students

34

Section 3: Strength of Prediction

35

36

ACCUPLACER Validity Presentation ResourcesResources

Admitted Class Evaluation Service (ACES™) http://professionals.collegeboard.com/higher-ed/validity/aces

ACES Placement Validity Study http://professionals.collegeboard.com/higher-ed/validity/aces/placement

Placement Validity Report for ACCUPLACER Sample// f / f /http://professionals.collegeboard.com/profdownload/aces-sample-placement-validity-report-

accuplacer.pdf

Predictive Validity of ACCUPLACER Scores for Course Placement: A M t A l iA Meta-Analysishttp://professionals.collegeboard.com/data-reports-research/cb/validity-accu-placement

37

Questions, Comments, Suggestions

• Researchers are encouraged to freely express their professional judgment. Therefore, points oftheir professional judgment. Therefore, points of view or opinions stated in College Board presentations do not necessarily represent official C ll B d iti liCollege Board position or policy.

• Please forward any questions, comments, and suggestions to:

mmkerns@collegeboard orgmmkerns@collegeboard.org

38