math assessment test ocma may, 2008. history of mat test originally developed in late 60’s

MATH ASSESSMENT TEST

OCMA

May, 2008

HISTORY OF MAT

• Test originally developed in late 60’s

HISTORY OF MAT

• Test originally developed in late 60’s • Original purpose was to verify foreign credentials of

applicants

HISTORY OF MAT


applicants• Name changed by 1970 to Math Inventory test and given

to all students taking first semester math

HISTORY OF MAT



to all students taking first semester math• Purpose at this time was to collect data on students to

determine their math skill level

HISTORY OF MAT




determine their math skill level• Math Learning Centre developed in 1973 and MAT now

had a remedial component

HISTORY OF MAT




determine their math skill level• Math Learning Centre developed in 1973 and MAT now

had a remedial component• Remedial modules were further developed during the

mid 70’s

ORIGINAL TEST

• Modular multiple choice test• Students given overall grade on the test + any modules

that were weak • Remedial work assigned only on those topics that were

weak• Work completed through the Math Learning Centre

during the semester

Past & Present

• Diagnostic test with a remedial component

• Push to use test as an assessment tool for streaming

Past & Present


• Students had higher level math at entry


• Student coming in with more varied math background

Past & Present


• Students had higher level math at entry

• Students scored higher on average on the MAT


• Student coming in with more varied math background

• Students scoring lower on the MAT

Original Business and Technology Versions

Business1. Fractions2. Decimals3. Percents4. Order of Operations and Law

of Signs5. Equations

Original Technology and Business Versions

Business1. Fractions2. Decimals3. Percents4. Order of Operations and Law

of Signs5. Equations

Technology1. Fractions2. Decimals3. Percents4. Order of Operations5. Laws of Signs6. Exponents and Radicals7. Algebraic Fractions8. Equations and Formulas9. Mensuration10. Trigonometry and Geometry11. Logarithms

Current Computerized Version1. Fractions2. Decimals3. Percents4. Order of Operations5. Laws of Signs6. Exponents7. Basic Algebra8. Equations9. Radicals10. Algebraic Fractions11. Mensuration12. Trigonometry and Geometry13. Logarithms

It is possible to select different modules by the program of the students.



FALL 2007

Business Students did only

Topics 1-5

Technology students did Topics 1-10, as research showed these were the critical modules.



FALL 2007Business Students did onlyTopics 1-5 Technology students did Topics 1-10, as research showed these were the critical modules. (54 questions)

FALL 2008Business Students will add Topics 7 & 8 (40 questions)

PAST AND PRESENT

PAST

1. Written on first day of class

PRESENT

1. Students write prior to semester start – available from Feb.

PAST AND PRESENTPAST


2. Pencil and Paper

PRESENT1. Students write prior to semester

start – available from Feb.

2. Computerized test completed in a proctored site. (Practice tests available online with built in remedial.)

PAST AND PRESENT

PAST


2. Pencil and Paper

3. Results given in weeks 2 or 3

PRESENT1. Students write prior to semester

start – available from Feb.

2. Computerized test completed in a proctored site. (Practice tests available online with built in remedial.) Results given on exit from test

3. Results given on exit from test

PAST AND PRESENT

PAST1. Written on first day of class

2. Pencil and Paper

3. Results given weeks 2 – 3

4. Remedial done during semester through the MLC

PRESENT1. Students write prior to

semester start – available from Feb. Practice tests available online with built in remedial

2. Computerized test completed in a proctored site. (Practice tests available online with built in remedial.)

3. Results given on exit from test

4. Remedial completed online prior to September 30.

Current Research

• The push to change the use of the assessment test for streaming meant that the test needed to be “tested”.

• We are currently doing – Item Analysis on each question in the test (new)– Validity of the test as a predictor of success in

semester 1 college math (this is ongoing from previous years)

Item Analysis of Test(Creating good questions in a multiple choice test)

1. The incorrect options, distractors, must be a clue to the nature of the misunderstanding and thus prescriptive of appropriate remediation

2. Each question must discriminate correctly between the top 27% and the bottom 27% of students.

Validity

• Test validity refers to the degree to which the test actually measures what it claims to measure, in this case, success in first semester college math courses.

• It is also the extent to which inferences, conclusions, and decisions made on the basis of test scores are appropriate and meaningful.

ITEM ANALYSIS DISTRACATORS

The good and the bad!

43210

40

30

20

10

0

43210

90

80

70

60

50

40

30

20

10

0

Q7

Perc

ent

Q8

Fractions Module: questions 7 & 8

Green is correct answer

Discrimination of Question 7

RankQ7

321101010

90

80

70

60

50

40

30

20

10

0

Perc

ent

Chart of Rank, Q7

Percent within levels of Rank.

bottom 27%

middle 54%top 27%Green is

correct answer.

Question 7

• Each distractor for this question represented a common error in dealing with mixed fractions

• The question discriminates correctly between the top 27% of students and the bottom 27 % of students

Question 8

• This question did not give much useful information since over 90% of students answered correctly (index of difficulty is low)

• The distractors were reviewed and changed to reflect more typical mistakes that students make when multiplying fractions

Item Analysis

We are • Reviewing each question in this manner to ensure that each

question adds to our knowledge of any weaknesses students may have i.e. it has appropriate distractors

Item Analysis

We are • Reviewing each question in this manner to ensure that

each question adds to our knowledge of any weaknesses students may have i.e. it has appropriate distractors

• Reviewing the discrimination of each question to ensure that the question is not misleading (discrimination – top 27% should have a higher percent correct than bottom 27%)

Item Analysis

We are • Reviewing each question in this manner to ensure that

each question adds to our knowledge of any weaknesses students may have i.e. it has appropriate distractors

• Reviewing the discrimination of each question to ensure that the question is not misleading (discrimination – top 27% should have a higher percent correct than bottom 27%)

• Changing our remedial to give specific mini lessons based on the wrong answer the student selects. (Currently it is a more general remedial on a topic such as adding fractions.)

Validity of Test for Placement

• This test was designed as a diagnostic tool to assess areas of weakness and provide remedial assistance.

• Over time we have shown that there is a positive linear relationship between the test score on our assessment and success in first year college mathematics

• BUT is it strong enough i.e. does it predict well enough to use as a placement test?

2004 Research Questions

1. Does attendance affect performance in first semester math?

2. Are basic math skills on entry to the college a factor in determining student success?

PassedLow Attendance

Independent Learners

PassedHigh Attendance

Passers

FailedLow Attendance

Failures

FailedHigh Attendance

Heart Breakers

Attendance

Correlation between final grade and attendance was statistically significant . (r=0.467, p=0.000)

Quadrant

ASSES

SM

ENT

(%)

85

80

75

70

65

60

55

Interval Plot of ASSESSMENT vs Quadrant95% CI for the Mean

One-way ANOVA: ASSESSMENT versus QuadrantSource DF SS MS F PQuadrant 3 54369 18123 69.33 0.000Error 1148 300084 261Total 1151 354453

100908070605040302010

100

80

60

40

20

0

MAT %

MA

101 %

60

60

S 22.8520R-Sq 18.7%R-Sq(adj) 18.6%

Business Students Fall 2007MA101 % = 21.45 + 0.5797 MAT %

but passed MA101Type 1: Failed MAT

but failed MA101Type 2: Passed MAT

Correlation coefficient r=0.43 and p=0.000

MAT% AS A PREDICTOR OF SUCCESS IN MA101MAT CONTINUES TO BE A SIGNIFICANT PREDICTOR OF SUCCESS IN 2007

Validity of Test for Placement

• What does that mean to students, if they are streamed into an extra math course using this test?

MAT AS A PLACEMENT TEST(extra math course)

Able to Pass first Semester Math

True False

Streamed into an extra math course based on Assessment Test

Yes Correct Decision1

(helps students succeed)

No Correct Decision

1 This is the group we are trying to help. They do not have the basic math skillsrequired to pass the regular first semester math course.

AS AN ASSESSMENT TEST

Able to Pass first Semester Math

True False

Streamed into an extra math course based on Assessment Test

Yes TYPE 1 ERROR Correct Decision

No Correct Decision TYPE 2 ERROR(issues other than math)

Type 1 Error(able to pass, extra math

course)

Type 2 Error(can’t pass, no extra course)

Student is required to do the extra math course even though they are able to pass the first semester course.

Student is not streamed into the extra math course when the course is required.

Cost:1.time: student is an unnecessary semester behind in math2.cost of additional course that they did not require

Cost:1.time: student is a semester behind BUT still does not have basic math skills required to pass course

COST OF ERRORS TO STUDENTS OR COLLEGE

?QUESTIONS

math assessment test ocma may, 2008. history of mat test originally developed in late 60’s

Documents