Advancing Assessment of Quantitative and Scientific

Reasoning: A Progress Report on an NSF Project

Donna L. Sundre, Douglas Hall, Karen Smith,

Advancing Assessment of Quantitative and

Scientific Reasoning

Donna L. Sundre

Center for Assessment and Research Studies (CARS)

James Madison Universitywww.jmu.edu/assessment/

Overview of talk

Current NSF Research project

History of the test instrument

Phase I: Generalizability of the instrument

Phase II: Assessment Practice and Validity

Results from some of our partners:

James Madison University

Truman State University

St. Mary’s University

Current NSF Project

3-year grant funded by National Science Foundation: “Advancing assessment of scientific and quantitative reasoning”

Hersh & Benjamin (2002) listed four barriers to assessing general education learning outcomes: confusion; definitional drift; lack of adequate measures, and misconception that general education cannot be

measured

This project addresses all of these concerns with special emphasis on the dearth of adequate measures

Partner Institutions

Michigan State University: State-supported; Research institution

Truman State University: State-supported; Midwestern liberal arts institution

St. Mary’s University (Texas): Independent; Roman-Catholic; Hispanic Serving institution

Virginia State University: State-supported; Historically Black institution

Objectives of NSF project

Explore psychometric quality and generalizability of the QR and SR instruments

Build scientifically based assessment plans Build assessment capacity at partner institutions Develop new assessment models for adoption and adaptation Document potential barriers to assessment practice and explore

solutions Create scholarly communities of assessment practitioners to

sustain work

History of the instrument

Tests have been under development since 1997 at JMU Quantitative Reasoning (QR- 26 items) and Scientific Reasoning (SR- 49 items)

Designed to measure 8 general education learning objectives

Test information and manuals available at

http://www.jmu.edu/assessment/resources/prodserv/cbts.htm

Project phases

Phase I: First Faculty institute (conducted July 2007 at JMU); followed by data collection, identification of barriers, and reporting of results

Phase II: Assessment practice and validity studies; research questions developed at July 2008 Faculty Institute; dissemination of findings and institutional reports

Early content validity evidence

Results strongly support generalizability of test items Truman State: 100% of items mapped Michigan State: 98% (1 item not mapped) Virginia State: 97% (2 items unmapped) St. Mary’s: 92% (5 items not mapped)

Mapping of items alone is not sufficient

Balance across objectives must be obtained

Teams then created additional items to cover identified gaps in content coverage 14 for MSU; 11 for St. Mary’s; 10 for Truman State; 4 for VSU

Research at JMU

Highlights of our Findings: Grades in relevant courses are positively correlated with QR

and SR scores Student QR and SR scores improve with additional course work

AP and JMU credits show greater improvement Transfer credits do not show as marked gains

Students completing their requirements perform better than those who have not

Sophomores and juniors score higher than entering first year students

Research at JMU

Highlights of our Findings: Sophomore students who have completed 3 or 4 courses score

higher than sophomores who have not. We have established faculty ‘standards’ for performance

Many of our students are not meeting those high expectations

Of those completing requirements: QR: 70% SR: 73% These percentages are much higher than those observed for

entering students or students who have not completed their requirements

We ‘filter’ our data using motivation Effort scores This removes about 30-35 scores out of 1,100

Research Plan

Administration of QRSR to incoming freshman classes (Fall 2007 & 2008)

Administration to students with junior standing Spring 2008 and 2009

Link results to various student groups and other academic data

QRSR Freshman Results

Fall 2007

Mean QRSR (%)

Fall 2008

Mean QRSR (%)

All Freshmen 61.2 58.1

Schools

Business 60.9 57.3 b (-0.33)

Human & Soc Sci 59.1 a (-0.39) c 57.9 b (-0.27)

Sci Eng & Tech 63.8 61.5

a – p<0.05 vs. SET; b – p<0.01; c – Effect size

QRSR Score Correlations

Fall 2007 Fall 2008

SAT- Total 0.64 0.60

ACT Composite 0.69 0.63

CCTST a 0.49 0.50

a – California Critical Thinking Skills Test

QRSR Junior Results

Spring 2008 Mean QRSR Score (%)

Spring 2009 Mean

QRSR Score (%)

All Juniors 57.2 59.7

Schools Business 51.7 a (-0.63) b 53.6 a (-0.63) b

(-0.53 ) c

Human & Soc Sci 55.2 a (-0.50) 61.3

Sci Eng & Tech 62.7 63.2

a – p<0.01 vs. SET; b – vs. SET; c – vs. HSS

Test Scores and Student Motivation Level

• Student Opinion Survey (SOS) developed by JMU

• 10 items – 1-5 scale

• Score Range 10-50

• 3 scores• Effort• Importance• Total Motivation

Freshman QRSR And Motivation Levels

Motivation Level Fall 2007

Mean QRSR (%)

Fall 2008

Mean QRSR (%)

No Response 57.8 (N=93) 58.2 (N=103)

10-19 62.3 (N=7) 50.4 (N=12)

20-29 58.4 (N=85) 54.6 (N=80)

30-39 62.7 (N=196) 60.3 (N=204)

40 or higher 66.9 (N=45) 64.6 (N=54)

Spring 2009 Junior QRSR And Motivation Levels

Motivation Level Mean

QRSR (%)

Mean

Cum GPA

No Response (N=40) 62.3 2.90

10-19 (N=11) 45.6 3.05

20-29 (N=85) 57.5 3.24

30-39 (N=98) 60.6 3.11

40 or higher (N=17) 67.6 3.27

Junior QRSR And Motivation Levels

Motivation Level Mean QRSR Score (%)

Mean Total Motivation

Business 53.6 28.86 a (-0.48)

Human & Soc Sci 61.3 29.93

Sci Eng & Tech 63.2 32.08

a – p<0.05 vs. SET

Truman State University

QRSR results

Institution Characteristics

• Public liberal arts

• Highly selective

• High economic diversity

• Low ethnic diversity – predominately white

• Long history of assessment

• Good infrastructure for data collection

Questions

1. Reliability of QRSR v CAAP?

2. Correlations with number of science and quantitative classes?

3. Correlations with ACT?

4. Comparison of majors v nonmajors

5. Correlation with STAT 190 performance?

6. Comparison of Juniors’ scores to first-year students’?

QRSR Administration• Juniors

– Part of normal junior testing – Spanned two academic years: Fall 07- Spring 09– All Jr.’s participate – roughly 50/50 between JMU and

CAAP science and math– Paper-pencil administration– 2283 total

• Smaller scale study of First-year students– Invitations to instructors of first-year experience– Online administration– 135 total

• Both versions include 10 additional items for coverage of outcomes

RQ 1: How does the reliability of QRSR compare to the CAAP?

Overall reliability is comparable

CAAP: .84 - .86*

QRSR Juniors

2007-2008: .80 (calculated) 2008-2009: .81 (calculated)First-year students

Fall 2008 .86 (calculated)(*http://www.act.org/caap/pdf/handbook/Chapter4.pdf ) (No item data available)

– Reliability of outcome-based SUBSCALES (Juniors 07-08)

• Physical science outcome 1 – 38 items: .639• Physical science outcome 2 – 26 items: .579• Physical science outcome 3 – 9 items: .460• Physical science outcome 4 – 8 items: .189

• Life science outcome 1 – 38 items: .639• Life science outcome 2 – 26 items: .579• Life science outcome 3 –19 items: .563• Life science outcome 4 – 5 items: .174• Life science outcome 5 – 16 items: .518

• Math outcome 1 – 27 items: .666• Math outcome 2 –5 items: .274• Math outcome 3 – 22 items: .609• Math outcome 4 – 5 items: .439• Math outcome 5 – 5 items: .396

RQ 2: How do QRSR scores and CAAP scores correlate with number of classes taken (at

Truman) in science and quantitative areas?  CAAPMath CAAPSci QRSR

#AGSC -0.053 -0.030** -0.008

#BIOL 0.156** 0.215** 0.118**

#CHEM 0.270** 0.277** 0.146**

#CS 0.203** 0.091* 0.068*

#ECON 0.127** -0.103* 0.061

#MATH 0.190** -0.011 0.054

#PHYS 0.316** 0.279** 0.148**

#POL -0.032 0.01 0.109**

#PSYC -0.068 -0.063 -0.023

#SOAN -0.151** -0.032 -0.021

#STAT 0.125** -0.053 0.039

RQ 3: How do ACT science and math subscores correlate with science and math

subscores on the two assessment instruments?

CAAP Math CAAP Sci. JMU

ACT Math .685 .543 .517

ACT Sci. .516 .635 .518

ACT Comp. .568 .655 .618

CORRELATIONS

RQ 4: Does QRSR discriminate science/math majors from non science/math majors?

Yes.• Science and Math majors : 85.3%

• Other majors average 79.7%

These differences are statistically significant for the overall score (t(584) = 5.85, p < .001) and for each of the outcome subscores.

RQ5: Does STAT 190 predict student performance of the QRSR?

Too few students without STAT 190 credit to test those with the course v those without.

Correlation with Truman STAT 190 course grades

QRSR: .318

CAAP MATH: .374

CAAP SCI: .282

RQ6: How do scores of Juniors compare to those of first-year students?

First-year scores and junior scores are significantly different, p < .01, effect size .215 (for junior 07-08)

0

10

20

30

40

50

60

70

80

First Year Juniors 07-08 Juniors 08-09

Raw

sco

re m

ean

Challenges

• Collecting data from first-year students

• Estimates of student motivation

• Sharing the model outside quantitative and scientific disciplines

• Using the data in a changing curriculum

Uses & Future directions

• Considered as part of gen ed curriculum reform

• Data analysis from 08-09 juniors continues

Thank you for coming!

Questions??

All slides will be made available from the NASPA website in a week or two.