ASSESSING GE GOALS AND CAPACITIES WITH CONCEPT INVENTORIES:ONE PATH FORWARD . . . PART I

1. Overview: General Education

2. Overview: Concept Inventories

3. Confluence: SLCI Results from S13 GE Natural Sciences Courses

4. Confluence: Math Concept Inventories for GE Quantitative Reasoning

5. Moving Forward: Next steps and segue into Part II

Assessment takes a village . . . Catherin Atkins, College of Science

Janet Bowers, MathematicsGeoff Chase, Undergraduate Studies

Douglas Deutschman, BiologyReynaldo Monzon, STAR

Chris Rasmussen, MathematicsStephen Schellenberg, Geological Science

Kathy Williams, CTL. . . with thanks to S13 GE Nat. Sci. Faculty

and Ed Nuhfer of Humboldt State

Fun facts:~1/3 of BS/BA is GE

WASC is interested in GEGE is integral part of mission

Goals and Capacities (G&Cs) of SDSU General Education Program

Essential Capacities Developed through General Education1. Construct, analyze, and communicate arguments2. Apply theoretical models to the real world3. Contextualized phenomena4. Negotiate differences5. Integrate global and local perspectives6. Illustrate relevance of concepts across boundaries7. Evaluate consequences of actions

Goals for Natural Sciences (Three Foundation, One Exploration)1. Explain basic concepts and theories of the natural sciences2. Use logic and scientific methods to analyze the natural world

and solve problems3. Argue from multiple perspectives about issues in natural

sciences that have personal and global relevance4. Use technology in laboratory and field situations to connect

concepts and theories with real-world phenomena

Goals for Quantitative Reasoning (One Foundation)1. Apply appropriate computational skills and use basic

mathematical concept to analyzed problems in the natural and social sciences

2. Use methods of quantitative reasoning to solve and communicate answers to real-world problems

Are these G&Cs being assessed at the

course level?If so, how?

How can these G&Cs be introduced, developed,

and demonstrated within and across courses?

How can we assessthese G&Cs at aprogram level?

One Path Forward for Programmatic Assessment andCurricular Revision is the Concept Inventory

Definition:Collection of questions designed to assess student understanding of the

foundational knowledge, concepts, and procedures for a given topic, discipline, etc.

We hypothesize that such concept inventories provide a means for programmatic assessment and curricular revision that will serve our students, faculty, and institution

Goals: • Assess scientific habits of mind and literacy

• Identify gaps in approach understanding• Inform curricular reform

Inform on multiple levels in multiple ways when linked with demographics:From synoptic overview of student population and needs to

insights from distribution of distractor responses

Concept inventories have traditionally been discipline-focused,but recent efforts include metadisciplinary assessment of science literacy

Two Example Questions from 25-Item Science Literacy Concept Inventory(SLCI of Nuhfer et al., pers. comm.)

6. To help us to understand the lunar phases, we have set up a basketball, a baseball, and a golf ball to represent respectively the Sun, Earth and the moon. What method of science are we employing? A. Experiment. Moving the balls can allow us to measure the size of the

shadow that one ball casts on another ball.B. Modeling. Moving the balls helps us to perceive the positions of the

celestial bodies that might explain the observed phases.C. Multiple working hypotheses. Moving the balls can allow us to determine

whether the lunar phases were different during the ice ages.D. None. Moving the balls differs from reality to such an extent that it is an

ineffective way to understand lunar phases.

24. Which of the following assumptions is important to all sciences?A. Humans can understand the physical world through laws they can discover.B. The experimental method is the only valid way to test hypotheses.C. Life is not governed by the same physical laws as non-living systems.D. Random events have no role in the actual physical world.

GE Capacities:

1 2 3 4 5 6 7

GE Goals (Nat. Sci.):

1 2 3 4

GE Capacities:1 2 3 4 5 6 7

GE Goals (Nat. Sci.):1 2 3 4

2,794 InitialRecords

SLCI?870(31%) No

Yes

Non-Response Bias:Female 25.2%Male 39.1%

GEOL303 9.2%BIOL100 18.3%BIOL326 43.3%ASTR310 56.4%

Native?548

(20%)No

Yes

CHEM100?216(8%) Yes

No

Missing Data67

(2%)Yes

No

1,093 Retained Records (39%)

SAT/ACT Not Reported:Native 3.8%Transfer 60.8%

Taking Foundations:Native 71.6%Transfer 31.4%

First Generation:CHEM100 29.3%All Others 19.0%

Minority:CHEM100 73.7%All Others 54.3%

Summary of Results from SCLI Deployment across Suite ofGE Foundation and Exploration Courses during early Spring 2013

Demographics Student Preparedness and Performance

SDSU “Treatment”

Age SAT (ACT) Score Student Level

Gender GPA Foundations, Exploration

Ethnicity, Race Total Units Completed

First Generation Course Modality

English as 1st Language Major (Sci., Eng., Other)

Service Area Residence Hall

Overall SLCI Performance and

Potential Explanatory Variables

Correlations of Explanatory Variables with SLCI Performance

Univariate (Unadjusted) Multivariate (R2=20.5%)Explanatory

Variable R2 F p R2 F p

Age 0.5% 1.93 0.12 0.0% 0.19 0.91Gender <.1% 0.42 0.51 0.2% 2.25 0.13Ethnicity / Race 4.1% 4.57 <.001 1.2% 1.63 0.09First Generation 1.3% 7.14 <.001 0.0% 0.33 0.72English as 1st Lang. 0.4% 4.78 0.03 0.1% 1.27 0.26Service Area <.1% 0.15 0.70 0.2% 2.66 0.10SAT (ACT) Score 12.6% 157 <.001 7.1% 95.3 <.001GPA 6.3% 73.4 <.001 1.9% 25.2 <.001Student Level 1.8% 6.70 <.001 0.1% 0.52 0.67Found/Exploration 1.1% 11.6 <.001 0.0% 0.11 0.74Units Completed 2.4% 27.1 <.001 0.2% 2.93 0.09Course Modality 0.3% 1.76 0.17 0.6% 3.74 0.02Major (Sci, Eng, Oth) 0.4% 1.98 0.14 0.1% 0.74 0.48Residence Hall <.1% 0.19 0.66 0.1% 1.21 0.27

Correlations of SAT (ACT) and GPA with SLCI Performance

R2 = 12.6% R2 = 6.3%

Freshman Sophomore Junior Senior Total

Exploration 2 (<1%)

66(21%)

154(49%)

91(29%) 313

Foundation 340(44%)

351(45%)

62(8%)

27(3%) 780

Aside: Snapshot of when Foundation and Exploration

courses are being takenby our students . . .

Implications forscaffolding, advising, and

degree progress?

Explorations

R2 = 5.2%R2 = 6.0%

Foun

datio

ns

R2 = 7.0%R2 = 15.0% Patterns, trends, and explanatory hypotheses?

SLCI Performance Across Courses

Course 18 19-21 22+ Sci. Major Units Earned

BIOL32x 0% 76% 24% 30% 95 (+/- 29)

ENVS100 43% 53% 4% 14% 42 (+/- 23)

Course 18 19-21 22+ Science Major

BIOL32x 0% 76% 24% 30%

ENVS100 43% 53% 4% 14%

Course 18 19-21 22+

BIOL32x 0% 76% 24%

ENVS100 43% 53% 4%

ENVS100BIOL32x

COURSE2$

700 800 900 1,000 1,100 1,200 1,300 1,400SAT (ACT) Score

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

SLC

IENVS100BIOL32x

COURSE2$

700 800 900 1,000 1,100 1,200 1,300 1,400SAT (ACT) Score

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

SLC

I

Performance difference cannot be attributed to incoming SAT scores

Performance difference may be partially explained by differences in age, major, units earned, and

other confounded variables

End-Member Comparison

Performance difference cannot be attributed to incoming SAT scores

Performance difference may be partially explained by differences in age, major, units earned, and

other confounded variables

End-Member Comparison

ENVS100BIOL32x

COURSE2$

700 800 900 1,000 1,100 1,200 1,300 1,400SAT (ACT) Score

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

SLC

IENVS100BIOL32x

COURSE2$

700 800 900 1,000 1,100 1,200 1,300 1,400SAT (ACT) Score

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

SLC

I

Univariate (Unadjusted) Multivariate (R2=35.9%)

Variable R2 F p R2 F pCourse 19.9% 27.4 <.001Gender

SAT (ACT) Score

GPA

Student Level

Units Completed

Total 19.9%

End-Member Comparison

Performance difference cannot be attributed to incoming SAT scores

Performance difference may be partially explained by differences in age, major, units earned, and

other confounded variables

End-Member Comparison

ENVS100BIOL32x

COURSE2$

700 800 900 1,000 1,100 1,200 1,300 1,400SAT (ACT) Score

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

SLC

IENVS100BIOL32x

COURSE2$

700 800 900 1,000 1,100 1,200 1,300 1,400SAT (ACT) Score

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

SLC

I

Univariate (Unadjusted) Multivariate (R2=35.9%)

Variable R2 F p R2 F p

Course 19.9% 27.4 <.001 4.7% 8.22 0.01

Gender 0.1% 1.00 0.32

SAT (ACT) Score 3.7% 5.75 0.02

GPA 2.6% 3.93 0.03

Student Level 0.1% 0.14 0.94

Units Completed 0.1% 0.10 0.77

Total 19.9% 11.3%

End-Member Comparison

Performance difference cannot be attributed to incoming SAT scores

Performance difference may be partially explained by differences in age, major, units earned, and

other confounded variables

End-Member Comparison

ENVS100BIOL32x

COURSE2$

700 800 900 1,000 1,100 1,200 1,300 1,400SAT (ACT) Score

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

SLC

IENVS100BIOL32x

COURSE2$

700 800 900 1,000 1,100 1,200 1,300 1,400SAT (ACT) Score

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

SLC

I

Majority of the R2 is not uniquelyassignable since it is shared amongrelated variables (i.e., confounded)

End-Member Comparison

Univariate (Unadjusted) Multivariate (R2=35.9%)

Variable R2 F p R2 F p

Course 19.9% 27.4 <.001 4.7% 8.22 0.01

Gender 0.1% 1.00 0.32

SAT (ACT) Score 3.7% 5.75 0.02

GPA 2.6% 3.93 0.03

Student Level 0.1% 0.14 0.94

Units Completed 0.1% 0.10 0.77

Total 19.9% 11.3%

Demographics Student Preparedness and Performance

SDSU “Treatment”

Age SAT (ACT) Score Student Level

Gender GPA Foundations/Exploration

Ethnicity / Race Total Units Completed

First Generation Course Modality

English as a 1st Language Major (Sci, Eng, Other)

Service Area Residence Hall

A Working Model . . . Limited by the Nature of the Current Data

Next Steps: Deploy more broadly over multiple semesters to allow focuson science literacy gains through time within individuals

SLCI

Alternate or Parallel Concept Inventory?Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012)

Alternate or Parallel Path?Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012)

Alternate or Parallel Path for GE Natural Sciences?Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012)

Ongoing Efforts:Programmatic: Mapping Concept Inventory skills onto GE G&Cs

Administrative: Implementing deployment across GE Nat. Sci. coursesGrass-Roots: Using course content to introduce, practice, and master GE G&C

TECH

TRADITIONAL

EXPERIENCED

FLIPPED

September 2012Pre-Test

December 2012Post Test

TECH

TRADITIONAL

EXPERIENCED

FLIPPED

From GE Natural Science to GE Quantitative Reasoning:Fall 2012 Calculus Concept Inventory

Goals: • Assess scientific habits of mind and literacy

• Identify gaps in approach understanding• Inform curricular reform

Operate and inform on multiple levels in multiple ways:Synoptic overview of student population to

insights from distribution of distractor responses

Pre-Post Test Gains:Fall 2012 Calculus Concept Inventory

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

-.100

-.050

.000

.050

.100

.150

.200

.250

16. The drawing represents a loaf of bread with a slice shown x inches from the left-hand end of the bread. Which of the following graphs could represent the volume V of the bread to the left of the slice as a function of the distance x from the left-hand end of the slice?

Example Question

Operate and inform on multiple levels in multiple ways:Synoptic overview of student population

ASSESSING GE GOALS AND CAPACITIES WITH CONCEPT INVENTORIES:ONE PATH FORWARD . . . PART I

1. Overview: General Education

2. Overview: Concept Inventories

3. Confluence: SLCI Results from S13 GE Natural Sciences Courses

4. Confluence: Math Concept Inventories for GE Quantitative Reasoning

5. Moving Forward: Next steps and segue into Part II

Assessment takes a village . . . Catherin Atkins, College of Science

Janet Bowers, MathematicsGeoff Chase, Undergraduate Studies

Douglas Deutschman, BiologyReynaldo Monzon, STAR

Chris Rasmussen, MathematicsStephen Schellenberg, Geological Science

Kathy Williams, CTL. . . with thanks to S13 GE Nat. Sci. Faculty

and Ed Nuhfer of Humboldt State

Fun facts:~1/3 of BS/BA is GE

WASC is interested in GEGE is integral part of mission