COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS

Rebecca Orr, Ph.D.

Professor of Biology

Agenda• Efforts made to improve student success:

• Online homework using MasteringBiology• Peer tutoring• Online quizzing •  

• Strategies for data collection:• Preparing to begin• Modifying as collection progresses• Involving your students

• Case study: Impact of pre-exam quizzing on student outcomes

• Lessons learned

Student Success Challenges

• Have you ever heard: • “I studied so hard, and it just didn’t work!”

• My question: • How can student success be improved in the introductory biology (majors) course?

My Evolution of Mastering Use

• Began using Mastering to create assignments and made them available for optional student use. • Student feedback indicated a trend:

• Successful students were using Mastering • Course evaluation surveys by students said:

“Make Mastering mandatory!”• Less successful students did not tend to utilize

Mastering

• Moved to making Mastering mandatory by assigning a portion of lecture credit (10%) to Mastering.

Is There a Significant, Positive Correlation Between Completing MasteringBiology and Exam Scores?

• Students required to complete MasteringBiology homework assignments (10% lecture grade).

• Overall exam average was compared to percentage MasteringBiology completed by end of semester.

• Spearman's Rank Order correlation was run:(rs(138) = 0.571, P = .000)

• A statistically significant, positive correlation between percentage of MasteringBiology homework completed and exam average was found.

Results

• Students completing more work in MasteringBiology were more successful on exams.

Problem: How to Increase Student Success in Introductory (Majors) Biology?

• My Strategy, Spring 2011 semester:

• Evaluate the efficacy of providing peer tutors vs. requiring MasteringBiology homework (using exam scores to measure)• Offered optional, online pre-exam quizzes to all using

MasteringBiology

• Results:

• Peer tutoring appears to have no benefit over offering required MasteringBiology homework.

• Significant, positive correlation between participating in optional quizzing and increased exam score observed.

Cognitive Science of Learning

• Introduction of Desirable Difficulties:

“Conditions of instruction that appear to create difficulties for the learner, slowing the rate of

apparent learning, often optimize long-term retention and transfer.”

Testing Serves as Learning Events

• Testing as a learning vehicle vs. an assessment vehicle.

• Evidence shows: Practice of recalling information is more powerful than re-presentation of information.

• Reality is: Students re-present information to themselves (i.e., reading a chapter over and over again, underlining things, etc.), focusing on storage rather than retrieval.

Can An Online Homework Platform Be Used For Quizzing to Increase Student Success?

• Introduced required pre-exam quizzing delivered via MasteringBiology

• Students required to take pre-exam quizzes during the Fall 2011 semester (n=199) and Spring 2012 semester (n=174)

• My Question: Does taking pre-exam quizzes using

Mastering result in significantly higher exam scores?

Preparation

• Before beginning, set up spreadsheet to collect data necessary to answer questions.

• My predesigned columns included:• Name (deleted prior to sending to statistician)• Exam # and grade• Number of pre-exam quizzes taken• Grade on each pre-exam quiz• Set up color coding to designate “drops” and “not in my course”

• Data gathered by exporting grade book from Mastering.

Student Participation Compliance in Required Pre-exam QuizzingFall 2011 & Spring 2012

% Students Completing Quizzes prior to Exam

% Quizzes Completed

Exam 1

n= 362

Exam 2

n= 339

Exam 3

n= 321

Exam 4

n= 305

100% 78.5 59.9 62.9 59.3

67% 16.3 22.4 * *

50% * * 26.5 25.9

33% 3.0 11.5 * *

0% 2.2 7.4 10.3 14.8

Comparison of Exam Averages100% Quiz Takers to 0% Quiz Takers

Fall 2011

Spring 2012

Comparison of Exam Averages by Percentage of Quiz Taking: Spring 2012

Analysis by Groups

• Scores on Exams 1 and 2 were averaged, and students were grouped by these averages.

• Groups Established:• Pass: 70%+• Fail: <70%• Low (“weak student”): <60%• Middle (“average student”): 60%-80%• High (“strong student”): >80%

• Student scores were compared by group for 100% quiz takers vs. 0% quiz takers.

Performance of Passing vs. Failing Students: 100% Quiz Takers vs. 0% Quiz Takers

Exam 3

Exam 4

Performance by Low, Middle and High Students: 100% Quiz Takers vs. 0% Quiz Takers

Exam 3

Exam 4

Data Analysis Often Generates New Questions!

• Can students’ compliance with pre-exam quiz requirement be increased?

• If percent pre-exam quiz compliance increases, will significant gains in student success in the course be observed?

Impact of Data Collection on Course Approach • Tell the students about the data!

• Ask them to predict! Who earns higher exam scores:• Quiz takers or non quiz takers?• Quiz takers that fail their quizzes or non quiz takers?

• Show them the data, and ASK:• “What group do you want to be in?!”

The blue group? Or the red group?!

Pre-exam Quiz Participation Rate Increased Significantly in Fall 2012 vs. Spring 2012

Sprin

g: 100% Parti

cipati

on

Fall:

100% Parti

cipati

on

Sprin

g: 0% P

articip

ation

Fall:

0% Parti

cipati

on0

1020304050607080

Exam 1Exam 2Exam 3Exam 4

Percentage of ABC vs. DFW Grades Earned in Spring 2012 vs. Fall 2012

ABC DFW0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

SpringFall

Study Results• Student participation in pre-exam quizzing results in

significantly higher exam grades.

• Benefit realized to student is based on participation (rather than performance) in quizzing.

• Participation in pre-exam quizzing results in significantly higher exam scores for students of diverse abilities.

• Stronger students may see slightly greater benefits from pre-exam quizzing (as evidenced by exam averages).

• Significant increase in student compliance with required quizzing was accompanied by significantly higher percentage of ABC rates vs. DFW rates.

Student Perception of Mastering: Homework and Pre-Exam Quizzing

Mas

terin

g Hom

ewor

k Help

ed M

e Kee

p ...

Mas

terin

g Hom

ewor

k In

crea

sed

Und...

Quiz

zes

Incr

ease

d Und

erst

andin

g

Quiz

zing

Result

s Le

d to

+ S

tud.

..

Plan to

con

tinue

quiz

zing

in fu

ture

0102030405060708090

100

(% S

tud

ent

Ag

reem

ent)

Study Conclusions

• Pre-exam quizzing using online platform seems to provide a viable testing environment for student learning.

• Using online platform allows instructors more time to teach!

• All students showed positive gains in exam scores as a result of participating in pre-exam quizzing.

Lessons learned

• Begin process of gaining IRB approval right away.

• Consult with your statistician about what you want to investigate before you start.

• Speak with colleagues about your investigation to get a feel for questions they may raise.

• If introducing a new teaching method or technology, consider giving a student survey to gather their thoughts and comments.

Lessons learned (cont.)• Design your data collection tools before beginning the

study.• Gather and organize more than you think you need!• Stay current in updating master spreadsheet throughout

the semester:• Falling behind can become overwhelming!

• New questions often arise as you review the data.

• Modifications in what you record may be necessary. Much easier to do this “as you go” than to go back and wade through the entire semester’s worth of data!

• Tell students about your study- involve them in the process when possible.

Benefits of Studying Student Success Data

• Allows you to identify what you are doing that is truly effective.

• Sharing results with students motivates them and helps them to feel that your requirements will yield positive gains for them.

• Provides intellectual challenge to you that provides additional purpose and interest!

Acknowledgements• Shellene Foster

Professor of Mathematics, Collin College

• Pushpa Ramakrishna, Ed.D. Professor of Biology, Chandler-Gilbert Community College

References• 1. Bjork, E. L., & Bjork, R. A.  (2011). Making things hard on yourself, but in a good way:

Creating desirable difficulties to enhance learning.  In M. A. Gernsbacher, R. W. Pew, L. M. Hough, & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56-64).  New York: Worth Publishers.

• 2. Halamish, V., & Bjork, R. A. (2011). When does testing enhance retention? A distribution-based interpretation of retrieval as a memory modifier. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 801-812.