B E N J A M I N G A M B O A , R E S E A RC H A N A LY S TC R A F T O N H I L L S C O L L E G E

RESEARCHING:ALPHA TO ZETA

SESSION OBJECTIVES

• Participants will be able to:• Apply proper controls and create a dataset for a

research study• Evaluate multiple statistical analyses, such as

statistical and practical significance, logistic regression, and segmentation modeling, for appropriateness to the study• Facilitate conversations around findings to

effect change at their college

MOTIVATION

• Communications professor observed 100% success rates in short-term courses• Considerable literature on accelerated and

compressed coursework• Title III STEM Grant was considering compressed

sequencing

CONTROLS

• Reviewed literature for methods• Determined 8-week control would provide most

data and usable results• Other controls: college, course, course length,

faculty, course type, and term (fall and spring only)• Disaggregation: age, gender, and ethnicity

CREATING DATASET

• A total of 4,592 records (over 5 years) were identified for the treatment group• Applying the controls, 11,002 records were

identified for the control group• Variables created: Course success, cumulative

prior GPA (continuous), cumulative prior GPA (normalized), student’s age in term, first-time students

STATISTICAL TECHNIQUES

• Statistical significance: p-value• Practical significance: effect size (Cohen’s d)• Predictive analytics: logistic regression &

segmentation analytics

STATISTICAL TECHNIQUES

• Statistical significance: p-value• Practical significance: effect size (Cohen’s d)• Predictive analytics: logistic regression &

segmentation analytics

STATISTICAL SIGNIFICANCE

• P-value is a well known statistic that faculty and decision-makers understand (or at least think they do)• Remember, the p-value is impacted by the

sample size!

• Use responsibly

STATISTICAL TECHNIQUES

• Statistical significance: p-value• Practical significance: effect size (Cohen’s d)• Predictive analytics: logistic regression &

segmentation analytics

PRACTICAL SIGNIFICANCE

• Effect size (Cohen’s d): difference of the two means divided by the pooled standard deviation

• Using Cohen as a guide:• Small effect ≥ 0.20• Medium effect ≥ 0.50• Large effect ≥ 0.80

STATISTICAL TECHNIQUES

• Statistical significance: p-value• Practical significance: effect size (Cohen’s d)• Predictive analytics: logistic regression &

segmentation modeling

PREDICTIVE ANALYTICS

• Logistic Regression• Predicts binary outcome (i.e. success, no

success)• Standard package in statistical software• Supports (A LOT) of continuous and

dichotomous predictor variables• Assumes lack of relationship between predictor

variables

PREDICTIVE ANALYTICS

• Logistic Regression• Dummy coding• Multicollinearity• Missing values• Overfitting the model• Selecting the cut off value• Choosing the best model• Goodness-of-fit test• Interpretation of results

PREDICTIVE ANALYTICS

• Segmentation Modeling• Classification tree (CRT) algorithm• Standard in some packages, add-on for others• Partitions cases along predictor variables where

similar outcomes are grouped together• Visual output that is easy to describe and

understand

RESULTSC

om

pre

ssed c

ours

es

0% 25% 50% 75% 100%

75%

69%

Success Rates by Course Length

RESULTS

Higher GPA

Lower GPA

0% 25% 50% 75% 100%

85%

77%

70%

57%

Traditional courses

Condensed courses

Success Rates by Course Length for Students with Higher and Lower than Average Prior GPAs

RESULTS

Reading

English

0% 25% 50% 75% 100%

92%

88%

85%

71%

Traditional courses

Compressed courses

Success Rates by Course Length and Subject

RESULTS

Predictor β p Exp(β)

Course Length .44 <.001 1.553

Prior GPA .73 <.001 2.083

• Course length has an odds ratio of 1.553 and a positive β coefficient meaning a student enrolled in a compressed course is one and a half times more likely to succeed than a student enrolled in a traditional-length course.

• Prior cumulative GPA has an odds ratio of 2.083 and a positive β coefficient meaning as a student is two times more likely to succeed than a student with a GPA .73 points lower.

RESULTS

Segmentation model best predicted success for:1) continuing students with GPAs greater than 3.028 and2) students with GPAs between 2.457 and 3.028 enrolled in

condensed courses

1 2

CLOSING THE RESEARCH LOOP

• Publish and distribute a full report and dashboard

CLOSING THE RESEARCH LOOP

• Presentation to faculty chairs committee• Left the statistical language out• Focused on:• Correlated relationships• Variables that best predicted course success• Implications on teaching and learning

CLOSING THE RESEARCH LOOP

Fall 2014

Math 090

Math 942

Math 090

Math 942

Math 095

Math 952

Spring 2015

Math 095

Math 952

Transferable Math

Math 090

Math 095

Fall 2015

Transferable Math

Math 090

Transferable Math

Spring 2016

Math 095

CLOSING THE RESEARCH LOOP

• Provided impetus to grant program to identify interested faculty for a pilot program• Completed first term of Fast Track Math

Outcome

Non-Fast Track MATH-942

Fast TrackMATH-942 Effec

t Size (d)

P-value# N % # N %

Success in MATH-942 22 50 44.0 41 61 67.2 0.47 0.014

Retain to MATH-952 7 22 31.2 41 41 100.0 1.59 <0.001

Success in MATH-952 5 7 71.4 24 41 58.5 -0.26 0.523

Success in Sequence 5 50 10.0 24 61 39.3 0.66 <0.001

FINAL QUESTIONS AND THOUGHTS

BUT WHEN I DO, THEY’RE YOURS

I DON’T ALWAYS ANSWER QUESTIONS

RESOURCES

DesJardins, S.L. (2001). A comment on interpreting odds-ratios when logistic regression coefficients are negative. The Association for Institutional Research, 81, 1-10. Retrieved October, 15, 2006 from http://airweb3.org/airpubs/81.pdf

George, D., & Mallery, P. (2006). SPSS for windows step by step: A simple guide and reference (6th ed.). Boston: Allyn and Bacon.

Harrell, F.E. (2001). Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer Science+Business Media, Inc.

RP Group (2013). Suggestions for California Community College Institutional Researchers Conducting Prerequisite Research. Retrieved January 29, 2014 from http://www.rpgroup.org/sites/default/files/RPGroupPreqreqGuidelinesFNL.pdf

Tabachnick, B.G. & Fidell, L.S. (2007). Using Multivariate Statistics (5th ed.). Boston: Pearson Education.

Wetstein, M. (2009, April). Multivariate Models of Success. PowerPoint presentation at the RP/CISOA Conference, Tahoe City, CA. Retrieved January 28, 2014 from http://www.rpgroup.org/sites/default/files/Multivariate%20Models%20of%20Success.pdf

Wurtz, K. A. (2008). A methodology for generating placement rules that utilizes logistic regression. Journal of Applied Research in the Community College, 16, 52-58.