Analysis and Communication of US News Rankings using Monte Carlo Simulations: A Comparison to Regression Modeling

Presented by Chris Maxwell

Purdue University

AIR 2010

Introduction

What changes in submitted data most influence our US News rankings?

• Identify key data elements• Provide realistic expectations of future rank

This presentation will focus on the US News graduate program in education rankings

Results will also be presented for graduate business and national universities rankings

Import into Excel and use ordinary linear regression (OLS) to model the US News score:

Initial Analysis

Started with US News data from website:

OLS Problems

Variable rejections

Multicollinearity

Model variability - which model is “right” ?

Counterintuitive results

OLS Problems (continued)

Models can be extremely accurate, but communication of results becomes very problematic

Is there another way to model the score using the same data?

US News Methodology

US news scores are z-score based:• (observation - mean)/standard deviation

In general, each institution’s z-scores are:• multiplied by the US News weight• totaled• the highest total is scaled to 100

Not all calculation details are known and some data are missing

Monte Carlo Simulation

Can a US News-type equation be simulated that calculates the US News scores?

•18 unknowns, but 50 observations…

The equation framework is input into an iterative Excel VBA program

Reasonable ranges are defined for the 18 unknown standard deviations and “means”

Monte Carlo Simulation (continued)

For each iteration (~40,000) in a run:•Randomly chose all unknowns•Compute score for each institution•Rescale so top score is100•Compute sum of squared errors

The best-fit equation is saved, algebraically rearranged, and compared to regression

Refine the model and repeat the process

Model Comparisons

Graduate Education 2009: Regression Monte Carlo

Intercept -34.0 -35.8Peer Survey 8.45 8.80

Superintendent Survey 8.70 9.48GRE Verbal 0.053 0.033

GRE Quantitative 0.000 0.013Acceptance Rate -11.50 -9.54

Student/Faculty Ratio 0.002 -0.182Doctorates Produced/Faculty -2.73 -2.25

Research Funds (millions) 0.500 0.496Funds/Faculty (thousands) 0.026 0.027

R Squared 97.6% 97.4%

Model Comparisons (continued)

Graduate Business 2009: Regression Monte Carlo

Intercept -145 -145Peer Survey 8.91 8.36

Recruiter Survey 6.04 5.78Undergraduate GPA 17.5 17.4

GMAT score 0.071 0.074Acceptance Rate -2.11 -2.04

Starting Salary 1.7E-04 1.9E-04Employed at Graduation 10.8 10.2

Employed at 3 months 27.5 27.6

R Squared 99.96% 99.95%

Model Comparisons (continued)

Graduate Education 2010: Regression Monte Carlo

Intercept -38.1 -36.6Peer Survey 11.2 10.7

Superintendent Survey 7.96 8.52GRE Verbal 0.024 0.021

GRE Quantitative 0.016 0.016Acceptance Rate -7.40 -8.97

Student/Faculty Ratio -0.420 -0.343Doctorates Produced/Faculty -1.79 -2.13

Research Funds (millions) 0.417 0.420Funds/Faculty (thousands) 0.028 0.027

R Squared 99.2% 99.1%

National Universities 2009: Regression Monte Carlo

Intercept -96.8 -86.1Peer Survey 11.3 11.1

Graduation Performance 10.4 10.3Classes under 20 13.5 12.6

Classes 50 and over -6.50 -5.06Student/Faculty ratio 0.065 -0.054

Full-time Faculty 0.05 1.41SAT 25th 0.014 0.006SAT 75th -0.001 0.006

High School Rank 10.1 11.7Acceptance Rate -2.74 -2.84

Faculty Terminal Degrees 8.80 8.13Graduation Rate 19.0 16.3

Retention Rate 5.82 7.04Alumni Giving 19.1 16.2

IPEDS Finances ratio (log) 11.15 8.76Faculty Salaries 4.2E-05 7.5E-05

R Squared 99.6% 99.5%

National Universities 2009 (stdevs): Actual Monte Carlo

Peer Survey 0.7 0.5Graduation Performance 7% 11%

Classes under 20 14% 11%Classes 50 and over 6% 9%

Student/Faculty ratio 4.3 4.2Full-time Faculty 10% 16%

SAT 25th 140 143SAT 75th 134 153

High School Rank 27% 12%Acceptance Rate 21% 12%

Faculty Terminal Degrees 9% 8%Graduation Rate 18% 22%

Retention Rate 9% 13%Alumni Giving 10% 7%

IPEDS Finances ratio (log) 0.24 0.26Faculty Salaries $16,700 $21,100

Summary

Conclusions

Cautions

Questions / Discussion