Nigel WardUniversity of Texas at El Paso

Fifth International Conference on Intelligent Technologies December 3, 2004

Dealing with Uncertainty in a Model of

Computer Science Graduate Admissions

(a 12 minute pre-banquet talk

at a small 3-day gathering

of soft-computing researchers)

The Problem

10,000+ CS grad school applicants a year

many wasted applications

some disappointed applicants

A Solution

enable applicants to predict acceptance decisions,using a web tool

a model of applicant strength + models of admissions criteria

demonstration

The Acceptance Estimator Concept

How to Combine GRE Scores?How to Combine GRE Scores?

Two common styles: avg/sum and min:

“we expect a GRE V+Q+A of at least 2100”

“we expect at least 600 V, 700 Q and 650 A”

A compromise: stronger scores weighted less, but not zero*

1.33 for weakest, 1.0 for middle, .67 for strongest

(an ordered weighted averaging operator)

* cf Carlsson, Fuller and Fuller in Yager and Kacprzyk, 1997

Sample ComputationSample Computation

raw value (RV)

normalized value NV

rank R

ranking factor RF

contribution level CL

verbal 600 100 #1 .67 67

quantitative 650 0 #3 1.33 0

analytical writing

4.5 62 #2 1.00 62

Explaining Apparent DiversityExplaining Apparent Diversity

admissions policy for department x

standardmodel ofapplicantstrength

> GQ

department-specificthreshold

x

omissions

simplific

ations

guesses

fog

X’s published

admissions

policy and

statistics

spin

Estimating the Scaling ParametersEstimating the Scaling Parameters

To apply OWA, we must normalize scores first

what is the GRE Q score corresponding to a 3.7 UTEP GPA?JNTU

y = 0.001x + 0.6692

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

-100 -50 0 50 100 150 200

GRE

GPA 系列1

線形 (系列1)

GRE Composite

GP

A

JNTU

Mumbai

y = 8E-06x + 0.5651

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

-300 -250 -200 -150 -100 -50 0 50 100 150

GRE

Grades 系列1

線形 (系列1)

Mumbai

UTEP

y = -0.0039x + 2.9309

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

-120 -100 -80 -60 -40 -20 0

GRE

Grades 系列1

線形 (系列1)

U. Texas at El Paso

Weighting the ScoresWeighting the Scores

factor

GRE V

IW

.7

GRE Q 1.0

GRE AW .7

GPA (if US) 2.5

GPA (JNTU, Madras) 2.5

GPA (other Indian) 2.0

…

letters of recommendation varies

∑CL x IW i

∑IW ii

ii

CGRE =

Complexities in RecommendationsComplexities in Recommendations

• commeasurate with GREs and GPA

• can be a plus or a minus

• are fundamentally optional

• are not expected to have specific points

so no ranking factors

• vary in influence

so the importance weight computation is vital

Modeling RecommendationsModeling RecommendationsLeading recommender is a describing you as a

= weight

scaling factor = warmth score

Summary of the ComputationSummary of the Computation

Subtract Baseline and Scale

Raw to get Normalized Value:

Weight and Sum:

Order Normalized Values

and apply Ranking Factors

to get Contribution Levels:

∑CL x IW i

∑IW ii

ii

GQ =

NV = (RV - BV ) x SFi i ii

CL = NV x RFi ii

where r is rank, n is number of scores

RF =2 r - 1

3 n - 1( 1+ )

i

Factors in Admissions DecisionFactors in Admissions Decisionss

In the Model• GREs• GPA• in-major or recent GPA• major• letters of recommendation• statement of purpose• scholarships• group membership

Not in the Model• undergrad school• GRE subject test (CS)• TOEFL• nationality/culture• specific coursework• research match• publications• etc.

EvaluationEvaluation

55 UTEP applicant datafiles

accept / rejectcompare

compute GQ score

> -25?

applicant features accept/reject decisions

51/55 successes

with failuresexplicable

Modeling Other DepartmentsModeling Other Departments

compute GQ score

applicant data

> accept / reject

threshold for school X

published data for school X

compute GQ score

- 250

- 200

- 150

- 100

- 50

0

50

100

150

200

0 1 2 3 4 5

NRC Effectiveness

CG

RE

overall

Does the Model Work for DepartmentDoes the Model Work for Departments?s?

- 250

- 200

- 150

- 100

- 50

0

50

100

150

200

0 1 2 3 4 5

NRC Effectiveness

CG

RE

overalltrend

Does the Model Work for DepartmentDoes the Model Work for Departments?s?

Thus selectivity, as measured by the model, correlates with desirability, somewhat

The Diversity Behind the NumbersThe Diversity Behind the Numbers

Minimum scores of 550, 600 and 3.5 on the verbal, quantitative, and analytical writing sections, respectively (U. of Delaware)

Most students admitted have earned scores in excess of 650 for the Analytical and Quantitative parts (Columbia)

Average scores of successful applicants to this program for Fall 2002: GRE: 560 verbal, 770 quantitative (U. of Houston)

- 250

- 200

- 150

- 100

- 50

0

50

100

150

200

0 1 2 3 4 5

NRC Effectiveness

CG

RE

overall

- 250

- 200

- 150

- 100

- 50

0

50

100

150

200

0 1 2 3 4 5

NRC Effectiveness

CG

RE

overalltrend

- 250

- 200

- 150

- 100

- 50

0

50

100

150

200

0 1 2 3 4 5

NRC Effectiveness

CG

RE

averageconj. of minsmost- aboveminimum sumaverageconj. of mins

Averages, Minimums, and ThresholdsAverages, Minimums, and Thresholds

inferred threshold

- 250

- 200

- 150

- 100

- 50

0

50

100

150

200

0 1 2 3 4 5

NRC Effectiveness

CG

RE

overall

- 250

- 200

- 150

- 100

- 50

0

50

100

150

200

0 1 2 3 4 5

NRC Effectiveness

CG

RE

overalltrend

- 250

- 200

- 150

- 100

- 50

0

50

100

150

200

0 1 2 3 4 5

NRC Effectiveness

CG

RE

averageconj. of minsmost- aboveminimum sumaverageconj. of mins

Averages, Minimums, and ThresholdsAverages, Minimums, and Thresholds

inferred threshold

threshold vs. min: ~30 (0.15 GPA points)==> departments don’t take risks (?)

avg vs. threshold: ~20 (0.1 GPA points)==> departments don’t have much variety (?)

A View of the Applicant PoolA View of the Applicant Pool

Number ofApplicants

Overall Applicant Strength (GQ score)

minimum average

acceptees

A Blurred ViewA Blurred View

Number ofApplicants

Applicant Strength measured by GREs only

minimum average

acceptees

Modeling Other DepartmentsModeling Other Departments

compute GQ score

applicant data

> accept / reject

threshold for school X

published data for school X

compute GQ score

Modeling Other DepartmentsModeling Other Departments

compute GQ score

applicant data

> accept / reject

threshold for school X

published data for school X

compute GQ score

adjustment

3010soft minimums

4010hard minimums

20-20average

40-30most above

marginadjustmentdescription

Presenting UncertaintyPresenting Uncertainty

Some Sources of UncertaintySome Sources of Uncertainty

• user interface errors

• lack of information about the applicant

• incorrect fundamental assumptions

• incorrect GQ-model parameters

• incorrect modeling of departments’ criteria

• inadequate information on departments

Try it Yourself!Try it Yourself!

http://www.cs.utep.edu/admissions/

Future WorkFuture Work

• verification on data from more departments

• better parameter estimates on more data

• a more parameterized version to model different departments better

• a centralized clearinghouse?

Benefits for UTEPBenefits for UTEP

• better informs potential UTEP applicants

• increases site traffic, and applicant pool?

• increases Google score

• shows we understand student needs

• makes the world a better place