presentation 20 august 2014 (departmental meeting)

An Analysis of Grades in the Computer Science andSoftware Degree Programmes

20 August 2014

Hans Hüttel1 and Mikkel Meyer Andersen2

1Department of Computer Science

2Department of Mathematical Sciences

Aalborg UniversityDenmark

Part I

Background

24

An Analysis of Grades

Hans Hüttel and MikkelMeyer Andersen


A cause for concern

In 2012 there was concern about increasing failure rates incertain courses in the degree programmes in computer scienceand software. These were

I Discrete Mathematics (2nd semester) (DM)I Algorithms and Data Structures (3rd semester) (AD)I Syntax and Semantics (4th semester) (SS)I Computability and Complexity (5th semester) (BK)

Is there a pattern to these observations? Is itconnected to the backgrounds of students?

24




The four upper secondary education pro-grammes in Denmark

I The STX and HF programmes consist of a broad range ofsubjects in the fields of the humanities, natural scienceand social science.

I The HHX programme focuses on business andsocio-economic disciplines in combination with foreignlanguages and other general subjects.

I The HTX programme has its focus on technological andscientific subjects in combination with general subjects.

There are 146 schools with STX and/or HF, 60 offering HHXand 38 with HTX.

24




Admission requirements

I Prospective students must have passed certain subjects,including the level A course in maths, in order to beadmitted to our degree programmes in computer scienceand software.

I In each of the secondary programmes, prospectivestudents will have received four grades in each subject

I To pass, the mean of these four grades must be at least 2.Thus, you can pass if you receive e.g. the grades 4, 02, 02and 00 in maths.

24




The data material

We have obtained our data material from STADS.I 287 students have information about maths grades from

high schoolI 138 students have complete CS/SW grade information

(DM, AD, SS, BK)I 52 students overlap (high school grades and complete

CS/SW grades)

All students are enrolled in the current version of our degreeprogrammes (post-2010).Until 2012 no information was available concerning high schoolgrades.

The AAU system for registering student data

24




What are we analyzing?

We consider the following derived information.

MATHS denotes the mean of all 4 high school mathsgrades

GotU is set to Yes if, at any time, the grade ’U’(no-show at exam) was given in CS/SW courses(DM, AD, SS, BK), else No

Failed is set to Yes if, at any time, a grade -3 or 00 wasgiven in CS/SW courses (DM, AD, SS, BK), elseNo

Part II

Do students’ secondary school grades matter?

24




Where do students come from?

n Percent

HF 6 2.1HHX 8 2.8HTX 179 62.4STX 94 32.8

HF

HHX

HTX STX

24

68

1012

MATH

No significant difference between mean of MATHS for STX and HTX.

24




A connection with failure/no-show?

All 287 students (some had not passed all 4 courses that weconsider):

●●

No Yes

24

68

1012

GotU

MATH

No Yes2

46

810

12

Failed

MATH

GotU Mean Median SD Q025 Q975

No 7.25 7.62 2.90 2 12.00Yes 5.07 5.00 2.15 2 10.55

Failed Mean Median SD Q025 Q975

No 7.35 7.75 2.73 2.40 12Yes 5.93 6.00 2.97 1.68 12

Two-sided t tests for H0 of equal means in both tables are statistically significant.

24




School grades vs. failure at AAU

Only 52 students with complete data (none of which had a U):

●

●

No Yes

24

68

1012

Failed

MATH

Failed Mean Median SD Q025 Q975

No 8.60 8.50 2.16 3.53 12Yes 6.02 5.75 2.81 2.50 12

Two-sided t test for H0 of equal means is statistically significant.

24




School grades vs. average grade

HF

HH

X

HTX ST

X

−20

24

68

1012

CS/

SE g

rade

mea

n

n = 52

24




Correlations

Variable 1 Variable 2 n Correlation p value

MATHS DM 239 0.515 < 10�10

MATHS AD 125 0.408 2.3 · 10�6

MATHS SS 117 0.414 3.4 · 10�6

MATHS BK 53 0.597 2.4 · 10�6

24




Conditional independence

This model is based on students with complete information, i.e.n = 52.

GradeDM

GradeBK

MATH

GradeSS

GradeAD

MATHS and (SS, AD) are conditionally independent given DM and BK.

24




What if we had restricted admission?

A cut-off at grade 4:

MATHS < 4 (20%) MATHS � 4 (80%)

Failed No 26 159Yes 31 71

54% of those with MATHS < 4 failed and 31% of those with MATHS � 4 failed.

A cut-off at grade 7:

MATHS < 7 (48%) MATHS � 7 (52%)

Failed No 75 110Yes 62 40

45% of those with MATHS < 7 failed and 27% of those with MATHS � 7 failed.

Both contingency tables are statistically significant according to Fisher’s exact test.

24




What if we had restricted admission?

Comparison of average grades in the four courses (n = 52)FA

LSE

TRU

E

−20

24

68

1012

MATH >= 4

CS/

SE g

rade

mea

n

●●

FALS

E

TRU

E

−20

24

68

1012

MATH >= 7

CS/

SE g

rade

mea

n)

CS/SW grade mean Mean Median SD Q025 Q975

MATHS < 4 1.56 1.50 2.29 -1.37 4.83MATHS >= 4 5.91 5.88 3.55 -0.19 11.46

MATHS < 7 2.37 2.00 2.05 -1.16 5.8MATHS >= 7 6.89 7.75 3.47 -0.30 11.5

Part III

Computer Science vs. Software

24




Course grades

DM Mean Median SD Q025 Q975

CS 5.84 5.5 3.58 0 12SW 4.56 4.0 3.86 0 12

AD Mean Median SD Q025 Q975

CS 3.36 2 3.66 0 12SW 2.99 2 3.53 0 10

SS Mean Median SD Q025 Q975

CS 7.13 7 3.82 0 12SW 6.29 7 4.49 -3 12

BK Mean Median SD Q025 Q975

CS 6.45 7 4.30 0 12SW 5.13 7 4.82 -3 12

No statistically significant mean difference.

24




An overall picture

Overall Mean Median SD Q025 Q975

CS 5.70 4 4.08 0 12SW 4.74 4 4.35 -3 12

The difference of the means of 0.954 now becomes statisticallysignificant (p = 0.009). Mean difference 95% confidenceinterval is [0.24; 1.67].

24




Conditional independence

GradeDMGradeSS

GradeAD

GradeBK

Note the same structure as before, where MATHS wasincluded. This is a different sample of students (with completegrades for DM, AD, SS and BK).

24




Pairwise correlations

Variable 1 Variable 2 n Correlation

DM AD 138 0.638DM SS 138 0.667DM BK 138 0.608AD SS 138 0.620AD BK 138 0.544SS BK 138 0.817

All correlations statistical significant different from 0 with p value < 10�10.

Part IV

Some tentative conclusions

24




Some observations

I Data analyses based on data from STADS can be quiterevelatory!

I Maths grades from secondary school do matter both withrespect to failure and average grades.

I The independence model points towards there being aseparate problem for the Algorithms and Data Structurescourse.

24




What we cannot (and should not) do

I We cannot do anything about whatever happens in theDanish secondary school system.

I We cannot blame the course Discrete Mathematics forwhatever problems we may see.

24




What we might want to do

I Ensure a notion of progression through a collaborative,long-term joint effort by the lecturers involved in thecourses.

I Actively discourage students with low maths grades fromapplying.

I Conduct qualitative interviews to find out more about theunderlying rationales.

I Use data from STADS to perform other data analyses.

Do we want a notion of restricted admission?

Should we include an analysis of drop-out rates? !(Probably.)

All groups [focus groups comprised of drop-outs and MSc students] compare computer science with medical school. Medicine has a reputation as being a demanding degree programme, and it is difficult to be admitted to the programme is difficult. Because of this you know that you have to work hard in order to finish your degree. Entrance to the computer science programmes is easy, no-one is aware of what the studies really imply and therefore one has no expectations wrt. what is really required of you. The participants therefore suggest a change of the image that the degree programmes in computer science have. If one knows that getting a computer science degree is just as demanding as getting a degree in medicine, more people will realize right from the start that they do not have what it takes. (My translation)

A quote from the report Undersøgelse af frafaldet på datalogiuddannelserne

(SFI, 2009)

presentation 20 august 2014 (departmental meeting)

Education