gees nss workshop 2010 insights into interpreting the nss mark langan, alan fielding & peter...

GEES NSS Workshop 2010

Insights into interpreting the NSS

Mark Langan, Alan Fielding & Peter Dunleavy

Manchester Metropolitan UniversityFaculty of Science & Engineering

Fielding, A.F., P.J.Dunleavy and A.M. Langan (2010) Effective use of the UK's National Student (Satisfaction) Survey (NSS) data in science and engineering subjects. Journal of Further and Higher Education, 33, 347-368.

What makes a student satisfied?


Structure

• Research evidence based on a range of quantitative approaches using national dataset (science subject focus)

• Discussion about implications of using NSS for decision-making in H.E.


• Based on Course Experience Questionnaire and considered robust in terms of the statistical measures; internal consistency, construct validity and concurrent validity.

• Measures six dimensions: teaching; assessment and feedback (sometimes considered separately); academic support; organisation and management; resources; and, personal development.


•Five point scale with 21 ‘items’ and separate overall satisfaction rating (Q22)

•Thorough overview can be found in Surridge 2007 and Marsh and Cheng 2008. Take home message: the outputs are hierarchical in nature and not designed for simplistic league tables

•Note: satisfaction is a complex concept to measure and there are many approaches


DataLevel 3 (closest to Programme/Dept)NSS data froma)2007;b)2008; c) 2009.

Science subjects data pruned to remove subjects not taught at MMU (e.g. medicine).

Still very large data sets (>40,000 cases per survey)


NSS QuestionsTeaching (Teach)Q1 Staff are good at explaining things.Q2 Staff have made the subject interesting.Q3 Staff are enthusiastic about what they are teaching.Q4 The course is intellectually stimulating.

Assessment fairness (Fairness)Q5 The criteria used in marking have been clear in advance.Q6 Assessment arrangements and marking have been fair.

Assessment feedback (Feedback)Q7 Feedback on my work has been prompt.Q8 I have received detailed comments on my work.Q9 Feedback has helped me clarify things I did not understand.

NSS QuestionsSupport (Support)Q10 I have received sufficient advice and support with my studies.Q11 I have been able to contact staff when I needed to.Q12 Good advice was available when I needed to make study choices.

Management & Organisation (Org)Q13 The timetable works efficiently as far as my activities are concerned.Q14 Any changes in the course or teaching have been communicated effectively.Q15 The course is well organised and is running smoothly.

NSS QuestionsLearning Resources (Resources)Q16 The library resources and services are good enough for my needs.Q17 I have been able to access general IT resources when I needed to.Q18 I have been able to access specialised equipment, facilities or

room when I needed to.

Personal Development (PD)Q19 The course has helped me present myself with confidence.Q20 My communication skills have improved.Q21 As a result of the course, I feel confident in tackling unfamiliar

problems. Overall satisfaction (Overall)Q22 Overall, I am satisfied with the quality of the course.

SatisfactionSatisfaction is % of students answering 4 or 5 to a question.e.g. Q 1 Biology MMU – 95% of students were satisfied

Note that dissatisfaction may tell a different story.


Subject differences (general/national)

Subject differences confound simple comparisons, examples from 2009.

Medians for Qs 7, 8 & 9 plus 13, 14 & 15.


90.087.585.082.580.0

92.5

90.0

87.5

85.0

82.5

80.0

2007 Satisfaction Scores

2008, 2009 S

atisf

act

ion S

core

s

20082009

Variable

s25

s31

s59

s32

s12

s40

s11

s37

s35

s25

s31s59s32

s12

s40s11

s37

s35

Scatterplot of 2008, 2009 vs 2007

Subject differences (science/national)

Biology results (2008)

What answers are correlated with Q22?

Approach• Use % in agreement with a question (answers

4 & 5 on 5 point scale)• Simple correlation (ignoring subject)• Correlation allowing for subject differences

(ANCOVA)• Repeat for each year• Calculated for national dataset (three years)


Annual national trends (ranks)

Overall satisfaction is consistently related to:

Teaching Quality, Support and Organisation.

It only weakly related to Resources and Assessment, particularly feedback.

2009 2008 2007Teach 3.8 4.3 3.8 Assessment fairness 14.5 15.0 15.0 Assessment feedback 16.3 17.0 16.0Assessment 15.4 16.0 15.5Support 7.0 8.7 6.3Organisation 7.3 8.0 8.0Resources 20.0 14.7 20.0Personal development 11.7 13.0 11.7

Mean Rank


Subject differences (Feedback Qs)Question

Subject prompt detailed explained

Biological Sciences r 0.018 -0.130 -0.205

Physical Sciences r *0.440 *0.385 *0.589

Physical Geography r *0.675 *0.377 *0.566

Mathematical Sciences r 0.328 *0.460 *0.533

Computer Sciences r 0.226 0.103 *0.353

Mechanically based Engineering r 0.117 -0.190 0.192

Electrical and Electronic Engineering r 0.249 -0.233 -0.150

Technology r *0.728 0.090 0.237

Human Geography r 0.274 0.348 *0.433


Predictive model (Forest Tree Analysis)

• Decision and regression trees use a supervised learning algorithm which must be provided with a training set that contains cases with class labels or values for classification.

• We used a new variant of regression trees called FTA (software called ‘RandomForests’). Robust method with fewer constraints than traditional regression methods, for example allowing different factors to be explored in their influence on overall satisfaction within different subgroups.


Regression Trees (an example)based on http://www.dtreg.com/classregress.htm

Predicts property value

Effectiveness of Q1-21 to predict overall satisfaction (Q22) Predicting questionnaire item Inc MSE (%)Q15 - The course is well organised and is running smoothly 119.89Q1 - Staff are good at explaining things 71.45Q4 - The course is intellectually stimulating 66.71Q14 - Any changes in the course or teaching have been communicated effectively 60.79Q10 - I have received sufficient advice and support with my studies 55.34Q11 - I have been able to contact staff when I needed to 43.40Q3 - Staff are enthusiastic about what they are teaching 40.08Q2 - Staff have made the subject interesting 38.26Q12 - Good advice was available when I needed to make study choices 35.27Subject 32.35Q6 - Assessment arrangements and marking have been fair 20.10Q17 - I have been able to access general IT resources when I needed to 18.73Q19 - The course has helped me present myself with confidence 17.35Q18 - I have been able to access specialised equipment, facilities or room when I 15.41Q16 - The library resources and services are good enough for my needs 15.34Q20 - My communication skills have improved 13.29Q13 - The timetable works efficiently as far as my activities are concerned 13.16Q7 - Feedback on my work has been prompt 10.49Q9 - Feedback on my work has helped me clarify things I did not understand 6.65Q5 - The criteria used in marking have been clear in advance 6.60Q21 - As a result of the course, I feel confident in tackling unfamiliar problems 3.32Q8 - I have received detailed comments on my work 3.04

Predictive model (FTA shown as ranks)

Predictor 2007 (%)

2008 (%)

2009 (%)

Teaching 27.7 27.8 24.8 Fairness 10.7 13.6 6.9 Feedback 3.5 1.7 3.1Assessment 14.3 15.3 10.0Support 18.9 18.9 17.5Organisation 26.2 23.9 25.9Resources 2.3 4.2 6.8Personal Development 10.6 9.9 15.0


1. Q15 The course is well organised and is running smoothly.

2. Q4 The course is intellectually stimulating.3. Q1 Staff are good at explaining things.4. Q14. Any changes to the course or teaching

have been communicated effectively.5. Q10 I have received sufficient advice and

support with my studies.

Top five predictors (best first)


Q22 ‘under-performers’Actual Predicted Residual SE1 SE2 SE3 Subjects

Q22 ‘as expected from Q1-Q21’

Q22 ‘over-performers’

University Groupings

Group n Mech Eng

ComSci

Allied Med

Elec Eng

Biol M&S EGS Hum Geog

Chem All

Million+ 85 72.6 71.2 74.2 71.9 75.9 92.0 84.3 91.1 75.7

Alliance 117 75.9 71.3 79.8 74.3 84.1 85.8 84.8 87.9 89.1 80.0

None 151 79.0 82.2 81.7 80.7 87.1 90.4 89.6 86.3 89.3 84.2

Russell 139 86.8 85.7 83.7 86.1 91.3 85.6 88.7 84.7 90.7 87.3

1994 84 83.4 88.0 89.8 90.0 89.7 90.3 88.6 91.5 92.7 88.9

All 576 78.7 79.7 80.6 80.7 86.4 87.6 87.6 87.7 90.5 84.5

n = 65 115 56 57 63 48 69 61 42

Mean overall Q22 for university groups

Conclusions

• Subject differences (e.g. mathematical content)• Institutional differences• False assumptions (e.g. enhancing feedback

directly enhances Q22)• Institutional effects• Satisfaction is a complex measure related to L&T

practices• Teaching and Organisation prominent


Ramsden quotes

Quotes from Ramsden (2007)

Higher Education Academy Annual Conference 2010

“… [The NSS] is not a measure of satisfaction so much as a window into how our designs for learning are experienced by students.

From these insights we assemble thepractical measures we may take toenhance the quality of theirexperiences.”



“... it is not simple to know what to do. Current experiences, unlike satisfaction, are a mixture of previous experiences and the environment as it is now...

... so sometimes we will need to adjust expectations or consider altering previous experiences in order to improve quality.”



“I cannot agree with the idea, for example, that because students are slightly less positive about feedback on assessed work in the NSS than about the quality of teaching...

... we should rush to bully academics into providing more feedback more quickly.”



“From this it also follows that students do not have a ‘right’ to be satisfied. They are themselves part of the experience. ..

... Students decide their own destinies and we can only add or subtract value at the margins.”

gees nss workshop 2010 insights into interpreting the nss mark langan, alan fielding & peter...

Documents

survey gees nss workshop

approaches gees nss

years gees nss workshop

nss questions teaching

nss mark langan

satisfaction satisfaction

subject correlation

teaching assessment