Student Feedback on, and Evaluation of, their Learning

ExperienceMichael Prosser

Institute for Teaching and Learning

The University of Sydney

2

INTRODUCTION

Some distinctions in the way we talk about and describe students learning experiences in higher education

A. Consumer and client centred views of student learning experience

- ratings of student satisfaction with teaching and services

B. Learner centred views of student learning experience

- indicators of student perceptions and experiences relate to learning approaches and outcomes

In this presentation I wish to focus on the experience of students as learners – learner centred views

3

A. Design of learning experiences:

- focus on the way student experiences are designed by the teacher

B. Student experiences of the design of student learning experiences:

- focus on the way the student experiences the design

4

It is not (just) the way we design our courses and programs in higher education that relates to the quality of student learning but how our students experience and understand those designs.

Finding out, understanding and responding to the way students experience our courses and programs – the student experience of our courses and programs

Working towards aligning the way our student experience our courses and programs with the way we design those courses and programs

5

CHARACTERISTICS OF THE STUDENT (e.g. previous experiences, current understanding) STUDENTS'

PERCEPTIONS OF CONTEXT (e.g. good teaching, clear goals)

COURSE AND DEPARTMENTAL LEARNING CONTEXT (e.g. course design, teaching methods, assessment)

STUDENTS' APPROACHES TO LEARNING (how they learn e.g. surface/deep)

STUDENTS' LEARNING OUTCOMES (what they learn quantity/quality)

  OVERVIEW OF THE STUDENT LEARNING PERSPECTIVE

Figure 1: Model of Student Learning

6

Student Approaches to Learning

A key variation represented by:

Surface Approach:Intention to reproduce

• rote memorise information needed for assessment• failure to distinguish principles from examples• treat tasks as external impositions• focus on discrete elements without integration

 Deep Approach:Intention to understand

• meaningfully memorise information for later use• relate new ideas to previous knowledge• relate concepts to everyday experiences• relate evidence to conclusions 

7

Students’ Experiences of The Learning Context

Student experiences of:

• Quality of teaching – including quality of feedback

• Clearness of course goals and standards

• Workload - too high to understand

• Assessment - reproduction and not understanding

are key aspects of student experiences found to relate to how they approaches their studies and to their learning outcomes

Entwistle and Ramsden, 1983

8

EXAMPLE FROM MEDICINEQuantitative Survey

Study of over 8000 students in first year subjects around Australia, including 618 first year medical students in three different first year courses

Amongst the data collected were students responses to:

• Ramsden's Course Experience Questionnaire

• Biggs’ Study Process Questionnaire

1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

9

Approaches to StudySurface Approach 

32. Although I generally remember facts and details, I find it difficult to fit them together into an overall picture

 Deep Approach 

28. I try to relate ideas in this subject to those in other subjects, wherever possible Biggs’ Study Process Questionnaire

10

Student Experiences Of Learning Environment

Good Teaching15. The staff made a real effort to understand difficulties students might be having with their work.

Clear Goals and Standards1. It was always easy to know the standard of work expected6. I usually had a clear idea of where I was going and what was expected of me in this subject.

Appropriate Workload25. The sheer volume of work in this subject meant that it couldn't all be thoroughly comprehended (-).

Appropriate Assessment8. To do well in this subject, all you really need is a good memory (-).

Paul Ramsden’s Course Experience Questionnaire

11

Analyses showed that within each course: 

1.   A deep approach is associated with experiences that the teaching is good and the goals and standards are clear 2.   A surface approach is associated with experiences that the workload is too high and assessment tests reproduction

That is, variation in students’ experiences of the learning environment within subjects is associated with the approaches to study within subjects

– within subject variation in experience is not measurement error.

12

Factor Analysis of Experiences of Learning Context and Approach to Study

________________________________________________Scale Subject A Subject B Subject C

Factors Factors Factors________ ________ ________1 2 1 2 1 2

________________________________________________ExperiencesGood teaching .84 .77 .79Clear Goals & Stan .68 .70 .69App. Workload -.73 -.68 -.69App. Assessment -.70 -.60 -.60ApproachesDeep Approach .75 .84 .80Surface Approach .62 .64 .72________________________________________________

13

EXAMPLE FROM MATHEMATICSQualitative Survey

First year university mathematics students, large research intensive university, analyses of short open ended written statements by students identified:

Key differences in the experiences of Mathematics:

Fragmented Experience A. Maths as numbers, rules and formulaeB. Maths as numbers etc with applications to problems

Cohesive ExperienceC.Maths as a way of thinkingD.Maths as a way of thinking for complex problem solvingE. Maths provides insights for understanding the world

14

Experience of Maths

Quote

A: Numbers Maths is the study of numbers and the application of various methods of changing numbers

C: Logical system

Maths is the study of logic. Numbers and symbols used to study life in a systematic perspective and requires the mind to think in a logical and precise way

E: Complex / World

Techniques for thinking about observable, physical phenomena in a quantitative way and also for thinking more abstractly with little or no relation to the directly observable universe

15

Approach to Study of Maths

Quote

A: Learning by rote memorisation, with an intention to reproduce

(surface)

I liked calculus because I could remember formulas which is how I used to study. I would rote learn all the formulas and summarise all my theoretical notes

E: Learning with an intention of gaining a coherent understanding of the theory and looking for situations where the theory will apply

(deep)

Read the relevant theory and try to get on the same “wavelength” as the person who actually discovered it. Before I attempt any problems I try to think where you can use the concept: i.e. what the concept was invented for. Then I attempt problems (on my own)

16

Relationship between Experience of Mathematics and Approach to Studying Mathematics

______________________________________________Experience Approach Total

_______________Surface Deep(A & B) (C, D & E)

______________________________________________Fragmented (A & B) 179 (91%) 17 (9%) 196

Cohesive (C, D, E) 4 (10%) 36 (90%) 40  Total 183 53 236______________________________________________Chi-square=126, p<.001; Phi=.57; ES=2.1

Crawford, K., Gordon, S., Nicholas, J. and Prosser, M. (1998) Qualitatively different experiences of learning mathematics at university. Learning and Instruction, 8, 455-468.

17

Scale Item Item

Fragmented Experience

Cohesive Experience

9. Mathematics is figuring out problems involving numbers

17. Mathematics is a logical system which helps explain the things around us

Defining Items from the Experiences of Mathematics Questionnaire

18

Cluster Analysis of Experiences and Understandings_____________________________________________________________Variables Cluster 1 Cluster 2

(n=147) (n=127)_____________________________________________________________Prior Experiences and Understandings Fragmented experience of Maths 0.31 -0.46 Cohesive experience of Maths -0.39 0.45 Prior Surface Orientation 0.34 -0.44 Deep Orientation -0.56 0.59 Academic Ranking (TER) -0.29 0.35Experiences and Approaches Good Teaching -0.37 0.37 Clear Goals and Standards -0.23 0.24 Appropriate Workload -0.30 0.32 Appropriate Assessment -0.12 0.12 Surface Approach 0.44 -0.45 Deep Approach -0.54 0.57Post Experiences and Understanding Fragmented experience of Maths 0.34 -0.41 Cohesive experience of Maths -0.27 0.27 Achievement (Final Mark in Maths) -0.34 0.40

___________________________________________________

Cluster Analysis of Variables (medium to large differences)_____________________________________________________________Variables Cluster 1 Cluster 2

(n=147) (n=127)_____________________________________________________________Prior Experiences and Understandings Fragmented experience of Maths High Low Cohesive experience of Maths Low High Prior Surface Orientation High Low Deep Orientation Low HighExperiences and Approaches Good Teaching Low High Clear Goals and Standards Small difference Appropriate Workload Low High Appropriate Assessment Small difference Surface Approach High Low Deep Approach Low HighPost Experiences and Understanding Fragmented experience of Maths High Low Cohesive experience of Maths Low High Achievement (Final Mark in Maths) Low High

___________________________________________________

20

EXAMPLE FROM OXFORD UNIVERSITY(Honours: n=428, Approach: n=2320)

_________________________________________________Experience Outcome Approach

Honours _________________(1,2a,2b) Surface Deep(+ve)

________________________________________________Good teaching Large Low HighClear Goals and

Standards Large Low HighAppropriate

Assessment Medium Low HighAppropriate

Workload Medium LowHigh

_________________________________________________http://www.learning.ox.ac.uk/iaul/pdf/OLCPFinal.pdf

21

In summary:

Relationships between students experiences of their course, their approaches to study and their learning outcomes

View of student learning experiences:

• learner centred (learning) cf consumer / client centred (satisfaction)

• student experience of learning designs cf design of learning experiences

22

EXAMPLE FROM THE UNIVERSITY OF SYDNEY

Graduates:

Graduate Course Experience Questionnaire - Programpopulation survey of all coursework graduates

Postgraduate Research Experience Questionnaire - Programpopulation survey of all research graduates

23

Currently enrolled students:

Student Course Experience Questionnaire – whole yearstratified random sample of all enrolled coursework

students

Student Research Experience Questionnaire – whole yearpopulation survey of all enrolled research students

Unit of Study Evaluation - coursecompulsory triennial survey of all units of study

24

Supplemented and supported by:

Focus group discussions to further explore issues identified in surveys

Academic Board Reviews of Faculties interview groups of postgraduate and undergraduate students to better understand their experiences

Performance based funding model for teaching to reward Faculties enhancing the student experience

Faculty Teaching and Learning Plans with a focus on Student Experience

25

IMPLICATIONS FOR EVALUATING STUDENT LEARNING EXPERIENCES

1. Try to understand how and why student experience their courses and programs they way they do – quantitative survey, open-ended responses and follow up interview and focus groups

2. In responding , try to respond to why, not how, they experience their courses and programs the way they do – eg workload, assessment

3. Implement institutional policies, procedures and strategies that recognise and reward teachers, schools and departments and faculties efforts to respond to and enhance students learning experiences