Great excavations

Unearthing institutional datasets to strengthen and inform university practice

Clare Milsom, Martyn Stewart & Elena ZaitsevaLiverpool John Moores University

National Student SurveyQualitative comments

National Student SurveySatisfaction scores

Module feedback

First destinationstatistics Progression data

Completion data

Benchmarking

Grade profiles

League tables

Enrolment data

Demographic data External examiner comments

Student feedback

Student Union Surveys

Attendance data

Problems with combining data sources

Organisational separation

Conceptual separation & validity

Poor representation

A role for institutional researchers to transform information into knowledge, applying analytical approaches to uncover the knowledge residing in data, and improving the processes through which this knowledge is transferred throughout the university

Terenzini 1993; Davenport & Prusak 1998; Teodorescu 2006 , Yorke 2009

‘The university is awash with data: but is it awash with institutional intelligence?’Longden & Yorke (2009, p66)

Unearthing meaning in data

CASE STUDY 1 : Library

Sources of data:

1. Mirror NSS: first and second - February 2. NSS 2011: third year students – Jan to April3. Student registration survey 2011: all students – end Sept mid Oct4. Student enrolment 2011: all students

CASE STUDY 1 : Library

NSS and ‘Mirror’ Surveys: two most semantically important themes

29 500 words 33 066 words 86 362 words

Year 1 (mirror) Year 2 (mirror) Year 3 (NSS)

Favourable Course Favourable Course Favourable CourseUnfavourable Lectures Unfavourable Time Unfavourable Library

Total 3533 respondents

1191 students left positive comments1318 students left negative comments

CASE STUDY 1 : Library

computersworkstudentscourseworktime feedbackLackfacilities

CASE STUDY 1 : Library

library is great and has fantastic atmosphere (first year student)

library resources are fantastic! (first year student)

Library's & library staff are great-good range of electronic resources & staff always willing to take time to help-even if you feel silly asking obvious question they sort of make up for the lack of enthusiasm & investment in resources elsewhere would give 10/10 defo. Big up library!!! (first year student)

The library and computer resources have always been reliable, even off campus and any problems have been dealt with quickly. Admin staff are always helpful and on hand. (second year student)

The Library resources, including staff are supportive to my learning needs. (second year student)

I'm grateful for the library and its resources, I make full use of them when I have an assignment to do. (second year student)

Lack of working computers at crucial times in the year, also the regular broken printers makes it a long and difficult process to work within the library. (second year student)

Lack of available/working computers between the hours of 11am-5pm in the library has occasionally disrupted work, especially during group projects where meet up time is restricted. (second year student)

Also, staff in the library are sometimes a bit rude and speak to you like you are in primary school, it’s quite patronising. (second year student)

Yes within the library silent zones there is still too much talking, this impacts on my abilty to concentrate (first year student)

just lack of space in the library to complete work and print work off nearer to deadline dates and around dinner times (first year student)

The Electronic Library can be confusing especially when searching through journals for a particular topic (first year student)

CASE STUDY 1 : Library

Favourable Neutral UnfavourableProvided, needs and

courseworkClosely linked to University Computers, available,

people, assignmentsLaw, English, Business,

FinanceHealth, Nursing, Pharmacy,

environmental sciences, Forensic

Education, ITT, Outdoor education, Dance, Tourism,

CASE STUDY 1 : LibraryCASE STUDY 1 : Library

AldamLibrary associated with

university

Aldham

•Library being open 24 h is appreciated

•PC access remains a problem - lot of them (PCs) are out of order; waiting time to get access to computer – minutes (15 mins being mentioned)

•Lack of up to date books; too much old material in the library

CASE STUDY 1 : Library

Avril

Students’ life is placement – centred:

Library and IT facilities not available over Christmas, [but] we had assignments to do. Think it is really hard having to do assignments and revise for exams while on placement!

Avril

Course associated with skills. University close to placement.

CASE STUDY 1 : Library

MarshCourse associated with

university

Marsh•The library isn't big enough – lack of general workspace and computers (40% of comments); waiting time to get access to computer – hour(s)

•Resources (mainly books) are limited or non-available - There are daily fights in the libraries for resources (30% of comments)

•Level of support/help should be geared towards needs of a particular student - we need some input from real people in the LRC who are switched on and not IT specialists whom expect you to grasp it first time; moody library staff

CASE STUDY 1 : Library

Student registration survey 2011: opens comments relating to support

Staff FavourableEmail UnfavourableModules Unfavourable

CASE STUDY 1 : Library

FavourableFriendlyFastEfficientHelpful

Enrolment open text comments

CASE STUDY 1 : Library

CASE STUDY 2: ‘Feedback on my work has been prompt’

National Student Survey (Q7):‘Feedback on my work has been prompt

• Low satisfaction score (sector-wide & institutionally).

• Many universities , including LJMU, introduce feedback turn-around policy

• Rise in satisfaction score of institution (5 percent points) & in 53% of programmes, but persists as area of most dissatisfaction

Predictors of ‘Overall satisfaction’ : Year 1 and 2 Student Experience Survey 2012

Relational frequency analysis of the content of free text comments from the 2011 National Student Survey, LJMU, generated by the semantic analysis software LeximancerTm

‘Promptness’ & ‘late’ tied up in wider conceptions of utility of feedback

Interventions wider than policy intervention: Better communication of function of feedback, Feedback as development: formative feedback, feed-forward, Make programme assessment ‘maps’ visible.

CASE STUDY 2: ‘Feedback on my work has been prompt’

‘But when I mean late feedback I mean it’s too late in the course..’

Well the work is handed in – so what’s the point of the feedback? ‘

‘Can’t change the work’. It’s too late to do anything with it’

Data are representations of reality. Some better than others.

- Combined data – unveiling ‘real-world’ complexities: interactions, perceptions, priorities

- Looking beyond simplistic procedural interventions; cultural and behavioural change

‘a fraught juggling act between ‘economies of performance’ (results; league tables; KPIs)

and

‘ecologies of practice’ (situated ethos and ideals;

cultures & practices)

Stronach et al (2002), plus Harris (2005), Clegg (2008), Cheng (2010), Gale (2011), Mathieson (2011)

Supporting staff in use of data for enhancement: Studies on development of professional identity in a performance culture….

Cheng, M. 2010. Audit cultures and quality assurance mechanisms in England: a study of their perceived impact on the work of academics, Teaching in Higher Education, 15, no.3: 259-271.

Clegg, S. 2008. Academic identities under threat. British Educational Research Journal, 34, no.3, 329–345.

Davenport, T.H. & Prusak, L. (1998). Working knowledge: how organizations manage what they know. Boston, MA: Harvard Business School Press

Gale, H. 2011. The reluctant academic: early-career academics in a teaching-oriented university, International Journal of Academic Development, 16, no. 3: 215-227.

Harris, S. (2005). Rethinking academic identities in neo-liberal times. Teaching in Higher Education, 10, no. 4: 421–433.

Longden, B. & Yorke, M. (2009) Institutional research: what problems are we trying to solve? Perspectives: Policy & Practice in Higher Education, 13 (3), pp 66-70.

Mathieson, M. 2011. Developing academic agency through critical reflection: a sociocultural approach to academic induction programmes, International Journal for Academic Development, 16, no.3: 243-256

Stronach, I. B. Corbin, O. McNamara, S. Stark and T. Warne. 2002. Towards an uncertain politics of professionalism: teacher and nurse identities in flux. Journal of Education Policy. 17, no. 1: 109-138.

Teodorescu, D. (2006) Institutional researchers as knowledge managers in universities: envisioning new roles for the IR profession, Tertiary Education & Management, 12, pp75-88.

Terenzini, P.T. (1993) On the nature of institutional research and the knowledge and skills it requires, Research in Higher Education, 34, 1-10.