creating a coherent performance indicator framework for the higher education student lifecycle in...
TRANSCRIPT
Creating a coherent performance indicator framework for the higher education student
lifecycle in Australia
Sonia Whiteley
The Social Research Centre & The Australian Centre for Applied Social Research Methods
EAIR 37th Annual Forum in Krems, Austrla30 August to 2 September 2015
About the Social Research Centre
• We are a private, for-profit company owned by ANU Enterprise (Australian National University)
• Our services include: survey design and execution, qualitative research, survey data management, statistical consulting and analytical reporting.
EAIR 2015 2
About the Australian Centre for Applied Social Research Methods
• The Australian Centre for Applied Social Research Methods (AusCen) provides national leadership in social research methods and training by: Building a world-class team of researchers and graduate students
in social research methodology, applications and techniques Developing and validating new and cost-effective data collection
methods Increasing the availability and access to secondary data for
research across Australia, and Producing a more sophisticated Australian skills base via training
and educational activities.
EAIR 2015 3
Overview
1. Higher education performance indicators in Australia
(a very short history)
2. The Quality Indicators for Learning and Teaching
Survey Program
3. Proposed indicator framework & measures
EAIR 2015 4
Quality in higher education
• Until the 1980’s debates about the quality of higher education were essentially invisible with QA processes managed by each institution
• The first wave of ‘questions about quality’ resulted from sectoral diversification and public funding
• The second wave is occurring in the context of massification, regulation and an orientation towards market assessments of quality
• The focus has shifted from internal peer review to external performance indicators
EAIR 2015 5
Higher education performance indicators‘
EAIR 2015 6
Linke 1991Perceived Teaching Quality
Graduate Employment Status
Chalmers 2007
Graduate Satisfaction
Student Involvement and Engagement
Employer Satisfaction
Graduate Full-time Employment
Graduate Participation in Further Studies
Graduate Starting Salaries
DEEWR 2009Student Experience
Quality of Learning Outcomes
Higher education performance indicators (2)
Unclear why there have been difficulties creating a sustainable set of learning and teaching performance indicators
Lack of consensus on the nature of the indicators?
Retro-fitting of existing data against measures?
Resistance to increased reporting burden?
EAIR 2015 7
Quality Indicators for Learning and Teaching
• QILT is the outcome of 20 years of work on higher education performance indicators
• The most recent review suggested the indicators should be: Fit for purpose Consistent Auditable Transparent Timely
…to provide a robust and reliable measure of teaching performance throughout the Student Life Cycle.
EAIR 2015 8
The Student Life Cycle
EAIR 2015 9
Our role in QILT
• The Social Research Centre was commissioned by the Department of Education and Training as the independent administrator of QILT.
This involves: Collecting data Reporting on survey outcomes Creating, monitoring and updating the QILT website.
EAIR 2015 10
QILT Website
EAIR 2015 11
QILT Website
EAIR 2015 12
QILT Website
EAIR 2015 13
What are the QILT surveys?
• The QILT program consists of:
Student Experience Survey (SES) - measuring the engagement of
current students with the higher education system
Graduate Outcomes Survey (GOS) – examining graduates’ labour
market outcomes, and
Employer Satisfaction Survey (ESS) – assessing the employer’s
opinion of the graduates’ generic skills and work readiness.
EAIR 2015 14
What are the QILT surveys? (2)
• The QILT survey suite focuses on:
Commencing and completing undergraduate university students –
Student Experience Survey (SES)
University graduates – Graduate Outcomes Survey (GOS)
Employers of recent university graduates – Employers Satisfaction
Survey (ESS)
• All are cross sectional, point-in-time surveys except the GOS, which is longitudinal.
EAIR 2015 15
QILT in the context of the student life cycle
EAIR 2015 16
QILT outcomes
For the first time in Australia we will have:• an integrated and coherent set of surveys providing
feedback at key junctures of the student experience and beyond
• a centralised and independent approach to survey administration
• a program of work that encompasses the entire higher education sector, rather than just focusing on the universities and
• a comprehensive and current website containing headline indicators from each of the surveys.
EAIR 2015 17
The QILT Survey Program
The Australian Government’s desire to have objective KPIs that demonstrate students’ engagement with and experience of courses, and the quality of those
courses from the students’ perspective.The three QILT surveys work together to provide a holistic and coherent picture across the entire
student life cycle.
EAIR 2015 18
QILT indicator framwork
There are three sets of indicators underlying the Quality in Learning and Teaching framework:
• Student engagement
• Graduate destinations, and
• Graduate readiness.
EAIR 2015 19
QILT indicator sets
EAIR 2015 20
Indicator set Assessment Stage of cycle Benchmarking
SES Student engagement Formative Process Yes
GOS Graduate destinations Summative Outcome Yes
GOS - L Graduate destinations Summative Secondary
outcome Yes
ESS Graduate readiness Summative Output No
Student Experience Survey - engagement
The Student Experience Questionnaire (SEQ) focuses on aspects of the university experience that are:
Measureable; Linked to learning and development outcomes; and Able to be influenced by universities.
And measures five conceptual domains: Teaching Quality Learner Engagement Student Support Learning Resources, and Skills Development.
EAIR 2015 21
Student engagement indicators
EAIR 2015 22
Measure
Headline indicator
Overall quality of educational experience
% of students satisfied with the overall quality of their educational experience
Sub-indicator
Teaching Quality % of students satisfied with the quality of the teaching they have experienced
Learner Engagement % of students satisfied with their engagement with learning at their institution
Student Support % of students satisfied with the support they received at their institution
Learning Resources % of students satisfied with the learning resources provided by their institution
Skills Development % of students satisfied with the skills development they experienced through university studies
Graduate Outcomes Survey - destinations
The higher education sector has collected data on graduate destinations since 1972 which has been used to:
Provide a detailed picture of the labour market Support continuous improvement through internal and
external benchmarking, and Contribute to performance measurement frameworks and
key performance indicators.The Graduate Outcomes Questionnaire will examine destinations and provide a more contextualised understanding of graduate outcomes.
EAIR 2015 23
Graduate Destinations ‘model’
EAIR 2015 24
Graduate destination indicators
EAIR 2015 25
Measure
Headline indicator
Employment % of graduates working one hour per week or more
Study % of graduates in full-time study
Sub-indicator
Time-related underemployment
% of graduates that are working less than 35 hours per week, want more work and are ready to start work immediately
Qualification-related underemployment
% of graduates involuntarily employed in an occupation with an AQF level below their attained AQF level
Contextual indicator
Fully employed % of graduates that are employed and do not want to work more hours
Qualification-related underemployment
% of graduates reporting a mismatch between their education and the education required for their current position
% of graduates reporting a mismatch between their skills and the skills required for their current position
Graduate readiness
• Graduate readiness from an employers perspective is one of the most creative components of QILT
• Graduates rate their readiness and then provide contact information for their supervisor to offer similar feedback
• Recruitment for the employer survey is ‘challenging’ and provides many opportunities for innovation
• The Employer Satisfaction Questionnaire (ESQ) is in the early stages of development and validation
EAIR 2015 26
Graduate readiness indicators
EAIR 2015 27
Measure
Headline indicator Employer satisfaction % of employers who agree that the graduate was
prepared for their jobSub-indicator
Foundation skills % of employers who agree that the graduate demonstrates foundation skills
Adaptive Skills % of employers who agree that the graduate demonstrates adaptive skills
Teamwork & Interpersonal Skills
% of employers who agree that the graduate demonstrates teamwork and interpersonal skills
Technical Skills & Domain Specific Knowledge
% of employers who agree that the graduate demonstrates technical skills and domain specific
knowledgeEmployability Skills % of employers who agree that the graduate
demonstrates employability and enterprise skills
Graduate readiness indicators (2)
Additional contextual indicators – readiness expectations % match between employers and graduates agreeing
that the graduate demonstrates foundation skills % match between employers and graduates agreeing
that the graduate demonstrates adaptive skills % match between employers and graduates agreeing
that the graduate demonstrates teamwork and interpersonal skills
EAIR 2015 28
Integration of indicator sets
EAIR 2015 29
Student engagement
Graduate readinessGraduate’s perspective
Graduate readinessEmployer’s perspective
Skills Development
Foundation Skills
Adaptive Skills
Teamwork & Interpersonal Skills
Foundation Skills
Adaptive Skills
Teamwork & Interpersonal Skills
QILT data collection periods
EAIR 2015 30
Domain 2015 2016 2017
Student Experience Survey August – September
August – September
August – September
Graduate Outcomes Survey April (trial) and October rounds
April andOctober rounds
April andOctober rounds
Employer Satisfaction SurveyApril (instrument trial) andOctober rounds
April andOctober rounds
April and October rounds
Key challenges
Key challenges include (but are not limited to):• effectively managing a data collection program that is
active for 10 months of the year, covering an in-scope population of up to 1 million potential respondents
• including new and diverse stakeholders in unfamiliar quality and data management processes,
• centralising disparate legacy approaches to collecting graduate feedback, and
• supporting institutions to effectively engage with and analyse the data.
EAIR 2015 31
To summarise…
QILT is an ambitious, national program of centralised surveys to support the measures underpinning the indicator
framework. The ability to link views from key stakeholders during and after contact with the higher education system
provides an unprecedented opportunity to understand student experiences and outcomes.
EAIR 2015 32
