A lens on descriptive and learning analytics at UNISA

Mr Glen Barnes & Mr Dion van Zyl

Department of Institutional Statistics & Analysis Unisa

SAHELA Conference - June 2013

Objectives

• A recent report for the VP: Academic teaching &

Learning

– Progress on all initiatives relating to teaching & learning

– Placement within the context of the approved conceptual

model

• Selected results pertaining to the areas of:

– Descriptive analytics

– Predictive analytics

– Learning analytics Numerous contributors acknowledged …

Learning

Analytics

A conceptual-hypothetical model

• For enhancing student success

– Student walk at the core

– Both student and institution as situated agents

– Identifies shaping attributes and conditions for ‘fit’ towards

success

– Implementation framework

• Conceptual modelling

• Metrics & measures

• Data, information & analytics

• Interventions

• Monitoring & evaluation

FRAMEWORK FOR THE

MANAGEMENT OF STUDENT EXPERIENCE, SUCCESS, THROUGHPUT &

GRADUATENESS

Conceptual

Modeling

Identifying

what is

relevant,

measurable,

available &

actionable

Statistical &

Analytic

Modeling

producing

Actionable

Intelligence

Data

Gathering:

“Broad”

Tracking/

Intelligence

System

Proactive

Learner

Support

Interventions

Monitoring &

Evaluation

Systematic gathering and disseminating of information

• Identifying modules ‘at risk’

• Further cohort retention and throughput studies – Re-defining throughput timelines

• Exam success – Sitting view

– Module/Course success view

– Degree credit view

• Graduate success – Academic graduation view

– HEMIS graduation view

– Ceremony graduation view

Identifying modules ‘at risk’

• To refine the old definition of ‘high risk’ and apply a nuanced metric for different

requirements

• Objectives include:

– The prioritising of funding for additional interventions (face-to-face tutoring)

– Revision of e-Tutor provisioning (student : tutor ratios)

– College specific interventions

• Estimation of a ‘Risk Appetite’

– Curriculum based (compulsory, pre-requisite modules for qualifications)

– Student based (curriculum and electives)

Driver Description Metric

ENR Enrolment ranking on tota l enrolments Enrol led

REG Registration ranking on regis tered modules Registered

ATT Attri tion ranking Cancel led/Enrol led

PTN Participation in the exam Wrote/Registered

NPR Exam Pass Rate Passed/Wrote

CSR Module or Course Success Rate Passed/Registered

GSR Gross Success Rate Passed/Enrol led

Tota l ENR

ENR Rank

(No)

1 CSET EUP15 0 1 (0 ) ETHICAL INFORM ATION AND COM M UNICATION TECHNOLOGIES FOR … 5 26 631 0

2 CEMS ECS15 0 1 (1) ECONOM ICS IA 5 19 502 22,93

3 CEMS FAC15 0 2 (1) FINANCIAL ACCOUNTING PRINCIPLES, CONCEPTS AND PROCEDURES 5 18 952 25,1

4 CEMS ECS15 0 1 (2 ) ECONOM ICS IA 5 17 478 30,93

5 CEMS FAC15 0 2 (2 ) FINANCIAL ACCOUNTING PRINCIPLES, CONCEPTS AND PROCEDURES 5 17 132 32,29

6 CEMS ECS16 0 1 (1) ECONOM ICS IB 6 12 934 48,88

7 CLAW CLA15 0 1 (1) COM M ERCIAL LAW IA 5 12 664 49,95

8 CEMS MNB15 0 1 (1) BUSINESS M ANAGEM ENT IA 5 12 011 52,53

9 CLAW CLA15 0 1 (2 ) COM M ERCIAL LAW IA 5 11 053 56,32

10 CEMS INM10 13 (1) INTRODUCTION TO THE ECONOM IC AND M ANAGEM ENT ENVIRONM ENT 1A 5 10 785 57,38

NQFNo Colle ge Code (Se me ste r) Module De sc ription

One or more drivers … Module ranking index

Module

Risk Index

Module

Risk

Category

Student

‘Risk Appetite’

Qualification

‘Risk Appetite’

Risk index

Risk index

Risk index

Risk index

Compulsory

Modules

Risk index

Risk index

Risk index

Risk index

Elective

Modules

Towards a ‘Risk Appetite’

Curriculum Risk

Student Risk

Barriers to Graduation

Cohort retention & throughput studies

• Increased awareness and need for these

• Identified the need to move away from aggregated

analyses

– Regular updates

– Analyses per qualification

– Move to specialisations and majors within qualification

• Tracking system and analysts geared to update all

cohorts in a short period

– Move to ‘provisional’ more recent data

– Use of the ‘academic’ graduate view Less reliance on HEMIS

Constrained by IT capacities / hardware

Statistical analysis and predictive modelling

• Profiling cohorts

• Researching student access to ICTs

– ICT sophistication index

• Understanding the habits and behaviours of students

• Understanding dropouts

• Predicting module enrolments

• Predicting student registrations

• Predicting students ‘at risk’

Profiling Cohorts

• To develop and refine a survey questionnaire that can be used to profile students

according to academic and non-academic risk indicators defined in the conceptual

model

• To evaluate most appropriate method(s) of data collection (single vs. multi-phase

approach) that can facilitate the roll out to the broader student body

• To evaluate aspects of validity and reliability relating to the measuring of items and

constructs

• To conduct exploratory analyses to identify academic and non-academic factors

explaining student success

Educational Background: Results

50% 40% 60%

Exam Pass Rate

Did receive career guidance at school

EPR 53%/59%

Career guidance

received

54%/50%

Mothers education:

Post Matric

30%/25%

n-275

20

11

n-301

20

12

Fathers education:

Post Matric

33%/31%

First in family to

enrol in HE

40%/52%

Did not receive career guidance at school

EPR 43%/50%

Highest level of education: Mother Post

Matric

EPR 61%/50%

Highest level of education: Mother

Matric or below

EPR 44%/57%

Highest level of education: Father Post

Matric

EPR 59%/59%

Highest level of education: Father Matric or below

EPR 42%/43%

Not the first person in family to enrol in HE

EPR 53%/55%

First person in family to enrol in

HE

EPR 45%56%

2011/2012

Exam Pass Rate: 2011/2012

p=0,020*

p=0,020*

Life Circumstances: Results

n-275

20

11

n-278

20

12

Exam Pass Rate

Family

38%/43%

Single

63%/57%

50% 40% 60%

Life stage grouping: Family

EPR 36%/40%

Life stage grouping: Single

EPR 56%/50%

Life stage grouping

Number of

financial

dependents (1+)

45%/48%

None financial dependants

EPR 57% /56%

(1+) Financial dependants

EPR 38%/53%

Sufficient funds

for studies

51%/55%

Disagree: sufficient funds for

studies

EPR 46%/49%

Agree: sufficient funds for studies

EPR 51%/61%

2011/2012

Exam Pass Rate: 2011/2012

p=0,003*

p=0,004*

Researching student access to ICTs

• The key research questions were:

– what is the extent of ICT access among Unisa students?

– what are the technological capabilities of these students?

The 2011 research culminated in a proceeding at the ODL

Conference in September 2012 entitled ‘Student Access to and

Skills in Using Technology in an ODL Context’

Moving toward identifying an ‘ICT sophistication index’

Conceptual: ICT Sophistication How Unisa students adopt & engage

Ab

ilit

y t

o e

ng

ag

e w

ith

IC

T (

hard

ware

& s

oft

ware

)

Success

64%

Typically connect via own computer,

mobile phone and tablet devices (at

home)

Highest levels of ICT engagement for

academic purposes

Communicating with fellow students

and lecturers via ICT lower than

expected!

Highest LSM

Parents also studied

ICT adoption (access/ownership to ICTs in general)

Higher adoption

Higher ability

Higher adoption

Lower ability

Lower adoption

Lower ability

Lower adoption

Higher ability

Success

50%

Success

39%

Success

37%

High prevalence of connectivity via own

computer, mobile phone and tablet devices

(at home)

Mid to high levels of ICT engagement for

academic purposes

Communicating with fellow students and

lecturers via ICT 1 in 2

Mid to high LSM

Large proportion of parents also studied

Connectivity via own computer, mobile

phone and tablet devices lowest

Traditional - Pen & paper?

Lowest levels of ICT engagement for

academic purposes

Communicating with fellow students and

lecturers via ICT higher than expected

Lower LSM

Low proportion of parents also studied

Connectivity via own computer, mobile

phone and tablet devices (at home) low

More work and Internet cafe

High levels of ICT engagement for

academic purposes

Communicating with fellow students

and lecturers via ICT highest 6 in 10)

Lower LSM

Low proportion of parents also studied

Conceptual: ICT Sophistication Interventions

Ab

ilit

y t

o e

ng

ag

e w

ith

IC

T (

hard

ware

& s

oft

ware

)

Success

64%

ICT adoption (access/ownership to ICTs in general)

Higher adoption

Higher ability

Higher adoption

Lower ability

Lower adoption

Lower ability

Lower adoption

Higher ability

Success

50%

Success

39%

Success

37%

Predicting Students ‘At Risk’

• e-Tutor pilot project module ACN203S in 2012

• Attempting to deal with both academic and non-academic

factors

• Retain ‘early alert’ factors and as few as possible

• Move from CHAID analyses to logistic regression

• Model built on 2011 cohort

– Results for 2011 and 2012

• Aim to have sequenced parameters

– Follow the student walk

Students ‘At Risk’ - Results Structural Demographics

Gender

Race

Home language

English (1,14)

Other (0,81)

Nationality

Age

Correspondence language

College/School CEMS - Accounting Sciences (0,3)

CSET – Sciences (0,3)

CEDU - Educational Studies (2,3)

CEDU - Teacher Education (2,1)

Academic period (semester, year)

CESM

Entering status Transfer (T) (1,2)

Entering (E) (1,0)

Non-entering/Returning (N) (1,4)

Exam sitting May/June (15,6)

Qualification type – UG (0,96)

Tutorial attended – Yes (1,2)

Module repeat – No (7,5)

Accuracy

2011 = 73,0%

2012 = 72,1%

2012 success using the 2011 model

Variables to be considered…

Structural Demographics

ICT sophistication

Living Standard Measure (LSM)

Employment status

Life stage grouping

Parents in HE

Credit load

Matric certificate

English matric mark

Entrance testing

Assignments

‘Risk Appetite’

Registration Tuition & learning Exam

Learning analytics??

Student Support and Interventions

• Academic Literacies

– Reading and writing, numeracy - are provided at the various

regional offices, face-to-face 2013, online 2014

• The Integrated Tutor model (e-Tutors)

– Pilot in 2012, all NQF 5 in 2013

• Measurement of under-preparedness

– More use of NBT and similar testing, test all 2014 students in

two colleges

• Interactive study materials

– Podcasts, vodcasts & simulations

Links to profiling, ICT sophistication and early alerts

Ongoing Monitoring & Evaluation

• Enrolment management

– Setting enrolment targets per qualification

• Setting module success targets

• Setting cohort success targets

• The impact of the re-admissions policy

• Monitoring concessions to improve throughput

• Monitoring student satisfaction

• Undertaking module evaluations

Underpinning Requirements

• Re-think of the metrics to be measured

• Re-assess those being used

• Highlight the need for improved ‘systems’

– Operations

– ICT

– Interventions/support

• Results to inform

– Procurement and Development of IT

Structures Processes Systems

Conclusions

• A lens …

– Change to reflect on the various initiatives

– Opportunity to share some of the results

– Systematically move ahead under the ‘cover’ of the

conceptual model

– Indication of our involvement in areas of analytics and the

move to ‘learning’ analytics

Acknowledgements:

Mr G Barnes, Director: Information and Analysis, DISA

Mr D van Zyl, Manager: Information Services, DISA

Ms Y Chetty, Director: Institutional Research, DISA

Ms H Liebenberg, Senior Specialist: Institutional Research, DISA

Prof EO Mashile, Executive Director: Tuition and Facilitation of Learning

Prof G Subotzky, Past Executive Director: Institutional Statistics Analysis

Dr P Prinsloo, Directorate for Curriculum and Learning Development (DCLD)

Registration – Entrance testing/evaluation

Bridging / Alternative pathways (Attitudes, habits, ability, resource use)

ICT

sophistication

Interventions

Training & interventions

Structures Processes Systems

Observe the frequency patterns …

Determine appropriate categories …

Risk Index

Risk

Category