1

Strategies for large-scale assessment: an institutional analysis of research and practice in a virtual university

Gavin Sim, Naveed Malik, Phil Holifield

2

Introduction

Background of VU Infrastructure Strategy

– Admissions– Assignments– Examinations

Future Strategy Conclusions

3

Background Virtual University Government Funded Project Courses delivered via satellite television and a

Learning Management System (Intralearn) Utilise lead academics to develop courses Students attend Learning Centres across

Pakistan to view programmes and use the Internet

Evidence that the needs of learners in remote areas can be served via local learning centres (Mayes, 2001)

4

Pak Institute of Management & Computer Science – Lecture

5

Pak Institute of Management & Computer Science - Lab

6

Background - Cont.

Current Cohort 1700 capabilities of 30750 Universities in Pakistan full year course

followed by exams VU this strategy not feasible, students need

to be able monitor progress and obtain feedback

Marking time would be 212 days for typical exam scripts

New Strategy required utilising the technology

7

Infrastructure Guidelines highlight need to identify technical

issues (Stephens, Bull et al. 1998) VU identified, distributed cohort - connectivity,

scalability, security & authenticating users Academic Intranet Courses and assessment delivered via the LMS Intralearn requires a ColdFusion backend and a

DBMS which Microsoft SQL Server is used Intralearn and SQL-Server are hosted on

separate servers creating a 3-Tier approach

8

Infrastructure - Cont.

9

Infrastructure - Cont. Scalability an Issue Max bandwidth between Learning Centres and VU

Hub 2Mbps can increase to 155Mbps The LMS needs an adaptable architecture Custom built software has been developed for

submitting the assignments (http upload). Asynchronous assessment is required due to the

variability in connectivity between centres Students can use any software - this is then

encrypted, renamed and forwarded to the tutor for grading

10

Infrastructure – Cont.

11

Infrastructure - Cont.

Delivery of exams - security issues Security flaws in sending emails such as

authentication and encryption (Hatton, Boyle et al. 2002)

Due to student numbers an asynchronous approach required

Software developed and deployed at centres Exam sent to centres encrypted and placed on

local server Same file returned encrypted after the exam

12

Strategy - Admissions Admissions scheduled for spring and autumn Test administered to 5000 students comprising

MCQ and essay questions. Marking outsourced to National Testing Centre Found no advantage in using this approach due

to bias for those exposed to testing before Test anxiety could adversely effect their grades

(Cassady & Johnson 2002) Admissions based on public examination results

13

Strategy – Assignments

Account for 15% of marks and students usually have between 5-10 per semester.

Regular testing is vital (Croft, Danson et al. 2001)

A series of bad marks can lead to a downward spiral in learning and confidence (Taras, 2002)

The assignments are structured to build student confidence

CS101 first assignment send an email and use the discussion board

14

Strategy – Assignments Cont.

Submission of assignments deadline plus one day

Academics are responsible for designing the assignments

Essay style formats usually used rather than objective testing

Tutors responsible for marking 250-300 assignments per week

Number of tutors is proportional to the number of students enabling scalability

15

Strategy – Assignments Cont.

Solutions are posted on LMS no feedback is provided

Self-reflection against a model answer enables students to gain personal insight of their understanding (Peat, Franklin et al. 2001)

Can ask questions concerning assignments on moderated discussion boards

16

Strategy - Exam

Two exams per module mid-term 35% and final exam 50%

Examiners are responsible for authenticating student ID at learning centres

Exams consist of both MCQ and Essay style questions

MCQ issues surrounding guessing (Harper, 2003)

Essays subjectivity with different markers

17

Strategy – Exam Cont.

Having varied question styles help produce fairer test

MCQ marked automatically whilst essays are marked by tutors

The students type their essays improving legibility for the tutors

This method enables the full range of cognitive skills to be tested as defined by Bloom (1956)

Debate whether objective testing can test higher cognitive skills in all subject domains

18

Strategy - Exams Cont.

In the majority of instances Synthesis and Evaluation promote divergent thinking and answers cannot be determined in advance (Bloom 1971)

Higher Cognitive Skills tested in Linguistics (Reid 2002)

Results are posted on the LMS Any queries are directed to staff within the VU

19

Future Strategy

Group projects planned for final year Students will work with others outside their

geographical area Tutors responsible for grading projects Explicit guidelines will need to be drawn up Ways to overcome gaps in face-to-face

presentations and lab demonstrations are sought

20

Conclusions

Infrastructure and strategy need to be developed in parallel

A strategy of combining CAA and tutor marked assessments is feasible

The scalability of the strategy and infrastructure should enable successful future expansion

Future analysis and research in the field of CAA may produce solutions that overcome some of the drawbacks

21

Questions