Thinking the unthinkable:

USING PROJECT RISK MANAGEMENT WHEN INTRODUCING COMPUTER-

ASSISTED ASSESSMENTS

Ian Harwood & Bill Warburton

University of Southampton, UK.

Thinking the unthinkable…

• Introduction

• Overview of PRM and risk efficiency

• Methodology / Case background

• Identified risks

• Conclusions

Introduction

• Implementing CAA is a risky activity

• Limited coverage in CAA literature

• Aims of this paper:– Build links between CAA and PRM literature– Show how formal PRM can be beneficial

Viewing CAA introduction as a project• Project has been defined as:

'a temporary endeavour undertaken to create a unique product or service' PMI (2000)

• A project is:‘a step into the unknown, fraught with risk

and uncertainty’ Lock (2000)

Increasing interest in Risk

• Now in the 'risk society' (Beck, 1992).

• Broad base of risk literature covering:– environmental issues (GM foods, nuclear waste)

– data processing

– security management (terrorism, insurance)

– project management

– health and safety (design criteria, disaster planning)

– finance and economics (derivatives, risk/return)

What is ‘Risk’?

• ‘a choice rather than a fate’ Bernstein (1996)

• risk or uncertainty?– risk is present when future events occur

with measurable probability– uncertainty is present when the likelihood

of future events is indefinite or incalculable Knight (1921) ‘Risk, Uncertainty and Profit’

Thinking the unthinkable…

• Introduction

• Overview of PRM and risk efficiency

• Methodology / Case background

• Identified risks

• Conclusions

PRM interest groups

• Association for Project Management: Risk Management Specific Interest Group (http://www.eurolog.co.uk/apmrisksig/)

• Centre for Risk Research, School of Management, University of Southampton, UK (http://www.management.soton.ac.uk/risk/default.asp)

• The Project Management Institute: Risk Management Specific Interest Group (http://www.risksig.com/).

The Project Risk Analysis and Management process (Simon et al 1997)

Identify risks

Assess risks

Plan riskresponses

Write reports

Focus PRAMDefine project

Risk Register&

Risk Model

DocumentedRisk Management

Procedure

… part of broaderproject plans

RiskAssessment

Reports

Reduce risks bychange to project

Achievementof projectobjectives

Risk efficiency

• From theory of portfolio management in Economics (Markowitz, 1959)

• ‘Risk’ and ‘Return’ inextricably linked

• Now used in the project environment– CCTA ‘Project Risk Management’

document

Theoretical risk efficiency framework

Risk

Ret

urn

A

Brisk efficient frontier

Key:

B

A C1…6: :

:

Minimum risk

Maximum return

Cases 1 to 6

feasibleCAA

applications

x : Desired risk efficiency

x RI2

RI2 : Case 2 risk inefficiency(as an example)

C1

C4C3

C2

C5

C6

Risk efficiency migration due to PRM

Risk

Ret

urn

A

Brisk efficient frontier

Key:

B

A

C1…6

:

::

Minimum risk

Maximum return Cases 1 to 6

feasibleCAA

applications

x : Desired risk efficiency

x

C1

C4

C3

C2

C5

C6

Thinking the unthinkable…

• Introduction

• Overview of PRM and risk efficiency

• Methodology / Case background

• Identified risks

• Conclusions

Overview of cases

• Background to Soton MLE strategy

• Roll out of PerceptionCase Date

(m/y) Discipline Formative /

summative usage

Number / level of students

Contribution to overall

assessment 1 05/03 Science Summative 80 2nd year UG,

16 3rd year UG 20%

2 10/03 Humanities Diagnostic 60 1st year UG N/A 3 11/03 Healthcare Form. & Sum. c.300 PGR

100 PGR Completion is a

course requirement 4 11/03 Soc. Sci. Form. & Sum. 280 1st year UG 30% 5 12/03 Science Summative 30 2nd year UG 1% 6 01/04 Science Summative 60 1st year UG 20%

Methodology

• Case study approach (Yin, 2003)

• Semi-structured interviews

• Cross-case analysis (Miles and Huberman, 1994)

• (retrospective) risk register generated

Thinking the unthinkable…

• Introduction

• Overview of PRM and risk efficiency

• Methodology / Case background

• Identified risks

• Conclusions

Identified risks- before & during delivery Table 2•Wrong assessment published (C3,5)

•Authors ‘reinvent the wheel’ (C5)

•Publication date missed (C1)

•Communication gaps (C1,3)

•Environmental issues (C4)

•Cheating (C1,2,3,5,6)

•Failure at point of delivery- scalability issues (C3,4)

•Tests submitted prematurely (C2)

Identified risks- during & after delivery Table 2•Informal appeals- protest about fairness of CAA (C1,5)

•Formal appeals:-workstation issues (C1,2,6) -unfamiliar test environment (C1)

-test construction (C1)-negative marking (C6)-QA issues (C5)

CAA abandoned:-changes in course architecture (C3) -staff turnover (C6)-weak implementation (C5)-perceived unfriendliness (C5)

‘Proactive’ responses: mitigating actions 1 Table 2•Pre-emptive Load-test at realistic levels

•Allow sufficient time to fix the problem

•Run a formative version of the test

•Keep channels of communication open

•Help staff to comply with internal CAA procedures

•Ease of use- capture & address author feedback

•Keep channels open between authors

•Comply with internal CAA procedures

•Use prefabricated templates

‘Proactive’ responses: mitigating actions 2 Table 2•Make training a prerequisite for access to the service.

•Help staff to comply with internal CAA procedures

•Use random selections from large question pools

•Allocate sufficient time to test workstations before exam

•Use a class register to book sufficient workstations

•Practice with formative versions

•Maintain tutorials & practice tests on public workstations

•Help tutors choose appropriate assessment methods

‘Reactive’ responses: Contingency Plans Table 2•Have a paper copy

•Substitute marks from formative assessments

•Capture and address author feedback

•Discount compromised items

•Re-issue test

•Discount compromised items

•Use trained invigilators

•Have a spare workstation area available

•Monitor and address student feedback

Thinking the unthinkable…

• Introduction

• Overview of PRM and risk efficiency

• Methodology / Case background

• Identified risks

• Conclusions

Benefits of formal PRM

• Enables better informed (and more believable) plans, schedules and budgets

• Increases the likelihood of a project going to plan

• Allows more meaningful assessment of contingencies

• Contributes to the build-up of statistical information to assist in better management of future projects

• Enables a more objective comparison of alternatives

• Identifies, and allocates responsibility to, the best risk owner

Benefits of formal PRM cont…• Improves general communication

• Leads to a common understanding and improved team spirit

• Assists in the distinction between good luck / good management and bad luck / bad management

• Helps develop the ability of staff to assess risks

• Focuses attention on the real and most important issues

• Facilitates greater risk-taking, thus increasing the benefits gained

• Demonstrates a responsible approach to customers

Risk Management Maturity (Hillson, 1997)

• Four levels:1) naïve2) novice3) normalised4) natural

• Assists in identifying areas for improvement in risk management

Decay of risk exposure through iterative PRM

Preparation risks

Aut

hori

ng r

isks

Delive

ry ri

sks

C1C4

C3

C2

C5

C6

C1

C4

C3

C2

C5C6

Realised risk exposurein this case study

Potential risk exposureusing project risk mgt.

Preparation risksA

utho

ring

ris

ks

Delive

ry ri

sks

Closing thoughts…

Six 'first time' applications of Perception replace paper-based exams- riskier than future repeat applications– iterative approaches to formalised project risk

management can build institutional corpi of CAA risk knowledge

– trend of total risk exposure progresses from risk inefficient to more risk efficient

– a full-scale system could be released that already catered for the greatest risks

– institution becomes more risk mature (Hillson, 1997)

Any questions…?