Proving to improve: how Accounting uses double blind peer review to assure

academic standards

UA – Summit of Deans Councils Canberra

28 February 2014

www.slideshare.net/markalistairfreeman

Achievement Matters: External Peer Review of Accounting Learning Standards Australian Business Deans Council Institute of Chartered Accountants in Australia CPA Australia Office for Learning and Teaching Project team : Adelaide (B.Howieson), Deakin (K.Watty), RMIT (B.O’Connell,

P.de Lange), Sydney (M.Freeman), UWA (P. Hancock), UWS (A.Abraham) Participants from 17 providers

Websites achievementmatters.com.au disciplinestandards.pbworks.com

Disclaimer

The views expressed in this presentation do not necessarily reflect the views of any of these stakeholders

article.wn.com

We expect Bachelor of X in your faculty is up for scrutiny by TEQSA in 2015. Ready?

Challenge

Standard: “a definite level of excellence or attainment..... so established by authority, custom, or consensus” (Sadler, 2012)

Learning outcome: ‘knows, understands and can do’

www.shutterstock.com

Definitions

“The idea that a single external examiner could make a comparative judgement on the national, and indeed international, standard of a programme has always been flawed” 6 experienced EE in 4 disciplines . Each given five 2i/2ii Findings

– Only 1 jointly highest (of 5) by all 6 EE in discipline – 9 of 20 ranked both best and worst (of 5)

2012

2013

“Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities” (Tenet 6: Price et al, 2008)

Manifesto

7

Scott Ray 4/4/09 http://www.flickr.com/photos/15075617@N05/3418528723

1.2 There are robust internal processes for design and approval of the course of study which take account of external standards and requirements, e.g. published discipline standards …, and comparable standards at other higher education providers.

Legislation

Martin Morris 20/2/12 http://www.flickr.com/photos/martin55/6929721749/

5.5 “The academic standards intended to be achieved by students and the standards actually achieved by students in the course of study are benchmarked against similar accredited courses of study.”

outcomes

External AoL

9

http://vimeo.com/channels/testingtogether/66041940

Every uni should have systemic policy for external moderation but not necessary same every discipline

TEQSA Chief

Accounting graduate

Knowledge

Application

Judgement Communication & Teamwork

Self management

Bachelor graduates are able to exercise judgement under supervision to solve routine accounting problems in straightforward contexts using social, ethical, economic, regulatory and global perspectives

Masters

minimal

emerging/advanced complex

^

2010

Assess Enter Compare Pre-F2F

F2F

Consensus Agree

Post-F2F

Apply

Confirm

2011-14 Calibrate

Calibration – Task validity Written communication

Individual confidence pre-workshop

Individual results pre-workshop

• Min & max (n=26)

• Mean ±1 SD

Group results at workshop

• Small groups (n=5)

• Consensus

91%

NA A

NA A

Calibration – UG student 1 Written communication

Individual confidence pre-workshop

Individual results pre-workshop

• Min & max (n=26)

• Mean ±1 SD

Group results at workshop

• Small groups (n=5)

• Consensus

NM M

NM M

78%

....S2, S3, S4, S5

Calibration workshops Date Location Learning standards calibrated Level

Jul 2011 Darwin • Written Communication Bachelor

Sep 2011 Melbourne • Written Communication Master

Feb 2012 Adelaide • Written Communication Bachelor Sep 2012 Sydney

• Knowledge • Written Communication

Master

Feb 2013 Adelaide

• Knowledge • Oral Communication

Master

Jul 2013 Perth

• Application • Judgement • Oral Communication

Bachelor

Feb 2014 Adelaide • Application • Judgement • Teamwork

Master

Jul 2014 Sydney • Teamwork • Self management

Bachelor

Martin Morris 20/2/12 http://www.flickr.com/photos/martin55/6929721749/

Live review

1

Live review report

1

Appendix

𝐵𝐵𝐵𝐵𝐵𝐵𝐵𝐵𝐶𝐶𝐵𝐵𝐵 ?

Your uni predicts your main Bachelor / Masters (coursework) degree will be selected by TEQSA as a sample degree for scrutiny in 2015. How would you evidence external AOL?

1. Lobby QVS to make that degree/major a focus for 2014 2. Select partner dept to verify sample assignments & exams

in a final year unit for 5% students in each grade band 3. Pay an experienced external academic to assess a sample 4. Lobby deans council to establish a national ‘double blind

peer review’ benchmarking process that includes calibration

5. Do nothing about LO - rely on other indicators like CEQ

Thank you

Achievement Matters project aims

1. External peer-reviewed data to prove & improve 2. External assurance of learning model developed 3. Professional development 4. Model disseminated

Provider selects reviewers

Project selects standards & degree for

review

Provider selects tasks

Project generates random student

ID to submit

Provider submits de-identified data

Reviewers calibrate

3 anonymous reviewers submit (Project allocates

2 externals)

Provider uses results

(Project ensures consensus)

External calibration

& review

Participant benefits • External reviews

– Prove (TEQSA, AACSB, CPA) – Improve

• Calibration – Professional development – Improve assessment & feedback – Sharing resources (eg. www.writersdiet.ac.nz/)

• Enhanced collaboration across sector & with profession • Process

– Cultural fit (research; AOL; not CEQ) – Defensible (valid, reliable, replicable) – Efficient

Participant costs

• Time – Preparing submission – Getting calibrated – Reviewing submission – Unpacking report – Disseminating locally

• Direct – Workshop travel (if not already going to AFAANZ) – Accommodation (extra night)

Challenges

Project • Identifying and archiving samples & evidence • Hygiene factors eg. de-identifying, file size, automating feedback • Calibrating national shared understandings of standards - BIGGEST • Ensuring feedback is constructive • Staying on track (ie. not distracted eg. AACSB obligations) • Tracking dissemination and impact • Sustainability

Wider context • TEQSA, AQF, HESP • New government

Conclusions

• Drivers – Regulatory framework – Accreditation

• Model – Promotes direct evidence for proving then

improving – Positive benefit cost ratio

• Sustainability