Setting and assessing standards

Mark Freeman

mark.freeman@sydney.edu.au mark.freeman@abdc.edu.au

Status, options and challenges

Outline

1. Regulatory context

2. Setting learning standards

3. Assessing learning standards

4. Evaluating moderation options

5. Q & A

Support for this project has been provided by the Australian Business Deans Council, the Institute of Chartered Accountants in Australia, CPA Australia and the Australian Government Office for Learning and Teaching. The views expressed in this presentation do not necessarily reflect the views of any of these stakeholders.

I have taught Snoopy to whistle

I can’t hear him whistle

I said that I had taught him, not that he had learned

Definitions

Standards

“a definite level of excellence or attainment, or a definite degree of any quality viewed as a prescribed object of endeavour or as the recognised measure of what is adequate for some purpose, so established by authority, custom, or consensus” (Sadler, 2009)

Learning outcomes

“the expression of the set of knowledge, skills and the application of the knowledge and skills a person has acquired and is able to demonstrate as a result of learning” (AQF, 2011)

Learning standards

“the explicit levels of attainment required of and achieved by students and graduates, individually and collectively, in defined areas of knowledge and skills” (DEEWR, 2011)

It’s coming!

39%

TEQSA

Provider

Qualifications

Teaching and Learning

Research

Information

‘Fitness-for-purpose’ ‘standards’ (ie. internal external) Higher Education Standards Panel consult Ministers Commissioners Teaching standards separate from learning standards

Threshold

“At this point [the learning and teaching standards] are not threshold standards [but] what the government may choose to do in the future remains to be seen.“

The Australian 24 Aug 2011

TEQSA legislation

...take account of external standards.. e.g. published discipline standards...

...standards intended ...and ..actually achieved ....are benchmarked

...awards ...meet the corresponding specifications ...described in the AQF

TEQSA regulatory risk framework

T & L Standards Discussion Paper - principle 3 & 5

“TEQSA is not the only custodian of standards, nor are higher education institutions. This responsibility is distributed and shared more widely, including with disciplinary communities and professional associations”

“Institutional standards for teaching and learning will differ but all institutions must meet or surpass national standards”

Setting learning standards“Discipline communities will ‘own’ and take responsibility for implementing teaching and learning standards (working with professional bodies and other stakeholders where appropriate) within the academic traditions of collegiality, peer review, pre-eminence of disciplines and, importantly, academic autonomy” DEEWR (2009, p. 32)

9 discipline groups in 4 waves – 11 sets defined

Jul’09 Arts, Social Sciences & Humanities

Business, Management & Economics

Engineering & ICT

Jul’10 Architecture, Design & Building

Science

Feb’10 Creative & Performing Arts

Health, Medicine & Vet Science

Law

Feb’11 Education

Business, Management and Economics

Accounting Banking, Finance & Related Fields

Business Information Studies Business Management

Economics and Econometrics Hospitality Management

Human Resource Management

Industrial Relations

International Business Organisation Management

Marketing Sport and Recreation

Tourism Management Missing (eg. Logistics)

Degrees• Bachelor • Master (Entry) • Master (Advanced)

Cycles1. Agenda setting2. Awareness raising3. Consultation4. Dissemination

Engagement• 2,100 participants nationally • 38 Australian universities• 21 private/other providers• 20 others (eg professional

and peak bodies)

+

+

+

Judgement

Knowledge

Application

Communication & Teamwork

Self Management

Judgement

Knowledge

Application

Communication & Teamwork

Self Management

Judgement

Knowledge

Application

Communication & Teamwork

Self Management

+

+

+

Quantitative

Public sector

Professional Small business

Regional

Judgement

Knowledge

Application

Communication & Teamwork

Self Management

Chalk + talk learning

Online learning Problem-based learning

Team-based learning

Provider CProvider B Provider DProvider A

Indiv Small group Indiv

Q1 & 2

Assessing learning standards

Assessing learning standards

1. Perceptions – employers, graduates, professional bodies

“Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities” (Tenet 6: Price et al, 2008)

“many graduates already subjected to skills testing for employment”

2. Common test – ACER, CLA, AHELO

3. External moderation – UK, Go8, Krause, ABDC-Prof Bodies

eg. AGS/CEQ; professional body accreditation

Moderation initiatives

QVS Krause-Scott et al Achievement Mat

Scope Multiple Multiple Accounting

Level Bachelor Bachelor Bach + Mast

HEI grouping Go8 11 across Start 10 across

Reviewers 1 academic 1 academic 2 aca/professionals

Data selection Stratified Stratified Randomised

Sample size 5% HD/D/C/P/F 1 HD/D/C/P/F per partner 5

Products All unit’s tasks All unit’s tasks Specific threshold

Intent Verification (QA) Qlty assurance & Qlty Enhancement (QE)

QA & QE

Authority Top-down Top-down Bottom-up

Achievement Matters Project

Aims

1. Evidence of accounting academic standards • External, double-blind, peer-reviewed

• Benchmark against national consensus (Bachelor & Master)

• All HEP types

2. A model process for obtaining and using evidence • Assessing inputs & outputs

• Quality enhancement & assurance

3. Professional learning and capacity building

Rationale: Improve, self-regulate, avoid perverse options

Pilot: Adelaide, Curtin, Deakin, Griffith, Monash, RMIT, Southern Cross, Sydney, USQ, UWA, UWS

Graduates of a Bachelor/Master (Entry) degree would be expected to justify and communicate accounting advice and ideas in straightforward/diverse collaborative contexts involving both accountants and non-accountants.

Pilot cycle: Threshold standard written communication

Master (entry): Diverse = Several competing or new qualitative perspectives and/or quantitative perspectives characterised by considerable data items, over multiple variables and knownrelationships between them.Bachelor: Straightforward = few qualitative perspectives and/or quantitative perspectives characterised by considerable data items over multiple variables and known relationships between them

23

Assess Enter Compare

Pre-F2F

F2F

Consensus Agree

Post-F2F

Implement

Reaching consensus on assessment task validity

24

Assess Enter Compare

Pre-F2F

F2F

Consensus Agree

Post-F2F

Apply • to assignment if student• to marking if faculty

Calibrating and grading to the standard

I’m confident rating assessment requirements and students’ work

Calibration – task validity

NA A

NA A

Individual results pre-workshop

• Min & max (n=26)

• Mean ±1 SD

Group results at workshop

• Small groups (n=5)

• Consensus

Calibration – UG student 1

Individual results pre-workshop

• Min & max (n=26)

• Mean ±1 SD

Group results at workshop

• Small groups (n=5)

• ConsensusNM M

NM M

Calibration – PG student 1

Individual results pre-workshop

• Min & max (n=26)

• Mean ± 1 SD

Group results at workshop

• Small groups (n=5)

• ConsensusNM M

NM M

Confirmation – PG student 5

Individual results at workshop

• Min & max (n=20)

• Mean ±1 SD

Group results at workshop

• Small groups (n=5)

• ConsensusNM M

NM M

Participant feedback

Having to enter my feedback into SPARK caused me to reflect on the reasons for my judgement

I expect this project will help establish national agreement on academic standards between accounting degree providers and with employers

Impact on academics

Strongly StronglyDisagree Agree

Pre-workshop 1

Pre-workshop 3

Post-workshop 3

33

Indiv Small group Indiv

Q3 & 4

ChallengesTEQSA

1. How will standards be set and monitored in a way that is sensible, fair, accepted and still economic?

HEI

2. How should we engage in disciplines setting standards?

3. How should agreed disciplinary learning standards be implemented into our curriculum?

4. How can we best participate in collaborative initiatives assessing achievement against national benchmarks?

5. How should evidence from our participation in national moderation projects be reported and used?

6. What systems changes and professional development are needed here to prepare for the standards agenda?

Thank you