Helping Undergraduate Students Across Disciplines and Cultures Actively Engage and Collaborate as Equal Members of a Community of Scholars: Peer Review Within an E-Learning Environment

Dr Sue TaylorSchool of Accountancy, QUT Business School, Brisbane, Australia

Presentation: Extracted From Submitted Paper as Detailed Below – Full Details of Literature References Used in This Presentation Can Be Located Within This Paper

HELPING UNDERGRADUATE STUDENTS ACROSS DISCIPLINES AND CULTURES ACTIVELY ENGAGE AND COLLABORATE AS EQUAL MEMBERS OF A COMMUNITY OF SCHOLARS: PEER REVIEW WITHIN AN E-LEARNING ENVIRONMENT

Introduction and Background

External and Internal Tensions in Higher Education:

• Increased calls for public accountability in the Higher Education sector at the global level;

• Current move by the OECD to rank universities globally based on the quality of their teaching and learning outcomes;

• Universities have therefore prioritised the need for increased transparency in assessment practices;

• However, the precise explicit systems and procedures articulated within this accountability agenda seriously neglect the quality of student learning outcomes and have failed to engage students;

• Current assessment practice has thus been evaluated as seriously deficient.

Introduction and Background (cont’d)• Institutions and their teaching staff therefore face

numerous challenges such as:

• financial restrictions;

• increasing student numbers;

• increasingly educationally and culturally diverse and time-poor student population; and

• the resulting fragmentation of academic programmes across flexible learning options.

Introduction and Background (cont’d)Is There A Resolution?:Innovative assessment design, including new

paradigms of student engagement and learning and pedagogically based technologies have the capacity to provide some measure of relief from these internal and external tensions by significantly enhancing the learning experience for students.

Within this innovative assessment design context, a key concept of interest is the creation of an opportunity for students to help each other in the form of group or collaborative learning.

Introduction and Background (cont’d)

These collaborative group processes encourage peers to act as positive role models within a non-intimidating, informal environment.

Thus, any discussion of the use of peer review within an e-learning environment is both important and timely.

Study Motivations and the Purpose of Results Dissemination

• The creation of peer-reviewed assessment tasks in both Case Study One - AYB200 – Financial Accounting (2010) and Case Study Two - AYB227 – International Accounting (2011) was based on the acknowledgment that assessment is the single biggest influence on how students approach their learning.

Study Motivations and the Purpose of Results Dissemination (cont’d)

 • While the key objective of both reformulated assessment

tasks was to benefit students by increasing their desire to learn, it was also expected that accounting educators would develop new skills as mediators and moderators in the process of assisting students to take ownership of their own learning.

• QUT was also expected to benefit through the development of assessment tasks which were constructively aligned with the desired learning outcomes within a context of minimal resource requirements.

Study Motivations and the Purpose of Results Dissemination (cont’d)

• The accounting profession was also considered to be a beneficiary with post-peer-reviewed assessment task students entering the profession with increased technical accounting skills and non-technical skills in communication, teamwork, problem-solving and self-management.

• Dissemination of the results of this project to the broader community of accounting educators was also anticipated with the creation then of an opportunity for increased dialogue related to new and, potentially, more relevant modes of student assessment.

What is Student Peer Review and What Are the Benefits? 1) Benefits of Peer Review – Social Constructivism

• The social constructivist model of assessment argues that both students and tutors should be actively engaged with every stage of the assessment process in order for them to truly understand the requirements of the process, and the overall criteria and standards being applied.

• Within the educational literature, peer-marking using model answers was found to be particularly effective in improving students’ work and in students’ positive perceptions of the value of the activity. The 1996 study by Orsmond, Merry & Reiling highlighted that students benefited from peer reviews by becoming more critical and working in more structured ways.

What is Student Peer Review? 2) Benefits of Peer Review – Social Constructivism (cont’d)

• In addition, the benefits of peer assessment have been highlighted as including the fact that students become more confident, independent and reflective learners, and they obtain a deepened understanding of the required learning

Benefits of Peer Review (cont’d) – 2) Peer Learning Networks

• A second benefit of peer review is that it has the potential to assist students from culturally and educationally diverse backgrounds in adjusting to university, with peers potentially acting as positive role models within a non-intimidating, informal environment.

• As highlighted in Ladyshewsky and Gardner “communications between peers are less threatening than those that involve supervisors or authorities. Hence, enhanced disclosure, discussion and deeper learning outcomes are possible”.

Benefits of Peer Review (cont’d) – 2) Peer Learning Networks

The advantages of these learning networks then include:

• additional assistance with challenges from peers; more perspectives on problems;

• access to expertise;

• more meaningful participation;

• the creation of an informal environment as opposed to the highly structured lectures and tutorials run by perceived authority figures.

Benefits of Peer Review (cont’d) – 3) Generation of an Iterative Cycle of Learning through Formative Feedback

• As highlighted by Pearce, Mulder & Baik (University of Melbourne), the level of the qualitative and quantitative feedback normally available to students involved in a major project is limited to a final summative grade from the time-poor, academic staff.

• This approach is ineffective as there is no further opportunity for students to improve on their assignment. This means there is little motivation for them to reflect on, or learn from this feedback.

 

Benefits of Peer Review (cont’d) – 3) Generation of an Iterative Cycle of Learning through Formative Feedback

A third key benefit of the peer review process is its simultaneous ability to reduce the marking loads of staff while creating opportunities for students to become involved in a continuous cycle of evaluating the work of their peers during its formative stages.

The benefits of this evaluation process then include that it: provides students with a valuable perspective on their own work; encourages them to revise it; promotes a sense of community and collaboration; and helps students to become equipped for lifelong, independent learning.

Limitations of Peer ReviewStudent Concerns:

• While the available literature highlights a wide range of benefits of peer review, there are a range of potential impediments to implementing student peer review including:

• with students rather than staff marking the work, issues of validity, reliability, bias and fairness will arise;

• students may dislike evaluating another student’s work;

Limitations of Peer Review (cont’d)Student Concerns (cont’d)

• students can resent being required to review and comment on other students’ work believing that staff are paid to complete these tasks;

• they may lack confidence in their own ability to evaluate their peers’ work and may similarly doubt the competence of other student reviewers; and

• and some students may, on a cost versus benefit analysis, feel that the time taken to provide a peer review is not compensated for by the comments received by the peer review they will receive in exchange.

Limitations of Peer Review (cont’d)

Staff Concerns: • Of concern to academic staff is that any teaching

innovation must be achieved within short time frames to ensure minimal impact on their key disciplined-based research and publication agendas.

• Thus, the key issue for staff in terms of the adoption of any peer review process related to the time needed to administer the peer review process in class on a manual basis.

Limitations of Peer Review (cont’d)• E-Learning Technology - A Remedy?

• As highlighted in the on-line, PRAZE support manual by Pearce, Mulder & Baik, their development of an e-Learning based, peer review process at the University of Melbourne (PRAZE), was motivated by the desire to provide students with feedback that promotes a genuinely reflective cycle of learning.

• “...Students benefit both by being the recipient of comments on their own work but also through critically reviewing the work of others and reflecting on its positive and negative aspects.”

Limitations of Peer Review (cont’d)

E-Learning Technology - A Remedy? • In addition to providing a continuous cycle of

formative feedback, an e-Learning, peer review process has the potential to minimise the major staff concerns related to the time costs associated with its administration

Limitations of Peer Review (cont’d)• E-Learning Technology - A Remedy?

• Student concerns related to uncertainty as to whether they have the skills and experience to mark the work of their peers are also potentially minimised given the highly structured, step-by-step marking guide that can be included within the online review process.

• By providing an efficient and easy-to-use online tool for time-poor students, the costs related to any peer review completed are also minimised.

Limitations of Peer Review (cont’d)• E-Learning Technology - A Remedy? • While e-learning technology is capable of resolving a number of

the issues of concern amongst students and staff related to the peer review process, where does the use of technology leave the social benefits of a sense of belonging to a university community, which are mostly acquired through face-to-face contact?

• Is it possible to create a virtual sense of belonging and is this online technology equally as effective?

• Case Study Two as detailed below highlights the key features of an independent trial of the PRAZE process and its ability to engage students in a sense of online community. While Case Study One outlines the costs and benefits of a staff-led peer review process.

Case Study One – 2010 - AYB200 Background - The Dilemma:

• Financial Accounting (AYB200) - first "serious" accounting subject offered in the Bachelor of Business Degree at QUT for students majoring in accounting (approx. 1400 students per year);

• 2008 assessment review requirement – maximum of three assessment tasks/semester;

• Minimum of 20% per task;

• No hurdles allowed;

AYB200 Case Study Background - The Dilemma (cont’d):

• The remaining AYB200 assessment tasks were a 20% graduate capability based, Letter of Advice and a MC mid-semester exam (25%) and a final exam (55%) which required detailed journal entry responses and financial statement construction etc.

• Focus in final exam on real world, practical cases

• No marks for weekly, tutorial work or attendance led to students regarding this process as an unimportant step

AYB200 Case Study – Three-Tiered Assessment – The Results:

• Very low attendance rates (25-30%/semester); • Very low completion levels of weekly tutorial

work even amongst attendees;

• Inability of tutors to actually work constructively through material in tutorials (e.g. in small student groups) given the absence of this preliminary work;

AYB200 Case Study Background - The Dilemma (cont’d):

Additional Notes:• AYB200 cohort from a wide range of different

cultural backgrounds with difference symbol systems such as language, logic and mathematical systems and different opportunities for social interaction with more knowledgeable others;

• Significant percentage of students with English as a second language;

• Initial BSB100 Accounting subject often exempted from degree requirements to attract international and TAFE students – “prior” subjects were often “theory-based”;

AYB200 Case Study Background - The Dilemma (cont’d):

Additional Notes:

• AYB200 therefore, first, “real” accounting subject undertaken – complex international issues to address;

• Tutors – combination of full-time and sessional staff and Honours students; and

• No staff with educational qualifications – no internal access to teaching and learning advisors at this time.

AYB200 Case Study – Three-Tiered Assessment – The Results:

• Very low engagement levels with available feedback mechanisms in terms of either accessing staff during consultations or the available tutorial solutions (e.g. only 15% of student cohort downloaded material); and

• Consistently high failure rates (22-24%/semester) – 2008 and 2009 – six semesters (inc. Summer offerings)- in a yearly cohort of 1400 students – this is loss of approx. 300 students per year – with all the distress that this entails for the students and their families and the staff.

Peer Review Methodology – Case Study One - AYB200 : Staff- Led, In-Class, Peer Assessment Task Introduced (cont’d)

• Following consultations with the Head of School and tutorial staff, an in class, peer marking task was introduced;

• Task Characteristics – Three-Week Rotation - “Grouped” On-line Material Released – Lecture Content (Week One Delivery) + Week One Tutorial Work (Week Two In-Class Discussions) + Peer-Marked Assessment Task on Lecture One Content – Marked at the Beginning of Week Three’s Tutorials

Methodology: Peer Review Case Two – AYB 227 (2011) – E-Learning - PRAZE – University of Melbourne

The AYB227 – International Accounting cohort of students:

• Characterised by a considerable diversity in terms of both cultural and discipline-based knowledge with students coming from international business, finance, management and accounting backgrounds;

• Rapidly increasing numbers from approximately 25 students to over 450 students per year as a result of high levels of interest in international business activities within our employer groups, the staff and our students;

• The most valuable resources within the subject, as is highlighted by the peer review literature, was this actual student diversity itself;

• Thus, the key objective within this subject was to actively engage students in the assessment process that they could share their different cultural and discipline-based backgrounds.

 

Methodology: Peer Review Case Two – E-Learning - PRAZE – University of Melbourne (cont’d) - Case Study Two – AYB227 – International Accounting (cont’d)

• As the assessment task for the subject involved a major, cross-cultural written report, a formative based peer review process was introduced to provide students with a genuinely reflective, summative cycle of feedback from their peers prior to the final submission of the project for grading purposes.

• However, of concern, were both a range of time-related, administration issues raised by full time staff in Case Study One, and the still rapidly growing numbers within AYB227.

• In addition, the negative issues raised by students within Case Study One in terms of their preference that the peer-review process did not restrict normal, in-class, tutorial activities needed to be addressed. Thus, at my request, the School of Accountancy, QUT Business School, become a trial member of the University of Melbourne’s PRAZE e-Learning project.

• Students submitted their draft only answers to the project tasks – anonymously – i.e. using student numbers only – in Week Seven – Marks were awarded for this submission.

• Students were provided with a four-day, “staggered” submission process to allow for other commitments etc.

• Prior to the submission phase, an in-class practice session took place so that the students were able to gain confidence ion their ability to review each other’s work.

• Once the draft project answers were submitted these were allocated randomly by the computer to student peers within the class.

• The peer review process then took place over another four-day, “staggered” window - and was guided by a series of step-by-step questions which were necessary to answer to complete the final project for grading – additional marks were awarded for this review.

Methodology: Peer Review Case Two – E-Learning - PRAZE – University of Melbourne (cont’d) - Case Study Two – AYB227 – International Accounting (cont’d)

The Results – Case Study One AYB200 – Financial Accounting (2010)

• As highlighted in Figs. One and Two, from a student perspective, the outcomes of the assessment change were significantly successful with:

• a decline in overall failure rates of 15%; • an increase in weekly work completion, tutorial attendance

and engagement rates to approx. 100%; • and a 100% and 85% approval rating by the AYB200 cohort in

post-exam, in-class and on-line confidential surveys respectively.

• However, while sessional staff fully supported the peer review process (non-research focus), full time staff sought to have it removed from AYB200 given the time lost away from research activities - substantial administration time costs.

Fig. 1: Case Study One Student Results - Peer Review – Staff Administered – Semester One 2010 - AYB200 - Pre and Post Exam Cohorts - Positive Student Responses

Work S

ignif.

Inc.

Exam H

elp

Peer &

Self A

sses

s Erro

rs

Revise

Tool

Study H

abit

Exam Form

at Help

Less

Exa

m Load

Early E

rrors

Ident.

Three W

eek C

ycle

Preset

Forms

Acces

s to H

elp

Attend

Real C

ases

0

5

10

15

20

25

30

35

Case Study One - Staff Administered - Peer Review Task - ResultsPre-Exam LEX Students = 48 Students/129 Comments -

Post Exam On-Line Students =31 Students/93 Comments - Post Exam In-Class Students = 25 Students/131 Comments

Fig. 2: Case Study One Student Results - Peer Review – Staff Administered – Semester One 2010 - AYB200 – Final Exam Results –

Pre and Post Assessment Task

 

High Distinction Distinction Credit Pass Marginal Fail Fail Low Fail0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

6%

13%

33%

39%

5%3%

1%

Case Study One - Staff Administered - Pre and Post Peer Reviews - Results - Final Exam Grades

• 102 students of the 126 students in the off-semester cohort submitted their drafts;

• 99 students of these 102 students then completed their assigned reviews;

• At the time of submitting this paper and presentation, an interim survey had been completed with 42 student responses received as set out in Fig. 3.

The Results: Peer Review – E-Learning - PRAZE – University of Melbourne (cont’d) - Case Study Two – AYB227 – International Accounting (cont’d)

The Results (cont’d) - Case Study Two – AYB227 (2011) – e-Learning Administered

• Consistent high levels of student support as with Case Study One;

• However, in contrast to Case Study One, also very high level of staff support in this case with staff delighted at ease of use and low time costs associated with the PRAZE process and the excellent quality of the reports generated on student progress through the submission and review stages;

• As highlighted in Fig. 3, students also found great value in both receiving and providing peer-based feedback. In addition, the students were overwhelmingly in support of the e-Learning process which they found, almost without exception, to be very easy to use and “interesting”. Of great delight to many students was the high quality of the feedback received from their peer reviewer which provided them with useful ideas for their final draft.

Fig. 3 – Case Study Two – E-Learning Administered – AYB227 – 42 Peer Review Student Evaluations – Voluntary and Confidential – Anonymous PRAZE Submission and Review Process

Work M

otiva

tor

Easy S

ubmiss

.

Easy P

eer R

eview

(PR)

Asses

s. Tas

k Clar

ified

PR Com

ms. Valu

able

Marks A

warded

Were

Fair

Pr Proc

ess -

Rete

ntion

0

5

10

15

20

25

30

35

40

45

22

4038

14

21

34 33

Case Study Two - e-Learning Administered, Anonymous Peer Review - AYB227 - 42 Student Responses - Volun-tary and Confidential

Yes - Agree Yes - Strongly Agree No Comment Didn't Upload No

Summary and Conclusion The two alternate versions of peer review conducted within Case

Study One (staff, in-class led) and Case Study Two (e-Learning administered) provided an excellent opportunity for the independent evaluation of one of the key themes identified within the peer review literature review.

That is, did the adoption of the PRAZE technology serve to benefit the time saving interests of full time academic staff, only to fail to actively engage students and staff in a collaborative process as equal members of a community of scholars?

Alternatively, was the e-Learning process successful in genuinely creating a virtual sense of belonging even in the absence of “human tutors”?

 

Summary and Conclusion (cont’d)• In the staff administered, AYB200 Case Study, the

journeys of the AYB200 student cohort and the sessional staff were very similar with each group providing overwhelmingly positive responses to the peer-marked assessment task process.

• Regardless of this success, however, the project was placed at risk as a result of full time staff concerns related primarily to “time lost” away from promotion-based research commitments and expectations. The Head of School in this case received strong demands from full time staff to remove the peer review task from the subject.

Summary and Conclusion (cont’d) By contrast, in the AYB227 – International Accounting - Case

Study Two, the full time staff concerns related to “time lost” were significantly removed.

The PRAZE system delivered an extremely efficient, effective and easily mastered (by both staff and students) peer-review process which allowed the students to engage widely in a collaborative, community of scholars’ environment with minimal involvement of “human tutors”.

A preliminary, in-class practice marking workshop allowed the students to both benefit from the face-to-face contact with staff and their peers gained in Case Study One, and to gain confidence in their ability as peer reviewers.

Summary and Conclusion (cont’d)

 

Many students highlighted their preference for an anonymous evaluation process so that comments could be more “honest”.

The feedback mechanism within the PRAZE system also facilitated an open, post review dialogue between the “reviewed” and the “reviewers”.

Summary and Conclusion (cont’d)

In conclusion, the two case studies provide clear, independent evidence that innovative assessment design, does have the capacity to provide some measure of relief from the internal and external tensions currently faced by higher education institutions, their academic staff and students.

 

Summary and Conclusion (cont’d) 

 The clear “winners” in the e-Learning-based, peer review process are ALL the relevant stakeholders.

• Students - an increased desire to learn;

• Accounting Educators - new skills as mediators and moderators;

• QUT - constructively aligned assessment tasks – utilising minimal resources;

• Accounting Profession - post-peer-review students - increased technical accounting and technology skills and non-technical skills in communication, teamwork, problem-solving and self-management; and

• Broader Community of Accounting Educators- increased dialogue related to new and, potentially, more relevant modes of student assessment.

 

The Way Forward? Continuing the Peer-Assessment Journey - A Win-Win Balance