The formative use of e-assessment

Some early implementations, and

Suggestions for how we might move on

Andrew Boyle

Initial stuff – rationale, definitions, etc.

Introduction

• Early predictions re e-assessment– e-assessment will lower barriers between formative and summative

assessment– Early years of e-assessment coincided with renewed interest in

formative assessment (FA)– Now e-assessment needs to move to ‘mature’ phase

• Literature review– Important role in FA research– Attempt to summarise where we have got to with formative e-

assessment (eFA)– Group implementations to establish coherence– Move towards critique

Definitions• FA

– All those activities undertaken by teachers, and/or by students, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged

– Contrasted with summative assessment Summarises learning Used for recording and reporting the amount of learning but not for feeding back

into learning– Important strand in both school and HE assessment research

Slightly different emphases

• e-assessment includes– Tests that are delivered on-screen– e-portfolios– Electronic discussion boards, forums and so on

Research aims

• Identify types of implementations used frequently by researchers and developers working in the field

• Suggest ways in which the body of research evidence might be expanded

• Underlying aim to delineate those areas where eFA provides a distinctive input

• Balance ‘e-assessment as transformative’ and more sceptical views

Method and scope

• Review is inclusive, rather than excluding– ‘Best evidence synthesis’– Seeks authentic, faithful and convincing results– Does not comply with one or more ‘objective’ criteria

• A thorough review of eFA literature– Backed up by a selective review of FA literature– Not a general review of e-assessment

Background findings re ‘plain’ FA

FA research is about several things

• Interaction– Teacher-learner; learner-learner

• Feedback is central– Speech – questions and answers– Comment-only marking on written work

Focus on features of the work, not the individual Permit learner to ‘close the gap’ between current and desired performance

• Features of FA items and tests relatively little studied– Can have more (or less) than one correct answer– Distractors explicitly connected to incorrect or incomplete conceptions– Item responses provide clues to effective action

Claims about FA

• Associated with learning gains– Claimed to be one of the strongest associations

between educational intervention and learning gains– Some question about whether all reported gains are

caused by the FA intervention

• Disseminating messages to practitioners key– Evidence that changes in practice have been patchy

Organisation of eFA findings

FindingsAreas for

further work Key issues

Finding 1

Electronic technologies provide a range of new tools that classroom teachers can use to create formative assessments to suit their and their students’ needs.

• Variations on the theme of MCQ• Sophisticated tasks – rich in interactivity and multimedia• Test designs specific to e-assessment• e-portfolios to closely integrate formative and summative

assessment• Communications tools such as electronic discussion boards

and forums for self- and peer feedback in e-learning courses

Characteristics of formative & summative items

• eFA implementations take item and task types from summative assessment– Cf. Wiliam’s characteristics of good formative items

• Need to build on research into FA item types• Establish distinctive characteristics of ‘good’ eFA

items/tasks

eFA as exam revision

• FA research emphasises several facets• Many eFA implementations equate FA with

revision or practice testing– Revision not a bad thing per se

Can reduce test anxiety Support distance learners

– But is a reduced concept compared to totality of FA

Need to write own test questions

• Early implementations of eFA often involve innovators developing own questions– Realistic for all teachers to write their own questions?– If teachers select from pre-written banks of questions,

is anything lost? Would they be able to tailor the formative

assessment to their learners’ needs? What sort of skill is selecting balanced formative

tests from an item bank?

Finding 2

e-assessment functionality permits formative feedback to be given in variety of ways not possible in plain FA.

• Variety of feedback methods in eFA test applications• Advantages of e-portfolios for providing feedback

– Encourages teachers and learners to interact about drafts– Process of generating work forefronted, rather than merely concentrating on the final

product– Communications tools can give feedback varied by: channel, recipient, formality, etc.

Can target learners of different styles

• E-learning technologies can facilitate giving of feedback to distance learners– Contribute to online discussions (peer feedback)– Reflective journal (self-assessment)– Evaluate peers’ work (either formally or informally)

Principles for feedback in e-tests

• Need to systematise understanding of e-test feedback– Does ‘comments not grades’ apply to eFA?

If so, one of the most obvious benefits of e-assessment (rapid right/wrong info) is removed.

– Is engagement with rich media and interactivity synonymous with deep learning? Could learners be distracted – clicking through

sites without really truly processing content?– Could we relate feedback to students’ learning styles?

Practicality of e-portfolio feedback

• Need logistical studies that investigate practicality of using e-portfolios– Ensure that teachers can provide high-quality

feedback and potential is not lost– ICT elements of portfolios should reduce burden

Impact of cultural factors in e-communication

• Students giving and receiving feedback need to understand cultural norms– Students working at distance may bring different prior

assumptions to feedback– Culture can also mean academic culture

Concern if students have not internalised academic norms Requirement to style switch between varying e-discourses

could make it harder for some to internalise ways of writing

Finding 3

eFA applications can be used remotely. This provides a resource which is not easily replicated via pencil-and-paper materials.

• Advantages of asynchronicity for undergraduates:– Reduced examination stress– Popular and motivating– Free up teacher time– Allow students to increase self-regulation for tertiary study– Asynchronous online discussions facilitates enhanced reflection

• Areas needing further clarification– Effect of learning styles and motivation of usage

Already self-regulating students benefit more Usage differs between intrinsically interested & ‘pragmatists’

– Clarify most common usage patterns

Better studies of learning gains

• Many papers assert attainment benefits of eFA– Esp. for self-access use

• Claims undermined by research design– Small cohorts– Test difficulty not equated– Variables confounded

• Does eFA provide attainment benefits over and above ‘plain’ FA?– It might – there are also claims that use of ICT in schooling is

associated with improved attainment

Self-access for primary and secondary students?

• eFA as self-access strongly associated with tertiary• What issues for self-access use by school pupils?

– Greater obligation on teachers to moderate feedback– Role of parents in online learning– Effective use to support ‘personalised learning’?

Superordinate ‘key issues’

Define how the ‘e’ goes beyond plain FA, AND

Define genuine distinctiveness of eFA as opposed to summative e-assessment

•Use of instruments by practitioners

•Provision of feedback

•Need to demonstrate learning gains

eFA must not be a reduced version of ‘plain’ FA

•eFA as exam revision

•Provision of feedback

Beware of how eFA may impose novel burdens on teachers and students•Requirement of teachers to write own test questions•Need to provide manageable systems for feeding back via e-portfolios

Need to understand how teachers and students adapt to novel roles that eFA requires

•Cultural demands of using e-communication tools

•Strong element of independent working and self-assessment

•Thanks for listening.

•Questions/comments …

Features of FA items and tests• Can have more (or less) than one correct

answer• Distractors explicitly connected to incorrect or

incomplete conceptions• Item responses provide clues to effective action