Functional, frustrating and full of potential:Learners’ experiences of a prototype for

automated essay feedback

Bethany Alden Rivers1, Denise Whitelock1, John T. E. Richardson1,

Debora Field2, and Stephen Pulman2

1The Institute of Educational Technology, The Open University2Department of Computer Science, University of Oxford

The SAFeSEA Project...

Supportive Automated Feedback for Short Essay Answers

OpenEssayist + EssayAnalyser

OpenEssayist...

• analyses drafts of students essays.

• offers instantaneous, automatic, individualised feedback.

• scaffolds learning through skill development.

• promotes self-regulation.

‘Views’Draft Overview Structured version of the essay, highlighting key words,

phrases and sentences

Key words and key phrases

Frequency of most used words and phrases

Key sentences Most important sentences in the essay

Key word dispersion

Distribution of key word and phrases across the essay

Word cloud Cluster of key words in different colours and sizes, according to their frequency

Word limit Number of words in each section compared to number of words in the essay

Word count Pie chart showing number of words in each section

Organise the key words

User can group key words and phrases in this page, which shows these in different colours in the Draft Overview.

e.g. Draft Overview

Evaluation Phase 1: Interviews with Users

• Postgraduate students at the UK Open University

• Used OpenEssayist between September 2013 to February 2014

• Completed evaluation survey (online)

• Contacted for follow-up interviews

The Case Study

Three storylines:

Two students who took part in the

evaluation phase

+

One student who opted out

Three questions:

Usefulness?

Potential?

Target user?

Maria’s story: “It encourages you to think but it’s too bewildering for a novice

learner.”

• Female, mid-50s, professional background in linguistics

• Planning is the most important part of writing.

• She has clear strategies for using feedback.

• Potential for the system to helps students with their writing style, word choice and essay structure

• Inappropriate tool for a beginner or for someone who was ‘not so familiar with ICT’

• OpenEssayist could be a catalyst for peer support

Robert’s story: “It could be useful but mainly for students

who are less confident.”

• Lifelong student

• Opted out of evaluation study

• Different approaches to planning and writing depending on the length of the essay.

• Has a clear process for assimilating feedback

• Beneficial for students with learning disabilities

• Helps a user focus on ‘what bits might be important’, like ‘structure or synthesis’.

Karina’s story: “Worrisome, confusing and fascinating: this system is for the

younger generation, not for mature learners.”

• On early retirement after career in technology

• ‘I always found that writing the essay was the hardest thing.’

• Wishes there was more dialogue around essay-writing

• First thoughts were of fascination and intrigue

• Needs an in-built narrative

• Should offer word choices, like a thesaurus

• Could provide exemplars to aide reflection

Findings: Usefulness?

Maria was the most positive about what the system could already do, in relation to the usefulness of the ‘views’.

Karina brought certain expectations of what the system was going to do. When these hopes were not met, this caused disappointment.

Findings: Target User?

‘It’s nice—but it’s not for me.’

Based on these three ‘stories’ it would seem that this system is most suitable for a traditional-aged university student in Year 2 or 3 of undergraduate study, who does not feel sure about his or her skills at essay-writing

Other current testing sites

• University of Hertfordshire

• British University in Dubai

• The Open University

Sun unavailable but what about the moon?

• Hints before writing?

• R.C.T.

• 2 essays

• F(1,41) = 3.23, p = 0.04 for hints

Towards the sun

• Give automatic feedback

• Use margin comments as a tutor

Margin comments

16

Category systems• Previous categorisation

schemes (Hyland, 2001; Perpignan, 2003; Whitelock et al., 2004; Brown & Glover, 2006; Nelson & Schunn, 2009)

• Concerned with:– Feed forward– Socio-emotive feedback

• SAFeSEA concerned with:– Decoding a margin

comment– Expression of opinion

through NL

Moving forwards towards a comment generator

• 3 layer coding designed by Debora Field

• Opinion, marker attitude, linguistic act

• In trials

Next steps

• Further evaluation testing• RCT on different types of

feedback• Motivated strategies for

learning questionnaire (MSLQ) adapted by Richardson (2006)

• Moving towards ‘advice for action’

Contact

Professor Denise Whitelock

Denise.Whitelock@open.ac.uk

www.open.ac.uk/researchprojects/safesea