assessment feedback systems and interactive learning environments the open university, walton hall,...

Assessment Feedback Systems and Interactive Learning Environments

The Open University, Walton Hall, Milton Keynes MK7 6AA

Professor Denise [email protected]

DMW Congress Colombia November 2014

I hate marking but want the tasks and feedback to assist student learning


The e-Assessment and automatic feedback Challenge

• Constructivist Learning – Push

• Institutional reliability and

accountability – Pull


DMW Congress Colombia November 2014DMW Congress Colombia November 2014

www.storiesabout.com

www.storiesabout.com/creativepdp

[email protected]


MCQs: Variation on a theme


Example of LAPT Certainty-Based Marking

UK cabinet ministers demo exercise

showing feedback,

University College, Tony Gardner-Medwin

Drug Chart Errors and Omissions

Medicines Administration Assessment,

Chesterfield Royal Hospital

MCQs: High Stakes Assessment


Example of practice “Thinking Skills Assessment" (TSA) question, Admissions Interview, Cambridge Assessment, Steve Lay

Example of practice “Thinking

Skills Assessment"

(TSA) feedback, Admissions Interview,

Cambridge Assessment, Steve Lay

Scaffolding and High Stakes assessment

• Math for Science

• Tutor less course

• Competency led

• No point to cheat

• Web home exam

• Invigilation technologies


Self diagnosis

• Basic IT skills, first year med

students (Sieber, 2009)

• Competency based testing

• Repeating tests for revision

• Enables remedial intervention


Theoretical drivers for Assessment and Feedback

Piaget (1930) Individual manipulating surroundings --

mental representations can be tested Induction of rules and

their testing ... Nuffield science

Cognitive psychology (70s 80s) How are these mental

representations stored? Classification development and how

this can go wrong. .... Misconceptions and mal rules........

Student modelling .... AI

Bruner (1982) assessment tasks will match competency

levels depending on level of help...SCAFFOLDING

Pask conversation model used by Laurillard


Scaffolding and Interactivity; The Open University Science Foundation Course

• Interaction, Feedback loops

• Tell, Explore, Check

• Predict, Look and Explain

• Entering the discourse of a subject via audio feedback

• Scaffolded text feedback (Bruner & Woods)

• SHOW ME button


Interactive Tasks

• Games

• Simulations

• Making the abstract concrete

• Directing the sequence of an

animation


Interactivity and Cognitive Change Scores


Theoretical drivers: Social Constructivism

• Vygotsky (1978) individuals shape cultural settings

which shapes minds no longer individual

• Activity theory (Engstrom 1987) Tool mediation

• Situated Cognition (Lave and Wenger 1991)

Authentic assessment

• Peer interaction and assessment

• Learning conversations Laurillard (2002)


Characteristics Descriptor

Authentic Involving real-world knowledge and skills

Personalised Tailored to the knowledge, skills and interests of each student

Negotiated Agreed between the learner and the teacher

Engaging Involving the personal interests of the students

Recognise existing skills Willing to accredit the student’s existing work

Deep Assessing deep knowledge – not memorization

Problem oriented Original tasks requiring genuine problem solving skills

Collaboratively produced Produced in partnership with fellow students

Peer and self assessed Involving self reflection and peer review

Tool supported Encouraging the use of ICT


Elliott’s characteristics of Assessment 2.0 activities

Authentic Assessment: Building e-portfolios on a chef’s course


Food preparation for e-portfolio

Modern Apprenticeship in Hospitality and

Catering, West Suffolk College, Mike Mulvihill

Evidence of food preparation skill for e-portfolio

Modern Apprenticeship in Hospitality and Catering,

West Suffolk College, Mike Mulvihill

Peer Assessment and the WebPA Tool

• Loughborough (Loddington et al, 2009)

• Self assess and peer assess with given criteria

• Group mark awarded by tutor

• Students rated:

• More timely feedback

• Reflection

• Fair rewards for hard work

• Staff rated:

• Time savings

• Administrative gains

• Automatic calculation

• Students have faith in the administrative system


Mobile Technologies and Assessment

• MCQs ,PDAs Valdiva &

Nussbaum(2009)

• Polls,instant surveys

• Simpson & Oliver (2007)

• Draper (2009) EVS


Gains from Formative Assessment

• Mean effect size on standardised tests between 0.4 to 0.7

(Black & Williams, 1998)

• Particularly effective for students who have not done well

at school http://kn.open.ac.uk/document.cfm?docid=10817

• Can keep students to timescale and motivate them

• How can we support our students to become more

reflective learners and engage in formative assessment

tasks?


Collaborative formative assessment with Global Warming

DMW, Institute of Educational Technology, September 1997DMW, Institute of Educational Technology, September 1997


Global Warming


Next: ‘Yoked’ apps via BuddySpace

Student A

Student B(‘yoked’, butwithout full screen sharingrequired!)


Global Warming: Simlink Presentation


LISC: Aily Fowler

Kent University ab-initio Spanish module

•Large student numbers

•Skills-based course

•Provision of sufficient formative assessment meant

unmanageable marking loads

•Impossible to provide immediate feedback

• leading to fossilisation of errors


The LISC solution: developed by Ali Fowler

A call system designed to enable

students to:

•Independently practise sentence

translation

•Receive immediate (and robust) feedback

on all errors

•Attend immediately to the feedback

(before fossilisation can occur)


How is the final mark arrived at in the LISC System?

The two submissions are unequally weighted

•Best to give more weight to the first attempt

• since this ensures that students give careful

consideration to the construction of their first

answer

• but can improve their mark by refining the answer

•The marks ratio can vary

(depending on assessment/feedback type)

• the more information given in the feedback, the

lower the weight the second mark should carry


Heuristics for the final mark

If the ratio is skewed too far in favour of

the first attempt…

• students are less inclined to try

hard to correct non-perfect

answers

If the ratio is skewed too far in favour of

the second attempt…

• students exhibit less care over the

construction of their initial answer


What about emotional support in the feedback?

• Difficult at times to receive written feedback

• Not just a cognitive response

• How can Bales help?


Coding into Categories

• Bales analysis

• Psychology 1950s

• Analyses talk

• Includes socio-emotive categories

• Flander’s (1970) categories

inappropriate as also includes

classroom control


Bales Categories

• Four main groupings

• A. Positive reactions; agreeing

and boosting the other person

• B. Directing/teaching

• C. Questions: requesting

information, clarification etc

• D. Negative reactions:

disagreement


Coding the tutor comments

Categories Specific Examples

Positive Reactions

A1

A2

A3

1. Shows solidarity

2. Shows tension release

3. Shows agreement

Jokes, gives help, rewards others

Laughs, shows satisfaction

Understands, concurs, complies, passively accepts

Attempted Answers

B1

B2

B3

4. Gives suggestion

5. Gives opinion

6. Gives information

Directs, proposes, controls

Evaluates, analyses, expresses feelings or wishes

Orients, repeats, clarifies, confirms

Questions

C1

C2

C3

7. Asks for information

8. Asks for opinion

9. Asks for suggestion

Requests orientation, repetition, confirmation, clarification

Requests evaluation, analysis, expression of feeling or wishes

Requests directions, proposals

Negative Reactions

D1

D2

D3

10. Shows disagreement

11. Shows tension

12. Shows antagonism

Passively rejects, resorts to formality, withholds help

Asks for help, withdraws

Deflates others, defends or asserts self

Bales’ Interaction Process


Identifying trends: H801

Graph to show conflated Bale’s

categories against mean number

of incidences in H801 scripts

0 5 10 15 20 25

A Pass 1

A Pass 2

A Pass 3

A Pass 4

B Pass 1

B Pass 2

B Pass 3

B Pass 4

C Pass 1

C Pass 2

C Pass 3

C Pass 4

D Pass 1

D Pass 2

D Pass 3

D Pass 4

Ba

les'

In

tera

ctio

na

l C

ate

go

rie

s a

t e

ach

Pa

ssL

leve

l

Number ofIincidences


Identifying trends: H801

Pie Chart to show the mean

number of incidences per pass per

conflated Bales' Interactional

Category for all four levels of pass

in H801 scripts

5.96

17.13

5.73

1.61

A

B

C

D

Key:

A = Positive reactions

B = Responses

C = Questions

D = Negative reactions


Identifying trends

A

B

C

D

A

B

C

D

A

B

C

D

A

B

C

D

Pie Charts to show the mean number of incidences per conflated Bales Interactional

Category for ‘Pass 1’ and ‘Pass 4’ in the following courses:

Key:A = Positive reactions C = QuestionsB = ResponsesD = Negative reactions

Pass 4

Pass 1

B820 S103

A

B

C

D

A

B

C

D

H801


What is Open Mentor?

“An open source mentoring tool for tutors”

“Open source” = free and easy to use, and to embed in an

institutions infrastructure and working practices

“mentoring” = designed to help people learn how to give

feedback effectively, through reflection and social networks

“tutors” = primarily intended for teaching staff, but with clear

applications for those involved in quality


How Open Mentor handles comments

“Good work”

“Yes, well done”

“Yes, but is this useful?”

“Can you explain what you mean”

“This does not follow”

A = positive reactions

A = positive reactions

B = attempted answers,

and not a positive reaction

C = questions

D = negative reactions


Explaining OpenMentor’s Rules

Four categories

•A – Positive Reactions

•B – Attempted Answers

•C – Questions

•D – Negative Reactions


‘A’ - Positive Reactions

Category Examples of Rules Examples of commentsA - Positive Reactions1. Shows solidarity A1 ...excellent... Excellent Conclusions.

A1 ...(good|comprehensive)... Good, you are drawing on hard facts here.A1 ...nicely... Very nicely stated. Your analysis is thorough and your

conclusions consistent regarding the attractiveness of the budget airline sector. This is a good example of critical thinking.

A1 ...well presented... Very well presented diagram with interesting information.A1 ...effective use... Effective use of the case material here.A1 …well (structured|stated)… Report very well structured.A1 ...(well|clear)(ly)*

(structured|structure|summary| summarised|presented|presentation)...

The corporate vs. business unit strategy is well presented and nicely tied to strategies.

A1 ...reasonable.... A reasonable structure as listed in your table of contents.A1 ...useful point(s)... Generally useful points in this section.

2. Shows tension release A2 ... a helpful...

A2 …(thanks|thank you)…3. Shows agreement A3 ...yes... Yes, the intellectual reactions are both real.

A3 ...indeed... Indeed – if it has one basic strategy it is surely differentiation, though it still has to control costs.

4. Praise then direction A4 good...but... Good model, good quote, but be careful about what industry you analyse ??


‘B’ Attempted Answers

Category Rules Examples of commentsB - Attempted Answers

B4 …perhaps… Perhaps even better here to explain the link in your mind between "analysis of stratgeies" and "strategic issues".

B4 …requires…B4 …take care… Take care with your STEP analysis not to make it too industry

focussed.B4 …useful to… Innovation is closely linked to structure and culture- it would be

useful to see some book 6-8 concepts here too.

B4 …you (might|ought)… You ought to have explicitly stated these.

B4 Don’t|never … Don’t introduce new frameworks just for the sake of it in the conclusion. The conclusion should be pulling together what went before.

B4 Please (see|refer to|look at)... Please make sure to read and understand the question correctly

5. Gives opinion B5 I (am|think)... I think I can see where you are going, though a numbered report format might have demonstrated the approach better

B5 This is.... this is an introduction rather than a “summary”B5 ...sounds...like... This sounds as if it could be very popular!!B5 ...not sure... I am not sure about the balance between the environmental

analysis and the review of the resources, capabilities (power, culture, structures and systems) as raised in the question.

B5 I (thought|agree|suggest)… I thought it was because they did not need any external input and saw a significant market sector they could address themselves.

B5 I (do|don’t) think… I don’t think this exercise has helped to develop your analysis. I also think that the development of the perspectives is superficial

6. Gives information B6 …(demonstrates|shows) this….B6 Also… Also, cross link to Leadership issues, Pettigrew on Strategic

Thinking tooB6 ...Q1... etc Q1 = 59/100


Is the rule set generic?

Comments Classified from Test Data

0102030405060708090

100

B820 TestSet 1

B820 TestSet 2

B820 TestSet 3

A850 M878 S809

Test Data Set

% o

f co

mm

ents

cla

ssif

ied


What can we learn from modelling tutors marking to construct a formative e-assessment tool?

• Open Comment project builds on

the work of OpenMentor

• Free text entry for History and

Philosophy students

• Immediate feedback (in context)

to students

• Influenced by ELIZA

(Weizenbaum, 1963)


Stuart Watt

Changed to reduce the "modelled on" description

Open Comment addresses the problem of free text entry

• Automated formative assessment tool

• Free text entry for students

• Automated feedback and guidance

• Open questions, divergent assessment

• No marks awarded

• For use by Arts Faculty


Causal models of explanation

First step:

• Identification of question types where

students exhibit causal reasoning

Looked for questions with:

• Salient variables

• Description of these variables

• Identification of trends

• Identification of relationship between

the variables i.e. causality


Praise for effort and not just ability

• Praise for ability per se can hinder

learning (Mueller & Dweck, 1998)

• Praise = being clever

• Negative feedback now

without ability

• Disempowering and demoralising


Praise and motivation: feedback and self regulation

• “Feeling good” feedback part of many guidelines

• Mueller & Dweck (1998) Praise for effort as well as

performance

• Raven’s test given to USA school children

• First test feedback was praise

• Second test most difficult. Half praised for effort,

half praised for ability

• Third test medium difficulty

• On the third test pupils praised effort increased score by

1. Praised for ability decreased 1 mark (scale 0-10)


How does feedback effect mindsets?

1. Your intelligence is something very basic about you

that you can’t change very much

2. You can learn new things but you can’t really change

how intelligent you are

3. No matter how much intelligence you have you can

always change it quite a bit

4. You can always substantially change how intelligent

you are


Mindsets (Dweck, 2006)

• Fixed mindset

• Super sensitive about being

wrong

• Always trying to prove

themselves

• Growth mindset

• Stretch themselves

• Confront obstacles as

challenges

• Lack of tension when

learning as they know they

are novices and can

improve


Stages of analysis by computer of students’ free text entry for Open Comment: Advice with respect to content (socio-emotional support stylised example)

Stage 1a: Detect errors

E.g. Incorrect dates, facts. (Incorrect inferences and

causality is dealt with below)

•Instead of concentrating on X, think about Y in order to

answer this question Recognise effort (Dweck) and

encourage to have another go

•You have done well to start answering this question but

perhaps you misunderstood it. Instead of thinking about X

which did not…….. Consider Y


Computer analysis continued

Stage 2a: Reveal first omission

Consider the role of Z in your answer Praise

what is correct and point out what is missing

Good but now consider the role X plays in your

answer

Stage 2b: Reveal second omission

Consider the role of P in your answer Praise

what is correct and point out what is missing

Yes but also consider P. Would it have

produced the same result if P is neglected?


Final stages of analysis

• Stage 3:request clarification of key point 1

• Stage 4: request further analysis of key point 1

– stages 3 and 4 repeated with all the key points

• Stage 5:request the inference from the analysis

of key point 1 if it is missing

• Stage 6:request the inference from the analysis

of key point 1 if it is not complete

• Stage 7:check the causality

• Stage 8:request all the causal factors

are weighted


Open Comment drivers for reflection

• Students are able to find facts similar to X

• Know how X might be disputed

• Are able to make predictions about X

• Know how to use X in an argument

• Know how far X can be pushed

• Supported with tools and strategies for effort


SAFeSEA

Professor Denise Whitelock

Professor John Richardson

Professor Stephen Pulman

An automated tool supporting

online writing and assessment

of essays providing

accurate targeted feedback

SAFeSEA: Supportive Automated Feedback for Short Essay Answers


About SAFeSEA

• No tutor support for drafts of first assignment

• Reduce dropout rate with automatic feedback?

• Effect of summaristion

• What are the beneficial factors?

• Correlate measures of learner activity and

essay improvement

• http://www.open.ac.uk/researchprojects/safesea/


Talk Back

• Checking understanding by ‘talk back’

• Summaries in OpenEssayist

• Key words = key ideas

• www.open.ac.uk/researchprojects/safesea/


OpenEssayist: What it tells you

The system’s focus is to present summaries of your own work

in different ways, to encourage you to reflect constructively on

what you have written.

In other words Open Essayist tells you from its analysis what

are the most important or key points in your essay. You can

then think about whether that was what you intended to

emphasis in your essay. If not then you can make the

appropriate changes.

A very important aspect of the OpenEssayist system is that it

will not tell you what to write, or how to rewrite sections of

your essay, or even what is correct or incorrect in your essay.


ESSAY

GRAPHICS

ANALYSIS

OpenEssayist: How it gives feedback

Three aspects of your essay are analysed by the system:

• the structure of the essay (which paragraphs constitute

the introduction, the conclusion, the discussion

sections, etc.),

• the key words and key phrases of your essay (which

are the most important words and phrases, the ones

that are most representative of your essay's overall

meaning)

• the key sentences of your essay (which are whole

sentences that are most representative of your essay's

overall meaning).


Sample key phrases dispersion plot


Visualisations as a Thinking Tool

• Visualising text (Bertin, 1981;

Johnson et al, 1993)

• Free text visualisation still

problematic

• Do users need training with

visualisations?

• Convention vs. Instruction?


Short text for illustration of Rainbow Diagrams


Sentence graph of short text


Pretend essay: 10 identical paragraphs


Pretend essay: 50 identical sentences


Stanford University Boothe Prize essay


OU essay awarded high grade


OU essay awarded low grade


Good vs. bad?

Bad:

•not densely connected•red nodes (conclusion)

not central•few links between violet (intro)

and red nodes


Good:

•densely connected•red nodes (conclusion)

central•close links between violet

(intro) and red nodes

Findings

“One of the clues was talking about the way the color nodes, that the red ones are at the end. If they are a good connection with a good explanation for each paragraph. They should be connected together but the darker ones should be in the middle. But you see the color groups together so for me it automatically pulls my eye to this page because all the colors are closer together and more in the middle. So that would be the student course assignment essay with the highest mark.”

“You don’t show me anybody’s text. You are not revealing anyone else’s essay. So students cannot plagiarize. But you are saying ‘Look hang on, this is the way this essay connects together.’ That’s what telling a good story is about this linking.”


Rainbow diagrams related to mark awarded

• Multivariate analysis of variance on marks

awarded to 45 students

• Submitted two essays

• Rainbow diagrams produced from these essays

and rated as high, medium or low attainment

• Covariate showed a significant relationship with the marks

• F(1, 43) = 5.92, p = .01 using a directional test

• Essays rated as high would be expected to receive 8.56

percentage points more than essays rated as medium

• 17.2 percentage points higher than essays rated

from rainbow diagrams as low


Is NLP the bridge between Computer Assisted Assessment and Learning Analytics?

• Automatic marking

• Recognising text

• Refining text


How about feedback first?

• Hints before writing?

• R.C.T.

• 2 essays

• F(1,41) = 3.23 p = 0.04 for hints


Creating teaching and learning dialogues: towards guided learning supported by technology

• Learning to judge

• Providing reassurance

• Providing a variety of signposted

routes to achieve learning goals

• Provide socio-emotive support


Feedback: Advice for Action

• Students must decode feedback and

then act on it Boud (2000)

• Students must have the opportunity to

act on feedback Sadler (1989)

• Gauging efficacy through student action


Characteristics Descriptor

Authentic Involving real-world knowledge and skills

Personalised Tailored to the knowledge, skills and interests of each student

Negotiated Agreed between the learner and the teacher

Engaging Involving the personal interests of the students

Recognise existing skills Willing to accredit the student’s existing work

Deep Assessing deep knowledge – not memorization

Problem oriented Original tasks requiring genuine problem solving skills

Collaboratively produced Produced in partnership with fellow students

Peer and self assessed Involving self reflection and peer review

Tool supported Encouraging the use of ICT

Elliott’s characteristics of Assessment 2.0 activities

Advice for Action


The 4Ts Pyramid

Tool Development

Training of staff

TransformationTasks

TransferLearning


Training

• Question development

• Quality REAQ

• Plagiarism software

• Advice for Action

• Socio e-emotive content

• Maintain empathy with the Learner


e-Assessment Futures

• Research issues

• Adaptive testing

• Automatic marking

• Learning Analytics Data mining

• Web 2.0, 3.0 …


Feedback to prompt self reflection

• Analysis must prompt “Advice for

Action”

• Self reflective discourse with

computer feedback

• Visual representation of feedback

can open a discourse between

tutor and student

• Prompt peer to peer discourse


References 1

Whitelock, D. (2010) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global. pp. 319–342.

Whitelock, D. (2008) ‘Accelerating the assessment agenda: thinking outside the black box’. In Scheuermann, F. & Pereira, A.G. (eds.) Towards a Research Agenda on Computer-Based Assessment: Challenges and Needs for European Educational Measurement, published by the European Commission, Joint Research Centre. pp15-21 ISSN. 1018-5593


References 2


Five assessment special issues


assessment feedback systems and interactive learning environments the open university, walton hall,...

Documents