The Use of ICT for the Assessment of Key Competences

Irish Presidency Conference

Better Assessment and Evaluation to Improve Teaching and Learning

19-20 March 2013

Christine RedeckerYves Punie

IPTS: Part of Joint Research Centre of the EC: 7 Research Institutes across Europe

Mission: “to provide customer-driven support to the EU policy-making process by developing science-based responses to policy challenges that have both a socio-economic as well as a scientific/technological dimension”

IPTS- Who we are

Future Learning Spaces in 2020

2006 2007 2008 2009 2010 2011 2012 2013

Learning 2.0 in formal E&T Learning 2.0 in informal settings

Creativity & Innovation in schools in EU

eLearning in the EU10 MS

- Digital Competence- Active Ageing & Learning

Tell-Net Teacher Networking

Review ICT

Impact

IPTS Studies on ICT for Learning

The Future of Learning

ICT for the Assessment of Key Competences

Digital Competence

Mainstreaming Creative Classrooms

Open Education in Europe

1to1 Learning initiatives in Europe

Mapping Technologies for Learning

DC Follow up

Where we stand

Multiple Choice

Short Answer

Generation 1Automated administration

and/or scoring of conventional tests

Adaptive TestingGeneration 2 Generation 4

Intelligent Tutors

Automated Feedback

Personalised feedback and

tutoring

Learning 2.0

Generation 3Continuous integrated assessment

Data Mining & Analysis

Behavioural tracking

Generation Re-inventionTransformative strategy: use technology to

change test formats

Peer assessment

Collaborative multimedia learning environments

Online collaborationVirtual laboratories

Games

Simulations

Virtual Worlds

Replication of complex real life situations within a

test

Bun

ders

on, I

nouy

e &

Ols

en (

1989

). T

he f

our

gene

rati

ons

of c

ompu

teri

zed

educ

atio

nal

mea

sure

men

t. In

R. L

. Lin

n (E

d.),

Edu

cati

onal

mea

sure

men

t (T

hird

ed.

, pp.

367

-407

). N

ew

Yor

k: M

acm

illa

nB

enne

tt, R

. E. (

2010

). T

echn

olog

y fo

r L

arge

-Sca

le A

sses

smen

t. In

P. P

eter

son,

E. B

aker

&

B. M

cGaw

(E

ds.)

, Int

erna

tion

al E

ncyc

lope

dia

of E

duca

tion

(3r

d ed

., V

ol. 8

, pp.

48-

55).

fo

rd: E

lsev

ier.

1990 1995 2000 2005 2010 2015 2020 2025Efficient testing Personalised learning

Learning Analytics

Computer-Based Assessment (CBA) Technology-enhanced learning

Current use of ICT in assessment in schools

Central recommendations on the use of ICT in pupil assessment in compulsory education in primary and general secondary education (ISCED 1, 2, 3), 2009/10Source: Eurydice (2011). Key Data on Learning and Innovation through ICT at School in Europe 2011, Education, Audiovisual and Culture Executive Agency.

There is scope for improvement

1st Generation CBA: Problem-Solving

http://www.worldclassarena.org/v5/flash/09_year_old/backtofront.swf

World Class Arena Sample Test: Problem Solving (9 years)

Examples

http://www.worldclassarena.org/v5/flash/13_year_old/pyramids.swf

Problem Solving 13 years

Generation Re-Invention: Problem-Solving

Examples

1st Generation CBA: Situational Judgement

EPSO sample situational judgement test questions; http://europa.eu/epso/discover/prepa_test/sample_test/index_en.htm

Examples

Generation Re-Invention: e-tray exercise

EPS

O s

ampl

e as

sess

men

t cen

tre

e-tr

ay e

xerc

ise

http

://eu

ropa

.eu/

epso

/dis

cove

r/pr

epa_

test

/sam

ple_

test

/inde

x_en

.htm

Examples

Future eAssessment Strategies

Multiple Choice

Short Answer

Generation 1Automated administration

and/or scoring of conventional tests

Adaptive TestingGeneration 2

Generation 4

Intelligent Tutors

Automated Feedback Personalised feedback and

tutoring

Learning 2.0

Generation 3Continuous integrated assessment

Data Mining & Analysis

Behavioural tracking

Generation Re-inventionTransformative strategy: use technology to

change test formats

Computer-Based Assessment (CBA) Technology-enhanced learning

Peer assessment

Collaborative multimedia learning environments

Online collaborationVirtual laboratories

Games

Simulations

Virtual WorldsReplication of complex

real life situations within a test

Bun

ders

on, I

nouy

e &

Ols

en (

1989

). T

he f

our

gene

rati

ons

of c

ompu

teri

zed

educ

atio

nal

mea

sure

men

t. In

R. L

. Lin

n (E

d.),

Edu

cati

onal

mea

sure

men

t (T

hird

ed.

, pp.

367

-407

). N

ew

Yor

k: M

acm

illa

nB

enne

tt, R

. E. (

2010

). T

echn

olog

y fo

r L

arge

-Sca

le A

sses

smen

t. In

P. P

eter

son,

E. B

aker

&

B. M

cGaw

(E

ds.)

, Int

erna

tion

al E

ncyc

lope

dia

of E

duca

tion

(3r

d ed

., V

ol. 8

, pp.

48-

55).

fo

rd: E

lsev

ier.

Learning Analytics

1990 1995 2000 2005 2010 2015 2020 2025Efficient testing Personalised learning

Towards Generation 3: Datamining

Signalshttp://www.itap.purdue.edu/tlt/signals/signals_final/index.htm

Similar: GPS at Northern Arizona University

http://resourceconnect.nau.edu/grade-performance-status.asp

Examples

Towards Generation 3: Behaviour Tracking

Example: SNAPPSNAPP is a software tool that allows users to visualize the network of interactions resulting from discussion forum posts and replies. Teachers can rapidly identify patterns of user behaviour. SNAPP can extract all user interactions from various commercial and open source learning management systems, such as BlackBoard and Moodle.

Examples

Towards Gen 4: Intelligent Tutoring Systems

GnuTutor (http://gnututor.com/)

OpenSource version of AutoTutorhttps://www.youtube.com/watch?v=k6HDEEnYuIA&feature=player_embedded

AutoTutorhttps://www.youtube.com/watch?v=aPcoZPjL2G8

Examples

Generation 1 Generation 2

Generation 4Personalised feedback and

tutoring

Learning 2.0

Generation 3Continuous integrated assessment

Generation Re-invention

Computer-Based Assessment (CBA) Technology-enhanced learning

Collaborative multimedia learning environments

Bun

ders

on, I

nouy

e &

Ols

en (

1989

). T

he f

our

gene

rati

ons

of c

ompu

teri

zed

educ

atio

nal

mea

sure

men

t. In

R. L

. Lin

n (E

d.),

Edu

cati

onal

mea

sure

men

t (T

hird

ed.

, pp.

367

-407

). N

ew

Yor

k: M

acm

illa

nB

enne

tt, R

. E. (

2010

). T

echn

olog

y fo

r L

arge

-Sca

le A

sses

smen

t. In

P. P

eter

son,

E. B

aker

&

B. M

cGaw

(E

ds.)

, Int

erna

tion

al E

ncyc

lope

dia

of E

duca

tion

(3r

d ed

., V

ol. 8

, pp.

48-

55).

fo

rd: E

lsev

ier.

- mainstream use for high stakes testing- further development of innovative test formats- increased ease of use at school level - trend towards automated scoring

→ cultural change: increased acceptance of CBA

Automated administration and/or scoring of conventional tests

Adaptive Testing

Replication of complex real life situations within a test

- Multimedia collaborative learning environments support key competence

development.

- Peer and portfolio assessment evolve naturally.

-Data mining and analysis will make integrated assessment and tutoring possible

in the future.We are in a transitional period: Moving from the “Testing Paradigm” to the (future) “Tutoring Paradigm” in which explicit

testing could become obsolete

Technological trends1990 1995 2000 2005 2010 2015 2020 2025

Efficient testing Personalised learning

Key Questions

1. Which ICT tools, formats, environments, devices etc. are most suitable for the assessment of which Key Competences?

2. How can ICT be used to support competence-based assessment strategies that adequately capture the skills and attitudes dimensions of all eight Key Competences and their transversal nature?

Assessing Key Competences

F = Formative AssessmentD = Diagnostic AssessmentS = Summative Assessment

1 Communication in the mother tongue DS FD FS

2 Communication in foreign languages DS FS

3a Mathematical competence DS FD FDS FDS

3b Basic competences in science and technology FS FDS

4 Digital competence FDS

5 Learning to learn FS FDS

6 Social and civic competences FS

7 Sense of initiative and entrepreneurship FS

8 Cultural awareness and expression FS FS

CB

A

ePor

tfol

ios

Pee

r A

sses

smen

t

Sel

f A

sses

smen

t

Qu

izze

s /

sim

ple

ga

mes

Vir

tual

wor

lds

gam

es

Sim

ula

tion

s

Inte

llig

ent

Tu

tors

etc

.

Focus Areas

•PART 1

•INSPIRATION

Mother Tongue Literacy1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

Matching spoken and written languagehttp://www.familylearning.org.uk/sight_word_games.html

Foreign language learning

2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

http

://w

ww

.nic

kjr.

de/g

ames

/107

-sw

iper

s-bu

chst

abie

r-bu

ch

Mathematical games3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

Mathematical games using simulation for secondary school educationSource: http://hotmath.com/games.html.

5 Learning to Learn CBA (F) Peer Assessment (FS) Self Assessment (FS)

http://spark.uts.edu.au/

Example: SparkPlus

Peer Assessment for Collaborative Work

• http://www.youtube.com/watch?v=Tp8v2UBbmN4

www.moodle.org

Moodle is a free web application that educators can use to create effective online

learning sites.

Moodle StatisticsRegistered sites: 77,213Countries: 229Courses: 7,116,466Users: 66,349,750Teachers: 1,292,731Enrolments: 56,306,274Forum posts: 118,413,272Resources: 63,891,942Quiz questions: 147,057,434

Peer Assessment using moodle

ePortfolios

1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)http://sites.google.com/site/eportfolioapps/overview

/model-k-12

ePortfolios

ePortfolios1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)

http://sites.google.com/site/eportfolioapps/overview

/levels

Level 1 - ePortfolio as Storage Collection: Creating the Digital Archive (regularly – weekly/monthly)A Focus on Contents & Digital Conversion

Level 2 - ePortfolio as Workspace Collection + Reflection: Immediate Reflection on Learning & Artifacts in Collection (regularly) A Focus on Process & Documentation of Learning

Level 3 - ePortfolio as Showcase Selection/Reflection/Direction (each semester? End of year?)A Focus on Product & Documentation of Achievement

→ A (formative/summative) assessment approach on the basis of artefacts → Focus on production and reflection

A game approach to arts

8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)

http://pheon.org; http://pheon.org/intro.html; http://americanart.si.edu/multimedia/games/pheon.cfm.

Pheon is a multimedia scavenger hunt that tests players’ ingenuity through exploring the collection of the Smithsonian American Art Museum, making objects, and texting through the galleries

ePortfolios meetVirtual reality

Science: Molecular Workbench

3b Basic competences in science and technology CBA (S) Virtual Worlds / games (FS) Simulations (FDS)

http://mw2.concord.org/public/part2/diffat/page3.cml

3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

Simulations for mathematics

Sample task from US future New State Tests

(2014/15)

http://www.k12center.org/rsc/pdf/a-sea-change-in-assessment-letter-size.pdf

Cognitive Tutor (Carnegie)

3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

http://www.youtube.com/watch?v=RR4HPrGlUz8&feature=related

Science: Quest Atlantis

3b Basic competences in science and technology CBA (S) Virtual Worlds / games (FS) Simulations (FDS)

Example: Quest Alantis

Quest Atlantis (QA) is an international learning and teaching project that uses a 3D multi-user environment to immerse children, ages 9-16, in educational tasks. QA combines strategies used in the commercial gaming environment with lessons from educational research on learning and motivation.

Citizenship6 Social and civic competences CBA (DS) Peer/Self Assessment (FDS) Virtual Games / Simulations (FDS)

Educational game, designed to help teach concepts in citizenship, geography, and media. Developed by Serious Games International, it has detailed lesson plans and assignments for students.

Example: Child labour in Bangladesh:

http://www.youtube.com/watch?v=mf9Ju86IVYw

Global Conflicts http://www.globalconflicts.eu/

Examples

•PART 2

•CRITICAL INSPECTION

Digital Competence

4 Digital competence CBA (S) Simulations (FDS)

Internet and Communication Core Certification (IC3) exam sample questionhttp://www.certiport.com/Portal/common/htmllibrary/Sales-Video-Demos/demo_ic3.html

→ Focus on knowledge

CBA forDigital Competence

Digital Competence: IC3

4 Digital competence CBA (S) Simulations (FDS)

Inte

rnet

and

Com

mun

icat

ion

Cor

e C

erti

fica

tion

(IC

3) e

xam

sa

mpl

e qu

esti

onht

tp:/

/ww

w.c

erti

port

.com

/Por

tal/

com

mon

/htm

llib

rary

/Sal

es-

Vid

eo-D

emos

/dem

o_ic

3.ht

ml

→ Focus on operational skills

CBA forDigital Competence

Entrepreneurship Questionnaire

7 Sense of initiative and entrepreneurship CBA (DS) Peer/Self Assessment (FDS) Virtual Games / Simulations (FDS)

?https://

www.careeringahead.com.au/survey/Entrepreneurship/

Generic/questions.asp

→ Focus on personality/disposition as a static fact

Entrepreneurship Questionnaire

Conclusions

ICT offers powerful, engaging assessment formats, in particular:

CBA ePortfoliosSelf / Peer

AssessmentQuizzes /

simple games Virtual worlds/ gamesSimulations

Basic literacy(foreign/mother tongue)

Basic mathematical competence

Learning to Learn

Creative writing(foreign/mother tongue)

Cultural awareness and expression

Scientific competence

Digital competence

Social and civic competences

Educational software

Mathematical competence

CBA lends itself to simplified, knowledge-based assessment→ Pedagogy comes first! Careful selection of question formats

ICT Assessment approaches are underdeveloped for:

Learning to Learn Sense of initiative and entrepreneurship Social and civic competences

Caveat:

The Paradigm Shift

Technology

1990 1995 2000 2005 2010 2015 2020 2025Focus on: Efficient testing Focus on: Personalised learningEra of Computer-Based Assessment (CBA) Era of Technology-enhanced learning

PedagogyEfficient Testing

Validity of Scores

QuickFeedback

Authentic Contexts

CBA(Gen.1)

AdaptiveTests

(Gen.2)

AutomaticScoring

Gen. Re-Invention

Learning2.0

Immersive Learning

LearningAnalytics

IntelligentTutors

? ? ? ?

→ Primacy of Pedagogy: Pedagogical guidelines for intergrated

assessment

• Thank you very much for your attention!

• Christine Redecker• Yves Punie

• Christine.Redecker@ec.europa.eu• Yves.Punie@ec.europa.eu

http://ipts.jrc.ec.europa.eu/publications/pub.cfm?id=5719

further information ……

http://is.jrc.ec.europa.eu

Newsletters: http://is.jrc.es/pages/newsletter.html

Publications: http://is.jrc.ec.europa.eu/pages/Publications.html

Contact: jrc-ipts-is-NEWS@ec.europa.eu