technology-based assessment: challenges and promises

Gyngyvr MolnrUniversity of Szeged http://www.staff.u-szeged.hu/~gymolnar

Technology-based Assessment: Challenges and Promises

6th EDEN Research WorkshopBudapest, Hungary, 24-27 October, 2010

Research Group on the Development of Competencies, Hungarian Academy of Sciences

Institute of Education, University of Szeged

Developing Diagnostic Assessments Project

Outline

Need for change Possibilities and promises of TBA Implementing TBA in international projects



The need for change

Technology abounds Change - Live, learn,

communicate New insight into the

learning process Potential to transform

education



The need for change

New expectation, lifestyle - the Net Generation digitalnatives => calls for different type of teaching

Schifting from generationgap to generation lap

New demands on the education



The need for change Existing models of

assessment fail New assessments are

required

Multi-sensory feature Retention rates Technology alone will not

result in higher achievement



Technology is not the issue - > catalyst

The need for change

Data about education quality and its impact on economic output

After 4 cycles of PISA Economic modeling E.g. how the GDP would

change according to the change in educational output

Result: relatively small improvements large impact on future





0

200

400

600

800

1000

1200

Mex

ico

Turk

eyG

reec

eP

ortu

gal

Italy

Luxe

mbo

urg

Spa

inU

nite

d S

tate

sP

olan

dN

orw

ayS

lova

k R

epub

licO

EC

DH

unga

ryD

enm

ark

Ger

man

yIc

elan

dFr

ance

Irela

ndS

wed

enA

ustri

aS

witz

erla

ndB

elgi

umC

zech

Rep

ublic

Uni

ted

Kin

gdom

Aus

tralia

New

Zea

land

Can

ada

Net

herla

nds

Japa

nK

orea

Finl

and

Catching up with Finland(in percent of GDP)

% currentGDP

Why is assessment so important?

You can not improve what you can not measure(Schleicher, 2010)



The feedback in education ICT growing in every segment of education International tendencies (OECD, ETS, NCES) in

educational assessment PPT reached its barrier Human resources, cost of money and time TBA solve this problem NEED to use technology to improve the efficiency of

assessment Improved quality of data and increased speed of

assessment Better conditions to improve the educational system



Possibilities and promises of TBA(1) Reach stimulus material (multimedia tools) New item formats New construct (PS in technology reach environment) New areas (dynamic assessment) assessing dynamic

MicroDYN approach example



Possibilities and promises of TBA

(2) TBA rises new issues in assessment:

background data testing prozess - e.g. educational data mining log file analyses - issues of reaction time, sequence

annotations regarding eye and face tracking



Possibilities and promises of TBA(3) offers new assessment methods (adaptive testing)

the difficulty of the test tailors dynamically to the students ability level

items are selected from an item bank dependent at a difficulty appropriate for the student

provides more time-efficient and accurate assessments.



-4

-3

-2

-1

0

1

2

3

4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

logi

tsca

le

item

Adaptive testing

Person ability level



(4) increases motivation - students with high and low ability

level - flow experience (5) changes the whole assessment process (item generation,

scoring, data-processing, information flow, feedback and the speed of assessment)

rapid and precise feedback (6) poses new questions and problems if TBA replace PP

improved quality and increased quantity of data and increased speed of assessment



Possibilities and promises of TBA

Implementing TBA in internationalprojects

Europe: growing attention in several European countriesand the European Union also facilitates these attempts

At national level Luxemburg leads the way: able to assess an entire cohort simultaneously automatic generation of the feedback to the teachers University of Luxembourg and the Public Research

Center Henri Tudor TAO (Testing Assist par Ordinateur)




Germany DIPF TBA project : adapts and further develops the TAO platform establish a national standard for technology-

assisted testing




In Hungary 2008 first media effect study: Inductive reasoning 5th grader (PP + CB) TAO platform Result: global achievement highly correlate items,

persons behave differently 2009 - a large-scale project was launched by the Center

for Research on Learning and Instruction - University of Szeged - develop an online diagnostic assessment system for the first six grades of primary school in reading, mathematics and science



Developing DiagnosticAssessments Project

the development of assessment frameworks devising a large number of items both on paper and on

computer - building item banks research on comparing the achievements on the tests

using different media. long-term project aims: an online formative assessment

system for the first six grades of primary school First phase: pilot the system in 200 schools study the related technological and methodological

issues in detail




US - many instances CBA and online assessment so wide spread - testing

industry - engaged in writing and developing task and different testing software

Despite of this in 2009 Microsoft, Cisco and Intel -transforming global educational assessment and improving learning outcomes

Teaching of 21st Century Skills project




Aim: resolve the gap - exists between what goes on in schools and what goes on in the real world

better measure the skills - needed for the 21st-century




Specify high-priority skills, competencies, - 21st century -turn these into measurable standards and assessment framework

Examine innovative learning environments and formative assessments - draw implications for ICT-based summative assessments

Identify methodological and technological barriers support solutions

implementation of these standards and methodologies, pilot test, make recommendations for broader educational assessment reform.



OECD PISA

Started to implement CBA 2006 CBAS 2009 - ERA 2012 - problem solving 2015 - completely to CBT still a lot of open questions



Despite of the advantages - in large-scale national and international assessment - faces several technological challenges

complexity of issues - no simple and clear solution Several technological means available in schools diversity, compatibility, connectivity require further

considerations



But the future is more promising



Thank you for your kind attention!http://www.staff.u-szeged.hu/~gymolnar


and remember:Without data, you are just another person with an Opinion (Schleicher, 2010)
Slide Number 1OutlineThe need for changeThe need for changeThe need for changeThe need for changeSlide Number 7Why is assessment so important?The feedback in educationPossibilities and promises of TBASlide Number 11Possibilities and promises of TBASlide Number 13Possibilities and promises of TBAAdaptive testingPossibilities and promises of TBAImplementing TBA in international projectsImplementing TBA in international projectsImplementing TBA in international projectsDeveloping Diagnostic Assessments Project Implementing TBA in international projectsTeaching of 21st Century Skills projectTeaching of 21st Century Skills projectOECD PISA Slide Number 25But the future is more promising http://www.staff.u-szeged.hu/~gymolnar

technology-based assessment: challenges and promises

Education

development of competencies

education quality

education ict

change technology

educationresearch group

future research group

change data

change possibilities