a comparison of online and paper-based test results sanja Čandrlić, sanjac@inf.uniri.hr...

A comparison of online and paper-based test results

Sanja Čandrlić, sanjac@inf.uniri.hr

Martina Ašenbrener Katić, masenbrener@inf.uniri.hr

Presentation outline

• Introduction• Research and testing environment• Research results• Conclusion• Future work

214th Workshop SE Education and Reverse EngineeringSinaia

Introduction

• Comparative research of paper-based testing vs. online testing• ICT more and more used, also in education

– Many advantages– Knowledge assesment as an important part of educational process

• The aim of the research: – Can online tests replace traditional paper based tests?– What types of questions should online test contain?

• Online test

314th Workshop SE Education and Reverse EngineeringSinaia

Online testing

• Reasons to move to online testing?– Automated correction -> more students, less time, the computer program

evaluates the tests (for some questions)

• Objective & subjective question types– True/false– Multiple choices – Connections– Essay– Short answer

414th Workshop SE Education and Reverse EngineeringSinaia

Question types

514th Workshop SE Education and Reverse EngineeringSinaia

Research environment

• - a customized Moodle solution, adapted to the needs of UniRi

• 1231 tests included in this research • 3 courses in the field of information systems design:

– Data Modeling, – Process Modeling – Information Systems

614th Workshop SE Education and Reverse EngineeringSinaia

Research environment

• Online testing since 2009– first online tests included a certain percentage of subjective questions– Data Modeling, Process Modeling

• Online testing since 2011 – intention to ease the correction – online tests with only objective questions – Information Systems course– the number of students and teachers' teaching load increased

• Can online tests consisting exclusively of objective questions completely replace paper-based tests?

714th Workshop SE Education and Reverse EngineeringSinaia

Test properties and participants

• 15 – 25 questions from different categories in online tests and paper-based tests

• Time limit to solve the tests, both for online and paper-based tests• Shuffled questions in online tests• Online tests set up to be solved in a ‘safe window’

• Single major and Double major program of informatics

8

14th Workshop SE Education and Reverse EngineeringSinaia

Tests set up

9

14th Workshop SE Education and Reverse EngineeringSinaia

 Paper-based tests,

all 3 coursesOnline tests

DM, PMOnline tests

IS

Question types

essay, short essay,

listings, image drawing,

short answer questions

multiple choice,true/false,

connection, essay,

short answer questions

multiple choice,true/false, connection

Duration 60 minutes 20 minutes 20 minutes

Nr of questions 15 (on average) 20 (on average) 25 (on average)

Nr of tests 753 310 168

Questions and points set up

10

14th Workshop SE Education and Reverse EngineeringSinaia

Paper-based tests DM & PM online tests IS online tests

Essay Objective questions Objective questions

Short essay Multiple answer Multiple answer

Listings True/false True/false

Drawing Connection Connection

Short answer Subjective questions

Essay

Short Answer

50% points

50% points

70%

30%

100%

Main hypothesis

Descriptive statistical analysis:• High variance coefficient for paper-based test results – arithmetic mean

cannot be considered as representative central tendency indicator• Low value of quartile deviation coefficient – median is a better indicator

of central tendency

H: There is no significant difference in median values of the results achieved during MudRi online tests and traditional paper-based tests

1114th Workshop SE Education and Reverse EngineeringSinaia

Test results

1214th Workshop SE Education and Reverse EngineeringSinaia

1. Box-plot diagrams for Data Modeling course (left online tests, right paper-based tests)

.2.4

.6.8

1

Box-plot diagram for DM course

(left online, right paper-based)

0.2

.4.6

.81

Box-plot diagram for PM course

(left online, right paper-based)

Box-plot diagram for IS course

(left online, right paper-based)

1. Box-plot diagrams for Information Systems course (left online tests, right paper-based tests)

0.2

.4.6

.81

Research results

• Normality analysis indicated the use of nonparametric statistic methods for hypothesis testing

Mann-Whitney test: • Data Modeling course – no significant statistic difference in median

values (p=0.2401) between online and paper-based tests. • Process Modeling course – no significant statistic difference in median

values (p= 0.3777) between online and paper-based tests. • Information Systems course – the difference is extremely significant

(p=0.0066).1314th Workshop SE Education and Reverse EngineeringSinaia

Conclusion

• For DM & PM the H is confirmed• For IS the H is rejected

• IS – better results -> lower levels of knowledge, based on recognition• Paper-based tests:

– higher max points -> subjective factor, partially correct, “hidden knowledge”– min points = 0 - > students give up

1414th Workshop SE Education and Reverse EngineeringSinaia

Conclusion (2)• Transition from paper-based to online tests: Essay-> objective q.• To explain, to describe, ... 30 % of subjective q., manual correction• Advantages of online tests:

– automated correction, – easy to use (after forming the question bank)– immediate test results for the students– students prefer to use the keyboard– comparative overview of test results (for the teacher)

• Advantages of paper-based tests: – easy to prepare– „drawing” questions

1514th Workshop SE Education and Reverse EngineeringSinaia

Future work

• Efforts to further improve methods for online assessment• Revising the question bank• Analysis of students’ answers• Too easy questions (100% correct)• Too hard questions (potentially unclearly stated, incomprehensible to

the students)• Plan to update the question bank with sets of different types of

questions for formative assessment – to test their knowledge during preparation for the tests

1614th Workshop SE Education and Reverse EngineeringSinaia

• Thank you for your attention!

• Questions?

1714th Workshop SE Education and Reverse EngineeringSinaia