a comparison of online and paper-based test results sanja Čandrlić, [email protected]...
TRANSCRIPT
A comparison of online and paper-based test results
Sanja Čandrlić, [email protected]
Martina Ašenbrener Katić, [email protected]
Presentation outline
• Introduction• Research and testing environment• Research results• Conclusion• Future work
214th Workshop SE Education and Reverse EngineeringSinaia
Introduction
• Comparative research of paper-based testing vs. online testing• ICT more and more used, also in education
– Many advantages– Knowledge assesment as an important part of educational process
• The aim of the research: – Can online tests replace traditional paper based tests?– What types of questions should online test contain?
• Online test
314th Workshop SE Education and Reverse EngineeringSinaia
Online testing
• Reasons to move to online testing?– Automated correction -> more students, less time, the computer program
evaluates the tests (for some questions)
• Objective & subjective question types– True/false– Multiple choices – Connections– Essay– Short answer
414th Workshop SE Education and Reverse EngineeringSinaia
Question types
514th Workshop SE Education and Reverse EngineeringSinaia
Research environment
• - a customized Moodle solution, adapted to the needs of UniRi
• 1231 tests included in this research • 3 courses in the field of information systems design:
– Data Modeling, – Process Modeling – Information Systems
614th Workshop SE Education and Reverse EngineeringSinaia
Research environment
• Online testing since 2009– first online tests included a certain percentage of subjective questions– Data Modeling, Process Modeling
• Online testing since 2011 – intention to ease the correction – online tests with only objective questions – Information Systems course– the number of students and teachers' teaching load increased
• Can online tests consisting exclusively of objective questions completely replace paper-based tests?
714th Workshop SE Education and Reverse EngineeringSinaia
Test properties and participants
• 15 – 25 questions from different categories in online tests and paper-based tests
• Time limit to solve the tests, both for online and paper-based tests• Shuffled questions in online tests• Online tests set up to be solved in a ‘safe window’
• Single major and Double major program of informatics
8
14th Workshop SE Education and Reverse EngineeringSinaia
Tests set up
9
14th Workshop SE Education and Reverse EngineeringSinaia
Paper-based tests,
all 3 coursesOnline tests
DM, PMOnline tests
IS
Question types
essay, short essay,
listings, image drawing,
short answer questions
multiple choice,true/false,
connection, essay,
short answer questions
multiple choice,true/false, connection
Duration 60 minutes 20 minutes 20 minutes
Nr of questions 15 (on average) 20 (on average) 25 (on average)
Nr of tests 753 310 168
Questions and points set up
10
14th Workshop SE Education and Reverse EngineeringSinaia
Paper-based tests DM & PM online tests IS online tests
Essay Objective questions Objective questions
Short essay Multiple answer Multiple answer
Listings True/false True/false
Drawing Connection Connection
Short answer Subjective questions
Essay
Short Answer
50% points
50% points
70%
30%
100%
Main hypothesis
Descriptive statistical analysis:• High variance coefficient for paper-based test results – arithmetic mean
cannot be considered as representative central tendency indicator• Low value of quartile deviation coefficient – median is a better indicator
of central tendency
H: There is no significant difference in median values of the results achieved during MudRi online tests and traditional paper-based tests
1114th Workshop SE Education and Reverse EngineeringSinaia
Test results
1214th Workshop SE Education and Reverse EngineeringSinaia
1. Box-plot diagrams for Data Modeling course (left online tests, right paper-based tests)
.2.4
.6.8
1
Box-plot diagram for DM course
(left online, right paper-based)
0.2
.4.6
.81
Box-plot diagram for PM course
(left online, right paper-based)
Box-plot diagram for IS course
(left online, right paper-based)
1. Box-plot diagrams for Information Systems course (left online tests, right paper-based tests)
0.2
.4.6
.81
Research results
• Normality analysis indicated the use of nonparametric statistic methods for hypothesis testing
Mann-Whitney test: • Data Modeling course – no significant statistic difference in median
values (p=0.2401) between online and paper-based tests. • Process Modeling course – no significant statistic difference in median
values (p= 0.3777) between online and paper-based tests. • Information Systems course – the difference is extremely significant
(p=0.0066).1314th Workshop SE Education and Reverse EngineeringSinaia
Conclusion
• For DM & PM the H is confirmed• For IS the H is rejected
• IS – better results -> lower levels of knowledge, based on recognition• Paper-based tests:
– higher max points -> subjective factor, partially correct, “hidden knowledge”– min points = 0 - > students give up
1414th Workshop SE Education and Reverse EngineeringSinaia
Conclusion (2)• Transition from paper-based to online tests: Essay-> objective q.• To explain, to describe, ... 30 % of subjective q., manual correction• Advantages of online tests:
– automated correction, – easy to use (after forming the question bank)– immediate test results for the students– students prefer to use the keyboard– comparative overview of test results (for the teacher)
• Advantages of paper-based tests: – easy to prepare– „drawing” questions
1514th Workshop SE Education and Reverse EngineeringSinaia
Future work
• Efforts to further improve methods for online assessment• Revising the question bank• Analysis of students’ answers• Too easy questions (100% correct)• Too hard questions (potentially unclearly stated, incomprehensible to
the students)• Plan to update the question bank with sets of different types of
questions for formative assessment – to test their knowledge during preparation for the tests
1614th Workshop SE Education and Reverse EngineeringSinaia
• Thank you for your attention!
• Questions?
1714th Workshop SE Education and Reverse EngineeringSinaia