Mohammad AL-Smadi, Christian Gütl

Institute for Information Systems and New Media (IICM)

Graz University of Technology, Austria

msmadi;cguetl@iicm.edu

Past, Present and Future of e-Past, Present and Future of e-Assessment: Assessment:

Towards a Flexible e-Assessment Towards a Flexible e-Assessment SystemSystem

OutlineOutlineMotivations

An Introduction to e-Assessment.

Overview about e-Assessment History.

e-Assessment Challenges and Difficulties.

A Proposal for an Enhanced e-Assessment System.

Conclusions.

Outlook.

2ICL2008 Conference: 24-26 Sep. Villach-Austria.

© AL-Smadi and Gütl 2008

Background & Motivations 1Background & Motivations 1• Gütl WG: Technology-Enhanced Learning• Skill & Competence-based LearningSkill & Competence-based Learning• Skill detection/assignment (Nussbaumer et al., 2007)

• Content-based: knowledge and action component• Social metadata creation (Web 2.0 technologies)• User behavior (attention metadata)

• E-assessment (Gütl, 2007)

• Automatic knowledge & skill assessment and feedback provision• Content-based: statistical & NLP• Different types of Assessment

• Skill-based adaptive e-learning (Nussbaumer et al., 2007b)

• Learning content• Learning path• Assessment

AL-Smadi and Gütl 2008ICL2008 Conference: 24-26 Sep. Villach-Austria.3

Background & Motivations 2Background & Motivations 2The increase in the staff workload:

Increasing number of students supervised by the same staff.

Reducing time and effort.Appropriate assessment information and effective

decisions.(Dietal et al, 1991).The increase of pressure to provide assessments that

are fair, reliable and effective. (Brown et al, 1996)

AL-Smadi and Gütl 2008ICL2008 Conference: 24-26 Sep. Villach-Austria.4

e-Assessment Strategies and Typese-Assessment Strategies and TypesAssessment strategies (Bransford et al, 2000):

Formative Assessment: Part of the learning process, feedback provision.

Summative Assessment: At the end of a learning activity, judging and discriminating between

students.E-Assessment types (Charman & Elms, 1998):

Computer Based Assessment: Test delivery and feedback provision are done by the computer.Computer Assisted Assessment:More general, the whole assessment process involving test delivery,

test marking, grading, analysis, reporting and feedback provision are handled by the computer

5ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

Historical Overview of e-Assessment 1Historical Overview of e-Assessment 11960’s when:

PLATO (Programmed Logic for Automatic Teaching Operations) project has

been started at the University of Illinois.TICCIT

(Time-Shared, Interactive, Computer-Controlled, Information Television),which has been started in 1967.

The “Automatic Grader” (Hollingsworth,1960).The use of computers to automatically assess the students’

programming assignments .1980’s and the effect of micro-computers (Reiser,2001):

An increasing interest of using computers in instruction. Assessment Systems in other fields appeared after soon:

Mathematics (Rottmann &Hudson,1983) Chemistry (Myers,1986).

6ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

Historical Overview of e-Assessment 2Historical Overview of e-Assessment 2

ICL2008 Conference: 24-26 Sep. Villach-Austria.7

Since 1990’s and afterwards, the impact of the World Wide Web (WWW). Blackboard.com

provides automatic grading of multiple choice and True/False questions.

Systems such as QUIZIT (Tinoco et al,1997) , WebCT.com , ASSYST (Jackson &Usher,1997) and PILOT (Bridgeman et al, 2000) are also examples of web-based systems with ability of online testing and grading.

e-Examiner (Guetl, 2007): A tool to support the assessment process by automatically

generating test items for open-ended responses, marking students’ short free-text answers and providing feedback.

AL-Smadi and Gütl 2008

E-Assessment Challenges in Modern E-Assessment Challenges in Modern Learning Settings 1Learning Settings 1Digital Mimes

Imitating the conventional assessment. limitation in exercises types. (Elliot, 2008)

Assuming that the students have to retain the context related information in their memories.

To overcome this problem:Open-web examinations (Ridgway et al, 2004).Modernizing e-assessment systems to be influenced by the

abilities of Web 2.0 (Elliot, 2008).

8ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

E-Assessment Challenges in Modern E-Assessment Challenges in Modern Learning Settings 2Learning Settings 2Mentality Change & Culture Evolution: (Prensky, 2001)

Our educational systems:The rapid change in our culture The use of technology in our modern life activities.

Multi-tasked students & Outdated educational systems: Digital Natives and Digital Immigrants. The gap between students and our educational systems.

To overcome this problem:Teachers have to learn the new language and style of their students.Content should be changed.New methods of learning like Game-based learning should be used.

9ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

E-Assessment Challenges in Modern E-Assessment Challenges in Modern Learning Settings 3Learning Settings 3Security & Privacy:

Preventing CheatingSeveral solutions (Barker & Lee, 2007):

The simplest and cheapest of using passwords The more complex and expensive techniques of adding

physical instruments to the assessment system such as, Biometric authentication and using video conferencing setups.

Maintaining users’ privacy:Protect specific users’ data not to be accessed by other users.Providing adequate information of students’ performance to

teachers while keeping specific details secret.

10ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

E-Assessment Challenges in Modern E-Assessment Challenges in Modern Learning Settings 4Learning Settings 4Assessment & Feedback as a Mean to Learn:

Encourage students to further progress and learn. (Dochy &

McDowell, 1997):Portfolio-assessment, self-assessment and peer-assessment.

Feedback as a mean to learn (Klassen, 2001):Spirit of formative assessments .Achieving objectives.A mirror for learning.

Technology has a major influence on feedback:Computers can (Charman & Elms, 1998):

Facilitate tracking user’s behavior.Provide immediate or timely feedback.

11ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

E-Assessment Challenges in Modern E-Assessment Challenges in Modern Learning Settings 5Learning Settings 5Standards & Specifications:

Why Standards? Interoperability.Reusability. Manageability.Accessibility. Durability.

Many organizations and consortia:Dublin Core Metadata Initiative (DC).The Institute of Educational and Electronics Engineers (IEEE).The Instructional Management System Global Learning Consortium

(IMS GLC).The Advanced Distributed Learning Initiative (ADL).

E-assessment systems should be designed to be flexible and support most of these standards.

12ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

E-Assessment Challenges in Modern E-Assessment Challenges in Modern Learning Settings 6Learning Settings 6E-assessment Automation:

Most of the developed e-assessment tools are related to specific part(s) of the assessment cycle or limited to some type(s) of assessment.

e-assessment tools should:Support the entire lifecycle of assessment by (semi-) automatic

methods. Be designed based on the learning goals.

13ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

Overall Conceptual ArchitectureOverall Conceptual Architecture

14ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

Conclusions 1Conclusions 1Reducing time and efforts spent on students’ assessment is

a rationale to use e-assessment technology. There is an increasing pressure on teachers in further and

higher education to provide assessments that are fair, reliable, efficient and effective.

Appropriate assessment information provides an accurate measure of student performance that enable students, teachers, administrators and other key stakeholders to make effective decisions.

Because of the gap between the students and the educational systems form one side and the gap between the students and the educators from the other side, the current applied educational systems are not appropriate anymore.

15ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

Conclusions 2Conclusions 2Assessment is not only used for measuring and judging

students work. Furthermore it is a mean for learning, where different types of assessment such as self and peer assessment can be used.

Feedback has been considered to be a mirror for learning where both students and educators can use this mirror to see what they have done or have not during courses.

The more standards the e-assessment system supports the more flexible it would be.

E-assessment tools should support the entire lifecycle of assessment by (semi-) automatic methods.

Based on this review, we have introduced an abstract level for a flexible e-assessment system .

16ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

OutlookOutlookBased on the first conceptual architecture, we will start

an in-deep requirements analysis and will start the first phase of a prototype implementation.

17ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008

Thank You for Your Attention!Thank You for Your Attention!Questions are Welcome.Questions are Welcome.

Mohammad AL-Smadi, Christian Gütl

Institute for Information Systems and New Media (IICM)

Graz University of Technology, Austria

msmadi;cguetl@iicm.edu

18ICL2008 Conference: 24-26 Sep. Villach-Austria. AL-Smadi and Gütl 2008