Game design and the gamification of content:The VirtualSign project

This work was supported by the Engineering Institute of Oporto and GILT (Graphics, Interaction and Learning Technologies) under grant No. 60973089, a project FCT (Fundacao para a Ciencia e Tecnologia).

1. Implementation of a bidirectional translation system for Portuguese Sign Language (LGP)

2. Development of a game focused primarily on educational and didactic aspects

Main Goals

Scope:• configuration and movement of the hands;• body position/inclination;• facial expressions;

Sign Language

Configuration and Movement of the Hands

Hand shape Palm orientation

Body Position

Facial Expression

System Components

Depth Sensor (Kinect) Data Gloves (5DT)

Translation through an avatar

Dynamic recognition of the Gestures

Sign Language Serious Game

Virtual Sign Application

1- Finite Automata

2- Algorithms for Hierarchical Classification

3- Sequence Alignment Algorithms

Game Play

1. The inventory script. 2. The graphical interface. 3. The map has objects, and those

objects contain scripts on them.4. The inventory consists of 42 spaces5. The handling of collisions, between

objects.

Score • Player’s score is incremented during

the game;• The shorter the time it takes between

the acquisition of two objects, the greater will be the score.

Gamification

1. Points: studies at the University of Chicago show that points are fantastic motivators and can be used to reward users/students across multiple levels or dimensions of a gamified activity.

2. Levels: these are often defined as point thresholds, so the students (or users) can use them to indicate a higher status and have access to bonus content.

Gamification

3. Challenges, badges, achievements, and trophies: the introduction of goals in an activity makes students (users) feel like they are working toward a goal.

4. Leader boards or “high-score table”: in the context of gamification, high-score tables are used to track and display desired actions, using completion to drive valued behavior. In VirtualSign score was based on objects acquired and achievement times.

VirtualSign Achievements

1. Gesture recognition using Kinect and Data Gloves2. Gestures classifiers3. Automatic Classification of Gestures4. 3D avatar to represent text in PSL5. Application chat6. Serious Game for PSL

Game Application Issues

1. The high development costs and an uncertain market make investment in educational games and innovations too risky for producers;

2. Institutions resist adopting innovations and do not want to make unnecessary changes and investments, including the use of new technologies for learning;

3. Instructors, institutions and publishers do not (traditionally) want to replace textbooks with educational games;

Game Application Issues

4. The value of specific educational technologies (games, simulations, etc..) have not been proven in many cases;

5. Parents and teachers still have very negative attitudes about the use of games in the classroom;

6. Games are especially suited to teach higher order skills that are not typically assessed through examinations (multitasking, decision-making, strategic vision, etc.);

7. Easy access to computers and the Internet cannot be taken for granted in many educational institutions.

Conclusions

• Success in the development of a system that allows the translation of textual content into sign language, and the other way round, based on the use of data gloves and the Kinect, to recognize the Portuguese alphabet using automatic classification.

• Connected to this, the development of a gamification framework and a game model aimed at teaching Portuguese Sign Language showed to be viable options in the design of a learning experience.

Video Presentation

1’ 35’’

VirtualSign