A Multisensory and Multimodal Game for Children with Cochlear Implants

Maria Carmela SogonoSupervisor: Dr. Deborah RichardsITEC810 Information Technology ProjectDepartment of Computing, Macquarie University

2

Agenda• Problem• Related work• Methodology• Outcomes– Design principles– Instructional design– Game evaluation

• Conclusion

3

ProblemEven with hearing aids, quality is still not the same as with normal hearing

Can affect brain development in children

Problems in areas such as sound localisation

Can a multimodal, multisensory game be used to test, train, and improve sound localisation acuity?

Photo source: http://www.flickr.com/photos/adwriter/1305191984/

4

Why focus on children?

• 35:10000 children• Normal hearing is important for brain

development

MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM

MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM

MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM

5

Why focus on SSD?

There is less help available for children with SSD.

Photo source: http://www.firefliesandjellybeans.com/2012/05/can-you-hear-me-now.html

6

What is sound localisation?The ability to identify where sound has come from

Photo source: http://www.flickr.com/photos/reverses/5098858054/

7

Baha

Processes sound from the impaired side and sends it to the hearing ear

Boneanchored hearing aid

Photo source: http://products.cochlearamericas.com/diagram/baha-system

8

Would it be possible to improve sound localisation acuity for children with SSD and Bahas?

9

How?GamesMultisensoryMultimodal

• To appeal to the target audience• To make it easier to track, collect,

then analyse data

• Supported by perceptual motor training theory

• Supported by technological advancements

10

Can we design games to improve sound localisation acuity for children with SSD and Bahas?

Would it be possible to improve sound localisation acuity for children with SSD and Bahas?

11

Relevant work• Research studies testing sound

localisation acuity• Existing sound localisation games

developed at Macquarie University• Other games developed for

children with SSD

Photo source: http://www.flickr.com/photos/driek/5412231453/

12

Methodology1. Review of relatedliterature

13

Methodology2. Analysis of existing work

14

Methodology3. Development of design principles

15

Methodology4. Development of instructional design template

16

Methodology5. Development of game evaluation guidelines

17

Design principles1. The learning environment shall be

realistic.2. The stimulus shall be meaningful.3. The game shall start at an easy level.4. When the game level of difficulty

increases, the increase shall be incremental.

5. The game shall incorporate motivational elements.

6. The game shall provide feedback after each trial.

18

7. The game shall propose a challenge.8. The game shall make use of fantasy.9. The game shall incorporate sensory and

cognitive curiosity.10.The game shall adapt to the user’s progress.11.The game shall allow for the repetition of

tasks.12.The game shall track user data.13.The game shall use bright colours.

Design principles

19

14.The game shall be visually appealing to the target audience.

15.The game shall make use of high-quality audio.

16.The game shall allow for sound to originate from at least 2 distinct sources.

17.A non-accented, trained voice shall be used.

18.The game shall make use of multiple modalities.

Design principles

20

Interaction modalitiesModality Description TechnologyVision Visual perception Monitor

TouchscreenProjector & screen3D goggles

Audio Hearing SpeakersHaptic Deals with the sense of touch;

currently only used to provide pressure

JoystickWand

Action, manipulation

Directed by limbs and body KeyboardMouseGamepadJoystickWii controllerKinectTouchscreen

21

Instructional design templateTitle <Title of the game module>Description <Describes the game in one or two sentences>Objectives To improve skills in localization acuity through training

To track user progress whilst trainingGame Style <Describes what game elements are incorporated>

Game genres: Puzzle (Hidden object, reveal the picture) Action (Maze game, rhythm action game, shooter games) Strategy (Time management) Adventure

Background <Describes the game’s backstory>Initial Setup <Describes how the game is set up in the beginning. What a new player will see,

hear, or experience after opening the game for the first time? >Game Model <Describes game progression. What will the player have to do? What happens

when the player answers correctly or incorrectly?> Four general models of game flow: Sound identification Sound evasion Sound tracking Sound training

22

Instructional design templateSuccess Condition

<Describes how the game or a level within the game is won>

Fail Condition

<Describes how the game or a level within the game is lost>

Challenges <Describes how the game levels increase in difficulty> There are several ways to increase difficulty: Include increasing instances of the stimulus Decrease angular distance between the instances Include background noises Include additional elements onscreen to increase distraction Decrease stimulus visibility (for example, it is hiding behind

another object) Include a time limit Include a “number of times” limit

Additional Interaction

<Describes possible interaction styles that are available if the player has access to additional hardware>

23

Sample game modules

Bug KillerTreasure HuntWild Animal EvasionDance Beat

Shooter, soundidentification

Adventure, sound tracking

Maze, sound evasion

Rhythm action, sound training

Game evaluation• Game design

• Usability

• Content

24

“Does the game follow design principles and best practices?”“Is the game usable?”

“Does the game teach what it’s supposed to?”

25

Game design evaluationModule Name: <Module name>Does the module have… Rating…a realistic learning environment? <1-5>Explanation: <Describe how the game satisfies or does not satisfy this condition.>

…meaningful stimulus? <1-5>Explanation:<Describe how the game satisfies or does not satisfy this condition.>

…motivational elements? <1-5>Explanation: <Describe how the game satisfies or does not satisfy this condition.>

…feedback? <1-5>Explanation: <Describe how the game satisfies or does not satisfy this condition.>

26

Game design evaluation…a challenge? <1-5>

Explanation: <Describe how the game satisfies or does not satisfy this condition.>

…elements of fantasy? <1-5>Explanation: <Describe how the game satisfies or does not satisfy this condition.>

…adaptation to the user’s progress? <1-5>Explanation: <Describe how the game satisfies or does not satisfy this condition.>

…repetition of tasks? <1-5>Explanation: <Describe how the game satisfies or does not satisfy this condition.>

...multiple modalities? <1-5>Explanation: <Describe how the game satisfies or does not satisfy this condition.>

27

Game modules evaluated

Bug KillerTreasure HuntWild Animal EvasionDance Beat

4 3.22

3.89

3.11

28

Conclusion

29

• Augmented reality• Multiplayer games• Haptic, large scale mats• Games for adults

Recommendations

30

Thank you!For questions, contact maria-carmela.sogono@students.mq.edu.au