Technology and Health Care 21 (2013) 97–111 97DOI 10.3233/THC-120712IOS Press

Duckneglect: Video-games based neglectrehabilitation

R. Mainettia, A. Seddab, M. Ronchettia, G. Bottinib,c and N.A. Borghesea,∗aDepartment of Computer Science, Università degli Studi di Milano, Milano, ItalybPsychology Department, University of Pavia, Pavia, ItalycCognitive Neuropsychology Center, Niguarda Ca’ Granda Hospital, Milano, Italy

Received 30 August 2012Accepted 26 October 2012

Abstract.BACKGROUND: Video-games are becoming a common tool to guide patients through rehabilitation because of their power ofmotivating and engaging their users. Video-games may also be integrated into an infrastructure that allows patients, dischargedfrom the hospital, to continue intensive rehabilitation at home under remote monitoring by the hospital itself, as suggested bythe recently funded Rewire project.OBJECTIVE: Goal of this work is to describe a novel low cost platform, based on video-games, targeted to neglect rehabili-tation.METHODS: The patient is guided to explore his neglected hemispace by a set of specifically designed games that ask him toreach targets, with an increasing level of difficulties. Visual and auditory cues helped the patient in the task and are progressivelyremoved. A controlled randomization of scenarios, targets and distractors, a balanced reward system and music played in thebackground, all contribute to make rehabilitation more attractive, thus enabling intensive prolonged treatment.RESULTS: Results from our first patient, who underwent rehabilitation for half an hour, for five days a week for one month,showed on one side a very positive attitude of the patient towards the platform for the whole period, on the other side a signif-icant improvement was obtained. Importantly, this amelioration was confirmed at a follow up evaluation five months after thelast rehabilitation session and generalized to everyday life activities.CONCLUSIONS: Such a system could well be integrated into a home based rehabilitation system.

Keywords: Adaptive video-games, neglect rehabilitation, video processing, monitoring, space exploration, mirror images, bodyscheme

1. Introduction

Visuospatial neglect is usually associated to right brain damage and it is characterized by the inabilityto interact with the stimuli located in the contra-lesional side of space [1,31,38]. Post-stroke recovery isnegatively affected by the presence of visuospatial neglect: patients with this diagnosis are more impairedin terms of independence; they get lower scores on disability tests and require longer rehabilitationperiods [20,32]. This may be due to the multi-componential character of the neglect syndrome, as it mayaffect various domains, such as perception and mental representation in multiple sensory modalities.

∗Corresponding author: N. Alberto Borghese, Department of Computer Science, Università degli Studi di Milano, Milano,Italy. Tel.: +39 02 503 1632; E-mail: alberto.borghese@unimi.it.

0928-7329/13/$27.50 c© 2013 – IOS Press and the authors. All rights reserved

AUTH

OR

COPY

98 R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation

Neglect influences most everyday life activities such as eating, reading and getting dressed [1,24]. Inthe light of these functional implications and of the neglect syndrome frequency, ranging from 13% to18% of patients who have had right hemisphere stroke [11], it is not surprising that the development ofefficient rehabilitation techniques is an important aim of the present research on neglect.

Since now, however, none of the systems targeted to neglect rehabilitation has shown optimal resultsin terms of applicability, long term outcome and impact on daily living [9,14,33,36]. Neglect has beentreated by means of pharmacological intervention, different physiological sensory stimulation (vestibu-lar, optokinetic, electro-nervous stimulation, transcutaneous mechanical vibration, prism adaptation), orcognitive-behavioral training. However, these techniques have shown several limitations. Pharmacolog-ical treatment may induce side-effects [4] while physiological stimulation has shown no evidence of anyprolonged recovery from neglect [33]. A partial exception is represented by prism adaptation that how-ever it is still at an experimental level [30]. Positive effects have been observed in traditional behavioraland cognitive training. These usually employ paper and pencil tasks or their computerized equivalent,but amelioration is task specific and it tends not to be transferred to daily life activities [9].

Virtual Reality (VR) has more recently been proposed as an effective alternative to traditional behav-ioral and cognitive rehabilitation as it can integrate, with high flexibility, motor and cognitive training ina daily life scenario [14,34,40]. In particular, the work of Sun and co-workers [29] used a street cross-ing paradigm for rehabilitation and was aimed at creating ecological VR situations that would promptthe patient to explore his left hemispace. However, the treated patients lost their ability to consider theleft part of the space soon or later after rehabilitation and continuous follow-up sessions are possiblyrequired for maintaining recovery. Moreover, classical VR systems, based on head mounted displays(HMD) and gloves have high costs [41,47] and are invasive for the patient. HMD, moreover, restrictthe visual field and create a conflict between vergence and accommodation [48]. For these reasons VRrehabilitation platforms have been mainly restricted to hospitals and laboratories prototypes.

The game industry has recently offered a set of powerful, yet cheap, instruments to reliably track hu-man movement: Nintendo Wii remote and balance board,1 Sony Playstation Eye-cam and Move,2 andMicrosoft Kinect3 have revolutionized the way people interact with a graphical scenario and these de-vices have rapidly become a major source of inspiration for the researchers working in rehabilitationwho immediately recognized the potential of these new technologies (see [8] for a recent review). More-over, such technology could make rehabilitation at home a real option as being explored by recentlyfunded EU projects Rewire4 and Cupid.5 Video games in fact, appear a viable solution to support in-tensive training as it allows linking improvement with a reward mechanism represented by incrementalsuccess at some task or goal. Therefore a computerized game-like method constitutes a powerful tool toprovide participant repetitive practice, feedback about performance and motivation to endure practice. Inaddition, it can be adapted based on an individual participant’s baseline motor or cognitive performanceand be progressively augmented in task difficulty. The use of gaming elements can also be used to takepatients’ attention away from any pain resulting from their injury or movement. This occurs the more apatient feels involved in an activity and again, allows a higher level of participation in the activity, as thepatient is focused on achieving goals within the game [12,41].

1http://www.nintendo.com.2http://us.playstation.com/ps3/playstation-move/.3http://www.xbox.com/KINECT.4http://www.rewire-project.eu/.5http://www.cupid-project.eu/.

AUTH

OR

COPY

R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation 99

Fig. 1. Four of the environments used in the Duckneglect platform: a) mountain, b) and d) cupboard with different objects, andc) seaside. The targets are (in order): cherries without leaves, coca cola cans, cups with left handle and white seashells. Thecorrespondent distractors are: cherries with leaves, sprite cans, cups with right handle and yellow seashells. In panel (a) depictsthe user while performing the exercise. The ‘

√’ green sign indicates that the subject did grasp the object correctly. The yellow

trajectory indicates the hand movement. This trajectory is visible only to the experimenter in the analysis carried out off-lineand not to the patient. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/THC-120712)

We show here how using a platform constituted of a web-cam, a PC, a TV display or a video projector,adequate video-games and tracking software, a reliable rehabilitation system can be obtained. The pa-tient is guided to explore his left-hemispace by a set of specifically designed games that ask the patientto reach targets, with an increasing level of difficulties and a decreasing set of cues (both visual andauditory). A balanced reward system and music played in the background make the game even moreattractive. We have termed this platform Duckneglect from one of the elements used inside one of thegames used for training. Results on our first patient show that he did recover from neglect using theparadigm presented here and that he maintained such recovery also at the follow up five months afterthe last rehabilitation session. Moreover, he was always willing to start rehabilitation sessions for all therehabilitation period of one month, that reinforce the idea that game based rehabilitation can be indeeda good match for intensive and prolonged rehabilitation.

2. Materials and methods

Duckneglect is aimed at guiding the patients through visual search tasks: the patient is instructed tohit with the hand controlateral to the neglected space one or more objects belonging to a certain class(targets) that appear on the screen inside a virtual scenario and to avoid distractors (Fig. 1).

2.1. The Duckneglect architecture

The setup of Duckneglect is shown in Fig. 2a. The patient comfortably sits on his wheelchair with awhite wall behind him. He watches a projection screen positioned at 1.5 m in front of him. A virtual

AUTH

OR

COPY

100 R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation

Fig. 2. The Duckneglect set-up is shown in panel (a). The patient sits in his wheelchair (A) with a white curtain behind him (B).A projection screen of 2 × 2 m placed at 1.5 m from him (C). A screen projector (D) is connected to a host PC (E). A webcam(F) surveys the scene. The virtual environment is projected on the screen on an area of about 1.8 m by 1.2 m, the dimensionof the targets and of the distractors is about 180 mm × 250 mm, with a visual angle of approximately 8 degrees. The set-up iscompleted by two loud speakers at the patient’s side, not shown in the picture. (Colours are visible in the online version of thearticle; http://dx.doi.org/10.3233/THC-120712)

scene is projected on the screen by a projector driven by a host PC. A web cam, connected to the samePC, is positioned frontally aimed at the patient’s face. The camera used here is a PS3 Eye-camera fromSony with 640 × 480 pixels, RGB color coding, and a rate of 30 frames per second (fps). Two loudspeakers are positioned at the patient’s side to provide spatialized sound.

The PC runs the Duckneglect rehabilitation software that loops through three main steps. First, thevideo stream is acquired from the camera and it is processed in real-time to extract the patient silhou-ette from the background. The silhouette is then pasted over the actual virtual scene inside which thepatient does the current rehabilitation exercises. Lastly the composite image is displayed (Fig. 1a). Thiscycle runs at 30 frames per second guaranteeing that the patient sees himself moving inside the virtualenvironment with a fluid motion.

2.2. The game play

Before rehabilitation can start, calibration has to be carried out. This has two goals: enabling a robustseparation of the patient silhouette from the background and guide patient into the desired position inwhich he will do the exercises. Calibration is obtained in four steps.

In the first step a set of forty background images is acquired without the patient inside the field of viewof the camera. The RGB images are converted into gray level images, the median value is computed foreach pixel, p(x,y), to obtain a single calibration image, Ical(x,y).

The patient is then asked to step in and to position himself such that his silhouette matches the darksilhouette shown on the screen (Fig. 3a). In this phase, the patient silhouette is extracted as the sub-setof pixels in the incoming image, I(x,y) that have a large difference in their gray level with respect to thevalue that they had in the background image:

{x, y} : |I(x, y) − Ical(x, y)| > Thdef (1)

where Thdef is set to a very conservative value, here equal to 20 gray levels. If the patient’s head istoo high or too low with respect to the position in the dark silhouette, the patient’s chair is regulated

AUTH

OR

COPY

R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation 101

Fig. 3. (a) The dark silhouette that guides the patient in the correct place with superimposed the current patient silhouette. Thissame display allows verifying the capability of the patient to explore the game space. (b) The patient silhouette is pasted insidethe virtual rehabilitation scenario. Notice that the patient sees himself mirrored inside the screen according to the set-up inFig. 1. In this screenshot the patient is touching a stylized cow, partially hidden by his hand, that represents one of the possibletargets of this particular game. Please notice that the background is the same of Fig. 1a. This demonstrates the possibility tocreate different scenarios changing the game elements. The choice of the game elements is carried out here through controlledrandomization. The region outside which collisions are controlled is highlighted as a dashed black line. The red circle representsthe patient center from which the hand position is searched (see text). (Colours are visible in the online version of the article;http://dx.doi.org/10.3233/THC-120712)

accordingly. If his silhouette appears smaller or larger than the dark one, his chair is moved forwards ofbackwards accordingly.

Once the patient is put in place, the threshold is refined. For each pixel a foreground average image,Ifor(x,y) is computed averaging forty frames in which the patient is present inside the scene. For allthe pixels for which: | Ifor(x,y) –Ical(x,y) | > 10 gray levels, the optimal threshold, th*, that separatesthe background from the patient foreground is computed using Otsu thresholding [25]. As this mightnot always produce the best results, the threshold can be further adjusted through a slider until robustsilhouette detection is obtained. This manual adjustment was rarely required.

Finally, the patient is asked to reach the borders of the image and, in case, the position of his chairadjusted. In case of hemipleagia, that often occurs in neglect patients, only the not paretic arm can beused. The overall calibration procedure requires few minutes to be completed.

After calibration, the rehabilitation games can start. Each RGB image of the video stream is firstconverted into a gray level image and thresholded with th* to obtain a binary image. This is associatedto the alpha channel of the acquired image and works as a binary transparency mask [22]. Therefore apixel of the screen will have the value of the acquired image when the corresponding mask bit is equalto one (pixel belonging to the patient silhouette) or the value of the virtual scene when the mask bit isequal to zero: the RGB value of the pixels belonging to the virtual scene are overwritten with those ofthe silhouette creating the composite image in Fig. 3b.

Collision detection is a critical task. To determine when the patient correctly hits a target, we have notonly to identify the collision between the patient’s hand and the target, but also to evaluate if this collisionis meaningful. To this aim, we check that the patient’s hand overlaps significantly with the target andthat this overlap lasts significantly over time. To this aim, we define a circular collision area centeredin each target of radius 20 pixels (targets have size 64 × 64 pixels). The definition of a collision areaassociated to the patient’s hand is more challenging as the hand is not a static item. We first determine thecurrent position of the fingers in the acquired image as the foreground pixel most distant from the patientcenter identified on the silhouette (the red circle in Fig. 3b). A circular collision region, centered in such

AUTH

OR

COPY

102 R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation

Fig. 4. The collision detection mechanism implemented in Duckneglect. The degree of overlap between the hand collisionregion and the target collision region determines the occurrence of the collision. In this case, a true collision is detected only inthe upper right condition. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/THC-120712)

identified finger position with a radius of 10 pixels is defined. Because of the construction mechanism,such circle moves solid with the patient’s hand. A collision is detected when the fingers collision regionis entirely contained inside the target collision region and the overlap lasts at least 5 frames (150 ms)(Fig. 4). To make collision detection even more robust, we first observe that it can be simplified if weconsider that collision with a target occurs inside a region that does not overlap with the patient’s upperbody. Accordingly, a search region, with a hemi-ellipse shape, anchored to the calibration silhouette ofthe patient’s head and trunk is defined (Fig. 3b), and collisions are searched only outside this area.

The whole software application has been developed under OpenSource paradigm to maximize soft-ware diffusion. OpenCV library6 has been used for image processing, SFML7 for graphics and FoxToolkit8 for the Graphical User Interfaces (GUI) that guide through calibration and gaming.

2.3. The game design

A strong effort has been put in defining a set of games that were simple but at the same time enter-taining and effective for neglect rehabilitation. We have here identified simple “grasping” games as themost adequate, as they allow varying the scenarios and the targets in a very controlled way. A set ofdifferent scenarios have been designed that recall pleasant environments that are usually related to nicetimes in everyday life (Fig. 1). During rehabilitation, music is played according to the actual scenario,chosen soft and calm, inside the repertoire of new age music. We used virtual objects (both targets anddistractors) that can be experienced in everyday life, such as cans on the shelves of a house kitchen orseashells on the sand. Distractors are required to make the game more entertaining and force the patientto focalize his attention on the objects to be reached.

The mini-games progress through three hierarchical stages: in the first stage, both visual and auditorycues are provided to the patient during rehabilitation. Visual cues are constituted of blinking of the target,while auditory cues are represented by spatialized sound: the sound intensity from the two speakersdepends on the azimuth of the target on the screen with respect to the subject. Both cues are activated

6http://opencv.org/.7http://www.sfml-dev.org/.8http://www.fox-toolkit.org/.

AUTH

OR

COPY

R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation 103

when no target or distractor is hit after 10s. This means that the patient is not performing any activityor he is not able to find the target. In the second stage only visual cues are provided while in the thirdstage no cues are given. Each stage has a set of difficulty levels that depend on the number of targetsand distractors. At the first level we have one target and no distractors, at second level one target andfour distractors, at the third level one target and six distractors, at the fourth level, three targets and sixdistractors and at the last level six target and six distractors. The patient is admitted to the next stagewhen at least 90% of the targets of all the levels of the current stage have been hit. A new environmentis chosen randomly among the ones available for this new stage and the game starts again. When a levelhas been completed, the program gives a feedback to the patient, writing on the screen “Bravo!” (Good!)and displays a score that is inversely proportional to the total amount of time required to reach the targetsfor that level.

Particular care has been put in the rewarding mechanism. When the patient hits a target, a green ‘√

’symbol is displayed besides the target (Fig. 1a) and a nice beep is played. After 200 ms the target and the‘√

’ symbol disappear from the screen and points are added to the patient’s score. When the patient hitsa distractor, a red cross is displayed besides the distractor hit for 200 ms as a warning for having made awrong action. Afterwards the red cross disappears but the distractor remains in its place on the screen.No negative points are attributed to wrong actions as the primary goal here is to always encourage thepatient in going on with rehabilitation.

Once in a time a surprise effect is produced [42]: when the patient hits the target, instead of receivingthe usual positive reward and seeing the ‘

√’ symbol displayed on the screen, an object, different from

the targets and the distractors, appears on the screen. This was a balloon or a bird flying towards the skyin the games implemented here and gives a score bonus to the patient.

To avoid that the game becomes boring and that the patient over-learns a given behaviour, two differentmechanisms have been developed. The first mechanism is the randomization of the scenes. A singlerehabilitation session is constituted of a sequence of games, each game uses a different virtual scenethat is picked randomly among those available. The second mechanism is related to the position of thetargets and the distractors. These will never be in the same position on the scene. We have designed a4x4 matrix for the left and right region and defined which of these positions are reachable (they do notoverlap with the patient silhouette): ten valid positions on the right side of the patient and ten on his left.The targets and the distractors are randomly assigned to these positions according to a uniform randomdistribution, checking that the same number of targets and distractors is going to be displayed in eachposition at the end of each game, such guaranteeing a balanced set of stimuli. In this way, the patientin confronted each time with a different virtual environment and a different distribution of the targetmaking the game more attractive on one side and on the other side this avoid learning the position of thetargets and the distractors, transforming the task from visual search to a memory task.

We have relied extensively on scripting language to define the game, in terms of scenes, targets,distractors, scoring and reward system, such that modifications can be obtained by simply modifying atext (XML) file.

3. Results

Duckneglect has been tested on a first neglect patient, IB, a 65 years old right-handed male (18 yearsof education), who suffered from a right fronto-temporal intraparenchymal haemorrhagic lesion in 2009.IB has been diagnosed with chronic neglect at each neuropsychological evaluation he underwent since2009. Moreover, he showed a severe hemiplegia for the left side of the body. He underwent a session of

AUTH

OR

COPY

104 R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation

Fig. 5. Results on the Albert cancellation test[2] before (a) and after Duckneglect treatment. In this test, uncrossed lines on theleft side of the sheet are an index of neglect.

classical cognitive rehabilitation with no improvement either in neuropsychological tests or in everydaylife activities. The patient was then referred to the Neuropsychological Center of Niguarda hospital inNovember 2010, one year after the lesion. The diagnosis of chronic neglect was here confirmed by meansof the classical tests for peripersonal space neglect: the Albert Test, the Diller Test and the Line bisectionTest (see Table 1). The Mini Mental State Examination (MMSE) and the Attentional Matrices were alsoadministered to the patient as indexes of global cognitive functioning. Before including the patient in thestudy, we also performed the Token Test [17], to ensure normal comprehension abilities, necessary tounderstand the Duckneglect task instructions. IB scored 32/36, classified as non pathological accordingto the Italian norms. Written informed consent was obtained for all the cognitive evaluations and therehabilitation sessions, and the treatment has been conducted in accordance with the Declaration ofHelsinki. The same tests were performed immediately after the last rehabilitation session.

IP was treated with Duckneglect for 1 month, 5 days a week for an average of 30 minutes a day in2011 at the meeting room of the Department of Computer Science. There was a significant improvementof peripersonal neglect on the line bisection task at which patient showed a significant error reduction.Interestingly, IB showed an improvement also at the MMSE and the Attentional Matrices suggestingthat rehabilitation targeted to neglect had also a general positive effect on cognition. As reported byseveral other rehabilitation studies [6,7,19,33], there was a significant improvement of the performanceon these tests after each rehabilitation session, already in the first week of treatment. However, this earlyimprovement was not stable until the end of the rehabilitation period. More interestingly, the follow upassessment at five months after the treatment showed that improvement remained stable in the patientas shown also in Table 1. This proved that the treatment was effective both for neglect and generalcognition.

More in details, score measured before and immediately after treatment demonstrates an ameliorationof cognitive functions as shown by both the MMSE (particularly, IB ameliorated at the sentence readingand at the attention and calculation subtests) and the Attentional Matrices. As for neglect behavior, IBameliorated at the line bisection test, for which the midpoint shifted from +6.01 cm (tCRAWFORD =1.871, p = 0.03) to +2.66 cm (tCRAWFORD = 0.896, p = 0.191) and at the Albert Test (differenceof less than 2 omissions between right and left). In particular, in this test, IB missed the twelve linesleftmost before the treatment (the lines contained inside the rectangle in Fig. 5a), while he did miss anyafter treatment (Fig. 5b), scoring 100%.

At the follow up, IB achieved the same score in the Albert Test, the line bisection and the MMSE inwhich he maintained the amelioration reached with the treatment. Moreover, the Attentional Matrices

AUTH

OR

COPY

R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation 105

test, five months after rehabilitation, resulted inside the normal range and not anymore under threshold.IB showed no improvement in the Diller test just after rehabilitation, while at the follow-up he was ableto find few ‘

√’ symbols on the left and much more on the right. Overall, these results, may suggest some

idiosyncrasies of the patient for this particular task.From the game point of view, IB was 100% accurate at all game levels and stages throughout the reha-

bilitation period. The time to complete the tasks and the number of distractors hit progressively reducedfrom the first to the fourth week of treatment. IB’s reaction time with audiovisual cues significantly de-creased for all the levels, showing the fastest responses at the fourth week of treatment. On the otherhand with visual or no cues IB did improve only for the levels that contained more targets (levels three,four and five). As an additional remark, several tasks in everyday life became possible. For instance, hewas once requested by his wife to read her the title of the editorial article, that in the actual newspaperhe was reading (“Corriere della Sera”) is located in the leftmost column of the first page, and he did readit with no difficulty.

Most important, he was never tired exercising, he was very excited when he could succeed to terminateone game achieving the highest level of difficulty and was trying hard to locate all targets when especiallyat the beginning of the rehabilitation session this was difficult for him. He was eager to be able tocontinue rehabilitation at home as well as to participate to each rehabilitation session. In this respect thegame did indeed reach the goal of not boring the patient.

4. Discussion

Duckneglect is based on three key elements: games to guide rehabilitation, hands-free motion trackingand the display of mirror images.

4.1. Game design

Some of the games available in the entertainment market, like Nintendo Wii Party9 and Wii fit,10

have been recently tried for rehabilitation [3,21], but it was soon recognized that games created forentertainment, do not completely match the rehabilitation needs: they have a fixed interaction pace,often quite fast, a wealth of targets and distractors and they may easily produce strain and anxiety [41,46]. For these reasons games specific to rehabilitation have started to be developed [10,12,26,34,38].However, most of such games do not fully take into account basic game design principles [42] and, as aresult, rehabilitation games generally lack the appealing of commercial games because of their limitedgraphics and gameplay and become soon boring or have limited impact. Moreover, the rehabilitation aimshould be clearly matched inside the gameplay to make the games most effective. In particular, we havekept the game play as simple as possible so that the patient could better focus on the visual search task,hidden under the game. For neglect patients, revisiting the classical visual search tasks [9,14,29] throughgames ensures theoretical validity of the rehabilitation program. Furthermore, Duckneglect records agreater amount of data compared to classical paper and pencil techniques, such as reaction times, handtrajectory and velocity profile, that enable a more detailed analysis of the patient’s performance duringthe rehabilitation exercises.

9http://wiiparty.nintendo.com/.10http://www.nintendo.com/games/detail/hoiNtus4JvIcPtP8LQPyud4Kyy393oep.

AUTH

OR

COPY

106 R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation

The visual search tasks were realized here leveraging the state-of-the-art game design theory [16,26,38,42,43]: meaningful play, flow theory and sense of presence have all incorporated in the design ofDuckneglect.

Meaningful play states that each game action must have a direct and clearly distinguishable feedbackas well as a reasonably lasting effect. The relationship between the action and its outcome in the gameshould be discernible with a clear feed-back in the immediate, with no appreciable delay. This helpsthe patient in understanding what he can and cannot do. It was achieved here through a clear video-audio feed-back of the success (target hit) or failure (distractor hit) of a patient’s action and on a robustidentification of the collision of the patient’s hand with the targets and the distractors. We have observedthat when the red cross is displayed on the screen upon reaching for a distractor, the patient tends toobserve more carefully the scene, possibly to understand how to improve.

The second basic principle is the theory of flow that establishes a relationship between the challengelevel and the patient’s skills: the game should not be too difficult or too simple. According to [16], whenthe user’s skills are matched to the challenge level posed by the game, the user enters into a state ofcomplete focus and immersion in the game, state in which he loses track of time. Recent studies alsoshow that physical pain is also reduced when flow state is reached.11 To achieve this, we have designeddifferent stages and levels with a progressive difficulty, increasing according to patient’s impairment.The difficulty is defined by the cues available: visual and/or auditory or no cues, and on the number oftargets and distractors in the scene. Only upon completion of one stage/level, the patient can progressto the next stage/level, such guaranteeing that an adequate challenge level is maintained throughout arehabilitation sessions. Moreover, the game pace is controlled by the patient himself as a large timeamount is assigned to him to explore the space searching for targets. To increase the flow state, attractivegraphics that reproduces calm scenarios with fine details has been realized (cf. Fig. 1). Targets anddistractors were designed large enough to make interaction clear; a surprise element has also been addedto increase variability and keep the attention alive. The controlled randomization in the choice of thescenarios and in the position of targets and distractors makes the patient feeling challenged by an alwaysdifferent situation. The scoring system reflects also the rehabilitation nature of the games [10,12,26,38]:no negative points or “death” is given to the patient inside the game, but a positive score is always issued,with a score increasing inversely proportional with the time required to the patient to reach the target.The score obtained in the different sessions is also shown to the patient to demonstrate his improvementover time. Music is played during the rehabilitation sessions to make the patient more comfortable duringthe exercises and avoided full silence that is never present in everyday life. All these elements concur tocreate a flow experience [16,42]. This is particularly important because when the patient is completelyfocused on the game everything else vanishes, hiding the burden of therapeutical repetitive tasks andthe difficulties arising from his impairment, under the entertaining experience of a game, that can evencreate an engaging rehabilitative situation. This is even more important if we consider that post-strokepatients often fall into depression [33]. This has allowed to train IB for one month and still having himeager to play the game: the most common comment from the patient at the end of many rehabilitationsessions was: “already done?”.

4.2. Mirror images

The sense of presence is another strong point of the Duckneglect approach: as the patient sees himself(mirrored, Fig. 1a) on the screen, he has really the strong perception of being himself the actor in thegame.

11http://www.gq.com/news-politics/newsmakers/201202/burning-man-sam-brown-jay-kirk-gq-february-2012.

AUTH

OR

COPY

R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation 107

This is associated to the use of “hands-free” tracking of motion that does not require attaching anydevice to the patient, allowing the most natural interfacing with the games, more natural than that of-fered, for instance, by the gloves used in Virtual Reality approaches [12,14,34,41]. Moreover, hands-freetracking can potentially enable patients of doing rehabilitation autonomously in a more natural way. Theresults obtained are very similar to those of the Gesturetek IREX system12 that is the state of the artfor hands-free tracking in rehabilitation. However, such system is not customizable to different patholo-gies and costly and for this reasons, its diffusion has been limited to clinical centers. We improved this,assembling an equivalent rehabilitation platform using off-the-shelf components and robust real-time im-age processing to identify the patient’s silhouette in the incoming images. The silhouette is then pastedinside the virtual environment through the alpha channel associated to the RGB image.

This solution is both particularly computationally efficient and effective as can be seen in Fig. 1a. Thecritical processing step is background subtraction. This task becomes particularly difficult when pulsedor direct lights are present in the room. The first ones produce image flickering and the second ones,produced for instance by light spots, may cast shadows of the patient over the background that makethe silhouette not clean. The view of a jagged image of himself is disturbing for the patient. This wasconfirmed when in a very bright day in which the orientation of the external sun light (associated tothe particular hour of the day) combined with internal lights, created a lighting condition for which thethreshold did not allow a good separation of the silhouette from the background. We had to close theblinds to start the rehabilitation session that day. This is also an indirect confirmation of the “uncanny”valley effect [43] that states that the display of a human being is pleasant either when it is clearly acopy of the subject or when it is clearly an avatar that does not pretend to be a humanoid being; not wellrepresented human beings fall inside the uncanny valley where the display is perceived as unpleasant. Weremark here that the state of the art in background subtraction [45] does not allow completely solving thisproblem through image processing techniques and we have preferred to work on the set-up: a lightingcondition that creates a diffused constant light guarantees the optimal situation. Such condition could beeasily achieved in the meeting room of our Department, where rehabilitation took place, but it is moredifficult to recreate in patients’ homes. We did make some preliminary tests that showed that, in somecases, a white projection screen or a white sheet behind the patient was needed, and further tests arerequired. We have also investigated the use of 3D cameras like Microsoft Kinect13 to make silhouetteextraction more robust with respect to lighting conditions [25]. However, the quality of the silhouetteproduces is lower than that obtained through background subtraction and the cost of Kinect is about 10times that of the Web-cam used here. For these reasons the introduction of 3D cameras for tasks that areessentially bi-dimensional, as visual search is, should be carefully evaluated and possibly avoided.

4.3. Implementation issues

Collision detection is a critical task when using hands-free paradigm. We have not used more com-plex game engines like Panda 3D14 or Unity15 that provide embedded collision detection functionalitiesbetween 3D surfaces, but we have preferred to use standard Open GL library and developed our owncollision detection function as we want to reliably measure when a meaningful collision between the

12http://www.gesturetekhealth.com/products-rehab-irex.php.13http://www.microsoft.com/en-us/kinectforwindows/.14https://www.panda3d.org/.15http://unity3d.com/.

AUTH

OR

COPY

108 R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation

Table 1Patient IB scores at neglect tests, before, immediately after, and 5 months after the treatment

Before treatment After treatment 5 months follow upMMSE 20* 27 26.46Attentional matrices 13* 20* 47Line bisection + 6.01∗ + 2.66 + 1.64

Left Right Left Right Left RightAlbert test 12∗ 18 11 12 16 15Diller V test 0∗ 39 0∗ 36 5∗ 52

Asterisks and bold characters indicate pathological scores. For the Mini Mental State Examination (MMSE) [23] and Atten-tional matrices pathological scores are defined using Italian normative values: for the MMSE, the cut off value is set to 23.8 [44].For the Attentional Matrices, the cut off value is set to 23.9 [18]. For the Albert line cancellation test a two or more targetsomissions difference between left and right indicates neglect [2]. For the Diller test, a four or more target omissions differencebetween left and right indicates neglect. Finally, line bisection has been evaluated with the Crawford t test for single cases [15].

patient and a virtual object occurs. To this aim, we identify that a collision has occurred when a certaindegree of overlap between the patient’s hand and a virtual object is detected (Fig. 4). Hands trackingwas carried out here with the simplified mechanism described in Section 2.3. This does work well whenhands do not overlap with the body on the camera image and this is the reason for which collisions aresearched only outside the area identified in Fig. 3b. To extend hands tracking beyond this area, the use ofgloves with colored patches (blue for the right glove and green for the left one) has been proposed [12].However, as wearing gloves can be a difficult task for hemiparetic patients and an almost impossible onefor hemiplegic ones, like most neglect patients are, such solution has been ruled out here as it wouldlargely limit usability. On the other side, interesting hands movements occur outside the body area andall targets and distractors are placed on the left and right hemispace of neglect patients (cf. Fig. 3b).Therefore, the simple tracking mechanism adopted here is powerful enough for the tasks used here.

4.4. Clinical results

As it can be seen from Table 1, IB ameliorated after Duckneglect treatment; more interestingly hisimprovement persisted and even increased several months after completing Duckneglect rehabilitation,although in this period he did not receive any other form of rehabilitation. This supports the accumulat-ing evidence that intensive rehabilitation does allow recovering cognitive functions even in the chronicphase [33,34,46]. Moreover, as suggested in [28], the improvement after Duckneglect treatment can beascribed to the fact that the patient has reached a threshold above which spontaneous recovery is trig-gered: the patient starts exploring the neglected space spontaneously that indeed would decrease hisneglect.

Interestingly IB improved not only on neglect but also on global cognitive functions as shown inTable I. This improvement, confirmed by the neuropsychological tests, has also been corroborated byobservations reported by the patient’s family: IB was able to read the whole newspaper again and toexplore the surrounding space more efficiently. These results support the tenet that a wider and enrichedscenario including realistic objects as stimuli, rather than abstract geometric targets, and seeing himselfas an actor on the screen is more motivating and encouraging for patients engaged in motor recoveryprograms [6,11,36,41]. In this respect, Duckneglect might enable a greater generalization of cognitivestrategies with respect to classical cognitive and behavioural training.

Another key characteristic of Duckneglect is mirror imaging: the patient sees himself on the screen“mirrored”, that is flipped with respect to the vertical midline (Fig. 1a, 2c and 3b) and follows his ownmovements in real-time. Such mirror images seem to improve the performance of right brain damaged

AUTH

OR

COPY

R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation 109

patients with neglect when reaching objects located in the contralesional, ignored space [37], as theyrequire specific processing of the visual information mediated by adequate body mental representations.Such mechanism develops early in childhood as humans are familiar with mirrored body since the firstmonths of life [5].

Notably, although trained only in the far space (beyond the “by hand reachable” space), IB showedthe remission of neglect also for peri-personal space. One may speculate that the visual and kinestheticmultisensory integration prompted by the real-time feed-back provided by the mirroring effect (seeinghis own upper limb reflected in the far space while reaching objects), might facilitate the spatial remap-ping between far and near space [6,39]. In our patient the hypothesized spatial remapping seems to occurdue to the “action feedback” in the absence of the presence of a tactile or visual continuity obtained bymeans of a tool allowing to reach the far space (i.e. a long stick or a laser pointer), the patient hand itselfbecomes the effective “tool”.

We remark here that very recently a more subtle identification of neglect is proposed, that is basedon the number and complexity of cognitive tasks required to the patient at a given time [7]. Duckne-glect can be easily extended to providing multiple tasks that have been envisaged for a more adequaterehabilitation in these cases.

5. Conclusion

Duckneglect has two goals: enable intensive rehabilitation and explore the possibility to move reha-bilitation at patient’s home where the patient is most comfortable and he can find the best time windowin the day to train, avoiding commuting to specialized centers or hospitals.

Preliminary results both in terms of clinical outcome and in terms of compliance with the patient showthat video-games, designed according to neuropsychological theory on one side and on best game designpractice on the other side, could be the best solution to provide intensive rehabilitation.

The architecture of Duckneglect is kept as simple as possible: the slide projector used in our meetingroom and the projection screen can be substituted by a standard TV screen connected to the PC. Sucha platform can open the door for massive deployment at home. However, to make this a reality a feed-back to the hospital is required to monitor the patient progression as suggested in the EU funded projectsRewire16 and Cupid17

Acknowledgements

We thank Dr Davide Cicerale for his help in collecting the data. This work was partially sup-ported by Rewire, project funded by the EC under the FP7 framework, contract N. 287713. Web site:http://www.rewire-project.eu.

References

[1] Adair JC, Barrett AM. Spatial neglect: clinical and neuroscience review: A wealth of information on the poverty ofspatial attention. Ann New York Academy of Sciences 2008; 1142: 21.

16http://www.rewire-project.eu.17http://www.cupid-project.eu.

AUTH

OR

COPY

110 R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation

[2] Albert M. A simple test of visual neglect. Neurology. 1973; 23: 658.[3] Baltaci G, Harput G, Haksever B, Ulusoy B, Ozer H. Comparison between Nintendo Wii Fit and conventional reha-

bilitation on functional performance outcomes after hamstring anterior cruciate ligament reconstruction: prospective,randomized, controlled, double-blind clinical trial, Knee Surg Sports Taumatol Arthrosc. 2012.

[4] Barrett AM, Crucian GP, Schwartz RL, Heilman KM. Adverse effect of dopamine agonist therapy in a patient withmotor-intentional neglect. Arch Phys Med Rehab. 1999; 80(5): 600.

[5] Beis JM, Andre JM, Barre A, Paysant J. Mirror images and unilateral spatial neglect. Neuropsychologia 2000; 39(13):1444.

[6] Berti A, Frassinetti F. When far becomes near: remapping of space by tool use. J Cogn Neurosc. 2000; 12(3): 415.[7] Bonato M, Priftis K, Marenzi R, Umiltà C, Zorzi M. Deficits of contralesional awareness: A case study on what paper

and pencil tests neglect. Neuropsychologia 2012; 26(1): 20.[8] Borghese NA, Mainetti R, Pirovano M, Lanzi PL. An integrated low-cost system for at home rehabilitation, Proc 18th

IEEE Conference on Virtual Systems and Multimedia, VSMM2012, 2012.[9] Bowen A, Lincoln NB. Rehabilitation for Spatial Neglect Improves Test Performance but Not Disability. Stroke 2007.

38: 2869.[10] Burke J, McNeill M, Charles D, Morrow P, Crosbie J, Mc- Donough S. Optimising engagement for stroke rehabilitation

using serious games. The Visual Computer. 2009; 25: 1085.[11] Buxbaum LJ, Ferraro MK, Veramonti T, Farne A, Whyte J, Ladavas E. Hemispatial neglect: Subtypes, neuroanatomy,

and disability. Neurology. 2004; 62(5): 749.[12] Cameirã MS, Bermúez i Badia S, Duarte Oller E, Verschure PF. Neurorehabilitation using the virtual reality based

Rehabilitation Gaming System: Methodology, design, psychometrics, usability and validation. J NeuroEng Rehab. 2010;22: 7.

[13] Cappa SF. Neglect rehabilitation in stroke: not to be neglected. Europ. J. Neurol. 2008; 15(9): 883.[14] Castiello U, Lusher D, Burton C, Glover S, Disler P, Improving left hemispatial neglect using virtual reality, Neurology

2004; 62: 1958.[15] Crawford JR, Garthwaite PH, Porter S. Point and interval estimates of effect sizes for the case-controls design in neu-

ropsychology: rationale, methods, implementations, and proposed reporting standards. Cogn Neurops. 2010; 27(3): 245.[16] Csikszentmihalyi M. Flow: The psychology of optimal experience. Harper Perennial, 1991.[17] De Renzi E, Vignolo LA. The token test: A sensitive test to detect receptive disturbances in aphasics. Brain 1962; 85:

665.[18] Diller L, Weinberg J. Hemi-inattention in rehabilitation: The evolution of a rational remediation program. Advances in

Neurology. 1997; 18: 63.[19] Di Monaco M, Schintu S, Dotta M, Barba MS, Tappero R, Gindri P. Severity of unilateral spatial neglect is an independent

predictor of functional outcome after acute inpatient rehabilitation in individuals with right hemispheric stroke, ArchPhys Med Rehab. 2011; 92(8): 1250.

[20] Farne A, Bonifazi S, Ladavas E. The role played by tool-use and tool-length on the Plastic Elongation of peri-hand space:A single case study. Cogn Neurops. 2005; 22(3): 408.

[21] Flynn S, Palma P, Bender A. Feasibility of using the Sony PlayStation 2 gaming platform for an individual poststroke: acase report. J Neurol Phys Ther. 2007; 31(4): 180.

[22] Foley JD, Van Dam A, Hughes JF, Feiner SK. Computer Graphics: Principles and Practice, 2nd edition, Addison-Wesley,2011.

[23] Folstein MF, Robins LN, Helzer JE, The Mini-Mental State Examination. Arch Gen Psych. 1983; 40(7): 812.[24] Frosio I, Mainetti R, Palazzo S, Borghese NA. Robust Kinect for rehabilitation. Proc GIRPR Conference, 2012.[25] Gonzalez, RC, Woods RE. Digital image processing, third edition. Prentice Hall, 2007.[26] Goude D, Bjäk S, Rydmark M. Game design in virtual reality systems for stroke rehabilitation. Stud Health Techn Inf.

2007; 125: 146.[27] Halligan PW, Marshall JC. How long is a piece of string? A study of line bisection in a case of visual neglect. Cortex.

1988; 24(2): 321.[28] Han CE, Arbib MA, Schweighofer N. Stroke rehabilitation reaches a threshold. PLoS Comp Biol. 2008; 4(8): 100.[29] Jaehun K, Kwanguk K, Deog Young K, Won Hyek C, Park Chang-Ik O, Suk Hoon H, Kiwan J, Ku W, Sang K. Virtual

Environment Training System for Rehabilitation of Stroke Patients with Unilateral Neglect: Crossing the Virtual Street.Cyberpsych Behav. 2007; 10(1): 7.

[30] Karnath H, Milner AD, Vallar G. The cognitive and neural bases of spatial neglect. Oxford [Oxfordshire]. Oxford Uni-versity Press, 2002.

[31] Katz N, Hartman-Maeir A, Ring H, Soroker N. Functional disability and rehabilitation outcome in right hemispheredamaged patients with and without unilateral spatial neglect. Arch Phys Med Rehab. 1999; 80(4): 379.

[32] Kerkhoff G. Modulation and rehabilitation of spatial neglect by sensory stimulation. Progr Brain Res. 2003; 142: 257.

AUTH

OR

COPY

R. Mainetti et al. / Duckneglect: Video-games based neglect rehabilitation 111

[33] Langhorne P, Coupar F, Pollock A. Motor recovery after stroke: A systematic review. The Lancet Neurology. 2009; 8:741.

[34] Laver KE, George S, Thomas S, Deutsch JE, Crotty M. Virtual reality for stroke rehabilitation. Cochrane DatabaseSystem Review. 2001; 9 CD008349.

[35] Measso G, Cavarzeran F, Zappala G, Lebowitz B, Crook T, Pirozzolo F. Il Mini Mental State Examination: studionormativo di un campione random della popolazione italiana. Develop Neuropsyc. 1983; 9(2): 77.

[36] Milner AD, McIntosh RD. The neurological basis of visual neglect. Curr Opinion Neurol. 2005; 18(6): 748.[37] Neppi-Modona M, Rabuffetti M, Folegatti A, Ricci R, Spinazzola L, Schiavone F. Bisecting lines with different tools

in right brain damaged patients: the role of action programming and sensory feedback in modulating spatial remapping.Cortex 2007; 43(3): 397.

[38] Pirovano M, Mainetti R, Baud-Bovy G, Lanzi PL, Borghese NA. Self-Adaptive Games for Rehabilitation at Home. ProcIEEE Conference on Computational Intelligence and Games, CIG; 2012.

[39] Ramachandran V, Altschuler E, Stone L, Al-Aboudi M, Schwartz E, Siva N. Can mirrors alleviate visual hemineglect?Medical Hypotheses. 1999; 52: 303.

[40] Rode G, Pisella L, Rossetti Y, Farne A, Boisson D. Bottom-up transfer of sensory-motor plasticity to recovery of spatialcognition: Visuomotor adaptation and spatial neglect. Prog Brain Res 2003; 142: 273.

[41] Rose FD, Brooks BM, Rizzo AA. Virtual reality in brain damage rehabilitation: Review. Cyberps Behav, 2005; 8(3):241.

[42] Schell J. The Art of Game Design: Book of Lenses. Elsevier, 2008.[43] Seyama J, Nagayama RS. The Uncanny Valley: “ffect of Realism on the Impression of Artificial Human Faces”. Presence.

2007; 16(4): 337.[44] Spinnler M, Tognoni G. Standardizzazione e taratura italiana di test neuropsicologici. Ital J Neurol Science. 1987; 8(6).[45] Stauffer C, Grimson EL. Learning Patterns of Activity Using Real-Time Tracking, IEEE Trans Pattern Anal. Mach

Intell.2000; 22(8): 747.[46] Thornton M, Marshall S, McComas J, Finestone H, McCormick A, Sveistrup H. Benefits of activity and virtual re-

ality based balance exercise programmes for adults with traumatic brain injury: perceptions of participants and theircaregivers. Brain Injury 2005; 19(12): 989.

[47] Tsirlin I, Dupierrix E, Chokron S, Coquillart S, Ohlmann T. Uses of virtual reality for diagnosis, rehabilitation and studyof unilateral spatial neglect: Review and analysis, Cyberps behav social networking. 2009; 12(2): 175.

[48] Wann JP, Rushton S, Mon-Williams M. Natural problems for stereoscopic depth perception in virtual environments,Vision Research 1995; 35(19): 2731.

AUTH

OR

COPY