B

Bv

CMa

b

h

�����

a

ARRA

KVVSRB

1

ip

G

vT

0h

Journal of Neuroscience Methods 209 (2012) 290– 298

Contents lists available at SciVerse ScienceDirect

Journal of Neuroscience Methods

journa l h omepa g e: www.elsev ier .com/ locate / jneumeth

asic Neuroscience

uilding virtual reality fMRI paradigms: A framework for presenting immersiveirtual environments

harles Muellera,∗, Michael Luehrsa, Sebastian Baeckea, Daniela Adolfa, Ralf Luetzkendorfa,ichael Luchtmannb, Johannes Bernardinga,1

Department for Biometrics and Medical Informatics (IBMI), Otto-von-Guericke-University, Medical Faculty, Leipziger Str. 44, 39120 Magdeburg, GermanyKlinik für Neurochirurgie, Otto-von-Guericke-University, Leipziger Str. 44, 39120 Magdeburg, Germany

i g h l i g h t s

We develop informatical concepts, which allow the creation of own VR-fMRI paradigms.We develop neuroinformatical techniques which provide real-time VR-fMRI studies.We embed an easy-to-handle integration concept for virtual environment files.We validate the application in a real-time VR-fMRI study with spatial memory topic.Subjects indicate higher interaction and more attention than in common fMRI studies.

r t i c l e i n f o

rticle history:eceived 12 February 2012eceived in revised form 18 May 2012ccepted 23 June 2012

eywords:irtual realityirtual environmentstimulus presentationeal-time fMRIrain–computer interface

a b s t r a c t

The advantage of using a virtual reality (VR) paradigm in fMRI is the possibility to interact with highlyrealistic environments. This extends the functions of standard fMRI paradigms, where the volunteer usu-ally has a passive role, for example, watching a simple movie paradigm without any stimulus interactions.From that point of view the combined usage of VR and real-time fMRI offers great potential to identifyunderlying cognitive mechanisms such as spatial navigation, attention, semantic and episodic memory,as well as neurofeedback paradigms. However, the design and the implementation of a VR stimulusparadigm as well as the integration into an existing MR scanner framework are very complex processes.To support the modeling and usage of VR stimuli we developed and implemented a VR stimulus appli-cation based on C++. This software allows the fast and easy presentation of VR environments for fMRIstudies without any additional expert knowledge. Furthermore, it provides for the reception of real-time

data analysis values a bidirectional communication interface. In addition, the internal plugin interfaceenables users to extend the functionality of the software with custom programmed C++plugins. The VRstimulus framework was tested in several performance tests and a spatial navigation study. Accordingto the post-experimental interview, all subjects described immersive experiences and a high attentionalload inside the artifical environment. Results from other VR spatial memory studies confirm the neuronalactivation that was detected in parahippocampal areas, cuneus, and occipital regions.

. Introduction

In the last years, several functional magnetic resonance imag-ng (fMRI) studies used virtual reality (VR) environments whenresenting their stimuli. Virtual reality refers to an artificial

∗ Corresponding author. Permanent address: Hegelstrasse 6, 39104 Magdeburg,ermany. Tel.: +49 391 40 59061; fax: +49 391 40 59061.

E-mail address: charles.mueller79@gmail.com (C. Mueller).1 Institut Für Biometrie und Medizinische Informatik, Medizinische Fakultät, Uni-

ersität Magdeburg, Leipziger Str. 44, 39120 Magdeburg, Germany.el.: +49 391 67 13537; fax: +49 391 67 13536.

165-0270/$ – see front matter © 2012 Elsevier B.V. All rights reserved.ttp://dx.doi.org/10.1016/j.jneumeth.2012.06.025

© 2012 Elsevier B.V. All rights reserved.

computer-generated environment, with which users act and inter-act as if in a known real environment. Furthermore, users canexperience things that would otherwise be very difficult or evenimpossible in a magnetic resonance scanner or with an electroen-cephalograph.

The big advantage of virtual environments lies in the presen-tation of realistic stimuli. Instead of passively watching a simplemovie stimulus, subjects can interact actively with the paradigm,for example, navigating and exploring an artificial environment.

Referred to this, several virtual environment studies have been usedto investigate the role of the hippocampus and the parahippocam-pal area in topographical, spatial and episodic memory processes(Aguirre, 1998; Maguire et al., 2006; Mellet et al., 2010). In

scienc

awD

r(ocmtet(c(

mr2vaet

ritfsatssetdtrpftwtsfP

2

2

ms

rst

cisbSM

C. Mueller et al. / Journal of Neuro

ddition, the location of so-called place cells in the human brainas researched with virtual environments (Aguirre et al., 1996;oeller et al., 2010).

Another example of how virtual environments are used in neu-osciences are neurofeedback studies and human–brain-interfacesHBI). By using operant conditioning, subjects learn to regulate theirwn brain activation with the aid of a neurofeedback signal. In suchases, fMRI image data need to be analyzed in real-time, whicheans as fast as they are acquired, i.e., within a single repetition

ime (TR) (Gembris et al., 2000; Weiskopf et al., 2007; Scharnowksit al., 2009). Specific examples include a classification algorithmo control the movement of an avatar in a two-dimensional mazeYoo et al., 2004) or reducing the pain intensity in the right anterioringulate cortex with neurofeedback training and a virtual flameDeCharms, 2005).

Similarly, several psychological studies used virtual environ-ents in their therapies: burn pain therapy (Hoffman et al., 2000),

eduction of claustrophobic symptoms (Garcia-Palacios et al.,007), and therapy of posttraumatic stress disorder based on theirtual reality exposure therapy (Rothbaum et al., 1999). The basicspect of these psychological therapies is the immersive experi-nce, which helps to evoke relevant neuronal activity and provideshe essential feeling of being inside of a real world.

Alongside these benefits, the main difficulty of using virtualeality in neuroscientific experiments are the time-consumingmplementation, the combination of neuroscientific and compu-ational methods, and the integration into an existing acquisitionramework. In spite of the increasing number of neuroscientifictudies using virtual reality, few virtual environment stimuluspplications can handle such experiments in their entirety. Sincehe technical complexity and the required knowledge in computerciences demand high personnel and financial resources, severalcientific groups work with virtual environment stimuli using staticnvironments from computer games. These game frameworks con-ain only one virtual environment, whose technical restrictionsictate the design of the neuroscientific experiment. Our aim waso implement an adaptive, flexible, and extendable virtual envi-onment stimulus application that includes several scenes for fMRIaradigms. The user should be able to import a VR scene with just aew mouse clicks and present this scene to a subject. Furthermore,he application should provide a highly realistic stimulus outputhich fulfills the immersive requirements necessary for using vir-

ual reality in psychological therapies (Regenbrecht et al., 1998). Toupport real-time fMRI experiments we integrated functionalitiesor exchanging data values with external applications (Mathiak andosse, 2001).

. Materials and methods

.1. Experimental infrastructure

For the presentation of virtual environments we used a multi-odal stimulus environment concept that comprises three basic

ystems. See Fig. 1 for a detailed overview of this concept.The first basic system is the presentation system which is rep-

esented by the virtual environment stimulus application. Thisystem provides visual and auditory stimuli and presents them tohe subject.

The second is the data acquisition system represented in ourase by the magnetic resonance imaging scanner which producesmage data for the data analysis system. Implementation of the

ystem and all test experiments were conducted on a whole-ody MRI scanner (3T Trio, Software Version Numaris Syngo VA35,iemens Medical Systems, Erlangen, Germany) at the University ofagdeburg, Clinic for Neurology. An eight-element phased array

e Methods 209 (2012) 290– 298 291

coil (Siemens Medical Systems, Erlangen, Germany) was used forimaging. The vendor’s EPI BOLD sequence and the correspondingreconstruction programs were modified to export each volumedataset immediately after acquisition and internal motion correc-tion in real time to the host computer of the MRI scanner (Posseet al., 1999; Weiskopf et al., 2004, 2007; Hollmann et al., 2008).This software function performs the image export to any accessi-ble computer in the local area network. For the synchronizationbetween presentation system and data acquisition system we useda synchronization signal that is sent from the host computer on themagnetic resonance scanner to the presentation computer.

The last basic system is represented by the data analysis system.Therefore, we used the commercial real-time data analysis softwareTurbo BrainVoyager (Brain Innovation, 2012). To ensure that dataanalysis system and stimulus presentation were processed simul-taneously, we distributed both systems on two personal computers.For the real-time data analysis transfer, a bidirectional client/serverTCP communication protocol was implemented and tested (Luehrset al., 2011).

Data analysis was performed on a computer (Pentium IV,3.0 GHz, 2 GB Random Access Memory (RAM), Windows XP) con-nected to the local area network via 100 Mbit/s. For the first testsof the real-time fMRI framework we embedded the TCP commu-nication protocol into a Turbo BrainVoyager plugin to transfer dataanalysis values in real time to our stimulus application. The presen-tation of visual and auditory VR stimuli also took place on a separatepersonal computer (stimulus computer, Pentium IV, 3.0 GHz, 3 GBRAM, Windows XP). Visual information was projected with a videoprojector on a transparent screen and viewed via a 45◦ mirrormounted on the receiver coil. The statistics computer and the VRstimulus computer were connected directly via a 100 Mbit/s Ether-net crossover cable. Apart from the interface for the real time exportof the image data, which had to be implemented for the vendor-specific scanner system, all components were independent of thescanner system.

2.2. VR stimulus application

The simulation of a virtual environment generally requirestwo software components. The first is a virtual scene or a three-dimensional model representing the pure environment, e.g. objectsand architectural structures. For the creation of our virtual scene weconstructed several single three-dimensional models and mergedthem into one comprehensive model using the professional designsoftware Autodesk, Inc., 3ds Max. Furthermore, this process allowsthe custom modeling and integration of structural elements, suchas geographical landmarks or human character models. The sec-ond component is the so-called game engine framework, whichcalculates all interactions between the subject and the three-dimensional model. It also simulates human behaviour and realisticenvironmental conditions, such as collision detection, the physi-cal characteristics of the specific environment, visual and auditorystimuli, and the behavioral patterns of computer-simulated char-acters and their movements. In addition, building event-markersand interactive elements is supported by a virtual event sys-tem. Trigger event classes observe the main character movementsand initiate predefined animations relative to the current posi-tion. To cover these comprehensive functions, we integrated theTrinigy Inc. Vision Game Engine, version 7.6.4. It is typically usedto build professional computer games and has been provided tous for non-commercial and scientific use. All important parts ofthe Vision Game Engine were directly embedded in our C++source

code programmed with Microsoft Visual Studio 2008, Microsoft.NETFramework 2.5. To simulate the physical effects inside the vir-tual environment we used the free Nvidia, Corp., PhysX, version2.8.1. We extracted the necessary software libraries only and

292 C. Mueller et al. / Journal of Neuroscience Methods 209 (2012) 290– 298

Fig. 1. The technical infrastructure for the real-time fMRI setting. The components inside of the box depict the vendor-specific measurement system. During the dataacquisition process, the image data will be motion-corrected and transformed by the vendor specific image calculation environment. Afterwards the reconstructed imagedata will be transferred in real time to the host computer. Host computer, presentation system and data analysis system are connected via 100 Mbit local area network( ccess op ation

t

iFnmuteaf

Fhiai

LAN). To analyze the MRI image data in real time the data analysis system takes aresents the virtual environment to the subject in the MRI scanner. The synchronizhe experiment protocol.

ntegrated them into our virtual environment stimulus framework.or implementing the graphical user interface, the TCP commu-ication interface, our plugin interface, and for the several dataanagement functions we used the open source version of the

ser interface framework Nokia, Corp., Qt, version 4.6.3. To keephe virtual environment stimulus application well organized and

xtendable we designed our software as a multi-layered three-tierrchitecture software. In this hierarchical concept the user inter-ace represents the highest layer in the software framework. The

ig. 2. The software architecture of the VR stimulus application. The software design is bighest tier in the hierarchy is represented by the user interface layer, which includes al

t. The second is the application logic layer, which includes the MRI experiment data stond the plugin management. The third is the communication layer. The TCP communicatnterface are located here.

n the image data at the host computer. At the same time the presentation systemsignal connects the host computer with the presentation system and synchronizes

second is represented by the application logic layer, which con-trols the main process of the Vision Game Engine. The third andlast is defined by the communication layer, which governs thecommunication processes, e.g. TCP communication protocol, syn-chronization signal and Microsoft DirectX interface for UniversalSerial Bus button-press event control. For further details, see Fig. 2.

To transfer the real time fMRI data analysis values to the VR stim-ulus application, e.g. for neurofeedback paradigms, we developeda multi-thread and bidirectional communication protocol based

ased on the classical hierarchical model of the three-tier architecture concept. Thel graphical components and all controls to the functionalities implemented belowrage, the game engine processes, the collision detection, the physical simulation,

ion, MRI trigger signal functionality, our plugin interface, and the Microsoft DirectX

scienc

oCcapcrtdsaaeatBf

2

usotawcpteinasws

ahwfl3uoierirajtist

sTfiwbm

w

C. Mueller et al. / Journal of Neuro

n a standardized client-server architecture and the Transmissionontrol Protocol (TCP). The communication protocol works with alassical handshaking model, which avoids communication errorsnd supports a more flexible data transfer. In the communicationrocess both client and server can switch their roles in real time. Thelient that sends requests to the server can also switch to a serverole and serve requests from the former server process. By defaulthe virtual environment application acts as a server and receivesata analysis values from the data analysis software. On the clientide the data analysis software connects to the server side after eachnalyzed MRI volume and sends the data analysis values to the VRpplication. The communication protocol is designed for using anxtendable configuration. It ensures the transport of several datanalysis values from multiple regions of interest (ROI) at the sameime. To integrate this communication functionality in the TurborainVoyager software, we implemented a communication pluginor Turbo BrainVoyager 3.0 (Luehrs et al., 2011).

.3. Virtual reality experiments

To evaluate the interaction between the above described mod-les we conducted several fMRI-VR experiments, one tutorialession and two consistent main sessions for each subject, withur VR framework and the communication plugin integrated inhe Turbo BrainVoyager 3.0 (Brain Innovation, 2012). Four subjectsged 23–31 years, one female and three male, all right-handed,ere examined after giving written consent to the local ethics

ommittee. All subjects controlled an avatar from a first-personerspective and were shown how to navigate through an unknown,hree-dimensional virtual city map. In addition, they were asked toxplore and memorize the virtual environment with all topograph-cal characteristics. No specific restrictions were given regardingavigation: the subjects were free to fully interact and becomecquainted with the virtual environment. The avatar that repre-ented the virtual character or person in the virtual environmentas controlled by pressing four MRI-compliant buttons inside the

canner: move forward, move left, move right, and move backward.The CityMap environment used was modelled by Trinigy Inc.

nd consisted of three streets connected to each other, and severalouses and backyards, see Fig. 2(C). The subjects were measuredith the adapted EPI BOLD sequence (TR = 2000 ms, TE = 30 ms,ip angle = 77◦, 34 slices, matrix: 64 × 64, spatial resolution:

mm × 3 mm × 3 mm). All paradigm settings of this fMRI-VR stim-lus were designed completely with the graphical user interfacef the implemented VR stimulus application. The designed exper-ment consisted of two conditions: resting condition (a), and a VRxploring condition (b). Altogether we acquired 132 volumes. Weepeated a block of 11 volumes 12 times with 4 scans in the rest-ng condition (a) and 7 scans in the VR exploring condition (b). Theesting condition (a) was signalized by showing a black screen with

red cross in the center. In the VR exploring condition (b) the sub-ect had full control over the avatar and navigated the streets ofhe CityMap. When the resting condition (a) began, navigation wasnterrupted by showing the black screen again. After this break, theubject continued navigation at the same point as before interrup-ion.

Furthermore, we evaluated the interaction between the VRtimulus application and real-time fMRI data analysis of theurbo BrainVoyager software. This application analyzed all volumesor each subject and thresholded the scans to display activ-ty correlated to the VR paradigm in real time. All image data

ere motion-corrected, spatially smoothed and linearly detrended

efore analysis with the recursive least squares general linearodel (GLM) algorithm (Brain Innovation, 2012).In addition to the online data processing, an offline analysis

ith BrainVoyager QX 2.2 was performed in order to quantify the

e Methods 209 (2012) 290– 298 293

observed neuronal activity. EPI scans were corrected for motionand coregistered to the acquired T1-weighted structural image.This image was spatially normalized into the stereotactic spacedescribed by Talairach and Tournoux. After the normalization, allEPI images were spatially smoothed using a Gaussian kernel with 4mm full width at half maximum (FWHM). A GLM was applied to thetime course. The block-design described above modeled the activa-tion blocks and showed the results for each subject. Furthermore, apost-experimental interview was performed in order to assess thelevel of user acceptance and immersive impressions.

2.4. Application performance tests

To ensure the computational accuracy and to obtain the quan-tifiable overall performance of this multi-layered software, thesingle performance of each layer had to be analyzed separatelyand evaluated in relation to the complete software framework(Eeckhout, 2010). For this reason, all layers presented in Fig. 2 weretested in several performance tests including capacity tests andstress tests. In order to ensure the safe execution of the imple-mented classes, all embedded algorithms were evaluated first byprocessing generated test data. After the successful evaluation withthese test data, the same algorithms were tested under exper-imental conditions processing data from the VR-fMRI study. Allperformance tests processed on generated test data were con-ducted on a personal computer (Intel Core 2 E8400 3 GHz; 3 GBRAM; Nvidia GeForce 9800GT with 512 MB memory, MicrosoftWindows XP Service Pack 3) and a laptop computer (Mobile IntelCore 2 P8400 2.4 GHz; 3 GB RAM, Mobile Intel 4 Series ChipsetGraphics Controller with 256 MB memory). The performance testsprocessed on VR-fMRI data were conducted on the experimentalsetup described in Section 2.1.

According to the user interface layer performance tests, weevaluated the entire layer process chain with regard to paradigmcreation processes and consistency check algorithms. For this rea-son, 20 generated data sets containing experimental paradigmsin block design and event-related design with a capacity of min-imum 20 to maximum 25 experimental conditions and an amountof measurement points of 1000 were loaded. During these tests,all process calls and the complete memory usage of the stimulusapplication were logged. The final user interface performance testswere performed during VR-fMRI experiments described in Section2.3.

When testing the performance and the computational accuracyof the application logic layer with generated test data we easedthe performance measurement by splitting this complex layer intovarious classes and tested each class separately. Afterwards, theinteraction between third-party classes, e.g. Vision graphic engineclasses, and self-programmed classes was evaluated. For this rea-son, three different virtual environment scenes on both computerswere analyzed with respect to their scene loading time, their usedsystem memory (RAM) load, and their graphical device memoryload. Furthermore, we analyzed the utilization of the central pro-cessing unit and of the graphical processing unit before and duringthe presentation of this three different virtual scenes. The threescenes were differentiated on the basis of their complexity andtheir richness of detail. To indicate the complexity of a scene weused three variables: the number of textures, the number of models,and the number of index buffers and vertex buffers. Based on thesecriteria the ViewerMap was classified as a low-detailed virtualenvironment, the CrossingMap as a moderately-detailed virtualenvironment, and the CityMap as a high-detailed virtual envi-

ronment. The final performance measurements were performedduring the virtual reality fMRI experiments.

In case of the communication layer, the performance andthe stability of the TCP transfer protocol was tested during a

2 scienc

cdrupomatvduTftvcRafiarof

2

setdttp

nofiteyasloletftttorfiatafiwap(

94 C. Mueller et al. / Journal of Neuro

omplex VR presentation. In both tests with generated data andata from VR-fMRI experiments we observed the long-term accu-acy, the technical feasibility, and the correct data transfer forpcoming neurofeedback experiments. Since the comprehensiveresentation of VR environments and the simultaneous exportf mean values from multiple ROI mean a high loss of perfor-ance, the interaction and the time-critical processing of both

lgorithms were evaluated in several data transfer sessions. Forhis reason, 10 generated data sets with a total amount of 1000alues each were simulated as 10 ROI and used for the mean valueata transfer. In an interval of 2000 ms, which corresponds to thesual repetition time in our experiments, the Turbo BrainVoyagerCP plug-in sent the corresponding value of the actual volumeor each simulated ROI. At the same time the stimulus applica-ion presented the virtual environment and received the meanalues from the Turbo BrainVoyager software in real time. Con-erning the performance tests during VR-fMRI experiments, threeOI localized in the right primary motor cortex, in the left primaryuditory cortex, and in the middle prefrontal cortex were definedor ROI signal transfer. During these experiments, the TCP plug-n accessed the provided Turbo BrainVoyager interface functionsnd sent the mean values of the actual volume for each ROI. Alleceived mean values and the processor utilization were logged inrder to analyze the performance of the underlying communicationramework.

.5. User acceptance tests

To ascertain the functionality and the usability of the VRtimulus application we conducted a user acceptance test withight neuroscientists. To that end, we evaluated the entire sys-em in four categories: the number of errors during stimulusesign phase; the number of errors during stimulus presenta-ion phase; the time required to finish the two given exercises;he subjective judgement on the software usability by thearticipants.

At the beginning of the test all participants estimated their basiceuroscience skills on the basis of four variables on a scale fromne (poor skills) to ten (excellent skills): working experience in theeld of neurosciences in years; experience in creating neuroscien-ific stimuli; experience in using neuroscientific stimulus software;xperience in using data analysis software and real-time data anal-sis software. First we calculated a threshold value of 3.0 by creating

fictitious default inexperienced user with the highest acceptablecore. For each participant an experience level value was estab-ished using a weighted sum of the mentioned four values. Basedn the value of the predefined threshold the calculated experienceevel value distinguished the inexperienced (less than 3.0) from thexperienced participants (greater than or equal than 3.0). All par-icipants received two tasks asking them to design and to adapt anMRI experiment using the VR stimulus application. In the first taskhey had to create an fMRI block design experiment paradigm withhree stimulus conditions and a given number of experimental set-ings, e.g. repetition time (TR), number of conditions, and resolutionf time in milliseconds or volumes. In the second task they wereequired to test the PRT file import function and to load a prede-ned BrainVoyager PRT file with an event-related experiment andll given paradigm settings. For further details, see Fig. 3(A). Afterhe loading process the participants had to adapt the paradigm in

predefined manner and to save the modifications in a new PRTle. Finally, all participants were given a related questionnaire in

hich they were asked to estimated the application functionality

nd usability. During this subjective assessment process partici-ants rated the software on a scale from one (poor usability) to tenexcellent usability) and described potential flaws in the software.

e Methods 209 (2012) 290– 298

3. Results

3.1. Application performance tests

The highest layer in the hierarchy, the user interface layer,was tested in several capacity tests and stress tests. All paradigmssettings either from generated test data or from the VR-fMRI exper-iment were loaded and processed successfully. The data obtainedwere used to analyze the paradigm creation process and the consis-tency check algorithm embedded in this software layer. In addition,the observation of corresponding process calls and the evaluation ofmemory usage showed no data loss or performance loss for the userinterface layer on both desktop computer and laptop computer.

In the application logic layer tests we documented and analyzedthe performance during several virtual environment presenta-tions. The results indicate a significant time-saving at the secondload of the specific virtual environment map. This is mainly dueto the memory of the graphical processing device which retainsinformation about the first loading process and uses it for thesecond. Another observation was the disparity in performancebetween personal and laptop computer. Both computers use thesame amount of random access memory (RAM) but the perfor-mance of the personal computer with a more powerful graphicaldevice is clearly higher than that of the laptop computer beforeand during the virtual environment presentation. The discrepancyin performance between personal and laptop computer is not sig-nificantly higher and allows a virtual environment presentation onlaptop computers with a more comparable hardware configurationas well. All virtual reality paradigms either from generated test dataor from the VR-fMRI experiment were loaded and presented suc-cessfully. The observation of process calls indicated no interactionerrors between self-programmed classes and third-party classes.

The lowest layer in the hierarchy, the communication layer, wastested for fast and reliable data transfer with generated test dataand data from the VR-fMRI experiment. The simulation of 10 ROIby using the Turbo BrainVoyager TCP plug-in combined with 10generated data sets was successfully realized. All 1000 values foreach simulated ROI were sent successfully with the implementedTCP client plug-in integrated into the Turbo BrainVoyager applica-tion. The stimulus application received all sent values and loggedthe associated timestamps. After the successful evaluation withgenerated test data, the communication layer was tested underexperimental conditions processing the human subject data fromVR-fMRI experiments. Therefore, three ROI described in Section 2.4were selected for the real-time ROI signal transfer via TCP. Con-cerning the experimental setup, the respective mean value of eachROI was sent within one TR to the VR stimulus application. Duringexperiment acquisition, the virtual reality stimulus presentationwas never interrupted and received all sent mean values withina time delay smaller one millisecond. These short transmissiontimes are mainly a consequence of using an optimized and self-implemented communication protocol combined with a 100 Mbit/sEthernet crossover cable for direct data transfer. Furthermore, theaverage increase in the processor utilization from the experimentpreparation phase to the virtual environment presentation phasewas 3.7% and indicated an exceptional performance for this layer. Inall data transfer sessions with both generated data and human sub-ject data no significant performance loss or data loss error occurred.All timestamps and communication information recorded by thedata transfer protocol confirmed the reliability of time-critical pro-cesses necessary for upcoming neurofeedback experiments.

3.2. User acceptance testing

During the tests the framework showed no software crashesor stability errors. All experimental paradigm settings either in

C. Mueller et al. / Journal of Neuroscience Methods 209 (2012) 290– 298 295

Fig. 3. The VR stimulus application offers the possibility to load and present several virtual environments and link them with fMRI or real-time fMRI paradigm settings.The experiment paradigm tab (A) includes all settings to define an fMRI or real-time fMRI experimental setup. For fast experimental design it is possible to import/exportparadigm settings from/to a BrainVoyager protocol (PRT) file. The virtual scene 3D-maze represents a self-modeled virtual environment scenario, which was imported intoour virtual environment framework and used for a real-time fMRI neurofeedback study (B) (Thoms et al., 2011). The scene CityMap was provided by Trinigy Inc. and usedfor the mentioned VR experiments in Section 2.3 (C). In addition, the framework provides the real-time adaption of artificial structures and characters. According to this, ther s fromA study

bceetebiuiae1iau8ptbdvapvopciv

eal-time weather effect framework analyzes incoming real-time data analysis value more extensive evaluation of this framework will be performed in a forthcoming

lock design or event-related design were created and loaded suc-essfully. Several application tabs assisted the participants in thexperiment creation process. Warning dialogs indicated input datarrors and helped users to avoid inconsistent experimental set-ings. Moreover the implemented tooltips for all control buttonsased software handling and were seen by all participants as aig advantage of the stimulus software. All participants could fin-

sh the given exercises without any problems. We evaluated thesability of the VR stimulus software for both experienced and

nexperienced users. Of the eight participants, four were classifieds inexperienced and four as experienced users. The inexperi-nced participants worked on the given exercises for 15 min,5 min, 16 min, and 17 min, respectively. The experienced partic-

pants worked on the given exercises for 15 min, 17 min, 18 min,nd 20 min. Furthermore the inexperienced participants rated thesability of the software with 6 points, 8 points, 8 points, and

points while the experienced participants its usability with 7oints, 7 points, 8 points, and 8 points, respectively. According tohe experience level, these results show no significant differencesetween both groups. In addition, three of the eight participantsescribed the user interface as well structured and the tooltips asery helpful for experiment creation. Moreover, two participantsssessed the fast import of experimental settings via BrainVoyagerrotocol files and the automatic block-design value calculation asery positive and useful. Two participants criticized the structuref the user interface and the arrangement of several buttons. One

articipant mentioned the missing tab highlighting to indicate theurrent location on the experiment creation dialog. All of their crit-cal comments will be taken into account in future versions of thisirtual environment stimulus application.

Turbo BrainVoyager and adapts weather influences inside the virtual environment. (D).

3.3. Virtual reality experiments

The VR stimulus application was successfully integrated into theSiemens 3T Trio scanner infrastructure. At the beginning of eachexperiment, we established and tested the TCP connection betweenthe Turbo BrainVoyager real-time data analysis software and theVR stimulus application. Twelve VR-fMRI experiments with foursubjects, one tutorial session and two consistent main sessions foreach subject, were successfully acquired. The stimulus paradigmwas successfully predesigned in preparation for the VR-fMRI exper-iments and correctly loaded at the beginning of each experiment.During the stimulus presentation no technical or graphical prob-lems occurred. All subjects were able to navigate through the virtualenvironment and reported no difficulties in memorizing the topo-graphical characteristics of the artificial world.

In the real-time data analysis with the Turbo BrainVoyager soft-ware, all subjects showed activation in the occipital region as wellas in the precuneus and the right middle frontal gyrus. As noted,all results focus on the contrast between navigation and baselineand verify the functionality of the implemented VR stimulus appli-cation.

In addition to the on-line data processing with Turbo BrainVoy-ager, an off-line BrainVoyager QX data analysis was performed inorder to quantify the measured BOLD signal. Since the technicalvalidation of the VR stimulus application was the main purpose,the average change in brain activity from four subjects across all

sessions was analyzed using a fixed effect analysis. FDR-correctedactivation clusters (p < 0.05) exceeding a spatial extent of six vox-els were considered significant. For a detailed view of the clusteranalysis, see Table 1.

296 C. Mueller et al. / Journal of Neuroscience Methods 209 (2012) 290– 298

Table 1Results of the fixed-effects group analysis for the VR-fMRI CityMap experiment. Brain regions listed showed significant activation for the virtual environment memorizingtask compared with the baseline. The data show FDR-corrected activation clusters (p < 0.05) exceeding a spatial extent of six voxels. Columns show anatomical regions,stereotactic coordinates, cluster size, cluster maximum value, and the corresponding Brodmann area.

Anatomical region Coordinates Cluster size Cluster max. value Brodmann area

X Y Z

RH lingual gyrus −21 −54 4 1125 10.144 18LH lingual gyrus −19 −61 2 6 5.707 19RH cuneus 5 −90 10 16 5.262 18LH cuneus −1 −75 12 491 8.768 23LH cuneus −10 −79 7 13 7.462 17LH parahippocampal gyrus −22 −52 4 77 6.160 30RH precuneus 6 −60 38 57 4.854 7RH middle frontal gyrus 48 8 41 114 5.316 8

tfmhtpmplfo

Ccpoaa

Fli

RH middle occipital gyrus 36 −64 7

LH middle occipital gyrus −51 −70 4

LH fusiform gyrus −40 −61 −14

Significant activity across all subjects was observed bilaterally inhe frontal, parietal, and the medial lobes and confirmed the resultsrom the on-line data analysis. The maximum cluster value and the

ajority of significantly activated voxels were located in the rightemisphere. In the occipital lobe, temporal activation was found inhe left lingual gyrus as well as in parts of the left cuneus and the leftarahippocampal gyrus. These activation clusters extended to theiddle occipital gyrus and smaller parts of the right parahippocam-

al region. Several medial temporal activations were found in theeft parahippocampal gyrus extending to the intersection with theusiform gyrus. In the frontal cortex, right-sided activations werebserved in the middle frontal gyrus. For further details, see Fig. 4.

Furthermore all subjects were familiar with fMRI experiments.oncerning the results from the post-experimental interview, theyonsidered the following aspects of the VR-fMRI paradigm as very

ositive: the high interaction with the paradigm, the new freedomf movement and the possibility to explore the virtual environmentlone, the persistently high level of attention to the paradigm inccordance with the constant influence of salient VR stimuli, and

ig. 4. Results of the fixed-effects group analysis for the VR-fMRI CityMap experiment.

ingual gyrus, cuneus, precuneus and middle temporal gyrus (B) on transverse, sagittal

mage of a single subject in Talairach space.

16 5.060 1948 4.383 19

7 3.569 37

the immersive impressions, which occurred during the explorationof the virtual environment.

4. Discussion

The virtual environment stimulus application was successfullyimplemented and evaluated during 12 real-time fMRI experimentswith four subjects and several performance tests. Neuroscientistswere able to design and to load several fMRI experiments either inblock design or event-related design and could apply experimentalsettings to the provided virtual environment scenes.

The implemented plugin framework provides high expandabil-ity and a custom access to internal functionalities. Plugins haveto be programmed in a predefined structure as a C++based direct

linked library. This gives the user the possibility to integrate theminto the VR stimulus application via the simple plugin–loader inter-face. In addition the integrated TCP communication protocol offersan enormous number of possibilities for real-time data exchange

Activity of the right and left parahippocampal region (A) as well as the activity inand coronal planes. Functional imaging data are superimposed on a T1-weighted

scienc

atoprsctftaw(twypi

tnoaatsum

tnctos

rlbeaiiHfmtf

tBcfrnpr1csBatita

C. Mueller et al. / Journal of Neuro

nd neurofeedback studies. According to the weather effect adap-ion presented in Fig. 3(D), the real-time modification of artificialbjects and characters based on the neurofeedback signal transferrovides new possibilities for social interaction studies and neu-ofeedback training. Forthcoming real-time VR-fMRI experimentshould lead to a more detailed evaluation and measurable resultsoncerning this virtual environment adaption framework. Besidehis more advanced neurofeedback approach, a classical real-timeMRI neurofeedback study has already been conducted. In detail,his study investigated the aspects of neurofeedback training using

self-modeled virtual maze integrated in our VR stimulus frame-ork as well as the implemented TCP communication protocol

Thoms et al., 2011), see Fig. 3(B). With the implementation ofhe communication protocol, we opted for a distributed solutionith a separate VR-stimulus computer and a separate data anal-

sis computer. For that reason our concept avoids the high loss oferformance that normally occurs on one computer with compute-

ntensive processes.The results of the performance tests confirm the stability and

he software quality of the implemented VR software and showo loss of performance. Furthermore, they show the importancef the graphic device. The graphical processing unit contributes

high amount of computational power and optimizes the over-ll performance of the entire process, especially when simulatinghree-dimensional virtual environments. For a virtual environmenttimulus presentation the use of a powerful graphical processingnit with a high amount of memory is hence strongly recom-ended.Despite the advantages of our concept some aspects still need

o be improved. Before a virtual environment can be used in aeuroscientific experiment it has to be designed in a very time-onsuming process. Three VR scenes are already provided withhe framework, but all users have the possibility to create theirwn virtual worlds and load them with the implemented stimulusoftware.

The algorithms for the neurofeedback avatar control allow onlyather restricted control mechanisms, such as turn right, turneft, move forward and move backward. These algorithms wille extended to support more complex decisions within a virtualnvironment. In addition the integration of artificial intelligencelgorithms will allow interactions with non-person charactersnside the virtual world and give the facilitate research on socialnteractions at a higher level of visualization (Mathiak et al., 2011).owever, stimulation in virtual environments is not restricted to

MRI experiments. With some modifications, the same approachay be used for other neuroscientific modalities such as elec-

roencephalography (EEG), magnetoencephalography (MEG) orunctional near infrared imaging (fNIR) studies.

In our VR experiments four subjects successfully navigatedhrough a virtual city map by using button press events. The TurborainVoyager showed the results of on-line data analysis that indi-ate the functionality of the VR stimulus software. Although onlyour volunteers were measured, the off-line results confirm theesults from other studies and show activation for topographicalavigation and spatial and episodic memory processes in parahip-ocampal gyrus, in posterior cingulate cortex, in medial parietalegions (precuneus), and occipital temporal areas (Maguire et al.,997; Aguirre, 1998; Mellet et al., 2010). Furthermore, the resultsonfirm the observation from previous studies using computer-imulated environments (Aguirre et al., 1996; Maguire et al., 2006;aecke et al., 2009). Referred to the post-experimental interview,ll subjects described a higher feeling of motivation and atten-

ion in comparison to conventional fMRI experiments. In addition,mmersive impressions were often described by the volun-eers and confirmed the presented virtual environment paradigms immersive and highly realistic. Based on these immersive

e Methods 209 (2012) 290– 298 297

impressions, the implemented VR stimulus application fullfils themost important requirement necessary for psychological VR ther-apies (Regenbrecht et al., 1998; Hoffman et al., 2000). Since theintegrated Trinigy Inc. Vision Game Engine is a commercial soft-ware product, the free download of the implemented virtual realitystimulus application will not be supported. However, Trinigy Inc.provides the usage of this graphic engine for non-commercial andscientific use. Anyone who admits this contractual point is allowedto use the implemented virtual reality software for free.

In summary, it can be stated that various VR solutions offer somefunctionalities but the complete set of functionalities of our frame-work was not accessible until now. Many solutions already usecommercial and hand-made virtual environments for fMRI experi-ments (Aguirre et al., 1996; Maguire et al., 2006; Mellet et al., 2010;Doeller et al., 2010), but no detailed information are available aboutthe technical setup and restrictions of the software solutions. Fur-thermore the presented scientific results are not replicable for thosewithout access to the technical specification or the software.

One of the commercial VR stimulus solutions for neurobe-havioral and functional MRI studies is VR World 2, which uses adrag-and-drop VR development concept (Baumann et al., 2003).However, VR World 2 is no longer available as a standalone softwareand provides no information about real-time fMRI support, infor-mation exchange between experimentally relevant applications, orcustom access to internal functionalities. Moreover, the specificvirtual world has to be purchased and cannot be modified after-wards. Other VR stimulus systems use several components fromthird companies as well. In particular, the software 3DVIA, Corp.,Virtools (Mellet et al., 2010) or the open source software Reactor-Man (Beck et al., 2010) are often used for developing and presentingvirtual environments.

Again, all aforementioned software solutions, especially thecommercial ones, offer great possibilities for designing virtual envi-ronments but do not have the flexibility and the expandabilityto facilitate research on a huge spectrum of neuroscientific ques-tions, e.g. real-time fMRI with neurofeedback topics, or real-timeadaptable stimulus paradigms. Furthermore, VR stimulus softwareshould make possible highly realistic virtual environments, but freeavailable game design software does not provide the necessarycomplexity and quality in the graphical presentation. Hence, theenvironment looks very artificial and subjects never experience theessential immersive impressions (Regenbrecht et al., 1998). For thatreason the professional game engine, that was integrated into ourstimulus framework provided immersive artifical environmentsthat were considered to be very realistic by the volunteers.

Acknowledgement

This work was supported by the Ministry of Education, Saxony-Anhalt, Germany. Grant-number: 5162/AD/0308T.

References

Aguirre GK. Neural components of topographical representation. Proc Natl Acad SciUSA 1998;95(3):839–46.

Aguirre GK, Detre JA, Alsop DC, D’Esposito M. The parahippocampus subserves topo-graphical learning in man. Cereb Cortex 1996;6(6):823–9.

Baecke S, Luetzkendorf R, Tempelmann C, Mueller C, Adolf D, Scholz M, et al. Event-related functional magnetic resonance imaging (efMRI) of depth-by-disparityperception: additional evidence for right-hemispheric lateralization. Exp BrainRes 2009;196:453–8.

Baumann SB, Neff C, Fetzick S, Stangl G, Basler L, Vereneck R, et al. A virtual realitysystem for neurobehavioral and functional MRI studies. Cyberpsychol Behav2003;6(3):259–66.

Beck L, Wolter M, Mungard N, Vohn R, Staedtgen M, Kuhlen T, et al. Evaluation of

spatial processing in virtual reality using functional magnetic resonance imaging(fMRI). Cyberpsychol Behav Soc Netw 2010;13(2):211–5.

Brain Innovation. Turbo BrainVoyager 2012. URL http://www.brainvoyager.com/.DeCharms RC. Control over brain activation and pain learned by using real-time

functional MRI. Proc Natl Acad Sci USA 2005;102(51):18626–31.

2 scienc

D

E

G

G

H

H

L

M

M

M

98 C. Mueller et al. / Journal of Neuro

oeller CF, Barry C, Burgess N. Evidence for grid cells in a human memory network.Nature 2010;463(7281):657–61.

eckhout L. Computer architecture performance evaluation methods. Synth LectComput Archit 2010;5(1):1–145.

arcia-Palacios A, Hoffman HG, Richards TR, Seibel EJ, Sharar SR. Use of virtual realitydistraction to reduce claustrophobia symptoms during a mock magnetic reso-nance imaging brain scan: a case report. Cyberpsychol Behav 2007;10(3):485–8.

embris D, Taylor J, Schor F, Frings W, Suter D, Posse S. Functional magnetic res-onance imaging in real time (FIRE): sliding-window correlation analysis andreference-vector optimization. Magn Reson Med 2000;43:259–68.

offman H, Patterson D, Carrougher G. Use of virtual reality for adjunctive treat-ment of adult burn pain during physical therapy: a controlled study. Clin J Pain2000;16(3):244–50.

ollmann M, Mönch T, Mulla-Osman S, Tempelmann C, Stadler J, Bernarding J. Anew concept of a unified parameter management, experiment control, and dataanalysis in fMRI: application to real-time fMRI at 3 T and 7 T. J Neurosci Methods2008;175(1):154–62.

uehrs M, Mueller C, Bernarding J. A real-time fMRI communication interface forTurbo-BrainVoyager 3.0 using the TBV 3.0 plugin interface. In: Proc. Hum. BrainMapp.; 2011.

aguire EA, Frackowiak RSJ, Frith CD. Recalling routes around London: activation ofthe right hippocampus in taxi drivers. J Neurosci 1997;17(18):7103–10.

aguire EA, Nannery R, Spiers HJ. Navigation around London by ataxi driver with bilateral hippocampal lesions. Brain 2006;129(11):

2894–907.

athiak K, Klasen M, Weber R, Ackermann H, Shergill S, Mathiak K.Reward system and temporal pole contributions to affective evaluationduring a first person shooter video game. BMC Neurosci 2011;12(66). ,http://dx.doi.org/10.1186/1471-2202-12-66.

e Methods 209 (2012) 290– 298

Mathiak K, Posse S. Evaluation of motion and realignment for functionalmagnetic resonance imaging in realtime. Magn Reson Med 2001;45(1):167–71.

Mellet E, Laou L, Petit L, Zago L, Mazoyer B, Tzourio-Mazoyer N. Impact of the vir-tual reality on the neural representation of an environment. Hum Brain Mapp2010;31(7):1065–75.

Posse S, Wiese S, Gembris D, Mathiak K, Kessler C, Grosse-Ruyken M, et al. Enhance-ment of BOLD-contrast sensitivity by single-shot multi-echo functional MRimaging. Magn Reson Med 1999;42(1):87–97.

Regenbrecht H, Schubert TW, Friedmann D. Measuring the sense of presence andits relations to fear of heights in virtual environments. Int J Hum Comput Int1998;10(3):233–49.

Rothbaum BO, Hodges L, Alarcon R, Ready D, Shahar F, Graap K, et al. Virtual real-ity exposure therapy for PTSD Vietnam veterans: a case study. J Trauma Stress1999;12(2):263–71.

Scharnowksi F, Mathiak K, Weiskopf N. Echtzeit-fMRT. Klin Neurophysiol2009;40:214–21.

Thoms A, Mueller C, Bernarding J. A brain–computer interface for navigation througha virtual environment in real-time fMRI studies. In: Proc. Hum. Brain Mapp.;2011.

Weiskopf N, Mathiak K, Bock S, Scharnowski F, Veit R, Grodd W, et al. Principlesof a brain–computer interface (BCI) based on real-time functional magneticresonance imaging (fMRI). IEEE Trans Biomed Eng 2004;51(6):966–70.

Weiskopf N, Sitaram R, Josephs O, Veit R, Scharnowski F, Goebel R, et al. Real-time

functional magnetic resonance imaging: methods and applications. Magn ResonImaging 2007;25(6):989–1003.

Yoo S-S, Fairneny T, Chen N-K, Choo S-E, Panych LP, Park H, et al.Brain–computer interface using fMRI: spatial navigation by thoughts. NeuroRe-port 2004;15(10):1591–5.