Human-Centered Interfaces for Large, High-Resolution Visualization Systems

David VanoniDepartment of Computer Science and Engineering

University of California, San Diegodvanoni@cs.ucsd.edu

AbstractTo address the challenge of visualizing and work-

ing with the massive amounts of digital data producedtoday, large, high-resolution visualization systems havebeen developed that present information to users atscales beyond what is possible with traditional com-puting systems. While these systems provide an excel-lent resource for handling substantial visual datasets,they also present an interesting challenge when consid-ering effective methods for user interaction. Becausestandard desktop interfaces do not inherently scale upfor practical use in these large environments, herein welook at a human-centric approach for designing inter-faces and present a survey of interaction techniques thathave been developed for leveraging the large physicaland visual space afforded by these systems. We observehow these various approaches account for the needsand facilities of the human within large-scale interac-tion spaces and consider possibilities for new interfacesbased on emerging technology trends.

1. Introduction

With the ever-growing amount of digital data pro-duced every day, the question arises of how to ef-fectively visualize, consume, and analyze these data.Large, high-resolution visualization systems offer aunique platform for tackling this problem. These sys-tems typically consist of an array of digital projectorsor monitors combined into an immense display spaceaimed at presenting visual information to users at scalesnot possible with traditional computing systems.

The substantial space provided by large-scale dis-plays offers many advantages when dealing with mas-sive amounts of visual data but also raises interest-ing new challenges when considering how to effec-tively interact with these “human scale” systems [3].

Typical desktop computing interfaces employ a “win-dows, icons, menus, pointer” (WIMP) design accompa-nied by the time-honored keyboard and mouse, but un-fortunately this tried-and-true interaction method failsto scale up and leverage the available space. Effec-tive interfaces for these systems cannot simply evolvefrom scaling up existing techniques, but rather theymust harness the natural tendency of humans, as spa-tially located creatures, to explore and operate withinspace [35].

Interaction design for large, high-resolution displaysystems must not only leverage the affordances of thetechnology but also consider the needs and facilitiesof the human. This design strategy, known as human-centered design, is defined by the International Orga-nization for Standardization (ISO) as an “approach tosystems design and development that aims to make in-teractive systems more usable by focusing on the use ofthe system and applying human factors/ergonomics andusability knowledge and techniques” [28]. Applying ahuman-centric perspective to the design of interfaces forlarge-scale visualization systems provides guidance fordeveloping usable, effective interaction techniques.

This paper explores various approaches that havebeen developed for interacting with large display sys-tems, focusing on human usability. We will begin withan overview of large, high-resolution visualization sys-tems, identifying both the usability benefits and inter-action challenges that have been observed when usingthese systems. We will then discuss two specific inter-action themes that arise in discussions of interface de-sign for these systems: embodied interaction and ubiq-uitous computing. This will be followed by an in-depthinvestigation of multiple categories of interaction tech-niques focusing on how they rely on embodied interac-tion and ubiquitous computing to provide effective in-terfaces. Finally, we consider opportunities for futureresearch based on emerging technology trends.

1

2. Large, High-Resolution VisualizationSystems

Large-scale visualization systems have been in ac-tive development for over two decades. One of the firstsuch systems to gain popularity was the CAVE Auto-matic Virtual Environment (CAVE) [15]. The originalCAVE system consisted of a cube-shaped room madeup of 3-meter by 3-meter rear-projection screen walls.While this system provided an impressive immersiveexperience, it suffered from the limited display reso-lution of the projectors. Since then, various CAVEsystems have emerged, such as the NexCAVE, whichare powered by multiple LCD display panels providingvastly increased pixel density [19].

The NexCAVE represents just one of many visu-alization systems that have adopted the approach ofcombining multiple commodity displays into a uni-fied visualization platform [20]. Beyond benefitingfrom improved maintainability and increasingly afford-able components such as desktop computer monitors orhigh-definition television displays, these tiled displaysystems aim to provide unprecedented pixel real estatefor visualization tasks. Over the past decade, variousframeworks have emerged for driving the clusters ofcomputers powering these pixels, such as SAGE [54],CGLX [21], and DisplayCluster [33]. These frame-works provide different approaches for delivering vi-sual content to the displays and managing a varietyof input devices used to interact with and managethe content displayed. With multiple software frame-works available and established methods for construct-ing tiled display systems [44], researchers have beenable to develop impressively large visualization sys-tems. Currently, the world’s highest resolution tileddisplay system, Texas Advanced Computing Center’sStallion (Figure 1), features a 16× 5 array of 30-inchDell LCD monitors for a combined resolution of 328megapixels [66].

As large display systems continue to appear inmore and more environments, numerous applicationshave been developed that aim to take advantage ofthe immense physicality and high visual detail af-forded by these systems. Such applications range fromdigital brainstorming [24] to visualizing and compar-ing high-resolution multi-spectral and geospatial im-agery [50, 73] to interactive museum installations [1],among many others [45]. These applications target aunique platform that provides various usability benefitsbut also results in a number of interaction challengescompared to that of a typical desktop or laptop com-puter system.

Figure 1: TACC’s Stallion, the world’s highest resolu-tion display system weighing in at 328 megapixels [66].

2.1. Usability Benefits

Large, high-resolution visualization systems pro-vide unique opportunities for enhanced usability due totheir high visual resolution and the substantial physicalspace available around them. Space plays a vital rolewhen interacting with these systems, both in terms ofthe floor space in front of the display in which users canact and the screen space used for displaying visual infor-mation. Because of our physical presence in the world,the way in which we manage the space around us is “anintegral part of the way we think, plan and behave,” al-lowing us to simplify choice, simplify perception, andsimplify internal computation [35].

The high resolution afforded through these displaysystems naturally presents a “focus+context” interfaceto the user. A user can easily focus on the details of aspecific region by moving closer to the display and thensimply step back to obtain an overview of the entire vi-sualization. This natural interaction scheme allows auser to maintain an understanding of the context of spe-cific details within the larger visual space because of thecontinuous transition between the two. Focus+contextinterfaces have been shown to provide both better per-formance and higher subjective satisfaction comparedto other techniques such as overview+detail and zoom-ing/panning which involve a visual disconnect betweenthe detail and overview viewpoints [10].

Utilizing this focus+context interaction techniqueinvolves physically navigating the space in front of thedisplay, and numerous studies have revealed the ben-efits of such navigation compared to virtual navigationtechniques such as zooming and panning. Ball et al. per-formed user studies comparing the use of a tiled high-resolution display with that of a standard monitor andfound that users were able to find and compare targetsfaster within a visualization of finely detailed data whenusing the larger display [5]. The participants preferred

2

using physical navigation with the large display as op-posed to virtual navigation with the small display whichresulted in loss of context and increased frustration.

In a similar experiment, user performance wasstudied using a range of tiled-display sizes [8]. Thestudy found that as display size increased (i.e. ad-ditional display columns were added) virtual naviga-tion decreased and performance times improved. Thelarger displays provided more overview and details atonce which helped to guide physical navigation. Partic-ipants again preferred physical over virtual navigationand were observed to first physically navigate as muchas possible before resorting to virtual navigation.

Additional studies have shown that users also bene-fit from the increased visual space of large displays. Tanet al. performed multiple experiments comparing theuse of a standard desktop monitor to a large wall-sizeddisplay while positioning participants so as to keep thesame visual angle between both displays [65]. Theyfound that users performed better on spatial orientationtasks such as 3D navigation and mental map formationand memory which they attributed to the ability of thelarge display to immerse users within the problem spaceand bias them into using more efficient cognitive strate-gies.

Andrews et al. also observed cognitive benefits ofincreased display space with users of a large, high-resolution workspace [2]. They observed how partici-pants used the large space to support external memoryand organization. The visibility and persistence of ob-jects in space provide cues to the user that aid in retriev-ing information, and the high-resolution of the displayprovides quick access to highly detailed information, re-quiring the user to remember less. The user benefitsfrom reduced cognitive load as fewer context switchesare needed when large amounts of information are avail-able at a glance. The large display space also promotesarranging on-screen items, providing users a “semanticlayer” in which they can encode additional meaning inthe spatial relationships between these items.

2.2. Interaction Challenges

While the increased space made available throughlarge, high-resolution visualization systems can en-hance how we perceive visual information, it alsomakes these systems challenging to control and inter-act with. Kenneth Moreland points out the irony of howlarge-format display systems aim to increase the dataflow from the computer to the human yet tend to simul-taneously reduce the data flow from the human to thecomputer [40].

Various studies [16, 45] have identified multiple

challenges faced when interacting with large, high-resolution display systems, which we summarize here:

Losing track of the cursor. In order to quickly tra-verse a large screen space and avoid having torepeatedly clutch and reposition the input device,higher cursor acceleration is typically used. Thisincreased acceleration makes it difficult to keeptrack of the cursor during movement. A station-ary cursor also poses a problem as it can be hard tofind the cursor in such a large visual space.

Reaching distant objects. With increased screen realestate it becomes harder and more time-consumingto reach distant objects and move objects acrossthe entire screen.

Managing space and layout. Increased display sizeresults in increased space available to position on-screen items. It can become difficult to mentallykeep track of everything that is happening on thescreen at once and can result in complex multi-tasking behavior. With tiled-display systems thedisplay bezels can also result in visual discontinu-ities and complicate layout management.

Failing to leverage the periphery. Larger displays of-fer a much larger periphery that can be leveragedto support user activities in ways not possible onstandard displays.

Transitioning between interactions. Users may wantto work up close to the display when investigat-ing details but then interact from a distance whenperforming tasks involving an overview of the vi-sualization. Ideally, interaction techniques shouldbe able to support a smooth transition between in-teractions for up-close and distant tasks.

Moreland claims that “not until we discover appro-priate interaction paradigms will large-format displaysbecome truly useful” [40]. With unique challenges toaddress and the potential for improved visualizationworkflows, large display systems present an interest-ing opportunity for studying and developing human-centered interfaces.

3. Interaction Themes

The availability of increased interaction and visual-ization space provided by large, high-resolution displaysystems sets these systems apart from standard comput-ing interfaces. Andrews et al. observe that these sys-tems are “human scale” and thus the physicality of thehuman body is an important element when considering

3

interaction [3]. Understanding how humans, as spatiallylocated creatures, can utilize this space to effectively in-terface with the system presents an interesting researchchallenge. Here we explore two interaction themes thatarise in discussions of interface design for large-scalevisualization systems: embodied interaction and ubiq-uitous computing.

3.1. Embodied Interaction

Paul Dourish presents one definition of embodiedinteraction as “interaction with computer systems thatoccupy our world, a world of physical and social real-ity, and that exploit this fact in how they interact withus” [22]. Embodied interaction stresses the connectionbetween the physical body and the mind and focuses onhow humans generate and communicate meaning andunderstanding through physical interactions. Interfacesthat utilize embodied interaction principles aim to en-hance performance and insight by exploiting embodiedresources such as motor memory, proprioception, spa-tial memory, peripheral vision, and optical flow [6, 7].Research into the use of large, high-resolution visual-ization systems has revealed various ways in which hu-mans benefit from the use of these embodied resources.

Ball et al. propose that the performance benefitsthey observed due to physical navigation give supportfor embodied interaction theory; users understand thedisplay as a physical real-world object in their interac-tion space that they can navigate with respect to [8].“By better utilizing embodied resources such as spatialmemory, proprioception, and optical flow, people canmore efficiently navigate large information spaces withless disorientation, thus enhancing performance by alle-viating the cognitive resources to focus on the analytictask at hand” [6].

Andrews et al. claim that large displays also helpto ease cognition by enabling users to arrange objectsin space and use the perceptual system to recognizemeaning in arrangements of objects rather than havingto memorize these characteristics and relationships [2].They note that the ability to arrange objects also har-nesses the brain’s ability to encode location informationalongside non-spatial information such as text.

Beyond the cognitive benefits seen as large displayusers take advantage of space, various interaction tech-niques and devices also aim to leverage users’ embod-ied resources. For example, devices that enable the userto interact with on-screen virtual items by touching orpointing at them present a natural interaction approachthat mimics how humans interact with physical objectsin our everyday lives. This approach attempts to makethese devices and interactions easier to learn how to use

by building upon our experience with using our bodiesto interface with the physical world [4].

3.2. Ubiquitous Computing

Mark Weiser proposes that “the most profoundtechnologies are those that disappear,” enabling us touse them without thinking and focus beyond them onnew goals [71]. By providing multiple intuitive waysto interface with computing systems that integrate withour environment, we can be empowered to more easilyperform tasks and worry less about the specific tech-nologies involved in these tasks. Ubiquitous computingis the concept that computing devices can be locatedanywhere and everywhere and can take make differentforms, embedded into the natural human environmentso that they are both “invisible” yet always accessible.

Large, high-resolution visualization systems havethe potential to be an important component involvedin ubiquitous computing environments. As these sys-tems present a large space for non-conventional inter-action, users can leverage multiple devices of variousform factors that provide different modalities for inter-action. In the literature this form of interaction is of-ten referred to as “multimodal” interaction. Various re-search projects have developed multimodal interactionenvironments, such as the Stanford iRoom [32] and theWILD Room [11], that utilize large displays systemsalong with a combination of devices used to both con-trol on-screen contents as well as supply new contentto be displayed. These environments are meant to behighly collaborative and invite multiple users to interactwith the system together using the interface that bestsuits their current task.

The themes of embodied interaction and ubiquitouscomputing provide insight into how technologies andinterfaces to these technologies can be designed so thathumans can use them as effectively as possible. Suchinterfaces must leverage our understanding of how weinteract with the physical world and integrate into thisworld as seamlessly as possible.

4. Interaction Techniques

The increased physical presence of large displaysystems poses an interesting challenge when consider-ing effective methods for interaction. These systems areinherently different from standard desktop displays, andthus standard desktop interaction techniques can proveinadequate. The keyboard and mouse have long beenstaples of desktop computing, optimized for the tradi-tional “windows, icons, menus, pointer” (WIMP) userinterfaces found on all modern consumer operating sys-

4

tems. Unfortunately, applying this approach to large-scale display systems restricts the user from movingaround in the physical space, which is one of the majorbenefits of these environments. The mouse is limitedto 2D movement and provides only a small number ofinput buttons, whereas humans can perform many morecomplex gestures with their bodies (e.g. pointing, press-ing, grasping, rotating, snapping, and other movementsin full 3D space). Traditional input devices prohibitusers from taking full advantage of the 3D interactionspace available with large visualization systems.

As such, new forms of interaction are required tofully realize the potential available through these sys-tems. Interfaces for wall-sized displays have been atopic of research since the early 1990s when researchersat Xerox PARC developed Liveboard [23], a large inter-active display designed to support collaborative groupmeetings and presentations. The Liveboard system actsas a digital whiteboard which allows interaction througha stylus, enabling users to digitally draw on the boardand interact with on-screen items. With the goal of en-abling anyone to easily walk up and start using the sys-tem, Liveboard represents one of the first attempts atexploring large display interfaces with a specific focuson human usability.

In the remainder of this section we explore vari-ous interaction techniques that have been developed forinterfacing with large-scale visualization systems. Weobserve how different techniques attempt to address theinteraction challenges inherent in these non-traditionalworkspaces and how they incorporate the concepts ofembodied interaction and ubiquitous computing to en-hance the user experience.

4.1. Pen Input

As we saw with Liveboard, pen-based input wasone of the earliest forms of interaction developed forlarge-scale display systems. Pen input is designed toprovide direct interaction with the screen and can beseen as a natural evolution of the WIMP paradigm ap-plied to larger displays. Rather than using a mouse ona fixed surface, pen input enables the user to controlthe pointer of a WIMP-style user interface by operatingdirectly on the display, leveraging the physical spaceavailable. It provides a very simple and natural meansof interaction that can easily be adopted by users be-cause of the familiarity with analog pens used in every-day tasks.

While the general concept of pen-based input issimple enough, various usability issues arise when putinto practice. Elrod et al. designed the Liveboard penwith several input buttons, much like those found on

Figure 2: Using the FlowMenu to perform a zoomcommand via a continuous pen stroke. (Figure takenfrom [25].)

a standard mouse, but observed that users found thesebuttons to be awkward and avoided using them. Thisled to the conclusion that input should be based solelyon the pen’s position and when it comes into contactwith the screen. Reducing the physical input channelsin this way means that either the graphical interface dis-played must provide additional options for triggeringthe various functions and states previously performedthrough the buttons, or new approaches, such as gestureinterpretation, must be implemented.

4.1.1. Minimizing Visual Clutter. Attempting to fol-low the WIMP paradigm for pen input could result in amultitude of on-screen widgets needed to perform vari-ous functions. These widgets add undesired visual clut-ter to the display and quickly become cumbersome toreach for and activate via the pen. In an effort to addressthese issues Guimbretiere et al. developed the Flow-Menu [25], a visual command-entry system that pro-vides a minimally invasive graphical interface for flu-idly performing a number of functions with simple penstrokes. The FlowMenu is presented as a radial menuthat provides a hierarchy of options which can be tra-versed and activated by drawing pen strokes throughthe desired menu items (Figure 2). Rather than havinga collection of graphical widgets continuously clutter-ing the screen, the FlowMenu condenses the necessarycommands into a minimal interface that is optimizedfor pen gestures and only presented when needed. Asusers become more familiar with the system they caneven rely on motor memory to perform “eyes-free” in-teractions for commands involving simple sequences ofstrokes.

FlowMenu presents a clever evolution of menu-based interfaces that is well suited for pen-based inputand applications dominated by a focus on visual con-tent where obstructions should be minimized. How-ever, it also suffers from various limitations such as vi-sual obstruction by the user’s hand and inaccessibilitywhen activated near screen borders. As visualizationsystems continue to grow in scale, the localized inter-action method provided by FlowMenu fails to scale ac-cordingly as items that are out of reach of the user can-

5

Figure 3: Using Gesture Select to select distant objects:User identifies a distant target to select (1), draws an ini-tial stroke in the general direction of the target (2a), con-tinuation marks appear on the targets (2), user contin-ues the initial stroke by drawing the continuation markof the desired target (2b), the target is selected on penup (3). (Figure taken from [14].)

not be easily accessed.

4.1.2. Reaching Distant Objects. Several approacheshave been taken to address the issue of pen-based ac-cess to distant objects on large displays. With the vac-uum [12], Bezerianos et al. provide a virtual “vacuumcleaner” that brings toward it on-screen items that residewithin the arc of influence centered at the point of invo-cation and spanning the entire display. Users invoke thevacuum by touching the pen to the screen and drawingin the direction in which the arc should extend, afterwhich the arc can be expanded or redirected by furtherdrawing. Once the arc is defined, scaled-down prox-ies of the items residing within the arc are brought intoarm’s reach of the vacuum, allowing users to then selectan item via its proxy. Bragdon et al. present a similararc-of-influence technique called Gesture Select [14].Gesture Select allows the user to select a distant objectby drawing an initial stroke in the general direction ofthe target followed by the “continuation mark” corre-sponding to the desired target, as detailed in Figure 3.

User experiments revealed that both of these tech-niques are able to reduce the amount of time required toselect distant objects as compared to unaided selectionwhere users had to physically move first before beingable to select the target via pen. This result comple-ments the observation that users prefer to move as littleas possible to select remote items, even if the selectionmethod incurs a slight overhead.

While these techniques can provide improved se-lection time of distant objects, various issues suggestthat on-screen pen input does not provide an ideal in-terface for working with items spread across large dis-plays. With Gesture Select, users found it difficult todetermine which continuation mark was needed whenthe desired target was either too small, too far away, ortoo close to other targets, resulting in increased selec-

tion time and higher error rates. Additionally, as thevacuum technique gathers scaled-down proxies of on-screen items it changes the scale and spatial arrange-ment of the visualization; this can be disorienting andpossibly negate the benefits of arranging items across alarge spatial canvas to take advantage of visual cues andspatial memory.

The fidelity available through direct pen-based in-put lends itself more useful for localized interactionswhere a user is focused on up-close details within a spe-cific region of the display. As such, pen input is per-haps best suited as a technique that complements otherapproaches which enable easier interaction with the vi-sualization at larger overview scales.

4.2. Touch Input

Similar to pen input, touch-based input providesdirect interaction with screen content but removes theneed for additional devices such as pens. Whereas peninput is limited to a single point of contact and possiblya set of buttons, touch input can leverage the complexmovements that can be performed by the human hand.Multi-touch input, in which gestures can involve multi-ple fingers rather than just a single touch point, can en-able a multitude of interactions. By allowing direct ma-nipulation of on-screen objects, touch input leveragesour familiarity with real-world interactions, making theexperience seem almost instinctual. However, becausetouch input enables such a wide range of possible inter-actions, there is still much research to be done regardingthe best practices for designing intuitive touch-based in-terfaces [27].

4.2.1. On-Screen Touch Detection. Numerous ap-proaches have been developed to enable touch-basedinput on large-scale display systems. Ringel et al.implemented a touch input system for back-projectedSMARTBoard displays using behind-screen infrared(IR) illumination and a video camera equipped with anIR filter [55]. When a user touches the front of thescreen, her fingertip reflects the IR light which is thenvisible within the video feed and processed to deter-mine the touch location. Because this solution onlyworks with projection-based displays, a different ap-proach is required to enable touch interaction on tiled-display systems. One method of doing this is to arrangean array of cameras along the plane of the display. Thet-Frame [61] system uses the known locations of thesecameras and background subtraction-based image pro-cessing to triangulate the position of multiple fingers onthe display surface (Figure 4).

Schick et. al present an interesting extension of this

6

Figure 4: Touch detection via camera-based finger tri-angulation. (Figure taken from [61].)

camera-based approach by generating a full 3D recon-struction of the space in front of the display [58]. Byperforming voxel analysis on the scene captured by thecameras they are able to not only receive touch inputbut also provide users the ability to continue pointingat objects after their finger has left the display surface.Users found that this approach made it much easier toaccess all parts of the screen, alleviating one of the maindrawbacks when using touch input on large displays.

Unfortunately, camera-based touch systems requirecalibration and can suffer from low precision due to thelimited resolution of the cameras. A solution to these is-sues has emerged recently that involves the use of touchdetection “frames” developed by companies such as PQLabs. The PQ Labs Multi-Touch Wall [52] frame can beinstalled along the edges of displays up to 500 inchesdiagonal and can detect up to 32 touch points simultane-ous with an accuracy of 3 millimeters. The frame con-tains thousands of embedded IR LEDs with correspond-ing IR sensors to determine touch locations based on theobstructions of the IR beams. The precision availablewith these frames nicely complements high-resolutiondisplay systems, enabling users to interact with smallobjects and fine details on the screen [72].

4.2.2. Off-Screen Touch Surfaces. As with pen in-put, on-screen touch interfaces do not easily allow forworking with large-scale visualizations at an overviewscale. To address this, various approaches enable touch-based interaction at a distance from the screen. Oneapproach involves the placement of virtual overlays onthe visualization to specify the desired interaction re-gion. Touches are then mapped from the input de-vice to this region, allowing interaction with on-screen

Figure 5: Using a transparent touch frame, users seethe target on the distant display at their fingertips andperceive touching it directly. (Figure taken from [41].)

objects within the region. Input devices for detectinghands have ranged from capacitive touch monitors [57]to overhead camera-tracked desks [38]. Overall usersfind overlays to be intuitive even though there is nolonger direct manipulation of objects via touch. How-ever, it becomes tedious to continually reposition andresize the overlay for the task at hand, whereas withdirect manipulation users can select an object immedi-ately, assuming it is within physical reach.

In an effort to provide the feeling of direct ma-nipulation across an entire large-scale display, Muller-Tomfelde et al. [41] developed a transparent touchscreen that they placed in front of the display, allowingusers to see through the touch area and look at itemson the screen (Figure 5). This presents the illusion thatusers are directly touching on-screen items and allowsthem to easily access any area of the screen. How-ever, touch accuracy is highly dependent upon the user’sviewing angle as parallax comes into play, forcing theuser to stand very still in order to maintain the properalignment between the touch screen and the visualiza-tion.

Fixed, off-screen touch surfaces provide the benefitof interacting at an overview level, but limit the abilityof the user to physically navigate within the space andprohibit the user from performing fine-grained interac-tions with more detailed visualizations.

4.3. Mobile Devices

Within the past several years we have seen a sub-stantial proliferation of mobile devices such as smart-phones and tablets. These devices play a substantial rolein the widespread evolution of ubiquitous computing inwhich technology is easily accessible and found almost

7

everywhere. Due to their popularity, it makes sense toleverage mobile devices as an interaction medium thatis readily available.

In contrast to pen and touch input methods whichcan restrict the interaction space to within proximityof the display, smartphones and tablets free the user tomove about the entire space and benefit from physicalnavigation. These devices come equipped with a varietyof sensors – accelerometers, gyroscopes, microphones,cameras, multi-touch screens, etc. – as well as outputssuch as audio and vibration, enabling numerous possi-bilities for acquiring user input and providing feedback.They also provide a secondary display that can be usedto offload additional controls or information from themain display, alleviating clutter in the visualization andproviding the user a personalized miniature interactionspace.

4.3.1. Eyes-Free Interaction. One approach to mo-bile device-based interaction attempts to minimize theneed to look at the device, allowing the user to fo-cus on the visualization. Various implementations ex-ist that provide this “eyes-free” interaction style, suchas the levels of precision (LOP) cursor developed byDebara et al. [18]. The LOP-cursor leverages the multi-ple input modalities available on smartphones to con-trol an on-screen cursor with two levels of precision(Figure 6). The device’s pointing direction, calculatedbased on input from the on-board motion sensors, isused for coarse-grained placement of a rectangular con-trol canvas on the display. With the canvas positionedin the desired area of control, the user can lock theposition of the canvas by tapping and holding on thedevice’s screen and then perform high-resolution con-trol of the arrow cursor by dragging her finger on thescreen. Satyanarayan et al. [57] present a similar tech-nique using smartphones, but rather than providing ahigh-precision drag-able cursor they use a direct map-ping from the device screen space to the control overlayspace such that touches are visibly mirrored onto thelarge display within the control area.

Aimed at providing additional proprioceptive feed-back with mobile devices, Jansen et al. [30] designedcapacitive tangible controls that they affixed to multi-touch tablets. By adding these physical controls, suchas sliders, users can modify parameters of the visualiza-tion without looking at the device as often, relying moreheavily on tactile perception.

4.3.2. Personal Viewports. As opposed to eyes-freeinteraction, other approaches use the display of the mo-bile device as a key component of the interface to pro-vide a personal viewing window. One of the most basic

Figure 6: Using the LOP-cursor to position a rectangu-lar control canvas and then perform fine-grained cursormovement. (Figure taken from [18].)

solutions involves mirroring the large-scale visualiza-tion onto the device’s screen and providing touch-basedinput on the device [51]. Unfortunately this only allowscoarse-grained input. A zoom ability could be added,but this would detract from the large-scale visuals andoffer little if any advantage over direct on-screen touchinput.

Rather than mimicking the visuals already pre-sented on the main display, a secondary display can beespecially useful when it provides access to additionalinformation beyond the primary visualization. Apply-ing the concept of “magic lenses” [13], we can thinkof mobile devices as virtual windows or filters throughwhich we can view an alternate visualization that com-plements the primary (Figure 7). Advanced augmentedreality algorithms optimized for mobile devices [69]utilize the camera to perform natural feature tracking,enabling the device to determine its location relative tothe display without the need for external tracking sys-tems. This enables each user to have their own personalwindow through which they can augment their view ofthe full-scale visualization [62].

These viewports can also be used as a mechanismfor input. Interface elements available on the mobile de-vice can be used to add annotations to items displayedon the large screen so that users can deal with con-textual information without cluttering the main visual-ization [56]. Another technique developed by Jeon etal. [31] allows users to target on-screen items using thecamera of their smartphone and then move the item by

8

Figure 7: The mobile device acts as a “magic lens” thatprovides an alternative view of the visualization. (Fig-ure taken from [62].)

simply aiming the device at the desired location. Thisapproach provides immediate control of the visualiza-tion to ideally anyone who has a mobile device handy,although further work is still required to make this in-teraction as seamless as possible.

4.4. Pointing

Perhaps one of the most intuitive means of express-ing interest in a remote object is by pointing at it. Be-cause of this, a large body of research has investigatedhow to support pointing interactions with large-scaledisplay environments in which interacting with objectsfrom a distance occurs on a regular basis.

Ideally, pointing techniques will provide users thefreedom to move about the space and not require ad-ditional visual attention so that the interaction feels asclose as possible to natural pointing and does not dis-tract from the visualization. However, numerous chal-lenges arise when trying to develop natural pointingtechniques that can allow both rapid and accurate tar-get acquisition. Kopper et al. [36] identify several is-sues regarding high-precision pointing with large, high-resolution displays:

Natural hand tremor. The hand naturally trembles ata low amplitude, but when pointing from a distancethis can result in an on-screen movement of manypixels. Moving farther from the screen amplifiesthis issue.

Heisenberg effect. When pressing a button on a deviceheld in free-space, the user often slightly and unin-tentionally changes the position and orientation ofthe device. This is known as the Heisenberg effect,and it can cause the cursor to select unintentionallocations when a click occurs.

Mapping varies with distance. When standing close

to the display, changing the angle of the point-ing device results in a smaller on-screen movementthan when standing farther away from the display.This makes it difficult to perform small motionswhen standing far away from the display.

No parkability. With a traditional mouse, the user canstop moving the mouse and the cursor will remainin the same position. Pointing interfaces do notoffer this parkability unless some sort of toggle isused, making it unnatural or tiresome to maintaina set cursor position.

No supporting surface. Using a mouse on a support-ing surface allows the user to rest their handand more easily make fine positional adjustments.Pointing interfaces lack a supporting surface, lead-ing to fatigue and difficulty with performing smallmotions.

4.4.1. Modeless Techniques. Modeless pointing tech-niques involve a single interaction scheme in which theuser is not required to switch between different modesof operation. The most straightforward implementationof such a technique is know as “ray-casting” in whichthe position of the on-screen cursor is determined bythe intersection of a ray extending from the pointing de-vice with the screen. With Lumipoint [17], the ray isphysically defined by using laser pointers that intersectwith a projection-based screen, and the intersection po-sition is determined using cameras located behind thescreen. The pointing ray can also be virtual, extendingfrom devices that are either externally motion trackedor provide their own internal tracking via sensors.

Unfortunately, small items cannot be targeted reli-ably when using absolute ray-casting because of handtremor and limited input resolution [42]. Various filtersand thresholds have been applied to mitigate these is-sues but have limited benefit [68]. Another techniquedefines the virtual ray starting from the user’s eye lo-cation and then passing through the user’s hand. Thisperspective-based cursor has been found to improveperformance for tasks requiring more accuracy, but italso leads to occlusion of the visualization and greaterfatigue [34]. It also requires users to repeatedly switchbetween two very different focal lengths [42].

An alternative to ray-casting is relative pointingwhich involves a transfer function for mapping deviceor body movements into cursor movements. By imple-menting a dynamically adjusted gain within this trans-fer function, relative pointing techniques can providehigher precision while still enabling quick movementacross the entire screen space [42]. However, indi-rect mappings have been observed as being less intu-

9

Figure 8: Dual-mode pointing. Ray-casting used forcoarse-grained input (a). Activating precise mode viabutton press (b). Relative rotational movements controlcursor movements (c). Switching back to coarse modebe releasing button (d). (Figure taken from [42].)

itive [17], and recalibration is required as the relativeoffset between the input device and the cursor grows,forcing the user to repeatedly perform a clutch opera-tion [48].

4.4.2. Dual-Mode Techniques. Due to the limitationsof modeless pointing techniques, dual-mode techniqueshave been developed which aim to provide enhancedprecision by enabling the user to switch between coarse-and fine-grained input modes, as seen in Figure 8. Al-though switching between modes requires additionaltime, dual-mode techniques provide overall perfor-mance improvements when dealing with small targetsand alleviate recalibration issues so that users do nothave to clutch [42].

Kopper et al. [36] present two different implemen-tations of dual-mode pointing. The first, Absolute andRelative Mapping (ARM), works very similarly to thetechnique depicted in Figure 8 except that an additionaldevice is held in the non-dominant hand to perform thetoggle between the two modes so that the dominanthand can more easily perform the selection action. Theyfound that users easily understood how to use ARMfor interaction tasks but occasionally had trouble seeingwhat they were interacting with without moving closerto the display. When dealing with mostly overviewtasks, it was observed that users preferred to stay fartheraway from the screen and only move closer when ab-solutely necessary. To adapt for this, they implementedanother technique called Zoom for Enhanced Large Dis-play Acuity (ZELDA) in which activating the precisionmode displays a zoom window that not only providesfiner-grained cursor movement but also enhances visual

acuity, making it easier to see and work with smallerobjects while standing far from the screen.

4.4.3. Device-Free Techniques. While pointing is anatural mechanism for targeting items at a distance,ideally interfaces for large-scale displays will enable asmooth transition between distant and up-close interac-tion. Hand-held pointing devices can make this tran-sition awkward when the device does not have a clearapplication for up-close interaction with the display.Device-free pointing techniques alleviate this problemand provide a natural transition into close-range inputsuch as touch.

Vogel and Balakrishnan [68] present a device-freepointing implementation that relies on a motion-trackedglove. They propose several techniques for pointing in-cluding a dual-mode approach in which the user per-forms a pointing gesture with her hand for absolute ray-casting and then opens the hand to enable more pre-cise relative movement. To address the absence of inputbuttons, they also develop the AirTap gesture which al-lows the user to perform a click action by moving theindex finger down and back up, similar to how the fin-ger moves when clicking a mouse button.

Banerjee et al. [9] extend this glove-based track-ing with MultiPoint in which multiple points are trackedon either one or both hands. Several activation ges-tures were tested, with users preferring the unimanual“breach” gesture in which moving their hands forwardpast a threshold distance activates interaction with thepointed-at object. This technique essentially providesan invisible multi-touch screen floating in front of theuser, supporting actions such as dragging, scaling, androtating objects. It also maps naturally to multi-touchinput directly on the screen, providing a seamless tran-sition between interaction distances.

Unfortunately, while these techniques aim to bedevice-free they still require users to wear specially de-signed gloves tracked by an expensive tracking system.The RGB camera-based voxel analysis approach dis-cussed earlier [58] provides a truly device-less experi-ence but suffers from limited resolution and thus cannotachieve the precision required for fine-grained pointingtasks. Advances in marker-less 3D tracking [60] can po-tentially enable the high precision necessary for a verynatural device-free pointing technique for large, high-resolution displays.

4.5. Free-Space

Free-space interaction encompasses a variety ofdifferent techniques that make use of the human bodyoperating in space as the main mechanism for input.

10

This class of techniques, which can be thought of asa superset that includes the various body-based point-ing techniques previously discussed, is especially well-suited for large display interfaces as it enables users tofreely navigate the space to their advantage while alsoutilizing embodied resources. It presents many of thesame challenges that arise with pointing such as fatigueand lack of precision but offers a wide design space ofpossible gestures that can leverage users’ experienceswith everyday interactions.

4.5.1. Extending Reach. When interacting at a dis-tance, one of the most salient challenges is communi-cating the desired target to operate on. We have seenthat pointing provides one mechanism for accomplish-ing this, but the question arises as to what other body-based interfaces can enable users to extend their reachto items on the display.

One approach involves mapping the motions of theuser’s hands in 3D space into the 2D motions of twovirtual cursors on the display. Lehmann and Staadt [37]present an “interaction scaling” technique in which thismapping scales dynamically based on the user’s dis-tance from the display such that moving farther fromthe display provides a larger range of on-screen interac-tion space. This scaled interaction corresponds nicelywith the tendency of users to perform coarse-grainedoverview operations from farther away and more pre-cise operations closer up. Unfortunately, it also causescursor drift to accumulate with prolonged use, forcingusers to either trigger a cursor reset or physically moveto the appropriate distance so as to realign the cursorwith their position.

Extending the idea of mapping hands onto the dis-play, Shoemaker et al. [59] develop Shadow Reaching inwhich a shadow of the entire body is projected onto thedisplay (Figure 9), and different virtual lighting tech-niques enable multiple methods of distant reaching. Forexample, the orthographic lighting behavior presents ashadow with minimal distortion regardless of the user’sorientation, enabling precise control, whereas with the“user following” behavior the light source stays posi-tioned directly behind the user such that turning thebody enables reaching to distant areas of the screen.This technique also makes use of users’ spatial mem-ory and proprioception, allowing users to access virtualtools and control surfaces located on their body by sim-ply touching the corresponding body part. The body canalso serve as a container for digital information, provid-ing users a natural sense of ownership for personal filesthat they carry “within their body.”

Figure 9: Shadow Reaching used to access personalfiles stored “within the body.” (Figure taken from [59].)

4.5.2. Freehand. One of the great benefits of touchscreens is their affordance of instant access; anyone cansimply walk up and immediately interact with virtualitems without the need for anything but their hands. Theconcept of freehand free-space interaction extends thisimmediacy beyond the screen into the space, prefer-ably eliminating the need to acquire a device or per-form any calibration. The idealized conceptualizationof such interfaces essentially transforms the user into asort of magical conductor with the power to commandanything on the display with a wave of the arm.

Working toward the goal of “magical” freehand in-teraction involves various practical considerations suchas designing gestures that limit body movements towithin the range of users’ comfort zones, avoiding un-natural poses and reducing fatigue [53]. Because ofthe inherent lack of haptic feedback in freehand inter-actions, alternative forms of feedback – such as visualor auditory cues – can be implemented such that theyhelp guide the user through the task at hand while alsominimizing distraction [26].

Without an input device, physical body movementsmust be interpreted to communicate commands to thedisplay system, requiring the use of body tracking sys-tems. While marker-based tracking systems have beenused for many years, recent advances in marker-lesstracking systems such as the Microsoft Kinect [60] openup new potential for truly freehand input. Hespanholet al. [26] used a Kinect to perform a study comparingdifferent freehand gestures for selecting and rearrang-ing items on a large display based on analogous actionsperformed in the real world (Figure 10). They foundthat users preferred the grabbing gesture and were ableto perform the best with it once they overcame an ini-tial learning curve. While the dwelling gesture providedthe lowest barrier to entry, it also tended to slow down

11

Figure 10: Freehand gestures for selection and rear-rangement: (a) pushing; (b) dwelling; (c) lassoing;(d) grabbing; (e) enclosing. (Figure taken from [26].)

the user and was consequently thought to be an obstaclerather than a tool. The pushing gesture, while success-ful in other contexts such as touch screen input, failsto translate effectively to an environment in which notactile feedback is available.

For pan-and-zoom interactions, Nancel et al. [43]found that two-handed techniques outperformed uni-manual approaches as users had better control over theindividual functions. They also compared linear and cir-cular gestures for zooming and hypothesized that circu-lar gestures would enable finer control and eliminate theneed for clutching. Interestingly however, they foundthat users performed poorly with circular gestures dueto a lack of physical guidance, and linear gestures werebetter understood as they mapped more naturally to theconcept of pulling and pushing an object toward andaway from the user.

Results from these studies indicate that free-spacetechniques perform best when they enable virtual inter-actions that build on users’ familiarity with real-worldinteractions. As humans perform countless differentmovements to perform various tasks, an interesting areaof research still remains on how to best translate theseactions into effective human-computer communication.

5. Moving Forward

Given the number of different interaction tech-niques that have been researched and developed forlarge, high-resolution visualization systems, it is clearthat no one solution will suffice for every scenario.These systems offer an enormous range of interactionpossibilities with much to still explore. Ultimately,the effectiveness of an interface depends on the task athand; thus a truly usable interface for large-scale dis-play environments will ideally enable multimodal inter-action, offering a multitude of ways for users to interactwhile providing a fluid transition between interactiontechniques as tasks change.

The unique availability of space afforded by largedisplay systems provides multiple interaction scales thatusers naturally navigate between when performing dif-

ferent tasks [47]. Multimodal interfaces should buildon this behavior so as to provide the best possible formof input when operating within these different con-texts. By doing so there is also potential to balance thestrengths and weaknesses of individual interaction tech-niques by intelligently transitioning between techniquesbased on the task. For example, if users are inclined tomove closer to the display in order to investigate de-tails, the challenges of high-precision pointing can beavoided by employing a readily available high-precisiontouch detection frame.

Research on multimodal interfaces has been on-going for many years now [32], and as technologieshave evolved so too have the frameworks and infras-tructures that enable this type of distributed interac-tion [11, 29, 64, 70], even to the point of commercial-ization [46]. As digital devices continue to pervade ourdaily lives we edge ever-deeper into a world in whichcomputing is truly ubiquitous, and interfaces shouldcontinue to evolve in order to leverage this. These de-vices provide not only additional platforms for inter-action but also sources of data which can feed the vi-sualizations in question. Enabling personal devices tobecome part of this interactive ecosystem can help in-crease the accessibility of these systems so that any usercan easily and quickly start using them.

Beyond the immense popularity of laptops, smart-phones, and tablets, there has also been an increasinginterest in “wearable computing” with devices such asGoogle Glass [63], the Meta glasses [39], and the MYOarmband [67] recently emerging. These devices couldopen up a whole new range of possible interactions toexplore and integrate with existing techniques, but thechallenge will be to determine if they ultimately en-hance the usability of the overall system.

Another exciting development is the next iterationof the Kinect sensor that will be released later this yearalongside the Xbox One. The original Kinect has beenone of the most popular devices for developers exper-imenting with freehand free-space interaction, and thenew version promises to provide enhanced resolutionand tracking precision that enables it to pick up detailssuch as hand postures and shifts in body weight [49].Traditionally this sort of tracking required expensive,specialized systems involving wearable markers, andthe original Kinect simply did not have the precisionnecessary to enable fine-grained interactions. It will beinteresting to see if the new Kinect can enable usersto simply walk up and start using a large-scale displayfrom a distance, providing both immediate and preciseinteraction that has not yet been possible.

12

6. Conclusion

Large, high-resolution visualization systems offera unique platform for new scales of visual presentationand new forms human-computer interaction. Variousbenefits have been observed while using these systemsin such a way that capitalizes on the available space, yetnumerous challenges also arise when this space must benavigated and controlled effectively. We have surveyeda variety of different approaches for addressing thesechallenges and considered how these interaction tech-niques account for the needs and abilities of the humanactor, specifically with regard to principles of embod-ied interaction and the concept of ubiquitous comput-ing. While individual techniques can provide effectivesolutions for accomplishing certain tasks on large-scaledisplays, developing a truly usable interface requires adiverse ecosystem of interaction mechanisms that workseamlessly together. As new technologies continue toevolve and emerge, future research into interfaces forlarge visualization systems can leverage these advance-ments to provide even more effective methods for inter-action.

Acknowledgments

I would like to thank my Research Exam chair,Charles Elkan, and my committee members, Bill Gris-wold and Jim Hollan, for taking the time to review mywork in this field. I would like to acknowledge ScottKlemmer for kindly volunteering to serve on my com-mittee as well. I would also like to thank my advisor,Falko Kuester, for his input and advice, as well as mygood friends and colleagues at CISA3, Andrew Huynh,Vid Petrovic, Ashley Richter, Joe DeBlasio, John Man-gan, Aliya Hoff, and TW, for their feedback and sup-port. Finally, a very special and heartfelt recognitiongoes to Shirin Abdi for her never-ending encourage-ment during the process of completing this exam.

References

[1] J. Alexander, J. Barton, and C. Goeser. Transformingthe Art Museum Experience: Gallery One. In MW2013:Museums and the Web 2013, 2013.

[2] C. Andrews, A. Endert, and C. North. Space to Think:Large, High-Resolution Displays for Sensemaking. InProceedings of the SIGCHI Conference on Human Fac-tors in Computing Systems, pages 55–64, 2010.

[3] C. Andrews, A. Endert, B. Yost, and C. North. Infor-mation visualization on large, high-resolution displays:Issues, challenges, and opportunities. Information Visu-alization, 10(4):341–355, Aug. 2011.

[4] R. Ball, M. Dellanoce, T. Ni, F. Quek, and C. North.Applying Embodied Interaction and Usability Engineer-ing to Visualization on Large Displays. Technical re-port, Virginia Polytechnic Institute and State University,Blacksburg, Virginia. USA, 2006.

[5] R. Ball and C. North. Effects of Tiled High-ResolutionDisplay on Basic Visualization and Navigation Tasks.In CHI ’05 Extended Abstracts on Human Factors inComputing Systems, pages 1196–1199, New York, NewYork, USA, 2005. ACM Press.

[6] R. Ball and C. North. Realizing embodied interactionfor visual analytics through large displays. Computersand Graphics, 31(3):380–400, June 2007.

[7] R. Ball and C. North. The Effects of Peripheral Visionand Physical Navigation on Large Scale Visualization.In Proceedings of graphics interface 2008, pages 9–16,2008.

[8] R. Ball, C. North, and D. A. Bowman. Move to Im-prove: Promoting Physical Navigation to Increase UserPerformance with Large Displays. In Proceedings of theSIGCHI Conference on Human Factors in ComputingSystems, pages 191–200, 2007.

[9] A. Banerjee, J. Burstyn, A. Girouard, and R. Vertegaal.MultiPoint: Comparing laser and manual pointing as re-mote input in large display interactions. InternationalJournal of Human-Computer Studies, 70(10):690–702,Oct. 2012.

[10] P. Baudisch, N. Good, V. Bellotti, and P. Schraedley.Keeping things in context: a comparative evaluation offocus plus context screens, overviews, and zooming. InProceedings of the SIGCHI Conference on Human Fac-tors in Computing Systems, pages 259–266, 2002.

[11] M. Beaudouin-Lafon, S. Huot, M. Nancel, W. Mackay,E. Pietriga, R. Primet, J. Wagner, O. Chapuis, C. Pil-lias, J. R. Eagan, T. Gjerlufsen, and C. Klokmose. Mul-tisurface Interaction in the WILD Room. Computer,45(4):48–56, 2012.

[12] A. Bezerianos and R. Balakrishnan. The Vacuum: Fa-cilitating the Manipulation of Distant Objects. In Pro-ceedings of the SIGCHI Conference on Human Factorsin Computing Systems, pages 361–370, 2005.

[13] E. A. Bier, M. Stone, K. Pier, W. Buxton, and T. DeRose.Toolglass and Magic Lenses: The See-Through Inter-face. In Proceedings of the 20th annual conferenceon Computer graphics and interactive techniques, pages73–80, 1993.

[14] A. Bragdon and H.-S. Ko. Gesture Select: AcquiringRemote Targets on Large Displays without Pointing. InProceedings of the SIGCHI Conference on Human Fac-tors in Computing Systems, pages 187–196, 2011.

[15] C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. V.Kenyon, and J. C. Hart. The CAVE: audio visual expe-rience automatic virtual environment. Communicationsof the ACM, 35(6):64–72, 1992.

[16] M. Czerwinski, G. Robertson, B. Meyers, G. Smith,D. Robbins, and D. S. Tan. Large display researchoverview. In CHI ’06 extended abstracts on Human fac-tors in computing systems - CHI EA ’06, pages 69–74,

13

New York, New York, USA, 2006. ACM Press.[17] J. Davis and X. Chen. Lumipoint: Multi-User Laser-

Based Interaction on Large Tiled Displays. Displays,23(5):205–211, Nov. 2002.

[18] H. Debarba, L. Nedel, and A. Maciel. LOP-cursor:Fast and precise interaction with tiled displays usingone hand and levels of precision. In 2012 IEEE Sym-posium on 3D User Interfaces (3DUI), pages 125–132.Ieee, Mar. 2012.

[19] T. a. DeFanti, D. Acevedo, R. a. Ainsworth, M. D.Brown, S. Cutchin, G. Dawe, K.-U. Doerr, A. John-son, C. Knox, R. Kooima, F. Kuester, J. Leigh, L. Long,P. Otto, V. Petrovic, K. Ponto, A. Prudhomme, R. Rao,L. Renambot, D. J. Sandin, J. P. Schulze, L. Smarr,M. Srinivasan, P. Weber, and G. Wickham. The futureof the CAVE. Central European Journal of Engineering,1(1):16–37, Nov. 2010.

[20] T. a. DeFanti, J. Leigh, L. Renambot, B. Jeong, A. Verlo,L. Long, M. Brown, D. J. Sandin, V. Vishwanath, Q. Liu,M. J. Katz, P. Papadopoulos, J. P. Keefe, G. R. Hid-ley, G. L. Dawe, I. Kaufman, B. Glogowski, K.-U. Do-err, R. Singh, J. Girado, J. P. Schulze, F. Kuester, andL. Smarr. The OptIPortal, a scalable visualization, stor-age, and computing interface device for the OptiPuter.Future Generation Computer Systems, 25(2):114–123,Feb. 2009.

[21] K.-U. Doerr and F. Kuester. CGLX: A Scalable, High-performance Visualization Framework for NetworkedDisplay Environments. IEEE Transactions on Visual-ization and Computer Graphics, 17(3):320–332, Apr.2010.

[22] P. Dourish. Where the Action Is: The Foundations ofEmbodied Interaction. MIT Press, 2004.

[23] S. Elrod, R. Bruce, R. Gold, D. Goldberg, F. Halasz,W. Janssen, D. Lee, K. McCall, E. Pedersen, K. Pier,J. Tang, and B. Welch. Liveboard: a large interactivedisplay supporting group meetings, presentations, andremote collaboration. In Proceedings of the SIGCHIConference on Human Factors in Computing Systems,pages 599–607, 1992.

[24] F. Guimbretiere, M. Stone, and T. Winograd. Fluid inter-action with high-resolution wall-size displays. In Pro-ceedings of the 14th annual ACM symposium on Userinterface software and technology, pages 21–30, 2001.

[25] F. Guimbretiere and T. Winograd. FlowMenu: Combin-ing Command, Text, and Data Entry. In Proceedingsof the 13th annual ACM symposium on User interfacesoftware and technology, pages 213–216, 2000.

[26] L. Hespanhol, M. Tomitsch, K. Grace, A. Collins, andJ. Kay. Investigating intuitiveness and effectiveness ofgestures for free spatial interaction with large displays.In Proceedings of the 2012 International Symposium onPervasive Displays - PerDis ’12, pages 1–6, New York,New York, USA, 2012. ACM Press.

[27] A. Ingram, X. Wang, and W. Ribarsky. Towards theestablishment of a framework for intuitive multi-touchinteraction design. In Proceedings of the InternationalWorking Conference on Advanced Visual Interfaces -

AVI ’12, page 66, New York, New York, USA, 2012.ACM Press.

[28] International Organization for Standardization. ISO9241-210: Ergonomics of human-system interaction –Part 210: Human-centred design for interactive systems,2010.

[29] R. Jagodic, L. Renambot, A. Johnson, J. Leigh, andS. Deshpande. Enabling multi-user interaction in largehigh-resolution distributed environments. Future Gen-eration Computer Systems, 27(7):914–923, July 2011.

[30] Y. Jansen, P. Dragicevic, and J.-D. Fekete. Tangible re-mote controllers for wall-size displays. In Proceedingsof the 2012 ACM annual conference on Human Fac-tors in Computing Systems - CHI ’12, pages 2865–2874,New York, New York, USA, 2012. ACM Press.

[31] S. Jeon, J. Hwang, G. J. Kim, and M. Billinghurst. In-teraction with large ubiquitous displays using camera-equipped mobile phones. Personal and UbiquitousComputing, 14(2):83–94, Aug. 2009.

[32] B. Johanson, A. Fox, and T. Winograd. The InteractiveWorkspaces project: experiences with ubiquitous com-puting rooms. Pervasive Computing, IEEE, 1(2):67–74,2002.

[33] G. P. Johnson, G. D. Abram, B. Westing, P. Navr’til,and K. Gaither. DisplayCluster: An Interactive Visu-alization Environment for Tiled Displays. 2012 IEEEInternational Conference on Cluster Computing, pages239–247, Sept. 2012.

[34] R. Jota, M. A. Nacenta, J. A. Jorge, S. Carpendale, andS. Greenberg. A Comparison of Ray Pointing Tech-niques for Very Large Displays. In Proceedings ofGraphics Interface 2010, pages 269–276, 2010.

[35] D. Kirsh. The intelligent use of space. Artificial Intelli-gence, 73(1):31–68, 1995.

[36] R. Kopper, M. G. Silva, R. P. McMahan, and D. A. Bow-man. Increasing the Precision of Distant Pointing forLarge High-Resolution Displays. Technical report, Vir-ginia Polytechnic Institute and State University, 2008.

[37] A. Lehmann and O. Staadt. Distance-Aware Biman-ual Interaction for Large High-Resolution Displays.In Computer Vision, Imaging and Computer Graph-ics - Theory and Application, volume i, pages 97–111.Springer Berlin Heidelberg, 2013.

[38] S. Malik, A. Ranjan, and R. Balakrishnan. Interactingwith large displays from a distance with vision-trackedmulti-finger gestural input. In Proceedings of the 18thannual ACM symposium on User interface software andtechnology, UIST ’05, pages 43–52, New York, NY,USA, 2005. ACM.

[39] Meta. Meta Glasses. https://www.spaceglasses.com/, 2013.

[40] K. Moreland. Redirecting Research in Large-FormatDisplays for Visualization. In Large Data Analysis andVisualization (LDAV), 2012 IEEE Symposium on, pages91–95, 2012.

[41] C. Muller-Tomfelde, K. Cheng, and J. Li. Pseudo-directtouch: interaction for collaboration in large and high-resolution displays environments. In Proceedings of the

14

23rd Australian Computer-Human Interaction Confer-ence, pages 225–228, 2011.

[42] M. Nancel and E. Pietriga. Precision Pointing for Ultra-High-Resolution Wall Displays. Technical report, IN-RIA, 2011.

[43] M. Nancel, J. Wagner, E. Pietriga, O. Chapuis, andW. Mackay. Mid-air Pan-and-Zoom on Wall-sized Dis-plays. In Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems, pages 177–186,2011.

[44] P. A. Navratil, B. Westing, G. P. Johnson, A. Athalye,J. Carreno, and F. Rojas. A Practical Guide to LargeTiled Displays. In Proceedings of the 5th InternationalSymposium on Advances in Visual Computing: Part II,pages 970–981, Las Vegas, NV, USA, 2009.

[45] T. Ni, G. S. Schmidt, O. G. Staadt, M. A. Livingston,R. Ball, and R. May. A Survey of Large High-ResolutionDisplay Technologies, Techniques, and Applications.Virtual Reality, pages 223–236, 2006.

[46] Oblong Industries Inc. Mezzanine. http://www.oblong.com/mezzanine/, 2013.

[47] S. M. Peck, C. North, and D. Bowman. A MultiscaleInteraction Technique for Large, High-Resolution Dis-plays. In 2009 IEEE Symposium on 3D User Interfaces,pages 31–38, 2009.

[48] C. Pelling, T. Sko, and H. J. Gardner. Be Careful HowYou Point That Thing: Wiimote Aiming For Large Dis-plays. In Proceedings of the 21st Annual Conferenceof the Australian Computer-Human Interaction SpecialInterest Group: Design, pages 397–400, 2009.

[49] D. Pierce. The all-seeing Kinect: tracking myface, arms, body, and heart on the Xbox One.http://www.theverge.com/2013/5/21/4353232/kinect-xbox-one-hands-on, May2013.

[50] K. Ponto, K. Doerr, and F. Kuester. Giga-stack: Amethod for visualizing giga-pixel layered imagery onmassively tiled displays. Future Generation ComputerSystems, 26(5):693–700, May 2010.

[51] K. Ponto, K. Doerr, T. Wypych, J. Kooker, andF. Kuester. CGLXTouch: A multi-user multi-touch approach for ultra-high-resolution collaborativeworkspaces. Future Generation Computer Systems,27(6):649–656, June 2011.

[52] PQ Labs Inc. Multi-Touch Screen for LCD large screen,plasma TV display - Touch Screen Overlay. http://multitouch.com/product_wall.html, 2013.

[53] G. Ren and E. O’Neill. 3D selection with freehand ges-ture. Computers and Graphics, 37(3):101–120, May2013.

[54] L. Renambot, A. Rao, R. Singh, B. Jeong, N. Kr-ishnaprasad, V. Vishwanath, V. Chandrasekhar,N. Schwarz, A. Spale, C. Zhang, G. Goldman, J. Leigh,and A. Johnson. SAGE: the Scalable Adaptive Graph-ics Environment. In Proceedings of WACE, pages2004–2009, 2004.

[55] M. Ringel, H. Berg, Y. Jin, and T. Winograd. Barehands:Implement-Free Interaction with a Wall-Mounted Dis-

play. In CHI ’01 Extended Abstracts on Human Factorsin Computing Systems, pages 367–368, 2001.

[56] J. Sanneblad and L. E. Holmquist. Ubiquitous Graphics:Combining Hand-held and Wall-size Displays to Inter-act with Large Images. In Proceedings of the workingconference on Advanced visual interfaces, pages 373–377, 2006.

[57] A. Satyanarayan, N. Weibel, and J. Hollan. Using over-lays to support collaborative interaction with displaywalls. In Proceedings of the 2012 ACM internationalconference on Intelligent User Interfaces - IUI ’12, page105, New York, New York, USA, 2012. ACM Press.

[58] A. Schick, F. van de Camp, J. Ijsselmuiden, andR. Stiefelhagen. Extending Touch: Towards Interactionwith Large-Scale Surfaces. In Proceedings of the ACMInternational Conference on Interactive Tabletops andSurfaces, pages 117–124, 2009.

[59] G. Shoemaker, T. Tsukitani, Y. Kitamura, and K. S.Booth. Body-centric interaction techniques for verylarge wall displays. In Proceedings of the 6th NordicConference on Human-Computer Interaction ExtendingBoundaries - NordiCHI ’10, pages 463–472, New York,New York, USA, 2010. ACM Press.

[60] J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp,M. Cook, M. Finocchio, R. Moore, P. Kohli, A. Crimin-isi, A. Kipman, and A. Blake. Efficient Human Pose Es-timation from Single Depth Images. IEEE Transactionson Pattern Analysis and Machine Intelligence, PP(99),Oct. 2012.

[61] A. Soro, G. Paddeu, and M. L. Lobina. MultitouchSensing for Collaborative Interactive Walls. In Human-Computer Interaction Symposium, volume 272, pages207–212. Springer, 2008.

[62] G. Soros, H. Seichter, P. Rautek, and E. Groller. Aug-mented Visualization with Natural Feature Tracking. InProceedings of the 10th International Conference onMobile and Ubiquitous Multimedia, pages 4–12, 2011.

[63] T. Starner. Project Glass: An Extension of the Self. Per-vasive Computing, IEEE, 12(2):14–16, 2013.

[64] D. Stø dle, J. M. Bjø rndalen, and O. J. Anshus. ASystem for Hybrid Vision- and Sound-Based Interactionwith Distal and Proximal Targets on Wall-Sized, High-Resolution Tiled Displays. Human-Computer Interac-tion, 4796:59–68, 2007.

[65] D. S. Tan, D. Gergle, P. Scupelli, and R. Pausch. Phys-ically large displays improve performance on spatialtasks. ACM Transactions on Computer-Human Inter-action, 13(1):71–99, Mar. 2006.

[66] Texas Advanced Computing Center. VisualizationResources. http://www.tacc.utexas.edu/resources/visualization, 2013.

[67] Thalmic Labs Inc. MYO - Gesture control armbandby Thalmic Labs. https://www.thalmic.com/myo/, 2013.

[68] D. Vogel and R. Balakrishnan. Distant freehand point-ing and clicking on very large, high resolution displays.In Proceedings of the 18th annual ACM symposium onUser interface software and technology, page 33, New

15

York, New York, USA, 2005. ACM Press.[69] D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond,

and D. Schmalstieg. Real-Time Detection and Track-ing for Augmented Reality on Mobile Phones. IEEETransactions on Visualization and Computer Graphics,16(3):355–68, 2010.

[70] N. Weibel, A. Satyanarayan, A. Lazar, R. Oda, S. Ya-maoka, K.-U. Doerr, F. Kuester, W. G. Griswold, andJ. D. Hollan. HIPerFace: a Multi-channel Architectureto Explore Multimodal Interactions with Ultra-ScaleWall Displays. In 33rd International Conference onSoftware Engineering (ICSE), pages 1–10, 2011.

[71] M. Weiser. The computer for the 21st century. ScientificAmerican, 265(3):94–104, 1991.

[72] B. Westing, B. Urick, M. Esteva, F. Rojas, and W. Xu.Integrating multi-touch in high-resolution display envi-ronments. In State of the Practice Reports on - SC ’11,pages 8:1–8:9, New York, New York, USA, 2011. ACMPress.

[73] S. Yamaoka, K.-U. Doerr, and F. Kuester. Visualiza-tion of high-resolution image collections on large tileddisplay walls. Future Generation Computer Systems,27(5):498–505, May 2011.

16