![Page 1: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/1.jpg)
Using Tweek to Create Using Tweek to Create Graphical User Interfaces inGraphical User Interfaces in
Virtual RealityVirtual Reality
Patrick HartlingPatrick Hartling
IEEE VR 2003IEEE VR 2003
![Page 2: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/2.jpg)
VR Juggler — http://www.vrjuggler.org/
OverviewOverview
Problem description
Goals for Tweek software
Design overview
Implementation overview
Future work
![Page 3: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/3.jpg)
VR Juggler — http://www.vrjuggler.org/
Problem DescriptionProblem Description
Interaction in VR is primarily spatialNot all interactions map well to this method• Alphanumeric input• Fine-grained interaction• List selection
Per-application custom interfaces• Map abilities of specific devices to interaction• Typically use 3D input devices
Why abandon effective 2D interaction techniques in 3D space?
![Page 4: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/4.jpg)
VR Juggler — http://www.vrjuggler.org/
MotivationMotivation
Why have GUIs in virtual reality?• GUIs are everywhere in the computing world• Most computer users understand how to use
a GUI• Why not have GUIs in VR?
![Page 5: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/5.jpg)
VR Juggler — http://www.vrjuggler.org/
Issues with GUIs inIssues with GUIs inVirtual Reality (1/2)Virtual Reality (1/2)
Positives• Typically more flexible and more extensible
than other input devices–More input methods means less overloading of
button presses, gestures, etc.
• More portable between VR systems than hardware input devices
• Familiarity from desktop experience
![Page 6: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/6.jpg)
VR Juggler — http://www.vrjuggler.org/
Issues with GUIs inIssues with GUIs inVirtual Reality (2/2)Virtual Reality (2/2)
Negatives• No direct physical correspondence
– In VR, a user could grab an object with his/her hand or use a GUI
– No GUI in real world for moving the object
• Remote users may not see GUI [Park00]• Use of the interface may be difficult
– Collision detection– Placement
• Immersive interfaces have to re-invent GUI technology
![Page 7: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/7.jpg)
VR Juggler — http://www.vrjuggler.org/
Previous Work (1/3)Previous Work (1/3)
Many areas of previous work• Widget sets for VR• PDAs as input devices• Floating menus and heads-up displays
Virtual User Interface (VUI)• Originally developed at Iowa State University
[Heath98]• 3D immersive user interface• May be an open source release from Fuel Tech
![Page 8: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/8.jpg)
VR Juggler — http://www.vrjuggler.org/
Previous Work (2/3)Previous Work (2/3)
Java Interface to Virtual Environment (JAIVE)• Used Java interface on handheld computer
[Hill00]• Communicated with C++ applications• Interface could be extended from C++ using
JAIVE protocol
![Page 9: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/9.jpg)
VR Juggler — http://www.vrjuggler.org/
Previous Work (3/3)Previous Work (3/3)
PDA in a CAVE™ system• Kent Watsen, Naval Postgraduate School• Experimented with use of Palm PDA in CAVE
[Watsen99]
3DI Group• Doug Bowman, Virginia Tech• Investigating differences in system interaction
techniques between VE displays• Active on 3DUI mailing list
![Page 10: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/10.jpg)
VR Juggler — http://www.vrjuggler.org/
TweekTweek
Framework for the development of cross-platform, cross-language, scalable GUIs• GUIs communicate with remote applications• GUIs can be use on the desktop, on a PDA, or in an
immersive VR environment
Features• GUI panels are dynamically loaded components
– Panels use standard 2D GUI “widgets”
• Dynamic interface update and extension• Multi-language capabilities
![Page 11: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/11.jpg)
VR Juggler — http://www.vrjuggler.org/
GoalsGoals
Allow GUI scalability• Same GUI runs on the desktop, a PDA, and a
projection-based VR system• Must provide an easily extendable framework for
cross-platform, cross-device GUIs
Interact with VR applications written in any programming language• Fully separate the GUI from the application
Reuse existing GUI technology• We do not want to re-invent scrollbars, menus, etc.
Avoid cluttering immersive space
![Page 12: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/12.jpg)
VR Juggler — http://www.vrjuggler.org/
Same GUI, Multiple Uses (1/2)Same GUI, Multiple Uses (1/2)
PDA in Projection System• Offers force feedback• High-resolution display– Back-lit display is needed
• Widely available– A person with a PDA can walk into the VR system
and get the GUI dynamically
![Page 13: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/13.jpg)
VR Juggler — http://www.vrjuggler.org/
Same GUI, Multiple Uses (2/2)Same GUI, Multiple Uses (2/2)
Desktop simulation• Standard GUI interface used to control VR
software in simulator mode
Immersive 3D space• VRJ VNC incorporates 2D GUI into 3D
visualization–Modeled after 3Dwm (www.3dwm.org) use of VNC
and other VR VNC viewers
![Page 14: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/14.jpg)
VR Juggler — http://www.vrjuggler.org/
Design Overview (1/3)Design Overview (1/3)
Distributed Model/View/Controller
Cube rotation Cube translation
Network Communication
Controller
ModelView
![Page 15: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/15.jpg)
VR Juggler — http://www.vrjuggler.org/
Design Overview (2/3)Design Overview (2/3)
Observer design pattern• Simplified MVC: subject, observer• VR application is the subject
– Maintains user-defined state information– Per-application customizations
• GUI is the observer– Provides a view of subject’s state– GUI interactions manipulate application state
• Benefits– Distinct separation of application and GUI– Allows multiple GUIs to observe the same application
![Page 16: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/16.jpg)
VR Juggler — http://www.vrjuggler.org/
Design Overview (3/3)Design Overview (3/3)
Two-way communication between application and GUI• Application can send data to the GUI
GUI flexibility• A GUI can load any component it needs to
make interaction more effective• Not limited to direct manipulation of
application state
![Page 17: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/17.jpg)
VR Juggler — http://www.vrjuggler.org/
Implementation OverviewImplementation Overview
Language tools• C++ VR applications• Java-based GUI composed of JavaBeans– Beans can be “pushed” to the GUI while the user
is in the virtual space– Dynamic extension of interface
• CORBA communication between C++ and Java
![Page 18: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/18.jpg)
VR Juggler — http://www.vrjuggler.org/
Example Tweek InterfaceExample Tweek Interface
• Multi-purpose tool– Navigation–Map visualization– Data visualization– Information
storage to spreadsheet-friendly format
![Page 19: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/19.jpg)
VR Juggler — http://www.vrjuggler.org/
VRJ VNC: Embedding 2D GUIs VRJ VNC: Embedding 2D GUIs in 3D Spacein 3D Space
• Embed any 2D GUI in 3D world
• Uses VNC• Can be added
to existing VR Juggler applications
![Page 20: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/20.jpg)
VR Juggler — http://www.vrjuggler.org/
VRJ VNC: Presentations in VRVRJ VNC: Presentations in VR• Bring
PowerPoint presentation into the 3D environment
![Page 21: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/21.jpg)
VR Juggler — http://www.vrjuggler.org/
VRJ VNC: Embedding TweekVRJ VNC: Embedding Tweek
• Tweek Java GUI is just another window on a desktop
• Set VNC desktop size to give the desired embedded frame size
• Interaction with the GUI is translated to the desktop via VNC
![Page 22: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/22.jpg)
VR Juggler — http://www.vrjuggler.org/
Future WorkFuture Work
• Use Java Applets to allow Tweek to be used from a web browser–Make use of a familiar interface– Easy access to application GUI for remote
demonstrations
![Page 23: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/23.jpg)
VR Juggler — http://www.vrjuggler.org/
Online InformationOnline Information3DUI• http://www.mic.atr.co.jp/~poup/3dui.html
Tweek• http://www.vrjuggler.org/tweek/• Contains publications, guides, and
programmer references
![Page 24: Using Tweek to Create Graphical User Interfaces in Virtual Reality](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815721550346895dc4c0da/html5/thumbnails/24.jpg)
VR Juggler — http://www.vrjuggler.org/
ReferencesReferences
• [Park00] Park, K. et al. “Lessons Learned from Employing Multiple Perspectives in a Collaborative Virtual Environment for Visualizing Scientific Data”. In Proceedings of ACM Conference on Collaborative Virtual Environments, San Francisco, California, September 10–12, 2000.
• [Heath98] Heath, D. “Virtual User Interface (VUI): A Windowing System for VR”. In Proceedings of 2nd Immersive Projection Technology Workshop, Ames, Iowa, May 11–12, 1998.
• [Hill00] Hill, L., Usability of 2D Palmtop Interaction Device in Immersive Virtual Environment. Master’s thesis, Iowa State University, Ames, IA, 2000.
• [Watsen99] Watsen, K. “A Handheld Computer as an Interaction Device to a Virtual Environment”, In Proceedings of 3rd Immersive Projection Technology Workshop, Stuttgart, Germany, May 10–11, 1999.