![Page 1: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/1.jpg)
Collaborative Mixed Reality
Visualization of an Archaeological Excavation
Hrvoje Benko
Department of Computer ScienceColumbia University
MIT talk – January 12, 2005
![Page 2: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/2.jpg)
2
Multidisciplinary Team
![Page 3: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/3.jpg)
3
Motivation
►Excavation is destructive and physically “unreconstructable” process
►Need to preserve as much data as possible for analysis
►Data interpretation happens off-site
►Current tools focus on 2D data and do not incorporate 3D information
►Many experts—collaboration is a must!
![Page 4: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/4.jpg)
4
Archaeological Excavation at Monte Polizzo, Sicily, Summer 2003
Ian Morris, Director (Stanford University)
![Page 5: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/5.jpg)
5
Working in the field!
![Page 6: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/6.jpg)
6
Meshed 3D Model (13 scans)
![Page 7: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/7.jpg)
7
We have a 3D site model, now what?
Real Virtual
![Page 8: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/8.jpg)
8
High Resolution Images
DrawingsField Notes
Panoramic Images
Videos
GIS Data
![Page 9: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/9.jpg)
9
High Resolution Images
DrawingsField Notes
Panoramic Images3D Object Models
3D Site Model Videos
GIS Data
Database
![Page 10: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/10.jpg)
10
Two Problems
►How to combine all this data in one seamless environment?
►How to make it easy to interact with?
![Page 11: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/11.jpg)
11
VITA:Visual Interaction Tool for Archaeology
► Multiple users
► Multiple displays Projected tabletop
Handheld
High-resolution monitor
See-through head-worn
► Multiple interaction devices MERL DiamondTouch
table
EssentialReality P5 gloves
Speech input
6DOF tracker
![Page 12: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/12.jpg)
12
Design Considerations
►Use the most appropriate display for the given data
►Facilitate both human-system and human-human interaction
![Page 13: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/13.jpg)
13
Modular Architecture
AR Module
DT ModuleSCREEN
Module
HANDHELD
Module
AR Module …
DatabaseMessage
Facilitator
2D Modules
3D Modules
![Page 14: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/14.jpg)
14
AR Module Components
Head Tracker
Hand Tracker
(Intersense IS900)
Sony Head-Worn Display
(LDI-D100B)
Microphone
P5 Glove
(Essential Reality)
DiamondTouch Table
Connector
(MERL)
![Page 15: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/15.jpg)
15
Life-size Immersive Exploration
![Page 16: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/16.jpg)
16
3D Multimodal Interaction
► Provide natural interaction mechanism for our 3D environment
► Modalities
Speech: IBM ViaVoice 10
Gestures: EssentialReality P5 glove
Selection statistics: SenseShapes
► Focus on selection
Based on collaboration with Phil Cohen et al. (ICMI 2003) and SenseShapes (ISMAR 2003)
![Page 17: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/17.jpg)
17
VirtualTray
![Page 18: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/18.jpg)
18
Desktop Components
Touch-sensitive
Projective Display
(MERL DiamondTouch)
High-resolution
Display
Handheld Display
![Page 19: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/19.jpg)
19
World-In-Miniature
![Page 20: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/20.jpg)
20
Harris Matrix
![Page 21: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/21.jpg)
21
Enhanced Harris Matrix
![Page 22: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/22.jpg)
22
Enhanced Harris Matrix
![Page 23: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/23.jpg)
23
Cross-Dimensional Hybrid Gestures
►Synchronized 2D and 3D gestures
►Facilitate seamless transition across dimensions
To appear in IEEE VR 2005
![Page 24: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/24.jpg)
24
Cross-Dimensional Hybrid Gestures
Pull
Push
To appear in IEEE VR 2005
![Page 25: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/25.jpg)
25
Cross-Dimensional Hybrid Gestures
Pull
Pin
Drag
Rotate
Push
To appear in IEEE VR 2005
![Page 26: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/26.jpg)
26
Cross-Dimensional Hybrid Gestures
Pull
Connect
Scale
Disconnect
Push
To appear in IEEE VR 2005
![Page 27: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/27.jpg)
27
Handheld Focus-in-Context Display
►Movable high-resolution inset
Tracked by DiamondTouch
Projection suppressed in its bounds
Physical magic lens
![Page 28: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/28.jpg)
28
Tabletop Interaction
![Page 29: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/29.jpg)
29
User Feedback
►Overall very positive reaction
►Archaeologists benefited from:
Temporal–Spatial connection
Aggregated collection of all data
Accurate 3D model
Simple touch-based interactions
►Potential for increased collaboration
![Page 30: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/30.jpg)
30
Room for Improvement►Reduce wires►Reduce weight ► Eye occlusion hinders communication ►Missing data:
More objects, features, notes and pictures More scans during excavation (time-lapse spatial record)
►Missing features: Virtual scale measure (implemented since) Variable site model scaling Improved selection in world-in-miniature
![Page 31: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/31.jpg)
31
Current and Future Work
► Larger Site:
Summer 2004 - Thulamela, South Africa
► Personalized user experience based on expertise
► Environment management
![Page 32: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/32.jpg)
32
Acknowledgments► Special thanks to:
Peter Allen, Alejandro Troccoli, and Ben Smith (CU Robotics Lab)
Ian Morris and Trinity Jackman (Stanford Archaeology Center)
Lynn Meskell and James Conlon (CU Dept. of Archaeology)
Sajid Sadi and Avinanindra Utukuri (P5 glove)
Shezan Baig
Mitsubishi Electric Research Labs (DiamondTouch table)
Alias Systems
Microsoft Research
► Funded in part by NSF ITR Grants IIS-0121239 and IIS-00-82961, and Office of Naval Research Contracts N00014-99-1- 0394, N00014-99-1-0683, N00014-99-1-0249, and N00014-04-1-0005.
![Page 33: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/33.jpg)
Questions?
![Page 34: Collaborative Mixed Reality Visualization of an ... · SenseShapes (ISMAR 2003) 17 VirtualTray. 18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution](https://reader034.vdocuments.mx/reader034/viewer/2022042303/5ece31b16bbfcd25911792ce/html5/thumbnails/34.jpg)
34