c omputer v ision and s ystems l aboratory ssgrr - 2000 universite laval vision/ 1 ssgrr - 2000...
TRANSCRIPT
1
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
SSGRR - 2000International Conference on Advances in Infrastructure for Electronic Business, Science and Education on the Internet,
July 31 - August 6, L’Aquila, Italy
2
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Designing Virtual Environments for Critical Transactions and Collaborative Interventions:
the VERTEX / APIA Framework for Networked, Physics-Compliant Objects
Denis PoussartDenis LaurendeauFrançois BernierMartin Simoneau
Nathalie HarrisonDenis OuelletChristian Moisan
Computer Vision and Systems Laboratory and Interventional MRI Unit, CHUQ, Université Laval
Québec, CANADA
Supported, in part, by grants from NSERC, FCAR, the Institute for Robotics and Intelligent Systems and the Canadian Foundation for Innovation
www.gel.ulaval.ca/~vision/
3
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Critical Interventions represent cases where{errors, delays, lack of optimization}may have very negative consequences
for safetyfor the environmentfor healthfor costs …
In the future, as more and more complex situations arise, we may anticipate that operational support from Virtual Environments will become paramount and prevalent in the
planning training execution phases of delicate tasks
4
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
The inspection, maintenance and repair of hydroelectric facilities is just one example
5
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Computer Graphics
Virtualizing “reality”
HumanMachine
Interaction
Computer Vision
Automated generation of
geometry of objects and scenes
6
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
But something is missing ...
For critical tasks, visual illusion is not sufficient.
There is more to “real things” than just shape, or forms, even if they are augmented with some “behaviors”.
Accurate physical modeling, laws, and simulation capabilities must be integrated within the virtual environment.
Reality includes PHYSICS!
7
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Computer Graphics
Virtualizing “reality”
Computer Vision
HumanMachine
Interaction
Simulation
Physicsactual world
8
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
This opens up a huge question space!
what is relevant to be physically modeled?
what would be appropriate forms of models?
what level of detail?
perhaps multiple levels of detail, depending ...
how to develop scenarios ?
and brings about many integration issues ...
9
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
A project of Phase 3 of the Institute of Robotics and Intelligent Systems (IRIS) of the Network of Centers of Excellence program of Canada.
VERTEX: Virtual Environments: from 3D Representations to Task planning and EXecution
Objective is to optimize the execution of delicate tasks by combining the accurate simulation of actual scenes, tools and processes with advanced human machine interfaces.
CRIM
International Submarine Engineering Ltd.
We are exploring this approach in:
10
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
VERTEX
On site acquisition
11
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Geometric modeling, 3D and photometric
ModelingModels of behaviors
Models of augmented scenes
Tools, materials, processes
VERTEX
On site acquisition
12
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Geometric modeling, 3D and photometric
ModelingModels of behaviors
Models of augmented scenes
Tools, materials, processes
VERTEX
Planning
Task simulation in “VR” mode
Reactive Interaction Predictive evaluation
Task decomposition
Optimized scenarios
Acquisition sur le site
13
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Planning
Task simulation in “VR” mode
Reactive Interaction Predictive evaluation
Task decomposition
Optimized scenarios
Geometric modeling, 3D and photometric
ModelingModels of behaviors
Models of augmented scenes
Tools, materials, processes
TrainingSimulated scenarios
VERTEX
Acquisition sur le site
14
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
ExecutionTask supervision &
Teleoperation in augmented VR
modeReal time control of robot and tools
Geometric modeling, 3D and photometric
ModelingModels of behaviors
Models of augmented scenes
Tools, materials, processes
On site acquisition
Planning
Task simulation in “VR” mode
Reactive Interaction Predictive evaluation
Task decomposition
Optimized scenarios
TrainingSimulated scenarios
VERTEX
15
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
TrainingSimulated scenarios
Geometric modeling, 3D and photometric
ModelingModels of behaviors
Models of augmented scenes
Tools, materials, processes
On site acquisition
VERTEX
Planning
Task simulation in “VR” mode
Reactive Interaction Predictive evaluation
Task decomposition
Optimized scenarios
ExecutionTask supervision & Teleoperation in augmented
VR modeReal time control of robot and tools
16
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Design issue:
User-centric design
Who is the user?What are his / her needs?
Actually, complex interventions typically involve several users, of various types, with different needs, perspectives and internal models
Hix, D., Swan, E., Gabbard, J., McGee, M., Durbin, J., King, T. (1999) User-Centered Design and Evaluation of a Real-Time Battlefield Visualization Virtual Environment. In Proceedings of IEEE Virtual Reality '99
These users, acting cooperatively, might very well be in different locations
17
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
On site Acquisition
VERTEX
Modeling: off-line
Planning:interactive, soft real-time
VR
Execution: hard real-time
AR
18
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
it might just flow out of the action loop,
A key aspect of physics relates to the handling of time. Real time????
Design issue:
But different physical components may require different time resolution: run time optimization requires fine grain control of time.
and during the direct, immediate {supervision. control, execution} of the task (Augmented Reality), timing accuracy is mandatory: this is the realm of hard real time.
during strategic planning activities, it blends with predictive evaluation,
it may relate to factors which impact upon the user’s sense of interactivity, such as latency jitter,
In (critical) VE’s, time has many different flavors:
19
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Design objectives:
The (physics) simulation engine is at the core of the
system
Beside its predictable real-time behavior, the engine should be capable of supporting:
dynamic extensibility (non - stop)
internal coherence, robustness
multi-resolution behavioral modeling
precisely known degradation
20
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Design objectives (cont):
From an implementation point of view, the engine should seek
modularity, reusability
close match to current and foreseeable trends
- Moore’s law
- high-speed networking
capability to operate from heterogeneouscomponents with run-time binding
networked deployment (multiple users, geographica extent of tasks)
21
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Implementation choice:
Use the Common Object Request Broker Architecture (CORBA) as the software bus - the glue - in assembling the Vertex system.
Why? CORBA is “heavy”, with significant overhead ...
True, but as time will unfold
complexity of relevant problems
CPU, network performance will both and
and the benefits of a robust architecture a more and more significant asset. Silicon, bandwidth free!
22
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
network
Actors * Properties * Interactions Architecture
APIA
To insure time accuracy and conformity to physics, we locate the main driving loop in the simulation engine, “away” from the HMI component.
*
23
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
network
APIA
Maintains an on-going representation of the “world”
Implemented on a cluster of COTS (à la Beowulf) Runs (preferably) on hard real-time OS (OS’s)
Physics engine
A Lego-like approach, with hierarchical capabilities
May include heterogeneous components
24
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
network
APIA
Overall management
Scenario authoring
Repository of model objects …
Controller
25
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
network
APIA
I/O links to the actual physical world
Sensors &Actuators
26
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
network
APIA
Multiple and different views / interactions easily implemented
To suit the representation levels required by different types of users
HMI’s
27
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
network
APIA
Provides the physical glue between the components
RT-CORBA provides the logical glue Designed to fully exploit high performance networking QOS, CaNet3
Geographically - distributed computing, users
28
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
CA*net3 IPv6
29
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Implementation choice
Douglas C. Schmidt, Center for Distributed Object Computing,Washington University
Vertex uses ACE™ and TAO™, a CORBAimplementation under development at the Center for Distributed Object Computing,Washington University.
ACE / TAO is designed to support real-time networked applications, with rigorous control of task priorities and QOS.
30
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
VERTEX / APIA is currently being deployed in other areas, such as breast and liver cancer treatment through cryosurgery.
minimally invasive surgery shares many aspects of telerobotics,
a collaborative project with the Imaging Research Unit of Hopital St-François d’Assise (Dr. C. Moisan) and the Finite Element Research Group at Laval.
A generic approach ...
31
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
MRI Acquisition
3D Geometrical and Tissue Modeling
Modeling
Execution Task Supervision & Teleoperation in
AR modeReal-Time Cryogenic
Probe Control
VERTEX
Behavior Modeling
Models of Augmented Scenes
Tools, Materials and Processes
Planning
Task Simulation in VR mode
Reactive Interaction & Predictive Evaluation
Task Decomposition
Optimized Scenarios
TrainingSimulated scenarios
32
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
On-going 3D visualization of cryoprobelocation and orientation
Real-time display of (simulated) cold front spatial distribution
33
Computer Vision and
Systems Laboratory SSGRR - 2000UNIVERSITELAVAL
www.gel.ulaval.ca/~vision/
Current status andfuture work
Video