z

deti departamento de electrónica telecomunicações e informática universidade de aveiro

Projects in Virtual and

Augmented Reality@DETI

Paulo Dias

RVA 2019-2020

2

Outline

• 3D Reconstruction

• RVA

• VR Frameworks

• RVA@deti

– pSIVE (Platform for Setting up Interactive Virtual

Environments)

– Museu da Cidade de Aveiro

– VR4Rehab (Edição RVA 16-17 – videos)

– Continuous AR

– Bosch

– Produtech

– EdupARk

– …

How to start a presentation – Top 3

• What is Virtual and Augmented Reality

and what do you need to do it?

3

OR

• What are we doing at DETI in Virtual and

Augmented Reality besides teaching the

RVA course?

• 3: Start with a question!

How to start a presentation – Top 3

4

• 2: Shocking Facts

• AR/VR 3.33 Billion USD in 2015

• Expected 133.78 Billion USD by 2021

• Portuguese 217,6 billion USD in 2017

How to start a presentation – Top 3

5

• 3: Tell a story

• How did I come to this area?

• Ola, Eu sou o Paulo Dias!

6

3D reconstruction – Paulo’s PhD 1998 - 2002

7

• Available for small models

Shape Grabber

Minolta Vivid

Marble Statue of Aphrodite scanned with the

VIVID 910 using the rotary stage option

3D reconstruction

8

• For larger models (building, room) one possibility is to use “Laser Range scanners”.

Riegl

Zoller & Frölich

Cyra

SICK adaptado

Departamento de mecânica

3D reconstruction

9

Point cloud

Reflectance

1000 x 175 (Riegl LMS Z210)

3D reconstruction

10

Triangulation

Registration (ICP)

Digital Photos

Texture

IEETA model, 12 range, 12 Photographs

3D Model

3D reconstruction

11

Ieeta

Lab. Mecânica

Assemblée nationale Farmhouse Laveno

Sala dello scrutinio

Adapted Sick (Aveiro) Riegl (Italy)

3D reconstruction

3D Reconstruction – google Tango (discontinued)

12

Ieeta amphitheatre

Gab. 005

PhD - September 2003, 23

13

• Interaction with 3D content ?

14

Why Virtual Reality?

• 3D information is common (3D reconstruction,

games, CAD, Design, etc…)

• Interaction is still mainly 2D – Keyboard and

mouse since 70s’

• Virtual Reality: Attempt to go beyond classical

interaction and immerse user in 3D environment

Apple II – from 1977

15

Reality Augmented Reality Augmented Virtuality Virtual

Mixed Reality

Paul Milgram [Milgram et. al 1994]

Virtual reality Continuum

16

Projecto de 5º ano:

Mário Cruz

Paulo Moreira

2004-2005

Low cost environment for Virtual and Augmented Reality

17

Virtual reality

• Equipment

– Tracker 3DOF (Yaw, Pitch,

Roll)

– HMD SVGA stereoscopic

– Accelerometer– [Electrónica e

Sistemas Médicos / DETI]

pose detection.

• Graphicl libraries

– OpenGL

– VTK (Visualization Toolkit)

– OGRE

18

Augmented Reality

• Equipment

– Virtual reality + Camera

– Video see-through

• AR Toolkit

– Marker detecction

– Superposition of virtual elements

19

Magic book prototype - 2005

Results

20

Challenges in VR/AR

• User interaction with world – Input Devices

• User immersion in world – Output Devices

• =>No unique solution: several attempts

with different solutions

21

Input: Motion trackers

– Magnetic

– Mechanical

– Acoustic

– Optical

– Inertial

– Hybrids (Combination of previous)

Polhemus Fast-Track

Leap

3D Mouse - Logitech

InertiaCube 3 - Intersense

Finger Tracking - ART

Kinect

Phantom

Razor Hydra

22

Output

– Monitors

– Surround screens

– Workbenches

– Hemispherical Displays

– HMD

– Arm Mounted Displays

23

Hardware@deti - Input

– Intersense 3DOF Inertial Tracker [2009 - 1940€]

– Intersense 3DOF Inertial Track. BT [2012 -1437€]

– Wintracker [2009 - 1285€]

– Phantom Omni [2012 – ~3500€]

– Kinect v1.0 (3) [2012 - 300€]

– Kinect v2.0 (1) [2014 - 300€]

– Swissranger [2012 – ~3000€]

– Leap Motion (3) [2016 – 60€]

– Google tango HTC and Lenovo [2015 – ~500€]

– Orbtec Astra [2019 – ~250€]

• Trackers:

– 3DOF wired and wireless

– Wintracker 6DOF

– Google Tango

– Leap Motion

– Kinect 1/2

– WiiMote

– Razor Hydra

– Phantom

24

Virtual Reality - Hardware

25

Hardware@deti - Output

– I-glasses SVGA HMD [2004 - 800€]

– VR 2000 HMD [2012 - 1359€]

– Oculus DK1 [2014 - 420€]

– Oculus DK2 [2015 - 500€]

– Gear VR + Game Pad [2016 - 250€]

– Samsung Galaxy S7 [2016 - 650€]

– HTC VIVE [2016 - 1350€]

– Meta Glasses [2018 – ~1800€]

– Xperia Projector [2018 – ~3000€]

– Myo gesture ArmBand [2018 – ~100€]

• HMD:

– Oculus DK1,2

– HTC VIVE

– Samsung Gear VR

– HMD SVGA (2)• I-glass

• VR 2000

26

Virtual Reality@deti – Hardware

27

Hardware@deti - Input

Hardware session in a few weeks!

28

VR Frameworks Introduction - 2013

• Creating a Virtual Environment

– Cost (Time & Money)

– Specific knowledge

• Several frameworks/tools

– Specific implementation

– Particular architecture

• How to ease?

– Tools for ‘laymen’

– Flexible yet simple

Graphics engines

• OpenSceneGraph– open source C++ cross-platform toolkit for simulators, virtual reality and

games

– several NodeKits that fulfill a variety of purposes, such as particle

systems, text display, special effects, shadow , 3D interactive controls

,…

• OpenSG– real-time graphics programs, e.g. for virtual reality applications similar to

OpenSceneGraph

• VTK– Open Source library based in Object Oriented Programming

– multi-platform (Windows/Unix/Mac)

– C++ implementation with several wrappers for other languages (Tcl/Tk-

Python-Java)

• Many more …

29

Real-time physics engines

• Bullet– Free, open source

– Collision detection

– Soft and rigid body dynamics

• Newton Game Dynamics– Cross platform open source

– Simulates rigid bodies in games and other real-time application

• PhysX – multi-platform game physics

– support several devices (smartphones, GPU, CPU)

– Integrated in several game engines

– Acquired by Nvidia in 2008

– SDK provided to developers for free

• Many more…

30

3D Computer Graphics software

• Blender– free/open-source 3D computer graphics software used for creating

animated films, visual effects, art, 3D printed models, interactive 3D

applications and video games.

– include 3D modelling and several effects: texturing, raster graphics

editing, rigging and skinning, fluid and smoke simulation, particle

simulation, soft body simulation,…

– integrated game engine.

• OpenSpace3D– Free Open Source development platform for interactive real time 3D

projects

– User friendly, possible to create interactive 3D scene without writing any

code.

• 3D Studio Max– 3D Modelling, Animation & Rendering Software

– Autodesk, commercial

31

What else?

• Something missing?

32

Game engines

• Unity– cross-platform game creation system that includes a game engine and

an integrated development environment.

– Graphic working environment that facilitates 3D scene creation and

layouts

– Physics engine that provides collision detection, forces, torque and

physic materials

– Two licensing options Unity and Unity Pro

• Unreal Engine 4– Multiplatform game development toolkit supports state of the art

graphical fidelity

– full access to the source code of the engine

– In 2015 released for free usage with a 5% commission on commercial

products.

– PhysX 3.3 physics engine for physical simulation

33

Virtual Environment Frameworks - not desktop

• CCG Cave setup /Guimarães

34

Virtual Environment Frameworks - not desktop

• inVRs

• VRJuggler

• Vizard

• 3DVIA

• others

35

36

inVRs

– Institute of Graphics and Parallel ProgrammingJohannes Kepler University, Linz, Austria (2006)

– C++

– OpenSG and OpenSceneGraph as a scene graph engine

– Network distributed virtual world

– Pre-defined navigation and interaction techniques

– Configurable via XML

– Mostly used on academic environmentsAustrian/German Universities

37

Source: http://www.invrs.org/app_gallery.php

Safety Training - SAVE

Walkthrough

Entertainment - Space Trash

inVRs Applications

38

VR Juggler

– Iowa Center for Emerging Manufacturing Technology, Iowa State University, Iowa, United States (1998)

– C++

– Supports many graphics Engines – OSG, OpenSG, OpenGL, VTK…

– Huge variety of modules

– Network distribution through NetJuggler

– Extended over time – Ex.: VR JuggLua

– Used for many different purposes – Commercial / Academic

• Mina do Lousal (PT)

• Korea Ocean Research and Development Institute

– Configurable via VRJConf – Backend XML

39

Source:

Korea Ocean Research and Development Institute

Source: http://vrjuggler.org/, Last

VR Juggler

40

Vizard

Vizard Development Edition

Vizard Development Academic - Single License $ 3,990

Vizard Development Academic - SITE License 5 Seats $ 6,990

Vizard Enterprise Edition

Vizard Enterprise Academic - Single License $ 5,990

Vizard Enterprise Academic - SITE License 5 Seats $ 9,990

Vizard Plug-Ins

Vizard ARToolworks Add-on Development Edition $ 570

Vizard ARToolworks Add-on Enterprise Edition $ 7,990

• WorldViz, Santa Barbara - CA, United States

• Pyton Scripts + GUI

• OpenSceneGraph - Possible to expand vizard funcionalities

• Support many comercial Devices

• Physics Engine

• Research & Science (VHIL - Stanford U), Architecture & Construction (Archidimex - Netherlands)

41

Vizard

Applications

IDE

• Dassault Systèmes, Vélizy-

Villacoublay, FRANCE

• Former Virtools

• Closed Engine

• Many GUI Tools to work on models,

animations and scenes

• Price around 9000 USD + 3000 USD/yr

Source:

http://www.3ds.com/fileadmin/PRODUCTS/3

DVIA/3DVIAVirtools/demoshowcase/html/bro

wse.html?br=1

3DVIA Studio

42

Peripherals Interfaces

• Virtual Reality Periphery Network (VRPN)– network-transparent interface between applications

and physical devices used in VR

– client-server approach

– layer of abstraction that classifies inputs into several

categories (Analog, Button, Dial, Force Device,

Sound,Text, and Tracker), allowing to receive generic

input from different devices.

43

Other Frameworks

• FreeVR

• Avango NG

• DIVERSE

• ViRAL

• EON Studio

• …

44

Conclusion – VR frameWorks

• Difficult choice, many solutions

• Complex Architecture – Many Abstractions

• Often not trivial to Install / Configure

45

• Game engine emerging as alternative to

expensive frameworks – Are almost

standard in research today

Download and install – Break - EdupARk & Unity

http://edupark.web.ua.pt/

46

https://play.google.com/store/search?q=edupARk

https://unity.com/

z

deti departamento de electrónica telecomunicações e informática universidade de aveiro

Projects in Virtual and

Augmented Reality@DETI

Paulo Dias

RVA 2019-2020

48

Motivations for VAR projects

• Foster R&D in Virtual and Augmented

Reality areas

• Investigate new interaction and navigation

techniques

• Devise and demonstrate new scenarios for

the use of these technologies

49

Environment to configure RVA environment

Mestrados:

Danilo Souza

João Parracho

João Cardoso

João Pinto

Luis Afonso

Sérgio Eliseu

2013-2016

Interaction in Virtual reality

pSIVE – Danilo Souza 2013/2014

• Platform for Setting up Interactive

Virtual Environments (pSIVE) 2013/2014

– Virtual Visits

– Training

– Marketing

• Easy Creation

• Abstraction Layer

– Users – Frameworks

50

System Requeriments

• Hardware Independent

• Simple Configuration

– 3D Model Loading

– Attatch/Display Multimedia Information

– Technical aspects

– Reusable

• Well Defined Art Pipeline

– CAD softwares -> pSIVE

– Avoid calculations

• Cover part of IEETA’s devices

51

Architecture

• VR Juggler– Qualities met project requeriments

– Active Community and Develop

– Well documented

• OpenSceneGraph (OSG)– Easy-to-work-with solution

– Books available

– Very active project

– Support vast list of formats/software• 3D Studio Max, SketchUp, Maya, Blender...

• Why not Game Engines?

52

Platform Overview

53

Multimedia Module

54

Interaction Module

55

Menu Module

56

Case of Study

PRODUTECH-PTI / Tegopi

57

58

VR for training and formation

Update pSIVE – João Pinto 2014/2015

• Support for newer HMD’s (oculus rift)

– OpenScenGraph with osgOculusViewer

• Kinect support

– VRPN and FAAST

João Pinto 2014/2015

59

Architecture

60

VR – Full body interaction- 2015

61

Interação em RV – Full body interaction

62

Interação em RV – Manipulation

63

Mobile Interaction – Luis Afonso 2015-2016

• Move to Unity

– Oculus, leap and Android support

64

Architecture

65

66

VR Interaction with mobile devices

1- tablet for tracking of marker on HMD - Oculus

2- Leap Motion for hand tracking mounted on HMD

67

VR Interaction with mobile devices

68

User evaluation in Virtual and Augmented Reality

2006-2007

Mestrado em Gestão de Informação:

Angela Pimentel Amado

Projecto de licenciatura:

Jan-Willem Baggerman

Pauline van der Horst

Frank van Huussen

User evaluation

69

Virtual reality - User evaluation

70

Virtual reality - User evaluation - results

• Objective: Test navigation and visualization

• Methodology: Quantitative and qualitative

• 44 users (12 women / 32 men);

2 environments

RV x DV- HMD - Monitor

- Mouse - Keyboard

- Tracker - Mouse

results

- Preference for RV environment;

- RV causes more dyziness and disorientation;

- navigationa and visualization easy in both;

- better performance in DV.

71

2 Projectos 5º ano

Ricardo Casaleiro

Ricardo Seco

2006-2007

FCT Project

AcousticAVE – 2011-2014

Sound in VR

72

Sound in VR

• Auralization in closed spaces• Use of HRTF

• Computation of first order reflections ( ray-tracing)

• Computation of RT 60 with differente materials

• Visualization and auralization in VR

environments

• VTK, sound processing libraries (PortAudio,

libSoundFile, ...)

….(Burdea and Coiffet., 2003)

Stereo vs 3D Sound

73

74

Software to edit

acoustic properties in

3D models to compute

reverberation times

Sound in VR

75

3D reconstruction and auralization

Anta Pintada de Antelas

• Auralization with first order reflections and

HRTFs

Auralization: Reflections

76

Som directo

Primeiras reflexões

reverberação tardia

Tempo

dB

Tempo de reverberação

Atraso do som directo

Atraso das primeiras reflexões

Atraso das reverberação tardia

Auralization final demo AcousticAVE project

77

78

Bolseiros:

Bernardo Marques

João Alves

Tiago Araujo

Luis Afonso

Inês Cruz

2016-…

Augmented and Virtual Reality projects

Mestrados:

Ricardo Silva

Raphael Carvalho

Anna Andreikanich

Rafael Esteves

2017-…

VR4Rehab - Rovisco Pais

79

Continuous AR

80

Task 1: Catch the pumpkin Task 2: Count the spider

Task 3: Content selection Task 4: Information gathering

Continuous AR interfaces

AR for maintenance

82

AR for education

83

• EdupARk

Download and install - EdupARk & Unity

http://edupark.web.ua.pt/

84

https://play.google.com/store/search?q=edupARk

https://unity.com/

Unity Installation - Instalation

85

• Unity Installation

• Unity Account Asset store

Questions?

To do – Next week

• Install Unity

• Create Asset store account

• Remember to select for presentation:

– papers : IEEE VR2018, VRST, ISMAR / VR

– Or framework : Unreal, ARCore,

86

87

The people!• Beatriz Sousa Santos

• Guilherme Campos

• Joaquim Madeira

• Samuel Silva

• Os muitos alunos que trabalharam nos projectos:– Ricardo Casaleiro, Ricardo Seco

– Gustavo Corrente, Ana Cantanhede

– Angela Pimentel

– Jan-Willem Baggerman, Pauline van der Horst, Frank van Huussen

– Mário Cruz e Paulo Moreira

– Rui Palha, Filipe Duarte, Igor Cardoso, Tiago Sousa, João Parracho, João Cardoso

– André Oliveira

– Danilo Souza

– José Mendes, João Rodrigues, Ricardo Silva, José Sequeira, Vasco Santos, Miguel Vicente, Joel Pinheiro, André Alves, Luis Morais

– João Pinto, Sérgio Eliseu, Luis Afonso, João Cardoso

– José Ferrão, Miguel Ferreira, Pedro Coelho, Rafael Almeida, Diogo Carvalheira

– Raphael Carvalho, Anna Andreikavich, Ricardo Silva, Rafael Esteves, Inês Cruz

– João Alves Bernardo Marques,

– ...

Thanks

88

Questions?