intro to vr with unreal engine
TRANSCRIPT
Luis Cataldi - Epic Games
Intro to VR with
Unreal Engine
VFX house Framestore uses UE4 for “Field Trip to Mars”
2
VR with Unreal Engine is everywhere
3
VR with Unreal Engine is everywhereNASA: Building the ISS in Unreal Engine
4
VR with Unreal Engine is everywhere“The Quinn Building” from the South West College in the UK
6
VR with Unreal Engine is everywhereILMxLAB's Darth Vader VR Story Experience Teaser
And much more!
How can we learn to harness the power of
Unreal Engine?
VR Learning Resources for Unreal Engine: Starting Out:
• Oculus Quick Starts• SteamVR Quick Start• Google VR Quick Start• Gear VR Quick Starts
VR Platforms:
• Samsung Gear VR Development• Google VR Development• Oculus Rift Development• SteamVR Development
VR Topics:
• VR Cheat Sheets• VR Best Practices• Motion Controller Component Setup• VR Camera Refactor
VR Learning Resources for Unreal Engine:Video:
• 2015 UE4 - VR and Unreal Engine• Making Bullet Train and Going off the Rails in VR• VR Bow and Arrow Tutorial w/ Ryan Brucks - Training Livestream• Training Livestream - Sam and Wes' VR Stream: Cameras, Multiplayer, Tips and Tricks!• Creating Interactions in VR with Motion Controllers 1-3• Setting Up VR Motion Controllers• VR Networking and 3D Menus• Up and Running with Gear VR• Developing for VR • Integrating the Oculus Rift into UE4
Presentations:
• UE4 VR - Niklas Smedberg• Lessons from Integrating the Oculus Rift into UE4• Going Off The Rails: The Making of Bullet Train
Links:• Sam Deiter - 10 VR tips for Unreal Engine• Tom Looman’s - Getting Started with VR in Unreal Engine 4
VR Learning Resources for Unreal Engine:VR Editor Starting Out:
• Activating VR Mode
VR Editor Guides:
• Setting Up VR Editor from GitHub• Navigating the World in VR Mode• Working with Actors in VR Mode
VR Editor Reference:
• VR Editor Controls• Quick Select Menu• Radial Menu• Transforming Actors in VR• Editor Windows in VR Mode
•Unreal Engine 4.13 Release Notes
•Unreal Engine 4.12 Release Notes
•Unreal Engine 4.11 Release Notes
•Unreal Engine 4.10 Release Notes
•Unreal Engine 4.9 Release Notes
•Unreal Engine 4.8 Release Notes
•Unreal Engine 4.7 Release Notes
•Unreal Engine 4.6 Release Notes
•Unreal Engine 4.5 Release Notes
•Unreal Engine 4.4 Release Notes
•Unreal Engine 4.3 Release Notes
•Unreal Engine 4.2 Release Notes
•Unreal Engine 4.1 Release Notes
The Unreal Engine Release Notes:
Mitchell McCaffrey’s - Mitch VR Labs
• Mitch's VR Lab - an Introduction
• Mitch's VR Lab - Look Based interaction
• Mitch's VR Lab - Simple Teleportation Mechanic
• Mitch's VR Lab - Introduction to SteamVR
• Mitch's VR Lab - Simple Head IK
• Mitch’s UE4 Forum Post
Education Community VR for UE4:
Free Unreal Engine Courses:• Twin Stick Shooter• 3rd Person Power-Up Game with C++• 2D Sidescroller with Blueprints• Endless Runner with Blueprints• Unreal Match 3 Game
Education Community VR for UE4:
Free UE4 Community Youtube.com Learning Channels:• World of Level Design UE4 Fundamentals• Virtus Education series• Unreal Engine 4 Beginner Tutorials• Mathew Wadstein Tutorials• Leo Gonzales Unreal Basics• Tesla Dev Tutorials• UE4 Style Guide
Free UE4 Community Blueprints:• Communication Training - Zak Parrish• Blueprints Compendium - VOLUME II• BP_Compendium.pdf• Network Compendium
Free UE4 Community VR Learning Channels:• Unreal Engine VR Curriculum
Free UE4 Community ArchViz Learning Channels:• Architectural Visualization Tutorials
Education Community VR for UE4:Paid Elearning Courses:
• Unreal Engine 4: The Complete Beginner's Course• Learn to Code in C++ by Developing Your First Ga
me• Complete Introduction to Unreal 4• An Introduction to Creating Realistic Materials in U
E4• Master Blueprints in Unreal Engine 4 - Endless Ru
nner• Create a Helicopter Game Control System in Unre
al Engine 4• Unreal Essential Training - Lynda.com• Unreal: Learn Lighting - Lynda.com• 3dmotive - Unreal Engine courses• Pluralsight - Unreal Engine courses
Much more coming soon.
Improved metadata support● Skill level
● Engine version
● Sitemap filters
● Checkpoints
Learning Resources for Unreal Engine:
Learning Resources for Unreal Engine:
Getting the most value from the UE4 Launcher Learn Tab
Learning Resources for Unreal Engine:
Getting the most value from Content Examples
Recent Developments
Recent Developments
New: VR Project Template
We’ve added a new project template designed for Virtual Reality on desktop and console.
Recent Developments
New: VR Project Template
The template can be selected from the Blueprint tab as a new project is created.
Recent Developments
New: VR Project Template
The motion controller template provides examples for object interaction and manipulation as well as point based teleportation.
Recent Developments
At Epic, we drive engine development by creating content.
We use these as real-world test cases to drive our framework and optimizations to the engine.
Recent Developments
Instanced Stereo Rendering
Lets us use a single draw call to draw both the left and right eyes, saving CPU (and some GPU) time.
• Currently works on PC, and coming soon on PS4 and mobile platforms• Enable it in your project’s settings
Recent Developments
Instanced Stereo Rendering
Instanced stereo is available now for the deferred (desktop) renderer and the PS4.
We’re also implementing multi-view extension support for the forward (mobile) renderer, and improving PS4 support to better utilize the hardware.
Recent Developments
Hidden And Visible Area Mesh
We draw meshes to cull out geometry that you can’t see, and only apply post processing where you can.
For the deferred renderer, the visible area mesh is a bigger optimization!This is specific per-platform.
Recent Developments
Hidden And Visible Area Mesh
The Hidden Area Mask uses a mesh to early-out on pixels that aren’t visible in the final image.
Recent Developments
Hidden And Visible Area Mesh
The Visible Area Mask uses a mesh to constrain our post processing to the visible pixels only.
Recent Developments
Camera Refactor
As of 4.11, we’ve completely rewritten the camera system in order to make developing much easier!
• Camera Components now move exactly as the real HMD is moving
• You can attach components (meshes, UI, etc) directly to the camera component!
Platform Support:
Platform Support
As of 4.12, we support the following platforms out of the box:
• Oculus Rift• Steam VR (including the HTC Vive)• PSVR• OSVR (preview)• Samsung Gear VR• Google Daydream• Leap Motion
Platform Support
Create once, and deploy anywhere.
Mobile
Desktop / ConsoleOculus RiftHTC Vive / Steam VRPSVROSVR
Samsung Gear VRGoogle Daydream
Platform Support
All of these platforms go through UE4’s common VR interfaces, so you can make your content once, and deploy it anywhere.
• Unified Camera System• Motion Controller System• Optimized rendering paths• Low-latency optimizations
Oculus
Vive PSVROSVR
New native UE4 VR Forward Rendering options with multi-sample anti-aliasing (MSAA)
“Current state-of-the-art rendering often leverages screen-space effects, such as screen-space ambient occlusion (SSAO) and screen-space reflections (SSR). Each of these are well known for their realistic and high-quality visual impact, but they make tradeoffs that aren’t ideal in VR. Operating purely in screen-space can introduce incorrect stereo disparities, which some find uncomfortable.
Forward rendering lets us use multi-sample anti-aliasing (MSAA), adds arithmetic to our texture-heavy shaders (and removes GBuffer writes), removes expensive full-screen passes that can interfere with asynchronous timewarp, and offers a moderate speedup over the deferred renderer. Switching to a forward renderer has also allowed the easy addition of monoscopic background rendering, which can provide a substantial performance boost for titles with large, complex distant geometry.
https://developer.oculus.com/blog/introducing-the-oculus-unreal-renderer/
DESKTOP FORWARD RENDERERComing this Fall*
There are advantages to both!
The UE4 Deferred Render is a full-featured workhorse, but takes a bit of skill to fully leverage. Temporal Anti Aliasing limits how sharp your image can be.
A new UE4 Forward Renderer will be a specialized renderer, with less features, but faster baseline. Multi Sampling Anti Aliasing (MSAA) is the sharpest solution for anti-aliasing.
Design Considerations for VR
One of the biggest issues for working in VR is Motion/Simulation Sickness.
How is it caused?
en.wikipedia.org/wiki/Virtual_reality_sickness
Sensory conflict theory believes that sickness will occur
when a user's perception of self-motion is based on
incongruent sensory inputs from the visual system,
vestibular system, and non-vestibular proprioceptors, and
particularly so when these inputs are at odds with the user's
expectation based on prior experience.
Five typical causes of Motion/Simulation Sickness in VR
Read more about it
1. Non-forward movements• No unnatural movements
2. Awareness of Vection• When a large part of the visual field moves, a viewer feels like
he has moved and that the world is stationary
3. The feeling of accelerations
4. Too much camera YAW
5. Helped by adding a static reference frame
Things we CAN DO in Unreal Engine to improve VR Games and Experiences
You MUST maintain framerate
For the VR experience to feel smooth, your game needs to run on 75 hz (Oculus DK2) or even 90 hz. (HTC Vive and Oculus CV1) depending on the device. To see the current framerate type in “stat fps” or “stat unit” (for more detailed breakdown) in your console when running the game.
Here is where VR Instanced Stereo Can Help“Basically, we’re utilizing hardware instancing to draw both eyes simultaneously with a single draw call and pass through the render loop. This cuts down render thread CPU time significantly and also improves GPU performance. Bullet Train was seeing ~15 – 20% CPU improvement on the render thread and ~7 – 10% improvement on the GPU.” – Ryan Vance.
To enable this feature in 4.11 and above, go to your Project Settings and look for “Instanced Stereo” under the Rendering category.
Things to keep in at the front of your mind:
Check your performance constantly to ensure that
you are hitting your VR performance targets.
Things to keep in at the front of your mind:
○ Maintain a very simplistic approach to making your content.
○ Minimize complex shaders as best possible.
○ Add detail to the mesh within reason in lieu of relying of complex
shaders for surface details.
Things to keep in at the front of your mind:
LOD's and aggressive culling are a must to ensure
that you are hitting your VR performance targets.
Known issues and possible workarounds:
Screen Space Reflections(SSR)
SSR will work in VR but may not give you the results that you
want. Instead consider working with reflection probes.
Known issues and possible workarounds:
Normal Mapping Issues
When viewing Normal maps on objects in VR, you will notice that they do not
have the impact that they might have once had. This is because normal mapping
does not account for a binocular display or motion parallax. Because of this,
Normal maps have a tendency to look flat when viewed with a VR device.
Known issues and possible workarounds:
Parallax Mapping
Parallax mapping takes Normal mapping to the next level by accounting for depth cues,
Normal mapping does not. A Parallax mapping shader can better display depth
information, making objects appear to have more detail than they do. This is because no
matter what angle you look at, a Parallax map will always correct itself to show you the
correct depth information from that viewpoint. The best use of a Parallax map would be
for stone pathways and fine detail on surfaces. (Parallax maps often look better up close
whereas many normal maps will look passable at average distances.)
Known issues and possible workarounds:
Tessellation Shader Displacement
Tessellation Shader Displacement will displace 3D Geometry in real time by
adding details that are not modeled into the object. Tessellation shaders do a
great job of displaying information because tessellation shaders actually create
the missing detail by creating more vertices and displacing them in 3D Space.
Launching VR Preview:
Testing out your VR set is very straightforward, simply select “VR Preview” from the Play dropdown
button. By default the head tracking will work right away without any changes to your existing project or
template.
** Also new for 4.13 is Play from VR Editor, which can be enabled in Editor Preferences.
GPU Profiling:To capture a single frame with GPU timings press Ctrl+Shift+, or type in “profilegpu” in the console. This command collects accurate timings of the GPU, you will find that certain processes are a heavy burden on the framerate (Ambient Occlusion is one common example) when using VR.
See GPU Profiling & Performance and Profiling for documentation.
GPU Profiling:Unreal Engine 4.13 now also has a new experimental Real Time GPU Profiler!
•To enable it, enter r.GPUStatsEnabled 1 in the console•Then type stat gpu to launch it
Disable Heavy Post-Processors:Due to the demanding requirements of VR many of the advanced Post Processing features that you normally use should be disabled. To accomplish this, first you can consider disabling heavy post processors in your Project Settings first, or use the below sets per attribute.
• Add a Post Process(PP) volume to your level if there is not already one there.• Select the PP volume and enable the Unbound option so that these settings will be applied to the entire level.
• Expand each Settings of the Post Process Volume and disable any undesired active PP settings by enabling that property by clicking on it and then set the value from the default, usually 1.0, to 0 to disable the feature.
• Consider first disabling the biggest offenders to VR performance like Lens Flares, Screen Space reflections, Screen Space Ambient Occlusion, and anything else that might be impacting performance.
• While some of the features are disabled by settings in your .INI files, this ensures that performance will not be affected if the .INI is removed by mistake.
UE4 – Lighting for VR
• Dimmer lights & colors can help reduce simulation sickness.
• Make sure your Stationary / Dynamic Lights do not overlap.
• If using Dynamic Shadows, only have one shadowing light.
• Use Stat LightRendering to see current lighting cost.
• Use Static Lighting over Stationary or Dynamic, baked lights are the best option for VR
environments.
• Profile, Profile, Profile to ensure you are maintaining performance goals.
Fake shadows Wherever You Can!!Using cheats like fake blob shadow drop to simulate dynamic shadows are a good consideration for keeping VR project running at frame.
Blob shadow example. Image by Eric Chadwick
UE4 – Effects for VR• Mesh based VFX work the best for VR.
• Camera Facing particles do not hold up well in VR on their own due to the stereoscopic
view.
• The Dither Temporal AA Material Function can make Opacity masked objects look like
Translucent ones.
• Local Space rotation does not look correct in VR.
UE4 – Environments for VR
• Use Reflection probes instead of screen space reflections.
• Again… Texture Blob shadows are a cheap alternative to dynamic shadows.
• The Merge Actor Tool can help cut down on Static Mesh draw call without having to do work
outside of UE4.
Some very important things we all need to know about Unreal Engine.
The Unreal Engine Framework
The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController Class
GameState Class
PlayerState Class
The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController Class
GameState Class
PlayerState Class
The GameMode is the definition of
the game.
● It should include things like
the game rules and win
conditions.
● It also holds important
information about:
○ Pawn
○ PlayerContoller
○ GameState
○ PlayerState
The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController Class
GameState Class
PlayerState Class
The Pawn class is the base class
of all Actors that can be controlled
by players or AI.
● The Pawn represents the
physical location, rotation,
etc. of a player or entity
within the game.
● A Character is a special type
of Pawn that has the ability to
walk around.
The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController Class
GameState Class
PlayerState Class
A PlayerController is the interface between the Pawn and the human player controlling it.
● The PlayerController decides what to do and then issues commands to the Pawn (e.g. "start crouching", "jump").
● Putting input handling or other functionality into the PlayerController is often necessary.
● The PlayerController persists throughout the game, while the Pawn can be transient.
The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController Class
GameState Class
PlayerState Class
The GameInstance is a class who’s
state persists switching of levels,
game modes, pawns etc. Where
classes like GameMode or
PlayerController are being reset
and data stored in those classes is
removed.
The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController Class
GameState Class
PlayerState Class
A PlayerState is the state of a
participant in the game, such as a
human player or a bot that is
simulating a player. Non-player AI
that exists as part of the game
would not have a PlayerState.
The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController Class
GameState Class
PlayerState Class
The GameState contains the state
of the game, which could include
things like the list of connected
players, the score, where the
pieces are in a chess game, or the
list of what missions you have
completed in an open world game.
The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController Class
GameState Class
PlayerState Class
The HUD is the base object for
displaying elements overlaid on the
screen. Every human-controlled
player in the game has their own
instance of the AHUD class which
draws to their individual Viewport.
Object
Actor
Pawn
Character
Base building blocks in the Unreal Engine
Any object that can be placed into a level
Subclass of Actor and serve as an in-game avatar
Subclass of a Pawn that is intended to be used as a player character
CharacterMovementComponent
Pawn
Character
Default Pawn
Wheeled Vehicle
CapsuleComponent
SkeletalMeshComponent
Etc.
VehicleMovementComponent
SkeletalMeshComponent
PhysicsHandle
Etc.
SpectatorPawn
DefaultPawnMovementComponent
StaticMeshComponent
CollisionComponent, Etc.
Controller1-to-1 Relationship
How about programming interactivity for VR?
Programming VR Interaction with Blueprints
Blueprints in Unreal Engine is a complete visual scripting system based on the concept of using a node-based interface to create interactions from within Unreal Editor.
Programming VR Interaction with Blueprints
Learning Blueprints through Content Examples
Hey!! We need AUDIO for VR too!!
UE4 – Audio for VRAmbient Sound Actors in VR
Ambient Sound Actor can be used for
many purposes such as ambient
looping sounds and non-looping
sounds. Generally, the Ambient
Sound Actor conforms to the real
world where the closer you are to a
sound, the louder it will appear.
UE4 – Audio for VRSound Properties
You can assign a sound asset from
the Details panel by selecting an
asset from the Sound settings drop-
down menu or by highlighting a
sound asset in the Content Browser
and clicking the button.
UE4 – Audio for VRAttenuation Properties
Attenuation is the ability of a sound
to decrease in volume as the player
moves away from it.
It is advisable to use Sound
Attenuation objects whenever
possible, if for no other reason than
to give broad control over the
settings for many Actors.
UE4 – Audio for VRNew: Stereo Spatialization
3D spatialization is now possible for
stereo audio assets.
The 3D Stereo spread parameter
defines the distance in game units
between the left and right channels
and along a vector perpendicular to
the listener-emitter vector.
UE4 – Audio for VRAudio Volume
Audio Volumes allow you to control
and apply various sounds in your
level as well as provide an avenue
to create compartmentalized audio
zones where you can control what
is heard inside and outside of the
volume.
Additional toolsets in Unreal Engine to enhance VR:
Complete state of the art suite of AI Tools.
Additional toolsets in Unreal Engine to enhance VR:
Complete set of tools for animation and animation retargeting
84
VR with Unreal Engine is a lot of fun.Remember to go to check out the UE4
VR learning resouces for more great info.