augmented reality - let’s make some holograms! (uxd version)

www.cameronvetter.com 1

Augmented

Reality

Let’s Make Some

Holograms!

Goal of this Presentation


Learn AR Terminology

Build an Application

Highlight UXD Challenges

See Lots of Examples

www.cameronvetter.com

About Me Senior Software Architect


@PoshPorcupine

Presentation Roadmap


Holograms

Persistence

Control

Real World Understanding

Spatial Mapping

Plane Finding

Spatial Understanding

Gaze

Cursor

Gestures

Voice

Billboarding

Placement

Physics

World Anchors

Holographic Lens1268 x 720 Per Eye

3 Waveguide Lenses Per Eye

Intel Atom CPUX5-Z8100 1.4 GHZ

2GB Ram

Holographic Processing UnitCustom build Microsoft GPU

Sensors4 Environment Cameras

1 Depth Camera

1 Video Camera

4 Microphones

1 Light Sensor

What is the HoloLens?


Development Tools


Visual Studio 2015 Update 3

Windows 10 SDK

Unity 3d HoloLens Technical Preview

HoloLens Emulator

HoloToolkit / HoloToolkit-Unity

Real World

Understanding


World Coordinate System1 Unit = 1 Meter

User Start is OriginX=0, Y=0, Z=0

Hidden TrianglesCalled the Spatial Mesh

Spatial AnchorsAttach Hologram to Reference Points

in the Spatial Map

Spatial Mapping


Room Spatial MeshVisualized with Solids in the

Illustration

PhysicsHolograms and Real World interact

with each other through Game

Engine Physics

OcclusionWorks Both Ways

Plane FindingAbility to Find Planes in the Real

World

What do we get?


Uses Spatial MapSpatial Map Data used to find Planes

Type of PlanesWalls

Floor

Ceiling

Horizontal Surfaces

Other

Skipping this TopicWe now have Spatial Understanding

Plane Finding


World UnderstandingMore Detail than Spatial Map

Topology QueriesSimple Surface and Spatial Queries

Object Placement SolverConstraint Based Placement of

Object Sets

Shape QueriesLook for Real World Shape Matching

Custom Shape by the User



Demos


Spatial

Mapping

Spatial

Understanding

User Feedback of ProgressPresent the Spatial Map Visually as

the User Creates it

Know the Detail NeededLet the User Stop After the Minimum

is Obtained

Video DemoFragments by Rare Software

Fit Into User ExperienceBlend it into the App Experience

Mapping UXD


Holograms


Body LockedTags Along with the Display

Display LockedPositionally Locked to the Display

Use SparinglyClinginess is Frustrating for Users

Breaks IllusionHolograms are Intended to Blend,

Not Overlay

Billboarding


Relative to OriginOrigin is Starting Position, Not

Current Position

Relative to Real WorldMust take into Account, Real World

Objects

Spatial UnderstandingUsed to Accurately Place Objects

with Object Placement Solver

Plane FindingUsed to Quickly Find Ceiling, Tables,

Walls, and Other Surfaces

Placement


Tracks World StateReal World Objects and Holograms

Placement TypesOnFloor, OnWall, OnCeiling,

OnShape, OnEdge,

OnFloorAndCeiling, RandomInAir,

InMidAir, UnderFurnitureEdge

Placement ConstraintNearPoint, NearWall,

AwayFromOtherObjects,

AwayFromPoint

Placement RulesAwayFromPosition, AwayFromWalls,

AwayFromOtherObjects

Object Placement Solver


Unity Engine ProvidedAdd a RigidBody to Any Hologram to

Make it Observe Game Engine

Physics

Spatial Map ColliderEach Plane in the Spatial Mesh

Receives a Unity Collider

Not the Other WayReal World Objects are not Impacted

by Holograms

Real World ImpactHolograms are Impacted by Real

World

Physics


Demos


Placement

Solver

Physics

Not Too CloseToo Close and Users “Slip Inside” the

Holograms

Not Too FarLimited Display Resolution

Avoid 2D UITry Not to Create a 2D Interface in a

3D Environment

Focal PlanePlace Videos, Text, or Other 2D UI at

About 2 Meters

Placement UXD


TypographyPrint Design is Relevant Specifically

Billboard and Poster Design

Mock In VR or AR2D Prototypes Will Disappoint

User FeedbackAR Environment is New and

Distracting, Give the User Clarity

Be Strategic with TargetsDon’t Place UI Where User will

Accidently Click

Designing UI


Source:

James Hsu

of

Disney

Interactive

Control


Detects Using RaycastRaycast from User’s Eyes to Unity

Collider

Primary InteractionLooking At Holograms is the Primary

Way to Interact with Holograms

Real World has CollidersRemember During Spatial Mapping

the Real World is given Colliders

Invisible ObjectsAny Holograms missing a Collider

are not Selectable with Gaze

Gaze


Center of ViewDisplay Locked to Center of View

Requires GazeCursor Depends on Gaze, Expect

Both to be in Every Project

Stick with Standard CursorUse Microsoft Provided Cursors

Unless you Have a Good Reason to

Customize, TIP: You Don’t

SelectionPrimarily used for Selection

Confirmation in Conjunction with

Gaze and Gestures

Cursor


The Cursor

BloomTask Switch / Start Menu

Air TapSimilar to Primary Mouse Button

ManipulationTypically Used to Move or Drag

Items

HoldSimilar to Right Clicking

Gestures


Auxiliary ControlsThe Real Right Click

Use Windows 10 SpeechStandard Engine Built into Windows

System Level CommandsSelect == Air Tap

Face Me

Bigger

Smaller

Concise CommandsToo Long is Hard to Remember

Too Short is Hard to Differentiate

Voice Input


Demos


Gaze / Cursor Voice Input

Real World ParadigmsA Square Work Area with a Distinct

Front causes the User to Remain

Stationary

Use Physical SpaceUse the Whole Physical Space:

Floor, Ceiling, Walls, Floating…

Controls May be BlockedUse Ghosting to Show UI through

Out of Frame ContentUse Attention Directors: Spatial

Sound, Light Rays, Thought Bubbles

Interaction Design


Persistence


Frame of ReferenceRequires Stationary Frame of

Reference all Anchors are Tracked

Relative to

Attaches HologramHologram is Attached to Frame of

Reference, Remains Stationary as

Coordinate System Changes

PersistableHoloToolkit has built in Serializer

Separate Coordinate SystemAnchor has own Coordinate System,

adjusts as World Coordinate System

changes to Keep Hologram in Place

Spatial Anchors


Unity Specific for ARNot Hololens Specific, Unity Specific

Uses Spatial AnchorWorld Anchor is a Wrapped Spatial

Anchor

Persist Between SessionsSurvives App Restarts

Unity Anchor StoreWorld Anchors are Managed by

Unity Engine

World Anchors


LIVE DEMO!!!


01

02

03

04

05

06

Place Holograms

Map the Room

Use A Gesture

Shadows

Voice Commands

Physics

In Summary


Holograms

Persistence

Control

Real World Understanding

Spatial Mapping

Plane Finding


Gaze

Cursor

Gestures

Voice

Billboarding

Placement

Physics

World Anchors



[email protected]

@poshporcupine

Any Questions?

augmented reality - let’s make some holograms! (uxd version)

Technology