bringing mobile vision to wearables

1

Bringing Mobile Vision to Digital Eyewear

Giuliano Maciocci, Staff Product Manager Andrew Everitt, Principal Engineer Qualcomm Cambridge Ltd.

The Qualcomm Vuforia SDK is a product of Qualcomm Connected Experiences, Inc.

3

Digital play

Enabling apps to see

4

Digital Eyewear Shipments (Millions of units)

Most shipped to North America 2013/2014

North America, Western Europe and Asia Pacific are projected to be the main markets through 2019

Source: ABI Research

0

5

10

15

20

25

30

35

40

2013 2014 2015 2016 2017 2018 2019

Middle East/Africa Latin America Asia Pacific

Eastern Europe Western Europe North America

5

“Augmented Reality Will Be the Magic Key for Smart Glasses”

ABI Research, October 2013

6

Key use cases The potential impact of vision-based Digital Eyewear

Enterprise Education Shopping Gaming! Industrial and Logistics

Step-by step instructions

Maintenance tasks

Data visualization

Employee training

First responders

Schools and homes

Virtual experiments

Augmented textbooks

3D visualization

Better product previews In-situ product previews

Enhanced print advertising

More immersive experiences Environmental interaction

Live tabletop gaming

AR + Toys

Hybrid reality gaming

Real time data + accurate physical tracking for field and site-based applications

Facilitating active learning Immersive, scale-accurate product visualization. “Place before you buy.”

Taking vision-based gaming and toy companion applications to the next level of interaction.

7

Challenges Taking vision-based experiences beyond mobile

“Look-to” displays effectively place a screen in the corner of your vision. − Limited scope for moving beyond traditional mobile AR use cases.

See-through displays (both actual and camera-based) present new opportunities for immersive, line-of-sight AR – but also serious challenges. − Need to augment the real world, not a camera feed.

− Each user’s eyes are different.

− Binocular vision needs to be handled carefully (comfort, eye strain).

− Tracking latency becomes more apparent.

New ways of interacting need to be accommodated. − The touchscreen is gone.

8

10,000+

unique applications

100,000+

registered developers

An industry leading vision platform in mobile

Source: Qualcomm Connected Experiences, Inc.

125,000,000+

Vuforia-powered app installs

9

With a network of

preferred developers around the world

Global developer adoption

North America

Latin America

Europe

Middle East & Africa

India

SE / Asia Pacific

Japan & Korea China

13%

17%

26%

6% 7%

7% 9%

15%

Numbers reflect percentages of total Vuforia developers as of May 2014

10

And now, in stereo!

11

Introducing: Vuforia for Digital Eyewear

12

VDE Intro video Placeholder for VDE video

13

Vuforia SDK for Digital Eyewear Feature overview

User calibration API calculates the complex relationship between target, camera, display and eye

Brings the Vuforia vision platform to the latest Digital Eyewear

Video background removal for augmentation directly on real world targets in see-through Digital Eyewear

Stereoscopic content authoring on binocular Digital Eyewear

Unity extension and scene camera prefab supporting drag-and-drop stereo scene rendering

Virtual Buttons functionality allows direct on-target interaction

Samples for both OpenGL and Unity tailored to Digital Eyewear

14

15

Target Camera Display Eye

16

User calibration One-time calibration for all Vuforia SDK for Digital Eyewear applications

Solves a key challenge of AR for see-through eyewear: precise line-of-sight alignment of augmented content.

4-step calibration app (2 steps x eye). − Simple one-time user-driven task: aligning a virtual rectangle to a real-world Vuforia target.

− Generates a projection matrix for each eye, encapsulating camera-display-eye relationship.

− Projection matrices are stored locally in the user’s calibration profile.

− Stored projection matrices accessible via VDE API.

Calibration API: roll your own calibration app. − Calibration API lets you embed calibration routine into your own applications.

− Personalize the calibration process with your own targets, graphics, GUI.

− Store and access multiple calibration profiles for multiple users.

17

18

Stereoscopic rendering Rendering 3D content for binocular Digital Eyewear

Left/Right eye pose matrices supported in OpenGL.

Drag and Drop Unity prefab replaces ARCamera. − Side-by-side split-screen rendering.

− Supports simple conversion of Unity content to stereo.

− Leverages stored projection matrices in user’s calibration profile to ensure correct alignment and perspective correction of AR content over real world target.

Mobile AR (top) vs. Eyewear AR (bottom). Black background appears transparent on Digital Eyewear.

19

20

Video background removal Support for see-through Digital eyewear

Traditional mobile use case aligns AR content over camera’s video feed.

In see-through Digital eyewear, we need to align AR content directly over an actual real-world target, as seen by the user.

Easily toggle video background on or off depending on target device: off for see-through or on for occluded.

Video background removal + calibration achieves correct alignment over real-world targets.

21

22

Virtual Buttons Direct on-target interaction

Allows developers to designate regions of a physical target as buttons.

Touch the target to interact.

Placing a finger over the designated target region triggers an event.

Provides compelling interaction when Eyewear input methods are limited or not convenient.

23

24

OpenGL and Unity Samples Bootstrapping development for VDE

OpenGL samples: − Planar target sample + stereo

Unity samples: − Planar target sample with stereo rendering

− Virtual Buttons sample application

− Step-by-step instructional sample application

25

Supported Devices

26

Epson Moverio BT-200

27

ODG R-7

28

Samsung Gear VR Innovator Edition

29

API Overview

30

Digital Eyewear API New classes: QCAR::Eyewear, QCAR::EyewearCalibrationProfileManager, QCAR::EyewearUserCalibrator

A set of APIs to configure and provide rendering support for the Digital Eyewear device that your app is running on.

Query device for stereo support and enablement.

Switch between the device’s stereo and mono display modes (if supported).

Query whether device is see-thru.

Retrieve projection matrices for each eye using calibration profile (see-thru devices) or correction for stereo rendering (occluded devices).

Profile manager to manage up to 10 user profiles for calibration (see-thru devices only).

Calibrator class to support creating your own calibration experience (see-thru devices only)

31

Unity extension overview

32

33

34

35

36

37

38

39

40

Running on device

In Unity simply ‘Build And Run’ for Android to create an APK.

APK works on any Vuforia SDK for Digital Eyewear supported device.

Calibration for see-thru devices handled transparently by SDK.

Unity Extension handles stereo vs. mono rendering and removal of video background on see-thru devices.

On see-thru devices user calibration shared between all SDK applications.

41

Takeaways

42

Takeaways Vuforia SDK for Digital Eyewear

Delivers powerful, immersive 3D content to an emerging generation of stereoscopic see-through digital eyewear – not just a small screen in the corner of your field of view.

Enables enterprise applications on see-though digital eyewear such as the Epson Moverio BT-200 and powerful gaming, shopping and educational experiences on the new mobile VR/AR solution, the Samsung GearVR.

Addresses key challenges of developing AR experiences for digital eyewear, such as precise calibration and object alignment, enabling the display of content that is anchored to the real world and dynamically adjusts to the wearer’s eyes.

43

Call to action Join the vanguard of Digital Eyewear developers.

Think of a great vision-based experience for see-through or mobile AR/VR Digital Eyewear

− Enterprise

− Education

− Shopping

− Gaming

44

Sign up for the beta program

vuforia.com/beta

45

For more information on Qualcomm, visit us at: www.qualcomm.com & www.qualcomm.com/blog

©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm and Vuforia are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Uplinq and Smart Terrain are trademarks of Qualcomm Incorporated. All Qualcomm Incorporated trademarks are used with permission. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.

Thank you FOLLOW US ON: