©2013 SoftKinetic. All Rights Reserved.

S3498 – Gesture Processing Using 3D Depth Data On Mobile Platforms

Tim Droz SoftKinetic, Inc.

VP, US Operations

SoftKinetic DepthSenseTM Cameras

Patented Time-of-Flight sensors

Most advanced Depth cameras available

iisuTM middleware (“the interface is you”)

Whole body recognition

Pioneer in close interaction

– 2 Hand, 10 finger tracking

Powerful SDK available for developers

Content Internal team developing applications & UX frameworks

3rd party developer network

Hardware | Middleware | Content

SoftKinetic , Founded in 2007 with offices in Belgium, China, USA, and Korea, is the only independent end-to-end provider for natural gesture solutions

Body Recognition natural gesture solution How Does It Work?

Refine to user-level primitives iisu™ User API

iisu™ Volume Body parts

iisu™ Parts API

Depth map processing iisu™ World API

3D avatar Upper body & hands

iisu™ Remote

Standardized 3D depth map; Sensor calibration iisu™ Source API

iisu Gesture Recognition Middleware

Layer 0

Layer 1

Layer 2

Layer 3

Tegra 3 Solution TOF + Driver + iisu stack results in low latency

Your App Game, App, UI

Mobile Platform

DepthSense Camera Physical Layer

Events

Commands

iisu Gesture Recognition Frame Grabbing

SIMD in general Perform same instruction on multiple data simultaneously. Benefit from optimized performance, power, and

area for our embedded and media intensive gesture recognition software.

SoftKinetic Internal Library Platform independent SIMD code (SSE2, SSE3, SSE4.1, SSE4.2, AVX) for DepthSense™ camera drivers and

IISU™ middleware.

NEON-specific – DepthSense camera drivers Perform computations on 8 pixels (NEON) at a time in the camera driver to compute the Confidence, Depth,

Vertices and other 3D data and have an improvement of over 4 times the scalar implementation.

NEON-specific – iisu middleware We use SIMD instructions in iisu™ layers such as 3D Points Generation, Scene Segmentation, Signal Filtering,

Gaussian filtering, Depth Image Post Processing, Source 3D and have an improvement of over 2.5 times the scalar implementation.

Result iisu™ runs with a depth stream at 60 fps on most of ARMv7 platforms running Android or Linux.

SIMD Optimizations Tegra NEON boosts performance

Platform Acer Iconia A700 tablet, 1.3GHz Tegra 3, Android 4.1.1

SoftKinetic DS311 Camera – USB attached — QQVGA (160x120) depth, VGA color, stereo mic

— 15cm-4m operation distance

DepthSense SDK, iisu SDK

2 apps: Skeleton & Stuntman

SoftKinetic Android build targeted for: ARMv7 based CPU architecture

— Cortex A9, Cortex A15

Android OS at least v4.0.3

Demo Skeleton & Stuntman

More Pixels More x-y resolution in depth

More x-y resolution in color (RGB)

Wide FOV Best for small spaces (don’t need to rearrange the

furniture)

Full body from 1.5m

Higher Performance Silicon Better (finer) depth uncertainty

Higher fps (60+) = even lower lag

Smaller form factors Higher levels of integration (TI partnership)

Modules for PCs, Tablets, Smartphones, and more

What’s Next DepthSenseTM Sensors & Cameras

Large play-space, sees players closer to the Screen

Kinect sweet spot from 2m to 2.5m for single player!

Detection 4 to 6 player tracking

Hands & fingers from 10cm

Open/closed detection

3D Scanning

Object detection | tracking

Head tracking

Biometrics ID Face and volume identification

Robustness Increased flexibility with less false positives

Auto calibration

What’s Next iisuTM Gesture Recognition Middleware

Its already here Gaming IS mobile Allows people to have their content with them at all times

Multimodal Gameplay Close Interaction gesture & touch on mobile Plug in to TV’s & AIO’s for full body tracking It’s not about one or the other, it’s all of the above

Tegra 4+ (Logan) Platform GPGPU, OpenCL, CUDA –

— RGB+Z UV mapping on a scene, augmented reality

— Better graphics rendering in games/applications

— 4K Output

What’s Next Play Is The Mother Of Invention

Contact: Tim Droz tdr@softkinetic.com www.softkinetic.com www.youtube.com/SoftKinetic