tobias rothermel (pmd technologies): pmd tof – the swiss army knife of 3d sensing

c o n f i d e n t i a l© 2017 pmdtechnologies ag Infineon Technologies AG

pmdtechnologies – 2017-10-04 – V0118

pmd ToFthe Swiss army knife of 3D sensing

AWE Munich 2017

co

nfid

en

tia

l

© 2017 pmdtechnologies ag Infineon Technologies AG2

Why is the guy with the beard on the stage?

co

nfid

en

tia

l

To show that with

pmd’s ToF cameras

(e.g. CamBoard pico flexx)

you can do this:


co

nfid

en

tia

l


IR greyscale, Depthmap and point cloud

co

nfid

en

tia

l

© 2017 pmdtechnologies ag 5

… and this…

Color scheme: normal map color dependent on surface direction

Infineon Technologies AG

co

nfid

en

tia

l

6

… and much more

Secure face recognition

3D face scanning

Inside-out-tracking for 6DoF self localization

Hand and finger tracking gestures

AR/VR/MR

Surveillance & People counting

Robotics

Computational photography

Drones

Automation

© 2017 pmdtechnologies ag Infineon Technologies AG

co

nfid

en

tia

l


Pico family

https://www.youtube.com/watch?v=BHunyOy4A6c

https://www.youtube.com/watch?v=BHunyOy4A6c

co

nfid

en

tia

l

8

Google Tango

Asus Zenfone AR

„Giving mobile devices a human scale understanding of

space and motion“

Combination of sensors

Depth sensing

2D camera

Fisheye camera

IMU

Integrated pmd depth sensor

Up to 4m range

1% of distance = max. depth error

5 fps

<300mW power consumption

38k pixels

ToF module size 1cm x 2cm


co

nfid

en

tia

l

9

Tango use cases

Room Scanning

Augmented Reality Gaming

© Google

© Google

Indoor Mapping and Navigation

Utility


co

nfid

en

tia

l

10

Infotainment Gesture control

Driver Monitoring

In cabin sensing

Passenger detection

Pedestrian Detection

Autonomous parking

Gesture &

obstacle detection

Automotive

© Google

With friendly permission of


co

nfid

en

tia

l

• CamBoard pico flexx = ToF based depth sensing USB camera from pmd with an Infineon 3D image sensor inside

• ToF = Time of Flight• pmd = photonic mixer device = the pixel• pmd = pmdtechnologies ag

• + = best friends ;)


First things first

co

nfid

en

tia

l

Emission of electrically

modulated light

Electrical modulation signal

Detection of the reflected

modulated light

pmd Time-of-Flight (“indirect” ToF)

Convolution of reflected

modulated and the emitted light

Autocorrelation of the emitted and the reflected light

Goal: Determine where the “peak” is it is the

phase delay, which corresponds to the distance

© 2017 pmdtechnologies ag 12 Infineon Technologies AG

co

nfid

en

tia

l

[j, d]j=2pp

jmeas

Vacf,amplitude

Voffsetconvolu

ted s

ignal

S0 S1 S2 S3

2

20

2

13, SSSSV amplacf

[V]

Information derived in SW from raw data of each pixel

Distance information

Amplitude information

Intensity information

Voffset

2D b/w image based

on modulated light

2D b/w image based on ambient light

distance @ j>0, d ~ j

S0…3 measurement points of signal amplitude

20

13arctanSS

SSmeas

=j

13

Depth map

On chip On host system/ in AP


co

nfid

en

tia

l

14

What is inside a ToF camera?

VCSEL cover

housing

VCSEL

USB Chip

ToF Chip,

lens holder,

lens

thermal pad

housing


co

nfid

en

tia

l

15

CamBoard pico flexx - What you will get

© 2017 pmdtechnologies ag

• 3D resolution: 38k

• eye safety class 1

• VCSEL (laser) illumination

• measurement range: 0.1 to 4 m

• optimized opening angle: 62°x 45°

Hardware

• one SW for all platforms: Win, Linux, Android, MacOS

• easy approach: predefined use cases for kick start eval.

• systematically tested on product level

Software„Royale“

• Interface: drivers for USB available

• GUI: QT based for 2D and 3D presentation

• improved API: threading handled by Royale consistentlyInterfaces


co

nfid

en

tia

l

Approaches to the Royale software packages

16

CamBoard pico flexx: Royale library

royale viewer royale API

Purpose: getting started Purpose: reference source code for

device integration


co

nfid

en

tia

l

Structure of the API

17

Royale - API

User of camera data (camera HAL)

royale

Camera Driver

deliver raw

data

request

sensor data

get depth

data

configure

camera device

CameraManager ICameraDevice

CameraManager Main component which can queue

commands for connected cameras

ICameraDevice Interface which represents the camera

IDepthDataListener Interface to be implemented in the HAL to

retrieve the depth data

DepthData Contains the processed depth data


co

nfid

en

tia

l

Light-weight framework for time-of-flight cameras

Joint development by pmd and Infineon

C++ interface

C++11 based

Platform independent

Windows

Linux

Mac OS X

Android ARM

Linux ARM

Matlab interface

C# interface

Examples to connect to

OpenCV

OpenNI


Royale - Purpose

co

nfid

en

tia

l


Royale - Predefined use cases for the CamBoard pico flexx

Use caseRange

[m]

Frame

rate

Exposure

time (us)

Modulation

Frequencies

Indoor room reconstruction 1.0 – 4.0 5 fps 2000 80 & 60 MHz

Indoor navigation 1.0 – 3.0 10 fps 1000 80 & 60 MHz

3D object reconstruction 0.5 - 1.5 15 fps 700 80 & 60 MHz

Face reconstruction 0.3 – 1.0 25 fps 450 80 & 60 MHz

Small object recognition 0.3 – 2.0 35 fps 600 60 MHz

Hand tracking 0.1 – 1.0 45 fps 500 60 MHz

4 m 3 m 2 m 1 m 0 m

TANGOIndoor room reconstruction (5 frames/s, 2 ms int. time)

Room scanning, indoor navigation (10 frames/s, 1 ms int. time)

3D object reconstruction (15 frames/s, 0.7 ms int.)

Medium size object recognition, face reconstruction (25 frames/s, 0.45 ms int. t)

Remote collaboration, table-top gaming (35 frames/s, 0.6 ms int. time)

Hand tracking (45 frames/s, 0.5 ms int.)

co

nfid

en

tia

l


Thank you!

co

nfid

en

tia

l

ToF, pmdtechnologies and Infineon

Dresden

2002• pmdtechnologies

is founded

• HQ Siegen

2005

• efector pmd

• first mass product based on pmd ToF technology

• Industrial sensor developed with ifm electronic

2011

• New pmd branch in Dresden, Germany

2012

• More than 500,000 pmd sensors sold to date

• Ramp up of first commercially available ToF imager: pmdPhotonICs19k-S3

2013

• ifm electronic becomes 100% shareholder

• Infineon and pmd announce jointly developed 3D image sensor family

2016

• New pmd branch in San Jose, USA

2016/17

• ToF Tango phones on the market

• ToF HMDs on the market

© 2017 pmdtechnologies ag 21 Infineon Technologies AG

co

nfid

en

tia

l


How do pmd and Infineon cooperate?


co

nfid

en

tia

l

Consumer

AutomotiveIndustrial


What is pmdtechnologies?

target

• pmd: fabless IC company Time-of-Flight pixel design + ToF camera design

+13 years expertise/ IP/ key patents on ToF + calibration know-how

• Collaboration: Infineon (production & distribution)

jointly developed ToF image sensor IC

• Mother: ifm electronics

successfully sells

pmd-based

industrial sensors

since more

than 10 years.


co

nfid

en

tia

l

Chip Design

System Support

• Lens

• Illumination

• Calibration

• Production bring-up

3D ToF Camera Reference Designs


What is pmd doing?

• High integration SoC• Low power• Outdoor robust


tobias rothermel (pmd technologies): pmd tof – the swiss army knife of 3d sensing

Technology