recent developments in embedded vision: algorithms ......© 2017 embedded vision alliance 3 vision...

© 2017 Embedded Vision Alliance 1

Jeff Bier

Founder, Embedded Vision Alliance | President, BDTI

Vision Systems Design Webcast — January 25, 2017

Recent Developments in Embedded Vision:

Algorithms, Processors, Tools and

Applications


Computer vision: research and fundamental

technology for extracting meaning from images

Machine vision: factory applications

Embedded vision: thousands of applications

• Consumer, automotive, medical, defense, retail,

gaming, security, education, transportation, …

• Embedded systems, mobile devices, PCs and the

cloud

The Evolution of Vision Technology


Vision is the Sensing Opportunity

-

10,000

20,000

30,000

40,000

2010 2011 2012 2013 2014 2015 2016 2017 2018

Un

it p

er

Ye

ar

(M)

Sensor Unit Volume: Source: Semico Research, 2014

Microphone

Image

Gyroscope

Accelerometer

Ambient light

Proximity

Magnetometer

Pressure

Touch

Fingerprint

Chemical/gas

Temperature

Ultrasonic

IR

Biological

Humidity

Hall effect

UV

ECG

EMG

Ultrasonic

EEG

Source: Chris Rowen, Cadence


Vision is the Sensing Opportunity

-

10,000

20,000

30,000

40,000

2010 2011 2012 2013 2014 2015 2016 2017 2018

Un

it p

er

Ye

ar

(M)

Sensor Unit Volume: Source: Semico Research, 2014

Microphone

Image

Gyroscope

Accelerometer

Ambient light

Proximity

Magnetometer

Pressure

Touch

Fingerprint

Chemical/gas

Temperature

Ultrasonic

IR

Biological

Humidity

Hall effect

UV

ECG

EMG

Ultrasonic

EEG

0.00E+00

2.00E+12

4.00E+12

6.00E+12

8.00E+12

1.00E+13

1.20E+13

1.40E+13

2010 2011 2012 2013 2014 2015 2016 2017 2018D

ata

-Rate

We

igh

ted

Vo

lum

e p

er

Ye

ar

(M

un

its *

bit

s p

er

se

co

nd

)

Sensor Volume Adjusted for Data Rate

Source: Chris Rowen, Cadence


The Internet of Things That See: Compology

Video: bit.ly/2jnl8Po


• For embedded vision to achieve its potential, we need:

• High-performance, energy-efficient, inexpensive processors

• More-capable, more integrated and less expensive image sensors

• More capable and reliable algorithms

• Improved software development productivity

• More skilled engineers

Overcoming Critical Challenges


Processors


• Ideally, heterogeneous

processors combine:

• Most of the performance and

efficiency of specialize

processors

• Most of the ease of

development of CPUs

• But it comes at a cost:

• complexity

Trend: Heterogeneous Processors for

Performance and Efficiency

Performance/$

Performance/W

Development

Effort


Processor Chips:

• Analog Devices BF609

• Inuitive NU3000

• MobileEye EyeQ5

• Movidius (now Intel) Myriad 2

• NXP S32V

• Texas Instruments TDA3x, TDA2Ec

Trend: Vision-specific Processors

Processor Cores:

• Apical (now ARM) Spirit

• Cadence Vision P5, Vision P6

• CEVA MM-3101, XM-4, XM-6

• Synopsys EV5x, EV6x

• Vivante VIP7000,

GC7000-XS VX

Inuitive M3 Reference Design Movidius Myraid 2


Type of Processor Used for Vision Tasks—

Ranked as One of Top 3 (July 2016)

75%

64%

44%

31% 31% 26%

4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CPU GPU Mobile Vision-specific FPGA DSP Other


Maximizing Processing per Watt: ODG


Algorithms


• Infinitely varying inputs in many

applications

• Uncontrolled conditions: lighting,

orientation, motion, occlusion

• Leads to ambiguity…

• Leads to the need for complex,

multi-layered algorithms to extract

meaning from pixels

• Plus:

• Lack of analytical models means

exhaustive experimentation is

required

• Numerous algorithms and algorithm

parameters to choose from

Vision Algorithms Are Challenging

www.selectspecs.com


Embedded Vision Challenge: Algorithms

Source: xkcd.com


Source: hitl.washington.edu/artoolkit Source: xkcd.com

Embedded Vision Challenge: Algorithms


Deep Neural Networks: Learning Machines

Source: NVIDIA


Use of Neural Networks to Perform

Computer Vision Functions (July 2016)

Yes, extensively

19%

Yes, in a minor role 19%

Not yet, but planning to

37%

No 22%

Don't know 3%


Originally used solely for classification, deep

neural networks are now also being used for:

• Detection

• Segmentation

• Sequences (e.g., video captioning)

• Optical flow

• Visual motor control

Expanding Applicability of Deep Learning

Source: Long, Shelhamer, Darrell. CVPR’15

Source: Dosovitskiy et al., ICCV ‘15


Deploying Deep Learning: Nauto


Trend: Processors for Deep Learning

Degree of Specialization

Software Packages/

Libraries, Frameworks

Architecture Enhancements

Dedicated

Co-processor

None

Architecture Focus

Google

Tensor

Processing

Unit

Movidius

(now Intel)

Myriad 2

Nervana

(now Intel)

NVIDIA

Tegra X1


Sensors


Sensor Innovation Accelerates

Image Sensors World

ims—chips.de

Chronocam

Edn-europe.com

http://image-sensors-world.blogspot.com/2014/02/heptagon-reveals-array-camera.html


Proliferation of 3D Sensors

Infineon.com

Inuitive-tech.com

Scandy.co

Photographylife.com


Understanding the World in 3D: RetailNext

Retainlext.com

TechCrunch.com


Software Development


Developing efficient vision software is challenging, due to:

• Demanding, complex, ever-changing algorithms

• Complex, heterogeneous, cutting-edge processors

• Limited infrastructure (tools, libraries, frameworks, etc.)

Challenge: Vision Software Development

Qualcomm Conceptdraw hitl.washington.edu/artoolkit


• New Khronos standard API for acceleration of

vision applications

• Addresses heterogeneous processors (CPU,

vision co-processor, GPU, DSP, etc.)

• Promises portability across chips

• Developer specifies data flow graph of vision

algorithm kernels

• Kernels may be implemented in any language

• OpenVX framework partitions the graph

among processing engines, handles memory

transfers, etc.

• Designed for mobile, useful in other domains

Solution: Standards-based Abstractions

Vision

Accelerator

Application

Application

Application

Application

Vision

Accelerator Vision

Accelerator Vision

Accelerator

Khronos Group


Libraries and APIs Used for Vision Tasks—

Ranked as One of Top 3 (July 2016)

82%

47%

26% 26%

15%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

OpenCV OpenGL FastCV OpenVX Other


• Caffe

• Vuforia

Solution: Domain-specific Frameworks

Source: CEVA

Source: Gravity Jack


Vision-enabled Autonomy: DJI Phantom 4

dcrainmaker.com


Vision has been out of reach for most potential applications,

so few engineers in industry have vision expertise

The Embedded Vision Alliance (www.Embedded-Vision.com)

is a partnership of 60+ vision technology suppliers

Mission: Inspire and empower product creators to incorporate

visual intelligence into their products

The Alliance provides high-quality, practical technical

educational resources for engineers

• Alliance website offers tutorial articles, video “chalk

talks,” forums

• Embedded Vision Insights newsletter—news and updates

Register for updates at www.Embedded-Vision.com

Challenge: Skilled Engineers

http://www.embedded-vision.com/



http://www.embedded-visionsummit.com/




• Embedded vision enables software-defined sensors, uniquely capable of

delivering insights about the real world

• Accelerating advances in processors, sensors, algorithms and

development tools make it increasingly practical to deploy vision

• Embedded vision is becoming ubiquitous

• Vision is becoming a huge creator of value—for technology suppliers,

end-product developers, and users

Take-aways


The only industry event focused on enabling

developers to create “machines that see”

• “Awesome! I was very inspired!”

• “Fantastic. Learned a lot and met great people.”

• “Wonderful speakers and informative exhibits!”

Embedded Vision Summit 2017 highlights:

• Inspiring keynotes by leading innovators

• Practical technical, business and product talks

• Exciting demos of the latest technologies

• Visit www.EmbeddedVisionSummit.com for details

• Register by February 1 using discount code

vsdwebcast0125

Join us at the Embedded Vision Summit –

May 1-3, 2017—Santa Clara, California

http://www.embeddedvisionsummit.com/

http://www.embedded-vision.com/summit/register?aff=vsdwebcast0125


Embedded Vision Alliance Member Companies


Email me ([email protected]) for:

• A copy of these slides

• Links to videos of cool embedded-vision-based products

• Questions about the Embedded Vision Summit, May 1-3, 2017,

Santa Clara, California, USA

Questions?

Jeff Bier

President, BDTI

Founder, Embedded Vision Alliance

Chairman, Embedded Vision Summit

www.BDTI.com

www.embedded-vision.com

+1 925-954-1411

Walnut Creek, CA 94596 U.S.A.

mailto:[email protected]

http://www.bdti.com/




recent developments in embedded vision: algorithms ......© 2017 embedded vision alliance 3 vision...

Documents