towards embedded computer vision - new @ 2013

Wang, Yuan-Kai(王元凱) Electrical Engineering Department, Fu Jen University (輔仁大學電機工程系) Email: ykwang@mail.fju.edu.tw,

URL: http://www.ykwang.tw Slideshare: Search "Towards Embedded Computer Vision"

Towards Embedded Computer Vision

王元凱 Towards Embedded Computer Vision p.

Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

Context Awareness 2

事 Situation

High-Level Context Low-Level Contexts

MIT Oxygen

http://oxygen.lcs.mit.edu/ 1999

The Aware Home

http://awarehome.imtc.gatech.edu/

Ambient Intelligence v.s. Object Intelligence

Embedded system + Sensors

Wearable Computing Smart Home, Smart …

Embedded Systems "Without" Sensors

資料來源：資策會MIC ITIS計畫整理

Embedded Systems "With" Sensors

GPS Navigation

Wii Roomba

Exoskeleton

Driver assist Front view

Night vision

Blind corner

Blind spot

Rear view

Embedded Systems "With" Image Sensors

Google Glass

Smart Watch 10

Microsoft SenseCam

Observation Miniaturization

High Resolution

3D Capability

Embedded Computer Vision   Embedded System + Camera

+ Computer Vision Algorithm

Image Processing

Image Recognition

Image Capturing

Embedded Computer Vision   Embedded compute vision

  Implements computer vision algorithms on low-cost, low-power, constrained hardware

  Constrained hardware   Low-speed CPU   Low capacity memory   No floating-point (FPU)   Low-resolution image sensor

Research Status (1/3) 15

Research Status (2/3) H K K

Research Status (3/3)

DARPA Grand

Challenge

Google Car

Google Taxi

International Activities (1/2)   Special conferences

  IEEE Int. Workshop on Embedded Computer Vision

  IEEE Int. Conf. Distributed Smart Cameras   Special journal issues

  IEEE Journal of Selected Topics in Signal Processing, vol. 2, no. 4, Aug. 2008

  EURASIP Journal on Embedded Systems   Short courses in important CV conferences

  CVPR07&08: Distributed vision processing in smart camera networks

  ESC07: Embedded CV and smart cameras   ICASSP09: Distributed processing in smart

cameras

International Activities (2/2)   Research projects and Lab.

  Princeton Univ./Georgia Tech.: Embedded Systems Lab., Wayne Wolf

  Stanford Univ. Wireless sensor networks Lab.

  UCLA, CMU, MIT   Delft Univ. of Technology

SmartCam Project   Graz Univ. of Technology

Three Ways for ECV   CPU (Central Processing Unit)

  ARM, PowerPC   DSP (Digital Signal Processor)

  TI, ADI, NXP   FPGA (Field Programmable Gate Array)

  Altera, Xilinx

CPU   Stanford MeshEye

  ARM 7 (55MHz), ZigBee node   3 image sensors

  30x30 grayscale x 2   640x480 color x 1

Object detection

(http://wsnl.stanford.edu/smartcam.html)

"MeshEye:a hybrid-resolution smart camera mote for applications in distributed intelligent surveillance", IPSN-SPOTS, 2007

DSP   "Visual surveillance on DSP-based

embedded platform," Graz Univ. of Technology, 2008(Phd. dissertation)   TI C6414 (600MHz)   Applications: Adaboost face detection, vehicle

detection, license plate detection

FPGA (2/3)

"Hardware, Design and Implementation Issues on a

FPGA-Based Smart Camera," ICDSC, 2007

Hybrid: CPU+DSP   "Distributed Embedded Smart Cameras

for Surveillance Applications," IEEE Computer, 2006.

Developed for traffic surveillance

Hybrid: CPU+DSP by SoC   TI OMAP

Hybrid: CPU+FPGA   UCLA Cyclops

  MCU: Atmega128   CPLD: image processing   ECV app.: Hand gesture recognition

http://research.cens.ucla.edu/projects/2007/Multiscaled_Actuated_Sensing/Cyclops/

Hybrid: CPU+FPGA by SoC

  Xilinx Zynq

Challenges for ECV   Algorithm refinement

  Parallel computation   Function partition, Multi-threading

  Stream processing   Memory hierarchy optimization

  Hardware design   Pipeline, SIMD, board design

  Optimized programming skills   Fixed-point arithmetic   Memory management   Intrinsic commands

Our Experiences

Entertainment Robot (CPU)

Vision Sensor Network (CPU)

Smart Video Surveillance (DSP)

Wearable Camera (DSP)

Case 1 Entertainment

Robot CPU

Sony AIBO

Rolling Dice 32

Face Detection

Face Recognition

Facial Expression Recognition

Case 2 Embedded Vision

Sensor CPU

FJUCam

• Power consumption: 1W

•  Power sources: • 5V DC current • 5V Cell Battery

•  Weight: 35gm • Size: 6 x 4.5 x 5 (cm) (W x H x D)

Three Modules 1. Main board, 2. Lens module

3. Storage module

Face Detection

FJUCam Face Detection

Image Display

Cyclops Robot   Color tracking

Case 3 Video Surveillance

Mobile Video Surveillance

行動通訊網路(3G/GPRS)

網際網路

網路攝影機

監控伺服器

行動裝置(3G行動電話)GSM

數據機

Wap Push

傳遞視訊串流

PCBackground subtraction, ... •  2.8 GHz Intel CPU •  Software: C/C++ •  Frame rate: 10 fps

Background Subtraction Current Frame

BackgroundImage Update

Post Processing

Motion Object Image

Background Image

1+kM),(),(

yxByxPyxM

−= +

),()1(),(),(

yxPyxByxB

−+= αα

Speed up by (1) Circuit design, (2) Paralization

FPGA Speed Up   Comparisons

  PC: 2.8GHz CPU, C implementation   FPGA can speed up 500 times

Clock(Hz) FPS

Case 4 Wearable Camera

X-EYE ARM + DSP

Demo Video 46

Development

滑鼠

鍵盤

SD卡 USB-WIFI卡

讀卡機 USB-RS232

控制線

微投影機

自製外殼 USB

連接線

顯示器

觸控面板

筆電

移動電源

BeagleBoard

Camera USB

Hardware

Peripheral I/O

• USB Host

• JTAG

• DVI-D video out

• S-Video out

• SD/MMC+

• Stereo in/out

• RS-232 serial1

• Alternate power

• USB 2.0 HS OTG

OMAP3530 Processor • 600MHz Cortex-A8

• NEON+VFPv3 • 16KB/16KB L1 • 256KB L2 • 430MHz C64x+ DSP • 32K/32K L1 • 48K L1D • 32K L2

• Power VR SGX GPU • 64K on-chip RAM POP Memory • 256MB LPDDR RAM • 256MB NAND flash

7.6 cm

System

2010.04.25 49

Software

Next : UAV

Next : Sleep Monitoring 52

Next : Vehicular Object Detection

Future of Embedded Vision

Summary   Embedded CPU is not appropriate for

ECV   Pros

  Great flexibility   Fast development

  Cons   No float-point arithmatics   Interrupt-based architecture

  Smart phone with 4/8 cores is OK?

I Think …   Embedded compute vision

  Low-cost, low-power, minimal hardware

  High-Performance computer vision   Fast speed without cost,

power, and hardware constraints

From contradiction to convergence ?!

Constrained Resource

Abundant Resource

Yes by multicore

Contradiction

Challenges (1/2)   Algorithm decomposition

  Function decomposition   Partition serial part and parallel part   Data flow analysis and

data dependency analysis   Parallelism

  Loop unrolling   Multithreading   Pipeline

Challenges (2/2)   Performance analysis method

  For efficiency improvement   Implementation efforts

  Choose a good embedded platform for computer vision   Software issues   Hardware issues

  Programming skills   Multi-threading

Future Research Directions

Heterogeneous Multicore

ARM + DSP

ARM + FPGA

ARM + GPGPU

I Think Again …

  Is there a development platform for embedded computer vision with   Built-in camera

  AE, AF, AWB   High-performance processor

  Heterogeneous multicore   Software development environment:

  C, Matlab   OpenCV   Sample codes   Labs

Embedded Vision Alliance 61

  Ramesh Raskar

MIT Platform

  Andrew Adams

Stanford Platform

Taiwan Platform

for Embedded

Vision ?

本簡報授權聲明  此簡報內容採用

Creative Commons 「姓名標示 - 非商業性台灣 3.0 版」授權條款

 歡迎非商業目的的重製、散布或修改本簡報的內容，但請標明： (1)原作者姓名：王元凱； (2)圖標示：

 簡報中所取用的部份圖形創作乃截取自網際網路，僅供演講者於自由軟體推廣演講時主張合理使用，請讀者不得對其再行取用，除非您本身自忖亦符合主張合理使用之情狀，且自負相關法律責任。

towards embedded computer vision - new @ 2013

Education

lecture 15 ryuzo okada - vision processors for embedded...

reenergizing vision towards performance excellency

smart embedded vision with quantized neural...

embedded systems made in germany - vision …...embedded...

expanding towards greater vision

vision processing and neural networks - khronos group ·...

linux embedded applications in machine vision embedded linux...

implementing vision capabilities in embedded systems ·...

towards dependable embedded model predictive control

fundamentals of embedded computer vision: creating

cpf strategic vision towards 2030

towards closing the energy gap between hog and cnn...

may 2015 embedded vision summit introductory presentation

accelerating your embedded vision / machine learning ... ·...

towards manipulation-driven vision

hong kong 2030+: towards a planning vision and strategy...

towards embedded raids-on-chip

a vision-based dsp embedded navigation sensor -...

bdti esc embedded vision[1]

real-time and low latency embedded computer vision...