towards embedded computer vision - new @ 2013

Wang, Yuan-Kai(王元凱) Electrical Engineering Department, Fu Jen University (輔仁大學電機工程系) Email: [email protected],

URL: http://www.ykwang.tw Slideshare: Search "Towards Embedded Computer Vision"

Towards Embedded Computer Vision

1

王元凱 Towards Embedded Computer Vision p.

Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

Context Awareness 2

人

時

物

地

事 Situation

High-Level Context Low-Level Contexts



3

MIT Oxygen

http://oxygen.lcs.mit.edu/ 1999



4

The Aware Home

http://awarehome.imtc.gatech.edu/



Ambient Intelligence v.s. Object Intelligence

Embedded system + Sensors

5

Wearable Computing Smart Home, Smart …



6 6

Embedded Systems "Without" Sensors

資料來源：資策會MIC ITIS計畫整理



7 7

Embedded Systems "With" Sensors

GPS Navigation

Wii Roomba

Exoskeleton



8

Driver assist Front view

Night vision

Blind corner

Blind spot

Rear view

Embedded Systems "With" Image Sensors



9 9


Google Glass



Smart Watch 10



Microsoft SenseCam


11



Observation Miniaturization

High Resolution

3D Capability

12



13

Embedded Computer Vision   Embedded System + Camera

+ Computer Vision Algorithm

Image Processing

Image Recognition

Image Capturing



14

Embedded Computer Vision   Embedded compute vision

  Implements computer vision algorithms on low-cost, low-power, constrained hardware

  Constrained hardware   Low-speed CPU   Low capacity memory   No floating-point (FPU)   Low-resolution image sensor



A B

Research Status (1/3) 15



Research Status (2/3) H K K

16



17 17

Research Status (3/3)

DARPA Grand

Challenge

Google Car

Google Taxi



18

International Activities (1/2)   Special conferences

  IEEE Int. Workshop on Embedded Computer Vision

  IEEE Int. Conf. Distributed Smart Cameras   Special journal issues

  IEEE Journal of Selected Topics in Signal Processing, vol. 2, no. 4, Aug. 2008

  EURASIP Journal on Embedded Systems   Short courses in important CV conferences

  CVPR07&08: Distributed vision processing in smart camera networks

  ESC07: Embedded CV and smart cameras   ICASSP09: Distributed processing in smart

cameras



19

International Activities (2/2)   Research projects and Lab.

  Princeton Univ./Georgia Tech.: Embedded Systems Lab., Wayne Wolf

  Stanford Univ. Wireless sensor networks Lab.

  UCLA, CMU, MIT   Delft Univ. of Technology

SmartCam Project   Graz Univ. of Technology



20

Three Ways for ECV   CPU (Central Processing Unit)

  ARM, PowerPC   DSP (Digital Signal Processor)

  TI, ADI, NXP   FPGA (Field Programmable Gate Array)

  Altera, Xilinx



21

CPU   Stanford MeshEye

  ARM 7 (55MHz), ZigBee node   3 image sensors

  30x30 grayscale x 2   640x480 color x 1

Object detection

(http://wsnl.stanford.edu/smartcam.html)

"MeshEye:a hybrid-resolution smart camera mote for applications in distributed intelligent surveillance", IPSN-SPOTS, 2007



22

DSP   "Visual surveillance on DSP-based

embedded platform," Graz Univ. of Technology, 2008(Phd. dissertation)   TI C6414 (600MHz)   Applications: Adaboost face detection, vehicle

detection, license plate detection



23

FPGA (2/3)

"Hardware, Design and Implementation Issues on a

FPGA-Based Smart Camera," ICDSC, 2007



24

Hybrid: CPU+DSP   "Distributed Embedded Smart Cameras

for Surveillance Applications," IEEE Computer, 2006.

Developed for traffic surveillance



Hybrid: CPU+DSP by SoC   TI OMAP

25



26

Hybrid: CPU+FPGA   UCLA Cyclops

  MCU: Atmega128   CPLD: image processing   ECV app.: Hand gesture recognition

http://research.cens.ucla.edu/projects/2007/Multiscaled_Actuated_Sensing/Cyclops/



Hybrid: CPU+FPGA by SoC

  Xilinx Zynq

27



28

Challenges for ECV   Algorithm refinement

  Parallel computation   Function partition, Multi-threading

  Stream processing   Memory hierarchy optimization

  Hardware design   Pipeline, SIMD, board design

  Optimized programming skills   Fixed-point arithmetic   Memory management   Intrinsic commands



29

Our Experiences

Entertainment Robot (CPU)

Vision Sensor Network (CPU)

Smart Video Surveillance (DSP)

Wearable Camera (DSP)



30

Case 1 Entertainment

Robot CPU



31

Sony AIBO



Rolling Dice 32



33

Face Detection



34

Face Recognition



35

Facial Expression Recognition



36

Case 2 Embedded Vision

Sensor CPU



37

FJUCam

• Power consumption: 1W

•  Power sources: • 5V DC current • 5V Cell Battery

•  Weight: 35gm • Size: 6 x 4.5 x 5 (cm) (W x H x D)

Three Modules 1. Main board, 2. Lens module

3. Storage module



38

Face Detection

FJUCam Face Detection

Image Display



39

Cyclops Robot   Color tracking



40

Case 3 Video Surveillance

FPGA



41

Mobile Video Surveillance

FPGA

行動通訊網路(3G/GPRS)

網際網路

網路攝影機

監控伺服器

行動裝置(3G行動電話)GSM

數據機

Wap Push

傳遞視訊串流

PCBackground subtraction, ... •  2.8 GHz Intel CPU •  Software: C/C++ •  Frame rate: 10 fps



42

Background Subtraction Current Frame

-

BackgroundImage Update

Post Processing

Motion Object Image

Background Image

1+kB

kB

1+kP

1+kM),(),(

),(

1

1

yxByxPyxM

kk

k

−= +

+

),()1(),(),(

1

1

yxPyxByxB

kk

k

+

+

−+= αα

Speed up by (1) Circuit design, (2) Paralization



43

FPGA Speed Up   Comparisons

  PC: 2.8GHz CPU, C implementation   FPGA can speed up 500 times

Clock(Hz) FPS

CPU

FPGA

2.8G

25M

51

10



44

Case 4 Wearable Camera

X-EYE ARM + DSP



45

Goal

45



Demo Video 46



47

Development

滑鼠

鍵盤

SD卡 USB-WIFI卡

讀卡機 USB-RS232

控制線

微投影機

自製外殼 USB

連接線

顯示器

觸控面板

筆電

移動電源

BeagleBoard

Camera USB

Hub



48

Hardware

48 48

Peripheral I/O

• USB Host

• JTAG

• DVI-D video out

• S-Video out

• SD/MMC+

• Stereo in/out

• RS-232 serial1

• Alternate power

• USB 2.0 HS OTG

USER

OMAP3530 Processor • 600MHz Cortex-A8

• NEON+VFPv3 • 16KB/16KB L1 • 256KB L2 • 430MHz C64x+ DSP • 32K/32K L1 • 48K L1D • 32K L2

• Power VR SGX GPU • 64K on-chip RAM POP Memory • 256MB LPDDR RAM • 256MB NAND flash

RESET

7.6 cm



49

System

2010.04.25 49



50

Software



Next : UAV

51



Next : Sleep Monitoring 52



Next : Vehicular Object Detection

53



54

Future of Embedded Vision



55

Summary   Embedded CPU is not appropriate for

ECV   Pros

  Great flexibility   Fast development

  Cons   No float-point arithmatics   Interrupt-based architecture

  Smart phone with 4/8 cores is OK?



56

I Think …   Embedded compute vision

  Low-cost, low-power, minimal hardware

  High-Performance computer vision   Fast speed without cost,

power, and hardware constraints

From contradiction to convergence ?!

Constrained Resource

Abundant Resource

Yes by multicore

Contradiction



57

Challenges (1/2)   Algorithm decomposition

  Function decomposition   Partition serial part and parallel part   Data flow analysis and

data dependency analysis   Parallelism

  Loop unrolling   Multithreading   Pipeline



58

Challenges (2/2)   Performance analysis method

  For efficiency improvement   Implementation efforts

  Choose a good embedded platform for computer vision   Software issues   Hardware issues

  Programming skills   Multi-threading



59

Future Research Directions

Heterogeneous Multicore

ARM + DSP

ARM + FPGA

ARM + GPGPU



I Think Again …

  Is there a development platform for embedded computer vision with   Built-in camera

  AE, AF, AWB   High-performance processor

  Heterogeneous multicore   Software development environment:

  C, Matlab   OpenCV   Sample codes   Labs

60



Embedded Vision Alliance 61



  Ramesh Raskar

MIT Platform

62



  Andrew Adams

Stanford Platform

63



Taiwan Platform

for Embedded

Vision ?

64



本簡報授權聲明  此簡報內容採用

Creative Commons 「姓名標示 - 非商業性台灣 3.0 版」授權條款

 歡迎非商業目的的重製、散布或修改本簡報的內容，但請標明： (1)原作者姓名：王元凱； (2)圖標示：

 簡報中所取用的部份圖形創作乃截取自網際網路，僅供演講者於自由軟體推廣演講時主張合理使用，請讀者不得對其再行取用，除非您本身自忖亦符合主張合理使用之情狀，且自負相關法律責任。

66

towards embedded computer vision - new @ 2013

Education