김학일 인하대학교 정보통신공학과¸하대학교_김학일.pdf컬러영상처리, 3d...

2014. 11. 26

김 학 일

인하대학교정보통신공학과

김 학일

2

• Hakil (Hale) Kim– [email protected] / +82-32-860-7385

• Education– B.Sc. in Control & Instrumentation Eng., Seoul National

Univ., Korea, 1983– M.Sc. in ECE, Purdue Univ., West Lafayette, IN, US, 1985– Ph.D. in ECE, Purdue Univ., West Lafayette, IN, US, 1990

• Activities– 1990 ~ Current: Professor, College of Engineering, Inha Univ.– 2004 ~ 2012: Rapporteur, ITU-T SG17/Q.9 (Telebiometrics)– 2003 ~ Current : Editor, ISO/IEC JTC1/SC37– 2001 ~ Current: President, Korea Biometrics Association– 2004 ~ 2010: Associate Editor, Pattern Recognition Letter– 1998 ~ 1999/2005 ~ 2006: Visiting Scholar, Simon Fraser University, BC, Canada

기술 배경

3

제품최종검사

초정밀고속화

제품중간검사

공정검사(공정관리)

검사/계측 기술의 개념 변화

Classification

기술 배경

4

§ 최근의 기술현황

§카메라 : 초고해상도 디지털카메라와 함께 sub-pixel 알고리즘을 적용하여 최소1~2 µm 정도의 해상도를 확보§고속화를 위하여 DSP 보드사용§인식률을 향상시키기 위해컬러 영상처리, 3D 영상처리, X-ray 기술이 시도 됨

§ 최근의 기술현황

§카메라 : 초고해상도 디지털카메라와 함께 sub-pixel 알고리즘을 적용하여 최소1~2 µm 정도의 해상도를 확보§고속화를 위하여 DSP 보드사용§인식률을 향상시키기 위해컬러 영상처리, 3D 영상처리, X-ray 기술이 시도 됨

§ 일본 및 이스라엘과 같은기술 선진국

§AOI ( Automated Optical Inspection ) 시스템의 핵심기술인 영상처리 관련기술을 경쟁적으로 개발하고원천기술을 보호함으로써기술 무기화를 꾀함

§ 일본 및 이스라엘과 같은기술 선진국

§AOI ( Automated Optical Inspection ) 시스템의 핵심기술인 영상처리 관련기술을 경쟁적으로 개발하고원천기술을 보호함으로써기술 무기화를 꾀함

Machine Vision국내외 현황

기술 현황

5

국내 Vision SW의 90%가외국 기술에 종속

1

§ AOI 기술의 핵심인 Vision알고리즘에 있어서Matrox(MIL), Cognex(VisionPro), MVTec(HALCON) 등이 지배

CPU의 속도 향상과멀티코어 기술의 발전

2

§ 일반 CPU 의 Multi-core§ SIMD (Single Instruction Multiple Data) 기술의활용도가 높아짐§ GPU의 지속적인 발전으로병렬 처리 활용한 고속화

High-speed Image Processing

6

Distributed Memory

CPU

Shared Memory GPU

OS(32,64bit)

CUDAIPP

SSEn

OpenMP

SSE SSE2 SSE3

SSSE3 SSE4

MPI

OpenCV

HI

GH

S PE D P R O C

N

SS

E

I

G

CPU, GPU

OS

Library

PP

E

.

.

Shared memory Constant memory

Global memory

IPL (Intel)MMXSSE

EM64T

SSE3

SSE2 Texture memory

APU

OpenCLOpenACC

Parallel programming for heterogeneous CPU/GPU

Multi-core CPU를 이용한 OpenMP 프로그래밍

7

T0

T1

JOIN

FORK

JOIN

FORKMaster

thread

{Parallel region} {Parallel region}

OpenMP programming model – Fork and join model

MMX Register를 이용한 SIMD 프로그래밍

SIMD(Single Instruction Multiple Data) Programming

8

8 bit data allocated in 128 bit MMX register(16x8bit = 128bit)

GPU를 이용한 CUDA 프로그래밍

9

v A kernel is executed by a grid ofthread blocks

v A thread block is a batch of threads that can cooperate:• Sharing data through shared

memory• Synchronizing their execution

v Threads from different blocks operate independently

CUDA Program model

CPU-GPU를 결합한 OpenCL 프로그래밍

10

v OpenCL (Open Computing Language ) allows parallel computing on heterogeneous devices• CPUs, GPUs, Cell, DSPs, etc• Provides portable accelerated code

OpenCL memory model OpenCL program structure

CPU-GPU를 결합한 OpenCL 프로그래밍

11

Host program Kernel

int main(){clGetPlatformIDs(…);clGetDeviceIdIDs(…);clCreateContext(…):clCreateCommandQueue(…);clCreateProgramWithSource(…);clBuildProgram(…);clCreateKernel(…);clCreateBuffer(…);clEnqueueWriteBuffer(…);clSetKernelArg(…);clEnqueueNDRangeKernel(…);clEnqueueReadBuffer(…);clRelease(…);} .cpp

.cl

PCB Inspection

General Process

12

PCB pattern ImagePCB pattern Image

Position AlignmentPosition Alignment

Dynamic thresholding

Dynamic thresholding

MorphologicalsegmentationMorphologicalsegmentation

Detection & Classification of

defects

Detection & Classification of

defects

§ Shift and rotate

§ Binarize

§ Compare with reference image

§ Extract and classify features

Position Alignment using Distance Transform

13

}1,0{: 2 ®ÌW ZI

{ }1)( =WÎ= pIpO

},...1{},...1{ nn ´=W

Binary image

Image index

Object or foreground index

{ } { }0)(),(min),(min:)( ==Î= qIqpdOqqpdpD c

The distance transform (DT) is the transformation that generates a map Dwhose value in each pixel p is the smallest distance from this pixel to Oc

l Definition of Distance Transform


14

Chessboard City-block

Euclidean Quasi-Euclidean


15

Output results

Translation & rotation estimation using minimum distance matching methodTranslation by sub-pixel analysis Rotation by Hough transform Rotation by sub-pixel analysis

Distance Transform for accurate matching Model Target

Noise removing using morphological operation(option)

Binarization for Distance TransformModel (CAD or grab) Target(grab)

Low-pass filtering for noise removingGaussian Mean

Assumption: relatively small rotation


Distance Transform (1)

16

거리변환 기반의 패턴매칭


v Translation estimation using sub-pixel analysis

17-4

-20

24

-4-2

02

40.8

0.85

0.9

0.95

1

-4-2

02

4

-4-2

02

40.8

0.85

0.9

0.95

1

2),( ),(

22

),(

),().,(

),().,(

å å

å

Î Î

Î

++

++

=

Wji Wji

Wji

jyixji

jyixji

NCC

TM

TMl Rough translation using NCC

l Accurate translation based on matching score using sub-pixel analysis

Rough matching Accurate matching


18

Mθm-3

Mθm-2

Mθm-1

Mθm

Mθm+1

Mθm+2

Mθm+3

Mθm-1,1

Mθm-1,2

Mθm-1,3

.

.

.

Mθm+1,n-2

Mθm+1,n-1

Mθm+1,n

θ

Sub-pixel 기술을 이용한 거리변환 기반의 회전각도 계산



19

Model -2.9 degree rotated target image

-5 -4 -3 -2 -1 0 1 2 3 4 50.8

0.85

0.9

0.95

1-2.9 degree

Sub-pixelanalysis

-5 -4 -3 -2 -1 0 1 2 3 4 5

0.85

0.9

0.95

1

1.05

-3 degree

§ Rotation estimation using sub-pixel analysis



v Experimental results

20

l Experimental environments

l OS-Windows 7(64bit)

l CPU – Intel Core i7 920

l Memory – 8GB

l Make the target image DB using model data(CAD) for checking the translation and rotation information under the ground truth

l Etching by erosion

l Noise

l Smoothing

l Translation

l Rotation


v Experimental results for etching factor

21

00.2

0.40.6

0.81

1.21.4

1.61.8

2

00.2

0.40.6

0.81

1.21.4

0

0.5

1

1.5

2

2.5

X

table6

Y

q

SEM-HoughSEM-SRSMIL-NCCMIL-GMF

00.2

0.40.6

0.81

1.21.4

1.61.8

0

0.5

1

1.5

20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

X

table5

Y

q



v Experimental results for Gaussian noise

22

00.5

11.5

22.5

3

00.5

11.5

22.5

30

0.5

1

1.5

2

2.5

3

3.5

4

4.5

X

table7

Y

q


00.5

11.5

22.5

3

0

0.5

1

1.5

20

0.5

1

1.5

2

2.5

3

3.5

X

table8

Y

q



v Experimental results for Gaussian blurring

23

0

0.5

1

1.5

0

0.5

1

1.50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

X

table9

Y

q


00.2

0.40.6

0.81

1.21.4

1.61.8

2

00.2

0.40.6

0.81

1.21.4

0

0.5

1

1.5

2

2.5

X

table10

Y

q



v Experimental results for translation factor

24

00.1

0.20.3

0.40.5

0.60.7

0.8

0

0.2

0.4

0.6

0.8

10

0.02

0.04

0.06

0.08

0.1

0.12

X

table11

Y

q


00.1

0.20.3

0.40.5

0.60.7

0

0.1

0.2

0.3

0.4

0.50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

X

table12

Y

q



v Experimental results for rotation factor

25

00.1

0.20.3

0.40.5

0.60.7

0.80.9

1

0

0.2

0.4

0.6

0.8

10

0.2

0.4

0.6

0.8

1

X

table13

Y

q


00.1

0.20.3

0.40.5

0.60.7

0.80.9

0

0.2

0.4

0.6

0.8

10

0.5

1

1.5

2

X

table14

Y

q


Position Alignment (using Distance Transform

v Rotation factor in the real Grab images

26

120122

124126

128130

132134

136138

105

110

115

120

1250

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

X

table15

Y

q


7274

7678

8082

84

80

85

90

950

0.5

1

1.5

2

2.5

3

3.5

X

table16

Y

q


27< Conventional > < Proposed >

Distance Transform - Results


-0.20

0.20.4

0.60.8

1

-0.20

0.20.4

0.60.8

10

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

table14

SEMMIL-NCCMIL-GMF

Dx Dy

Dq

28

SEM

SEM

Crop the ROI & circle estimation

Distance Transform with Circle Estimation



Comparison for general images

29

Original NCC Using distance transform


Large Pattern

30

(578,579) (578,580)

6.01sec (2200 x 2200) 7.78sec (2200 x 2200)

Model Image(1118 x 1118)

NCC DT+NCC


Robustness to Noise

31

Original NCC NCC with distance transform without noise removing

NCC with distance transform and noise removing

(203,225) (206,223)

0.21sec 0.33sec 0.43sec

(204,224)

Threshold = 50, Ground Truth: (204,224)

32

Model Image Target Image

Feature extraction(Magnitude, Angle, Descriptor)

Reduce False Matching

Translation &Rotation estimation

Output Results


Similarity:Find Matching Points

Model ImageModel Image Target ImageTarget Image



Reduce False MatchingReduce False Matching



Output ResultsOutput Results





SIFT – Scale Invariant Feature Transform

Position Alignment using SIFT

33

Fast Pattern matching using SIMD operation


<source image : 4048x4057> <Template image : 224x208> <Matching Result>

Methods Time (ms) GainIPP 6.1 329.15 x 1

SSE2 ver. 155.68 x 2.1SSE4 ver. 46.55 x 7

< Performance Comparison>

Position Alignment using SIFTExperimental Results

34

21

Position Alignment using SIFTExperimental Results

35

65


Speed of SIFT by GPU

36

• Intel Core2 Quad Q6600 2.40GHz

• Windows XP, RAM-3GHzData

Size1 2 3 4 5 6 7 8 9 10

Model 256×256 256×256 256×256 256×256 256×256 256×256 141×144 141×144 256×256 256×256

Test 768×768 384×384 384×384 384×384 384×384 384×384 300×300 300×300 512×512 512×512

DataResult

1 2 3 4 5 6 7 8 9 10

Offset(x,y) (473,417) (192,193) (193,192) (195,191) (195,196) (193,196) (106,121) (135,127) (238,279) (232,259)

Angle(˚) 2.71 -0.63 -1.47 -1.38 0.70 -0.45 0.12 0.47 -0.24 -0.33

Score 0.90 0.88 0.63 0.89 0.87 0.82 0.99 0.97 0.77 0.95

# Features 14 96 17 19 16 48 13 19 15 36 20 36 6 14 6 17 15 69 12 53

4 4 3 3 7 7 4 4 5 3

Time(ms) 172 63 78 47 62 62 31 31 110 94

Model | TestMatchings

Dynamic Thresholding

Multi-level Thesholding

37

Parameters :Number of thresholds and groups

Show the result

å=

=>=L

iii

i ppNfp

1i1 ,0 ,

å-´+

´=W =

1)1( z

z

q

Nq

Nqiifg

)}(max{arg 2

10

* jj BMj z

s-<<

=

)}()max{(arg 2

1)1()(

*

**tt B

NjtNj zz

s ¢=-+<<

Input Image &Set parameter

OutputHistogram

CalculateGlobal histogram

CalculateGroup histogram

FindGroup threshold

FindGlobal threshold

38

Two-level thresholding


Experimental Results


Fast Dynamic Threshold using GPUs

39

0

200

400

init

init…

hist

mem

…gr

p…gr

p th

ror

i thr

img…

resu

l…

tota

l

init

init…

hist

mem

…gr

p…gr

p th

ror

i thr

img…

resu

l…

tota

l

init

init…

hist

mem

…gr

p…gr

p th

ror

i thr

img…

resu

l…

tota

l

0100200300400500600700800900

TSMO-32 GPU Processing time (Image size : 4048 x 4057)

TSMO-32 CPU Processing time (Image size : 4048 x 4057)

Segmentation

Fast Blob Analysis

40

면적 계산 및 출력텍스트형식(인덱스, 면적 ,라벨당 픽셀 갯수)

메모리 전송GPUàCPU

GPU 알고리즘 실행

메모리 전송CPUàGPU

입력영상

결과영상라벨링 영상(pMarker) Normalized image(pImg)

수렴여부 확인

라벨변화 여부

Kernel 함수2 실행(Scan Image)8방향 인접 값 고려

Kernel 함수1 실행(초기화)pImg(for area) pLut (LUT)

Thread 생성Threads: (8, 8, 1) Grid : (w/8, h/8)

NOYES

수행시간

체크

Segmentation

Fast Blob Analysis - Results

41

Original Image Blob Image

(msec)

(msec)

Defect Classification

Discolor Classification (1)

42

Input Image

Preprocessing

P2C Transform

Partition

Symmetry Uniformity

Classifier

Discolor Roughness

Histogram

Defect Classification

Discolor Classification (2)

43

가보필터 기반의 변색분류

제안한 변색분류 결과

Applications

Benchmarking of Various Libraries

44

SSE vs. Image Processing Libraries

OpenMP vs. CUDA

Applications

Fast Atan2 calculation

§ Atan2 is a time consuming function

45

Method Equation Time (ms)IPP 582.308

C++ 3,918.62

Proposed SSE

Equ.1 149.597

Equ.2 170.538

Equ.3 171.708

Equ.4 174.994

Equ.5 184.024

Equ.6 209.129

CUDA286.297 (Kernel

only :22.592)

< Data size : 100MB - The number of elements is 52,428,800 >

0

100

200

300

400

500

600

IPP SSEEqu.1

SSEEqu.2

SSEEqu.3

SSEEqu.4

SSEEqu.5

SSEEqu.6

CUDA

Time (ms)

모아레 스캔 방식으로 획득된데이터는 아크탄젠트 함수를이용하여 위상을 계산하는데 사용

Applications

Line Beam Profiling

§ Atan2 is a time consuming function

46

< Performance Comparison >

Methods Time (ms) Gain

Conventional 2681.69 x 1

SSE ver. 369.27 x 7.2

……Averaging

Applications

특징: Pixel 값을 Gamma correction을 통해 높이 값 보정

2D PCB 영상을 3D로 표현Profile line 위치

Profile line 표시

컬러바

Profile 표시

3D Visualization

Applications

Stitching (1)

§ Illumination Compensation

§ Stitching according to SIFT features

48

< Flow chart >

Applications

Stitching (2)

49

결론

50

ü 영상데이터 양의 증가와 동시에 처리속도 단축 요구에 따라 병렬

영상처리 기술의 개발은 필수

§ Multi-thread가 가능한 상용 Library 출시

§ Multi-core CPU를 이용한 OpenMP 프로그래밍

§ MMX Register를 이용한 SIMD 프로그래밍

§ GPU를 이용한 CUDA 프로그래밍

§ CPU-GPU를 결합한 OpenCL 프로그래밍

ü 외산 상용 Library를 능가하는 자체 프로그램 개발 가능

ü Machine Vision 기본 영상처리 Library 개발을 위한 국내 기업들의 협력

필요

Mobile Computer Vision

51

52

Most frequently used functions?

Inside Smartphone

53

Google’s ARA

http://www.qualcomm.com/snapdragon/processors/801

Manufacturers Product Model DeviceQualcomm Snapdragon Snapdragon 801, 805 Samsung GALAXY S5, LG G3

Qualcomm Snapdragon Snapdragon 800 Samsung GALAXY S4 LTE-A

NVIDIA Tegra Tegra 4 ZTE U988S

Intel Intel Atom Intel Atom Z2760 Motorola RAZR i XT890

Apple A Apple A6 iPhone 5

Samsung Electronics Exynos Exynos 5420, 5433 Samsung GALAXY Note II, Note IV

Texas Instruments OMAP OMAP 4460 Google GALAXY Nexus

Mobile Application Processor

54

Snapdragon Tier CPU GPU Display Devices Support

Snapdragon 8052.7 GHz Quad Krait 400 CPU

Adreno 420Ultra HD

4K external displays Supported

Samsung GALAXY S5 LTE-ALG G3 Cat.6

OpenGL ES 3.0DirectXOpenCLRenderscriptFlexRender™1


Adreno 3302560x2048 + 1080p4K external displays

SupportedSamsung GALAXY S5


Snapdragon 800 2.3 GHz Quad Krait 400 CPU

Adreno330

2560x2048 + 1080p4K external displays

Supported

Samsung GALAXY S4 LTE-ASamsung GALAXY Note 3LGE G2LGE Vu 3LG G Pro2LG G FlexPantech VEGA LTE A



Adreno 320

24-bit QXGA (2048x1536) + 1080p

external display Supported

Samsung GALAXY S4LG Optimus G ProPantech VEGA IRON


https://developer.qualcomm.com/discover/chipsets-and-modems/snapdragon

55

Mobile Application Processor

Snapdragon

u CPU, GPU, system memoryu Data parallel workloads in OpenCL C kernelsu Kernels can be built one or both of the CPU and GPU

56

• OpenCL Memory Model on Snapdragon– Shared memory between Host and Device– Avoid the copy between Host and Device

• Platform Architecture CPU(Krait)– VeNum co-processor– General-purpose SIMD architecture– 32kB local memory, 1MB global memory

• Platform Architecture GPU(Adreno)– Rendering graphical applications– SIMD processor– 8kB local memory, 32kB global memory

Going where?

57

http://connect.dpreview.com/post/1313669123/how-many-megapixels

• High resolution smartphone camera

– HTC One 2688 x 1520 (4MP)

– Apple iPhone 5, 5c, 5s 3264 x 2448 (8MP)

– Samsung Galaxy S4 4128 x 3096 (13MP)

– Nokia Lumia 1020 7136 x 5360 (38MP)

• Growth of Mobile AP– Mobile AP의 성능이 Pentium 4에 근접함

– Mobile AP의 시장 규모가 빠르게 성장함

– Wearable/Flexible 기기의 등장으로 지속적 성장

기대됨

• Mobile Parallel Computing– AP의 성장으로 Mobile 병렬 처리가 가능해짐

– OpenCL, OpenGLES와 같은 병렬 Library의 활용

– 고비용 비전 알고리즘의 실시간 처리 효과

2014년 AP 전망 및 64비트 발전 동향 / LIG 투자증권

Library manuf. OS supports interfaces Parallel processing Release Summary

OpenCV Intel

WindowsLinuxAndroidMac

C/C++PythonJavaMATLAB

CUDATBBOpenMPIPP

Version 2.4.8(2013/12/31)

Basic computer vision libraryEasy implementation

Vuforia QualcommiOSAndroidUnity 3D

C/C++JavaNet

• Vuforia SDK v2.8.7(2013/12) Augmented Reality mobile application

FastCV Qualcomm

WindowsMacLinuxWindows phoneWindows RT

C/C++Java • Version 1.4.0

(2013/12/03) Mobile optimized computer vision library

JavaCV Google Group

WindowsLinuxMacAndroid

Java ‘Parallel’ class JavaCV 0.7(2014/01/06) Java interface to OpenCV

OpenVX Khronos Group • CJava

WebCLOpenCL • Mobile Real time vision library

AR GPU computing

OpenGLES Khronos GroupAndroidiOSwebOS

C/C++ GPU processing OpenGL ES 3.0.3(2013/12/18)

Computer graphic rendering API for embedded system

OpenCL Khronos GroupWindowsLinuxAndroid

C/C++ GPU processing OpenCL 2.0(2013/07/22) CPU GPU parallel computation

58

Various libraries for Android

http://www.khronos.org/opengles/

OpenGLES

• Subset of OpenGL computer graphic rendering API

• Rendering 2D and 3D computer graphic

• Hardware accelerated using a GPU

• Designed for embedded system– Smartphones– Computer tablets– Video game consoles– PDAs

59

OpenCL

• CPUs and GPUs to accelerate parallel computation

• Portable code across different devices and architectures

• Dramatic speedups for computationally intensive APP

60

http://www.khronos.org/openvx

OpenVX

• Vision Hardware Acceleration Layer– Enables hardware vendors to implement

accelerated imaging and vision algorithms

– For use by high-level libraries or apps

• Focus on enabling real-time vision– On mobile and embedded systems

• Diversity of efficient implementations– From programmable processors, through

GPUs to dedicated hardware pipelines

61

http://docs.opencv.org/doc/tutorials/introduction/android_binary_package/android_dev_intro.html

Useful toolkit

• JDK (Java Development Kit)– For Java Developers. Includes a complete JRE plus tools for developing,

debugging, and monitoring Java applications

• Android SDK (Software Development Kit)– Android SDK provides you the API libraries and developer tools necessary to

build, test, and debug apps for Android

• Android NDK (Native Development Kit)– The NDK is a toolset that allows you to implement parts of your app using

native-code languages such as C and C++

62

http://www.tbray.org/ongoing/When/201x/2010/11/14/What-Android-Is

Android layout

• All the standard APIs are defined in terms of Dalvik classes

• Interface between Dalvik and native code using JNI interface

• implement using native-code languages such as C and C++ on NDK

63

OpenGL ES on Android

rgb2gray threshold boxFilter

GaussianBlur medianBlur bilateralFilter

64

OpenGL ES on Android

CannyEdge SobelFilter LaplacianFilter

warpPerspective buildPyramid SIFT

65

OpenCV와 OpenGL ES의 속도 비교

• ※ OpenCV는 OpenCV4Android 2.4.6 (SIFT와 SURF는 free version인 2.3.1)

• calcHist처럼 화소 간 dependency가 있는 경우 CPU 보다 느린 수행 시간을 보임

66

67

OpenCL on Android

• 실험 결과 (GalaxyS4 LTE-A )

u OpenCL Framework

Snapdragon 800 series

68

Exynos 5433 vs. Snapdragon 805

69

7% higher in performance

rgb2gray cvtColor CannyEdge

threshold boxFilter

OpenCL on Android

70

• Configuration & Demo

u OpenCL Framework

OpenCL on Android

71

SobelFilter Laplacian AdaptiveThres

GaussianBlur BuildPyramid

• Configuration & Demo

u OpenCL Framework

Face Detection on Mobile Phone

72

OpenCV (CPU) – 1.22 fps vs. OpenCL (GPU) – 5.15 fps

Human Detection on Mobile Phone

73

OpenCV (CPU) – 1.56 fps vs. OpenCL (GPU) – 1.92 fps

Video demo

74

Future Trend in Computer Vision

75

Kogeto’s Omni camera

76

Thank you for your attention

인하대학교 정보통신공학과교수 김 학 일

[email protected]://vision.inha.ac.kr

김학일 인하대학교 정보통신공학과¸하대학교_김학일.pdf컬러영상처리, 3d...

Documents