Hardware

Implementation

Chan Oh Park, Tahraoui Khaled

School of Information and

Abstract—In our related work, we proposed

Auto Focus for stereo camera. Instead of moving optical axis

Cognitive Stereo Auto Focus shifts one side image to the o

side for setting focus. In the shifting process,

be set on the proper place not to cause any side

by visual fatigue to user. To perform low visual fatigue focusing,

Cognitive Stereo Auto Focus analyzes the input scene and find

how many objects are in the image and how big they are. Based

on the analysis information, the algorithm set

user feels less visual fatigue. In this paper, we suggest

object analyzer hardware whose objective is analyzing disparity

map for generating information on the objects in the scene fo

Cognitive Stereo Auto Focus. Through experiment

confirmed that the performance of the suggested

enough for practical use.

Keywords-Auto convergence, histogram analy

census algorithm, stereo auto focus.

I. INTRODUCTION

Nowadays, the 3D media device industry is highly developing field. For this wide spread technologyimportant problem to solve is visual fatiguesevere side effects. Breakdown of the accommodation, high values of the parallax and crosstalk are the main factors that causes visual fatigue in 3D media. Most of the time producers can adjust those factors to cause low visual fatibefore the contents are shown to people. Unfortunately, stercamera displays raw 3D image, which conditions user will suffer from side effectsTo worsen, because of mobility of stereo camera devicescannot use chip-sets or high throughput processor for resynthesizing stereo image. In our related work visual fatigue problem in stereo camera, we proposed Cognitive Stereo Auto Focus (CSAF). CSAF seton the object where user feels less visual fatigue. To the best of our knowledge, CSAF is the first approach on stereo camera visual fatigue problem.

In this paper, we propose an object analysis algorithm with hardware implementation for CSAF. Analyzedhardware are used for predicting visual fatigue level. For brevity, in this paper, we omit the details on the visual fatigue analysis and the other parts of cognitive stereo auto focus

The rest of the paper is organized as follow

In Section 2, we start with a review of our whole system and our suggested protocol and the hardware architectureanalyzing disparity map. Then in Section 3,implementation of proposed algorithm on our test platform.

Hardware Mass Object Analyser

Implementation for Stereo Camera

Tahraoui Khaled, Jueng Hun Kim, and Jun Dong Cho

Sungkyunkwan University

School of Information and Communication

Suwon, Korea

copark@vada.skku.ac.kr

d Cognitive Stereo

for stereo camera. Instead of moving optical axis,

shifts one side image to the opposite

process, focus point should

not to cause any side effect produced

To perform low visual fatigue focusing,

the input scene and find

how many objects are in the image and how big they are. Based

sets the focus where

In this paper, we suggest a mass

e is analyzing disparity

map for generating information on the objects in the scene for

. Through experiment results, we

suggested system is

histogram analysis, visual fatigue,

ndustry is highly technology, the most

is visual fatigue, which causes . Breakdown of the accommodation, high

values of the parallax and crosstalk are the main factors that Most of the time 3D content

producers can adjust those factors to cause low visual fatigue shown to people. Unfortunately, stereo

means in some conditions user will suffer from side effects of visual fatigue.

of stereo camera devices, we sets or high throughput processor for re-

n our related work to settle the visual fatigue problem in stereo camera, we proposed

(CSAF). CSAF sets stereo focus less visual fatigue. To the best of

our knowledge, CSAF is the first approach on stereo camera

object analysis algorithm with ed results from the

ed for predicting visual fatigue level. For we omit the details on the visual fatigue

f cognitive stereo auto focus.

as follows:

of our whole system hardware architecture for

. Then in Section 3, we show the implementation of proposed algorithm on our test platform.

Finally, we conclude our paper and provide future research options in Section 4.

II. ANALYZING ALGORITHM

1) Introduction to Cognitive StereoCSAF is designed for less visual fatigue and more 3D effect

in stereo camera. Figure 1 shows overall process of the proposed focusing algorithm. Thecomes from the fact that large pormakes user feels less visual fatigue [1].

From the process of (a) in Figure 1map, which contains information on object size and depth. Then, we extract size and deptanalysis in (b). Finally, we find proper shift value D from visual fatigue prediction in (c), then this paper, we only deal with the algorithm and hardware architecture for the process (a) and (b).

Figure 1. (a) Disparity map, (b) Histogram analamount adjust, and

2) Histogram Analysis In this section, we describe

algorithm of a given disparity mapnumber of mass object, its portionfinding maximum point and defining boundaries between objects in disparity map histogramare necessary in finding the best focusing point

When a disparity map is given, lthe disparity map. Before analyzingfiltering (1) should be done as the analysis is based on differential method. ���� � ∑ ����� �

The cut-off frequency and acquired from � which will be introduced in later part of this section. For searching optimal points of histogramdefine derivative function of filtered histogram

���� � ����From the zero crossing points

optimum points and its histogram

Object Analyser

for Stereo Camera

Jun Dong Cho

Finally, we conclude our paper and provide future research

LGORITHM AND HARDWARE

Introduction to Cognitive Stereo Auto Focusing for less visual fatigue and more 3D effect

in stereo camera. Figure 1 shows overall process of the The main idea of our algorithm

large portion of object in comfort zone user feels less visual fatigue [1].

in Figure 1, we acquire a disparity map, which contains information on object size and depth. Then, we extract size and depth of objects by histogram

Finally, we find proper shift value D from visual fatigue prediction in (c), then display image in (d). In this paper, we only deal with the algorithm and hardware

e process (a) and (b).

(b) Histogram analysis, (c) Stereo Atuo Focus

, and (d) Display

the proposed histogram analysis of a given disparity map. Our system acquires the

object, its portion and sectioning points by finding maximum point and defining boundaries between

in disparity map histogram. The acquired information ry in finding the best focusing point in CSAF.

en a disparity map is given, let x[n] be the histogram of . Before analyzing the histogram, low pass

should be done as the analysis is based on

� ∑ ����� � (1) off frequency and filter coefficients �� , �� are

which will be introduced in later part of this For searching optimal points of histogram, we first

define derivative function of filtered histogram (2).

� ��� 1� (2) From the zero crossing points of H[n], we get possible

histogram value. The possible optimum

points value (3) represent a peak pixel numbers in that mass object and (4) its indexes. The boundary of each object is not yet decided.

����� � ���� � � � (3) ���� � � (4)

����� � � � H�n� � 0 andH�n 1� � ���� � ��� 1�� In (3), (4), suboptimum points still exist that are close to

each other resulting in too many sectioning points. Therefore, to unite all close suboptimum points, we judge whether the optimum value is valuable or not based on a threshold value comparison method (6).

Mlf�i� � $%&%'

��� 1� ()� *���� 1� � ����� ��+ ,��� 1� ����- � �./ 0 12334���� ()� *���� 1� 5 ����� ��+ ,��� 1� ����- � �./ 0 12334���� )6����78� � (6)

����� 7 � � MDR � Max Disparity Range � � 6���8�)+ C�D� When the indexes Ml[n] and Ml [n+1] are in the threshold

range, the point where the higher histogram point is kept and the other one is removed from the optimum list. Once the optimum points are decided, by defining the lowest valley of the histogram between two optimum points, we define boundaries of objects. In (7), ml represents the boundaries between two optimum point Mlf[n] and Mlf[n+1].

E��� � arg minG�HIJKJG�HILM ���� ,7- Summing the histogram values of an object defined by the

boundaries, we get the area of each object S[n] (8). The area information, center disparity value (6), boundary between objects (7) and object area (8), are used to predict visual fatigue level in CSAF.

O��� � P ��7�Q�ILM

RSQ�I ,8-

3) Hardware Architecture For real-time implementation, we exploit the high

throughput and parallelism of FPGA devices. Figure 2 represents the architecture of the mass object analyzer hardware. The hardware is mainly divided into three parts, camera interface, disparity map generator, and histogram analyzer. The procedure of our architecture is as follows.

Firstly, camera interface receives a pair of stereo image from optical devices and calculate disparity map. In area-based method [3], we basically suppose that the matching point is to lie on the epipolar line. Therefore, rectification [4] and camera

Figure 2. Hardware architecture

calibration [5] should be done before the matching process. For rectification, we used pre-stored warping coordinate information. As a result, without complicated matrix multiplications and fixed-point implementation, transforming one pixel is possible in one clock cycle with pipeline technique. Since the calculating disparity map takes the most of time in the whole process, we used specially designed fast census IP which is based on the paper [2], known for its suitability for real-time hardware implementation. Figure 3 shows the architecture of census IP.

Figure 3. Used FPGA platform and camera set

Secondly, we used embedded processor Microblaze that is connected to census IP which in turn conducts histogram analysis with the disparity map generated in the first step. We used Microblaze embedded processor for flexible programmability instead of designing special hardware.

III. IMPLEMENTATION RESULT

In this section, we represent object analysis result of sample images and hardware implementation on our platform. Implemented hardware operates at 100MHz on a Xilinx xc2vp30 FPGA with low resource utilization of 3304 LUTs (12%) and 16 BRAM blocks (11%) including Microblaze processor and census IP. The system guarantees at least 28 frames per second including 160 by 90 disparity map calculation on our test platform in Figure 4.

Figure 4. Used FPGA platform and camera set

Figure 5, 6 and 7 show the visualization of RTL simulation results of a sample image from our hardware. In figure 5, from the stereo image (left), the census IP calculates disparity map (right) and then histogram analysis is followed as seen in figure 6 and 7.

control

Image Data

Disparity map

Figure 5. Input stereo image pair and its disparity map from hardware

In figure 6, the analysis result is visualizedwhere four main objects are detected. Based on the analysis we reconstructed segmented disparity map in figure shows more example results from the hardware where we used middlebury stereo data sets [6].

Figure 6. Disparity map of input image and mass object anal

IV. CONCLUSION

In this paper, for Cognitive Stereo Auto

implemented mass object analyzer. The suggested algorithm

analyzes histogram of disparity map and finds the number and

portion of mass objects. We used Field Programmable Gate

Array device for real-time operation. Through the experiment

we confirm that hardware speed and algorithm performance

satisfies the criteria for practical use. As a future work,

remained to unite the implemented analyzer with visual

fatigue prediction block and focusing decision

Figure 8. Experimental results of middlebury stereo data sets: (a), (e) lef

Input stereo image pair and its disparity map from hardware

visualized into a graph sed on the analysis we

reconstructed segmented disparity map in figure 7. Figure 8 shows more example results from the hardware where we used

Disparity map of input image and mass object analysis result

uto Focusing, we

he suggested algorithm

histogram of disparity map and finds the number and

portion of mass objects. We used Field Programmable Gate

hrough the experiment

we confirm that hardware speed and algorithm performance

As a future work, it is

the implemented analyzer with visual

decision making block.

We are also planning to conduct s

much CSAF reduces visual fatigue.

Figure 7. Object segmentation based on the analysis.

REFERENCES

[1] M. Lambooij, W. IJsselsteijn, M. Fortuin, I. Heynderickx, “VisualDiscomfort and Visual Fatigue of Stereoscopic Displays: A Review”, Journal of Imaging Science and Technology , 2009 030201-(14)

[2] Chang, N.Y.-C. et al., " Algorithm and Architecture of Disparity Estimation With Mini-Census AdaptiveCircuits and Systems for Video Technology, vol.20, no.6, pp.7922010.

[3] A. Fusiello, E. Trucco, and A. Verri. A compact algorithm for rectification of stereo pairs. Machine Vision and Applications, 12(1):1622, 2000.

[4] Z. Zhang, "A Flexible New Technique for Camera Calibration," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, Nov. 2000.

[5] D. Scharstein and R. Szeliski, "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Vision, 2002.

[6] D. Scharstein and R. Szeliski, "Middlebury Stereo Datasetshttp://vision.middlebury.edu/stereo/data/

of middlebury stereo data sets: (a), (e) left-view image, (b), (f) disparity map, (c), (g) histogram analysis, (d), (h) segmented

disparity map.

We are also planning to conduct subject test to confirm how

reduces visual fatigue.

Object segmentation based on the analysis.

EFERENCES

M. Lambooij, W. IJsselsteijn, M. Fortuin, I. Heynderickx, “Visual Discomfort and Visual Fatigue of Stereoscopic Displays: A Review”, Journal of Imaging Science and Technology , 2009 – Vol. 53, Issue 3, pp.

C. et al., " Algorithm and Architecture of Disparity Estima- Census Adaptive Support Weight", Transaction on

Circuits and Systems for Video Technology, vol.20, no.6, pp.792-805,

A. Fusiello, E. Trucco, and A. Verri. A compact algorithm for rectification of stereo pairs. Machine Vision and Applications, 12(1):16–

Zhang, "A Flexible New Technique for Camera Calibration," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp.

D. Scharstein and R. Szeliski, "A Taxonomy and Evaluation of Dense Frame Stereo Correspondence Algorithms," Int'l J. Computer

Middlebury Stereo Datasets," Available: http://vision.middlebury.edu/stereo/data/

(b), (f) disparity map, (c), (g) histogram analysis, (d), (h) segmented