Image Fusion: Approaches in Hardware

- Kshitij Agrawal

Computer Vision Engineer

1Github | Linkedin | Twitter

Image Fusion Approaches

• Fusion Algorithms

• FPGA Based Architectures

• Hybrid Architectures

2

FUSION ALGORITHMS

3

4

HARDWARE ARCHITECTURRES

5

A Multi-sensor Image Fusion and Enhancement System for Assisting Drivers in Poor Lighting Conditions

- Li Tao, Hau Ngo, Ming Zhang, Adam Livingston and VijayanAsari (2005)

Flow

• DRC + Contrast enhancement – prior to fusion

• Color restoration – with chromatic info from CCD post fusion

• Nonlinear neighborhood dependent image enhancement

• wavelet transform based multiresolution image fusion

6

Hardware Utilization

• Pipelined enhancement architecture utilizes about 50% of the logic slices in Xilinx VirtexII XC2V2000-4FF896 FPGA

• At 67 MHz sustain a throughput rate of about 67 Mpx/sec or 63.9 1024×1024

• Driving applications – night or bad weather

7

Results of image enhancement, image alignment and image fusion: (a) original visible image; (b) enhanced visible image; (c) original thermal image; (d) ‘projective’ transformed visible image (enhanced) aligned with thermal image; (e) fused image with color restoration

8

Method of Image Fusion and Enhancement Using Mask Pyramid

- David C. Zhang, Sek Chai and Gooitzen Van der Wal (SRI , 2011)

Flow

• Mask pyramid based selection rule for LAP

• Acadia® II vision processor can align and fuse three 1280x1024 images in real time

• If any intelligent analysis from the source imager needs to be used, it can be programmed into mask pyramid.

• Flicker is a flashing effect occurs in fusion. This artifact is often seen in the fusion of LWIR and SWIR or Visible images, when both image sources contain reverse intensities at the same pixel locations. To improve temporal coherence and reduce flicker artifacts, a hysteresis mask is defined

9

FPGA System

10

(a) Visible image. (b) IR image noise. (c) Fusion without noise mask. (d) Fusion with noise mask. Note the increased contrast and speckle free sky.

11

Real-time Single FPGA-Based Multimodal Image Fusion System& Fast and Adaptive Bi-dimensional Empirical

Mode Decomposition for the Real-time Video Fusion - Michał Bartyś, Barbara Putz, Adrian Antoniewicz (2012)

Flow• Real-time fusion algorithm goals - effective data processing, smooth fused video output & low

complexity allowing for direct low-level implementation.

• Candidates: Laplacian pyramid method [14], shift-invariant discrete wavelet transform (SIDWT) method & FABEMD fusion algorithm [15, 16] based on the fast and adaptive bidimensionalmode decomposition.

12

13

•low cost, low power device•150K logic elements, •6.48 Mb of embedded memory, •360 18 × 18 multipliers, •475 user I/O arranged in 11 banks. •Running at 150MHz clock frequency. •Two banks of external DDRAM2 memories - 2x64Mx32b. •Synchronous burst SRAM (2Mx18b)

FPGA System

14

Performance

•Tests have been done with means of COTS kit based on Altera Cyclone IV FPGA

•Power dissipation of FPGA image fusion system is rated <4W.

•UFO intended to manage real-time image alignment and fusion operating at 50Hz and above.

Hardware Utilization

• Al times are achieved for input image resolution equal 640x480 pixels. FPGA is synchronized by 150MHz clock.

• Developed prototype of real-time fusion system is capable to process at 25 fps

• FPGA power dissipation rated <1W while performing real-time multimodal image fusion

15

*Antoniewicz “FPGA Implementation of Decomposition Methods for Real-Time Image Fusion”, Img Pr Com Challenges 4, Springer, 2013(Detailed implementation)

Examples of multimodal image fusion of images a:) TV image; b) IR image; c) TV image after morphological edge extraction; d) IR image after morphological edge extraction; e) fusion by averaging; f) fusion by SIDWT method; g) fusion by Laplacian pyramid; h) fusion by FABEMD method.

16

Others

Implementation of Image Fusion Techniques for Multi-Focus Images Using FPGA - MA. Mohamed and R.M EI-Den2 (2011)

• DCT, DWT and PCNN based Fusion• Xilinx Spartan 3AN • (1) "lab" of size 640x480; (2) "Pepsi" of size 512x512, and (3) "Toy" of size 512x512.

17

Design and Implementation of Image Fusion Technique Using DWT for Micro Air Vehicle - Dr. S. Narayana Reddy, C. Chandrasekhar (2014)

•Power consumption 0.10774W •Operates at a maximum frequency of 456.73MHz•Input 100x100 px @ 40Mhz

Hardware Utilization

18

HYBRID APPROACHES

19

• A mixed quasi-parallel structure with the DSP processors as well as FPGA matrices has been seriously taken into account.

• Evaluation of computational power of DSP processor chosen for eventual application in UFO system was done experimentally by application of special software tests. Investigations were arranged by means of the off-the-shelf video kit based on TMS320C6446 processor

20

Real-time Single FPGA-Based Multimodal Image Fusion System- Michał Bartyś, Barbara Putz, Adrian Antoniewicz (2012)

Performance

• Application of chosen DSP processor does not guarantee achievement of satisfactory processing times acceptable for implementation of image registering and fusion at video frame rates equal 40ms.

21

Image Fusion Real-time System Based on FPGA and Multi-DSP- Feng Qu, Bochao Liu, Jian Zhao, Qiang Sun (2013)

• In the final experiment, the image fusion of the five-band image with a frame rate of 15Hz, and a resolution of 1392 × 1040, is successfully completed in 41ms by this system.

• FPGA Alter Cyclone II - EP2C70F896, and DSP TI - TMS320C6416.

22

23

QUALITATIVE ANALYSIS

24

• Experimental Tests of Image Fusion for Night Vision - Yin Chen RS Blum (2005)

LAP and SIDWT gives best performance for Night vision

• Implementation of Image Fusion Techniques for Multi-Focus Images Using FPGA - MA. Mohamed and R.M EI-Den2 (2011)

25

Conclusion

• Best algorithm - Gradient pyramid, mask pyramid, FABEMD, SiDWT

• Least Hardware Complexity – LAP

• FPGA – low power (<1W), high throughput (25fps, 1208x1024)

• High Bandwidth - Need multiple DDR connected in parallel, SRAM is bonus

26