real-time edge-aware image processing with the bilateral grid

Real-Time Edge-Aware Image Processing with the Bilateral Grid

SIGGRAPH '07Jiawen Chen, Sylvain Paris, Frédo Durand

CSAIL - MIT

By Orazio Gallo, 01/29/08

Motivation

● Many computational photography applications require piecewise smooth filtering

● Bilateral filtering is effective but slow: minutes per megapixel

● Many approximations and optimizations have already been proposed (~seconds per megapixel)

● Real time?

Paper's Main Contribution

● The bilateral grid: a 3D data structure that naturally embeds edge information and allows for parallelization

● Now everything is real-time!● “Edge-aware” algorithms for different image

processing tasks

Talk Overview

● Super-quick overview of bilateral filtering (?)● The data structure and its use● Example apps

Bilateral Filtering

Edge + noise De-noised edge

x

(Adapted from Durand and Forsey 2002)

More on Bilateral Filtering

Input

Output

space

p

q

p

inte

nsity

inte

nsity

More on Bilateral Filtering

Input

Output

space

p

qinte

nsity

inte

nsity

p

Bilateral Grid

The Bilateral GridStandard 2D Image

x

y

• Nearest neighbor

• Bicubic

2D vs. Bilateral Grid Downsampling

x

y

Downsampling in 2D

x

y

Downsampling in 2D Bilateral Grid

Extra dimension compensates for aggressive down-sampling!!

2D vs. Bilateral Grid Downsampling

(70 x 70 x 10 for an 8 megapixel image)

Processing and Slicing● Processing:

any function that takes a 3D function as input!

● Slicing – getting back an image:use the original image to access the grid. The intensity level is obtained by interpolating along the z direction in the grid.

Bilateral Filter

Bilateral Filter on the Bilateral GridImage scanline

space

inte

nsityBilateral

Grid

space

inte

nsity

Bilateral Filter on the Bilateral GridImage scanline

Filtered scanline

Slice: query gridwith input image

BilateralGrid

Gaussian blur grid values

space

inte

nsity

Bilateral Filter on the Bilateral Grid

s is the geometric spread

r is the photometric spread

Bilateral Filter Performance

Image size: 2 MPixels● CPU

– Brute force: 10 minutes– State of the art ’06: 1 second [Weiss 06, Paris 06]

● Our Bilateral Grid with GPU– 2004 card (NV40): 28 ms (36 Hz)– 2006 card (G80): 9 ms (111 Hz)

Bilateral Filter Performance

Show:✔ bf.wmv

Grid Painting

Grid Painting

Grid PaintingImage scanline

space

inte

nsityBilateral

Grid

space

inte

nsity

Grid PaintingImage scanline

Output brush stroke

Query grid with input image: slicing

BilateralGrid

Grid Painting

Show:✔ gp_3d.wmv✔ gp_recut.wmv

Tone Mapping for HDRI

HDRI and Tone Mapping

● CRT displays 100:1● LCD displays up to ~1'000:1● Photographic prints up to ~1'000:1● Real world scenes: > 100'000:1

HDRI and Tone Mapping

From Wikipedia

HDRI and Tone Mapping

From Wikipedia

HDRI and Tone Mapping

The algorithm [Durand and Dorsey 02]

Color ratiosDetail

Large scale

Preserve

Linear remapping

exp

Log luminance

User-Driven, Local Tone Mapping

User-Driven, Local Tone Mapping

Show:✔ ToneMapping.wmv

Other Applications: Histogram Equalization

Matlab's adapthisteq (1.5s) Their result (100ms)

1760 x 2140, HDR

Other Applications: Transfer of Photographic Look [Bae 06]

● Artistic rendition from model– 2 bilateral filters

● Remove compression artifacts● Split into large-scale and small-scale layers

– 2 histogram transfers● Match statistics on layers

● 1.5 megapixels at 30 Hz

Model

Input

Output

Summary● Definition of the Bilateral Grid:

– Intuition, creation, processing, and slicing● Applications

– Bilateral filtering– Grid painting– Local tone mapping– Local histogram equalization– Transfer of Photographic look– ...

thanks!