Compositing & Blending

Tamara Berg Computational Photography

Many slides from: Alexei Efros & Derek Hoiem

Image Compositing & Blending

© NASA

This Class How do I put an object from one image into another?

Slide from: Derek Hoiem

Image Compositing

Recent news

Original

“Enhanced” Version

http://www.guardian.co.uk/world/2010/sep/16/mubarak-doctored-red-carpet-picture

Slide from: Derek Hoiem

Compositing Procedure 1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)

Composite by David Dewey

2. Blend them into the composite (in the right order)

Three methods

1.  Cut and paste

2.  Laplacian pyramid blending

3.  Poisson blending

Slide from: Derek Hoiem

Method 1: Cut and Paste

Slide from: Derek Hoiem

Method 1: Cut and Paste Method: •  Segment using intelligent scissors •  Paste foreground pixels onto target region

Slide from: Derek Hoiem

Method 1: Cut and Paste

Problems: •  Small segmentation errors noticeable •  Pixels are too blocky •  Won’t work for semi-transparent materials

Slide from: Derek Hoiem

Feathering Near object boundary pixel values come partly from foreground and partly from background

Slide from: Derek Hoiem

Method 1: Cut and Paste (with feathering)

Slide from: Derek Hoiem

Alpha compositing

+

Output = foreground*mask + background*(1-mask) Slide from: Derek Hoiem

Alpha compositing with feathering

Output = foreground*mask + background*(1-mask) Slide from: Derek Hoiem

Proper blending is key

Proper blending is key

Alpha Blending / Feathering

0 1

0 1

+

= Iblend = αIleft + (1-α)Iright

Setting alpha: simple averaging

Alpha = .5 in overlap region

Setting alpha: center seam

Alpha = logical(dtrans1>dtrans2)

Distance Transform bwdist!

Setting alpha: blurred seam

Alpha = blurred

Distance transform

Setting alpha: center weighting

Alpha = dtrans1 / (dtrans1+dtrans2)

Distance transform

Ghost!

Alpha Blending / Feathering

0 1

0 1

+

= Iblend = αIleft + (1-α)Iright

Affect of Window Size

0

1 left

right 0

1

Large windows -> lots of ghosting

alpha

Affect of Window Size

0

1

0

1

Small window –> seams are too obvious

Good Window Size

0

1

“Optimal” Window: smooth but not ghosted

Medium window -> just right!

What is the Optimal Window? To avoid seams

•  window = size of largest prominent feature

What is the Optimal Window? To avoid seams

•  window = size of largest prominent feature

To avoid ghosting •  window <= 2*size of smallest prominent feature

What is the Optimal Window? To avoid seams

•  window = size of largest prominent feature

To avoid ghosting •  window <= 2*size of smallest prominent feature

Natural to cast this in the Fourier domain

FFT

Fourier domain blending

FFT

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright)

FFT

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + …

FFT

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + … •  Feather corresponding octaves Fleft

i with Frighti

FFT

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + … •  Feather corresponding octaves Fleft

i with Frighti

•  Sum feathered octave images in frequency domain

FFT

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + … •  Feather corresponding octaves Fleft

i with Frighti

•  Sum feathered octave images in frequency domain •  Compute IFFT to get final blended result.

FFT

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + … •  Feather corresponding octaves Fleft

i with Frighti

•  Sum feathered octave images in frequency domain •  Compute IFFT to get final blended result.

FFT

Actually better implemented in the spatial domain

Image Pyramids

Known as a Gaussian Pyramid [Burt and Adelson, 1983] •  In computer graphics, a mip map [Williams, 1983]

- Reminder

A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose

Figure from David Forsyth

Gaussian Pyramid

Laplacian Pyramid

Lowpass Images

Laplacian Pyramid

Bandpass Images

Lowpass Images

Laplacian Pyramid

Lowpass Images

Laplacian Pyramid

Lowpass Images

Laplacian Pyramid

Lowpass Images

Bandpass Images

Pyramid Blending

0

1

0

1

0

1

Left pyramid Right pyramid blend

•  At low frequencies, blend slowly •  At high frequencies, blend quickly

Method 2: Laplacian Pyramid Blending

Implementation: 1. Build Laplacian pyramids for each image 2. Build a Gaussian pyramid of region mask 3. Blend each level of pyramid using region mask

from the same level

4. Collapse the pyramid to get the final blended image

)1(2112iiiii RLRLL −⋅+⋅=

Burt and Adelson 1983

Region mask at level i of Gaussian pyramid

Image 1 at level i of Laplacian pyramid

laplacian level

4

laplacian level

2

laplacian level

0

left pyramid right pyramid blended pyramid

Pyramid Blending

Blending Regions

Horror Photo

© david dmartin (Boston College)

Season Blending (St. Petersburg)

Season Blending (St. Petersburg)

Simplification: Two-band Blending Brown & Lowe, 2003

•  Only use two bands: high freq. and low freq. •  Blends low freq. smoothly •  Blend high freq. with no smoothing: use binary alpha

Low frequency (λ > 2 pixels)

High frequency (λ < 2 pixels)

2-band Blending

Linear Blending

2-band Blending

Practice Quiz Poisson Image Editing: 1) “Solving the Poisson equation computes the function whose gradient is the closest, in the L2 norm, to some prescribed vector field under given boundary conditions” In blending: What does function refer to? What is an example prescribed vector field?

2) How might you implement texture flattening using Poisson techniques? Photomontage: 3) What are the 2 main techniques used in this paper?

4) What is an image objective? Name 3 possible objectives. Seam Carving: 5) What is the definition of a seam?

6) What algorithm would you use to efficiently find the optimal seam for removal in an image? Describe generally how to implement this.

Break

Related idea: Poisson Blending A good blend should preserve gradients of source region without changing the background

Perez et al. 2003

Related idea: Poisson Blending A good blend should preserve gradients of source region without changing the background

Perez et al. 2003

HW 2!

What’s with Poisson?

Solving Poisson – computes the function whose gradient is the closest, in the L2 norm, to some prescribed vector field under given boundary conditions

What’s that mean?

Given a guidance field (e.g. gradient field from source image, or from a composite image)

Calculate the image that best fits this vector field

In general no image exists whose gradient exactly matches input. Best-fit image in least squares sense can be calculated by solving Poisson equation.

Perez et al. 2003

HW2: Gradient Domain Editing

General concept: Solve for pixels of new image that satisfy constraints on the gradient and the intensity •  Constraints can be from one image (for filtering) or more (for

blending)

HW2: Poisson Blending

A good blend should preserve gradients of source region without changing the background

Treat pixels as variables to be solved

•  Minimize squared difference between gradients of foreground region and gradients of target region

•  Keep background pixels constant

Perez et al. 2003

Example

Source: Evan Wallace

Gradient Visualization

Source: Evan Wallace

+ Specify object region

Other results

Perez et al. 2003

What do we lose?

Perez et al. 2003

Foreground color changes Background pixels in target region are replaced

Blending with Mixed Gradients Use foreground or background gradient with

larger magnitude as the guiding gradient

Perez et al. 2003

Concealment

Transparent Objects

NPR

Things to remember

Three ways to blend/composite 1.  Alpha compositing

–  Need nice cut (intelligent scissors) –  Should feather

2.  Laplacian pyramid blending –  Smooth blending at low frequencies, sharp at high

frequencies –  Usually used for stitching

3.  Gradient domain editing –  Also called Poisson Editing –  Explicit control over what to preserve –  Changes foreground color (for better or worse) –  Applicable for many things besides blending

Don’t blend, CUT!

So far we only tried to blend between two images. What about finding an optimal seam?

Moving objects become ghosts

Davis, 1998 Segment the mosaic

•  Single source image per segment •  Pick a good seam (cut) so that when you merge the

segments is looks ok.

- Details about this next week!

Kwatra et al, 2003

Putting it all together Compositing images

•  Have a clever blending function –  Feathering –  Center-weighted –  blend different frequencies differently

•  Choose the right pixels from each image –  Graph-cuts

Interactive Digital Photomontage

Aseem  Agarwala,  Mira  Dontcheva,  Maneesh  Agrawala,  Steven  Drucker,  Alex  Colburn,  Brian  Curless,  David  Salesin,  Michael  Cohen  

Challenges Find good seams between parts of images so

they can be joined with few visible artifacts Blend along seams to reduce or remove any

artifacts remaining after joining

Seam Objectives Measures suitability of a seam between two image regions

e.g. •  Match colors across seams •  Match colors and color gradients across

seams •  Match colors across seams, but prefer seams

that lie along edges

User Selection

Result of Pasting

Gradient Smoothing

Remove any remaining artifacts after merging source image parts

View input images as sources of gradient information rather than color.

Using graph cut labeling, form composite whose gradients best match source gradient vectors.

Final Result after smoothing

Extended Depth of Field

Input  Images  

Source  Map  

Extended Depth of Field

Relighting

Input  Images  

Source  Map  

Relighted Result

Visualizing Movement

Visualizing Movement

Selective Composites

Selective Composites

Selective Composites

Selective Composites

Background    reconstruc?on  

Background  reconstruc?on  

Religh?ng  

Composi?ng  and  Obstruc?on  removal  

Fictional Composites

Photo Clip Art

Jean-François Lalonde, Derek Hoeim, Alexei A. Efros, Carsten Rother, John Winn and Antonio Criminisi. Photo Clip Art, ACM Transactions on Graphics (SIGGRAPH 2007),

Photo Clip Art

Photo Clip Art

Street accident

Bridge

Painting

Alley

Failure cases

Shadow transfer

Porous objects

Failure cases

Best matching objects

Inverse of previous topic Removing content (as opposed to adding it)

Next…

Seam Carving for Content-Aware Image Resizing Shai Avidan Ariel Shamir

Slides  from:  Kshitij  Padalkar  

In  One  Line…  

Method  for  Content-­‐Aware  Image  Resizing    

Need  

Different  Displays-­‐Different  Resolutions-­‐(Different  Images??)  

HTML  allows  automatic  layout  of  the  text,  but  not  the  images  

Solu?ons  and  Other  Work  Crop  and  Scale  Saliency  Segmentation    

Defini?on…  

Remove  unnoticeable  pixels.  Energy  Function:  

� Remove Pixels with least energy �  Sort pixels by Energy, remove smallest pixels (optimum) �  Remove Column with smallest total Energy (column) �  Remove Seam with smallest total Energy (seam)

Seam  

Vertical  Seam:     �  an 8-connected path of pixels in the image from

top to bottom, containing one, and only one, pixel in each row of the image

Seam  

Energy  of  a  Seam  

� Minimum Energy Seam

Pixel  Removal  Criteria  Optimal  

Pixel  Removal  Criteria  Column  

Pixel  Removal  Criteria  Seam  

Discrete  Image  Resizing  Aspect  Ratio  Change  

•  Optimal  Seam  Order  

Image  Enlarging  Content  Amplification  Object  Removal    

Aspect  Ra?o  Change  Change  the  image  from  (n  X  m)  to  (n  X  m’)  

•  (assume    m’  <  m)  

Remove  m-­‐m’=c  seams  successively  

Seam Carving

Aspect  Ra?o  Change  Change  the  image  from  (n  X  m)  to  (n  X  m’)  

•  (assume    m’  <  m)  

Remove  m-­‐m’=c  seams  successively  

Scaling

Op?mal  Seam  Ordering  

Change  the  image  from  (n  X  m)  to  (n’ X  m’)  •  (assume    m’  <  m  and  n’  <  n)  

What  to  do?  What  is  the  correct  order  of  seam  carving?  •  Remove  vertical  seams  first?    •  Horizontal  seams  first?    •  Or  alternate  between  the  two?  

Op?mal  Seam  Ordering  

Transport  map  •  Matrix  of  size  n  X  m  •  Each  element  T(r,c)  holds  the  minimal  cost  needed  to  obtain  an  

image  of  size  n−r  X m−c    

Discrete  Image  Resizing  Aspect  Ratio  Change  

•  Optimal  Seam  Order  

Image  Enlarging  Content  Amplification  Object  Removal    

Image  Enlarging  

�  compute  the  optimal  vertical  (horizontal)  seam  ‘s’  on  I  and  duplicate  the  pixels  of  s  by  averaging  them  with  their  left  and  right  neighbors  (top  and  bottom  in  the  horizontal  case)  

Image  Enlarging  

� To  reduce  the  width  by  ‘k’,  compute  first  ‘k’  vertical  seams  and  duplicate  them  

� All  seams  duplicated,  50%  enlargement.  For  more  do  it  in  steps.  

Discrete  Image  Resizing  Aspect  Ratio  Change  

•  Optimal  Seam  Order  

Image  Enlarging  Content  Amplification  Object  Removal    

Content  Amplifica?on  � Scale the image, this will scale everything, content as

well as non-content. � Shrink the scaled-image by seam carving, this will

carve out the non-content part.

Discrete  Image  Resizing  Aspect  Ratio  Change  

•  Optimal  Seam  Order  

Image  Enlarging  Content  Amplification  Object  Removal  

Object  Removal  

� User marks the target object to be removed. � Then seams are removed from the image until all

marked pixels are gone. � To regain the original size, seam insertion is

employed.

Object  Removal  �  Find  the  missing  shoe…  

Object  Removal  � Object marking to remove unwanted results

Limita?ons  

� Too much content � No space for seam to avoid content

Announcements HW2 up later today Readings for next week on web page