Image Compositing and Matting

Introduction

Matting and compositing are important operations in the production of special effects. These techniques enable directors to embed actors in a world that exists only in imagination, or to revive creatures that have been extinct for millions of years.

During matting, foreground elements are extracted from a film or video sequence.

During compositing, the extracted foreground elements are placed over novel background images.

Traditional approaches to matting

Traditional approaches to matting include blue-screen matting and rotoscoping.

The former requires filming in front of an expensive blue screen under carefully controlled lighting

The latter demands talent and intensive user interaction.

Rotoscoping

Rotoscoping --- the process of tracking contours in a video sequence

Basic problem of blue screen matting

Given an image of a foreground object shot in front of a backing color (blue screen or green screen or other colors)

Obtain a matte of the foreground object so that the foreground object can be blended into a new background image using the matte to produce a new composite image.

Notations Let C = [R, G,B] denote a color with 0 ≤ R, G, B ≤ 1. Let denotes a transparency value with 0 ≤ ≤ 1. Foreground image color: Cf = [Rf ,Gf ,Bf ], f = 1

Backing (screen) color: Ck = [0, 0,Bk], k = 1 (assuming blue screen)

Original foreground object color: Co = [Ro,Go,Bo]

Background image color: Cb = [Rb,Gb,Bb], b = 1

Composite image color: Cc = [Rc,Gc,Bc]

Problem Statement

Given Cf and Cb at corresponding pixels, and Ck a known backing color, and assuming Cf = oCo + (1 − o)Ck

Determine o and Co, which then gives the composite color Cc = oCo + (1 − o)Cb at the corresponding point, for all points that Cf and Cb share in common.

Solution

Since Cf = oCo + (1 − o)Ck , we have:

4 unknowns, 3 equations…

Rf = oRo Gf = oGo

Bf = oBo + (1 − o)Bk .

Case 1: No Blue

There is no blue in Co, i.e., Bo=0, and Bk≠0. Then

This case is very restrictive. It rules out many colors, including grays

because grays have blue.

Case 2: Gray and Skin Color

Assume Ro=aBo (or Go=aBo) and Bk≠0 Then

where

Example 1: Gray

Then

This case applies to science fiction movie in which the spaceships are mostly gray.

Example 2: Skin Color

This case applies to human faces, hands, legs, etc.

Solution for General Cases

To solve the matting problem in general: Need to take the same image with two different

backing colors. This gives four equations for solving the four

unknowns Case 1: Use Two Different Shades of Blue. Case 2: Use Two Different Backing Colors.

Case 1

Use two different shades of blue Bk1 and Bk2 as backing colors.

Then

so

Case 2

Use two different backing colors Ck1 and Ck2.

Or

Over-constrained, 1 unknowns, three equations

Case 2: Solution 1

Adding up 3 equations:

Case 2: Solution 2

Apply least squares method Define:

Then

Least-squares

Least-Squares

Set

Case 1 Example

Case 2 Example

Other Developments in Matting

Matting Without Blue Screen A method proposed by Ruzon and Tomasi

User specify object region and boundary region. Alpha value of object region is set to 1. Alpha value of boundary region is computed by es

timating the contributions of neighboring objects’ colors.

Example

Bayesian Approach

Shadow Matting

Pull a matte of shadow. Acquire photometric and geometric properties

of the target scene by sweeping oriented linear shadows across it.

Then, composite the shadow onto the scene.

Example

Environmental Matting

Left: alpha matte. Middle: environment matte. Right: photo.

References1. Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski. A baye

sian approach to digital matting. In Proc. IEEE CVPR, pages II–264–II–271, 2001.

2. Y.-Y. Chuang, D. B. Goldman, B. Curless, D. H. Salesin, and R. Szeliski. Shadow matting and compositing. ACM Transactions on Graphics, 22(3):494–500, July 2003.

3. M. A. Ruzon and C. Tomasi. Alpha estimation in natural images. In Proc. IEEE CVPR, pages 18–25, 2000.

4. A. R. Smith and J. F. Blinn. Blue screen matting. In Proc. ACM SIGGRAPH, pages 259–268, 1996.

5. D. E. Zongker, D. M. Werner, B. Curless, and D. H. Salesin. Environment matting and compositing. In Proc. SIGGRAPH, pages 205–214, 1999.

Digital Compositing

Digital compositing means “digitally manipulated integration of at least two source images to produce a new image.”

The new image must appear realistic. It must be completely and seamlessly integrat

ed, as if it were actually photographed by a single camera.

Example 1

Example 2

More examples

http://www.beezlebugbit.com/digital/efx/efx_top.htm

Main Topics

Alpha blending: blending foreground and background

Keying: separating foreground and background Luma, chroma, difference keying

Rig removal: removing unwanted elements

Alpha Blending

C = [α F + (1 – α) B] If α = 1, then C = F, foreground is shown, i.e.,

foreground is opaque. If α = 0, then C = B, background is shown, i.

e., foreground is transparent. 0 < α < 1: semi-transparent, e.g., shadow, sm

oke, etc. If α ranges from 0 to 255, then the formula be

comes: C = [α F + (1 – α) B] / 255

Example: No Background

Example: With Background

Note

For shadow, must take fractional value (0 < α < 1). Otherwise, shadow looks unreal.

Boundary area

at boundary area should also be fractional. Otherwise, have dark fringes; unrealistic.

Summary

A good matte has fractional in shadow, and along object boundaries and shadow boundaries.

Keying

Separating foreground from background, creating a matte of foreground.

Also called pulling a matte (of foreground), or keying out (i.e., making transparent) background.

Recall: A good matte has fractional a in shadow, and alon

g object boundaries and shadow boundaries.

Basic methods

Luma keying: based on luminance (i.e., intensity)

Chroma keying: based on color (i.e., blue screen, green screen)

Difference keying: requires a clean plate, i.e., a background image without the foreground element.

Basic Idea

Compute difference between foreground and background (based on luma, chroma, or color)

Very small diff = 0. Very large diff = 1. Intermediate diff inte

rmediate

Luma Keying

Key out the background based on luminance. Useful when background has a uniform lumin

ance that is very different from foreground luminance.

Result

Chroma Keying

Key out the background based on color. Useful when background has a uniform color

that is very different from foreground color. Example: Image shot with blue screen.

Characteristics of blue screen image

Difference Keying

More general than luma and chroma keying. Key out background based on pixel-wise color di

fference between foreground and background footage.

Final Composition

Rig Removals

Rigs are equipment that support the actors or the props.

Sometimes, rigs cannot be removed by keying alone. So, have to apply masking technique to remove rigs. Need clean plate of background footage. If camera moves, then need motion-controlled came

ra: Computer controls camera to move the same way t

wice: Without foreground objects; get clean plate. With foreground objects.

Basic Idea

Apply a mask to mask out the rig. Then, replace pixels in masked area by corre

sponding pixels in clean plate background. If rig moves in footage, then have to animate

the mask accordingly.