poisson image editing - computer sciencelazebnik/research/fall08/jia_pan.pdfcomposition image i is...
TRANSCRIPT
Poisson Image Editing
SIGGRAPH 2003Patric Perez Michel GangnetAndrew Black
Most Frequently Used PDE
Wave Equation
Heat Equation
Most Frequently Used PDE
Poisson Equation, Steady State of Wave Equation and Heat Equation
Laplace’s Equation
Boundary Conditions
Dirichlet Boundary ConditionsSpecify the value of the function on a surface
Neumann Boundary ConditionSpecify the normal derivative of the function on a surface
Guided Interpolation
f: to be solved, f*: known regionv: guided field, g: v is prob. gradient of g
Simple Interpolation
Maximize the Smoothness
Solution: Laplace Equation with DirichletBoundary Conditions
Guided Interpolation
Interpolation-> minimization
Solution: Poisson Equation with DirichletBoundary Conditions
Relationship with Laplace case?
Or from vector field decomposition
Helmholtz-Hodge decomp.
Discrete Poisson SolverDiscretize the Minimization Directly
Partial Derivative
Partial Derivative for Interior Points
Discrete Poisson Solver
Linear System of EquationsGauss-Seidel Method with Successive OverrelaxationV-cycle MultigridDiscretize Laplacian with Discrete Laplacian of Gaussianhttp://www.tau.ac.il/~stoledo/taucs/ Taucs
Seamless Cloning :Importing Gradients
Importing Gradients from a Source Image
Discretize
Seamless Cloning Results
Seamless Cloning Results
Texture
Alignment
Transfer intensity only
Seamless Cloning: Mixing Gradients
Two ProposalsDefine v as Linear Combination of Source and Destination GradientsSelect Stronger one from Source and Destination Gradients (not conservative!)
Discretization
Mixing Gradients Results
Mixing Gradients Results
Mixing Gradients Results
Just reduce terrible things ! Another solution
Texture Flattening
Remain Only Salient Gradients
Discretization
Texture Flattening
Edge mask
Local Illumination Changes
Fattal Transformation
Local Color Changes
Mix two different colored version of original image
One provide f* outsideOne provide g inside
Local Color Changes
Monochrome of backgroundWhite in flower
Seamless Tiling
Select original image as gBoundary condition:
f*north=f*
south=0.5(gnorth+gsouth)Similarly for the east and west
Seamless Tiling
Discussion
Discussion
Discussion
Discussion
Drag-and-Drop Pasting
Leo Jiaya Jia Leo Jiaya Jia Jian Sun Jian Sun ChiChi--Keung TangKeung TangHeungHeung--Yeung ShumYeung Shum
The Chinese University of Hong KongMicrosoft Research AsiaThe H.K. University of Sci. & Tech.Microsoft Research Asia
SIGGRAPH 2006
Slides by the authors
Introduction to our method
Our method improves the Poisson image editing with
a new boundary optimization algorithm,an easier user interface,and an integration of alpha values.
Poisson equations in images
A case study
+
sftf
Poisson equations in imagesA case study
+
sftf
Poisson equations in imagesThe optimization problem in image blending [Perez et al. 2003]
Taking , we have
What does it imply?
0 00
2
pmin | | with | |f s tf f dp f f∂Ω ∂Ω∈Ω
∇ −∇ =∫ 0 00
2
pmin | | with | |tsf dp ff ff ∂Ω ∂Ω∈Ω
=∇ −∇∫ 0 00
2
pmin | | with | | f s tf f fdp f∂Ω Ω ∂Ω∈
∇ −∇ =∫sf∇
tf'
sf f f= −
0 00
2' p
min | ' | with ' | |f t sf dp f f f∂Ω ∂Ω∈Ω∇ = −∫ 0
t sf f−
Poisson equations in images
The minimization problem equals to solving the Laplace equation:
Image blending should take both the source and the target images into consideration.Property of solving the Laplace equation:
The variational energy will approach zero if and only if all boundary pixels satisfy , where k is a constant value.
0 0' 0 with ' | |t sf f f f∂Ω ∂ΩΔ = = −
0
2| ' |fΩ
∇∫0
( ) |t sf f k∂Ω− =
Poisson equations in images
Where is the optimal boundary ?Inside the user drawn regionOutside the object of interest
How is the object extracted?Lazy snapping or Grabcut [Rother et al. 2004] (September 23)
How to optimize it?Minimum color variance
∂Ω
2min (( ( ) ( )) ) , s.t. bluet sp
f p f p k∈∂Ω
− − ∂Ω∈∑
Boundary optimization
∂Ω and k are all unknownsAn iterative optimization
Initialize ∂Ω as the user drawn boundary.Given new ∂Ω, the optimal k is computed:
Given new k, optimize the boundary ∂Ω.Repeat the previous two steps until convergence.
2( , ) (( ( ) ( )) ) , s.t. bluet sp
E k f p f p k∈∂Ω
∂Ω = − − ∂Ω∈∑
( , ) 0E kk
∂ ∂Ω=
∂Shortest path problem
Boundary optimizationIn 2D graph, computing the shortest path between any two points: Dynamic ProgrammingOur problem is to compute a closed path
Boundary optimizationA shortest closed-path algorithm
Breaking closed boundary
Boundary optimizationA shortest closed-path algorithm
Boundary optimizationA shortest closed-path algorithm
Computation complexity O(N)
Boundary optimizationA shortest closed-path algorithm
Total computation complexity O(NM)
Boundary optimization discussion
OptimalityAvoiding that the path twists around the cut by selecting the initial cut position.
How to select the initial cut?Making it short to reduce O(MN)Passing smooth region
One example
Integrating fractional boundary
Fractional boundary is important in image composting: (transparency)
MattingComposition Image I is generated by foreground and background with alpha matteI = alpha*F + (1-alpha)*BMatting is a problem to get alpha,F,B from a given image I.User have to devide the image into three region: Foreground, Background and Unknown area. In foreground area, F = I, alpha = 1, B = 0; In background area, F = 0, alpha = 0, B = I. Our task is to get F,B,alphain unknown area.
Integrating fractional boundary
Where to use the fractional values?only the pixels where the optimized boundary is near the blue ribbon
∂Ω
Integrating fractional boundary
Where to use the fractional values?only the pixels where the optimized boundary is near the blue ribbon
fractional integration: the green region
otherwise: the yellow region
'
( , ) ( , ), ( 1, )( , ) ( (1 ) ) ( , ), ( 1, )
0 otherwise
x s
x x s t
f x y x y x y yellowv x y f f x y x y greenα α
∇ + ∈⎧⎪= ∇ + − + ∈⎨⎪⎩
Integrating fractional boundary
How to integrate the fractional values in Poisson blending?
A blended guidance field
'
( , ) ( , ), ( 1, )( , ) ( (1 ) ) ( , ), ( 1, )
0 otherwise
s
s
x
x x t
f x y x y x y yellowv x y f f x y x y greenα α
∇ + ∈⎧⎪= ∇ + − + ∈⎨⎪⎩
'
( , ) ( , ), (( , ) ( (1 ) ) ( , ), ( 1, )
0 otherwi
1, )
sex x s
x s
tv x y f f x y x y greef x y x y x y
nyellow
α α⎧⎪= ∇ + −
∇ ∈+ ∈⎨
+
⎪⎩
( , ) ( 1, ) ( , )x f x y f x y f x y∇ = + −
' ( (1 ) ) ( , ), ( 1, )( , ) ( , ), ( 1, )
( , )0 otherwise
x s t
x s
x
f x y x yf f x y x y gr
x y yellowv y nx eeα α
∇ +∇ + − +
∈= ∈⎧⎪⎨⎪⎩
Integrating fractional boundaryHow to integrate the fractional values in Poisson blending?
A blended guidance field
'
0 o
( , ) ( , ), (
therw
1, )( , ) ( (1 ) ) ( , ), ( 1,
s)i e
x s
x x s t
f x y x y x y yellowv x y f f x y x y greenα α
∇ + ∈⎧⎪= ∇ + − + ∈⎨⎪⎩
Integrating fractional boundaryHow to integrate the fractional values in Poisson blending?
A blended guidance field
Integrating fractional boundary
Final minimization:
New boundary: Solving the corresponding Poisson equation.
2* *p *
min | ' | with | |f tf v dp f f∂Ω ∂Ω∈Ω∇ − =∫
Results and comparison
Alpha blending
Results and comparison
Poisson blendingOur method
Results and comparison
Alpha blendingOur method
Results and comparison
Poisson blendingOur method
Poisson Image Editing Extended
SIGGRAPH 2006 sketch
Daniel Leventhal Brown Univ.Bernard Gordon Brown Univ.Peter G. Sibley Brown Univ.
Problem caused by texture
Alpha control
Alpha in [0, 1], background 0, and foreground 1.Obtain unselected area’s Alpha by blurring.
Implemented in YUV rather than RGB
Result
Luminance rescaling
Far from solved…
Only pure texture, how about mixture of texture and non-textureFar from perfect…
Parallel Method on Mesh
Poisson based mesh editingOptical Boundaries for mesh mergingExtend from image space to mesh manifold
Mesh Editing with Poisson-Based Gradient Field Manipulation
SIGGRAPH 2004Yizhou Yu UIUCKun Zhou MSRADong Xu Zhejiang Univ, MSRAXiaohan Shi Zhejiang Univ, MSRAHujun Bao Zhejiang Univ.Baining Guo MSRAHeung-Yueng Shum MSRA
Mesh merging
Deformation
Basic idea
Quite similar to the case of imageVector field decomposition can be extended to manifold
Discretization
NOT real gradient and divergence, but it works
Sparse linear system
Mesh Merging
Some ‘ugly’ detailsBoundary interactionBoundary correspondenceRe-parameterization
Mesh deformation
Interactively change guided vector fieldChange normal on a curvePropagate to other areas
Smooth normal field – smooth
Optimal Boundaries for Poisson Mesh Merging
SPM 2007Xiaohuang Huang Zhejiang Univ. & HUSTHongbu Fu HUSTOscar Kin-Chung Au HUSTChiew-Lan Tai HUST
Basic idea
Best boundary for Poisson MergingSimilar to ‘Drag-and-Drop Pasting’.
Optimal boundary
2( , ) (( ( ) ( )) ) , s.t. bluet sp
E k f p f p k∈∂Ω
∂Ω = − − ∂Ω∈∑
Mesh ‘ugly’ limitation
Correspondence between source and target images are trivialMeaningful correspondence between source and target meshes are VERY difficult.
Results
Failure?