region filling and object removal by exemplar based image inpainting
TRANSCRIPT
![Page 1: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/1.jpg)
Region Filling and Object Removal by Exemplar-Based Image Inpainting
Criminisi, A., Perez, P., and Toyama, K. (2004)
Lee, WoongheeM.S. student at the Big Data Mining Lab.
Department of computer science and engineering at the Hanyang UniversityOctober 4th, 2015
![Page 2: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/2.jpg)
Contents Prerequisite Key Observations Region Filling Algorithm Results and Comparisons
![Page 3: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/3.jpg)
Prerequisite Texture synthesis Inpainting
![Page 4: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/4.jpg)
PrerequisiteTo fill large image repetitively 2-D texture synthesis
inpainting
Exmaples from Ashikimin [1]
![Page 5: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/5.jpg)
PrerequisiteProperties
• Cheap cost to generate image• Effectively generating image• Difficult to fill holes in photos• A complex product of mutual
influences between different boundaries
texture synthesis
inpainting
![Page 6: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/6.jpg)
PrerequisiteTo fill holes in images by propagating linear structures (called isophote)
texture synthesis
inpainting
![Page 7: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/7.jpg)
PrerequisiteTo fill holes in images by propagating linear structures (called isophote)
Depends on Gestalt Law of Continuation
texture synthesis
inpainting
![Page 8: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/8.jpg)
PrerequisiteGestalt Law of Continuation texture synthesis
inpainting
![Page 9: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/9.jpg)
PrerequisiteGestalt Law of Continuation texture synthesis
inpainting
![Page 10: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/10.jpg)
PrerequisiteGestalt Law of Continuation
Human perceives a dotted line as a full line by implicit continuation.
texture synthesis
inpainting
![Page 11: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/11.jpg)
PrerequisitePropagation direction texture synthesis
inpainting
propagate along isophotes
![Page 12: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/12.jpg)
PrerequisiteProperties
• Effective to fill speckles, scratches, and overlaid text
• Causes noticeable blur to fill large regions
• Extremely slow (83’-158’ on a 384 X 256 image)
texture synthesis
inpainting
![Page 13: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/13.jpg)
Main Idea
To combine the advantages of “texture synthesis” and “inpainting”
![Page 14: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/14.jpg)
Key ObservationsA. Exemplar-Based Synthesis SufficesAlgorithm Core: Isophote-driven image-sampling process
![Page 15: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/15.jpg)
Key ObservationsA. Exemplar-Based Synthesis Suffices
![Page 16: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/16.jpg)
Key ObservationsB. Filling Order is Critical
artefacts
![Page 17: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/17.jpg)
Key ObservationsB. Filling Order is Critical
Onion peel(concentric-layer odering) causes “over shooting” → To achieve balancing between the structured regions and texture regions.
![Page 18: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/18.jpg)
Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ𝑝|
𝐷 𝑝 =|∇𝐼𝑝
⊥∙𝑛𝑝|
𝛼
Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω
![Page 19: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/19.jpg)
Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ𝑝|
𝐷 𝑝 =|∇𝐼𝑝
⊥∙𝑛𝑝|
𝛼
Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω
higher priority
lower priority
![Page 20: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/20.jpg)
Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ𝑝|
𝐷 𝑝 =|∇𝐼𝑝
⊥∙𝑛𝑝|
𝛼
Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω
similar priority
![Page 21: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/21.jpg)
Region Filling Algorithm2) Propagating Texture and Structure InformationAfter computing priorities, setting the highest priority Ψ 𝑝
To avoid diffusion, propagating image texture from the source region
Ψ 𝑞 = arg𝑚𝑖𝑛Ψ𝑞∈Φ𝑑(Ψ 𝑝, Ψ𝑞)
![Page 22: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/22.jpg)
Region Filling Algorithm3) Updating Confidence ValuesAfter filling the patch Ψ 𝑝, the confidence term is updated
𝐶 𝑝 = 𝐶 𝑝 , ∀𝑝∈ Ψ 𝑝 ∩ Ω
It does not require additional parameter to specify image.
![Page 23: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/23.jpg)
Region Filling AlgorithmThe 𝑡 indicates the current iteration.
![Page 24: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/24.jpg)
Region Filling AlgorithmProperties of the region filling algorithm
Recall 𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
The priority equation achieves balance of effects and an organic synthesis
![Page 25: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/25.jpg)
Region Filling AlgorithmProperties of the region filling algorithm
𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
• avoids an arbitrary fill order.• eliminates the risk of “broken-structure”
artefacts.• propagates strong edges.• reduces blocky and misalignment artefacts
without additional step.
![Page 26: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/26.jpg)
Region Filling AlgorithmImplementation Details
The target 𝛿Ω is manually selected.The normal direction 𝑛𝑝 is computed as1) Contour’s “control” points are filtered via
2D Gaussian kernel2) estimated as the orthogonal unit vector of
𝛿Ω
![Page 27: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/27.jpg)
Region Filling AlgorithmImplementation Details
The gradient ∇𝐼𝑝is computed as the MAX value in Ψ𝑝 ∩ 𝐼
Pixels are classified as belonging to• The target region Ω• The source region• The remainder
![Page 28: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/28.jpg)
Results and Comparisons
Experimental environment was a 2.5-GHz Pentium IV with 1GB of RAM.
To compare with the results of earlier work.
![Page 29: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/29.jpg)
Results and ComparisonsKanizsa Triangle and the Connectivity Principle
![Page 30: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/30.jpg)
Results and ComparisonsComparing Different Filling Orders
original image target region
raster-scan concentric
Harrison’s2 m 45 s
Ours5 s
![Page 31: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/31.jpg)
Results and ComparisonsComparing Different Filling Orders
original image target region
![Page 32: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/32.jpg)
Results and ComparisonsComparing Different Filling Orders
raster-scan concentric
![Page 33: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/33.jpg)
Results and ComparisonsComparing Different Filling Orders
Harrison’s45 m
Ours2 s
![Page 34: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/34.jpg)
Results and ComparisonsComparing Different Filling Orders
Using only data term leads the “over shoot”
![Page 35: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/35.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
original image target region
![Page 36: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/36.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
onion peel ours
![Page 37: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/37.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
onion peel ours
![Page 38: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/38.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
onion peel ours
![Page 39: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/39.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
onion peel ours
![Page 40: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/40.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
priority function for before image
Priority function is 0 for inside and 1 for outside Final priorities made the continuation of the pole
![Page 41: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/41.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
original image target region
![Page 42: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/42.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
Isophotes hits the thin boundary
![Page 43: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/43.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
ours traditional image inpainting (blurry)
![Page 44: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/44.jpg)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
target region
texture and structure inpainting (blurry) ours
![Page 45: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/45.jpg)
Comparison With Drori et al.
Results and Comparisons
Drori et al. (blurry)
![Page 46: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/46.jpg)
Examples on PhotographsResults and Comparisons
Drori et al. (blurry)
![Page 47: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/47.jpg)
Examples on PhotographsResults and Comparisons
Drori et al. (blurry)
onion peel ours
![Page 48: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/48.jpg)
Examples on PhotographsResults and Comparisons
ours
“bow-tie” effect
![Page 49: Region filling and object removal by exemplar based image inpainting](https://reader033.vdocuments.mx/reader033/viewer/2022052514/587180ce1a28ab230b8b779f/html5/thumbnails/49.jpg)
Thank you for listening