Mixed-Resolution Patch-Matching

Harshit Sureka

Guide: Dr. P. J. Narayanan

Patch-Matching

201/08/13

Scope of Patch-Matching Algorithms

Image credits: Image Montage (Simakov et al. 2008). Object Detection,Forgery Detection and Image Reshuffling (Barnes et al. 2010). Texture synthesis and Image completion (Xiao et al. 2011)01/08/13 3

Brute force v/s state-of-the-art

• Find the matching patches to the patch in red• Brute-force: 2125 msec

• State-of-the-art approaches: 1.5msec (1400x faster)

4

Image Credit: Kumar et al. (ECCV 2008)

01/08/13

The Patch-Matching Problem

Image Credit: Barnes et al. (SIGGRAPH 2009)

01/08/13 5

Types of Patch-Matching• Sparse

– Finding matches for a few key-points

• Dense– Matches for all patches in the

source Image• Local

– Search for matches in a specific region in the target image

• Global– Search everywhere for matches in

the target image

• Local is computationally less expensive

• But, will it suffice?

Sparse

Local Global

Dense

601/08/13

Where do similar patches lie?

• Our focus dense and global• Since a lot of patches lie at large distances

701/08/13

Next..

• What is a patch?• The formal patch-matching problem• Motivation for making patch-matching

algorithms faster• Types of patch-matching

Till now..

• Parameters of Patch-matching algorithms• Related Work

801/08/13

Parameters of Patch-Matching Algorithms

901/08/13

Previous Efforts

• To use general-purpose Approximate Nearest Neighbor (ANN) algorithms– KD trees– PCA trees– etc.

• They provide good solutions to general ANN problems

• Do not take advantage of the structure and properties specific to images

1001/08/13

Previous Efforts: PatchMatch

• Barnes et al., SIGGRAPH 2009• Randomized cooperative Hill-Climbing Search• Based on the property of coherence in image patches• 3 steps of the algorithm:

Image credit: Barnes et al. 2009

• Up-to 2x faster than all previous approaches

• Enables several interactive applications

• Not as accurate as one would hope

• Coherency Assumption: Not true in highly textured images

1101/08/13

Previous state-of-the-art: Coherency Sensitive Hashing (CSH)

• Korman and Avidan, ICCV 2011• Based on PatchMatch– Replaces the Random step of PatchMatch with a Hash

Table

• Combines cues of location (of coherence) and appearance in a novel manner

Type 1 Type 2 Type 312

Appearance Coherence

01/08/13

Coherency Sensitive Hashing (CSH)• Targeted propagation of information

• Lower RMS error values in early iterations compared to PatchMatch

• Faster and more accurate than Patchmatch

• Less dependent on the coherency assumption

• Lowest reconstruction errors– Current state-of-the-art

1301/08/13

Next..

• Introduction to Patch-Matching

• Previous techniques– General purpose ANN – PatchMatch– Coherency Sensitive Hashing (CSH)

Till now..

• Image Pyramids• Pyramids for Patch-Matching

1401/08/13

Pyramids• Image represented at multiple scales

15

Level 0

Level 1

Level 2

Level 3

Level 4512x512 32x32

01/08/13

Pyramids

• Pyramid coarse-to-fine framework is well established (Burt et al. 1984)

• Help a variety of problems– Image compression– Image alignment– Optical flow– Object tracking–Matching: do away with fine details, look at only

high level information

16Can they also be used for Patch-Matching? Why not!

01/08/13

Pyramid Patch-Matching• Novel patch-matching algorithm• Pyramid representation of original images

– Coarsest level image size determined by a threshold ‘Pthold’• Exhaustive global search at small images of coarsest level• Search localization at finer levels by up-sampling the

matches found at coarser levels

1701/08/13

Pyramid Patch-Matching (PPM)

• Coarsest level

18

• Finer levels

a* is the patch in IA(p+1) corresponding to a01/08/13

PPM: Real Image example

1901/08/13

Pyramid Patch Matching• Increase K– Best match at coarsest resolution may not be the best

match at original resolution– Store more number of matches– Wider search range at finer levels– Up to 2% better accuracy

20

• Reduce Patch-Size– Small size patches matched

at coarser resolutions

• Early Termination– Of distance calculation if it exceeds the distance of the

farthest patch currently in the matched list of patches– Up to 1.7x speed-up

01/08/13

Parallel Implementation• PPM processes each patch independently• Pyramid construction– Standard parallel implementation

• Coarsest level– Each thread assigned a block of source patches– Searches for best matches in the whole target image

• Finer levels– Up-sampling of matches; candidate-list generation– Each thread searches the candidate space of the patches

in its block• Near-linear time speed up w.r.t. number of cores

2101/08/13

GPU Implementation

• Pyramid construction on the GPU– Standard implementation

• Each patch is matched independently

• Distance calculation of each query patch with its candidate matches is also independent

2201/08/13

GPU Implementation• Algorithm– Exhaustive search at the coarsest image resolution– Localized search at finer resolution levels– For each patch

• threads equal to the number of candidate patches are invoked

• Each thread handles one vector-distance calculation– Up-sampling of matches; candidate-list generation

also in parallel• Additional 70x compared to implementation run

on 4-core (~250x compared to single-core)

2301/08/13

GPU Timings

• Values for CSH Dataset Images (shown in next slide)

• Brute force for Lena (256x256 image)• Most gain in speed achieved for brute-force

global exhaustive search

24

Search CPU GPU Speed-up

3 x 3 13.46 sec 278 msec 48x

5 x 5 20.45 sec 684 msec 30x

7 x 7 29.68 sec 1113 msec 26x

Brute-Force(Lena, 512x512 image)

356.63 sec 4.89 sec 72x

01/08/13

CSH Dataset for Experiments• 133 image-pairs from the dataset provided by Korman and Avidan, ICCV 2011 (CSH)• Both images of the same scene of a movie

• between 1 and 30 frames apart in the video

2501/08/13

ParametersVarying KNN and Patch-size

26

Varying Pthold

Varying Search-range

Error: RMS distance between RGB values of patches, averaged over all matches of a patch and over all patches in the image

Platform: Intel i7 920 processor, 2.67 GHz

Values over all 133 pairs of images in the CSH dataset (shown in next slide)

01/08/13

Experiment I• Proximity to ground-truth– Brute-force search for ground-truth matches– PPM captures up to 4% more ground truth

matches– Better performance for more number of matches

required

2701/08/13

Experiment II• Error (RMS error between RGB values of patches and their matches)

– Parameters set to achieve similar run-times– PPM achieves consistently lower error values than

CSH• Better performance than state-of-the-art– Much closer to ground truth (Calculated by Brute-force on GPU)

2801/08/13

Next..

• Patch-Matching and previous techniques• Pyramid Patch-Matching– Improvements: Increase K, Reduce patch-size,

early-termination– Experiments: State-of-the-art performance

Till now..

• Mixed-Resolution Vectors (MR-vectors)– Motivation for mixing resolutions

• Mixed-Resolution Patch-Matching (MRPM) 2901/08/13

Mixed-Resolution: Motivation

• Images at coarser resolutions contain approximate information

• Small objects may get smoothed out due to downsampling

• A match missed at the coarsest resolution will never be recovered later in the algorithm

• Solution: More information already exists at finer resolutions. Use it!

3001/08/13

Mixed-Resolution Vectors• Idea– Concatenate corresponding patch-vectors of finer

resolutions and current resolution• Including Information– Currently we pick a same-sized patch around the

center of the corresponding up-sampled patch– Available prior knowledge about the image or patches

can be used to pick information more wisely!

3101/08/13

MR-Vectors

• Primary contribution• Can include information from multiple levels

3201/08/13

Mixed-Resolution Patch-Matching

• In the framework of PPM, we replace traditional patch-vectors with MR-vectors

• Rest of the algorithm remains same

• We call this the “Mixed-Resolution Patch-Matching” Algorithm (MRPM).

3301/08/13

Comparison with increasing search range

• Accuracy increases with cost of time when using a wider search range around up-sampled matches

• Similar behavior is observed while mixing resolutions

34

Mixed-Resolution achieves lower error values with less cost of time

01/08/13

Parallel Implementation

• Fast and efficient parallel implementations of MRPM follow from parallel implementations of PPM

• Multicore: Near-linear time speed-up• GPU: All levels already on GPU memory

35

Search CPU GPU Speed-up

3 x 3 19 sec 442 msec 43x

5 x 5 30 sec 937 msec 32x

7 x 7 45 sec 1.73 sec 26x

01/08/13

PPM v/s MRPM

• Lower error values with marginal cost of time• Less effect of time cost on GPU performance

• Note: error not averaged over patch-size36

(l = 2)

01/08/13

Experiment I

• Proximity to ground truth

• Up to 8% more of the ground truth matches are captured by MRPM than CSH

• 4% more than PPM

3701/08/13

Best K Matches CSH PPM MRPM

Best Match 92.43% 92.48% 93.57%

Best 5 Matches 88.06% 89.7% 91.51%

Best 10 Matches 81.72% 85.69% 89.87%

Experiment II

• Error– Parameters set to achieve similar run time–MRPM achieves near optimal patch-matching

3801/08/13

Next..

• Patch-Matching and previous techniques• Pyramid Patch-Matching• Mixed-Resolution Vectors• Mixed-Resolution Patch-Matching

Till now..

• Applications• Conclusion

3901/08/13

Applications

• A variety of applications have a patch-matching engine at their core

• MRPM can act as that engine and help improve quality and speed of several of these applications

• We discuss 3 of these– Image de-noising– Image summarization (Retargeting)– Auto-crop

4001/08/13

Image denoising• Performs a weighted averaging

over similar patches to de-noise the image (Non-local means, Buades et al., 2005)

• Currently local due to computational constraints– Also improves accuracy in some

cases

• MRPM can be used to extend it to global

4101/08/13

Experiments

• Dataset of 36 images collected for evaluating denoising performance

42

Dataset of 36 images

Combining matches of local (non-local means) and global (MRPM)

01/08/13

Self-similar Image

• Evaluation: Method-noise• Less structure removed from Image

43

PSNR: 33.35

PSNR: 34.03

01/08/13

Image Retargeting (Summarization)

• Simakov et al. 2008, Bidirectional Similarity

4401/08/13

Results

• MRPM achieves better visual qualityof results

4501/08/13

Auto-crop

• Similar to Bi-directional Similarity– Coherence term muted: since all cropping

windows would anyway be coherent with the image

• Cropping based on completeness term similarity

4601/08/13

More Results

4701/08/13

Limitation: Image Reconstruction• Given an Image B and a dense patch-matching from A

to B. Reconstruct image A• Each pixel is replaces with the average of the

corresponding pixels that it is mapped to by all patches that contain it

4801/08/13

Summary

• Presented a Pyramid based approach to the problem of patch-matching– 3 Improvements to improve accuracy and efficiency

• Achieved state-of-the-art matching accuracy• Introduced the concept of Mixed-Resolution

Vectors• Proposed Mixed-Resolution Patch-Matching

outperforming state-of-the-art by up-to 8% more ground-truth matches captured

4901/08/13

Future work

• Include coherency cues in the framework of the algorithm– Up-sampling maintains some coherency

• Matching across all rotations and scale• Matching in videos

• Explore the scope of MR-vectors– We believe it is a generic concept and the idea can be

used in several other problems that use the multi-resolution technique

5001/08/13

Publication(s) Harshit Sureka, P. J. Narayanan. Mixed-Resolution Patch-

Matching. In the 12th European Conference on Computer Vision, ECCV 2012, 7th to 13th October 2012, Florence, Italy.

5101/08/13

Thank you!!

5201/08/13

Backup Slides

5301/08/13

Downsampling Method

• All, but the nearest-neighbor interpolation perform well

• Best: Gaussian Averaging

5401/08/13

Increase K• Storing more number of matches at coarser

resolutions than originally required• Increase in accuracy up-to 2%• More pronounced effects when less number of

matches are required (i.e. ‘K’ is low)

5501/08/13

Early Termination

• Early Termination– Of distance calculation if it exceeds the distance of

the farthest patch currently in the matched list of patches

– Up to 1.7x speed-up

5601/08/13

GPU Implementation Details

• Each patch represented as a 1-D vector called the Patch-vector

• Bottleneck: Vector-distance calculation

• Each thread performs only one vector-distance calculation of query patch vector with target image candidate patch vector

5701/08/13

GPU Implementation

• Distance stored in a distance array of the query patch

• Distances are stored in independent arrays for each source patch (in parallel, using Thrust library)

• Up-sampling of patches also done in parallel

5801/08/13

MRPM Algorithm

5901/08/13

Bidirectional Similarity

6001/08/13