Filling Algorithms

Pixelwise MRFs Chaos Mosaics

Patch segments are pasted, overlapping, across the image.

Then either:

● Ambiguities are removed by smoothing (Chaos Mosaics-MSR).

● Or a least cost path through the (chosen) overlapping images are found. Efros’01 uses dynamic programming, while Graphcut textures’03 uses… graphcut.

Pixel distributions are determined by comparision

with those with similar neighbourhoods.

These distributions are sampled from or heuristics are performed on them to determine how to fill them.

Synthesizing One Pixel

Infinite sample image

Generated image

Assuming Markov property, what is conditional probability distribution of p, given the neighbourhood window?

Instead of constructing a model, let’s directly search the input image for all such neighbourhoods to produce a histogram for p

To synthesize p, just pick one match at random

SAMPLE

p

Taken from Efros’ original presentation

Really Synthesizing One Pixel

finite sample image

Generated image

p

However, since our sample image is finite, an exact neighbourhood match might not be present

So we find the best match using SSD error (weighted by a Gaussian to emphasize local structure), and take all samples within some distance from that match

SAMPLE

Taken from Efros’ original presentation

The pixel metric doesn’t matter

One of the surprising things about this is that the choice pixel metric doesn’t seem to matter.

Efros uses || . ||2 2 on RGB space, I’ve been using || . ||1, while Criminisi uses a metric based on the CIELab colour space.

It doesn’t seem to matter which you use, presumably because we are only concerned with nearest neighbours, and they are all topologically equivalent.

Dynamic Programming solution

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

Chaotic Mosaics

Graphcuts solution

Chaotic Mosaics

Takes full advantage of the power of graphcuts method, it treats the whole image as one patch and finds optimal

joins along it.

Pros: Finds optimal (and often seamless) matches

Cons: Doesn’t find anything else, the recycling the optimal matches still

leaves you with tiling artefacts.

Graphcuts solution

Chaotic Mosaics

Image Quilting Graphcuts

● Onion skin or Outside in● The first.● The simplest?● Works with single textures or

simple convex filling regions● Just picks away at the image one

layer at a time

Pixel choice and Filling Algorithms

● Linear structure propagation● Onion skin +pushing in on linear textures● Better than Onion skin for multi textural

environments● When all you have is hammer, everything

starts to look like a nail. ~ Artefacts from trying too hard.

Pixel choice and Filling Algorithms

Missing Data Correction in Still Images and Image Sequences, Bornard et al. 2002

● Linear structure propagation● Onion skin +pushing in on linear textures● Better than Onion skin for multi textural

environments● When all you have is hammer, everything

starts to look like a nail. ~ Artefacts from trying too hard.

Pixel choice and Filling Algorithms

Missing Data Correction in Still Images and Image Sequences, Bornard et al. 2002

Filling AlgorithmsOnion Peel Vs. Linear propagation

Push InNow

Onion Peel

● Max. entropy fill● Consistent with MRF

assumptions.● Locally convex with a

minimum of occlusions at point of fill.

● Spirals in on simple shapes.

Pixel choice and Filling Algorithms

Why Coarse to Fine?

Standard Efros 15 pixelsStandard efros 21 pixels

New metric Efros 15 pixels

Efros uniform pixel weighting 15

C2f uniform pixel weightingCourse to fine 15 pixel nhood

The same but quicker...

Structures...Why Coarse to Fine?

StructuresAs TexturesWhy Coarse to Fine?

StructuresAs TexturesWhy Coarse to Fine?

Texture as structure?

?

Strong linear propagation comes from efros stylefills naturally.

Why Coarse to Fine?

Not readily apparent due to the onion skin fill

Efros

Linear propagation

The Efros Algorithm

with my pixel choice

No guarantee it’s any better than linear propagation

The algorithm often spots at the coarser levels that it has insufficient data to complete

No guarantee it’s any better than linear propagation

Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the higher levels is still needed.

No guarantee it’s any better than linear propagation

Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the higher levels is still needed.

No guarantee it’s any better than linear propagation

Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the lowest resolution is still needed.

No guarantee it’s any better than linear propagation

Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the lowest resolution is still needed.

No guarantee it’s any better than linear propagation

Annoyingly this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the lowest resolution is still needed.

No guarantee it’s any better than linear propagation

Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the lowest resolution is still needed.

No guarantee it’s any better than linear propagation

Compare it with a smaller fill

Is manifold learning possible?

?

Up to a point, we don’t care what colour the books are when propagating the shelf.

Similar structural edge patterns are apparent everywhere.

Why Coarse to Fine?

Why Coarse to Fine?

Problems

● Speed Massively slower(hours rather than seconds)

than patch based synthesis even with coarse to fine reducing neighbourhood size.

● Can we reduce the search space via image segmentation?

● Alternatively turn our soft coarse to fine constraints into something harder, by only testing pixels from the neighbourhoods of the k-closest fits at a coarser level.

Problems

Surprisingly, this could even increase robustness by preventing the growth of miss-fittings