edge-preserving multiscale image decomposition based...
Post on 29-Jun-2018
Embed Size (px)
Edge-preserving Multiscale Image Decomposition based on Local Extrema
INRIA / Grenoble UniversityCyril Soler
INRIA / Grenoble UniversityFredo Durand
(a) Input (b) Fine features boosted (c) Coarse features boosted (d) Scanline plots
Figure 1: Our multiscale decomposition of image (a) allows detail to be extracted based on spatial scale rather than contrast and preservesedges. (b) Boosting fine scale features increases the contrast of the pattern on the vase. (c) Boosting coarse scale contrast and suppressingfine features reduces the contrast of the pattern, while increasing the contrast of the vase with its background. (d) Scanline plots (rowsindicated using arrows in (a), (b) and (c)), illustrating the effect of the two equalizations (b) and (c). The dashed lines in the plots show twoexamples of edges that have been preserved.
We propose a new model for detail that inherently captures oscil-lations, a key property that distinguishes textures from individualedges. Inspired by techniques in empirical data analysis and mor-phological image analysis, we use the local extrema of the inputimage to extract information about oscillations: We define detail asoscillations between local minima and maxima. Building on the keyobservation that the spatial scale of oscillations are characterized bythe density of local extrema, we develop an algorithm for decom-posing images into multiple scales of superposed oscillations.
Current edge-preserving image decompositions assume image de-tail to be low contrast variation. Consequently they apply filtersthat extract features with increasing contrast as successive layersof detail. As a result, they are unable to distinguish between high-contrast, fine-scale features and edges of similar contrast that are tobe preserved.We compare our results with existing edge-preservingimage decomposition algorithms and demonstrate exciting applica-tions that are made possible by our new notion of detail.
Keywords: image decomposition, computational photography
A variety of applications in computational photography require adecomposition of an image into different scales. Traditional ap-proaches that use linear bases have evolved to accommodate theneed for respecting strong edges. Recent definitions of scales areusually based on spatial scale definitions combined with a notionon the range to differentiate strong edges [Tomasi and Manduchi1998; Durand and Dorsey 2002; Farbman et al. 2008; Lischinskiet al. 2006; Choudhury and Tumblin 2005]. Current approaches
e-mail: Kartic.Subr@inrialpes.fre-mail: Cyril.Soler@inrialpes.fre-mail: email@example.com
share a common notion of an edge large gradients, or large valuedifferences, where the definition of large might depend on the ap-plication. However, this notion of an edge makes it challenging tocapture fine details or textures that have fine spatial scale but highcontrast. For example, in Figure 1(d), some edges to be preservedare lower contrast than oscillations to be smoothed. Extracting thewhite dots on the vase as detail requires aggressive smoothing ofgradients, which would also blur single edges that are to be pre-served (see Fig. 2). This distinction between edges and oscillationsraises challenges in defining fully multiscale decompositions be-cause the interplay between spatial and edge consideration leads tounexpected results, as shown by Farbman et al. 
We propose a novel non-linear image decomposition that effec-tively extracts fine-scale features, regardless of their contrast, asdetail and yet preserves softer salient edges in the base layer. Incontrast to previous approaches that rely on magnitudes of pixeldifferences at their heart, our approach captures local image oscil-lations by considering local image extrema. A fine-scale textureis characterized by rapid oscillations (see Fig. 1) between minimaand maxima. Furthermore, the oscillation between extrema providecritical information that permit the distinction of individual edgesfrom oscillations. We obtain a multiscale decomposition by recur-sively smoothing the image while also progressively coarsening thescale at which extrema are detected.
1.1 Related work
Several image decomposition techniques have been proposed.Strategies that use linear filters [Burt and Adelson 1983; Rahmanand Woodell 1997; Pattanaik et al. 1998] produce halo artifacts atedges and have been succeeded by non-linear filters that preservestrong edges a popular choice being the bilateral filter [Tomasiand Manduchi 1998; Durand and Dorsey 2002; Choudhury andTumblin 2005]. Bae et al.  used the bilateral filter to sep-arate images into low- and high-contrast features and manipulatedthe layers independently to enhance photographic look. Fattal et
al.  presented a technique to enhance shape and surface de-tails of objects using bilaterally filtered representations of a set ofdifferently lit images. Our goal is to extract from a single image, ateach scale, the finest spatial oscillations as detail without assumingthem to be low-contrast oscillations.
Two approaches have been proposed for multiscale decompositionsusing the bilateral filter. One strategy is to progressively increasethe width of the range and spatial Gaussian through the coarsen-ing process. Chen et al.  used this technique to construct abilateral pyramid for progressive video abstraction. Another strat-egy [Fattal et al. 2007] recursively applies the bilateral filter tothe smoothed versions of the input image. This strategy decreasesthe width of the range-Gaussian during successive iterations so thatedges from preceding smoothing operations are not blurred duringthe coarsening.
In recent work, Farbman et al.  pointed out that, while thebilateral filter is effective at smoothing out low amplitude noise at afine scale, multiscale decompositions using the bilateral filter sufferfrom a variety of problems. Progressive widening of the range andspatial Gaussians through the coarsening process was shown to pro-duce halo artifacts at strong edges. To overcome some problems ofusing the bilateral filter in a multiscale decomposition, Farbman etal.  proposed a filter that smoothes an input image I by com-puting an image that is as close to I as possible while being smootheverywhere except at regions where the gradient of I is large. Theyused a weighted least squares filter, originally used to control ring-ing during deblurring of noisy images [Lagendijk et al. 1988]. Thenature of this optimization makes it impossible to preserve salientedges with lower contrast than the texture that is to be smoothed.
In summary, smoothing filters currently used in image decompo-sition algorithms assume detail is low-contrast. As a result, localvariation at different contrast levels are extracted as successive lay-ers of detail. Such layers of detail do not necessarily represent fine-scale spatial variation.
A notable exception, for 1D data, is empirical mode decomposi-tion [Huang 1998] a powerful data analysis tool originally pro-posed to decompose nonlinear, nonstationary signals into their in-trinsic modes of oscillations. The decomposition is achieved byiterative removal of the finest intrinsic oscillations as indicated bylocal extrema. This technique is popularly used on 1D data that donot contain sharp discontinuities. A few attempts at extending thetechnique to image decomposition [Nunes et al. 2003; Liu and Peng2005; Damerval et al. 2005] have uncovered a number of difficul-ties. One formidable challenge that has not been addressed is theneed to respect sharp edges. Another drawback of empirical modedecomposition is its poor handling of signals where oscillations atdifferent scales occur as bursts, in parts of the domain (the problemof intermittency [Li et al. 2005]).
We introduce novel definitions, based on local extrema, for edgesand detail that permit the distinction between highly contrastedtexture and single edges. Using these definitions we develop anedge-preserving smoothing algorithm that allows fine scale de-tail to be extracted regardless of contrast. We perform an edge-preserving multiscale decomposition by recursively applying thesmoothing algorithm on the base layer. The decomposition corre-sponds to features at different spatial scales with salient edges be-ing preserved. We compare our approach with existing decomposi-tions and demonstrate its effectiveness using applications. Figure 4places our novel algorithm in the context of existing approaches.
(a) Input (b) Our smoothing
(c) WLS Filter( = 13, = 0.2)
(d) WLS Filter( = 13, = 1.2)
Figure 3: The ubiquitous notion of edges as pixels with large gradi-ents does not allow disambiguation between fine-scale features andedges that are to be preserved, as shown by this example. (a) Thecontrast of the pattern on the flower vase is greater than acrossthe edges of the soft shadows and petal boundaries. (b) Usingour smoothing algorithm, the pattern is extracted as detail becauseof its fine scale, while coarser soft shadow- and petal-boundariesare preserved. (c) The weighted least square (WLS) filter does notsmooth the pattern if fidelity to strong gradients is retained. (d) Onthe other hand, the WLS filter necessarily blurs softer edges eventhough they are coarse-scale features while smoothing the patternon the vase.
2 Extrema-based multiscale decomposition
We present a novel smoothing algorithm which effectivelysmoothes highly contrasted oscillations while preserving salientedges. By applying this algorithm recursively on the smoothed im-age, we compute a multiscale decomposition of an input image intolayers at different scales of coarseness. In comparison with existingedge-preserving multiscale decompositions, our algorithm signif