Histogram Painting for Better Photomosaics

Brandon Lloyd, Parris EgbertComputer Science Department

Brigham Young University{blloyd | egbert}@cs.byu.edu

Abstract

Histogram painting is a method for applying local histo-gram transformations to an image without introducing dis-continuities at the boundary of the transformed region.Using interactive tools similar to those found in standardpaint programs, the histogram from one part of an imagecan be “painted” onto another. This is useful for improvingthe results of seam elimination algorithms in photomosaic-ing applications. When local intensity distributions differsignificantly at tile edges, a noticeable shift in appearancemay remain even after the seam is removed. Histogrampainting can be used to reduce these differences beforeremoving the seam. The technique can also be used toremove shadows from an image by “painting” the histo-gram from lit regions onto the shadowed portions.

Keywords: seam removal, shadow removal, histogrammatching.

1. Introduction

Images acquired from satellite or aerial photography havea limited field of view. Many overlapping image tiles areoften placed together to form a larger photomosaic. Visibleseams between individual tiles destroy the illusion that thephotomosaic is one continuous image. Seams can arisefrom intensity differences along the edges of the tilescaused by variations in atmospheric conditions, season,time of day, or changing surface features. Even when thetiles are photographed at nearly the same time, differencesin film, scanning, or post-processing can cause subtleintensity variations that create visible seams. A simplesolution to the problem is to perform a graduated cross-fade between adjacent tiles within the region of overlap.For this to produce satisfactory results, the features withinthe overlap region must line up perfectly. If they are mis-registered at all, the cross-fade will create a distractingghosting effect. Unfortunately, a cross-fade with imagesacquired by traditional techniques will almost alwayscause ghosting because of the perspective distortion intro-duced by the camera.

More sophisticated algorithms can perform seamlessblends between two image tiles without creating ghostingartifacts. However, if there are differences in the contrastor the intensity distributions of the tiles, one will still beable to distinguish the individual tiles in the photomosaic.Though the seam in Figure 1c is blended well, even thesubtle difference in contrast between the two halves of thesingle image betrays the fact that the image is actuallycomposed of two images blended together. Milgram [5]suggested performing a histogram equalization on bothimages before removing the seam. Histogram matching[4], which is a generalization of histogram equalization,could also be used to coerce the histogram of one image to

Figure 1: a) Two overlapping tiles. b) Mosaic withoutseam elimination. c) Though seam elimination improvesthe mosaic significantly, the contrast differences in theareas indicated by the arrows reveal the border betweenthe tiles. d) A uniform mosaic created by using histogrampainting.

a) b)

c) d)

match the other, or to match both to a predetermined histo-gram. The drawback of such an approach is that thesetransformations are applied globally to the whole tile andthis may not always be desirable. The two tiles in Figure 1,for example, come from a larger context wherein the cityregions have the appearance of the tile on the left and themountain regions have the appearance of the tile on theright. We would like to leave those regions as they are inorder to keep them consistent with surrounding tiles. Theinconsistent regions are the small city area in the upper leftcorner of the right tile which has too little contrast, and themountain area in the lower right of the left tile which hastoo much. Global transformations like histogram equaliza-tion and histogram matching cannot deal with this situa-tion. Furthermore, even if a global transformation could beperformed, local intensity distributions may still differafter the transformation.

We present a method called histogram painting whichis used to apply local histogram transformations. The mostimportant contribution of this paper is a method for apply-ing the local transformation without introducing sharp dis-continuities at the boundary of the transformed region. Forphotomosaicing applications, histogram painting allowsthe user the local control necessary to match the intensitydistributions more closely near the tile edges. Thisimproves the results of seam elimination algorithms whichare subsequently used to remove the intensity discontinui-ties. Since hand adjustments to a large number of tiles canbe tedious, we also present an automated algorithm thatcan be applied when the tiles have a significant amount ofoverlap.

2. Previous Work

The construction of a photomosaic begins with the regis-tration of image tiles. Brown [2] has written an excellentsurvey of registration techniques. Once the images arelined up correctly, additional processing is often necessaryto create a seamless photomosaic. Several researchershave explored methods for eliminating seams. Milgram[5] suggested an algorithm for selecting a seam within theoverlapping region between two tiles such that the intensi-ties match well at the seam. The seam path is constructedone row of pixels at time by assigning a cost to each posi-tion on the row based on the intensity differences betweenthe two tiles on either side of the position and the devia-tion from the already constructed path. Dynamic program-ming is used to determine the path with the lowest cost.Discontinuities may still occur at the selected seam points.These are eliminated by smoothly ramping intensities overa small range across the seam. Shiren et. al [7] laterimproved the technique by extending the seam-pointsearching algorithm to two dimensions within the over-lapped region. Araújo and Leite [1] use watersheds of thedifference image in the overlapping region to efficientlycalculate a seam along lines of maximum correlation.Peleg [6] used iterative relaxation to compute a seam elim-

ination function over each tile, which when added to theoriginal tiles, adjusts the intensities of each pixel to ensurethat they match at the edges. The function is constrained tobe as smooth as possible in the interior of the tile to mini-mize visible artifacts.

Burt et. al [3] used a multi-resolution algorithm to cre-ate seamless transitions between images that may be com-pletely unrelated. They observed that in a graduatedcrossfade between two images, if the width of the transi-tion zone T is large compared to the wavelength of thehighest spatial frequencies in the image, the ghostingeffect will be introduced. On the other hand, if T is smallcompared to the wave length of the lowest prominent fre-quency, a visible, albeit somewhat blurred step in intensitymay result. To blend two images together with neithersteps nor ghosting, the cross-fade must be performed sepa-rately on different spatial frequency bands with a transi-tion width appropriate for the frequencies contained ineach sub-band. The images to be blended are first decom-posed into two image pyramids. The highest frequencysub-band is at the base of the pyramid. Each level abovethe base contains the sub-band with half the frequency ofthe level below it. A new image pyramid is created byblending each level of the two source image pyramids witha transition width appropriate for the frequencies in thatlevel. Adding together each level of this new image pyra-mid yields the final, seamless image.

While all of these algorithms may be used to removetile seams, none of them alone is sufficient to achieve con-sistent intensity distributions across tiles, especially heter-ogenous tiles that consist of significantly different featuressuch as the mountains and city regions of Figure 1. We usehistogram painting to first achieve a smooth histogramtransition in intensity distribution between image tiles andthen apply any of the aforementioned seam eliminationtechniques to remove intensity discontinuities at the edges.

3. Histogram Painting

Histogram painting refers to the application of a local his-togram transformation interactively with the mouse or sty-lus. The user begins by selecting a source region withinthe area that will be modified and a target region whichhas the desired intensity distribution. A mapping is createdthat will transform the histogram of the source area to thehistogram of the target region. Once the mapping is estab-lished there are two techniques to apply the transforma-tion. For simple regions, the user outlines the area tomodify and specifies a transition width. The pixels in thecenter of the region are fully transformed with a smooth,uniform-width transition at the region boundary. The otherapplication technique affords the user more precise con-trol. Using brushes of varying sizes and opacities the usercan control the amount of tranformation that is applied.Figure 2 shows an image modified with the histogrampainting tool. Both application methods use an off-screen

image called the transition map to control the degree towhich each pixel is transformed.

3.1 The Transition Map.The transition map is used in conjunction with histogrammatching to produce the intensity transformations. Theintensity in the transition map is used to determine howmuch to transform the corresponding pixels in the image.Pixels in white areas undergo the full histogram transfor-mation while black areas remain untouched. Gray areasare only partially transformated. Figure 2b shows the tran-sition map used for the image in Figure 2a. Both applica-tion techniques modify the values in the transition mapand update the pixels in the image accordingly. For theregion with a fixed transition width, the transition map iscreated by first drawing the region filled with white on ablack transition map. The map is then filtered with a gaus-sian filter with a radius equal to the specified transitionwidth. For brushes, gaussian kernels are splatted on thetransition map as the brush is applied. The brushes canalso be used to “erase” the transformation by subtractingthe kernel. Painting operations continue to manipulate thesame transition map until the painting operations are “fro-zen.” At this point the transition map is cleared, the trans-formed pixel values are written to the image, and a newhistogram transformation must be selected. We interac-tively update the display in response to user brush strokes,using a two-dimensional look-up table (LUT) with whichto calculate transformed pixel values. The LUT is indexedby the intensity of the pixel in the original image and thecorresponding pixel in the transition map. This table isgenerated with histogram matching.

3.2 Histogram MatchingThe goal of histogram matching is to transform the inten-sity values of one image in such a way that its histogrammatches that of another image. A basic result from statis-

tics shows how this can be done. A random variable Xwith probability density function (pdf) can be trans-formed into a random variable with uniform distributionby using its cumulative density function (CDF) as a trans-formation function. The CDF for X is:

The inverse of the CDF of X will transform a uniform ran-dom variable U on the interval [0,1] into a random vari-able with the same distribution as X.

may not exist if has regions where the func-tion is flat. To ensure a unique inverse for each x we usethe function:

To transform X into a random variable Y with distribution we first use to transform X into a uniform ran-

dom variable. Then we use to transform from the

uniform random variable to . The complete transfor-mation is given by:

The process is represented graphically in Figure 3. By rescaling pixel intensities to the range [0,1] and

normalizing the area under the histogram curve to 1.0 wecan transform the histograms of the source image and thetarget image into pdfs and respectively. UsingEquation 1 we can create a LUT that will transform thedistribution of intensities of the source image to that of thetarget image.

We tried two approaches for generating intermediatehistogram tranformations. First, we tried using an interme-

Figure 2: a) A view of a typical histogram painting operation.The regions in the rectangle and oblong outline are used forthe target and source histograms respectively. b) The transi-tion map created by the user’s brush strokes.

f x( )

F x( ) f t( ) td

0

x

∫=

F 1– U( ) X=

F 1– u( ) F x( )

F* u( ) max x( ) f x( ) u={ }=

g y( ) F x( )

G* y( )g y( )

h X( ) G* F X( )[ ]= (1)

f x( ) g y( )

F x( ) G x( )

i0 i1 i0iα

Figure 3: Matching histogram CDF F(x) to G(x). The inten-sity i0 is mapped to i1 by the transformation. The dashed line

is the linearly interpolated CDF with parameter α = 0.5.With simple linear interpolation, iα would lie right in the

middle of i0 and i1

diate target CDF by linearly interpolating between thesource CDF and the target CDF, with the value in the tran-sition map as the interpolation parameter. We also triedlinearly interpolating between the identity LUT and theLUT generated by the histogram match to create an inter-mediate LUT. We prefer the second approach, not onlybecause it is simpler and easier to calculate, but moreimportantly, the intensities change at a constant rate withrespect to the interpolation parameter. This makes the his-togram painting tools easier to use because their behavioris more predictable.

For color images we perform histogram matchingseparately on the red, green, and blue channels. This doesnot always yield expected results. The user’s intuition ofhistogram painting is that it makes the intensities of onearea look like those of another. The user expects to see thedominant color of the source region be replaced by thedominant color of the target region. However, when thechannels are histogram matched separately, colors presentin neither of the regions can emerge. There is also auniqueness problem. Consider two identical images, onein color and the other in grayscale. If we attempt to recol-orize the grayscale with histogram painting by using thecolor image as the target the operation will fail becausedifferent colors may map to the same grayscale intensity.Despite its limitations, this simplistic approach to handlingcolor works very well for our application of cleaning upaerial photography in photomosaics.

4. Automatic Edge Histogram Blending

The seam elimination algorithms work best when the adja-cent images have similar intensity distributions in the areaof overlap. If the tiles have a significant amount of over-lap, we can use an automated algorithm to make a smoothtransition from the histogram of one image to the otherwithin the overlap region. The transition map is formed bycreating a smooth linear ramp across the region. The tran-sition map need not be created explicitly as its only pur-pose is to provide the interpolation parameter for thehistogram blend, and this can be be computed on-the-fly.For images with homogeneous features, it is sufficient touse the entire overlapping region in each image for thesource and target histograms. If there are heterogenousfeatures in the overlapping region, however, it is better touse an algorithm that adapts to changes in the local histo-gram. We obtained good results by using a sliding windowfor the histogram computation as shown in Figure 4. Ateach step the source and target histograms are taken from asmall window in the overlapping region. The histogramtransformation LUTs are calculated as usual. The mappingis used to transform n columns of pixels in the center ofthe window. The window is then shifted down n columns.The process is repeated along the length of the overlappingregion. The histograms can be efficiently updated by add-ing the pixels on the leading edge of the sliding window

and subtracting those on the trailing edge. The choice for ndepends on the images. If the intensity distributionschange rapidly along the length of the overlapping regionand n is too large, the algorithm can create noticeablesteps. If n is small then the histogram tranformation LUTsmust be calculated more often and the process takeslonger. We achieved satisfactory results using a value of nthat is approximately one-tenth of the width of the slidingwindow.

5. Shadow Removal

One of most serious intensity distribution disparitiesbetween adjacent image tiles occurs when one tile is shad-owed and the other is lit. Histogram painting can be usedto remove shadows under certain conditions. First, theremust be enough detail in the shadowed region. Applyinghistogram painting to a completely black shadow will onlycreate a lighter blob, because it cannot restore detail thatwas not present in the original image. Second, shadowremoval works best for large shadows like those createdby a mountain slope or cloud cover. It is far too tedious topaint out many small shadows such as those cast on a flatrocky terrain with the sun low in the sky. Third, we foundthat it was easier to handle soft rather than hard shadowedges. When the histogram from a lit region is paintedonto another lit region, the pixels are brightened exces-sively. It is extremely difficult to prevent this from hap-pening with a hard edge because if the brush crosses theedge by even one pixel there will be a bright halo sur-rounding the region that was shadowed. Soft edges areeasier to handle because they do not require as much preci-

Figure 4: Sliding window used in automatic edge histogramblending. Histograms are taken from the shaded region. Onlythe narrow band of pixels in the center of the region is trans-formed.

sion. One last consideration is that the objects in “relit”portions of the image will not cast shadows if these werenot present in the original image. This may actually bedesirable if the image is going to be used as input into aclassification algorithm or as a texture map on a modelwith novel lighting.

Shadow removal begins by picking regions in lit andshadowed areas of the image to use for the target andsource histograms, respectively. It is important that theseregions be as similar as possible and include all the fea-tures in the shadowed region. For example, in Figure 6awe picked regions that contained both trees and road. Theroads do not look right if we paint them with a histogramtransformation obtained from regions consisting only oftrees because the road-colored pixels are not contained inthe target and source histograms. Once we have estab-lished histogram transformation we proceed with thepainting operations. Large interior regions can be paintedquickly using a large brush at full opacity or the simpleregion fill tool. The edges require more precision. Softshadow edges are not as dark as the interior and requireless transformation. A small brush with low opacity pro-vides the necessary control. The brushes are additive sothat running the brush back and forth over a region slowlybrings the transformation to just the right level as theintensity builds up in the transition map. If we overshootand a region becomes too bright we slowly back off thetransformation by a similar operation with a low opacity

erase tool. Using this method it is possible to create aseamless transition between shadowed and lit areas.

6. Results

Figure 5 shows a seamless 8GB, meter-per-pixel resolu-tion photomosaic of a stretch of land nearly 90 miles long.The data is used for an interactive terrain visualizationprogram. Each tile was first histogram matched to a repre-sentative tile with similar features in order to equalize thetile brightness and contrast levels globally. We then usedhistogram painting to make the necessary large-scaleadjustments at a tile’s edges to match it with its neighbors.Automatic edge matching was then applied to all edgesbefore using a seam removal algorithm.

Figure 6 shows two images partially shadowed by amountain and by clouds. Using histogram painting theseshadows have been completely removed in just a few min-utes.

7. Conclusions and Future Work

Existing seam elimination algorithms successfully removeintensity discontinuities, but this is insufficient to createphotomosaics in which the constituent tiles are indistin-guisable. The intensity distributions must match as well.Histogram painting allows the histograms to be changedlocally with smooth transitions into untouched areas of the

Figure 5: A comparison of a distant and closeup view of a large data set before and afterhistogram transformations. Automatic edge histogram blending was applied to all of thetiles.

image. Histogram painting can remove shadows, contrastdifferences, and other inconsistencies that reveal tile loca-tions in a photomosaic. We present a simple method utiliz-ing a transition map to perform local pixel transformationswithout introducing discontinuities at the boundaries ofthe transformed regions.

There are several ways to make histogram paintingeasier to use. It can be quite tedious to get shadow edgesjust right. It would be useful to detect shadow edges andautomatically generate the transition map. When modify-ing large areas with subtle histogram shifts it would alsobe useful to use the local histogram around the currentposition of the brush in order to better adapt to local histo-gram changes. Finally, we would like to investigate betterways of performing histogram transformations on colorimages.

References

[1] F. Araújo Jr. and N. J. Leite, A Morphological Algorithm forPhotomosaicking, Proceedings VIII European Signal Pro-cessing Conference, Trieste, Italy, 1996, pp. 181-186.

[2] L. Brown, A survey of image registration techniques, ACMComputing Surveys, vol. 24, no. 4, 1992, pp. 326-376.

[3] P. Burt and E. Adelson, A Multiresolution Spline WithApplication to Image Mosaics, ACM Transactions onGraphics vol. 2, no. 4, 1983, pp. 217-236.

[4] K. R. Castleman, Digital Image Processing, (Upper SaddleRiver, New Jersey: Prentice Hall, 1996), pp. 91-93.

[5] D. Milgram, Adaptive Techniques for Photomosaicking,IEEE Transactions on Computers, vol. 26, 1975, pp. 1175-1180.

[6] S. Peleg, Elimination of Seams for Photomosaics, IEEEConference on Pattern Recognition and Image Processing,1981, pp. 426-429.

[7] S. Yang, L. Li, and G. Peng, Two Dimensional Seam-PointSearching in Digital Image Mosaicking, PhotogrammetricEngineering and Remote Sensing, vol. 55, no. 1, 1989, pp.49-53.

a) b)

c) d)

Figure 6: a) and c) Regions of images shadowed by mountains and clouds respec-tively. b) and d) The same regions with shadows removed.