Image processing applications through a variationalperceptually-based color correction related to RetinexJavier Vazquez-Corral1, Syed Waqas Zamir1, Adrian Galdran2, David Pardo3,4,5, Marcelo Bertalmıo1 ;1 Information and Communication Technologies Department, Universitat Pompeu Fabra, Barcelona, Spain2 Tecnalia, Computer Vision Group, Derio, Spain3 Department of Applied Mathematics, Statistics, and Operational Research, University of the Basque Country, Leioa, Spain.4 BCAM, Basque Center for Applied Mathematics, Bilbao, Spain5 IKERBASQUE, Basque Foundation for Science, Bilbao, Spain

AbstractThe retinex theory of color vision has been a light of inspira-

tion for color researchers since its inception 50 years ago. It hasbeen adapted to work for different goals such as shadow removal,high dynamic range imaging, or computational color constancy.Back in 2007 a variational perceptually-based color correctionmodel related to Retinex was presented by Bertalmıo and col-leagues. In this paper we first comment on this model and laterwe review different image processing applications that have beenobtained by performing small modifications to it, namely colorgamut mapping and image dehazing.

IntroductionThe Retinex theory of color vision [9] has been a very pow-

erful influence for color researchers for almost 50 years. Retinexpostulate can be over-summarised to the following idea: the colourof an object is not determined by the spectral composition of thelight reaching to the eye, but is determined by the comparisonbetween the three lightness values generated by this spectra at eachcone photoreceptor and the values generated by a white stimulusspectra (considered to be the spectra of maximum radiance) inall three bands. Edwin Land hypothesized that this effect washappening either at the retina, the cortex, or at both levels, andtherefore he created the portmanteau word ’Retinex’ to express theeffect.

Therefore, the main question that arises is: how do humanscompare a particular color to a white reference that might be faraway in the scene? Land and McCann [10] solved this issue bycomparing points of the scene that are far apart through paths. Thispath computation allowed the authors to define three main mecha-nisms that should be accomplished: ratio, product, and reset. Theproduct and ratio mechanisms are based on their proposal for thecomputation of the aforementioned comparison. First, this compar-ison is considered to be the ratio between the points, and second,this ratio is approximated through local comparisons among thepath, i.e. the ratio of the two end points can be decomposed asthe sequential product of ratios of each point of the path to thefollowing one. This leads to the reset postulate that claims that theratio between any point and its sequential one should be smallerthan a unity. If this does not happen, it means that the radianceof the latter point is larger than the radiance of the previous point,and we should reset the sequential product of ratios to one. In thisway, for any path, we will be always computing the ratio betweenthe end point and the point with maximum value along the path,

which is the approximation to the maximum radiance point [12].Retinex has been characterized using diverse mathematical

formulations during all these years. The straightforward randompath-based computation presented by Land and McCann [10] hasbeen modified to consider various models such as random sprays[13], Brownian motion [11, 14], or swarn termites movement [15].Retinex has been also mathematically defined using minimizationsbased on the Laplacian of the image [6] or variational-based energyminimizations [7].

Bertalmıo, Caselles, Provenzi and Rizzi presented a varia-tional perceptually-based color correction model [2] that was laterproved to exhibit the main mechanisms of the original Retinexexplained above: ratio, product, and reset [1]. In this paper, wewill review on this particular method and its modifications to dealwith different image processing applications. In particular, thismodel have been adapted to deal with color gamut mapping andimage dehazing.

This paper is organized as follows. In the next section we willreview the main characteristics of Bertalmıo -et al.- model. Laterin sections 3 and 4, we will explain the modifications performedto the model to deal with color gamut mapping and image dehaz-ing, respectively. Finally, conclusions are summed up in the lastsection.

Variational perceptually-based color correc-tion

Bertalmıo and colleagues [2] proposed an image energy func-tional whose minimization complies with some very basic visualperception principles, namely those of grey-world, locality and notexcessive departure from the original data:

E(I) =α

2 ∑x

(I(x)− 1

2

)2− γ

2 ∑x

∑y

w(x,y)|I(x)− I(y)|

+β

2 ∑x(I(x)− I0(x))

2 , (1)

where α,β and γ are constant and positive weights, I is a colorchannel (R,G or B), w(x,y) is a normalized Gaussian kernel ofstandard deviation σ , and I(x) and I(y) are two intensity levels atpixel locations x and y respectively.

This model can be understood as a competition between threedifferent terms. With a positive sign, there are two attachments:one to the original image and another to 1

2 , which is the expected

mean of the image after white patch. With a negative sign, there is acontrast maximization term. In this way, the images resulting fromthe model represent a contrast-enhanced version of the originalimage without deviating too far it and staying closer to 1

2 .The authors proved that the solution to the minimization of

Eq. (1) can be found as the steady state of the evolution equation

Ik+1(x) =Ik(x)+∆t

(α

2 +β I0(x)+γ

2 RIk (x))

1+∆t(α +β ), (2)

where the initial condition is Ik=0(x) = I0(x). The function RIk (x)indicates the contrast function:

RIk (x) =∑y∈I w(x,y)s

(Ik(x)− Ik(y)

)∑y∈I w(x,y)

(3)

where x is a fixed image pixel and y varies across the image. Theslope function s() is a regularized approximation to the sign func-tion, which appears as it is the derivative of the absolute valuefunction in the second term of the functional; in the original paperthe authors choose for s() a polynomial of degree 7. In this workwe review two image processing applications where the model ofBertalmıo -et al.- has been adapted to work, namely color gamutmapping and image dehazing.

Color gamut mappingColor gamut mapping deals with the problem of modifying

the gamut of an input image to make it consistent with the gamutof a particular device (printer, screen, projector, etc...). Generally,two different possibilities are considered: i) the gamut of theimage should be reduced to fit into a smaller destination gamut(called gamut reduction), or ii) the gamut of the image should beincreased to better cover a bigger destination gamut (referred asgamut extension). In a recent work, Zamir -et al.- [18] modifiedthe model of [2] to perform gamut reduction and extension; thisinvolved two significant modifications. First, they considered themean of the original image as the expected mean for the finalimage (this modification was already applied in [3]). In this way,the new functional was defined as

E(I) =α

2 ∑x(I(x)−µ)2− γ

2 ∑x

∑y

w(x,y)|I(x)− I(y)|

+β

2 ∑x(I(x)− I0(x))

2 , (4)

where µ represents the mean of the original image. This implies,that the evolution equation for Eq. 4 can be written as

Ik+1(x) =Ik(x)+∆t

(µα +β I0(x)+

γ

2 RIk (x))

1+∆t(α +β )(5)

Second, the authors showed that the sign of the contrast termmodifies the behavior of the model, so that it performs gamutreduction when the sign is positive and gamut extension when it isnegative. Let us explain in detail how they used this observation inboth cases.

Gamut reductionAuthors devised a double-iterative process for the gamut re-

duction case in order to select the best weight to be applied to

the contrast term. The double iterative process works as follows.At each iteration, Eq. (5) is run for some particular α , β , and γ

until the steady state is reached. The steady state of each iterationprovides some pixels of the final result. At iteration 1, values areset to β = 1, α = 0, and γ = 0, and therefore the original imageis obtained as the steady state. At this point, those pixels alreadybelonging to the destination gamut are selected for the final imageand leaved untouched for the following iterations. At iteration2, γ is decreased (for example, γ =−0.05) and α is increased inrelation to γ by |γ|20 . Eq. (5) is run again until steady state, and itis checked whether any of the colors that were outside the gamutat the previous iteration have been moved inside the destinationgamut. If this is the case, those pixels are selected for the finalimage and leaved unmodified for the following iterations. Thisprocedure continues by decreasing γ (and increasing α accord-ingly) until all the out-of-gamut colors come inside the destinationgamut.

Gamut extensionIn this case, the authors proposed to extend the gamut of the

original image by using a unique gamma value. For selecting thegamma value, Eq. (5) is applied to different values of γ and letthe original gamut exceed the destination gamut upto a certainthreshold level (Te). Later on, the colors that were placed outsidethe destination gamut in the previous stage are mapped back insideusing the gamut reduction algorithm presented previously.An example of both gamut reduction and extension with themethod of Zamir -et al.- is shown in Figure 1. The top row showsan original image (left) and its gamut reduced version (right), whilethe bottom row presents an original image (left) and its gamut ex-tended version (right).

Joint gamut reduction and gamut extensionAlthough much research has been devoted independently to

gamut reduction and gamut extension, the more general gamutmapping case where the intersection of source and target gamut isnot equal to one of the two gamuts has received little attention. Tofill this gap, Vazquez-Corral and Bertalmıo [17] recently showedthat by defining a smooth transition on the contrast enhancementparameter over the color space, it is possible to simultaneouslyreduce the input gamut in some areas of the color space whileincreasing it in others without introducing color artifacts or halos.This ability to modify at will the image colors allowed them toconvert the palette of colors of an original image into the paletteof colors of an unrelated second image, as can be seen in Figure2. This figure contains the original image (left), the target image(right), and the result obtained by Vazquez-Corral and Bertalmıo[17] (center).

Image dehazingImage dehazing, i.e. robustly recovering visual information

in bad weather conditions, is currently a key pre-processing taskfor many different computer vision applications such as tracking,video surveillance, or autonomous driving cars. Many differentmethods have been defined over the last years to handle this prob-lem, with the majority employing some form of constrast or satu-ration enhancement [16], [5]. This is the key connection betweenthe model of [2] and image dehazing methods.

Galdran -et al.- [4] modified the model of [2] in two different

Figure 1: Top row: Original image (left) and gamut reduced version (right) by Zamir -et al.- method [18]. Bottom row: Original image(left) and gamut extended version (right) by Zamir -et al.- method [18].

Figure 2: Original image (left), target image (right), and the result of the color modification performed by [17](center).

manners in order to perform image dehazing. First, they modifiedthe expected value of the image mean. To this end, they appliedthe Koschsmeider law of image formation in order to obtain theexpected mean of the clean image instead of the expected mean ofthe original image. Mathematically, they started considering theKoschsmieder law [8], that is written channel-wise as

I j(x) = J j(x)t(x)+(1− t(x))A j, (6)

where j ∈ {R,G,B}, x is a pixel location, I(x) is the observedintensity, J(x) is the scene radiance, corresponding to the cleanimage, transmission t(x) is a scalar quantity that is inversely relatedto the scene’s depth,and A is the airlight.

Galdran -et al.- rearranged the previous equation and took theaverage of each term leading to

mean(J j · t) = mean(I j)−mean((1− t)A j).

From this point, two crutial assumptions are considered: i) On ahaze-free image, we can expect colors to be independent of wherethe object is located in the scene. Thus, we can assume that J j andt are uncorrelated, and ii) t has a uniform distribution across theimage, i.e., depth values are equally distributed. From these twoassumptions, mean(J j) can be estimated as:

mean(J j)/2≈mean(I j)− (1/2) mean(A j).

Finally, as the airlight A can be roughly approximated by themaximum intensity value on each component, the estimation ofthe mean of the clean image is:

µj = mean(J j)≈ 2 mean(I j)−A j. (7)

Second, they introduced two new negative competing termsto couple the color channels. They correspond to the two inter-channel contrast differences (for example, in case of the red chan-nel, red versus green and red versus blue). Mathematically, theydefine the inter-channel contrast as

η

2

∫x,y

ω(x,y)|I j(x)− I j+1(y)| (8)

where as in the original model od Bertalmıo -et al.- [2] w(x,y)is a normalized Gaussian kernel of standard deviation σ , η isa constant and positive weight, and I j(x) and I j+1(y) are twointensity levels at pixel locations x and y from the color channelsj and j + 1, respectively. Galdran and colleagues showed thatby maximizing these new two terms (via a minimization of theenergy functional) they increase the difference in values betweenthe channels, therefore increasing the saturation of the images,which is related to the dark channel image dehazing idea [5].

The ensemble of both modifications lead to the followingimage energy functional

E(I j) =α

2

∫x(I j(x)−µ

j)2 +β

2

∫x(I j(x)− I j

0(x))2

− γ

2

∫x,y

ω(x,y)|I j(x)− I j(y)|

− η

2

∫x,y

ω(x,y)|I j(x)− I j+1(y)|

− η

2

∫x,y

ω(x,y)|I j(x)− I j+2(y)|, (9)

where j, j+1, and j+2 represent the three color channels.This new energy functional has as solution the steady state of

the following equation

Ik+1(x) =Ik(x)

1+∆t(α +β )

+∆t(

α

2 +β I0(x)+γ

2 RI j ,I j (x))

1+∆t(α +β )

+∆t(

η

2(RI j ,I j+1(x)+RI j ,I j+2(x)

))1+∆t(α +β )

(10)

In this case R(I j, I j+1) is defined as

RI j ,I j+1(x) =∑y∈I w(x,y)s

(I j(x)− I j+1(y)

)∑y∈I w(x,y)

(11)

Figure 3 illustrates the image enhancement produced by themethod devised in [4] by Galdran et. al. (right images) over theoriginal hazy images showing the cities of Bilbao and London (leftimages).

Conclusions and further workIn this paper we have reviewed the variational perceptually-

based color correction model of Bertalmıo and colleagues [2],and its applications to image processing applications, such ascolor gamut mapping and image dehazing. Further work is beingperformed to adapt the energy functional of [2] to take into accountdifferent priors such as natural image statistics or more complexcolor constancy formulations, and to improve its computationaltime.

AcknowledgementsJ. Vazquez-Corral. S.W. Zamir, and M. Bertalmıo were sup-

ported by European Research Council, Starting Grant ref. 306337,by the Spanish government, grant ref. TIN2012-38112, and bythe Icrea Academia Award. D. Pardo was partially funded by theProject of the Spanish Ministry of Economy and Competitivenesswith reference MTM2013-40824-P, the BCAM ”Severo Ochoa”accreditation of excellence SEV-2013-0323, the CYTED 2011project 712RT0449, and the Basque Government ConsolidatedResearch Group Grant IT649-13 on “Mathematical Modeling,Simulation, and Industrial Applications (M2SI)”

References[1] Marcelo Bertalmıo, Vicent Caselles, and Edoardo Provenzi. Issues

about retinex theory and contrast enhancement. International Journalof Computer Vision, 83(1):101–119, 2009.

[2] Marcelo Bertalmıo, Vicent Caselles, Edoardo Provenzi, and Alessan-dro Rizzi. Perceptual color correction through variational techniques.IEEE Transactions on Image Processing, 16(4):1058–1072, 2007.

[3] Sira Ferradans, Marcelo Bertalmıo, Edoardo Provenzi, and VicentCaselles. An analysis of visual adaptation and contrast perception fortone mapping. IEEE Trans. Pattern Anal. Mach. Intell., 33(10):2002–2012, 2011.

[4] Adrian Galdran, Javier Vazquez-Corral, David Pardo, and MarceloBertalmıo. Enhanced variational image dehazing. SIAM J. ImagingSciences, 8(3):1519–1546, 2015.

Figure 3: Original images (left) and images dehazed using Galdran -et al.- method [4].

[5] Kaiming He, Jian Sun, and Xiaoou Tang. Single image haze removalusing dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell.,33(12):2341–2353, 2011.

[6] Berthold K.P. Horn. Determining lightness from an image. ComputerGraphics and Image Processing, 3(4):277 – 299, 1974.

[7] Ron Kimmel, Michael Elad, Doron Shaked, Renato Keshet, and IrwinSobel. A variational framework for retinex. International Journal ofComputer Vision, 52(1):7–23, 2003.

[8] Harald Koschmieder. Theorie der horizontalen Sichtweite: Kontrastund Sichtweite. Keim & Nemnich, 1925.

[9] Edwin H. Land. The retinex. American Scientist, 52(2):247–264,1964.

[10] Edwin H. Land and John J. McCann. Lightness and retinex theory.Journal of the Optical Society of America, pages 1–11, 1971.

[11] Roberto Montagna and Graham D. Finlayson. Constrained pseudo-brownian motion and its application to image enhancement. J. Opt.Soc. Am. A, 28(8):1677–1688, Aug 2011.

[12] Edoardo Provenzi, Luca De Carli, Alessandro Rizzi, and DanieleMarini. Mathematical definition and analysis of the retinex algorithm.J. Opt. Soc. Am. A, 22(12):2613–2621, Dec 2005.

[13] Edoardo Provenzi, Massimo Fierro, Alessandro Rizzi, Luca De Carli,Davide Gadia, and Daniele Marini. Random spray retinex: A newretinex implementation to investigate the local properties of themodel. Image Processing, IEEE Transactions on, 16(1):162–171,Jan 2007.

[14] Alessandro Rizzi, Daniele Marini, Luigi Rovati, and Franco Doc-chio. Unsupervised corrections of unknown chromatic dominantsusing a brownian-path-based retinex algorithm. Journal of ElectronicImaging, 12(3):431–441, 2003.

[15] Gabriele Simone, Giuseppe Audino, Ivar Farup, Fritz Albregtsen,and Alessandro Rizzi. Termite retinex: a new implementation basedon a colony of intelligent agents. Journal of Electronic Imaging,23(1):013006, 2014.

[16] Robby T. Tan. Visibility in bad weather from a single image. In 2008IEEE Computer Society Conference on Computer Vision and PatternRecognition (CVPR 2008), 24-26 June 2008, Anchorage, Alaska,USA, 2008.

[17] Javier Vazquez-Corral and Marcelo Bertalmıo. Perceptually-inspiredgamut mapping between any gamuts with any intersection. In AICMidterm meeting, 2015.

[18] Syed Waqas Zamir, Javier Vazquez-Corral, and Marcelo Bertalmıo.Gamut mapping in cinematography through perceptually-based con-trast modification. J. Sel. Topics Signal Processing, 8(3):490–503,2014.

Author BiographyJavier Vazquez-Corral received the Ph.D. degree in computer

science from the Universitat Autonoma de Barcelona (UAB), Bellaterra,Spain, in 2011. He is a Postdoctoral fellow at Universitat Pompeu Fabra(UPF) in Barcelona. His research interests are related to the use of colorin image processing and computer vision problems. He is also interestedin bridging the gap between color in the human brain and its use incomputer-vision applications.

Syed Waqas Zamir received his B.Sc. in computer engineeringfrom COMSATS Institute of Information Technology, Pakistan, in 2010.It was later complemented by an M.Sc. in digital signal processingfrom Queen Mary, University of London, UK,in 2011. Currently,he isa Ph.D.student in image processing at the Universitat Pompeu Fabra,

Barcelona, Spain. His research interests include image processing, colorvision and computer vision. In particular, he is working in the developmentof perceptually inspired gamut mapping algorithms.

Adrian Galdran received a B.S. degree in Mathematics fromthe University of Valencia, Spain, in 2008, and a M.S. MathematicsInvestigation from the University of Valencia and the PolytechnicalUniversity of Valencia. On 2011, he got an M.S. in MathematicalModelization, Statistics and Computation at the Basque CountryUniversity. He is now finishing a PhD on Mathematical Image Processing,at the research company Tecnalia Research & Innovation, in collaborationwith the Department of Applied Mathematics at the University of theBasque Country. He is interested on image preprocessing, specially underadverse conditions, and variational techniques for computer vision andimage processing applications.

David Pardo is a Research Professor at Ikerbasque, the Universityof the Basque Country UPV/EHU, and the Basque Center for AppliedMathematics (BCAM). He has published over 110 research articles andhe has given over 190 presentations. In 2011, he was awarded as thebest Spanish young researcher in Applied Mathematics by the SpanishSociety of Applied Mathematics (SEMA). He leads a European Projecton subsurface visualization, several national research projects, as wellas research contracts with national and international companies. He isnow the PI of the research group on Mathematical Modeling, Simulation,and Industrial Applications (M2SI). His research interests includecomputational electromagnetics, petroleum-engineering applications(borehole simulations), adaptive finite-element and discontinuousPetrov-Galerkin methods, multigrid solvers, image restoration algorithms,and multiphysics and inverse problems.

Marcelo Bertalmıo received the Ph.D. degree in electrical and com-puter engineering from the University of Minnesota in 2001. He is anAssociate Professor at University Pompeu Fabra in Barcelona, Spain. Hisinterests are Image Processing and Computer Vision for digital cinema ap-plications, although he prefers the (analog) films of Joseph L. Mankiewiczand Ernest Lubitsch.