1057-7149 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TIP.2015.2393056, IEEE Transactions on Image Processing

Saliency-based color accessibility

Satohiro Tajima,1,2,3 and Kazuteru Komine3

January 14, 2015

Abstract Perception of color varies markedlybetween individuals because of differential ex-pression of photopigments in retinal cones.However, it has been difficult to quantify theindividual cognitive variation in colored sceneand to predict its complex impacts on the be-haviors. We developed a method for quan-tifying and visualizing information loss andgain resulting from individual differences inspectral sensitivity based on visual salience.We first modeled the visual salience for color-deficient observers, and found that the pre-dicted losses and gains in local image saliencederived from normal and color-blind modelswere correlated with the subjective judgmentof image saliency in psychophysical experi-ments, i.e., saliency loss predicted reducedimage preference in color-deficient observers.Moreover, saliency-guided image manipula-tions sufficiently compensated for individualdifferences in saliency. This visual saliency ap-proach allows for quantification of informationextracted from complex visual scenes and canbe used as an image compensation to enhancevisual accessibility by color-deficient individu-als.

Index Terms visual saliency, color vision,individual difference.

* Corresponding author; Email:

satohiro.tajima@gmail.com

1 Department of Neuroscience, University of Geneva. 1

rue Michel-Servet, Geneve, 1211, Switzerland.

2 Brain Science Institute, RIKEN. 2-1, Hirosawa, Wako,

Saitama, 351-0198, Japan.

3 Science & Technology Research Laboratories, Japan

Broadcasting Corporation. 1-10-11 Kinuta, Setagaya-ku

Tokyo 157-8510, Japan.

1 Introduction

The exclusivity of subjective experience partly de-pends on individual differences in sensory percep-tion. For example, congenital differences in the ex-pression of color-sensitive photopigments alter indi-viduals perception of complex visual scenes [1, 2].Thus, differences with regard to visual perceptionexist among individuals. A majority of humans aretrichromatic, i.e., different colors are encoded by thepopulation-activity of long-wavelength (L)-, middle-wavelength (M)-, and short-wavelength (S)-sensitivecones. Observers lacking one class of photopigmentsare deficient in their ability to discriminate colors;i.e., they are blind to the difference between a cer-tain pair of colors, such as red or green. This di-vergence in color vision is a major consequence ofgenetic polymorphisms [2] (Fig. 1a). There are cur-rently few practical methods to medically compen-sate such color-vision deficient (CD) observers so thatthey have common trichromatic vision, although genetherapies under development may soon provide a cure[3]. An alternative strategy to enhance visual acces-sibility for CD individuals is the compensatory de-sign of visual material with sufficient information tobe perceived by all viewers, including CD observers[4, 5, 6, 7].

Several graphical techniques can partially mimicthe visual experiences of CD individuals [8, 9] by re-placing each set of indistinguishable colors by a singlecolor chosen to be along one of the chromatic confu-sion lines of dichromats [10, 11, 12] (Fig. 1b). Thisprocedure provides precise chromatic metamers fordichromats. However, interpreting the simulation re-sults is problematic because the selection of repre-sentative colors is not uniquely determined but con-tains ambiguity that depends on the algorithm used.For example, various combinations of red, green, andyellow can be used to simulate the appearance ofthose colors in redgreen dichromat vision. More-over, it has been difficult to evaluate unstructured vi-sual data, such as natural images, directly from thesesimulations. It is difficult to choose the best imageparameters because the interpretation of a simula-

1

IEEE TRANSACTIONS ON IMAGE PROCESSING VOL: 24 NO: 3 YEAR 2015

1057-7149 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TIP.2015.2393056, IEEE Transactions on Image Processing

tions output depends on subjective impressions bythe viewer, and comparing subjective experiences likecolor perception among individuals is not straightfor-ward. A possible approach to comparing such sub-jective experiences is mapping the high-dimensionalperceptual content onto a common low-dimensionalsubspace and defining a perceptual difference met-ric as the distance measured in that subspace. Anave way to do this is to measure the chromatic con-trast (distance in a specific color space) in the sim-ulated images, but the problem of the ambiguity incolor mapping remains unsolved. In addition, a sim-ple color-difference analysis cannot account for coun-terexamples where dichromatic vision is superior tonormal trichromacy: dichromat observers sometimeshave advantages at finding camouflaged objects (e.g,predator/prey in the natural environment) against amulticolored background [13, 14, 15, 16, 17], and itsecological function is capturing attention in the con-text of animal and human behaviors [16, 17, 18, 19].

In this study, we propose visual saliency [20, 21,22, 23] as an alternative subspace in which to quan-tify perceptual differences (Fig. 1c). The conceptof visual salience was first proposed in the contextof cognitive science [20], and later specific computa-tional implementations were proposed [22]. Saliencyis widely used in studies on human visual attention,which is affected by the spatiotemporal context in vi-sual stimulus; for example, it has been intensively re-ported that saliency models predict many propertiesof bottom-up visual attention in trichromatic humanobservers [23, 24, 25]. Although generally the termsaliency can reflect various aspects of perceptualperformance, including eye movements, in this studywe will use this term to denote the visual conspicuitypredicted from image features. As implied by pre-vious studies [15], the notion of visual saliency po-tentially accounts for the complexity in the relation-ships between color vision and cognitive ability. How-ever, there have been surprisingly few studies thathave examined the saliency model and its behavioralimplications for observers lacking trichromatic colorvision, despite suggested importance of color infor-mation in object recognition by human [26, 27] andmachine vision [28, 29]; for example, psychophysicalstudy suggests that color information interacts withvisual object representation at multiple processinglevels [27]. In order to test whether and how muchthe proposed saliency analysis is relevant to humanperception, we developed a psychophysical methodto compare the perceptual content across observersin terms of visual salience, and the results suggestedthat a theoretically derived saliency-loss value (L) iscorrelated with the difference in subjective percep-

tual saliency reported by normal and CD observers.Moreover, further experiments revealed that this dif-ference in image preference could be ameliorated bymanipulating the visual stimuli so that there wouldbe a low saliency difference between trichromat andCD observers, thereby establishing a causal relation-ship between the quantified image salience and theperceptual judgments. This finding underscores thereliability of the present saliency-based approach formeasuring perceptual divergence ressulting from vari-ability in color vision and its potential for quantita-tively relating color vision polymorphism and behav-ioral divergence. The present approach also providesa promising and practical strategy for the design ofvisual content accessible by observers with a rangeof spectral sensitivities, including those with color-vision deficiency.

Figure 1: The considered problem and the proposed ap-proach. (a) The problem. Two observers with different color

visions have distinct perceptual experiences for the physically

identical stimulus. (b) Previous approach based on color con-

version. (c) The proposed approach based on visual salience.

2

1057-7149 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TIP.2015.2393056, IEEE Transactions on Image Processing

Figure 2: Computation of the saliency loss. (a) A model ofsaliency map for control observer having the common trichro-

matic vision. (b) Hypothetical computation of saliency maps

from the responses of dichromat observers, where the figure

shows the case of deuteranopia. CSA: center-surround antag-

onism.

2 Methods

2.1 Model of visual information diver-gence

To compare the color perception between trichro-matic and dichromatic observers when viewing com-plex visual scenes, we introduced the metric ofsaliency loss (L). If we have two salien