color in image & video processing applicationscat.cvc.uab.es/~joost/data/dagm_parti.pdf ·...

27/09/2010

1

Color in Image & VideoProcessing Applications

DAGM 2010Darmstadt

Joost van de Weijer

Universitat Autonoma de BarcelonaComputer Vision Center

Why use Color ?

photometric invariance discriminative power saliency detection

27/09/2010

2

Physics approach to color

R G B{ , , }

0 7

0.8

0.9

13-blue14-green

15-red16-yellow

17-magenta18

s

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

350 400 450 500 550 600 650 700 750

18-cyan

bc

e

e bc

s overview

THEORY

PART I: features• color spaces• color feature extraction• saliency detection.

APPLICATIONS

IMAGE CLASSIFICATION• combining color and shape

IMAGE SEGMENTATION• deviations from the

PART II: color constancy• bottom-up color constancy• top-down color constancy• color constant features

reflection model

OBJECT RECOLORING•complex reflection models

27/09/2010

3

Applications

segmentationclassification Object recoloring

human segmentation

Color Basics

2

- human vision- physics–based reflection model

27/09/2010

4

Human vision

Human vision

Humans have two types of photoreceptors: • rods: achromatic night time vision when light levels are low.• cones: color vision during day time when light levels are high.

27/09/2010

5

Human vision

Humans have two types of photoreceptors: • rods: achromatic night time vision when light levels are low.• cones: color vision during day time when light levels are high.

Normalized responsivity

S

M L

Normalized responsivity

opponent color system

The information from these receptors is combined in the retina as colour opponent signals through the optic nerve.

Retinal neurons appear to encode both achromatic contrast systemsRetinal neurons appear to encode both achromatic contrast systems and chromatic contrast systems.

2

g bR r

2

r bG g

2

r gB b

2 2

r gr gY b

RG R G BY B Y

Opponent mechanism [Itti PAMI 95]

((r,g,b) normalized RGB)

27/09/2010

6

opponent color system

The information from these receptors is combined in the retina as colour opponent signals through the optic nerve.

Retinal neurons appear to encode both achromatic contrast systemsRetinal neurons appear to encode both achromatic contrast systems and chromatic contrast systems.

+black-white

red-green

blue-yellow

+

+

-

Opponent mechanism Linear combination of RGB

B

G

R

O

O

O

111

211

011

3

2

1

red green

--

CIE 1931 standard observer

• computing the matching function for humans

• primary stimuli:

Image courtesy Bill Freeman

p yR=700nmG=546.1nmB=435.8nm

Projected illuminants at sub-intervals = 5nm

27/09/2010

7





27/09/2010

8





27/09/2010

9





27/09/2010

10



Experiment result(Commission International de l’Eclairage)

• Visuall field = 2 degrees• Interval = [380,780] • Sub-intervals = 5nm.


• primary stimuli:R=700nmG=546.1nmB=435.8nm

• Every position gives the mixing weights for the primary stimuli.• The resulting functions are• The resulting functions are the color matching functions

)(),(),( bgr

27/09/2010

11

Imaginary space: XYZ (Standard colorimètric Observer (CIE 1931))

Goal:• Avoid negative values in the color matching functions This simplifies

XYZ color space

Avoid negative values in the color matching functions. This simplifies the constructions of color measurement devices.• Correlate one of the functions to the intensity.

Chromaticity.exe

CIE diagram

ZYX

Yy

ZYX

Xx

ZYX

Zz

ZYX

31

31

31 ,,,,: zyxwhite

27/09/2010

12

• If two stimuli are presented, one of them at the center of the ellipse and

MacAdam Ellipses

them at the center of the ellipse, and the other one inside the ellipse, there is not a perceived difference.

• The space is not perceptually uniform (then the ellipses would be circles).

Perceptually uniform space

• CIELUV and CIELAB are examples of perceptually uniform color spaces.

kYY

if1611631

CIELUV

k

Y

Y

Y

Y

kY

Y

Y

Y

L

nn

nn

if 3.903

if 16116*

nn

ZY

Y

Yf

X

Xfa 500*

nn Z

Zf

Y

Yfb 200*

k xif 116

167.787x

k xif 3 xxf

008856.0k2*2*2** baLEab

27/09/2010

13

sRGB

• sRGB: standard RGB color space (HP, Microsoft, Pantone,Corel etc).

•Linear relation with XYZ space:

XR 498605372124063

• sRGB images are assumed to have a gamma=2.2.

• In case of unknown origin people often assume sRGB. For example in the case of data sets collected from the internet

Z

Y

X

B

G

R

linear

linear

linear

057.12040.00557.0

0415.08758.19689.0

4986.05372.12406.3

the case of data sets collected from the internet.

Device dependent color spaces

• RGB: output of majority of cameras and is dependent on the camera sensitivity response. • CMY(K): Is the standard generally applied for printing devices.

},,{0.43

},,{0.23

},,{3

2

2

2

BGRmaxBsiGR

BGRmaxGsiRB

BGRmaxRsiBG

H

HSI: human perception (others HSV, HSL)

BGRI 3

1

5.0

2

5.0

1

2

1

2

Lsi

LsiS

},,{},,{

},,{},,{

2

1

BGRminBGRmax

BGRminBGRmax

27/09/2010

14

Device dependent color spaces

• RGB: output of majority of cameras and is dependent on the camera sensitivity response. • CMY(K): Es the standard generally applied for printing devices.

HSI: often used in computer vision

B

G

R

O

O

O

3

1

3

1

3

16

2

6

1

6

1

02

1

2

1

3

2

1

Orthonormalopponent space:

O1

O3

G

B

f

f´

3

21

2

1arctan

22

OI

OOS

O

OH

333

S is not normalized with intensity !

O2RG

Projection of f on O3 is the intensity. Let f´ theprojection on the plane O1-O2, its length is thesatutation and its angle the hue.

f

Guidelines application of color spaces

• be aware that many color spaces are devise dependent (industrial applications).

• apply CIELAB or CIELUV only when perceptually uniformity of the color pp y y p p y yspace is relevant.

• when you want to decorrelate only luminance use opponent color space. HSI spaces introduces non-linearities.

Choose the color space which fits the photometric invariants your

trackingLearning color names

•Choose the color space which fits the photometric invariants your application requires.

27/09/2010

15

Reflection Models

2

Electromagnetic radiation spectrum

32101

))((log10 m

)(m )(nm610 1000

RadioTV

Radar123456789

10

RadarMicrowaves

Infrared

Visible lightUltraviolate

X rays10111213141516

710 100Gamma rays

27/09/2010

16

Pur

ple

gree

n

yello

w

oran

ge

visible light spectrum

400 450 500 550 600 650 700

P g y o

nanometers mnm 9101 Spectra.exe

http://askabiologist.asu.edu/research/seecolor/atable.htmlhttp://askabiologist.asu.edu/research/seecolor/atable.html slide credit: R. Baldrich

Surface reflectance

R G B{ , , }

0 7

0.8

0.9

13-blue14-green

15-red16-yellow

17-magenta18

s

?0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

350 400 450 500 550 600 650 700 750

18-cyan

bc

e

27/09/2010

17

Two types of reflection:

• Body reflection

• Surface reflection

Dichromatic reflection model

• Surface reflection

Body Reflectance

Normal

Metals:mainly specularreflection.

Colorant particles

Dielectric: specular and diffuse reflection. It includes materials that are coloured using pigments: paints, plastic, ceramic, fabric, paper, etc.

Reflecting materials

Body ReflectanceDiffuse reflection,

Surface ReflectanceSpecular reflection. The

reflection angle is similar toisotropic reflection. The spectral distribution

depends on colorants.

b b bf m c ( , ) ( ) ( )

reflection angle is similar to the incident angle. Its spectral distribution

depends on the illuminant.

s s s sf m c m h ( , ) ( ) ( ) ( )slide credit: R. Baldrich

27/09/2010

18

b sf f f ( , ) ( , ) ( , )

bf ( , ) : Reflected light by the object body. It depends on the pigments

Dichromatic reflection model:

bf ( , ) :

sf ( , ) : Reflected light from the surface. It has a SPD nearly the same as the incident light. (Specular or regular reflectance)

g y j y p p gused to colour the object and it’s the one that makes the object look coloured. (Diffuse reflectance)

: Angles that depend on light source position, observer and surface

The spectral and geometrical terms can be separated:

slide credit: R. Baldrich

ssbb cmcmf ,

sb ccf sb mm

Dichromatic Reflection Model

dichromatic model for matte surfaces:

bcf bm

RGB-histogram

27/09/2010

19

dichromatic model for specular surfaces:

Dichromatic Reflection Model

sb ccf sb mm

RGB-histogram

color spaces: normalized RGB

, , { , , }R G B

r g bR G B R G B R G B

• normalized RGB is given by:

• invariant for shadow and shading variations (matte surfaces):

bBb

bGb

bRb

bRb

cmcmcm

cm

BGR

Rr

bb

bg

br

br

ccc

c

normalized RGB

27/09/2010

20

color spaces: hue-saturation-intensity

3arctan

2

R Ghue

R G B

• defined as:

2 2 223sat R G B RG RB GB

3 R G Bi

sb

Bsb

Gsb

Rb

sbG

sbR

b

ccccccm

ccccmhue

22

3arctan

• hue is invariant for shading variations and specularities under white light:

hue

Take care of instabilities

• when working in different color spaces always take instabilities into account !• Error propagation is a convenient tool for instability evaluation:

q)(xq

xxx bestxx best

bestx

maxq

bestq

minq

q

q

it i tidithdth tS

22

q ...

:by defined isy uncertaint predicted The

).,...,(function compute to

,...,iesuncertaintingcorrespondwith measured are ,...,that Suppose

wu

wu

w

q

u

q

wuq

wu

27/09/2010

21


• when working in different color spaces always take instabilities into account !

• Error propagation is a convenient tool for instability evaluation:Error propagation is a convenient tool for instability evaluation:

Ex. 1

3

arctan2

R Ghue

R G B

2

2 3arctan

2

R Ghue

R G B

2 2 2

2 2 2 2hue hue hueh

2 2 2 22

1(assuming )R R G B

sat

2 2 2 2hue hue huehue R G B

R G B


• when working in different color spaces always take instabilities into account !

• Error analysis is a convenient tool for instability evaluation:

count

hue

The red bobsled is dominated by the blue sky and blue snow.

hue

hue

sat

saturation

27/09/2010

22

S.A. Shafer. Using color to separate reflection components. Color research and applications, 1985.

G.J. Klinker et al.. A physical approach to color image understanding. IJCV,

References: photometric invariants

1990.

M.J. Swain, D.H. Ballard . Color indexing. IJCV, 1991

T. Gevers, A.W.M. Smeulders. Color based object recognition. Pattern Recognition, 1999.

J.M. Geusebroek et al. Color Invariance. PAMI, 2001.

T. Gevers, H. Stokman. Robust histogram construction from colorinvariance for object recognition. PAMI, 2004.

J d W ij C S h id C l i l l f t t ti ECCV 2006 J. van de Weijer, C. Schmid. Coloring local feature extraction. ECCV, 2006

B.A. Maxwell et al. A Bi-illuminant Dichromatic Reflection Model for Understanding Images, CVPR, 2008.

T. Zickler et al. Color Subspaces as Photometric Invariants. IJCV, 2008.

10

Color Differential Structure

27/09/2010

23

differential-based computer vision

1. How do we combine the differential structure of the various color channels ?

2. How do we incorporate color invariance theory into the measurements of the differential structure while maintaining robustness ?

isoluminance

luminance gradient: isoluminant edges are not detected.

27/09/2010

24

Color Feature Detection

0-order structure 1-order structure

Color FeatureDetection

vector: xR xG+ 0=

from luminance to color

red green

+ =

+ =2‐channeltest‐image

22

22

yyyxyx

yxyxxx

GRGGRR

GGRRGR=tensor:

2

2

yyx

yxx

RRR

RRR

2

2

yyx

yxx

GGG

GGG+

DiZenzo. “Note on the Gradient of a Multi-Image”, Computer Vision, Graphics, and Image Processing, 1986.

27/09/2010

25

feature detection in oriented patterns

more tensor‐based features:

Harris corner points

symmetry points

oriented texturetraditional orientation estimation:

f f

(star and circle structures)

optical flow

orientation estimation

curvature estimation

...

arctan arctany y

x x

f f

f f

tensor‐based orientation estimation:

2 2 2 2

2 2arctan arctanx y x y

x y x y

f f f f

f f f f

differential-based computer vision

1. How do we combine the differential structure of the various color channels ?

2. How do we incorporate color invariance theory into the measurements of the differential structure while maintaining robustness ?

27/09/2010

26

• we differ between three types of edges1. material edge

Photometric Invariant Edge Detection

2. shadow/shading edge3. specular edge

• assumptions:1. white illumination2. neutral interface reflection3. shadows are not colored.

shadow

shading

material

specularity

Computation of quasi-invariance

IMAGE FULL FULL INVARIANT

nonlineartransformation

IMAGE FULL INVARIANT

FULL INVARIANT DERIVATIVE

DERIVATIVES

linear operationQUASI- INVARIANT DERIVATIVE

hue huex

27/09/2010

27

Dichromatic Model

• dichromatic model:bbCF ( )ssC

iim Cebbme CF ( )ssm C

body specular

colorant

bbm C

intensity illuminant

• first order photometric structure:bx

bxxxx mBGR CF },,{

( shadowmaterial shading ) specular

iixem C bb

xb

x emme C

Shadow-Shading-Specular Quasi-Invariant

B B B

R

G

opponent colors

R

G

spherical coordinates

R G

=

hue-saturation-intensity

specular variant

specular invariant

shading variant

shading invariant

shading-specular variant

shading-specular invariant

27/09/2010

28

spherical coordinates

• For matte surfaces :bbm cf B

shadow/shading directionR

G

• all shadow-shading variation is in the radial direction

0

uncertainty of cx

x

xrxr

x

xrxrspherical

xBxGxR

x

sin

0

0

0

sin

f

x

xx

sin

0

c

hue-saturation-intensity

• For specular surfaces : bbm cf ssm c

the hue direction

• there is no specular-shadow-shading variation in the hue-direction.

uncertainty of hx

0

0

0 xh

s

xixs

xixsxsh

hsi

xBxGxR

xf

0

0x

x

h

h

27/09/2010

29

invariant edge detection applications

Multi Image Applications • image retrieval

Colo Feature Extraction

Multi Image Applications

Single Image Applications

• image retrieval

• snakes

Color Feature Detection

• feature extraction

Instabilities

full quasiinvariant

test-image

h d h di i ishadow-shading invariance:

0},,{lim

BGR

specular-shadow-shading invariance:

}1,1,1{},,{lim

BGR

27/09/2010

30

Edge Detection

• experiments conducted on pantone colorset (1012) which is used to compose 500.000 edges.

• edge detection is based on the maximum response path of the derivative energy.

• edges are tested on- edge displacement.- percentage of missed edges.

• experiments conducted on pantone color set (1012) which is used to compose 500.000 edges.

shadow-shading:

f ll 0 21 2 0

Edge Detection

• edge detection is based on the maximum response path of the derivative energy.

• edges are tested on- edge displacement.- percentage of missed edges.

specular-shadow-shading:

full

quasi 0.043

0.21

0.99

2.0

full 0.85 9.8p g g

quasi 0.35 5.8

• Conclusion: Quasi invariants more than half the edge displacement, and have higher discriminative power.

27/09/2010

31

experiments : canny edge detection

luminance-gradient RGB-gradient

experiments : canny edge detection

shadow-shading quasi-invariant

shadow-shading-specular quasi-invariant

27/09/2010

32

Edge Classification

xFxF

xrF

shadow edges

xO3xr ,

Edge Classification

xF

specular edges

xO ,3F

27/09/2010

33

Edge Classification

red - object edgegreen-shading/shadow edgeBlue – specular edge

Photometric Invariant Corner Detection

• Harris corner detector combined with the quasi-invariants allows for photometric invariant corner detection

RGB specular-shadow-shading

shadow-shading

27/09/2010

34

experiments : Hough transform

RGB-gradient shadow-shading-specular quasi-invariant

references: color differential structure

S. DiZenzo. A note on the gradient of a multi-image. Computer Vision, Graphics, and Image Processing, 1986.

G. Sapiro and D. Ringach. Anisotropic diffusion of multivalued images with applications to color filtering. IEEE Image Processing, 1996.pp g g g,

J.M. Geusebroek et al. Color Invariance. IEEE Trans. Pattern Analysis and Machine Intelligence, 2001.

J. van de Weijer, Th. Gevers, J-M Geusebroek. Quasi-invariant edge and corner detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 2006.

J. van de Weijer, Th. Gevers, A.W.M. Smeulders, Robust Photometrical Invariant Features from the Color Tensor, IEEE T. Image Processing, 2006.

27/09/2010

35

52

Color Salient Features

Saliency Detection

• Goal: direct our gaze rapidly towards objects of interest in ourenvironment.

• Visual attention is know to be driven by both bottom up (image y p ( gbased) and top-down (task based) cues.

• Bottom-up saliency uses simple visual attributes such as intensity, contrast, color opponency, orientation , direction and velocity of motion.

• What matters is feature contrast rather than absolute feature strength (as in center surround systems).

27/09/2010

36

overview approach

L.Itty, C. Koch “Computational Modelling of Visual Attention”, Nature Reviews Neurosciende, 2001.

Computational Modeling of Visual Attention

L.Itty, C. Koch “Computational Modelling of Visual Attention”, Nature Reviews Neurosciende, 2001.

27/09/2010

37

black-white focus of detectors

luminance-based points color-based points

color distinctiveness

• the information content of an event, v, is equal to :

)log(log pppvpvI fff )log(log yx pppvpvI fff

yyyxxx BGRBGRBGRv

yxH ff ,• equation differential-based salient point detectors :

xxxx ggpp '' ffff Color Boosting Saliency:

To change from strength to information content of edges:

27/09/2010

38

statistics of color images:

• The statistics of is computed by looking of the 40.000 images of the Corel database.

xf

RGB opponent colorspace

• Isosalient surfaces can be approximated by aligned ellipsoids in decorrelated color spaces.

statistics of color images:

xxxx ggpp '' ffff Color Boosting Saliency:

RGB opponent colorspace

27/09/2010

39

• The statistics of is computed by looking of the 40.000 images of the Corel database.

xf

color boosting:

color boosting:

J. van de Weijer, Th. Gevers, A. Bagdanov, Boosting color saliency in image feature detection, IEEE PAMI 2006.

decorrelation whitening

tU

Color boosting:

color boosting:

derivativesRGB color space

derivativesopponent color space

derivativescolor boosted space

J. van de Weijer, Th. Gevers, A. Bagdanov, Boosting color saliency in image feature detection, IEEE PAMI 2006.

27/09/2010

40

bottom-up color attention:

examples:

input image color edges color boosted edgesbottom‐up attention

saliency points

input car-image

RGB-based (first 20 points) saliency boosting (first 4 points)

27/09/2010

41

generality approach: global optimal regions

RGB gradient

color boosting

experiment: image retrieval

Quantitative evaluation of color boosting on a retrieval experiment.

• Nister database: around 10.000 images• detector: DoG (color boosted)• descriptor: SIFT+hue

27/09/2010

42

experiment: image retrieval

retrieval score

theoretical maximum

amount of boosting

• color boosting improves results between 5-10 percent• the obtained maximum score is ‘equal’ to the theoretical maximum.

experiment: edge detection

Berkeley Segmentation Dataset and Benchmar

27/09/2010

43

The do’s and dont’s of Color Features

1. Take care in combining different channels: Tensor-based features solve the opposing vector problem.

2. Look at what kind of photometric invariance your problem needs:

D t t k d i ti f i l l

Quasi-invariants are more stable for feature detection.

3. When working with invariance take instabilities into account.

Use error analysis to find certainty measures for your invariants.

Do not take derivatives of circular color spaces.

Compute first derivatives, then color space transform.

4. When considering photometric invariance always also take discriminativepower into account.p

5. From information theory an optimal color space for salient feature detectioncan be derived.

6. Color information is highly corrupted in compressed data. In compression(jpeg, mpeg) chrominance is subsampled.

Questions ?

color in image & video processing applicationscat.cvc.uab.es/~joost/data/dagm_parti.pdf ·...

Documents