22

Color Filter Arrays: Representation and AnalysisISSN 1470-5559

RR-08-04 May 2008 Department of Computer Science

Yan Li, Pengwei Hao, and Zhouchen Lin

Color Filter Arrays: Representation and Analysis

Yan Li∗, Pengwei Hao†, and Zhouchen Lin‡

Abstract

A color filter array (CFA) used in a digital camera is a mosaic of spec-trally selective filters, which allows only one color (can be of a mixtureof primary colors) to be sensed at each pixel. There are two factors thatare crucial for the performance of a CFA: the demosaicking algorithm andCFA design. Demosaicking refers to the process of estimating the missingcolors, and CFA design specifies the size of the CFA pattern (one periodof the CFA) and which color to be sensed at each pixel. In this paper,we propose using the frequency structure as a representation of a CFApattern of an arbitrary rectangular size. The frequency structure is amatrix that records all the information of the multiplexing of the lumi-nance/chrominance components of the original image after it is filtered bythe CFA, modulated at certain frequency points in the frequency domain.The frequency structure can be easily obtained by computing the sym-bolic discrete Fourier transform (DFT) of the CFA pattern. It providesthe theoretical foundation for the analysis and design of the CFAs. Wegive some examples to show that characteristics of CFAs can be easilyunderstood by examining their frequency structures.

Keywords: color filter array (CFA), discrete Fourier transform (DFT),sampling, multiplexing, demosaicking

∗Y. Li is with the State Key Laboratory of Machine Perception, Peking University, Beijing100871, China (e-mail: yanli8083@gmail.com).

†P. Hao is with the Department of Computer Science, Queen Mary, University of London,London E1 4NS, U.K.,(e-mail: phao@dcs.qmul.ac.uk).

‡Z. Lin is with Microsoft Research Asia, 5th Floor, Sigma Building, Zhichun Road 49#,Haidian District, Beijing 100080, China (e-mail: zhoulin@microsoft.com).

1

1 Introduction

Nowadays, digital cameras have become very popular in daily life. In theory, tocapture a color image each pixel should have at least three sensors in order tosimultaneously sense at least three colors, e.g., red, green, and blue (R, G, andB), or cyan, magenta, and yellow (C, M, and Y). However, to reduce the cost,size, and complexity, most digital cameras use a single-chip sensor covered witha Color Filter Array (CFA) [1, 29]. A CFA is a mosaic of spectrally selectivefilters, which is of the same size as the sensed image and allows only one color tobe sensed at each pixel. Thereby, the sensed image, i.e., the CFA-filtered image,has only one color at each pixel. The missing two colors of each pixel have to beestimated by methods called demosaicking algorithms [1, 13] in order to recoverthe original full color image.

Numerous demosaicking algorithms have been proposed in the literature(e.g., [2, 8, 11, 12] and the reviews and comparisons in [13, 19, 24, 26], toname just a few). They all try to make full use of the spatial and spectralcorrelations in order to obtain sharp images with less artifacts [6], where thespatial correlation refers to the fact that within a homogeneous image region,neighboring pixels have similar color values, while the spectral correlation is thehigh correlation among different color channels.

As demosaicking algorithms depend on the configurations of the CFAs (e.g.,the size of CFA pattern and which color to sense at each pixel) heavily, ratherthan making the algorithms more and more complicated, an alternative way toimprove the quality of the demosaicked images is to optimize the configurationsof CFAs. Since a CFA is usually a periodic tiling of a much smaller array, whichis called the CFA pattern in this paper, the problem reduces to finding optimalCFA patterns. Many CFA patterns have been proposed [3, 4, 5, 7, 9, 10, 14,15, 17, 20, 21, 22, 23, 25, 27, 28, 30, 31, 32], among which some have been usedin commercial products [4, 27] and the most commonly used one is the Bayerpattern [4] (Fig. 1(a)).

Although there have been a large number of demosaicking algorithms andCFA patterns, the theoretical foundation for their design is still unavailable.Most of the demosaicking algorithms and CFA patterns were proposed based onsome empirical assumptions or observations, e.g., some of the algorithms adoptthe constant-hue assumption [16, 24] and some exploited the high-frequency cor-relation among the RGB channels [11, 12]. Moreover, most of the demosaickingalgorithms are specialized for the Bayer CFA, and the choice of CFA filter colorsis also limited (e.g., R, G, B, C, M, Y, W(white)). To compare the performanceof demosaicking algorithms and the characteristics of CFA patterns, doing ex-periments with the benchmark test images may be the only way (e.g., [20]). Thismay not be convenient and effective if one wants to find the optimal solutions.

In this report and its companion report [18], we propose a theoretical frame-work that can help analyze the characteristics of CFA patterns and design newCFA patterns with required characteristics in a systematic way. Our work isinspired by [2] and [8], which investigated the representation of images sampledwith a Bayer CFA and the demosaicking algorithms in the Fourier frequencydomain. The authors showed that an image sampled with a Bayer CFA is equiv-alent to the frequency domain multiplexing of a luminance component (luma)at the baseband and two chrominance components (chromas) at high frequencybands. Therefore, the full color image can be recovered by demultiplexing of

2

G B

R G

(a)

G C Y M

(b)

Figure 1: CFA patterns and their primary CFA patterns. (a) Bayer [4]. (b)Hamilton et al. [14].

luma and chromas, followed by a linear transformation. Their experimentsshowed that such a simple demosaicking algorithm could achieve comparableor even better performance than other complicated spatial-domain techniques.However, the theories in [2] and [8] are limited to the Bayer CFA only. Moreover,they can only deal with RGB primary colors. In comparison, our framework ismuch more general. It is based on a representation of CFAs called the frequencystructure. This representation makes it very simple and straightforward to vi-sualize the characteristics of a given CFA pattern, and thus provides anotherway to evaluate CFA patterns theoretically rather than experimentally. Usingour theories, we can handle various kinds of CFA patterns (with different sizesand different color patterns to sense, and not limited to RGB primary colors),recover the full-color image (demosaicking) for any rectangular CFA by sim-ple frequency-domain filtering and linear transform, and even design new CFApatterns with customizable characteristics and optimized parameters [18].

This paper is organized as follows. In Section 2, we derive the equivalence ofsampling with any rectangular CFAs in the spatial domain to the multiplexingof one luma and some chromas in the frequency domain. The multiplexing isrecorded by a matrix called the frequency structure of a CFA (pattern), whichis shown to be just the symbolic DFT of the CFA pattern. As examples, thefrequency structures of some existing CFAs are also presented. In Section 3, weshow that the characteristics of CFAs can be analyzed based on the frequencystructure. And the experiments show that with the analysis, the artifacts of thedemosaicked images can be easily understood. Finally, Section 4 concludes thispaper.

2 CFA Representation

In this section, we propose a symbolic representation of the CFA: the frequencystructure, which consists of the frequency components of a CFA-filtered imagecentered (or modulated) at certain frequency lattice points. Each frequencycomponent can be found to be a mixture of the spectra of different color channelsof the original image. The frequency structure is written in a matrix form thatcorresponds to the frequency lattice so that it is visually intuitive to see what the

3

frequency component looks like at each frequency lattice point. The spectrum ofthe CFA-filtered image is simply the sum of these frequency components whenthe components are shifted to be centered at the corresponding frequency points(frequency domain multiplexing). With this representation, it is easy to analyzethe characteristics of a CFA. We can also design various CFAs that meet ourrequirements and can be demosaicked in a systematic way.

2.1 CFA patterns

We view each CFA hCFA(x, y) as a tiling of its rectangular period called theCFA pattern hp(x, y). By the color that each pixel senses, a CFA pattern can bewritten as a matrix of symbols representing the colors C (e.g., C ∈ {R, G, B} orC ∈ {C, Y,M,G}) and accordingly it can be decomposed into base CFA patterns:

h(C)p (x, y) =

{1, if hp(x, y) = C,0, otherwise.

Then the sum of all base CFA patterns is an all-one matrix:∑

Ch(C)

p (x, y) = 1,∀x, y, (1)

and symbolically we can write:

hp =∑

Ch(C)

p · C. (2)

For example, the Bayer CFA pattern is (Fig. 1(a)):

hBayer =[

G RB G

](3)

and the corresponding base CFA patterns are respectively:

h(R)Bayer =

[0 10 0

], h(G)

Bayer =[

1 00 1

], h(B)

Bayer =[

0 01 0

]. (4)

Note that the colors of the CFA filters are not necessarily the three primarycolors (e.g., Fig. 1(b)). They may be mixtures of the three primary colors.Replacing CFA colors with the mixtures of the three primary colors, e.g., R, G,and B, we can also have primary CFA patterns h(R)

p , h(G)p , and h(B)

p , such that

hp = h(R)p · R + h(G)

p · G + h(B)p · B (5)

holds symbolically. It is easy to see that the primary CFA patterns are themixtures of the base CFA patterns, and the sum of all primary CFA patterns isalso an all-one matrix:

h(R)p (x, y) + h(G)

p (x, y) + h(B)p (x, y) = 1,∀x, y. (6)

For example, for the Bayer CFA pattern, the primary CFA patterns are the sameas the base CFA patterns. While for the Hamilton CFA pattern (Fig. 1(b)), theprimary CFA patterns are respectively:

h(R)Ham =

[12

12

0 0

], h(G)

Ham =[

0 12

12 1

], h(B)

Ham =[

12 012 0

], (7)

4

as M = (R + B)/2, Y = (R + G)/2, and C = (G + B)/2.Note that if colors C represent the primary colors, e.g., C ∈ {R, G, B}, Eqs.

(2) and (1) reduce to Eqs. (5) and (6), respectively. So we may also write Eqs.(2) and (1) if C represent the primary colors.

2.2 Spectra of CFA-filtered images

Let f(x, y) be the full color image of size (Nx, Ny) and the CFA pattern hp(x, y)be of size (nx, ny). Then the CFA-filtered image is:

fCFA(x, y) =∑

Cf (C)(x, y) · h(C)

CFA(x, y), (8)

where f (C)(x, y) is the color C component of f and h(C)CFA(x, y) is the correspond-

ing CFA of color C defined as the periodic replica of the primary CFA patternh(C)

p (x, y):h(C)

CFA(x, y) = h(C)p (x mod nx, y mod ny). (9)

From the linearity of the Fourier transform, we may first investigate the spec-trum of each color component f (C)

h = f (C) · h(C)CFA separately and then sum

them together to have the spectrum of a CFA-filtered image. Without loss ofgenerality, we assume that Nx and Ny are multiples of nx and ny, respectively.

We first compute the spectrum of h(C)CFA(x, y) :

H(C)CFA(ωx,ωy) = DFT [h(C)

CFA(x, y)]

=1

NxNy

Nx−1∑

x=0

Ny−1∑

y=0

h(C)CFA(x, y)e−i2π(xωx+yωy)

=1

NxNy

Nx/nx−1∑

lx=0

Ny/ny−1∑

ly=0

nx−1∑

x=0

ny−1∑

y=0

h(C)p (x, y)

· e−i2π[(nxlx+x)ωx+(nyly+y)ωy]

=

(1

nxny

nx−1∑

x=0

ny−1∑

y=0

h(C)p (x, y)e−i2π(xωx+yωy

)

·

nx

Nx

Nx/nx−1∑

lx=0

e−i2πnxlxωx

ny

Ny

Ny/ny−1∑

ly=0

e−i2πnylyωy

={

H(C)p (ωx,ωy) if nxωx ∈ Z and nyωy ∈ Z

0 otherwise

(10)

where H(C)p (ωx,ωy) is the DFT of the primary CFA pattern h(C)

p (x, y). Notethat here (ωx,ωy) takes discrete values in the square [0, 1)2 (at a stepsize of(1/Nx, 1/Ny) for H(C)

CFA and (1/nx, 1/ny) for H(C)p , respectively) instead of dis-

crete indices of the signals because we have found it more convenient to normal-ize the frequencies. The above equality shows that the spectrum of a CFA is asampling of the spectrum of its CFA pattern at frequency points (kx/nx, ky/ny),where (kx, ky) ∈ Z2.

5

Figure 2: 1D illustration of the frequency components of a color channel (thesuperscript C omitted). Each frequency component is a weighted spectrum ofthe color channel of the original image. The centers of the frequency compo-nents form a lattice. The spectrum of the CFA-filtered image is the sum ofthe frequency components by putting them at the frequency lattice points andcircularly shifting them due to periodicity. The dashed lines indicate the centersof spectra.

As multiplication in the spatial domain corresponds to the circular convolu-tion in the frequency domain, the DFT of f (C)

h can be found to be:

F (C)h (ωx,ωy) = DFT [f (C)(x, y) · h(C)

CFA(x, y)]

=F (C)(ωx,ωy) ! H(C)CFA(ωx,ωy)

=Nx−1∑

kx=0

Ny−1∑

ky=0

F (C)

(ωx −

kx

Nx,ωy −

ky

Ny

)· H(C)

CFA

(kx

Nx,

ky

Ny

)

=∑

kxnx/Nx ∈ Zkyny/Ny ∈ Z

F (C)

(ωx −

kx

Nx,ωy −

ky

Ny

)· H(C)

p

(kx

Nx,

ky

Ny

)

=nx−1∑

kx=0

ny−1∑

ky=0

H(C)p

(kx

nx,ky

ny

)· F (C)

(ωx −

kx

nx,ωy −

ky

ny

).

(11)

where F (C)(ωx − kx

nx,ωy − ky

ny

)has been circularly shifted. This implies that in

the frequency domain the spectrum F (C)h (ωx,ωy) of the CFA-filtered image is

a multiplexing of nxny frequency components centered at (kx/nx, ky/ny), kx =0, 1, · · · , nx − 1; ky = 0, 1, · · · , ny − 1, and each component is the spectrum ofthe original image weighted by the value of the spectrum of the CFA pattern atthe frequency point. Fig. 2 illustrates this observation in the 1D case.

6

(a) (b) (c)

Figure 3: Spectra of filtering on ‘lighthouse’ (nx = ny = 5). (a) The red channelF (R) of the original image. (b) The red channel F (R)

h of the CFA-filtered image.(c) The CFA-filtered image FCFA, which is the sum of three (red, green andblue) channels.

2.3 The frequency structure as the representation of CFA

We may use a matrix S(C) to record the frequency components of a color channelC of the CFA-filtered image:

S(C) =[H(C)

p

(kx

nx,ky

ny

)· F (C)(ω)

]

kx = 0, 1, · · · , nx − 1,ky = 0, 1, · · · , ny − 1.

(12)

We call this matrix the frequency structure of the primary CFA h(C)CFA of color C.

It contains all the information about the frequency components: the (kx, ky)-thentry is the frequency component centered at (kx/nx, ky/ny). Fig. 3(b) showsthe spectrum of the red channel of the CFA-filtered image ‘lighthouse’ which,according to (11), is the sum of all the frequency components by centering thefrequency components at their corresponding frequency points. At the frequencylattice, the profile of the frequency component of the red channel is clearlyvisible.

Then according to (8), we may define the frequency structure of a CFApattern as the following matrix:

SCFA =∑

CS(C). (13)

So the entries of the matrix SCFA are actually

SCFA(kx, ky) =∑

CH(C)

p

(kx

nx,ky

ny

)· F (C)(ωx,ωy), (14)

which is the multiplexing of the frequency components of all color channels. Forthis reason, we refer to the entries in SCFA as the multiplex components.

By applying DFT to both sides of Eq. (1), we have that:

∑

CH(C)

p

(kx

nx,ky

ny

)= δ(kx)δ(ky), (15)

which means that the sums∑

C H(C)p (kx/nx, ky/ny) of the coefficients for all

multiplex components (14) are zero, except the one at the baseband (frequency

7

(0, 0)), whose sum is 1. As in [2] and [8], we shall call the multiplex componentat the baseband the luminance component (luma) and the others the chromi-nance components (chromas). Fig. 3(c) shows the spectrum of the CFA-filtered‘lighthouse’ image, from which we can clearly observe one luma at the basebandand a number of chromas at high frequency points. These components revealthe profile of the frequency structure of the CFA pattern.

2.4 A symbolic formula to compute the frequency struc-ture

By the definition (14), it seems a little complex to compute the frequency struc-ture as we may have to compute the DFT of all the primary CFA patterns.However, we have found that there is a simple way to compute the frequencystructure of a CFA. To proceed, we have to introduce the symbolic DFT of asequence of symbols. For a string s = s0s1 · · · sN−1, its symbolic DFT is asequence of order-1 polynomials S = S0S1 · · ·SN−1, where

Sk =1N

N−1∑

l=0

sle−2πikl/N .

For the 2D case, the symbolic DFT can be defined in an analogous way. Withthis definition, we can claim that:

Theorem 1. If we rewrite F (C)(ω) as C, then the frequency structure SCFA isthe symbolic DFT of the CFA pattern hp.

Proof. The symbolic DFT of the CFA pattern hp is Hp = DFT [hp], where

Hp(kx, ky)

=1

nxny

nx−1∑

x=0

ny−1∑

y=0

hp(x, y)e−2πi(xkx/nx+yky/ny)

=1

nxny

∑

CC

nx−1∑

x=0

ny−1∑

y=0

h(C)p (x, y)e−2πi(xkx/nx+yky/ny)

=∑C

H(C)p (kx/nx, ky/ny) · C.

(16)

Hence, the claim is true by comparing the above with (14). "From this theorem, in the sequel, we also use C to represent the spectrum of

the color channel C of the original image.Similarly, we can also have S(C) = DFT [h(C)

p ] · C.

2.5 Examples of frequency structures

In this section, we give some examples of the frequency structures of a fewexisting CFA patterns. Thanks to Theorem 1, the frequency structures can beeasily computed.

The frequency structure of the Bayer CFA pattern (Fig. 4(a)) is

SBayer = DFT

[G RB G

]

=14

[R + 2G + B −R + B

R−B −R + 2G−B

].

(17)

8

G B

R G

(a)

G C Y M

(b)

G

B R G B R

G B R

(c)

G B R G

G B R

G

(d) (e)

Figure 4: Some existing CFA patterns. The second row are the spectra of image‘lighthouse’ sampled with the corresponding CFA patterns in the first row. (a)Bayer [4]. (b) Hamilton et al. [14]. (c) Diagonal Stripe [9]. (d) Yamanaka [31].(e) Parmar & Reeves [22].

Note that by convention the spectra of the CFA-filtered images are displayedwith (0, 0) being at the center (please be reminded that the spectra are periodicin both horizontal and vertical directions), while in the frequency structurerepresentation of CFAs, (0, 0) is at the top-left corner of the matrix. The readersshould circularly shift either of them in order to match them. Thus the frequencystructure SBayer means that the spectrum of any image sampled with the BayerCFA has a luma (R+2G+B)/4 at the baseband, and three chromas −(R−B)/4,(R−B)/4, and −(R− 2G + B)/4 centered (or modulated) at frequencies (1/2,0), (0, 1/2), and (1/2, 1/2), respectively.

Similarly, we can obtain the frequency structure of any rectangular CFA. Forexample, the frequency structure of the CFA pattern proposed by Hamilton, etal. [14] (nx = ny = 2, Fig. 4(b)) is

SHam = DFT

[R+B

2R+G

2G+B

2 G

]

=14

[R + 2G + B −G + B

R−G 0

],

(18)

and that of the Diagonal Stripe CFA pattern [9], [20] (nx = ny = 3, Fig. 4(c))is

SDiag = DFT

R B GG R BB G R

=

FL 0 00 0 FC1

0 FC2 0

,

(19)

where

FL

FC1

FC2

= TDiag ·

RGB

, and (20)

9

(a) (b) (c)

Figure 5: Average spectra of multiplex components (a) (R + G + B)/3, (b)(R−G)/2, and (c) (R + G− 2B)/4.

TDiag =16

2 2 22 −1− i

√3 −1 + i

√3

2 −1 + i√

3 −1− i√

3

, (21)

and that of Yamanaka’s CFA pattern [31] (nx = 4, ny = 2, Fig. 4(d)) is

SY am = DFT

[G R G BG B G R

]

=[

FL 0 FC1 00 FC2 0 −FC2

],

(22)

where

FL

FC1

FC2

= TY am ·

RGB

and (23)

TY am =14

1 2 1−1 2 −1−1 0 1

. (24)

We call TDiag and TY am above the multiplexing matrix, as it relates the multiplexcomponents and the spectra of primary color components. Some existing CFApatterns and the spectra of image ’lighthouse’ sampled with the correspondingCFA patterns are shown in Fig. 4. A more comprehensive list of frequencystructures of CFA patterns can be found in Table 1.

3 CFA Analysis

The analysis in Section 2.3 shows that the spectrum of a CFA-filtered im-age is the mixture of its multiplex components (14) modulated at frequencies(kx/nx, ky/ny), kx = 0, 1, · · · , nx − 1; ky = 0, 1, · · · , ny − 1. Thus the character-istics of a CFA depend on the features of the multiplex components themselvesand the crosstalk among them, which is determined by their arrangement in thefrequency domain. In the following, we first discuss the statistical characteris-tics of multiplex components, and then analyze CFAs based on their frequencystructures. With the analysis, the artifacts in the demosaicked images can beeasily understood, as will be shown by the experiments.

10

Table 1: Some existing CFA patterns and their frequency structures.

Inventor (year) CFA Pattern Frequency Structure Multiplexing Matrix T

Bayer [4](1976)

»

R GG B

– »

FL FC2

−FC2 FC1

–

1

4

2

4

1 2 1−1 2 −1−1 0 1

3

5

Dillon [7](1977)

2

6

4

W R W BB W R WW B W RR W B W

3

7

5

2

6

4

FL 0 0 00 0 0 −FC2

0 0 FC1 00 FC2 0 0

3

7

5

1

12

2

4

5 2 5−1 2 −1−3i 0 3i

3

5

Yamanaka [31](1977)

»

G R G BG B G R

– »

FL 0 FC1 00 FC2 0 −FC2

–

1

4

2

4

1 2 1−1 2 −1−1 0 1

3

5

Takashi, et al. [28](1985)

2

6

4

G RG BB GR G

3

7

5

2

6

4

FL 0F ∗

C2FC2

0 FC1

−FC2 F ∗

C2

3

7

5

1

8

2

4

2 4 2−2 4 −2

−1 − i 0 1 + i

3

5

Morimura, et al. [21](1986)

»

M C G YM Y G C

– »

FL FC1 0 FC1

0 FC2 0 −FC2

–

1

8

2

4

2 4 22 −4 2i 0 −i

3

5

Yamagami, et al. [30](1994)

2

6

4

W G W GR W B WW G W GB W R W

3

7

5

2

6

4

FL 0 FC1 00 FC2 0 FC2

−FC1 0 FC3 00 −FC2 0 −FC2

3

7

5

1

24

2

6

4

7 10 73 −6 3

−3i 0 3i1 −2 1

3

7

5

Bawolek, et al. [3](1999)

»

M WW Y

– »

FL FC2

FC2 FC1

–

1

24

2

4

10 7 72 −1 −10 −1 1

3

5

Hamilton, et al. [14](2001)

»

M YC G

– »

FL FC1

FC2 0

–

1

4

2

4

1 2 10 −1 11 −1 0

3

5

Gindele, et al. [10](2002)

»

W RB G

– »

FL FC1

FC2 FC3

–

1

6

2

6

4

2 2 2−1 −1 22 −1 −1−1 2 −1

3

7

5

Bean [5](2003)

»

C MB Y

– »

FL FC1

0 FC2

–

1

4

2

4

1 1 2−1 0 10 1 −1

3

5

Roddy, et al. [25](2006)

2

6

4

R G B CG R C BB C R GC B G R

3

7

5

2

6

4

FL 0 0 00 FC2 0 FC3

0 0 FC1 00 FC3 0 FC2

3

7

5

1

8

2

6

4

2 3 32 −3 10 −i i1 0 −1

3

7

5

Hirakawa, et al. [15](2007)

»

M Y C GG C Y M

– »

FL 0 0 00 F ∗

C2FC1 FC2

–

1

8

2

4

1 2 10 −1 1

1 + i −1 − i 0

3

5

Kodak [17](2007)

2

6

4

W B W GB W G WW G W RG W R W

3

7

5

2

6

4

FL F ∗

C30 FC3

F ∗

C3−FC2 −FC3 0

0 −FC3 FC1 −F ∗

C3

FC3 0 −F ∗

C3FC2

3

7

5

1

48

2

6

4

14 20 142 −4 26i −12i 6i

−3 − 3i 0 3 + 3i

3

7

5

FillFactory [9]

2

4

R B GG R BB G R

3

5

2

4

FL 0 00 0 FC1

0 FC2 0

3

5

1

6

2

4

2 2 22 −1 − i

√

3 −1 + i√

32 −1 + i

√

3 −1 − i√

3

3

5

Notes:

white: W = (R + G + B)/3cyan: C = (G + B)/2

magenta: M = (R + B)/2yellow: Y = (R + G)/2

, and T is the multiplexing matrix such that

2

6

6

6

4

FL

FC1

FC2

.

.

.

3

7

7

7

5

= T ·

2

4

RGB

3

5

11

G B R G

B R

G B R (a) (b)

Figure 6: The Anti-Diagonal Stripe CFA constructed by us and the spectrumof image ‘lighthouse’ sampled with it.

3.1 Statistics of multiplex components

We have mentioned in Section 2.3 that depending on the sum of the coefficients(15), there are two kinds of multiplex components: luma and chroma. Withoutloss of generality, we consider one luma (a) (R + G + B)/3, and two chromas:(b) (R−G)/2 and (c) (R+G−2B)/4. Fig. 5 shows the average spectra of eachmultiplex component computed from the natural color images in the 24-imageKodak test set as used in [8] and [13].

It can be observed that the luma is much larger than the chromas andoccupies the majority of the frequency space. In contrast, the chromas aresmall and compact in the low frequency band due to the so-called high spectralcorrelation in natural color images, i.e., there is high correlation among differentcolor channels, especially for the high-frequency subbands [6, 12]. Actually,based on the assumption that the high-frequency information in the R, G, and Bchannels is identical, an alias cancellation demosaicking algorithm was proposedin [11].

Fig. 5 also shows that the spectra are relatively strong along the horizontaland vertical axes, and that the region of large spectral values is diamond -shaped.This should result from the fact that there are more horizontal and vertical edgesin natural images.

The above statistics of luma and chromas are useful in analyzing the prop-erties of CFAs.

3.2 Analysis of CFAs

As all the multiplex components coexist in the same frequency space, it is in-evitable that aliasing occurs, which leads to demosaicking artifacts in the de-mosaicked images, such as blur, false color, and zippering. With the frequencystructure and the statistics of multiplex components, we can predict the per-formance of a CFA by analyzing at what frequencies and to what degree thefrequency aliasing occurs.

It is easy to see that the spectra around nonzero chromas will suffer morealiasing, and the amount of aliasing depends on the distance between the mul-tiplex components as well as their relative positions.

For example, the frequency structure SBayer (17) of the Bayer CFA shows

12

that luma and chromas overlap around frequencies (1/2, 0), (0, 1/2) and (1/2, 1/2).The chromas −(R−B)/4 at (1/2, 0), and (R−B)/4 at (0, 1/2) will suffer morealiasing because the luma is high in these two directions, as shown in Fig. 4(a).This is the major shortcoming of the Bayer CFA. The frequency structure SDiag

(19) (Fig. 4(c)) shows that the two chromas of the Diagonal Stripe CFA aremodulated at frequencies (1/3,−1/3) and (−1/3, 1/3), respectively. This CFAis good in the sense that the chromas are not on the horizontal or vertical axes.However, it is still unsatisfactory because the distance between the chromas andthe luma is small.

Similarly, we can analyze all CFAs. The frequency structures of some ex-isting CFA patterns are shown in Table 1, from which we can easily know thecharacteristics of each pattern.

3.3 Understanding demosaicking artifacts

With the analysis of spectral aliasing based on the frequency structure SCFA,it becomes easy to understand the artifacts of the demosaicked images in thespatial domain. Some experiments on the ability of the CFAs (and the de-mosaicking algorithms) to preserve various frequencies were conducted in [20].Their observations and conclusions become apparent if we investigate the fre-quency structures of the CFAs.

As examples, we compare the images demosaicked from three CFAs: Bayer(Fig. 4(a)), Diagonal Stripe (Fig. 4(c)) and Anti-Diagonal Stripe (Fig. 6). TheAnti-Diagonal Stripe CFA is constructed by us for comparison, whose chromasare modulated at frequencies (−1/3,−1/3) and (1/3, 1/3), respectively, whilethe chromas of Diagonal Stripe are modulated at frequencies (1/3,−1/3) and(−1/3, 1/3), respectively. We adopt the following universal frequency domaindemosaicking algorithm for all CFAs: first estimate each multiplex componentsby frequency-domain filtering and then reconstruct the red, green and bluecomponents by a linear transformation, as [2] and [8] did. Parts of demosaickedimages of Images 5 and 19 of the 24-image Kodak test set are shown in Fig. 7.

For the Bayer CFA, Fig. 7(a) shows that the demosaicked images havegreater error for the vertical and horizontal spokes in Image 5 and the picketfence (nearly vertical lines) in Image 19. These are caused by the crosstalkbetween chromas at (1/2, 0) (or (0, 1/2)) and the luma at the baseband. Forthe Diagonal Stripe CFA, Fig. 7(b) shows that the worst reconstructed areasare the spokes around the direction of −45o in Image 5 and the right side of thehouse roof in Image 19, which are related to the chromas at (1/3,−1/3) and(−1/3, 1/3). In contrast, for the Anti-Diagonal Stripe CFA, Fig. 7(c) shows thatthe spokes around the direction of 45o in Image 5 and the left side of the houseroof in Image 19 have more artifacts after demosaicking, which are related to thechromas at (1/3, 1/3) and (−1/3,−1/3). Thus, the experiments demonstratethat, based on the frequency structure of a CFA, we can analyze the crosstalkbetween multiplex components and thus understand the demosaicking artifacts.

3.4 Expected characteristics of a good CFA

The above analysis shows that the fundamental reason for demosaicking ar-tifacts is the crosstalk among multiplex components, especially between lumaand chromas, which are determined by the frequency structure of a CFA. Thus,

13

( c )

( b )

( a )

( o )

Figure 7: (o) Original images. (a)-(c) Parts of the demosaicked results of Images5 and 19, which are first sampled with the corresponding CFAs: (a) Bayer. (b)Daigonal Stripe. (c) Anti-Diagonal Stripe.

to reduce the demosaicking artifacts, a good CFA should make the crosstalk

14

among multiplex components as little as possible. This would require that thefrequency structure of the CFA should have some desired characteristics. Forexample, first, the distance between every two nonzero multiplex components,especially the luma and a chroma, should be as large as possible; second, thechromas should not be centered on the vertical or horizontal axes of the luma,where the luma has relatively high energy. A CFA design methodology to obtainCFAs of expected characteristics is presented in our companion report [18].

4 Conclusions

In this paper we derive a frequency domain representation of CFA-filtered im-ages, which is a multiplexing of one luma at the baseband and a number ofchromas modulated at certain high frequency points. A matrix, called the fre-quency structure, is defined to record the information of the multiplexing and isused as the representation of a CFA. It is shown to be just the symbolic DFT ofthe CFA pattern. With the representation, we can: (i). visualize the character-istics of a CFA; (ii). analyze its properties theoretically, such as understandingthe artifacts of the demosaicked images; and (iii). give some guidelines for CFAdesign. Our theory is the first one that is applicable to different CFAs.

Based on our theory, better CFAs with different sizes and color sensingconfigurations can also be designed [18]. With such CFAs, a simple and universalfrequency domain demosaicking algorithm which only involves linear filteringand linear transformation [2, 8] will suffice to demosaick CFA-filtered images,thus waiving the effort to design complicated and computationally intensivedemosaicking algorithms. We believe that our theory can also be applied torelated areas such as multispectral filter arrays and color display designs.

References

[1] J. Adams, K. Parulski, and K. Spaulding. Color processing in digital cam-eras. IEEE Micro, vol. 18, no. 6, pp. 20-31, 1998.

[2] D. Alleysson, S. Susstrunk, and J. Herault. Linear demosaicing inspired bythe human visual system. IEEE Trans. Image Process., 14(4):439-449, 2005.

[3] E.J. Bawolek, Z.F. Li, and R.D. Smith. Magenta-white-yellow (MWY) colorsystem for digital image sensor applications. U.S. Patent 5 914 749, Jun.1999.

[4] B.E. Bayer. Color imaging array. U.S. Patent 3 971 065, 1976.

[5] J.J. Bean. Cyan-magenta-yellow-blue color filter array. U.S. Patent 6 628331, Sep. 2003

[6] L. Chang and Y. P. Tan. Effective use of spatial and spectral correlations forcolor filter array demosaicking. IEEE Trans. Consum. Electron., 50(1):355-365, 2004.

[7] P.L.P. Dillon. Color Imaging Array. US Patent 4 047 203, Sept. 1977.

15

[8] E. Dubois. Frequency-domain methods for demosaicking of Bayer-sampledcolor images. IEEE Signal Process. Lett., 12:847-850, 2005.

[9] FillFactory. Technology image sensor: the color filter array.

[10] E.B. Gindele and A.C. Gallagher. Sparsely sampled image sensing devicewith color and luminance photosites. U.S. Patent 6 476 865 B1, Nov. 2002.

[11] J.W. Glotzbach, R.W. Schafer, and K. Illgner. A method of color filterarray interpolation with alias cancellation properties. Proc. IEEE Int. Conf.Image Processing, vol. 1, 2001, pp. 141-144.

[12] B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau. Color plane interpo-lation using alternating projections. IEEE Trans. Image Processing, vol. 11,no. 9, pp. 997-1013, Sept. 2002.

[13] B.K. Gunturk, J. Glotzbach, Y. Altunbask, R.W. Schafer, and R.M.Mersereau. Demosaicing: Color filter array interpolation. IEEE Signal Pro-cess. Mag., 22(1):44-54, 2005.

[14] J.F. Hamilton, J.E. Adams, and D.M. Orlicki. Particular pattern of pixelsfor a color filter array which is used to derive luminanance and chrominancevalues. U.S. Patent 6 330 029 B1, Dec. 2001.

[15] K. Hirakawa and P.J. Wolfe. Spatio-Spectral Color Filter Array Design forEnhanced Image Fidelity. Proc. IEEE Int. Conf. Image Processing, 2007.

[16] R. Kimmel, Demosaicing: image reconstruction from ccd samples. IEEETrans. Image Process., vol. 8, pp. 1221-1228, 1999.

[17] Kodak Press Release. [online]. June 2007. Available at:http://www.kodak.com/ or http://www.dpreview.com/news/0706/07061401kodakhighsens.asp

[18] Y. Li, P. Hao, and Z. Lin. Color Filter Arrays: A Design Methodology.Technical Report, Department of Computer science, Queen Mary, Universityof London, 2008.

[19] P. Longere, X. Zhang, P.B. Delahunt, and D.H. Brainard. Perceptual as-sessment of demosaicing algorithm performance. Proc. IEEE, vol. 90, no. 1,pp. 123-132, Jan. 2002.

[20] R. Lukac and K. N. Plataniotis. Color filter arrays: Design and performanceanalysis. IEEE Trans. Consum. Electron., 51(4):1260-1267, 2005.

[21] A. Morimura and H. Tanaka. Color solid-state imager with color filter hav-ing an overlapping segmented filter arrangement. U.S. Patent 4 630 106,Dec. 1986.

[22] M. Parmar and S. J. Reeves. A perceptually based design methodologyfor color filter arrays. Proc. Int. Conf. Acoustics, Speech, and Signal Proc.,473-476, 2004.

[23] M. Parmar and S. J. Reeves. Selection of optimal spectral sensitivity func-tions for color filter arrays. Proc. IEEE Int. Conf. Image Processing, 2006,pp. 1005-1008.

16

[24] R. Ramanath, W.E. Snyder, G.L. Bilbro, and W.A. Sander III, Demosaick-ing methods for Bayer color arrays. J. Electron. Imaging, vol. 11, no. 3, pp.306-315, July 2002.

[25] J.E. Roddy, R.J. Zolla, N.A. Blish, and L. Horvath. Four color image sens-ing apparatus. U.S. Patent 7 057 654Jun. 2006.

[26] T. Sakamoto, C. Nakanishi, and T. Hase. Software pixel interpolation fordigital still cameras suitable for a 32-bit MCU. IEEE Trans. Consum. Elec-tron., vol. 44, no. 4, pp. 1342-1352, Nov. 1998.

[27] Sony Press Release. [Online]. 2003. Available at:http://www.sony.net/SonyInfo/News/Press/200307/03-029E/.

[28] W. Takashi, and M. Shigehiro. Color imaging array and color imaging de-vice. U.S. Patent 4 500 914, Feb. 1985.

[29] M. Vrhel, E. Saber, and H.J. Trussell. Color image generation and displaytechnologies. IEEE Signal Process. Mag., vol. 22, no. 1, pp. 23-33, Jan. 2005.

[30] T. Yamagami, T. Sasaki, and A. Suga. Image signal processing apparatushaving a color filter with offset luminance filter elements. U.S. Patent 5 323233, June 1994.

[31] S. Yamanaka. Solid state color camera. U.S. Patent 4 054 906, 1977.

[32] W. Zhu, K. Parker, and M.A. Kriss. Color filter arrays based on mutuallyexclusive blue noise patterns. J. Vis. Commun. Image Represent., vol. 10,pp. 245-247, 1999.

17