theoretical foundations of transform coding · theoretical foundations of transform coding vivek k...

Theoretical Foundations ofTransform Coding

Vivek K Goyal

Divide and conquer. This principle is central tomany endeavors, ranging from childrenmanaging inconsistent parents to colonialpowers controlling native peoples. In engi-

neering and computational science it means breaking a bigproblem into smaller problems that can be more easily un-derstood and solved. Putting the pieces back togethergives a modular design, which is advantageous for imple-mentation, testing, and component reuse.

In signal processing, we are familiar with the divide andconquer strategy because its recursive application is the es-sence of fast Fourier transform (FFT) algorithms. Recur-sive division leads also to efficient algorithms forsearching, sorting, convolutions, and eigenvalue compu-tations. Thus, in both recursive and nonrecursive forms,this strategy is fundamental.

Everyday compression problems are unmanageablewithout a divide and conquer approach.Effective compression of images, for ex-ample, depends on the tendencies ofpixels to be similar to their neighbors orto differ in partially predictable ways.These tendencies, arising from the con-tinuity, texturing, and boundaries ofobjects, the similarity of objects in animage, gradual lighting changes, an art-ist’s technique and color palette, etc.,may extend over an entire image with aquarter million pixels. Yet the mostgeneral way to utilize the probable rela-tionships between pixels (later de-scribed as unconstrained sourcecoding) is infeasible for this many pix-els. In fact, 16 pixels is a lot for an un-constrained source code.

To conquer the compression prob-lem—allowing, for example, more than16 pixels to be encoded simultaneously—state-of-the-artlossy compressors divide the encoding operation into a se-quence of three relatively simple steps: the computation of alinear transformation of the data designed primarily to pro-duce uncorrelated coefficients, separate quantization of eachscalar coefficient, and entropy coding. This process is calledtransform coding. In image compression, a square image

with N pixels is typically processed with simple lineartransforms (often discrete wavelet transforms) of size

N .This article explains the fundamental principles of

transform coding; these principles apply equally well toimages, audio, video, and various other types of data, soabstract formulations are given. Much of the material pre-sented here is adapted from [14, Chap. 2, 4]. The detailson wavelet transform-based image compression and theJPEG2000 image compression standard are given in thefollowing two articles of this special issue [38], [37].

Source CodingSource coding is to represent information in bits, with thenatural aim of using a small number of bits. When the in-formation can be exactly recovered from the bits, the

source coding or compression is calledlossless; otherwise, it is called lossy. Thetransform codes in this article are lossy.However, lossless entropy codes appearas components of transform codes, soboth lossless and lossy compression areof present interest.

In our discussion, the “information”is denoted by a real column vectorx N∈R or a sequence of such vectors. Avector might be formed from pixel val-ues in an image or by sampling an audiosignal; K N⋅ pixels can be arranged as asequence of K vectors of length N. Thevector length N is defined such thateach vector in a sequence is encoded in-dependently. For the purpose of build-ing a mathematical theory, the sourcevectors are assumed to be realizations ofa random vector x with a known distri-

bution. The distribution could be purely empirical.A source code is comprised of two mappings: an en-

coder and a decoder. The encoder maps any vector x N∈Rto a finite string of bits, and the decoder maps any of thesestrings of bits to an approximation �x N∈R . The encodermapping can always be factored as γ α� , whereα is a map-ping from R N to some discrete set I and γ is an invertible

SEPTEMBER 2001 IEEE SIGNAL PROCESSING MAGAZINE 91053-5888/01/$10.00©2001IEEE

©20

01IM

AG

ES

TA

TE

mapping from I to strings of bits. The former is called alossy encoder and the latter a lossless code or an entropycode. The decoder inverts γ and then approximates xfrom the index α( )x ∈I. This is shown in the top half ofFig. 1. It is assumed that communication between the en-coder and decoder is perfect. (The last article of this issue[13] describes techniques that work when some transmit-ted bits are lost.)

To assess the quality of a lossy source code, we neednumerical measures of approximation accuracy and de-scription length. The measure for description length issimply the expected number of bits output by the encoderdivided by N; this is called the rate in bits per scalar sam-ple and denoted by R. Here we will measure approxima-tion accuracy by squared Euclidean norm divided by thevector length

( ) ( )d x xN

x xN

x xi ii

N

, � � �= − = −=∑1 12 2

1

.

This accuracy measure is conventional and usually leadsto the easiest mathematical results, though the theory ofsource coding has been developed with quite generalmeasures [1]. The expected value of d( �)x,x is called themean-squared error (MSE) distortion and is denoted byD E d= [ ( �)]x,x . The normalizations by N make it possibleto fairly compare source codes with different lengths.

Fixing N, a theoretical concept of optimality isstraightforward: A length-N source code is optimal if noother length-N source code with at most the same ratehas lower distortion. This concept is of dubious value.First, it is very difficult to check the optimality of a sourcecode. Local optimality—being assured that small pertur-bations ofα andβ will not improve performance—is oftenthe best that can be attained [16]. Second, and of morepractical consequence, a system designer gets to choosethe value of N. It can be as large as the total size of the dataset—like the number of pixels in an image—but can alsobe smaller, in which case the data set is interpreted as a se-quence of vectors.

There are conflicting motives in choosing N. Com-pression performance is related to the predictability ofone part of x from the rest. Since predictability can onlyincrease from having more data, performance is usuallyimproved by increasing N. (Even if the random variablesproducing each scalar sample are mutually independent,the optimal performance is improved by increasing N;however, this “packing gain” effect is relatively small[16].) The conflict comes from the fact that the computa-tional complexity of encoding is also increased. This isparticularly dramatic if one looks at complexities of opti-mal source codes. The obvious way to implement an opti-mal encoder is to search through the entire codebook,giving running time exponential in N. Other implemen-tations reduce running time while increasing memory us-age [29].

State-of-the-art source codes result from an intelligentcompromise. There is no attempt to realize an optimalcode for a given value of N because encoding complexitywould force a small value for N. Rather, source codes thatare good, but plainly not optimal, are used. Their lowercomplexities make much larger N’s feasible. A tutorial inan earlier issue of this Magazine [5] called this “the powerof imperfection.” The paradoxical conclusion is that thebest codes to use in practice are suboptimal.

Constrained Source CodingTransform codes are the most usedsource codes because they are easy to ap-ply at any rate and even with very largevalues of N. The essence of transformcoding is the modularization shown inthe bottom half of Fig. 1. The mapping αis implemented in two steps. First, an in-vertible linear transform of the sourcevector x is computed, producing y Tx= .Each component of y is called a trans-form coefficient. The N transform coeffi-cients are then quantized independentlyof each other by N scalar quantizers. Thisis called scalar quantization since eachscalar component of y is treated sepa-rately. Finally, the quantizer indexes thatcorrespond to the transform coefficients

10 IEEE SIGNAL PROCESSING MAGAZINE SEPTEMBER 2001

αx

x T

i γ

γ

bits

γ−1

γ−1 i βx̂

x̂

y1 i1 i1α1 β1

y2 i2 i2α2 β2

yN iN iNαN βN

U

y1^

y2^

yN^

� 1. Any source code can be decomposed so that the encoder is γ α� and the de-coder is β γ� −1, as shown at top. γ is an entropy code and α and β are the encoderand decoder of an N-dimensional quantizer. In a transform code, α and β eachhave a particular constrained structure. In the encoder, α is replaced with a lineartransform T and a set of N scalar quantizer encoders. The intermediate y i s arecalled transform coefficients. In the decoder, β is replaced with N scalar quantizerdecoders and another linear transform U. Usually U T= −1.

The standard theoretical modelfor transform coding has strictmodularity , meaning that thetransform, quantization, andentropy coding blocks operateindependently.

are compressed with an entropy code to produce the se-quence of bits that represent the data.

To reconstruct an approximation of x, the decoder es-sentially reverses the steps of the encoder. The action ofthe entropy coder can be inverted to recover the quantizerindices. Then the decoders of the scalar quantizers pro-duce a vector �y of estimates of the transform coefficients.To complete the reconstruction, a linear transform is ap-plied to �y to produce the approximation �x. This final stepusually uses the transform T −1 , but for generality thetransform is denoted U.

Most source codes cannot be implemented in the twostages of linear transform and scalar quantization. Thus, atransform code is an example of a constrained sourcecode. Constrained source codes are, loosely speaking,source codes that are suboptimal but have low complex-ity. The simplicity of transform coding allows large valuesof N to be practical. Computing the transformT requiresat most N 2 multiplications and N N( )−1 additions. Spe-cially structured transforms—like discrete Fourier, co-sine, and wavelet transforms—are often used to reducethe complexity of this step, but this is merely icing on thecake. The great reduction from the exponential complex-ity of a general source code to the (at most) quadraticcomplexity of a transform code comes from using lineartransforms and scalar quantization. See Box 1 for moreon constrained source codes.

The Standard Model and Its ComponentsThe standard theoretical model for transform coding lookslike the bottom of Fig. 1. It has the strict modularity shown,meaning that the transform, quantization and entropy cod-ing blocks operate independently. In addition, the entropycoder can be decomposed into N parallel entropy coders sothat the quantization and entropy coding operate independ-ently on each scalar transform coefficient.

This section briefly describes the fundamentals of en-tropy coding and quantization to provide background forour later focus on the optimization of the transform. Thefinal part of this section addresses the allocation of bitsamong the N scalar quantizers. Sources for additional in-formation include [3], [8], [14], [16].

Entropy CodesEntropy codes are used for lossless coding of discrete ran-dom variables. Consider the discrete random variable zwith alphabet I. An entropy code γ assigns a unique bi-nary string, called a codeword, to each i ∈I. (See Fig. 1.)

Since the codewords are unique, an entropy code is al-ways invertible. However, we will place more restrictiveconditions on entropy codes so they can be used on se-quences of realizations of z. The extension of γ maps thefinite sequence ( , , , )z z zk1 2 … to the concatenation of theoutputs of γ with each input, γ γ γ( ) ( ), , ( )z z zk1 2 … . A codeis called uniquely decodable if its extension is one-to-one.A uniquely decodable code can be applied to message se-

SEPTEMBER 2001 IEEE SIGNAL PROCESSING MAGAZINE 11

Box 1Constrained and Unconstrained Source Codes

The difference between constrained and unconstrainedsource codes is shown by partition diagrams. In these di-

agrams, the cells indicate which source vectors are encodedto the same index and the dots are the reconstructions com-puted by the decoder. Four locally optimal fixed-rate sourcecodes with 12-element codebooks were constructed. Thetwo-dimensional, jointly Gaussian source is the same as thatused in Fig. 2 and Box 6.

The upper-left plot shows an unconstrained code. Thepartition shares the symmetries of the source density but isotherwise complicated because the cell shapes are arbitrary.Encoding is difficult because there is no simple way to getaround using both components of the source vector simulta-neously in computing the index.

Encoding with a transform code is easier because after thelinear transform the coefficients are quantized separately.This gives the structured alignment of partition cells and ofreconstruction points in the upper-right plot.

It is fair to ask why the transform is linear. In two dimen-sions, one might imagine quantizing in polar coordinates.Two examples of partitions obtained with separatequantization of radial and angular components are shown inthe lower plots, and these are as elegant as the partition ob-tained with a linear transform. Yet nonlinear transforma-tions—even transformations to polar coordinates—arerarely used in source coding. With arbitrary transformations,the approximation accuracy of the transform coefficientsdoes not easily relate to the accuracy of the reconstructedvectors. This makes designing quantizers for the transformcoefficients more difficult. Also, allowing nonlinear transfor-mations reintroduces the design and encoding complexitiesof unconstrained source codes.

Constrained source codes need not use transforms tohave low complexity. Techniques described in [8] and [16]include those based on lattices, sorting, and tree-structuredsearching; none of these techniques is as popular as trans-form coding.

D = 0.055

D = 0.116

D = 0.066

D = 0.069

quences without adding any “punctuation” to showwhere one codeword ends and the next begins. In a prefixcode, no codeword is the prefix of any other codeword.Prefix codes are guaranteed to be uniquely decodable.

A trivial code numbers each element of I with a dis-tinct index in { , , ,| | }0 1 1… I − and maps each element tothe binary expansion of its index. Such a code requires log | |2 I bits per symbol. This is considered the lack ofan entropy code. The idea in entropy code design is tominimize the mean number of bits used to represent z atthe expense of making the worst-case performance worse.The expected code length is given by

L E p i ii

( ) [ ( ( ))] ( ) ( ( ))γ γ γ= =∈∑� �z zI

,

where p iz ( ) is the probability of symbol i and �( ( ))γ i is thelength of γ( )i . The expected length can be reduced if shortcode words are used for the most probable sym-bols—even if it means that some symbols will havecodewords with more than log | |2 I bits.

The entropy code γ is called optimal if it is a prefix codethat minimizes L( )γ . Huffman codes, described in Box 2,are examples of optimal codes. The performance of an op-timal code is bounded by

H L H( ) ( ) ( )z z≤ < +γ 1 (1)

where

H p i p ii

( ) ( )log ( )z z z= −∈∑I

2 (2)

is the entropy of z.The up to one bit gap in (1) is ignored in the remainder

of the article. If H( )z is large, this is justified simply be-cause one bit is small compared to the code length. Other-wise note that L H( ) ( )γ ≈ z can attained by coding blocksof symbols together; this is detailed in any informationtheory or data compression textbook.

QuantizersA quantizer q is a mapping from a source alphabet R N toa reproduction codebook C I= ⊂∈{ � }xi i

NR , where I isan arbitrary countable index set. Quantization can be de-composed into two operations q = β α� , as shown in Fig.1. The lossy encoder α:R N → I is specified by a partitionofR N into partition cellsS x x ii

N= ∈ ={ | ( ) }R α , i ∈I. Thereproduction decoder β:I→R N is specified by thecodebook C. If N =1, the quantizer is called a scalarquantizer; for N >1, it is a vector quantizer.

The quality of a quantizer is determined by its distor-tion and rate. The MSE distortion for quantizing randomvector x ∈R N is

[ ]D E d q N E q= = −−[ ( , ( ))] ( ) .x x x x1 2

The rate can be measured in a few ways. The lossy en-coder output α( )x is a discrete random variable that usu-ally should be entropy coded because the output symbolswill have unequal probabilities. Associating an entropycode γ to the quantizer gives a variable-rate quantizerspecified by ( , , )α β γ . The rate of the quantizer is the ex-pected code length of γ divided by N. Not specifying anentropy code (or specifying the use of fixed-rate binaryexpansion) gives a fixed-rate quantizer with rateR N= −1

2log | |I . Measuring the rate by the idealized per-formance of an entropy code gives R N H= −1 ( ( ))α x ; thequantizer in this case is called entropy constrained.

The optimal performance of variable-rate quantizationis at least as good as that of fixed-rate quantization, andentropy-constrained quantization is better yet. Entropycoding adds complexity, however, and variable lengthoutput can create difficulties such as buffer overflows.


Box 2Huffman Codes

There is a simple algorithm, due to Huffman [22], forconstructing optimal entropy codes. One starts with

a graph with one node for each symbol and no edges.These nodes will become the leaves of a tree as edges areadded to make a connected graph.

At each step of the algorithm, the probabilities of thedisconnected sets of nodes are sorted and the two leastprobable sets are merged through the addition of a parentnode and edges to each of the two sets. The edges are as-signed labels of 0 and 1. When a tree has been formed,codewords are assigned to each leaf node by concatenat-ing the edge labels on the path from the root to the leaf.

Shown below is a Huffman code tree for symbols {1,2, 3, 4, 5, 6} with respective probabilities {0.3, 0.26,0.14, 0.13, 0.09, 0.08}. The codewords are boxed. Com-puting a weighted sum of the codeword lengths gives theexpected code length

L = ⋅ + ⋅ + ⋅ + ⋅ + ⋅ + ⋅=

03 2 0 26 2 014 3 013 3 009 3 008 32 44. . . . . .. bits.

This is quite close to the entropy of 2.41 bits obtained byevaluating (2).

0

1

0

1

0

1

1

0

1

3

4

2

5

6

0

1

γ(1) = 00

γ(3) = 010

γ(4) = 011

γ(2) = 10

γ(5) = 110

γ(6) = 111

Furthermore, entropy-constrained quantization is onlyan idealization since an entropy code will generally notmeet the lower bound in (1).

Optimal QuantizationAn optimal quantizer is one that minimizes the distortionsubject to an upper bound on the rate or minimizes therate subject to an upper bound on the distortion. Becauseof simple shifting and scaling properties, an optimalquantizer for a scalar x can be easily deduced from an opti-mal quantizer for the normalized random variablew x x x= −( ) /µ σ , whereµ x andσ x are the mean and stan-dard deviation of x, respectively. One consequence of thisis that optimal quantizers have performance

D g R= σ 2 ( ), (3)

whereσ 2 is the variance of the source and g R( )is the per-formance of optimal quantizers for the normalizedsource. Equation (3) holds, with a different function g,for any family of quantizers that can be described by itsoperation on a normalized variable, not just optimalquantizers.

Optimal quantizers are difficult to design, but locallyoptimal quantizers can be numerically approximated byan iteration in which α, β, and γ are separately optimized,in turn, while keeping the other two fixed. For details oneach of these optimizations and the difficulties and prop-erties that arise, see [5] and [16].

Note that the rate measure affects the optimal encod-ing rule becauseα( )x should be the index that minimizes aLagrangian cost function including both rate and distor-tion; for example

α γ λ β( ) ( ( )) ( )xN

iN

x ii

= + −

∈

argminI

1 1 2�

is an optimal lossy encoder for variable-rate quantization.(By fixing the relative importance of rate and distortion,the Lagrange multiplier λ determines a rate-distortionoperating point among those possible with the given βand γ.) Only for fixed-rate quantization does the optimalencoding rule simplify to finding the index correspond-ing to the nearest codeword.

In some of the more technical discussions that follow,one property of optimal decoding is relevant: The opti-mal decoder β computes

β( ) [ | ]i E Si= ∈x x ,

which is called centroid reconstruction. The conditionalmean of the cell, or centroid, is the minimum MSE esti-mate [31].

High Resolution QuantizationFor most sources, it is impossible to analytically expressthe performance of optimal quantizers. Thus, aside from


Box 3Uniform Quantization

The design of quantizers has a deep and fascinating the-ory. Nevertheless, the fact remains [16]: “Most

quantizers today are indeed uniform and scalar, but are com-bined with prediction or transforms.”

Though definitions of uniform quantization vary some-what, the archetype of rounding is always an example of uni-form quantization. Shown in (a) is the input-outputrelationship of a device that rounds to the nearest integermultiple of step size∆. To describe this in formal notation, theencoder could be α( ) ( / )x x= round ∆ , where round()⋅ de-notes rounding to the nearest integer, with correspondingdecoder β( )i i= ∆.

The other common uniform quantizer is shown in (b).This is a shifted version of the previous uniform quantizer.Variations in the definition of uniform quantization some-times allow only the encoder to have equal length cells oronly the decoder to have evenly spaced outputs and may alsoallow the decoder outputs to be shifted from the centers ofthe partition cells.

Uniform quantization of a uniform source provides a set-ting to see the typical trade-off between rate and distortion.Consider x uniformly distributed on the interval [ , )0 1 . Afixed-rate uniform quantizer, as on the right above, with Kcells and step size ∆ =1/ K quantizes x m m∈ −[( ) , )1 ∆ ∆ to( / )m −1 2 ∆ for m K=1 2, , ,… . I t has rateR N= = −log log2 2 ∆ and distortion

D x x dx x m dm

m

m

N

= − = − −

∫ ∫∑ −

=

( �)( )

2

0

1

11

212

∆∆

∆x = 1

122∆ .

Thus, D R= −( / )1 12 2 2 . When using the MSE distortion mea-sure, there will almost always be a 2 2− R factor.

− − ∆52

− ∆12

− ∆12

−− ∆12

− ∆32

− ∆32

−− ∆32

− ∆52

− ∆52

−− ∆52

q x( )

q x( )

3∆

3∆

−3∆

−3∆

2∆

2∆

−2∆

−2∆

∆

∆

− ∆

− ∆

x

x

32− − ∆ 1

2− − ∆

(a)

(b)

using (3), approximations must suffice. Fortunately, ap-proximations obtained when it is assumed that thequantization is very fine are reasonably accurate even atlow to moderate rates [10], [40]. Details on this “highresolution” theory for both scalars and vectors can befound in [7] and [16] and references therein.

Let f xx ( ) denote the probability density function(pdf) of the scalar random variable x. High resolutionanalysis is based on approximating f xx ( )on the intervalSiby its value at the midpoint. Assuming f xx ( ) is smooth,this approximation is accurate when each Si is short.

Optimization of scalar quantizers turns into findingthe optimal lengths for the Si s, depending on the pdff xx ( ). One can show that the performance of optimalfixed-rate quantization is approximately

( )D f x dx R≈ ∫ −112

21 3 3 2x

/ ( )�

.(4)

Evaluating this for a Gaussian source with variance σ 2

gives

D R≈ −32

22 2π σ .(5)

For entropy-constrained quantization, high resolutionanalysis shows that it is optimal for each Si to have equallength [9]. A quantizer that partitions with equal-lengthintervals is called uniform (see Box 3). The resulting per-formance is

D h R≈ −112

2 22 2( )x ,(6)

where

h f x f x dx( ) ( )log ( )x x x= −∫� 2

is the differential entropy of x. For Gaussian random vari-ables, (6) simplifies to

D e R≈ −π σ6

22 2 .(7)

Summarizing (4)-(7), the lesson from high resolutionquantization theory is that quantizer performance is de-scribed by

D c R≈ −σ 2 22 , (8)

where σ 2 is the variance of the source and c is a constantthat depends on the normalized density of the source andthe type of quantization (fixed rate, variable rate, or en-tropy constrained). This is consistent with (3).

The computations we have made are for scalarquantization. For vector quantization, the best perfor-mance in the limit as the dimension N grows is given bythe distortion rate function [1] (see [13, Box 3]). For aGaussian source this bound is D R= −σ 2 22 . The approxi-mate performance given by (7) is only worse by a factorof πe / 6 (≈153. dB). This can be expressed as a redun-dancy ( / )log ( / ) .1 2 6 02552 πe ≈ bits. Furthermore, a nu-merical study has shown that for a wide range ofmemoryless sources, the redundancy of entropy-con-strained uniform quantization is at most 0.3 bits per sam-ple at all rates [6].

Bit AllocationCoding (quantizing and entropy coding) each transformcoefficient separately splits the total number of bitsamong the transform coefficients in some manner.Whether done with conscious effort or implicitly, this is abit allocation among the components.

Bit allocation problems can be stated in a single com-mon form: One is given a set of quantizers described bytheir distortion-rate performances as

D g R R i Ni i i i i= ∈ =( ), , , , ,R 1 2 … .

Each set of available ratesRi is a subset of the nonnegativereal numbers and may be discrete or continuous. The prob-lem is to minimize the average distortion D N Dii

N= −=∑11

given a maximum average rate R N R ii

N= −=∑11

.

As is often the case with optimization problems, bit al-location is easy when the parameters are continuous andthe objective functions are smooth. Subject to a few othertechnical requirements, parametric expressions for theoptimal bit allocation can be found [27], [35]. The tech-niques used when the Ri s are discrete are quite differentand play no role in forthcoming results [36].

If the average distortion can be reduced by taking bitsaway from one component and giving them to another,the initial bit allocation is not optimal. Applying this rea-soning with infinitesimal changes in the component rates,a necessary condition for an optimal allocation is that theslope of each gi at R i is equal to a common constantvalue. A tutorial treatment of this type of optimizationappeared in an earlier issue of this Magazine [30].

The approximate performance given by (8) leads to aparticularly easy bit allocation problem with

g c i Ni i iR

ii= = ∞ =−σ 2 22 0 1 2, [ , ), , , ,R … . (9)

Ignoring the fact that each component rate must benonnegative, an equal-slope argument shows that the op-timal bit allocation is


The motivating principle oftransform coding is that simplecoding may be more effective inthe transform domain than in theoriginal signal space.

( ) ( )R R

c

ci

i

ii

N Ni

ii

N N= + +

= =∏ ∏12

122

1

1 2

2

21

1log log

/ /

σ

σ.

With these rates, all the Di s are equal and the average dis-tortion is

( ) ( )D cii

N N

ii

N NR=

= =−∏ ∏1

12

1

122

/ /σ .

(10)

This solution is valid when each R i given above isnonnegative. For lower rates, the components withsmallest ci i⋅σ 2 are allocated no bits and the remainingcomponents have correspondingly higher allocations.

Bit Allocation with Uniform QuantizersWith uniform quantizers, bit allocation is nothing morethan choosing a step size ∆ i for each of the N compo-nents. The equal-distortion property of the analytical bitallocation solution gives a simple rule: Make all of the stepsizes equal. This will be referred to as“lazy” bit allocation.

Our development indicates that lazyallocation is optimal when the rate ishigh. Numerical studies have shown thatlazy allocation is nearly optimal as longas the minimal allocated rate is at least 1bit [14], [15]. Entropy-constrained uni-form quantization with lazy bit alloca-tion is used in the numerical examples inthe following section.

Optimal TransformsIt has taken some time to set the stage,but we are now ready for the main eventof designing the analysis transform Tand the synthesis transformU. Through-out this section the source x is assumedto have mean zero, and R x denotes thecovariance matrix E T[ ]xx , where T de-notes the transpose. The source is often,but not always, jointly Gaussian.

A signal given as a vector inR N is im-plicitly represented as a series with re-spect to the standard basis. An invertibleanalysis transformT changes the basis. Achange of basis does not alter the infor-mation in a signal, so how can it affectcoding efficiency? Indeed, if arbitrarysource coding is allowed after the trans-form, it does not. The motivating princi-ple of transform coding is that simplecoding may be more effective in thetransform domain than in the originalsignal space. In the standard model,

“simple coding” corresponds to the use of scalarquantization and scalar entropy coding.

Visualizing TransformsBeyond two or three dimensions, it is difficult to visualizevectors—let alone the action of a transform on vectors.Fortunately, most people already have an idea of what alinear transform does: it combines rotating, scaling, andshearing such that a hypercube is always mapped to aparallelepiped.

In two dimensions, the level curves of a zero-meanGaussian density are ellipses centered at the origin withcollinear major axes, as shown in the left panels of Fig. 2.The middle panel of Fig. 2(a) shows the level curves of thejoint density of the transform coefficients after a more orless arbitrary invertible linear transformation. A lineartransformation of an ellipse is still an ellipse, though itseccentricity and orientation (direction of major axis) mayhave changed.


� 2. Illustration of various basis changes. The source is depicted by level curves of thepdf (left). The transform coefficients are separately quantized with uniformquantizers (center). The induced partitioning is then shown in the original coordi-nates (right). (a) A basis change generally induces a nonhypercubic partition. (b)A singular transformation gives a partition with unboundedn cells. (c) A Karhunen-Loève transform is an orthogonal transform that aligns the partitioning with theaxes of the source pdf.

The grid in the middle panel indicates the cell bound-aries in uniform scalar quantization, with equal step sizes,of the transform coefficients. The effect of inverting thetransform is shown in the right panel; the source densityis returned to its original form and the quantization parti-tion is linearly deformed. The partition in the original co-ordinates, as shown in the right panel, is what is trulyrelevant. It shows which source vectors are mapped to thesame symbol, thus giving some indication of the averagedistortion. Looking at the number of cells with apprecia-ble probability gives some indication of the rate.

A singular transform is a degenerate case. As shown inthe middle panel of Fig. 2(b), the transform coefficientshave probability mass only along a line. (A line segment is

an ellipse with unit eccentricity.) Inverting the transformis not possible, but we may still return to the original co-ordinates to view the partition induced by quantizing thetransform coefficients. The cells are unbounded in one di-rection, as shown in the right panel. This is undesirableunless variation of the source in the direction in which thecells are unbounded is very small.

Although better than unbounded cells, the parallelo-gram-shaped partition cells that arise from arbitrary in-vertible transforms are inherently suboptimal (see Box 4).To get rectangular partition cells, the basis vectors mustbe orthogonal. For square cells, when quantization stepsizes are equal for each transform coefficient, the basisvectors should in addition to being orthogonal have equal


Box 4Shapes of Partition Cells

The quality of a source code depends on the shapes ofthe partition cells { ( )α −1 i , i ∈I} and on varying the

sizes of the cells according to the source density. Whenthe rate is high, and either the source is uniformly distrib-uted or the rate is measured by entropy (H( ( ))α x ), thesizes of the cells should essentially not vary. Then, thequality depends on having cell shapes that minimize theaverage distance to the center of the cell.

For a given volume, a body in Euclidean space thatminimizes the average distance to the center is a sphere.But spheres do not work as partition cell shapes becausethey do not pack together without leaving interstices.Only for a few dimensions N is the best cell shape known[2]. One such dimension is N = 2, where the hexagonalpacking shown below (left) is best.

The best packings (including the hexagonal case) cannotbe achieved with transform codes. Transform codes can onlyproduce partitions into parallelepipeds, as shown for N = 2in Fig. 2. The best parallelepipeds are cubes. We get a hint ofthis by comparing the two rectangular partitions of aunit-area square shown below. Both partitions have 36 cells,so every cell has the same area. The partition with square cellsgives distortion 1 432 231 10 3/ .≈ × − , while the other gives97 31104 312 10 3/ .≈ × − . (The calculations are easy; see Box3.)

This simple example can also be interpreted as a prob-lem of allocating bits between the horizontal and verticalcomponents. The “lazy” bit allocation arising from equalquantization step sizes for each component is optimal.This holds generally for high-rate entropy-constrainedquantization of components with the same normalizeddensity.

Box 5Autoregressive Source

Autoregressive models are popular in many branchesof signal processing. Under such a model, a sequence

is generated as

x x z[ ] [ ] [ ]k k k= − +ρ 1

where k is a time index, z[ ]k is a white sequence, andρ ∈[ , )0 1 is called the correlation coefficient. It is a crude butuseful model for the samples along any line of a grayscaleimage, with ρ ≈0 9. .

An �N -valued source x can be derived from a scalarautoregressive source by forming blocks of N consecutivesamples. With normalized power, the covariance matrixis given elementwise by ( ) | |R ij

i jx = −ρ . For any particular

N andρ, a numerical eigendecomposition method can beapplied to Rx to obtain a KLT. (An analytical solutionhappens to also be possible [17], [34].) For N =8 andρ =0 9. , the KLT can be depicted as below:

Each subplot (value of k) gives a row of the transformor, equivalently, a vector in the analysis basis. This basis issuperficially sinusoidal and is approximated by a discretecosine transform (DCT) basis. There are various asymp-totic equivalences between KLTs and DCTs for large Nandρ→ 1 [23], [33]. These results are often cited in justi-fying the use of DCTs.

k = 1

k = 2

k = 3

k = 4

k = 5

k = 6

k = 7

k = 81 2 3 4 5 6 7 8

m

lengths. When orthogonal basis vectors have unit length,the resulting transform is called an orthogonal transform.(It is regrettable that of a matrix or transform, “orthogo-nal” means orthonormal.)

A Karhunen-Loève transform (KLT) is a particulartype of orthogonal transform that depends on thecovariance of the source. An orthogonal matrix T repre-sents a KLT of x if TR T T

x is a diagonal matrix. The diag-onal matrix TR T T

x is the covariance of y x=T ; thus, aKLT gives uncorrelated transform coefficients. KLT isthe most commonly used name for these transforms insignal processing, communication, and information the-ory, recognizing the works [24] and [26]; among theother names are Hotelling transforms [19] and principalcomponent transforms.

A KLT exists for any source because covariance matri-ces are symmetric, and symmetric matrices are orthogo-nally diagonalizable; the diagonal elements of TR T T

x

are the eigenvalues of R x . KLT’s are not unique: any rowof T can be multiplied by ±1 without changing TR T T

x ,and permuting the rows leaves TR T T

x diagonal. If theeigenvalues of R x are not distinct, there is additional free-dom in choosing a KLT. An example of a KLT is given inBox 5.

For Gaussian sources, KLTs align the partitioningwith the axes of the source pdf, as shown in Fig. 2(c). Itappears that the forward and inverse transforms are rota-tions, though actually the symmetry of the source densityobscures possible reflections.

The Easiest Transform OptimizationConsider a jointly Gaussian source, and assume U and Tare orthogonal and U T= −1 . The Gaussian assumption isimportant because any linear combination of jointlyGaussian random variables is Gaussian. Thus, any analy-sis transform gives Gaussian transform coefficients.Then, since the transform coefficients have the same nor-malized density, for any reasonable set of quantizers, (3)holds with a single function g R( ) describing all of thetransform coefficients. Orthogonality is important be-cause orthogonal transforms preserve Euclidean lengths,which gives d x x d y y( , �) ( , �)= .

With these assumptions, for any rate and bit allocationa KLT is an optimal transform.

Theorem 1 ([11], [15]): Consider a transform coderwith orthogonal analysis transformT and synthesis trans-form U T T T= =−1 . Suppose there is a single function gto describe the quantization of each transform coefficientthrough

[ ]E g R i Ni i i i( � ) ( ), , , ,y y− = =2 2 1 2σ … ,


Box 6Optimizing the Synthesis Transform U

From a glance at Fig. 1, it is natural to assume that thesynthesis transform should be U T= −1. Theorem 3

provides sufficient conditions for this to be optimal, butthese are not always satisfied.

As an example, consider a two-dimensional,zero-mean, jointly Gaussian source with

Rx =

116

13 3 33 3 7

.

Level curves of the pdf are shown with dotted ellipses.Rotating by 30° would give independent transform coef-ficients, but suppose no transform (or the identity trans-form) is used. The conditions of Theorem 3 are notsatisfied: Even if the β i ’s (scalar quantizer decoders) areoptimal, a synthesis transform other than T I− =1 may beoptimal.

Withα i ’s (scalar quantizer encoders) that are uniformwith step size ∆ =1, the partitioning is as shown.

The outputs of the scalar quantizers (•) are not at thecenters of the cells, but they still form a separablecodebook; the cell centroids (∗) are not separable. Theoptimal synthesis transform

U =

0 970 00890083 0 944. .. .

gives a better codebook (�) and reduces the distortion by4.2%.

Transform codes are easy toimplement because of a divideand conquer strategy; the trans-form exploits dependencies inthe data so that the quantizationand entropy coding can besimple.

where σ i2 is the variance of y i and R i is the rate allocated

to y i . Then for any bit allocation( , , , )R R R N1 2 … there isa KLT that minimizes the distortion. In the typicalcase where g is nonincreasing, a KLT that gives

( , , , )σ σ σ12

22 2… N sorted in the same order as the bit allo-

cation minimizes the distortion.Since it holds for any bit allocation and many families

of quantizers, Theorem 1 is stronger than several earliertransform optimization results. In particular, it subsumesthe low-rate results of Lyons [28] and the high-rate re-sults that are reviewed presently.

Recall that with a high average rate of R bits per com-ponent and quantizer performance described by (9), theaverage distortion with optimal bit allocation is given by(10). With Gaussian transform coefficients that are opti-mally quantized, the distortion simplifies to

( )D c ii

N NR=

=−∏ σ 2

1

122

/,

(11)

where c e= π / 6 for entropy-constrained quantization orc = 3 2π / for fixed-rate quantization. The choice of anorthogonal transform is thus guided by minimizing thegeometric mean of the transform coefficient variances.

Theorem 2: The distortion given by (11) is minimizedover all orthogonal transforms by any KLT.

Proof: Applying Hadamard’s Inequality [18, 7.8.1] toR y gives

( )( )( )det det det detT R T RTi

i

N

x y= ≤=

∏σ 2

1

.

Since det T =1, the left-hand side of this inequality is in-variant to the choice of T . Equality is achieved when aKLT is used. Thus KLTs minimize the distortion.

Equation (11) can be used to define a figure of meritcalled the coding gain. The coding gain of a transform is afunction of its variance vector, ( , , , )σ σ σ1

222 2… N , and

the variance vector without a transform, diag( )R x :

( )( )

coding gain = =

=

∏∏

( )/

/

R xi

N

ii

N

ii

N N

1

1

21

1σ

.

The coding gain is the factor by which the distortion is re-duced because of the transform, assuming high rate andoptimal bit allocation. The foregoing discussion showsthat KLTs maximize coding gain. Related measures arethe variance distribution, maximum reducible bits, andenergy packing efficiency or energy compaction. All ofthese are optimized by KLTs [33].

More General ResultsThe results of the previous section are straightforward,and Theorem 2 is well known. However, KLTs are not al-ways optimal . With some sources, there arenonorthogonal transforms that perform better than anyorthogonal transform. And, depending on thequantization, U T= −1 is not always optimal, even forGaussian sources. This section provides results that apply


Box 7Orthogonality versus Independence

Two qualitative intuitions arise from high resolutiontransform coding theory: a) try to have independ-

ent transform coefficients; and b) use orthogonal trans-forms. Sometimes you can only have one or the other.

Suppose the source x is uniformly distributed on therhombus shown below left. For a ≠ 0, there is no or-thogonal transform that gives independent transformcoefficients. The best orthogonal transform is a 45° ro-tation (right top) and the nonorthogonal transform

Ta

aa

=−

−−

11

112

gives independent transform coefficients (right bot-tom).

By computing differential entropies of the transformcoefficients and using (6), one can show that the opti-mal high-rate performance using the orthogonal trans-form is

D e a R= − −16

1 22 2( ) .

Computing the effect of nonsquare cells as in Box 4,one can show that the performance with thenonorthogonal transform is

D a R= + −13

1 22 2( ) .

For a e e( )/( )− +2 2 , it is better to use thenonorthogonal transform; otherwise, it is better to usethe KLT. The optimal transform is neither orthogonalnor produces independent transform coefficients.

The situation is yet more complicated if we do notuse high resolution theory. For a =0 and very low rates,the rotated coordinates are better than the original coor-dinates even though the components are independent inthe original coordinates [14].

OrthogonalTransform

IndependentTransform

Coefficients

1 + a

1 + a−1 + a

−1 + a

1 a−1 a−

−1 a−

− −1 a

√2(1 a)−

√2(1 a)−−√2(1 + a)

√2(1 a)−−

1

−1 1

−1

without the presumption of Gaussianity or orthogonalityand examples to show the limitations of these results.

The Synthesis Transform UInstead of assuming the decoder structure shown in thebottom of Fig. 1, let us consider for a moment the bestway to decode given only the encoding structure of atransform code. The analysis transform followed byquantization induces some partition of R N , and the bestdecoding is to associate with each partition cell its cen-troid. Generally, this decoder cannot be realized with alinear transform applied to �y. For one thing, some scalarquantizer decoderβ i could be designed in a plainly wrongway; then it would take an extraordinary (nonlinear) ef-fort to fix the estimates.

The difficulty is actually more dramatic because even ifthe β i s are optimal, the synthesis transform T −1 appliedto �y will generally not give optimal estimates. In fact, un-less the transform coefficients are independent, there maybe a linear transform better suited to the reconstructionthan T −1 .

Theorem 3 ([14]): In a transform coder with invertibleanalysis transform T, suppose the transform coefficientsare independent. If the component quantizers reconstructto centroids, then U T= −1 gives centroid reconstructionsfor the partition induced by the encoder. As a further con-sequence, T −1 is the optimal synthesis transform.

Examples where the lack of independence of transformcoefficients or the absence of optimal scalar decodingmakes T −1 a suboptimal synthesis transform are given in[14]. One of these examples is presented in Box 6.

The Analysis Transform TNow consider the optimization of T under the assump-tion that U T= −1 . The first result is a counterpart to The-orem 2. Instead of requiring orthogonal transforms andfinding uncorrelated transform coefficients to be best, itrequires independent transform coefficients and finds or-thogonal basis vectors to be best. It does not require aGaussian source; however, it is only for Gaussian sourcesthat R y being diagonal implies that the transform coeffi-cients are independent.

Theorem 4 ([14]): Consider a trans-form coder in which analysis transformTproduces independent transform coeffi-cients, the synthesis transform is T −1 ,and the component quantizers recon-struct to their respective centroids. Tominimize the MSE distortion, it is suffi-cient to consider transforms with or-thogonal rows, i.e., T such that TT T is adiagonal matrix.

The scaling of a row of T is generallyirrelevant because it can be completelyabsorbed in the quantizer for the corre-sponding transform coefficient. Thus,

Theorem 4 implies furthermore that it suffices to con-sider orthogonal transforms that produce independenttransform coefficients. Together with Theorem 1, it stillfalls short of showing that a KLT is necessarily an optimaltransform—even for a Gaussian source.

Heuristically, independence of transform coefficientsseems desirable because otherwise dependencies that wouldmake it easier to code the source are “wasted.” Orthogonalityis beneficial for having good partition cell shapes. A firm re-sult along these lines requires high resolution analysis.

Theorem 5 ([14]): Consider a high-rate transform codingsystem employing entropy-constrained uniformquantization. A transform with orthogonal rows that pro-duces independent transform coefficients is optimal, whensuch a transform exists. Furthermore, the norm of the ithrow divided by the ith quantizer step size is optimally a con-stant. Thus, normalizing the rows to have an orthogonaltransform and using equal quantizer step sizes is optimal.

For Gaussian sources there is always an orthogonaltransform that produces independent transform coeffi-cients—the KLT. For some other sources there are onlynonorthogonal transforms that give independent trans-form coefficients, but for most sources there is no lineartransform that does so. Through an example, Box 7 con-siders whether orthogonality or independent transformcoefficients is more important, when you have to choosebetween the two. There is no unequivocal answer.

More examples that explore the limitations of Theo-rem 5 are given in [14]. In particular, it is demonstratedthat even an orthogonal transform that produces inde-pendent transform coefficients is not necessarily optimalwhen the rate is low.

Departures from the Standard ModelPractical transform coders differ from the standard modelin many ways. This is discussed in detail in two articles ofthis Magazine [37], [38]. One particular change has sig-nificant implications for the relevance of the conventionalanalysis and has lead to new theoretical developments:Transform coefficients are often not entropy coded inde-pendently.


xT T −1

i1 i1γ1j1 j1α1 β1

i2 i2γ2j2 j2α2 β2

iN iNγNjN jNαN βN

γ11−

γ21−

γN−1

x^

� 3. An alternative transform coding structure introduced in [12]. The transform T op-erates on a vector of discrete quantizer indices instead of on the continuous-valuedsource vector, as in the standard model. Scalar entropy coding is explicitly indi-cated. For a Gaussian source, a further simplification with γ γ γ1 2= = ⋅⋅⋅ = N can bemade with no loss in performance.

Allowing transform coefficients to be entropy codedtogether, as it is drawn in Fig. 1, throws the theory intodisarray. Most significantly, it eliminates the incentive tohave independent transform coefficients. As for the par-ticulars of the theory, it also destroys the concept of bit al-location because bits are shared among transformcoefficients.

The status of the conventional theory is not quite sodire, however, because the complexity of unconstrainedjoint entropy coding of the transform coefficients is pro-hibitive. Assuming an alphabet size of K for each scalarcomponent, an entropy code for vectors of length N hasK N codewords. The problems of storing this codebookand searching for desired entries prevent large values of Nfrom being feasible. Entropy coding without explicitstorage of the codewords—as, for example, in arithmeticcoding—is also difficult because of the number of symbolprobabilities that must be known or estimated.

Analogous to constrained lossy source codes, joint en-tropy codes for transform coefficients are usually con-strained in some way. In the original JPEG standard [32],the joint coding is limited to transform coefficients withquantized values equal to zero. This type of joint codingdoes not eliminate the optimality of the KLT (for aGaussian source); in fact, it makes it even more importantfor a transform to give a large fraction of coefficients withsmall magnitude. The empirical fact that wavelet trans-forms have this property for natural images (more ab-stractly, for piecewise smooth functions) is a key to theircurrent popularity. Another article in this issue [39] dis-cusses this and related developments at length.

Returning to the choice of a transform assuming jointentropy coding, the high rate case gives an interesting re-sult: Quantizing in any orthogonal analysis basis and usinguniform quantizers with equal step sizes is optimal. All themeaningful work is done by the entropy coder. Given thatthe transform has no effect on the performance, it can beeliminated. There is still room for improvement, however.

Producing transform coefficients that are independentallows for the use of scalar entropy codes, with the attendantreduction in complexity, without any loss in performance. Atransform applied to the quantizer outputs, as shown in Fig.3, can be used to achieve or approximate this. For Gaussiansources, it is even possible to design the transform so thatthe quantized transform coefficients have approximately thesame distribution, in addition to being approximately inde-pendent. Then the same scalar entropy code can be appliedto each transform coefficient [12]. Some of the losslesscodes used in practice include transforms, but they are notoptimized for a single scalar entropy code.

Historical NotesTransform coding was invented as a method for conserv-ing bandwidth in the transmission of signals output bythe analysis unit of a ten-channel vocoder (“voice coder”)[4]. These correlated, continuous-time, continu-ous-amplitude signals represented estimates, local in

time, of the power in ten contiguous frequency bands. Byadding modulated versions of these power signals, thesynthesis unit resynthesized speech. The vocoder waspublicized through the demonstration of a related device,called the Voder, at the 1939 World’s Fair.

Kramer and Mathews [25] showed that the total band-width necessary to transmit the signals with a prescribedfidelity can be reduced by transmitting an appropriate setof linear combinations of the signals instead of the signalsthemselves. Assuming Gaussian signals, KLTs are opti-mal for this application.

The technique of Kramer and Mathews is not sourcecoding because it does not involve discretization. Thus,one could ascribe a later birth to transform coding.Huang and Schultheiss [20], [21] introduced the struc-ture shown in the bottom of Fig. 1 that we have referredto as the standard model. They studied the coding ofGaussian sources while assuming independent transformcoefficients and optimal fixed-rate scalar quantization.First they showed thatU T= −1 is optimal and then thatTshould have orthogonal rows. These results are subsumedby Theorems 3 and 4. They also obtained high-rate bit al-location results.

SummaryThe theory of source coding tells us that performance isbest when large blocks of data are used. But this same the-ory suggests codes that are too difficult to use—becauseof storage, running time, or both—if the block length islarge. Transform codes alleviate this dilemma. They per-form well, though not optimally, but are simple enoughto apply with very large block lengths.

Transform codes are easy to implement because of adivide and conquer strategy; the transform exploits de-pendencies in the data so that the quantization and en-tropy coding can be simple. For jointly Gaussian sources,this works perfectly: the transform can produce inde-pendent transform coefficients, and then little is lost byusing scalar quantization and scalar entropy coding.

As with source coding generally, it is hard to make useof the theory of transform coding with real-world signals.Nevertheless, the principles of transform coding certainlydo apply, as evidenced by the dominance of transformcodes in audio, image, and video compression.

AcknowledgmentsComments from M. Effros, A.K. Fletcher, anonymousreviewers, and the guest editors are gratefully acknowl-edged.

Vivek K Goyal received the B.S. in mathematics and theB.S.E. in electrical engineering (both with highest dis-tinction), in 1993 from the University of Iowa, IowaCity. He received the M.S. and Ph.D. degrees in electricalengineering from the University of California, Berkeley,in 1995 and 1998, respectively. He was a Research Assis-


tant in the Laboratoire de Communicat ionsAudiovisuelles at Ècole Polytechnique Fédérale deLausanne, Switzerland, in 1996. He was with LucentTechnologies’ Bell Laboratories as an intern in 1997 andagain as a Member of Technical Staff from 1998 to 2001.He is currently a Senior Research Engineer for DigitalFountain, Inc., Fremont, CA. His research interests in-clude source coding theory, quantization theory, andpractical robust compression. He is a member of theIEEE, Phi Beta Kappa, Tau Beta Pi, Sigma Xi, Eta KappaNu and SIAM. In 1998 he received the Eliahu JuryAward of the University of California, Berkeley.

References[1] T. Berger, Rate Distortion Theory. Englewood Cliffs, NJ: Prentice-Hall,

1971.

[2] J.H. Conway and N.J.A. Sloane, Sphere Packings, Lattices and Groups, 3rded. (Grundlehren der mathematischen Wissenschaften, vol. 290). New York:Springer-Verlag, 1998.

[3] T.M. Cover and J.A. Thomas, Elements of Information Theory. New York:Wiley, 1991.

[4] H.W. Dudley, “The vocoder,” Bell Lab. Rec., vol. 18, pp. 122-126, Dec. 1939.

[5] M. Effros, “Optimal modeling for complex system design,” IEEE SignalProcessing Mag., vol. 15, pp. 51-73, Nov. 1998.

[6] N. Farvardin and J.W. Modestino, “Optimum quantizer performance for aclass of non-Gaussian memoryless sources,” IEEE Trans. Inform. Theory, vol.IT-30, pp. 485-497, May 1984.

[7] A. Gersho, “Asymptotically optimal block quantization” IEEE Trans. In-form. Theory, vol. IT-25, pp. 373-380, July 1979.

[8] A. Gersho and R.M. Gray, Vector Quantization and Signal Compression.Norwell, MA: Kluwer, 1992.

[9] H. Gish and J.P. Pierce, “Asymptotically efficient quantizing,” IEEE Trans.Inform. Theory, vol. IT-14, pp. 676-683, Sept.1968.

[10] T.J. Goblick, Jr. and J.L. Holsinger, “Analog source digitization: A com-parison of theory and practice,” IEEE Trans. Inform. Theory, vol. IT-13, pp.323-326, Apr. 1967.

[11] V.K. Goyal, “High-rate transform coding: How high is high, and does itmatter?,” in Proc. IEEE Int. Symp. Inform. Theory, Sorrento, Italy, June2000, p. 207.

[12] V.K. Goyal, “Transform coding with integer-to-integer transforms,” IEEETrans. Inform. Theory, vol. 46, pp. 465-473, Mar. 2000.

[13] V.K. Goyal, “Multiple description coding: Compression meets the net-work,” IEEE Signal Processing Mag., vol. 18, pp. 74-93, Sept. 2001.

[14] V.K. Goyal. Single and Multiple Description Transform Coding with Basesand Frames. Philadelphia, PA: SIAM, 2001.

[15] V.K. Goyal, J. Zhuang, and M. Vetterli, “Transform coding with back-ward adaptive updates,” IEEE Trans. Inform. Theory, vol. 46, pp.1623-1633, July 2000.

[16] R.M. Gray and D.L. Neuhoff, “Quantization,” IEEE Trans. Inform. The-ory, vol. 44, pp. 2325-2383, Oct. 1998.

[17] U. Grenander and G. Szegö, Toeplitz Forms and Their Applications. Berke-ley, CA: Univ. of California Press, 1958.

[18] R.A. Horn and C.R. Johnson, Matrix Analysis. Cambridge, U.K.: Cam-

bridge Univ. Press, 1985. (Reprinted with corrections 1987.)

[19] H. Hotelling, “Analysis of a complex of statistical variables into principalcomponents,” J. Educ. Psychology, vol. 24, pp. 417-441, 498-520, 1933.

[20] J.-Y. Huang, “Quantization of correlated random variables,” Ph.D. disser-tation, Yale Univ., New Haven, CT, 1962.

[21] J.J.Y. Huang and P.M. Schultheiss, “Block quantization of correlatedGaussian random variables,” IEEE Trans. Commun. Syst., vol. 11, pp.289-296, Sept. 1963.

[22] D.A. Huffman, “A method for the construction of minimum redundancycodes,” Proc. IRE, vol. 40, pp.1098-1101, Sept. 1952.

[23] A.K. Jain, “A sinusoidal family of unitary transforms,” IEEE Trans. Pat-tern Anal. Machine Intell., vol. PAMI-1, pp. 356-365, Oct. 1979.

[24] K. Karhunen, “Über lineare methoden in derWahrscheinlichkeitsrechnung,” Ann. Acad. Sci. Fenn., Ser. A.1.: Math.-Phys.,vol. 37, pp. 3-79, 1947.

[25] H.P. Kramer and M.V. Mathews, “A linear coding for transmitting a set of cor-related signals,” IRE Trans. Inform. Theory, vol. 23, no. 3, pp. 41-46, Sept. 1956.

[26] M. Loève, “Fonctions aleatoires de seconde ordre,” in ProcessusStochastiques et Mouvement Brownien, P. Levy, Ed . Paris, France:Gauthier-Villars, 1948.

[27] D.G. Luenberger, Optimization by Vector Space Methods. New York:Wiley, 1969.

[28] D.F. Lyons III, “Fundamental limits of low-rate transform codes,” Ph.D.dissertation, Univ. Michigan, Ann Arbor, MI, 1992.

[29] N. Moayeri and D.L. Neuhoff, “Time-memory tradeoffs in vectorquantizer codebook searching based on decision trees,” IEEE Trans. SpeechAudio Processing, vol. 2, pp. 490-506, Oct. 1994.

[30] A. Ortega and K. Ramchandran, “Rate-distortion methods for image andvideo compression,” IEEE Signal Processing Mag., vol. 15, pp. 23-50, Nov.1998.

[31] A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd ed.New York: McGraw-Hill, 1991.

[32] W.B. Pennebaker and J.L. Mitchell, JPEG Still Image Data CompressionStandard. New York: Van Nostrand, 1993.

[33] K.R. Rao and P. Yip, Discrete Cosine Transform: Algorithms, Advantages,Applications. San Diego, CA: Academic, 1990.

[34] W.D. Ray and R.M. Driver, “Further decomposition of theKarhunen-Loève series representation of a stationary random process,”IEEE Trans. Inform. Theory, vol. IT-16, pp. 663-668, Nov. 1970.

[35] A. Segall, “Bit allocation and encoding for vector sources,” IEEE Trans.Inform. Theory, vol. IT-22, pp. 162-169, Mar. 1976.

[36] Y. Shoham and A. Gersho, “Efficient bit allocation for an arbitrary set ofquantizers,” IEEE Trans. Acoust. Speech Signal Processing, vol. 36, pp.1445-1453, Sept. 1988.

[37] A. Skodras, C. Christopoulos, and T. Ebrahimi, “The JPEG 2000 still im-age compression standard,” IEEE Signal Processing Mag., vol. 18, pp. 36-58,Sept. 2001.

[38] B.E. Usevitch, “A tutorial on modern lossy wavelet image compression:Foundations of JPEG 2000,” IEEE Signal Processing Mag., vol. 18, pp.22-35, Sept. 2001.

[39] M. Vetterli, “Wavelets, approximation, and compression,” IEEE SignalProcessing Mag., vol. 18, pp. 59-73, Sept. 2001.

[40] R.C. Wood, “On optimum quantization,” IEEE Trans. Inform. Theory,vol. IT-15, pp. 248-252, Mar. 1969.


theoretical foundations of transform coding · theoretical foundations of transform coding vivek k...

Documents