bayesian estimation for continuous-time

8/13/2019 Bayesian Estimation for Continuous-Time

1/51

Bayesian Estimation for Continuous-Time

Sparse Stochastic Processes


2/51

Abstract

We consider continuous-time sparse stochastic processes from which we have only a

finite number of noisy/noiseless samples. Our goal is to estimate the noiseless samples

(denoising) and the signal in-between (interpolation problem). By relying on tools from the

theory of splines, we derive the joint a priori distribution of the samples and show how this

probability density function can be factorized. The factorization enables us to tractably

implement the maximum a posteriori and minimum mean-square error (MMSE) criteria as two

statistical approaches for estimating the unknowns. We compare the derived statistical methods

with well-known techniques for the recovery of sparse signals, such as the norm and Log ( -

relaxation) regularization methods. The simulation results show that, under certain conditions,

the performance of the regularization techniques can be very close to that of the MMSE

estimator.

INTRODUCTION

THE recent popularity of regularization techniques in signal and image processing is motivated

by the sparse

nature of real-world data. It has resulted in the development of powerful tools for many problems

such as denoising, deconvolution, and interpolation. The emergence of compressed sensing,

which focuses on the recovery of sparse vectors from highly under-sampled sets of

measurements, is playing a key role in this context [1][3]. Assume that the signal of interest is a

finite-length discrete signal also represented by as a vector) that has a sparse or almost sparse

representation in some transform or analysis domain (e.g., wavelet or DCT). Assume moreover

that we only

have access to noisy measurements of the form , where denotes an additive white Gaussian

noise. Then, we would like to estimate . The common sparsity-promoting variational techniques

rely on penalizing the sparsity in the transform/analysis domain [4], [5] by imposing where is the

vector of noisy measurements, is a penalty function that reflects the sparsity constraint in the

transform/ analysis domain and is a weight that is usually set based on the noise and signal

powers. The choice of is one of the favorite ones in compressed sensing when is itself sparse [6],

while the use of , where stands for total variation, is a common choice for piecewise-smooth


3/51

signals that have sparse derivatives [7]. Although the estimation problem for a given set of

measurements is a deterministic procedure and can be handled without recourse to statistical

tools, there are benefits in viewing the problem from the stochastic perspective. For instance, one

can take advantage of side information about the unobserved data to establish probability laws

for all or part of the data. Moreover, a stochastic framework allows one to evaluate the

performance

of estimation techniques and argue about their distance from the optimal estimator. The

conventional stochastic interpretation of the variation method in (1) leads to the finding that is

the maximum a posteriori (MAP) estimate of . In this interpretation, the quadratic data term is

associated with the Gaussian nature of the additive noise, while the scarifying penalty term

corresponds to the a priori distribution of the sparse input. For example, the penalty

is associated with the MAP estimator with Laplace prior [8], [9]. However, investigations of the

compressible/sparse priors have revealed that the Laplace distribution cannot be considered as a

sparse prior [10][12]. Recently in [13], it is argued that (1) is better interpreted as the minimum

mean-square error (MMSE) estimator of a sparse prior.

Though the discrete stochastic models are widely adopted for sparse signals, they only

approximate the continuous nature of real-world signals. The main challenge for employing

continuous models is to transpose the compressibility/sparsity concepts in the continuous domain

while maintaining compatibility with the discrete domain. In [14], an extended class of

piecewise-smooth signals is proposed as a candidate for continuous stochastic sparse models.

This class is closely related to signals with a finite rate of innovation [15]. Based on infinitely

divisible distributions, a more general stochastic framework has been recently introduced in [16],

[17]. There, the continuous models include Gaussian processes (such as Brownian motion),

piecewise-polynomial signals, and -stable processes as special cases. In addition, a large portion

of the introduced family is considered as compressible/sparse with respect to the definition in

[11] which is compatible with the discrete definition.

In this paper, we investigate the estimation problem for the samples of the continuous-time

sparse models introduced in [16], [17]. We derive the a priori and a posteriori probability density

functions (pdf) of the noiseless/noisy amples. We present a practical factorization of the prior

distribution which enables us to perform statistical learning for denoising or interpolation

problems. In particular, we implement the optimal MMSE estimator based on the message-


4/51

passing algorithm. The implementation involves discretization and convolution of pdfs, and is in

general, slower than the common variational techniques. We further compare the performance of

the Bayesian and variational denoising methods. Among the variational methods, we consider

quadratic, TV, and Log regularization

Techniques. Our results show that, by matching the regularizer to the statistics, one can almost

replicate the MMSE

Performance. The rest of the paper is organized as follows: In Section II, we introduce our signal

model which relies on the general formulation of sparse stochastic processes proposed in [16],

[17]. In Section IV, we explain the techniques for obtaining the probability density functions

and, we derive the estimation methods in Section III. We study the special case of Lvy

processes which is of interest in many applications in Section V, and present simulation results in

Section VI. Section VII concludes the paper SIGNAL MODEL In this section, we adapt the

general framework of [16] to the continuous-time stochastic model studied in this paper. We

follow the same notational conventions and write the input argument of the continuous-time

signals/processes inside parenthesis

(e.g., ) while we employ brackets (e.g., ) for discrete- time ones. Moreover, the tilde diacritic is

used to indicate the noisy signal. Typically, represents discrete noisy samples. In Fig. 1, we give

a sketch of the model. The two main arts are the continuous-time innovation process and the

linear operators. The process is generated by applying the shaping operator on the innovation

process . It can be whitened back by the inverse operator L. (Since the whitening operator is of

greater importance, it is represented by L while refers to the shaping operator.) Furthermore, the

iscrete observations are formed by the noisy measurements of . The innovation process and the

linear operators have distinct implications on the resultant process . Our model is able to handle

general innovation processes that may or may not induce sparsity/compressibility. The

distinction between these two cases is identified by a function that is called the Lvy exponent,

as will be discussed in Section II-A. The sparsity/ compressibility of and, consequently, of the

measurements , is inherited from the innovations and is observed in a transform domain. This

domain is tied to the operator L. In this paper, we deal with operators that we represent by all-

pole differential systems, tuned by acting upon the poles. Although the model in Fig. 1 is rather

classical for Gaussian innovations, the investigation of non-Gaussian innovations is nontrivial.

While the transition from Gaussian to non-Gaussian necessitates the reinvestigation of every


5/51

definition and result, it provides us with a more general class of stochastic processes which

includes compressible/sparse signals.

A. I nnovation Process

Of all white processes, the Gaussian innovation is undoubtedly the one that has been investigated

most thoroughly. However, it represents only a tiny fraction of the large family of white

processes, which is best explored by using Gelfands theoryof generalized random processes. In

his approach, unlike with the conventional point-wise definition, the stochastic process is

characterized through inner products with test functions. For this purpose, one first chooses a

function space of test functions (e.g., the Schwartz class of smooth and rapidly decaying

functions). Then, one considers the random variable given by the inner product

1) it is stationary, i.e., the random variables and are identically distributed, provided is a shifted

version of , and

2) it is white in the sense that the random variables and are independent, provided are non-

overlapping test functions (i.e., ). The characteristic form of is defined as

where represents the expected-value operator. The characteristic form is a powerful tool for

investigating the properties of random processes. For instance, it allows one to easily infer the

probability density function of the random variable , or the joint densities of . Further details

regarding characteristic forms can be found in Appendix A. The key point in Gelfands theory is

to consider the form

(the Lvy exponent) for to define a generalized innovation process over (dual of ). The class of

admissible Lvy exponents is characterized by the Lvy-Khintchine representation theorem [19],

[20] as


6/51

where for and 0 otherwise, and (the Lvy density) is a real-valued

density function that satisfies

In this paper, we consider only symmetric real-valued Lvy exponents (i.e., and ). Thus, the

general form of (4) is reduced to

Next, we discuss three particular cases of (6) which are of special interest in this paper.

1) Gaussian Innovation: The choice turns (6) into

which implies

This shows that the random variable has a zero-mean Gaussian distribution with variance .

2) Impulsive Poisson: Let and , where is a symmetric probability density function. The

corresponding white process is known as the impulsive Poisson innovation. By substitution in

(6), we obtain


7/51

where denotes the Fourier transform of . Let represents the test function that takes 1 on and 0

otherwise. Thus, if , then for the pdf of we know that (see Appendix A)

It is not hard to check (see Appendix II in [14] for a proof) that this distribution matches the one

that we obtain by defining

is a sequence of random Poisson points with parameter , and is an independent and identically

distributed (i.i.d.) sequence with probability density independent of . The sequence is a Poisson

point random sequence with parameter

if, for all real values , the random


8/51


9/51

The second important component of the model is the shaping operator (the inverse of the

whitening operator L) that determines the correlation structure of the process. For the generalized

stochastic definition of in Fig. 1, we expect to have

where represents the adjoint operator of . It shows that ought to define a valid test function for

the equalities in (15) to remain valid. In turn, this sets constraints on . The simplest choice for

would be that of an operator which forms a continuous map from into itself, but the class of such

operators is not rich enough to cover the desired models in this

paper. For this reason, we take advantage of a result in [16] that extends the choice of shaping

operators to those operators for which forms a continuous mapping from into for some .

1) Valid Inverse Operator :2) In the sequel, we first explain the general requirements on the inverse of a given

whitening operator L. Then, we focus on a special class of operators L and study the

implications for the associated shaping operators in more details. We assume L to be a

given whitening operator, which may or may not be uniquely invertible. The minimum

requirement on the shaping operator is that it should form a right-inverse of L (i.e., ,

where I is the identity operator). Furthermore, since the adjoint operator is required in

(15), needs to be

linear. This implies the existence of a kernel such that


10/51

Linear shift-invariant shaping operators are special cases that correspond to . However, some of

the operators considered in this paper are not shift-invariant. We require the kernel to satisfy the

following three conditions:

i) , where L acts on the parameter and is the Dirac function, ii) , for , iii) is bounded for all .

Condition (i) is equivalent to, while (iii) is a sufficient condition studied in [16] to establish the

continuity of the mapping , for all . Condition (ii) is a constraint that we impose in this paper to

simplify the statistical analysis. For, its implication is that the random variable is fully

determined by with, or, equivalently, it is independent of for. From now on, we focus on

differential operators L of the form, where D is the first-order derivative is the identity operator

(I), and are constants. With Gaussian innovations, these operators generate the autoregressive

processes. An equivalent representation of L, which helps us in the analysis, is its decomposition

into first-order factors as. The scalars are the roots of the characteristic polynomial and

correspond to the poles of the inverse linear system. Here, we assume that all the poles are in the

left half-plane. This assumption helps us associate the operator L to a suitable kernel, as shown

in Appendix B. Every differential operator L has a unique causal Green function [22]. The linear

shift-invariant system defined by satisfies conditions (i)(ii). If all the poles strictly lie in the left

half-plane (i.e., ), due to absolute inerrability of (stability of the system), satisfies condition (iii)

as well. The definition of given through thekernels in Appendix B achieves both linearity and

stability, while loosing shift-invariance when L contains poles on the imaginary axis. It is worth

mentioning that the application of two different right-inverses of L on a given input roduces

results that differ only by an exponential polynomial that is in the null space of L. 2)

Discretization: Apart from qualifying as whitening operators, differential operators have other


11/51

appealing properties such as the existence of finite-length discrete counterparts. To

Explain this concept, let us first consider the first-order continuous- time derivative operator D

that is associated with the finite- difference filter. This discrete counterpart is of finite length

(FIR filter). Further, for any right inverse of D such as, the system is shift invariant and its

impulse response is compactly supported. Here, is the discredited operator corresponding to the

sampling period with impulse response. It should be emphasized that the discrete counterpart is a

discrete-domain operator, while the discretized operator acts on continuous-domain signals. It is

easy to check that this impulse response coincides with the causal B-spline of degree 0 . In

general, the discrete counterpart of is defined through its factors. Each is associated with its

discrete counterpart and a discretized operator given by the impulse response. The convolution of

such impulse responses gives rise to the impulse response of (up to the scaling factor), which is

the discretized operator of L for the sampling period . By expanding the convolution, we obtain

the form for the impulse response of . It is now evident that corresponds to an FIR filter of lengthrepresented by with . Results in spline theory confirm that, for any right inverse of L, the

operator is shift invariant and the support of its impulse response is contained in [23]. The

compactly supported impulse response of , which we denote by , is usually referred to as the L-

spline. We define the generalized finite differences by


12/51

Sparsity/Compressibility The innovation process can be thought of as a concatenation of

independent atoms. A consequence of this independence is that the process contains no

redundancies. Therefore, it is incompressible under unique representation constraint. In our

framework, the role of the shaping operator is to generate a specific correlation structure in by

mixing the atoms of . Conversely, the whitening operator L undoes the mixing and returns an

incompressible set of data, in the sense that it maximally compresses the data. For a

discretization of corresponding to a sampling period, the operator mimics the role of L. Itefficiently uncouples the sequence of samples and produces the generalized differences, where

each term depends only on a finite number of other terms. Thus, the role of can be compared to

that of converting discrete-time signals into their transform domain representation. As we

explain now, the sparsity/compressibility properties of are closely related to the Lvy exponent

of . The concept is best explained by focusing on a special class known as Lvy processes that

correspond to (see Section V for more details). By using (3) and (53), we can check that the

characteristic function of the random ariable is given by . When the Lvy function is generated

through a nonzero density that is absolutely integrable (i.e., impulsive Poisson), the pdf of

associated with necessarily contains a mass at the origin [20] (Theorems 4.18 and 4.20). This is

interpreted as sparsity when considering a finite number of measurements. It is shown in [11],

[12] that the compressibility of the measurements depends on the tail of their pdf. In simple

terms, if the pdf decays at most inverse-polynomially, then, it is compressible in some

corresponding norm. The interesting point is that the inverse-polynomial decay of an infinite-

divisible pdf is equivalent to the inverse-polynomial decay of its Lvy ensity

With the same order [20] (Theorem 7.3). Therefore, an innovation process with a fat-tailed Lvy

density results in recesses with compressible measurements. This indicates that the slower the

decay rate of , the more compressible the measurements of .


13/51

D. Summary of the Parameters of the Model We now briefly review the degrees of freedom in

the model and how the dependent variables are determined. The innovation process is uniquely

determined by the Lvy triplet . The sparsity/compressibility of the process can be determined

through the Lvy density (see Section II-C). The values

and , or, equivalently, the poles of the system, serve as the free parameters for the whitening

operator L. As explained in Section II-B2, the taps of the FIR filter are determined by the poles.

MATLAB


14/51

INTRODUCTION TO MATLAB

What Is MATLAB?

MATLAB

is a high-performance language for technical computing. It integrates

computation, visualization, and programming in an easy-to-use environment where problems and

solutions are expressed in familiar mathematical notation. Typical uses include

Math and computation

Algorithm development

Data acquisition

Modeling, simulation, and prototyping

Data analysis, exploration, and visualization


15/51

Scientific and engineering graphics

Application development, including graphical user interface building.

MATLAB is an interactive system whose basic data element is an array that does notrequire dimensioning. This allows you to solve many technical computing problems, especially

those with matrix and vector formulations, in a fraction of the time it would take to write a

program in a scalar non interactive language such as C or FORTRAN.

The name MATLAB stands for matrix laboratory. MATLAB was originally written to

provide easy access to matrix software developed by the LINPACK and EISPACK projects.

Today, MATLAB engines incorporate the LAPACK and BLAS libraries, embedding the state of

the art in software for matrix computation.

MATLAB has evolved over a period of years with input from many users. In university

environments, it is the standard instructional tool for introductory and advanced courses in

mathematics, engineering, and science. In industry, MATLAB is the tool of choice for high-

productivity research, development, and analysis.

MATLAB features a family of add-on application-specific solutions called toolboxes.

Very important to most users of MATLAB, toolboxes allow you to learn and applyspecialized

technology. Toolboxes are comprehensive collections of MATLAB functions (M-files) that

extend the MATLAB environment to solve particular classes of problems. Areas in which

toolboxes are available include signal processing, control systems, neural networks, fuzzy logic,

wavelets, simulation, and many others.

The MATLAB System:

The MATLAB system consists of five main parts:

Development Environment:

This is the set of tools and facilities that help you use MATLAB functions and files. Many

of these tools are graphical user interfaces. It includes the MATLAB desktop and Command


16/51

Window, a command history, an editor and debugger, and browsers for viewing help, the

workspace, files, and the search path.

The MATLAB Mathematical Function:

This is a vast collection of computational algorithms ranging from elementary functions

like sum, sine, cosine, and complex arithmetic, to more sophisticated functions like matrix

inverse, matrix eigen values, Bessel functions, and fast Fourier transforms.

The MATLAB Language:

This is a high-level matrix/array language with control flow statements, functions, data

structures, input/output, and object-oriented programming features. It allows both "programmingin the small" to rapidly create quick and dirty throw-away programs, and "programming in the

large" to create complete large and complex application programs.

Graphics:

MATLAB has extensive facilities for displaying vectors and matrices as graphs, as well as

annotating and printing these graphs. It includes high-level functions for two-dimensional and

three-dimensional data visualization, image processing, animation, and presentation graphics. Italso includes low-level functions that allow you to fully customize the appearance of graphics as

well as to build complete graphical user interfaces on your MATLAB applications.

The MATLAB Application Program Interface (API):

This is a library that allows you to write C and Fortran programs that interact with

MATLAB. It includes facilities for calling routines from MATLAB (dynamic linking), calling

MATLAB as a computational engine, and for reading and writing MAT-files.

MATLAB WORKING ENVIRONMENT:

MATLAB DESKTOP:-


17/51


18/51


19/51

DIGITAL IMAGE PROCESSING


20/51

Digital image processing

Background:

Digital image processing is an area characterized by the need for extensive experimental

work to establish the viability of proposed solutions to a given problem. An important

characteristic underlying the design of image processing systems is the significant level oftesting & experimentation that normally is required before arriving at an acceptable solution.

This characteristic implies that the ability to formulate approaches &quickly prototype candidate

solutions generally plays a major role in reducing the cost & time required to arrive at a viable

system implementation.

What is DIP

An image may be defined as a two-dimensional function f(x, y), where x & y are spatial

coordinates, & the amplitude of f at any pair of coordinates (x, y) is called the intensity or gray

level of the image at that point. When x, y & the amplitude values of f are all finite discrete

quantities, we call the image a digital image. The field of DIP refers to processing digital image

by means of digital computer. Digital image is composed of a finite number of elements, each of

which has a particular location & value. The elements are called pixels.


21/51

Vision is the most advanced of our sensor, so it is not surprising that image play the

single most important role in human perception. However, unlike humans, who are limited to the

visual band of the EM spectrum imaging machines cover almost the entire EM spectrum, ranging

from gamma to radio waves. They can operate also on images generated by sources that humans

are not accustomed to associating with image.

There is no general agreement among authors regarding where image processing stops &

other related areas such as image analysis& computer vision start. Sometimes a distinction is

made by defining image processing as a discipline in which both the input & output at a process

are images. This is limiting & somewhat artificial boundary. The area of image analysis (image

understanding) is in between image processing & computer vision.

There are no clear-cut boundaries in the continuum from image processing at one end to

complete vision at the other. However, one useful paradigm is to consider three types of

computerized processes in this continuum: low-, mid-, & high-level processes. Low-level

process involves primitive operations such as image processing to reduce noise, contrast

enhancement & image sharpening. A low- level process is characterized by the fact that both its

inputs & outputs are images.

Mid-level process on images involves tasks such as segmentation, description of that

object to reduce them to a form suitable for computer processing & classification of individual

objects. A mid-level process is characterized by the fact that its inputs generally are images but

its outputs are attributes extracted from those images. Finally higher- level processing involves

Making sense of an ensemble of recognized objects, as in image analysis & at the far end of

the continuum performing the cognitive functions normally associated with human vision.

Digital image processing, as already defined is used successfully in a broad range ofareas of exceptional social & economic value.

What is an image?


22/51

An image is represented as a two dimensional function f(x, y) where x and y are spatial

co-ordinates and the amplitude of f at any pair of coordinates (x, y) is called the intensity of the

image at that point.

Gray scale image:

A grayscale image is a function I (xylem) of the two spatial coordinates of the image

plane.

I(x, y) is the intensity of the image at the point (x, y) on the image plane.

I (xylem)takes non-negative values assume the image is bounded by arectangle[0, a] [0, b]I:

[0, a] [0, b][0, info)

Color image:

Itcan be represented by three functions, R (xylem)for red,G (xylem)for greenandB

(xylem)for blue.

An image may be continuous with respect to the x and y coordinates and also in

amplitude. Converting such an image to digital form requires that the coordinates as well as the

amplitude to be digitized. Digitizing the coordinates values is called sampling. Digitizing the

amplitude values is called quantization.

Coordinate convention:

The result of sampling and quantization is a matrix of real numbers. We use two

principal ways to represent digital images. Assume that an image f(x, y) is sampled so that the

resulting image has M rows and N columns. We say that the image is of size M X N. The values

of the coordinates (xylem) are discrete quantities. For notational clarity and convenience, we use

integer values for these discrete coordinates.

In many image processing books, the image origin is defined to be at (xylem)=(0,0).The

next coordinate values along the first row of the image are (xylem)=(0,1).It is important to keep

in mind that the notation (0,1) is used to signify the second sample along the first row. It does

not mean that these are the actual values of physical coordinates when the image was sampled.


23/51

Following figure shows the coordinate convention. Note that x ranges from 0 to M-1 and y from

0 to N-1 in integer increments.

The coordinate convention used in the toolbox to denote arrays is different from the

preceding paragraph in two minor ways. First, instead of using (xylem) the toolbox uses the

notation (race) to indicate rows and columns. Note, however, that the order of coordinates is the

same as the order discussed in the previous paragraph, in the sense that the first element of a

coordinate topples, (alb), refers to a row and the second to a column. The other difference is that

the origin of the coordinate system is at (r, c) = (1, 1); thus, r ranges from 1 to M and c from 1 to

N in integer increments. IPT documentation refers to the coordinates. Less frequently the toolbox

also employs another coordinate convention called spatial coordinates which uses x to refer to

columns and y to refers to rows. This is the opposite of our use of variables x and y.

Image as Matrices:

The preceding discussion leads to the following representation for a digitized image

function:

f (0,0) f(0,1) .. f(0,N-1)

f (1,0) f(1,1) f(1,N-1)

f (xylem)= . . .

. . .

f (M-1,0) f(M-1,1) f(M-1,N-1)

The right side of this equation is a digital image by definition. Each element of this array

is called an image element, picture element, pixel or pel. The terms image and pixel are used

throughout the rest of our discussions to denote a digital image and its elements.


24/51

A digital image can be represented naturally as a MATLAB matrix:

f (1,1) f(1,2) . f(1,N)

f (2,1) f(2,2) .. f (2,N)

. . .

f = . . .

f (M,1) f(M,2) .f(M,N)

Where f (1,1) = f(0,0) (note the use of a monoscope font to denote MATLAB quantities).

Clearly the two representations are identical, except for the shift in origin. The notation f(p ,q)

denotes the element located in row p and the column q. For example f(6,2) is the element in the

sixth row and second column of the matrix f. Typically we use the letters M and N respectively

to denote the number of rows and columns in a matrix. A 1xN matrix is called a row vector

whereas an Mx1 matrix is called a column vector. A 1x1 matrix is a scalar.

Matrices in MATLAB are stored in variables with names such as A, a, RGB, real array

and so on. Variables must begin with a letter and contain only letters, numerals and underscores.

As noted in the previous paragraph, all MATLAB quantities are written using mono-scope

characters. We use conventional Roman, italic notation such as f(x ,y), for mathematical

expressions

Reading Images:

Images are read into the MATLAB environment using function imread whose syntax is

Imread (filename)

Format name Description recognized extension


25/51

TIFF Tagged Image File Format .tif, .tiff

JPEG Joint Photograph Experts Group .jpg, .jpeg

GIF Graphics Interchange Format .gif

BMP Windows Bitmap .bmp

PNG Portable Network Graphics .png

XWD X Window Dump .xwd

Here filename is a spring containing the complete of the image file(including any

applicable extension).For example the command line

>> f = imread (8. jpg);

Reads the JPEG (above table) image chestxray into image array f. Note the use of single

quotes () to delimit the string filename. The semicolon at the end of a command line is used by

MATLAB for suppressing output If a semicolon is not included. MATLAB displays the results

of the operation(s) specified in that line. The prompt symbol (>>) designates the beginning of a

command line, as it appears in the MATLAB command window.

Data Classes:

Although we work with integers coordinates the values of pixels themselves are not

restricted to be integers in MATLAB. Table above list various data classes supported by

MATLAB and IPT are representing pixels values. The first eight entries in the table are refers to

as numeric data classes. The ninth entry is the char class and, as shown, the last entry is referred

to as logical data class.

All numeric computations in MATLAB are done in double quantities, so this is also a

frequent data class encounter in image processing applications. Class unit 8 also is encountered

frequently, especially when reading data from storages devices, as 8 bit images are most

common representations found in practice. These two data classes, classes logical, and, to a


26/51

lesser degree, class unit 16 constitute the primary data classes on which we focus. Many ipt

functions however support all the data classes listed in table. Data class double requires 8 bytes

to represent a number uint8 and int 8 require one byte each, uint16 and int16 requires 2bytes and

unit 32.

Name Description

Double Double _ precision, floating_ point numbers the Approximate.

Uint8 unsigned 8_bit integers in the range [0,255] (1byte per

Element).

Uint16 unsigned 16_bit integers in the range [0, 65535] (2byte per element).

Uint 32 unsigned 32_bit integers in the range [0, 4294967295](4 bytes per

element). Int8 signed 8_bit integers in the range [-128,127] 1 byte per element)

Int 16 signed 16_byte integers in the range [32768, 32767] (2 bytes per

element).

Int 32 Signed 32_byte integers in the range [-2147483648, 21474833647] (4

byte per element).

Single single _precision floating _point numbers with values

In the approximate range (4 bytes per elements)

Char characters (2 bytes per elements).

Logical values are 0 to 1 (1byte per element).

Int 32 and single required 4 bytes each. The char data class holds characters in Unicode

representation. A character string is merely a 1*n array of characters logical array contains only

the values 0 to 1,with each element being stored in memory using function logical or by using

relational operators.

Image Types:

The toolbox supports four types of images:


27/51

1 .Intensity images;

2. Binary images;

3. Indexed images;

4. R G B images.

Most monochrome image processing operations are carried out using binary or intensity

images, so our initial focus is on these two image types. Indexed and RGB colour images.

Intensity Images:

An intensity image is a data matrix whose values have been scaled to represent

intentions. When the elements of an intensity image are of class unit8, or class unit 16, they have

integer values in the range [0,255] and [0, 65535], respectively. If the image is of class double,

the values are floating point numbers. Values of scaled, double intensity images are in the range

[0, 1] by convention.

Binary Images:

Binary images have a very specific meaning in MATLAB.A binary image is a logical

array 0s and1s.Thus, an array of 0s and 1s whose values are of data class, say unit8, is not

considered as a binary image in MATLAB .A numeric array is converted to binary using

function logical. Thus, if A is a numeric array consisting of 0s and 1s, we create an array B using

the statement.

B=logical (A)

If A contains elements other than 0s and 1s.Use of the logical function converts all

nonzero quantities to logical 1s and all entries with value 0 to logical 0s.

Using relational and logical operators also creates logical arrays.


28/51

To test if an array is logical we use the I logical function: islogical(c).

If c is a logical array, this function returns a 1.Otherwise returns a 0. Logical array can be

converted to numeric arrays using the data class conversion functions.

Indexed Images:

An indexed image has two components:

A data matrix integer, x

A color map matrix, map

Matrix map is an m*3 arrays of class double containing floating point values in the range

[0, 1].The length m of the map are equal to the number of colors it defines. Each row of map

specifies the red, green and blue components of a single color. An index ed images uses direct

mapping of pixel intensity values color map values. The color of each pixel is determined by

using the corresponding value the integer matrix x as a pointer in to map. If x is of class double

,then all of its components with values less than or equal to 1 point to the first row in map, all

components with value 2 point to the second row and so on. If x is of class units or unit 16, then

all components value 0 point to the first row in map, all components with value 1 point to the

second and so on.

RGB Image:

An RGB color image is an M*N*3 array of color pixels where each color pixel is triplet

corresponding to the red, green and blue components of an RGB image, at a specific spatial

location. An RGB image may be viewed as stack of three gray scale images that when fed in to

the red, green and blue inputs of a color monitor

Produce a color image on the screen. Convention the three images forming an RGB color

image are referred to as the red, green and blue components images. The data class of the

components images determines their range of values. If an RGB image is of class double the

range of values is [0, 1].


29/51

Similarly the range of values is [0,255] or [0, 65535].For RGB images of class units or

unit 16 respectively. The number of bits use to represents the pixel values of the component

images determines the bit depth of an RGB image. For example, if each component image is an

8bit image, the corresponding RGB image is said to be 24 bits deep.

Generally, the number of bits in all component images is the same. In this case the

number of possible color in an RGB image is (2^b) ^3, where b is a number of bits in each

component image. For the 8bit case the number is 16,777,216 colors.

We consider continuous-time sparse stochastic processes from which we have only a finite

number of noisy/noiseless samples. Our goal is to estimate the noiseless samples (denoising) and

the signal in-between(interpolation problem). By relying on tools from the theory of splines, we

derive the joint a priori distribution of the samples and show how this probability density

function can be factorized. The factorization enables us to tractably implement the maximum a

posteriori and minimum mean-square error (MMSE) criteria as two statistical approaches for

estimating the unknowns. We compare the derived statistical methods with well-known

techniques for the recovery of sparse signals, such as the 1 norm and Log (1-0 relaxation)

regularization methods. The simulation results show that, under certain conditions, the

performance of the regularization techniques can be very close to that of the MMSE estimator.

decomposition (POD), reduction based on inertial manifolds, and reduced basis

methodshave been used in this context. However, the solution of the reduced

order optimal control problem is generally suboptimal and reliable error estimation

is thus crucial. A posteriori error bounds for reduced order solutions of optimal control problems

have been proposed for proper orthogonal decomposition (POD) and reduced basis surrogatemodels in [9] and [2, 7], respectively. However, all of these results have slight deficiencies, i.e.,

evaluation of the bounds in [9] requires a forwardbackward solution of the underlying high-

dimensional state and adjoint equations and is thus computationally expensive, the error

estimator in [2] is not a rigorous upper bound for the error, and the result in [7] only applies to

optimal control problems without control constraints involving stationary (time-independent)


30/51

PDEs. In this talk, we employ the reduced basis method [8] as a surrogate model for the solution

of optimal control problems. We consider the following simplified roblem

setting: Given D RP , we want to solve the optimal control problem (

yd Y and ud U are the desired state and control, respectively; D is a

measurable set; and > 0 is the given regularization parameter. It follows from

our assumptions that there exists a unique optimal solution to (1) [6]. In the reduced basis

methodology, we next introduce a truth finite element approximation space YN Y of very

large dimension N and reduced basis space YN YN of dimension N, where usually N N.

We denote the truth and reduced basis solutions to the optimal control problem (1) obtained

through a Galerkin projection onto the respective spaces by (yN , u N) YN U and

(yN , u N) YN U, respectively.

We first show, for the simple problem setting (1), how to derive rigorous and

efficiently evaluable a posteriori error bounds for the optimal control, ku N u

NkU, and the associated cost functional, |J(yN , u N; ) J(yN , u N; )| [3, 4]. We start

with the bound from [9], replace the required high-dimensional state and adjoint

solution by the solution to the associated reduced basis approximation, and bound

the error introduced by extending the standard reduced basis a posteriori error bounds. The error

bound for the cost functional is based on the standard result

in [1], again by extending standard reduced basis a posteriori error bounds. The

offline-online decomposition directly applies to the reduced basis optimal control

problem and the associated a posteriori error bounds: the computational complexity

in the online stage to evaluate (yN , u N) as well as the control and associated cost functional

error bounds depends only on N and is independent of N. Our approach thus allows not only the

efficient real-time solution of the reduced optimal control problem, but also the efficient real-

time evaluation of the quality of the suboptimal solution. Finally, we consider various extensions

of the problem setting (1) to elliptic

optimal control problems with distributed controls and to parabolic problems with multiple

(time-dependent) controls and control constraints [5]. We also present

numerical results to confirm the validity of our approach. In the parabolic case, for

example, we can guaranteethanks to our a posteriori error boundsa relative

error in the control of less than 1% whilst obtaining an average (online) speed-up


31/51

of approximately 580 for the solution of the reduced basis optimal control problem

compared to the truth finite element optimal control problem and an average

speed-up of approximately 400 for the solution of the optimal control problem and

evaluation of the error bound for the control and cost functional. A principal approach to

compute (2) is the Monte Carlo method. However, it is extremely expensive to generate a large

number of suitable samples and to solve the deterministic boundary value problem (1) on each

sample. To overcome this obstruction, the multilevel Monte Carlo method (MLMC) has been

developed in

[1]. From the stochastic point of view, it is a variance reduction technique which

considerably decreases the complexity. The idea is to combine the Monte Carlo

quadrature of the stochastic variable with a multilevel splitting of the Bochner

space which contains the random solution. Then, to compute (2), most samples

can be performed on coarse spatial discretizations while only a few samples must

be performed on fine spatial discretizations. This proceeding is a sparse grid

approximation of the expectation. If we replace the Monte Carlo quadrature by

another quadrature rule for high-dimensional integrals, we obtain for example

the multilevel quasi Monte Carlo method (MLQMC) or the multilevel Gaussian

quadrature method (MLGQ).

It is a classical topic to look to orthonormal basis of polynomials on a compact

set X of Rd, with respect to some Radon measure . For exemple : the one

dimensional interval (Jacobi), the unit sphere (Spherical harmonics), the ball and

the simplex (work of Petrushev, Xu, ...) In this framework, one can be interested

in the best approximation of functions by polynomials of fixed degree, in Lp(),

and to built a suitable frame for characterization of function spaces related to

this approximation. This constructions have been carried using special functions

estimates.

We will be interested by spaces where the polynomials give the spectral spaces of

some positive selfadjoint operator. Under suitable conditions, a natural metric

could be defined on X such that (X, , ) is a homogeneous space, and if the

associated semi-group has a good Gaussian behavior, then we could apply the

procedure developed in recent works by P. Petrushev, T. Coulhon and G.K., to


32/51

built such frames, and such function spaces.

Sample regularity and fast simulation of isotropic Gaussian random fields on the

sphere are for example of interest for the numerical analysis of stochastic partial

differential equations and for the simulation of ice crystals or Saharan dust particles

as lognormal random fields. In what follows we recall the results from [2], which

include the approximation of isotropic Gaussian random fields with convergence

rates as well as the regularity of the samples in relation to the smoothness of

the covariance expressed in terms of the decay of the angular power spectrum.

As example we construct isotropic Q-Wiener processes out of isotropic Gaussian

random fields and discretize the stochastic heat equation with spectral methods.

Before we state the results, we start with a short review of the basics

III. PRIOR DISTRIBUTION

To derive statistical estimation methods such as MAP and MMSE, we need the a priori

distributions. In this section, by using the generalized differences in (17), we obtain the prior

distribution and factorize it efficiently. This factorization is fundamental, since it makes the

implementation of the MMSE and MAP estimators tractable. The general problem studied in this

paper is to estimate at for arbitrary (values of the continuous process at a uniform sampling grid

with ), given a finite number of noisy/noiseless measurements of at the integers. Although we are

aware that this is not equivalent to estimating the continuous process, by increasing we are able

to make the estimation grid as fine as desired. For piecewise-continuous signals, the limit process

of refining the grid can give us access to the properties of the continuous domain. To simplify the

notations let us define

distribution of (a priori distribution). However, the are in general pairwise-dependent, which

makes it difficult to

deal with the joint distribution in high dimensions. This corresponds to a large number of

samples. Meanwhile, as will be shown in Lemma 1, the sequence forms a Markov chain of order

that helps in factorizing the joint probability


33/51

distributions, whereas does not. The leading idea of this work is then that each depends on a

finite number of . It then becomes much simpler to derive the joint distribution of and link it to

that of . Lemma 1 helps us to factorize the joint pdf of . Lemma 1: For and , where is the

differential order of L, 1) the random variables are identically distributed, 2) the sequence is a

Markov chain of order , and 3) the sample is statistically independent of and . Proof: First note

that

Since functions are shifts of the same function for various and is stationary (Definition 1), areidentically distributed.

Recalling that is supported on , we know that and have no common support for . Thus, due to the

whiteness of cf. Definition 1), the random variables and are independent. Consequently, we can

write

which confirms that the sequence forms a Markov chain of order . Note that the choice of is due

to the support of . If the support of was infinite, it would be impossible to find such that and were

independent in the strict sense. To rove the second independence property, we recall that


34/51

Condition (ii) in Section II-B1 implies that for . Hence, and have disjoint supports. Again due to

whiteness of , this implies that and are independent. We now exploit the properties of to obtain

the a priori distribution of . Theorem 1, which is proved in Appendix C summarizes the main

result of Section III.

Theorem 1:Using the conventions of Lemma 1, for we have In the definition of proposed in

Section II-B, except when all the poles are strictly included in the left half-plane, the operator

fails to be shift-invariant. Consequently, neither nor are stationary. An interesting consequence

of Theorem 1 is that it relates the probability distribution of

the non-stationary process to that of the stationary process plus a minimal set of transient terms.

Next, we show in Theorem 2 how the conditional probability of can be obtained from a

characteristic form. To maintain the flow of the paper, the proof is postponed to Appendix D.

Theorem 2:The probability density function of conditioned on previous is given by

where


35/51

SIGNAL ESTIMATION

MMSE and MAP estimation are two common statistical paradigms for signal recovery. Since the

optimal MMSE estimator is rarely realizable in practice, MAP is often used as the next best

thing. In this section, in addition to applying the two methods to the proposed class of signals, we

settle the question of knowing when MAP is a good approximation of MMSE. For the estimation

purpose it is convenient to assume that the sampling instances ssociated with are included in the

uniform sampling grid for which we want to estimate the values of . In other ords, we assume

that , where is the sampling period in the definition of and is a positive integer. This assumption

does impose no lower bound on the resolution of the grid because we can set arbitrarily close to

zero by increasing .To simplify mathematical formulations, we use vectorial notations. We

indicate the vector of noisy/noiseless measurements by . The vector stands for the hypothetical

realization of the process on the considered grid, and denotes the subset that corresponds to the

points at which we have a sample. Finally, we represent the vector of estimated values by .

MMSE Denoising It is very common to evaluate the quality of an estimation method by means of

the mean-square error, or SNR. In this regard, the best estimator, known as MMSE, is obtained

by evaluating the posterior mean, or . For Gaussian processes, this expectation is easy to obtain,

since it is equivalent to the best linear estimator [24]. However, there are only a few non-

Gaussian caseswhere an explicit closed form of this expectation is known. In particular, if the

additive noise is white and Gaussian (no restriction on the distribution of the signal) and pure

denoising is desired , then the MMSE estimator can be reformulated as

where stands for with , and is the variance of the noise [25][27]. Note that the pdf , which is the

result of convolving the a priori distribution with the Gaussian pdf of the noise, is both


36/51

continuous and differentiable. B. MAP Estimator Searching for the MAP estimator amounts to

finding themaximizer of the distribution. It is commonplace to reformulate this conditional

probability in terms of the a priori distribution. The additive discrete noise is white and Gaussian

with the variance . Thus,

In MAP estimation, we are looking for a vector that maximizes the conditional a posteriori

probability, so that (26) leads to

The last equality is due to the fact that neither nor depend on the choice of . Therefore, they play

no role

in the maximization. If the pdf of is bounded, the cost function (27) can be replaced with its

logarithm without changing the maximizer. The equivalent logarithmic maximization problem is

given by

By using the pdf factorization provided by Theorem 1, (28) is further simplified to


37/51

Gaussian pdf becomes more evident as decays faster. In the extreme case where is Gaussian, is

also Gaussian convolution of two Gaussians) with a different mean (which introduces another

type of bias). The fact that MAP and MMSE estimators are equivalent for Gaussian innovations

indicates that the two biases act in opposite directions and cancel out each other. In summary, for

super-exponentially decaying innovations, MAP seems to be consistent with MMSE. For heavy-

tail innovations, however, the MAP estimator is a biased version of the MMSE, where the effect

of the bias is observable at high noise powers. The scenario in which MAP diverges most from

MMSE might be the exponentially decaying innovations, where we have both a mismatch and a

bias in the cost function, as will

be confirmed in the experimental part of the paper

MMSE Interpolation

A classical problem is to interpolate the signal using noiseless samples. This corresponds to the

estimation of where

( does not divide ) by assuming . Although there is no noise in this scenario, we can still employ

the MMSE criterion to estimate . We show that the MMSE interpolator of a Lvy process is the

simple linear interpolator, irrespective of the type of innovation. To prove this, we assume


38/51

In the case of impulsive Poisson innovations, as shown in (10), the pdf of has a single mass

probability at . Hence, the MAP estimator will choose for all , resulting in a constant signal. Inother words, according to the MAP criterion and due to the boundary condition , the optimal

estimate is nothing but the trivial all-zero function. For the other types of innovations where the

pdf of the increments is bounded, or, equivalently, when the Lvy density is singular at the origin

[20], one can reformulate the MAP estimation in the form of (30) as

The parameters of the above innovations are set such that they all lead to the same entropy value.

The negative log-likelihoods of the first two innovation types resemble the and regularization

terms. However, the curve of for the Cauchy innovation shows a no convex log-type function. C.

MMSE Denoising As discussed in Section IV-C, the MMSE estimator, either in the expectation

form or as a minimization problem, is not separable with respect to the inputs. This is usually a

critical restriction in high dimensions. Fortunately, due to the factorization of the joint a priori

distribution, we can lift this restriction by employing the powerful message-passing algorithm.

The method consists of representing the statistical dependencies between the parameters as a


39/51

graph and finding the marginal distributions by iteratively passing the estimated pdfs along the

edges [29]. The

Transmitted messages along the edges are also known as beliefs, which give rise to the

alternative name of belief propagation. In general, the marginal distributions (beliefs) are

continuous-domain objects. Hence, for computer simulations we need to discredited them. In

order to define a graphical model for a given joint probability distribution, we need to define two

types of nodes: variable nodes that represent the input arguments for the joint pdf and factor

nodes that portray each one of the terms in the factorization of the joint pdf. The edges in the

graph are drawn only between nodes of different type and indicate the contribution of an input

argument to a given factor. For the Lvy process, the joint conditional pdf is factorized as Note

that, by definition, we have . For illustration purposes, we consider the special case of pure

denoising corresponding to . We give in Fig. 4 the bipartite graph associated to the joint pdf (46).

The variable nodes depicted in the middle of the graph stand for the input arguments . The factor

nodes are placed at the right and left sides of the variable nodes depending on whether theyrepresent the Gaussian factors or the factors, respectively. The set of edges also indicates a

participation of the variable nodes in the corresponding factor nodes. The message-passing

algorithm consists of initializing the nodes of the graph with proper 1D functions and updating

these functions through communications over the graph. It is desired that we eventually obtain


40/51

the marginal pdf on the th variable node, which enables us to obtain the mean. The details of the

messages sent over the edges and updating rules are given in [30], [31].

SIMULATION RESULTS

For the experiments, we consider the denoising of Lvy processes for various types of

innovation, including those introduced in Section II-A and the Laplace-type innovation

discussed in Appendix E. Among the heavy-tail -stable innovations, we choose the Cauchy

distribution corresponding to . The four implemented denoising methods are 1) Linear minimum

mean-square error (LMMSE) method or quadratic regularization (also known as smoothing

spline [32]) defined as

for given innovation statistics and a given additive-noise variance, we search for the best for each

realization by comparing the results with the oracle estimator provided by the noiseless signal.

Then, we average over a number of

realizations to obtain a unified and realization-independent value. This procedure is repeated

each time the statistics

(either the innovation or the additive noise) change. For Gaussian processes, the LMMSE

method coincides with

both the MAP and MMSE estimators.

2) Total-variation regularization represented as


41/51

where should be optimized. The optimization process is similar to the one explained for the

LMMSE method. In

our experiments, we keep fixed throughout the minimization steps (e.g., in the gradient-descent

iterations). Unfortunately, Log is not necessarily convex, which might result in a no convex cost

function. Hence, it is possible

that gradient-descent methods get trapped in local minima rather than the desired globalminimum. For heavy-tail innovations (e.g., -stables), the Log regularizer is either the exact, or a

very good approximation of, the MAP estimator. 4) Minimum mean-square error denier which is

implemented using the message-passing technique discussed in Section V-C. The experiments

are conducted in MATLAB. We have developed a graphical user interface that facilitates the

procedures of generating samples of the stochastic process and denoising them using MMSE or

the variational techniques. We show in Fig. 5 the SNR improvement of a Gaussian process after

denoising by the four methods. Since the LMMSE and MMSE methods are equivalent in the

Gaussian case, only the MMSE curve obtained from the message-passing algorithm is plotted.

As expected, the MMSE method outperforms the


42/51


43/51

TV and Log regularization techniques. The counter intuitive observation is that Log, which

includes a no convex penalty function, performs better than TV. Another advantage of the Log

regularizer is that it is differentiable and quadratic around the origin. A similar scenario is

repeated in Fig. 6 for the compound- Poisson innovation with and Gaussian amplitudes (zero-

mean and ). As mentioned in Section V-B, since the pdf of the increments contains a mass

probability at , the MAP estimator selects the all-zero signal as the most probable choice. In Fig.

6, this trivial estimator is excluded from the comparison. It can be observed that the performance

of the MMSE denoiser, which is considered to be the gold standard, is very close to that of the

TV regularization method at low noise powers where the source sparsity dictates the structure.

This is consistent with what was predicted in [13]. Meanwhile, it performs almost as well as the

LMMSE method at large noise powers. There, the additive Gaussian noise is the dominant term

and the statistics of the noisy signal is mostly determined by the Gaussian constituent, which is

matched to the LMMSE method. Excluding the MMSE method, none of the other three

outperforms another one for the entire range of noise. Heavy-tail distributions such as -stables

produce sparse or compressible sequences. With high probability, their realizations consist of a

few large peaks and many insignificant sam-


44/51

ples. Since the convolution of a heavy-tail pdf with a Gaussian pdf is still heavy-tail, the noisy

signal looks sparse even at large noise powers. The poor performance of the LMMSE method

observed in Fig. 7 for Cauchy distributions confirms this characteristic. The pdf of the Cauchy

distribution, given by , is in fact the symmetric -stable distribution with . The Log regularizer

corresponds to theMAP estimator of this distribution while there is no direct link between the TV

regularizer and the MAP or MMSE criteria. The SNR improvement curves in Fig. 7 indicate that

the MMSE and Log (MAP) denoisers for this sparse process perform similarly (specially at small

noise powers) and outperform the corresponding -norm regularizer (TV). In the final scenario,

we consider innovations with and . This results in finite differences obtained at that follow a

Laplace distribution (see Appendix E). Since the MAP denoiser for this process coincides with

TV regularization, sometimes the Laplace distribution has been considered to be a sparse prior.

However, it is proved in [11], [12] that the realizations of a sequence with Laplace prior are not

compressible, almost surely. The curves presented in Fig. 8 show that TV is a good

approximation of the MMSE method only in light-noise conditions. For moderate to large noise,

the LMMSE method is better than TV.

CONCLUSION

In this paper, we studied continuous-time stochastic processes where the process is defined by

applying a linear operator on a white innovation process. For specific types of innovation, the

procedure results in sparse processes. We derived a factorization of the joint posterior

distribution for the noisy samples of the broad family ruled by fixed-coefficient stochastic

differential equations. The factorization allows us to efficiently apply statistical estimation tools.

A consequence of our pdf factorization is that it gives us access to the MMSE

estimator. It can then be used as a gold standard for evaluating the performance of regularization

techniques. This enables us to replace the MMSE method with a more-tractable and

computationally efficient regularization technique matched to the problem without

compromising the performance. We then focused on Lvy processes as a special case for which

we studied the denoising and interpolation problems using


45/51

MAP and MMSE methods. We also compared these methods with the popular regularization

techniques for the recovery of sparse signals, including the norm (e.g., TV regularizer) and the

Log regularization approaches. Simulation results showed that we can almost achieve the MMSE

performance by tuning the regularization technique to the type of innovation and the power of

the noise.We have also developed a graphical user interface

in MATLAB which generates realizations of stochastic processes with various types of

innovation and allows the user to apply either the MMSE or variational methods to denoise the

samples1


46/51

APPENDIX A

CHARACTERISTIC FORMS

In Gelfands theory of generalized random processes, the process is defined through its inner

product with a space of test functions, rather than point values. For a random process and an

arbitrary test function chosen from a given space, the characteristic form is defined as the

characteristic function of the random variable and given by

As an example, let be a normalized white Gaussian noise and let be an arbitrary function in . It is

well-known that

is a zero-mean Gaussian random variable with variance . Thus, in this example we have that P

(52) An interesting property of the characteristic forms is that they help determine the joint

probability density functions for arbitrary

finite dimensions as

where are test functions and are scalars. Equation (53) shows that an inverse Fourier transform of

the characteristic

form can yield the desired pdf. Beside joint distributions, characteristic forms are useful for

generating moments too:


47/51

APPENDIX E

WHEN DOES TV REGULARIZATION MEET MAP?

The TV-regularization technique is one of the successful methods in denoising. Since the TV

penalty is separable with respect to first-order finite differences, its interpretation as a MAP

estimator is valid only for a Lvy process. Moreover, the MAP estimator of a Lvy process

coincides with TV regularization only if , where and are constants such that . This condition

implies that is the Laplace pdf . This pdf is a valid distribution for the first-order finite

differences of the Lvy process characterized by the innovation with and because


48/51

Where is the modified Bessel function of the second kind. The latter probability density function

is known assymmetric variance-gamma orsymm-gamma. It is not hard to check that we obtain

the desired Laplace distribution for. However, this value of is the only one for which we observe

this property. Should the sampling grid become finer or coarser, the MAP estimator would no

longer coincide with TV regularization. We show in Fig. 9 the shifted functions for various

values for the aforementioned innovation where.

ACKNOWLEDGMENT

The authors would like to thank P. Thvenaz and the anonymous reviewer for their constructive

comments which greatly improved the paper.

REFERENCES

[1] E. Cands, J. Romberg, and T. Tao, Robust uncertainty principles:

Exact signal reconstruction from highly incomplete frequency information,

IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489509, Feb.

2006.

[2] E. Cands and T. Tao, Near optimal signal recovery from random


49/51

projections: Universal encoding strategies,IEEE Trans. Inf. Theory,

vol. 52, no. 12, pp. 54065425, Dec. 2006.

[3] D. L. Donoho, Compressed sensing,IEEE Trans. Inf. Theory, vol.

52, no. 4, pp. 12891306, Apr. 2006.

[4] J. L. Starck, M. Elad, and D. L. Donoho, Image decomposition via

the combination of sparse representations and a variational approach,

IEEE Trans. Image Process., vol. 14, no. 10, pp. 15701582, Oct. 2005.

[5] S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo, Sparse reconstruction

by separable approximation,IEEE Trans. Signal Process.,

vol. 57, no. 7, pp. 24792493, July 2009.

[6] E. Cands and T. Tao, Decoding by linear programming,IEEE Trans.

Inf. Theory, vol. 51, no. 12, pp. 42034215, Dec. 2005.

[7] L. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based

noise removal algorithms,Phys. D, Nonlin. Phenom., vol. 60, pp.

259268, Nov. 1992.

[8] D. P. Wipf and B. D. Rao, Sparse Baysian learning for basis selection,

IEEE Trans. Signal Process., vol. 52, no. 8, pp. 21532164, Aug.

2004.

[9] T. Park andG. Casella, The Bayesian LASSO,J. Amer. Stat. Assoc.,

vol. 103, pp. 681686, June 2008.

[10] V. Cevher, Learning with compressible priors, presented at the

Neural Inf. Process. Syst. (NIPS), Vancouver, BC, Canada, Dec. 810,

2008.

[11] A.Amini,M. Unser, and F.Marvasti, Compressibility of deterministic

and random infinite sequences,IEEE Trans. Signal Process., vol. 59,

no. 11, pp. 51395201, Nov. 2011.

[12] R. Gribonval, V. Cevher, and M. Davies, Compressible distributions

for high-dimensional statistics,IEEE Trans. Inf. Theory, vol. 58, no.

8, pp. 50165034, Aug. 2012.

[13] R.Gribonval, Should penalized least squares regression be interpreted

as maximum a posteriori estimation?,IEEE Trans. Signal Process.,


50/51

vol. 59, no. 5, pp. 24052410, May 2011.

[14] M. Unser and P. Tafti, Stochastic models for sparse and piecewise-

smooth signals,IEEE Trans. Signal Process., vol. 59, no. 3, pp.

9891006, Mar. 2011.

[15] M. Vetterli, P. Marziliano, and T. Blu, Sampling signals with finite

rate of innovation,IEEE Trans. Signal Process., vol. 50, no. 6, pp.

14171428, Jun. 2002.

[16] M. Unser, P. Tafti, and Q. Sun, A unified formulation of

Gaussian vs. sparse stochastic processes: Part IContinuous-domain

theory, arXiv:1108.6150v1, 2012 [Online]. Available:

http://arxiv.org/abs/1108.6150.

[17] M. Unser, P. Tafti, A. Amini, and H. Kirshner, A unified formulation

of Gaussian vs. sparse stochastic processes: Part IIDiscretedomain

theory, arXiv:1108.6152v1, 2012 [Online]. Available: http://

arxiv.org/abs/1108.6152.

[18] I. Gelfand and N. Y. Vilenkin, Generalized Functions, Vol. 4. Applications

of Harmonic Analysis. New York: Academic, 1964.

[19] K. Sato, Lvy Processes and Infinitely Divisible Distributions.

London, U.K.: Chapman & Hall, 1994.

[20] F. W. Steutel and K. V. Harn, Infinite Divisibility of Probability Distributions

on the Real Line. New York: Marcel Dekker, 2003.

[21] G. Samorodnitsky and M. S. Taqqu, Stable Non-Gaussian Random

Processes. London, U.K.: Chapman & Hall/CRC, 1994.

[22] M. Unser and T. Blue, Self-similarity: Part ISplines and operators,

IEEE Trans. Signal Process., vol. 55, no. 4, pp. 13521363, Apr. 2007.

[23] M. Unser and T. Blu, Cardinal exponential splines: Part ITheory

and filtering algorithms,IEEE Trans. Signal Process., vol. 53, no. 4,

pp. 14251438, Apr. 2005.

[24] T. Blu and M. Unser, Self-similarity: Part IIOptimal estimation of

fractal processes,IEEE Trans. Signal Process., vol. 55, no. 4, pp.

13641378, Apr. 2007.


51/51

[25] C. Stein, Estimation of the mean of a multivariate normal distribution,

Ann. Stat., vol. 9, no. 6, pp. 11351151, 1981.

[26] M. Raphan and E. P. Simoncelli, Learning to be Bayesian without

supervision, inProc. Neural Inf. Process. Syst. (NIPS), Cambridge,

MA, Dec. 45, 2006, pp. 11451152, MIT Press.

[27] F. Luisier, T. Blu, and M. Unser, A new SURE approach to image

denoising: Interscale orthonormal wavelet thresholding,IEEE Trans.

Image Process., vol. 16, no. 3, pp. 593606, Mar. 2007.

[28] D. Madan, P. Carr, and E. Chang, The variance gamma process and

option pricing,Eur. Finance Rev., vol. 2, no. 1, pp. 79105, 1998.

[29] H.-A. Loeliger, J. Dauwels, J. Hu, S. Korl, L. Ping, and F. R. Kschischang,

The factor graph approach to model-based signal processing,

Proc. IEEE, vol. 95, no. 6, pp. 12951322, Jun. 2007.

[30] U. Kamilov, A. Amini, and M. Unser, MMSE denoising of sparse

Lvy processes via message passing, inProc. Int. Conf. Acoust.,

Speech, Signal Process. (ICASSP), Kyoto, Japan, Mar. 2530, 2012,

pp. 36373640.

[31] U. S. Kamilov, P. Pad, A. Amini, and M. Unser, MMSE estimation

of sparse Lvy processes,IEEE Trans. Signal Process., vol. 61, no. 1,

pp. 137147, 2012.

[32] M. Unser and T. Blu, Generalized smoothing splines and the optimal

discretization of the Wiener filter,IEEE Trans. Signal Process., vol.

53, no. 6, pp. 21462159, Jun. 2005.

bayesian estimation for continuous-time

Documents