Isao YamadaTokyo Institute of Technology

MIIS 2015 July 5-6, 2015, Xidian University, Xi'an China

Two Spices of Convex Optimization For Certain Inverse Problems

A touch of spice brings miracle flavors to our great meal !

Wasabi Sushi

I am going to introduce Two Mathematical Spices :

1. Hybrid Steepest Descent Method

2. Algebraic Phase Unwrapping

which will bring Deeper & Wider applications of

Convex Optimization to Inverse Problems !

Good Luck !

Gauss(1777-1855)

Fourier (1768-1830)

Kalman(1930-)

Information Sciences, e.g. Signal Processing,

are growing with Mathematics !

Wiener(1894-1964)

Shannon(1916-2001)

P.L. Lions (1956-)

1994 Fields medal

Terence Tao (1975-)

2006 Fields medal

Newton(1643-1727)

von Neumann(1903-1955)

Euclid(Mid.4th century BC -Mid.3rd century BC)

Extraction of Valuable Information from Observed Data !

has been a most successful strategy !

A Goal of Information Sciences

MinimizeSubject to

Optimization

We like BEST Solution !

Measurement(Degraded image)

Signal Processing

is searching for BEST

Restored Image

All available knowledge

Physical Constraints, Statistical Property, Perceptual Characteristic

e.g.

e.t.c.

Signal Analysis

Fourier, Wavelet, Curvelet, etc

Most information can be expressed in terms of a Point ( Stone) in a Vector Space !Example of

Optimization is similar to Mountain Climbing !

Minimize

Maximize

Vector Space

Find a Stone on the Highest place !

Many Signal Processing Researchersare TRYING to

How can we find BEST solutions ?

Step 1. Design an Optimization Problem

Step 2. Translate (P1) into a computationally tractable problem (P2)

Minimize

Subject to(P1)

Use all available knowledge

Step 3. Apply a suitable algorithm to (P2)

Best in what sense ?

(P1)

Collection of Computationally Tracable Problems ?

Translation Based on Variety of

Mathematics

(P2)

Step 2

in in

Main Stage for Step 2

David Hilbert

: Real vector space equipped with Real Hilbert Space

Inner Product

Norm

J. von Neumann

NOTE: Completeness always holds if

converges to if

You can define flexibly !

is Closed in

Proper

Lower Semi-continuous

Convex

Convex Optimization defined in Real Hilbert Space

Minimize

where

Let's focus on

How can we find BEST solutions ?

Step 1. Design an Optimization Problem

Step 2. Translate (P1) into a computationally tractable problem (P2)

Minimize(P1)Use all available knowledge

Step 3. Apply a suitable algorithm to (P2)

Best in what sense ?

where is nonexpansive i.e.

A Common Strategy in Step 2 for Convex Optimization

Translation Based on

Convex Analysis &

Monotone Operator Theory

(P2)Step 2

Minimize Find

(P1)

: a certain Nonexpansive Operator defined with proximity operators /gradients

: a certain Nonlinear Operators

where

Step 2

More precisely, the solution sets of convex optimization problems

can often be expressed as

Example (Step 2) : Convex Feasibility Poblem

Metric projection onto a closed convex set

is

For its best approximation in a closed convex set exists uniquely.

NOTE: Computational issue depends on shape of set !

Nonexpansive

s.t.

A Common Principle in Step 3

Suppose isNonexpansive

Thenfor any

Note:This fact has been a key of many algorithms.

Theorem(Krasnosel’skii-Mann, e.g. [Groetsch'72])

POCS *Bregman, Soviet Math. Dokl (1965)*Gubin, Polyak, Raik, USSR Comp.Math. Math Physics (1967)

Example (Step 3) : Convex Feasibility Poblem

can be seen as an example of K-M Alg !This is because

Most successful convex optimization algorithms : Proximal Splitting Algorithms

(Forward backward splitting/Primal-dual splitting/ADMM etc)

can also be seen as K-M Alg (Step 3)applied to

: computable Nonexpansive Operator defined with proximity operators /gradients

: certain computable Nonlinear Operators

where

Step 2

Metric projection

Proximity operator

Forward-Backward splitting operator

Douglas-Rachford splitting operator (ADMM: Dual Variant [Fortin-Glowinski '83])

Primal-Dual splitting Operator

Applicability of Nonexpansive Operators is incleasing with advances in Convex Optimization

Great !

Proper Lower semicontinuous Convex function.

Proximity Operator (J.J.Moreau '62)

Good News (Proximable functions)

Example 1

Example 2

Example 3

Let

FACT (Convex envelope of )

Largest convex lower bound !

Example 1

Convex envelope of a function

All mappings reviewed so far belong to Averaged Nonexpansive Operators !

: nonexpansive,

is

: nonexpansive map s.t.

In this case, K-M Alg. guarantees :

Then

Let

def

29

Theorem ([Ogura-Yamada'02 / Combettes-Yamada'14])

- averaged - averaged

Let

Then

A Key for Broad Applications of K-M Alg.

behind the strategies in the Modern Convex Optimization algs,

(a ) Nonexpansive Operators have incredible power to describe, as their fixed point set, the closed convex sets of great value in Hilbert Space

(b) K-M alg allows us to access only one unspecial point in the fixed point set, but does not reveal the other points in the gold mine !

suggests

The fixed point expression

Digital Camerascan describe a CAT in 3 dimensional space in terms of a digital data !

Digital camera can focus on the face of cat by zooming in the image !

Mathematicians can describe a closed convex set in Hilbert space in terms of nonexpansive operators !

Mathematicians can focus on the intersection of multiple closed convex sets in Hilbert space by using nonexpansive operators !

K-M alg allows us to access only one unspecial point in the valuable closed convex set but does not reveal other points in the gold mine !

remain a mystery !

This situation of K-M algorithm is very similar to Fortune Cookie !

All look same but they are very different in many aspects !

Actually VERY DIFFERENT

K-M alg allows us to access only one unspecial point in the valuable closed convex set but does not reveal other points in the gold mine !

Results are VERY DIFFERENTin many aspects

Can we choose best one without crunching all cookies ?

The answer is YES ! Hybrid steepest descent method can accomplish this mission !

K-M algorithm

Hybrid Steepest Descent Method

The 1st Spice is here !

Hybrid Steepest Descent Method(Yamada et al,'96, Deutsch-Yamada'98, Yamada'01, Yamada-Ogura'04 etc)

can minimize over

where

Lipschitz ContinuousQuasi-Nonexpansive mapping

Slowly decreasing

Smooth Convex Function

1. This is a generalization of [P.L. Lions'77].2. This can select a very best solution among all fixed points !

More Spicy Principle in Step 3

If is nonexpansive,

is an extension of simplest case:

[Halpern'67/Reich'74/ Lions'77/Wittmann'92/Combettes'95/Bauschke'96]

can be seen as a remarkable extension of Projected Gradient Method [Goldstein '64].

Moreover, alternative form of HSDM

Hybrid Steepest Descent Method (HSDM) [Yamada'01]

Nonexpansive with bounded

Smooth Convex function, s.t.

Theorem (nonstrictly convex, [Ogura-Yamada'03]) Suppose

satisfies

where

Then

is desired to be minimized additionally.

Suppose

Minimize

Subject to

NOTE: is not in usually !

Hierarchical Optimization

Find a point inStandard Optimization

Find a Hardest Stone on the summit of a mountain !

(= Hierarchical Optimization)

: Bounded Linear Operator

Primal Dual Splitting

Primal Problem

Suppose

Dual Problem

(Condat '13/ Combettes, Vu, ... etc)

where

Under a certain mild condition,

Fenchel-Rockafellar Conjugate of

Example [Condat '13]

Define with

and by

where

Then

by

Krasnosel'skii-Mann Theorem guarantees :

SupposePrimal Dual Splitting is applicable to general case.

Masking Operatordiscarding 40%Pixels randomly

Gaussian Noise

Application to Image Restoration

Step 1

Total VariationData fidelity

[Yamada-Ono Eusipco'13]

Step 1a

Total VariationData fidelity

Step 1bPrimal Dual

splitting operator

Apply Krasnosel'skii-Mann algorithm orStep 2

Hybrid Steepest Descent Method to

Primal Dual splitting operator

+Krasnosel'skii-Mann

Primal Dual splitting operator

+Hybrid Steepest Descent Method

Promoting Sparsity of Curvelet Coefficients

Algebra helps Convex Optimization recover 2D Phase Surface from its Wrapped Samples

You do not have enough time ! Move on the 2nd Spice ! Otherwise …...

Smooth phase recovery from its wrapped samples (2D-Phase Unwrapping) has been challenging.

available only at simply connected

Smooth surfaceWrapped Samples

(Observable) (Unknown : must be estimated)

At each

Challenge in 2D Phase Unwrapping

is available.

Only

Find a smooth satisfying

2D-Phase Unwrapping

2D-Phase Unwrapping is a key for estimating Crucial physical information such as :

Degree of Magnetic Field Inhomogeneity in water / fat separation in MRI

Surface Topography as measured by - Interferometric Synthetic Aperture Radar (InSAR) - Interferometric Synthetic Aperture Sonar (InSAS)

Profile of Mechanical Parts by X-ray.

Simple Observation for Convex Optimization [Kitahara and Yamada'13]

where

is available at each

is available at each

is available at each

Theorem 1 [Kitahara-Yamada'13]

Let satisfy

satisfying

Then for an arbitrarily fixed

There exists a unique

NOTE

Poincaré (1854-1912)

This is scalar potential of a vector field !

Proposed approach to 2D-Phase Unwrapping

Step 2 (Numerical Integration)

where and

Step1 (Convex Optimization)

subject to

Compute

as an estimate of the unwrapped phase

Suppress Rapid Local Change of

On Step 1(Convex Optimization: Example) Find

All functions with continuous fourth derivatives differentiated no more than twice w.r.t. each variable [Pretlova '76].

Minimizes

Subject to

Bicubic SplineA Generalized Hermite-Birkhoff Interpolation Theorem

　

can be decomposed into a finite sum of integrals of the following type:

where

On Step 2 (Numerical Integration)

Polynomials

0 0.2 0.4 0.6 0.8 1-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

t

A(t

) [r

ad/]

Can we really accomplish the Numerical Integration ?

Indeed, we face difficulty as seen in a typical example of the unwrapped phase !

The answer is YES !Extend Sturm's idea on the use of my GCD algorithm !

Algebraic Phase Unwrapping [Yamada et al '98], [Yamada-Bose '02], [Yamada-Oguchi'11], [Kitahara-Yamada'15]

Sturm(1803-1855)

The 2nd Spice is here !

Theorem 2 (Algebraic Phase Unwrapping)

whereNumber of Sign Changes in Generated by

Algorithm 1

A Modification of the Euclidean Algorithm for Phase Unwrapping

A Modification of the Euclidean Algorithm for Phase Unwrapping

Polynomial Division may cause Certain Numerical instabilities !

Alternative Expression of Algebraic Phase Unwrapping

We need only

but do not need coefficients of for computation of

Key Observation

If the polynomial sequence is Regular, i.e,

Then

Note: More general relation is also given [Kitahara & Yamada '13].

Theorem 3 (Alternative Expression,[Kitahara & Yamada '13] )

We can express without using polynomial division.

Habitch '48

Collins '67

Brown-Traub'71

Algebraic Phase

Unwrapping

Thales(c. 624-546 BC) Euclid

(c. 323-283 BC) Fourier (1768-1830) Sturm

(1803-1855)

Poincaré (1854-1912)

A self-portrait of Hokusai (1760-1849)

at the age of 82

Numerical Example

2.5 km

30 m

14 m

Proposed method (STEP 1) Bivariate Spline Interpolation

Bicubic Interpolating Spline as a minimizer of

in some functional space

Estimate of Unwrapped Phase by Proposed Algebraic Recovery

Perfectly smooth phase surface is finally

obtained after the long journey !

Proposed method (STEP 2) Algebraic Phase Unwrapping

Topographic Mapping from Interferometric SAR

Elementary Geometry tells us

a way to translate phase into hight !

Topographic Mapping from Unwrapped Phase by the Proposed Method

Wow !Beautiful Mt.Fuji

Mathematical Developments of Hybrid Steepest Descent Method

● I. Yamada, The hybrid steepest descent method for the variational inequality problem over the intersection of fixed point sets of nonexpansive mappings, Studies in Computational Mathematics, vol. 8, 2001,pp. 473-504.

● I. Yamada, N. Ogura, N. Shirakawa, A numerically robust hybrid steepest descent method for the convexly constrained generalized inverse problems, Contemporary Mathematics 313, pp.269-305, AMS,2002.

● I. Yamada, N.Ogura, Hybrid steepest descent method for variational inequality problem over the fixed point set of certain quasi-nonexpansive mappings, Numer Funct Anal Optim, vol.25, pp. 619-655, 2004.

● C. Chidume, Geometric Properties of Banach Spaces and Nonlinear Iterations (Chapter 7:Hybrid steepest descent method for variational inequalities), Lecture Notes in Mathematics, vol.1965, Springer 2009.

● P.E. Mainge, Extension of the Hybrid steepest descent method to a class of variational inequalities and fixed point problems with nonself-mappings, Numer Funct Anal Optim, vol.29, pp.820-834, 2008.

Many more …..

Applications of Hybrid Steepest Descent Method

to Hierarchical Convex Optimizations in Signal Processing

1. I. Yamada, M. Yukawa, M. Yamagishi, “Minimizing the Moreau envelope of nonsmooth convex functions over the fixed point set of certain quasi-nonexpansive mappings," In: Fixed Point Algorithms for Inverse Problems in Science and Engineering (H.H.Bauschke, et al., eds.) , pp. 345-390, Springer-Verlag, 2011.

2. S. Ono, I. Yamada, “Hierarchical Convex Optimization with Primal-Dual Splitting," IEEE Trans. Signal Process., vol.63, no.2, pp.373-388, 2014.

3. I. Yamada and K. Oguchi, “High-resolution estimation of the directions-of-arrival distribution by algebraic phase unwrapping algorithms,” Multidimensional Systems and Signal Processing, vol.22, no.1-3, pp.191-211, 2011.

2. I. Yamada and N.K.Bose,“Algebraic phase unwrapping and zero distribution of polynomial for continuous-time systems,'' IEEE Trans. Circuits and Systems 1: Fundamental Theory and Applications, vol.49, no.3, pp.298-304, 2002.

1. I. Yamada, K.Kurosawa, H.Hasegawa and K.Sakaniwa, “Algebraic multidimensional phase unwrapping and zero distribution of complex polynomials – Characterization of multivariate stable polynomials," IEEE Trans. Signal Processing, vol.46, no.6, pp.1639-1664, 1998.

Algebraic Phase Unwrapping

4. D. Kitahara and I. Yamada, “Algebraic phase unwrapping along the real axis - Extensions and Stabilizations,” Multidimensional Systems and Signal Processing, DOI 10.1007/s11045-013-0234-7 (43 pages, April, 2013).

5. D.Kitahara and I. Yamada, Algebraic phase unwrapping over collection of triangles based on two-dimensional spline smoothing, ICASSP 2014.

6. D.Kitahara and I. Yamada, A virtual resampling technique for algebraic two-dimensional phase unwrapping, ICASSP 2015.

Minimize

Subject to

Generalized Hermite-Birkhoff Interpolation Problem

Why Piecewise Polynomial ?

is given as a Polynomial Spline of degree 2m-1 !

Multidimensional extensions & applications

([Atteia '68], [Laurent '69], [Ritter '69])

[Pretlova '76], [Wahba'90], 　[Unser'99], [Lai & Schumaker'07] etc

On Step 1 (contd)

Example 1 (Expression of Unwrapped Phase )

Algorithm 1 generates

Example 1 (Contd. )

Functional Data Analysis Process

1. Select Basis Set (Functional space behind Data )

2. Select Smoothing Operator (To surppress rapid local change of model function ) – e.g., differential equation– equivalent to a Bayesian prior over coefficients to estimate

3. Estimate coefficients to optimize some objective function

4. Model criticism, residual plots, etc.

5. Hypothesis testing