stability and conditioning in numerical · pdf filestability and conditioning in numerical...

Stability and Conditioningin Numerical Analysis

D. Trigiante

University of Florence

[email protected]

September, 2005.

Rodi – Stability and Conditioning. – p.1/73

"... It is by looking into the sameproblem from different points ofview that one arrives to acomplete insight of it. "

Euler


Stability vs. Conditioning

The terms stability and conditioning are used with a variety ofmeanings in Numerical Analysis.They have in common the general concept of the response ofa set of computations to perturbations arising from

the data,

the specific arithmetic used on computers.

They are not synonymous.


Stability

In Mathematics the notion of stability derives from thehomonymous notion in mechanics.

It regards the behavior of the motion of a system when it ismoved away from the equilibrium.Three ingredients enters in the definition, i.e.

the existence of a reference solution, i.e. the equilibrium;

the perturbation of the initial status (the initial conditions);

the duration of the motion, which is supposed to beinfinite.


Stability

The behaviors around the equilibrium may be many:

stability (marginal stability), uniform stability,

asymptotic stability (AS), uniform AS,

contractivity,

orbital stability,

instability,

� � �


General Perturbations

More general perturbations are also considered in thequalitative theory of dynamical systems.

The concept of stability under perturbation of the wholesystem, often called total stability, is also considered.

This kind of perturbation is frequent in Numerical Analysiswhere the source of errors, due to the computer arithmetic,can be seen as perturbation of the whole set of computations.

In other words, a numerical algorithm is not only perturbed by

the errors in the data, but also with respect to the errors arising

in the process of computations.


Conditioning

Many problems, however, do not last for a long (in principle,infinite) time, and/or do not have an equilibrium.

The above concept of stability do not apply, as it stands.

The non numerical analysis distinguishes such problems inwell posed and ill-posed, according whether the solutiondepends continuously on data or not.This is not enough for the Numerical Analysis purposes,where a more refined distinction is needed.The numerical analysts would like to know if suchdependence, although continuous, may result disastrous forthe error growth.

This requires the notion of Conditioning.


Stability and Numerical Analysis

Often, in the N.A. literature, two or more concepts aresuperimposed.For example, concepts such as discretization (in the casewhere the original problem is continuous), the stability of thealgorithms and the ability of the arithmetic systemimplemented on the computers to perform operations withacceptable relative errors are often strictly tight together.

We almost completely agree with the following statements

taken from Lax and Richmyer (written in 1956!).


Lax and Richmyer, 1956

We shall not be concerned with rounding errors..., but it will be

evident to the reader that there is an intimate connection be-

tween stability and practicality of the equations from the point

of view of growth and the amplification of rounding errors...

(Some authors) define stability in terms of the growth of round-

ing errors. However we have a slight preference for the (our)

definition, (i.e. independent on the rounding errors) because it

emphasizes that stability still has to be considered, even if the

rounding errors are negligible, unless, of course, the initial data

are chosen with diabolic care so as to be exactly free of those

components that would be amplified if they were present.Rodi – Stability and Conditioning. – p.9/73

Dahlquist

A similar concept was expressed in the same years byDahlquist.

"The most fundamental is the distinction between instability inthe underlying mathematical problem and instability in analgorithm for the (exact or approximate) treatment of theproblem".

We shall then avoid, when unnecessarily, to consider roundingerrors.


Classes of Problems

Asymptotically stable Problems (AS),

Marginally Stable Problems,

Unstable Problems,

Boundary Value Problems,


Algorithms and Asymptotic Stability

More often than it can be thought, numerical algorithms canbe considered as discrete dynamical systems around criticalpoints. (equilibria).

The spaces where such dynamics are placed may vary con-

siderably, ranging from

�� or

� �

to the space of

��

real or

complex matrices.


Example 1, trivial

��

where � � �

or

�

� is real or complex.

The equilibrium is at the origin and it is

AS for

� � �

stable for

� � �

unstable for

� � ��


Example 2, less trivial

��

where � � � �

and

� � � � � � � � � �

� � �

��

� �

�

� . . .. . .

......

. . .. . .

�

�

� ��

� � � � ��

The equilibrium is the origin in� �

and it is AS for

� � � stable for

� � �

and unstable for

� � ��

Not uniformly with respect to the dimension!

In the case of PDE, this makes the difference.


Example 3, time scales

��

�

�

� � ��

��The origin is A.S., but a generic solution approaches theequilibrium with two different modes, one slow and the otherfast.


Example 4, splittings

Iterative methods for the solution of linear systems

��

fit very well in the framework of AS dynamical systems.

The splitting techniques are nothing but strategies totransform the solution ��

� �

into an AS equilibrium point ofan appropriate dynamical system.

What is usually less emphasized is that, even in the so calleddirect methods, the AS plays a central role, especially whendealing with structured matrices.


What’s wrong with marginal stability?

We have said that AS appears often in Numerical Analysis,although problems which are naturally marginally stable oreven unstable appear as well.

Naturally means that they are unstable by themselves and notbecause of previous wrong choices, for example a wrongdiscretization.In principle, small perturbations may lead them to be unstable.In the real world, however, there are very important marginallystable systems which are far to be unstable and onlyperturbations of improbable nature (!) would turn them fromstability to instability. In other words, many marginally stableproblems need to be safely computed.


What’s wrong with unstable problems?

Unstable solution may be computed within the limit imposedby the computer arithmetic.

Computer arithmetic is able to represent relatively wellnumbers in a certain fixed range. When the numbers becomelarge such representation become poorer and poorer until itmay not have a single digit in common with the representednumber.

The use of large numbers is then unsafe on the computers.


Examples of unstable problems

The power method to find the maximum eigenvalue of amatrix is an unstable problem.In this case the required information is only partial, i.e. thegrow rate of the norm of a vector, instead of the norm ofthe vector itself.The algorithm is designed in order to avoid large numbers.

The Miller’s problem provides an example of unstableproblems.It arose in the fifties in computing recursively the values ofthe Bessel functions. It is worth to mention it because ofthe peculiar aspect of the devised stable algorithm.


Miller’s problem

In the simplest form the problem is

��

� � � � � � � ��

�� Approximate solution.Let

� � � � � � � � ��

� ��

��

� ��

��


Conditioning

The notion of stability is not enough. A more general conceptis needed.Let the discrete solution be � � � � � � � � � � ��

We shall consider the following two parameters:

��

� � ��

� ��

��

��

� �� (1)

They may provide two different types of information, especially

in the case of AS.


Qualitative vs Quantitative Information

Consider for example � � � � � ��

If we are interested in knowing the maximum value reachedby the solution, we immediately obtain

� � � ��

with

��

in case of asymptotic stability and

� � growingexponentially with

�

in the case of instability.We may be interested in knowing how fast the solution returnsto the equilibrium in case of A.S. Such information is providedby � � which, in this case becomes:

� ��

� � � � � �

� � � � � � �

in case of AS.


Information measure

� is a sort of information measure in the sense that itssmallness informs us that we are using large values of

�� i.e.we are computing terms already too small (smaller, forexample, than the machine precision).In the case of AS, by posing

� � � �� we have

� � � � ��

� � � � � �

� �

which, if

� �

is large, becomes � � � � � � � � � ��

The quantity

�

is already known in NA and is usually calledasymptotic rate of convergence. It is useful because itsinverse measures approximatively the number of iterationsneeded to approximate the equilibrium within a precision of

� � ��


� and machine precision

If the machine precision is

� � � �

then the number of iterationnecessary to reach the machine precision is

� � � � � ��

One has then

� �

� ��

��

It is useless to use

� � � � It follows that

� should be of the order of � ��

The ratio��

��

will be called stiffness ratio. In the example ��


Stiffness

Both

�� and � measure the sensitivity of the solution withrespect to change of the initial condition, although suchmeasure may change considerably.The problems will be classified as follows

well conditioned if

� � and � are small and of the sameorder;

stiff if �� is large (in the example > �);

ill conditioned if both parameters are large.

The definition of stiffness is unusual in this context. We willshow later that it fits with the usual definition in the context ofnumerical methods for ODE.


Example of stiff problem

0 5 10 15 20 25 30 35 40 45 500

1

2

3

4

5

6

7

8

9

10α=0.1

γd = 1/5 γ

d = 1/40

Figure 1: There is no need to use smaller value of ��


Entropy and information content

Let associate to � � � � � � � � � � � � � � �

the frequencies

� � � � � � � � � This can be obtained as usual by dividing theinterval

� � � � �

in subintervals and counting the numbers ofpoints in each of them. The entropy is given by

� � � � � � � ��

� � � � measures the expected values of information of thedistribution.It is maximum when the distribution is uniform, i.e. when ineach subinterval fall the same number of points, for exampleone.

The maximum value is attained at

� � � ��


0 50 1000

0.5

1

1.5

Entropy

N*+1


Optimal mesh

0 2 4 6 8 10 12 14 16 18 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

h1 h

2 h3

h4

To the distribution on the � axis, it will correspond a meshon the � axis.

. Rodi – Stability and Conditioning. – p.29/73

Reduction of stiffness

Stiffness may be reduced by reducing

�

. Consider themultiscale example. The two modes are

� �

and� � �

� �

After the fastest mode has become less then the machineprecision (say for � � � � ), on may compute the slower modenot at each step but every � steps, i.e. one may define a newvariable � � � �� In the new variable the slower modeequation becomes

� � � � � �

� � � � �

It is then enough to choose � such that � � ��

� � �

tohave comparable stiffness ratio for the two modes.


Multiscale case

0 10 20 30 40 500

1

2

3

4

5

6

7

8

9

10stiff

0 10 20 30 40 500

1

2

3

4

5

6

7

8

9

10non stiff

Time scale change a stiff problem to a non stiff one


Ill conditioned problem.

As example of ill conditioned problem consider the case of

��

where � � � �

and

� � � � � � � � �

The solution is

��

��

��

��

��

��

Even if

� � � the solution may become large for large

� In

this case both parameters� � and � are large.

This is not the case if��

��

��

�� Note that � � � �� is the root of

the polynomial associated to

�


Ill conditioned

0 10 20 30 40 50 600

0.5

1

1.5

2

2.5

3

3.5

4

4.5x 10

4 ||y||

��

��


More general problems

The same approach of using two conditioning parameters canbe used for more general problems.In the case of boundary value problems, the parameters

� �

and � can be defined as well. In other words, the notion ofconditioning continues to hold, as well as the notion ofstiffness.If ��

is a solution of a discrete BVP depending on someboundary condition � � � �

we can define

��

��

��

��

� ��

��

��

��

� ��

� �

where

��

�

is any norm in

� �


LDU factorization

Let

�

be a real

� � �

matrix. Consider the followingsequence,

� � � � � � � � � �

��

��

where

� � �

��

��

��

�

� � ��

��

� ��

��

��

��

� � � ��

� ��

� ��


��

��

��

�

� ��

� � �

with � � and

�� appropriately defined vectors. The final result is� �

This example differs from the others for the following reasons:

1. the underlying space is the space of real

� � �

matrices;

2. the motion is not autonomous, since the factor matrices

� � and

�� change at each step.

The problem here is in the possibility that the entries ofproduct matrices

� � and

�� may become large and then notwell represented in the computer arithmetic.


Grow factors

The dynamic is not uniquely defined in the sense that there isa certain freedom in the choice of both

�� and

�� One is able

to maintain small the entries of

�� by the pivoting strategy, but

still the entries of

�� and consequently

�� may grow.Parameters which monitor such grow, called grow factors,have been defined by Wilkinson and more recently by Amodioand Mazzia. When such parameters grow exponentially with�

the problem is said unstable. The parameter

� � nowbecomes:

��

� � ��

� ��

��

� � ��

which coincides with the grow factor used by Amodio andMazzia.


Tridiagonal Toeplitz matrices

Let

� � � � � � � � � � �

with � � � � real.�

is a tridiagonal

� � �

Toeplitz matrix.Consider the problem

� � � � � � � � � � where

� � � � � � � � � �

is the first unit vector in

� � It is equivalent to solve the

following discrete BVP, where � � are the entries of the vector � �

��

��


Tridiagonal Toeplitz matrix 2

The solution of the above problem can be expressed bymeans of the roots of the polynomial

� � � � � ��

Let � � � � � be such roots, the solution is � � � � � � � � � � � � Afterimposing the boundary condition, we obtain,

��

��

� ��

� �

��

��

� ��



1. in order to the solution being bounded, the two roots needto be distinct. Actually they should have distinct moduli inorder to prevent the denominator becoming too small;

2. the solution is essentially generate by the root of minimalmodulus � � �

3. if

� � � �

and

� � �

the solution is bounded with respectto

�

Even when � � � � � small perturbations of initial data, will cause

perturbation growing as � � � and not as the faster mode � � �



Both

�� and � behaves as � ��

They are bounded with respect to

�

when

� � ��

Considering that the vector

�

is the first unit vector in

� �� this

implies that the vector � is the first column of

� ��

By using sequentially the vectors

�� and applying similararguments, we arrive to the conclusion that

� � � � � � � �

is a necessary and sufficient condition to have

� � ��

boundeduniformly with respect to

�

Let

� �� and �� be the corresponding conditioning parameters,we have

� � � � � � � � � ��

� ��



When each

� �� is bounded with respect to

�� the matrix is saidwell conditioned (

� � � � � � � �

).When either

� � � � � � � � � or

� � � � � � � � � � � � ��

maygrow linearly with

� We say that the matrix

�is weakly well

conditioned.Let

� � � � � � � � � ��

� � � � ��

� � �

What kind of information may provide this parameter?Let us summarize with a specific example.


Stiffness and Toeplitz matrices

� � �

��

� � �

� � � � � � ......

. . . . . . �

� � � � � �

� � � � � ��

��

−10 −8 −6 −4 −2 0 2 4 6 8 10−1.5

−1

−0.5

0

0.5

1c(α)

α


−10 −9 −8 −7 −6 −5 −4 −3 −2 −1 0 1 2 3 4 5 6 7 8 9 100

1

2

3

4

5

6

7

8

9

10|z

1|

|z2|


−10 −5 0 5 1010

−5

100

105

1010

1015

kγk/γcond(A)


0 20 40 60 80 100−1

−0.5

0

α=−1

0

0 50 100

0

50

100

0 20 40 60 80 100−1

−0.5

0

α=−3

0 50 100

0

50

100

0 20 40 60 80 100−5

0

5

α=−1

.5

0 50 100

0

50

100


0 20 40 60 80 100−2

0

2x 10

6

α=1

0 50 100

0

50

100

0 20 40 60 80 100−1

−0.5

0

0.5

α=4

0 50 100

0

50

100

0 20 40 60 80 100−5

0

5x 10

14

α=10

0 50 100

0

50

100


More general Toeplitz matrices

Let

� � be a generic banded Toeplitz matrix.

The matrix

�

is well conditioned if the roots of the polynomialassociated to the matrix are such that

� � of them are insideand

� � are outside the unit disk.

Further generalization to define well conditioning in a regioncan be done as follows.


Conditioning in a region

Consider the matrix

� � in the previous example, and let

� � � � � � with � real and

�

complex.By posing

� � � � � � � the problem becomes

� � � � � � � � �Now the roots of the polynomial � � � � � � � � � � � � � � � � � � �

depend on the complex parameter�

In correspondence of the values of

�

such that

��

��

� ��

the

solution does not exist


Boundary locus

Suppose now that we ask to the system to be well conditionedfor all value of

�

in a prefixed region, for example� �

This means that we ask that for all

� � � � the two roots needto be one outside and one inside the unit circle.

The best way to check if the two roots remain in the sameposition with respect the unit disk is to monitor if they crossthe unit circle for

� � � � This leads to consider the map

� ��

� � �

and the curve (boundary locus)� � � � ��


The curve

�

is the locus of value of

�

where one or both theroots � � � � � � � � � � �

cross the unit circle.

If such curve completely lies in

� �� then the matrix

� ��

will be well conditioned in

�

.


0 0.5 1 1.5 2 2.5 3−2

−1.5

−1

−0.5

0

0.5

1

1.5

2BC with eigenvalues

real(λ)

imag

(λ)

Typical boundary locus


Generalization

Let

� � � � � � � � �

� � and

� � two banded Toeplitz matrices;

� � � � � �

� � �

the maximum value of the correspondingbandwidths.

� � � � and � � � � the two polynomial of degree

� �

� � � � � � � � � � � � � � � � �

� � � � ��

��

� � is the number of lower diagonals and

� � the number ofupper diagonals of

� �


Theorem.

� � is well conditioned in a region

�

of thecomplex plane, for all � � � � iff one of the two conditions holdtrue:

a)

� � � � �

b)

� � of the roots of the polynomial � � � � � �are inside the unit

disk and

� � are outside.

If

� � � �

have common points and if

�

is a Jordan curve, thenthe matrix is weakly well conditioned.


Conditioning of continuous problems

� � � � � � � � � � � � � � ��

The definitions of critical points, stability, asymptotic stabilityand instability apply to this case as well, only considering thatnow the variable

�

is continuous.

Moreover,the parameters

�� can be defined in a way very similar

to the discrete case.


Stiffness of continuous problems

� � � � � � � � � � � � � � ��

��

� � � � �

� �

�

� � � �

��

��

In the non scalar case, �� is the ratio of largest and smallest

eigenvalues.


Discrete representation

Definition. A continuous problem will be saidwell-represented by a discrete problem if

��

Let h � � � � � � � � � � � � � be the mesh.Both

�� and � depend on h

The request may lead to define the optimal mesh.

This has been discussed by Francesca Mazzia in her talk.


Not well represeted problem

��

� ��

�

� �

� � � � �

� ��

� � � � � � �

� � � � � � � � � � � � � � � ��

Let � be the unknown second component,

� � � � � � � �

� � �

� � � � � � � � ��

� � � � ��

� � � � � � � � � � � � � �

� � � � � � � � ��

� � � � ��


The solution of problem is

� � � � � � ��

��

�which corresponds to � � � � �. The problem is very wellconditioned as BVP

� ��

� �

but very ill conditioned withrespect to � � (shooting method).Let

� � � � � � ��

� � � � � � � � � � � � � � � ��


0

1

2

3

4

5

6

7

8x1043

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

mu

e


Test equations

In the process of constructing numerical methods the notionsof stability and asymptotic stability have played a central role.In fact the numerical methods have been essentially modeledon the two test equations

� � � � �

� � � � � � ��


Perron Theorem

In the first case the origin is a (marginally) stable equilibrium,while it is AS in the second case. There is a fundamentaldifference between the two cases.The second case may be considered as a modelrepresentative of more difficult (non linear) equations. Ageneral theorem of dynamical systems (Perron) support suchmodel.Theorem. Let

� � � � � � � ��

where � � � � � �

and

� � ��

��

� � � ��

If the �� is A.S. for the the linear part, then it is AS for thecomplete equation.


Perron & Ostrowsky

It is not by chance that a theorem essentially similar to theabove one was established independently by Ostrowski in thecontext of iterative procedures to find zeros of nonlinearequations (see Ortega ). The linear test equation is then morerepresentative.No similar general results are available in the case of marginalstability.Test equation � � � �

has, however, played an important roleessentially in proving the convergence of the methods.


LMM for ODEs

Let �� the the initial data and

� � � � � � � � � � � � � � � � � � � � � � � be the final data.A LMM applied to the test equation with the above boundaryconditions leads to the following discrete problem,

� � � � � � � � � � � � �

where

� � and

� � are Toeplitz matrices having

� � lower non

zero diagonals and

� � upper non zero diagonals. The case

� � � �

corresponds to the classical choice of using discrete

IVPs.


The non zero entries of the matrix

� � are the coefficients ofpolynomial � � while those of

� � are the coefficients of thepolynomial � To the matrix

� � � � � � � � � � � � we may applyeither the generalized root condition or the boundary locuscondition.

The representative polynomial is now � � � � � � � � � � � � � � � � � The

matrix

� � will be well conditioned if� � roots will be inside and

� � will be outside the unit disk. The convergence is ensured if

�

is a Jordan curve (this will prevent having double roots on the

unit circle) and

� � �


Example: The midpoint method

The midpoint rule may generate two different discretemethods, the classical one where the additional condition isplaced at the origin, and the BVM method where theadditional condition is placed at the end. In the first case thematrix

� � � � � is

��

� � � � �

��

��

� � � � �

. . . . . . . . .. . . . . . . . .

� � � � � � � � ��

��


while in the second case it is

��

� � � � �

��

� � �

� � � � �

� � � � �. . . . . . . . .

� � � � � ��

��

The representing polynomial � � � � � � � � � � � � � � �

is the same

in the two cases but

��

� � � � has two lower diagonals, while

��

� � � � has only one.Except for the values of � on the segment

� � � � �� the rootsof � � � � � � are always one outside and one inside the unit disk.

The matrix

��

� is well conditioned for � � � � � � while

��

� isnot.


Classes of BVMs

The result in the previous example is not isolated.

The BVM approach permits to obtain classes ofwell-conditioned numerical methods (

�

-stable methods)and also classes of perfect stable methods of any order.


Conservative Problems

Consider the non linear pendulum.

��

��

� �

� � ��

�The energy

� ��

� � � � � � ��

is a constant of the motion.

When the trapezoidal method is applied we get

� � � �

��

�

� ��

��

��

� � ��

��


BREATHING EFFECT

��

� � ��

��

� � � � � � � � ��

where

��

will be called volatile hamiltonian.

��

� ��

�

��

� � � � ��

� � � � � � � � � � � �


−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

h=2.09

h=0.8


0 50 100 150 200 250 300 350 400 450 500−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2Volatile Hamiltonian


Further conditions

Simplecticity

Conservation

Ergodicity

.....


Definitions

�� fixed

where � � � � � � The critical solutions (equilibria) are the

solutions of � � � � � �

Definition. The critical solution �� is stable if

��

such that

� �� for � � �

Definition. The critical solution �� is asymptotically stable(AS) if it is stable and, moreover,

� � � � ��

If

�

can be chosen independent on � � the stability or the ASstability is global.

Definition. The critical solution �� is unstable if it is not stable.


stability and conditioning in numerical · pdf filestability and conditioning in numerical...

Documents