arbenz peter,numerical methods for computational science and engineering

940
Numerical Methods for Computational Science and Engineering Numerical Methods for Computational Science and Engineering Lecture 1, Sept 18, 2014: Introduction Peter Arbenz Computer Science Department, ETH Z¨ urich E-mail: [email protected] NumCSE, Lecture 1, Sept 18, 2014 1/48

Upload: anthalya

Post on 16-Aug-2015

41 views

Category:

Documents


15 download

DESCRIPTION

maths

TRANSCRIPT

Numerical Methods for Computational Science and Engineering

Numerical Methods for Computational Scienceand EngineeringLecture 1, Sept 18, 2014: IntroductionPeter ArbenzComputer Science Department, ETH ZurichE-mail: [email protected]

NumCSE, Lecture 1, Sept 18, 2014

1/48

Numerical Methods for Computational Science and EngineeringIntroduction

Outline of todays lectureI

What is numerical methods for CSE

I

Survey of the lecture

I

Organization of the lecture (exercises/examination)

I

References

I

Start of the lecture

NumCSE, Lecture 1, Sept 18, 2014

2/48

Numerical Methods for Computational Science and EngineeringIntroduction

Scientific Computing

NumCSE, Lecture 1, Sept 18, 2014

3/48

Numerical Methods for Computational Science and EngineeringA CSE example

A CSE example: particle acceleratorsObserved phenomenonI

Charged particles are accelerated in electromagnetic fields.I

I

Cathode ray tubes: beam of electrons deflected by e-field tocreate image (classic TV set).

Particle accelerator is a device to propel charged particles tohigh speeds by applying electromagnetic fields.I

I

I

Paul Scherrer Institut: Numerous accelerator to investigatematerial properties, or for cancer treatment.CERN: Large hadron collider (LHC) to generate newelementary particles.Linear accelerators vs. cyclotrons. Demo: cyclo.m

NumCSE, Lecture 1, Sept 18, 2014

4/48

Numerical Methods for Computational Science and EngineeringA CSE example

A CSE example: particle accelerators (cont.)

NumCSE, Lecture 1, Sept 18, 2014

5/48

Numerical Methods for Computational Science and EngineeringA CSE example

A CSE example: particle accelerators (cont.)Model1. Electric fields are modeled by the time-harmonic Maxwellequations (periodic solution (E(x, t) = e(x)e it ), magneticfield eliminated, all pysical quantities = 1)curl curl e(x) = e(x),n e = 0,

x ,

div e(x) = 0,

x .

is accelerator cavity.2. Particles move according to Newtons law of motiondx(t)= v,dt

dv(t)q=(E + v B) .dtm0

where E = Eext + Eself . Eext is E-field from 1. (q charge, m0rest mass.)

NumCSE, Lecture 1, Sept 18, 2014

6/48

Numerical Methods for Computational Science and EngineeringA CSE example

A CSE example: particle accelerators (cont.)Discretization1. e-field in the Maxwell equation is a continuous function.Approximate it by a finite element function.This leads to a large sparsegeneralized eigenvalue problemAx = Mx.Size can be in millions.2. Particles are particles.Big issue is the computation of the self-field, i.e.,particle-particle interaction.NumCSE, Lecture 1, Sept 18, 2014

7/48

Numerical Methods for Computational Science and EngineeringA CSE example

A CSE example: particle accelerators (cont.)Discretization (cont.)I

Computing particle-particle interactions costs O(np2 )operations in the number of particles np .

I

Remedy: Approximately compute the electric potential (x)that is due to the charged particles. To that end we have tosolve the Poisson equation(x) =

1(x)0

with some boundary conditions.The quantity (x) on the right is the charge density which isin fact approximated on a regular grid.

NumCSE, Lecture 1, Sept 18, 2014

8/48

Numerical Methods for Computational Science and EngineeringA CSE example

A CSE example: particle accelerators (cont.)Also the solution (x) is obtained on this regular grid.

The gradient of (x) provides the force that acts on theparticle at position x,Eself = grad (x).NumCSE, Lecture 1, Sept 18, 2014

9/48

Numerical Methods for Computational Science and EngineeringA CSE example

A CSE example: particle accelerators (cont.)Solution algorithmsI

We need to solve systems of linear equationsAx = b,

I

I

I

where A is large and sparse, maybe positive definite.In a time-dependent problem we may have to solve such asystem in every time step.For the Poisson problem on regular grids there are fastPoisson solvers that are related to the fast Fourier transform.We need to solve eigenvalue problemsAx = Mx,where A and M are large and sparse to get the driving e-field.

NumCSE, Lecture 1, Sept 18, 2014

10/48

Numerical Methods for Computational Science and EngineeringA CSE example

A CSE example: particle accelerators (cont.)Efficiency, accuracyI

These problems are to be solved efficiently w.r.t. time andmemory.

I

Linear vs. nonlinear models.

I

Direct vs. iterative solution of linear systems.

I

High vs. relaxed accuracy.

I

Repeated solution of equal/similar problems may amortizeexpensive solution procedures.

NumCSE, Lecture 1, Sept 18, 2014

11/48

Numerical Methods for Computational Science and EngineeringSurvey of the lecture

Survey of the lecture1. Introduction2. Roundoff errors3. Nonlinear equations in one variable (2 lectures)4. Linear algebra basics (Read sections 4.14.4 in AscherGreif)5. Direct methods for linear system, pivoting strategies, sparsematrices (2)6. Linear least squares problems (2)7. Iterative methods for linear system (2)8. Eigenvalues and singular values (2)9. Nonlinear systems and optimization (3)10. (Piecewise) polynomial interpolation (3)NumCSE, Lecture 1, Sept 18, 2014

12/48

Numerical Methods for Computational Science and EngineeringSurvey of the lecture

Survey of the lecture (cont.)11. Best approximation12. Filtering algorithms, Fourier transform13. Numerical differentiation14. Numerical integration (2)15. Ordinary differential equations, initial value problems (3)

NumCSE, Lecture 1, Sept 18, 2014

13/48

Numerical Methods for Computational Science and EngineeringAbout this course

About this courseFocusIII

on algorithms (principles, scope, and limitations),on (efficient, stable) implementations in Matlab,on numerical experiments (design and interpretation).

No emphasis onI

I

theory and proofs (unless essential for understanding ofalgorithms)hardware-related issues (e.g. parallelization, vectorization,memory access)(These aspects will be covered in the courseHigh Performance Computing for Science and Engineeringoffered by D-INFK)

NumCSE, Lecture 1, Sept 18, 2014

14/48

Numerical Methods for Computational Science and EngineeringAbout this course

Goals Knowledge of the fundamental algorithms in numericalmathematics Knowledge of the essential terms in numerical mathematicsand the techniques used for the analysis of numericalalgorithms Ability to choose the appropriate numerical method forconcrete problems Ability to interpret numerical results Ability to implement numerical algorithms efficiently inMatlab

Indispensable: Learning by doing ( exercises)NumCSE, Lecture 1, Sept 18, 2014

15/48

Numerical Methods for Computational Science and EngineeringAbout this course

LiteratureUri Ascher & Chen Greif: A First Course in NumericalMethods. SIAM, 2011.http://www.siam.org/books/cs07/

NumCSE, Lecture 1, Sept 18, 2014

I

Excellent reference.

I

Main reference for large parts of thiscourse.

I

Target audience: undergraduate studentsin computer science.

I

I will follow this book quite closely.

16/48

Numerical Methods for Computational Science and EngineeringAbout this course

Literature (cont.)W. Dahmen & A. Reusken: Numerik fur Ingenieure undNaturwissenschaftler, Springer, 2006.A lot of simple examples and good explanations, but also rigorous mathematical treatment. Targetaudience: undergraduate students in science and engineering.

H.-R. Schwarz & N. Kockler: Numerische Mathematik.Teubner, 2006. 6. Auflage.Easy to read. Target audience: undergraduate students in science and engineering.

C. Moler: Numerical Computing with Matlab. SIAM 2004.Good reference for some parts of this course; Target audience: Matlab users and programmers. Seehttp://www.mathworks.ch/moler/.

W.Gander, M.J. Gander, & F. Kwok: Scientific Computing.An introduction using Maple and Matlab. Springer 2014.

NumCSE, Lecture 1, Sept 18, 2014

17/48

Numerical Methods for Computational Science and EngineeringAbout this course

PrerequisitesEssential prerequisite for this course is a solid knowledge in linearalgebra and calculus. Familiarity with the topics covered in thefirst semester courses is taken for granted, see K. Nipp and D. Stoffer, Lineare Algebra, vdfHochschulverlag, Zurich, 5 ed., 2002. M. Gutknecht, Lineare algebra, lecture notes, SAM, ETHZurich, 2009.http://www.sam.math.ethz.ch/~mhg/unt/LA/HS07/. M. Struwe, Analysis fur Informatiker. Lecture notes, ETHZurich, 2009.

NumCSE, Lecture 1, Sept 18, 2014

18/48

Numerical Methods for Computational Science and EngineeringOrganization

OrganizationLecturer:Prof. Peter Arbenz

[email protected]

Assistants:Daniel HuppChristian SchullerAlexander LobbeSharan JagathrakashakanAlexander BohnManuel MoserFabian ThuringTimo Welti

[email protected]@[email protected]@[email protected]@[email protected]@student.ethz.ch

NumCSE, Lecture 1, Sept 18, 2014

19/48

Numerical Methods for Computational Science and EngineeringOrganization

VenueClasses:Tutorials:

Mon 10.15-12.00 (CAB G11); Thu 10.15-12.00 (HG G5)Mon 13.15-15.00Thu 8.15-10.00

Please register (on course website) for tutorial groups untilSeptember 23th:http://www.math.ethz.ch/education/bachelor/lectures/hs2014/math/nummath_cse

Consulting hours: if needed, see the course website.

NumCSE, Lecture 1, Sept 18, 2014

20/48

Numerical Methods for Computational Science and EngineeringOrganization

Assignments The assignment sheets will be uploaded on the coursewebpage on Monday every week the latest. The exercise should be solved until the following tutorial class.(Hand them in to the assistant.)

NumCSE, Lecture 1, Sept 18, 2014

21/48

Numerical Methods for Computational Science and EngineeringOrganization

Examination Three-hour written examination involving coding problems tobe done at the computer onTBA Dry-run for computer based examination:Does not exist anymore.Try out a computer in the student labs in HG. Pre-exam question session:TBA

NumCSE, Lecture 1, Sept 18, 2014

22/48

Numerical Methods for Computational Science and EngineeringOrganization

Examination (cont.) Topics of examination:I All topics, that have been addressed in class or in ahomework assignment.I One exam question will be one of the homeworkassignment. Lecture slides will be available as (a single) PDF file duringthe examination. The Ascher-Greif book will be made available, too. The exam questions will be asked in English.

NumCSE, Lecture 1, Sept 18, 2014

23/48

Numerical Methods for Computational Science and EngineeringComputing environment: Matlab

Problem solving environment: MatlabII

II

We use Matlab for the exercises.Although most of the algorithm we are dealing with have beenimplemented in Matlab, it is useful when you program themyourselves.These (little) programs will be building blocks when you willsolve more complex problems in your future.Matlab helpII

II

Matlab commands help/docMatlab online documentation, e.g.,http://www.mathworks.nl/help/pdf_doc/allpdf.html

Numerous introductory textbooks / user guides / primersMy own Matlab introduction:http://people.inf.ethz.ch/arbenz/MatlabKurs/matlabintro.pdf

NumCSE, Lecture 1, Sept 18, 2014

24/48

Numerical Methods for Computational Science and EngineeringNumerical algorithms and errors

Numerical algorithms and errorsI

The most fundamental feature of numerical computing is theinevitable presence of errors.

I

The result of any interesting computation (and of manyuninteresting ones) is typically only approximate, and our goalis to ensure that the resulting error is tolerably small.

I

Example: How many loop iterations are there in this littleMatlab program?x = 0; h = 1/10;while x 0 small enough.

Similarly, for w = w (n) the expressionw = O(n log n)means that there is a constant C > 0 such that|w | Cn log nNumCSE, Lecture 1, Sept 18, 2014

as n .

38/48

Numerical Methods for Computational Science and EngineeringAlgorithm properties

Big-O and notationMore abstract:Class O(f ) of functions is defined asO(f ) = {g | c1 , c2 > 0 : N Z+ : g (N) c1 f (N) + c2 }The notation signifies a stronger relation than the O notation:a function (h) for small h (resp., (n) for large n) is ((h))(resp., ((n))) if is asymptotically bounded both above andbelow by .Example:O(h2 ) means at least quadratic convergence (see later). (h2 )is exact quadratic convergence.NumCSE, Lecture 1, Sept 18, 2014

39/48

Numerical Methods for Computational Science and EngineeringAlgorithm properties

Complexity IITo a certain extent, the asymptotic complexity allows to predictthe dependence of the runtime of a particular implementation ofan algorithm on the problem size (for large problems). Forinstance, an algorithm with asymptotic complexity O(n2 ) is likelyto take 4 as much time when the problem size is doubled.One may argue that the memory accesses are more decisive for runtimes than floating point operations. Often there is a lineardependence among the two. So, there is no difference in the Onotation.

NumCSE, Lecture 1, Sept 18, 2014

40/48

Numerical Methods for Computational Science and EngineeringElementary operations

ScalingScaling multiplication with diagonal matrices (with non-zerodiagonal entries) from left and/or right.It is important to know the different effects of multiplying with adiagonal matrix from left or right:DA

vs. AD

with Rnn 3 A, D = diag(d1 , . . . , dn )

Scaling with D = diag(d1 , . . . , dn )in Matlab:y = diag(d)*x;ory = d.*x;NumCSE, Lecture 1, Sept 18, 2014

41/48

Numerical Methods for Computational Science and EngineeringElementary operations

Elementary matricesMatrices of the form A = I + uvT are called elementary.Again we can apply A to a vector x in a straightforward and amore clever way:Ax = (I + uvT )xorAx = x + u(vT x)Cf. exercises.

NumCSE, Lecture 1, Sept 18, 2014

42/48

Numerical Methods for Computational Science and EngineeringElementary operations

Problem conditioning and algorithm stabilityQualitatively speaking:I

The problem is ill-conditioned if a small perturbation in thedata may produce a large difference in the result.The problem is well-conditioned otherwise.

I

The algorithm is stable if its output is the exact result of aslightly perturbed input.

NumCSE, Lecture 1, Sept 18, 2014

43/48

Numerical Methods for Computational Science and EngineeringElementary operations

An unstable algorithm

Ill-conditioned problem of computing output values y from inputvalues x by y = g (x): when x is slightly perturbed to x, the resulty = g (x ) is far from y .NumCSE, Lecture 1, Sept 18, 2014

44/48

Numerical Methods for Computational Science and EngineeringElementary operations

A stable algorithm

An instance of a stable algorithm for computing y = g (x): theoutput y is the exact result, y = g (x ), for a slightly perturbedinput, i.e., x which is close to the input x. Thus, if the algorithm isstable and the problem is well-conditioned, then the computedresult y is close to the exact y .

NumCSE, Lecture 1, Sept 18, 2014

45/48

Numerical Methods for Computational Science and EngineeringElementary operations

Unstable algorithmProblem statement: evaluate the integralsZ 1xndx, for n = 0, 1, 2, . . . , 30.yn =0 x + 10Algorithm development: observe that analytically, for n > 0,Z 1 nZ 1x + 10x n11yn + 10yn1 =dx =x n1 dx = .x + 10n00Also,Zy0 =0

1

1dx = log(11) log(10).x + 10

Algorithm:I Evaluate y0 = log(1.1).I For n = 1, 2, . . . , 30, evaluate yn =NumCSE, Lecture 1, Sept 18, 2014

1n

10yn1 .

46/48

Numerical Methods for Computational Science and EngineeringElementary operations

Unstable algorithm (cont.)Run the Matlab program Example1 6.m by Ascher and Greif tosee the catastrophic amplification of roundoff errors.This code is available fromhttp://www.siam.org/books/cs07/programs.zip.

NumCSE, Lecture 1, Sept 18, 2014

47/48

Numerical Methods for Computational Science and EngineeringElementary operations

Unstable algorithm (cont.)Roundoff error accumulationI

In general, if En is error after n elementary operations, cannotavoid linear roundoff error accumulationEn ' c0 nE0 .

I

Will not tolerate an exponential error growth such asEn ' c1n E0 ,

for some constant c1 > 1.

This is an unstable algorithm.

NumCSE, Lecture 1, Sept 18, 2014

48/48

Numerical Methods for Computational Science and EngineeringIntroduction

Goals for todayI

Unterstand how numbers are stored in a computer

I

Get a feeling for the almost random nature of rounding error

I

Understand how standard floating point systems are designedand implemented

I

Understand how roundoff errors can cause wrong results

References for todayI

Ascher & Greif: Chapter 2

I

N. J. Higham: Accuracy and stability of numerical algorithms.2nd ed. SIAM 2002.

I

W. Gander, M. Gander, F. Kwok: Scientific Computing.Springer 2014.

NumCSE, Lecture 2, Sept 22, 2014

2/34

Numerical Methods for Computational Science and EngineeringIntroduction

IntroductionOne of the important tasks of numerical mathematics is thedetermination of the accuracy of results of some computation.There are three types of errors that limit accuracy:1. Errors in the mathematical model of the problem to be solved.Simplified models are easier to solve (shape of objects,unimportant chemical reactants).2. Discretization or approximation errors depend on the chosenalgorithm or the type of discretization.I

May occur even when computing without rounding error:e =1+

I

111+++ 1! 2! 3!

is approximated by a finite number of terms (truncation error).Function is approximated by, e.g., a piecewise linear function

NumCSE, Lecture 2, Sept 22, 2014

3/34

Numerical Methods for Computational Science and EngineeringIntroduction

Introduction (cont.)II

A differential equation by a difference equation.Iteration is discontinued prematurely.

3. Rounding errors occur if a real number (probably anintermediate result of some computation) is rounded to thenext nearest machine number.The propagation of rounding errors from one floating pointoperation to the next is the most frequent source of numericalinstabilities.Since computer memory is finite practically no real numbercan be represented exactly in a computer.We discuss floating point numbers as a representation of realnumbers.

NumCSE, Lecture 2, Sept 22, 2014

4/34

Numerical Methods for Computational Science and EngineeringIntroduction

Motivating example: quadrature of a circleLets try to compute , the area of a circle with radius r = 1.We approximate by the area of an inscribed regular polygon:

n := 2nFn = cos 2n sin 2n

An = nFn = n cos

nnsin 22

as n

[See Gander, Gander, & Kwok: Scientific Computing. Springer 2014.]

NumCSE, Lecture 2, Sept 22, 2014

5/34

Numerical Methods for Computational Science and EngineeringIntroduction

Motivating example: quadrature of a circle (cont.)Of course, we cannot use to compute .The procedure is to compute A2n from An with an initial n0 forwhich An0 does not depend on . nn nn2n 2 cossin= sin n = sin.An = nFn =22222nsprn1 cos n1 1 sin2 nsin 2n = sin==.2223n0 = 6,6 = 2/n0 = 60 ,sin 6 =2

NumCSE, Lecture 2, Sept 22, 2014

6/34

Numerical Methods for Computational Science and EngineeringIntroduction

Motivating example: quadrature of a circle (cont.)

s=sqrt(3)/2; A=3*s; n=6;% initializationz=[A-pi n A s];% store the resultswhile s>1e-10% terminate if s=sin(alpha) smalls=sqrt((1-sqrt(1-s*s))/2); % new sin(alpha/2) valuen=2*n; A=n/2*s;% A = new polygonal areaz=[z; A-pi n A s];endfor i=1:length(z)fprintf(%10d %20.15f %20.15f %20.15f\n,z(i,2),z(i,3),z(i,1)end

NumCSE, Lecture 2, Sept 22, 2014

7/34

Numerical Methods for Computational Science and EngineeringBinary representation of numbers

IntegersAlso integers suffer from the finiteness of computers.Matlab represents integers by 32-bit signed ints (in the twoscomplement format) 30Pfalls a31 = 0 ai 2 i ,i=0a=30P(232 ai 2i ), falls a31 = 1i=0

Therefore the range is 2147483648 a 2147483647. Thesenumbers are given by intmin and intmax, respectively.Note that in Matlab the ints do not cycle when you shootacross intmax.Also note that you may be limited in problem size.NumCSE, Lecture 2, Sept 22, 2014

8/34

Numerical Methods for Computational Science and EngineeringBinary representation of numbers

Real numbersA number x R (in the binary number system) has the formx = (1.d1 d2 d3 dt1 dt dt+1 ) 2ee is an integer exponent, the binary digits di are either 0 or 1.d1 d2 d3+ 2 + 3 + 222In general, infinitely many digits are needed to represent a realnumber.1.d1 d2 d3 = 1 +

The choice of a binary representation is just one of manypossibilities. It is, indeed, a convenient choice when it comes tocomputers.NumCSE, Lecture 2, Sept 22, 2014

9/34

Numerical Methods for Computational Science and EngineeringBinary representation of numbers

Examples of real numbers1. 1.1012 2 = (1 + 1/2 + 1/8) 2 = 3.252. (10011.01)2 = 24 + 21 + 20 + 22 = 16 + 2 + 1 + 41 = (19.25)

PP1 m3. (0.010101 . . .)2 =22m = 41= 14 11 1 = 134m=1

4.

15

m=0

4

= (0.2) = (0.00110011 . . .)2

The last example is of interest insofar as it shows that to a finitedecimal number there may correspond a (nontrivial) infinite binaryrepresentation. (This is not true the other way round. Why?)So, one cannot assume that a finite decimal number is exactlyrepresentable on a binary computer.[See Gautschi: Numerical Analysis, 2012]NumCSE, Lecture 2, Sept 22, 2014

10/34

Numerical Methods for Computational Science and EngineeringBinary representation of numbers

Floating point representationGiven any real number in binary format

d1 d2 d3dt1 dtdt+1x = 1 + 1 + 2 + 3 + + t1 + t + t+1 + 2e222222e= (1.d1 d2 d3 dt1 dt dt+1 ) 2We somehow represent x in the computer asfl(x) = sign(x) (1.d1 d2 d3 dt1 dt ) 2efor some fixed t. Thus we need t bits for the so-called mantissa.Sign requires 1 bit of storage.Exponent range: L e U

NumCSE, Lecture 2, Sept 22, 2014

11/34

Numerical Methods for Computational Science and EngineeringBinary representation of numbers

Floating point representation (cont.)How big can the relative error|fl(x) x||x|become?

NumCSE, Lecture 2, Sept 22, 2014

12/34

Numerical Methods for Computational Science and EngineeringBinary representation of numbers

Error in floating point number representationI

Choppingfl(x) = (1.d1 d2 d3 dt ) 2e

I

Absolute error is bounded by 2t 2eRounding((1.d1 d2 d3 dt ) 2e ,fl(x) =(1.d1 d2 d3 dt + 2t ) 2e ,

0.dt+1 dt+2 . . . < 1/20.dt+1 dt+2 . . . > 1/2

In case of a tie (0.dt+1 dt+2 . . . = 1/2) round to nearest evennumber.Absolute error bounded by 21 2t 2e .Relative error bounded by rounding unit, machine precision

NumCSE, Lecture 2, Sept 22, 2014

1 = 2t .2

13/34

Numerical Methods for Computational Science and EngineeringIEEE floating point numbers

IEEE floating point numbersANSI/IEEE Standard 754-1985 for Binary Floating PointArithmetic. Acording to the IEEE standard a 32-bit float has thefollowing structure (from en.wikipedia.org)

The exponent has 8 bits, the mantissa 23 Bit. There is a sign bit.The value of a normalized 32-bit IEEE floating point number V isV = (-1)S x 2(E-127) x (1.M)Normalized means 0 < E < 255 = 28 1. (127 is called a bias.)NumCSE, Lecture 2, Sept 22, 2014

14/34

Numerical Methods for Computational Science and EngineeringIEEE floating point numbers

IEEE floating point numbers (cont.)I

double:1 sign bit11 bits exponent52 bits mantissaThe value of a normalized 64-bit IEEE floating point numberV isV = (-1)S x 2(E-1023) x (1.M)Normalized means, that 0 < E < 2047 = 211 1.

NumCSE, Lecture 2, Sept 22, 2014

15/34

Numerical Methods for Computational Science and EngineeringIEEE floating point numbers

IEEE floating point numbers (cont.)If the exponent has only zeros or ones, there is a special cases:III

0 (zero): e = 0, m = 0, s arbitrary.-Infinity, +Infinity: e = all ones, m = 0.e = all ones, m 6= 0: NaN

There are also non-normalized numbers.

NumCSE, Lecture 2, Sept 22, 2014

16/34

Numerical Methods for Computational Science and EngineeringIEEE floating point numbers

Rounding errors in IEEEParametersPrecisionsingledoubleextended

of IEEE Standard arithmetics with base E = 2.teminemax2423-1251282 6 1085352 -10211024 2 1 101663 -16381 16384 264 5 1020

From the above we can derive the following statements

LemmaIf x 6= 0 is a normalized floating point number and fl(x) obtainedafter rounding with t digits, then|fl(x) x| 2et /2|fl(x) x| 2t /2 xNumCSE, Lecture 2, Sept 22, 2014

(max. absolute rounding error)(max. relative rounding error)

17/34

Numerical Methods for Computational Science and EngineeringRounding errors

Rounding errorsWe assume that all numbers are normalized.Let t be the length of the mantissa.Between powers of 2, the floating point numbers are equidistant.

Here, the length of the mantissa is t = 2 and 2 e 2.

DefinitionMachine precision = 2(t+1) (half of Matlabs eps)This is half the distance of the numbers between 1 and 2.NumCSE, Lecture 2, Sept 22, 2014

18/34

Numerical Methods for Computational Science and EngineeringRounding errors

Rounding errors are randomNote how the signof the floating pointrepresentation errorat nearby argumentst fluctuates as ifrandomly: as afunction of t it is anon-smootherror.

Error in sampling exp(t)(sin(2t) + 2) in single precision.

NumCSE, Lecture 2, Sept 22, 2014

19/34

Numerical Methods for Computational Science and EngineeringRounding errors

Floating point arithmeticImportant to use exact rounding: if x and y are machine numbers,thenfl(x y ) = (x y )(1 + 1 ),fl(x y ) = (x y )(1 + 2 ),

|i | .

fl(x/y ) = (x/y )(1 + 3 ),In other words: The result of a basic operation with two floatingpoint numbers yields a result that is correct up to a relative errorsmaller than .Thus, the relative errors remain small after each such operation.This is achieved only using guard digits (intermediate higherprecision).

NumCSE, Lecture 2, Sept 22, 2014

20/34

Numerical Methods for Computational Science and EngineeringRounding errors

Guard digitFloating point system with = 10 and t = 4. So, = 21 103 . Letx = .1103, = 1.103 101 ,

y = 9.963 103 .

Then, x y = .100337. Hence, exact rounding yields 0.1003.Relative error:

|0.1003370.1003|0.100337

0.37 103 < .

However, if we were to subtract these two numbers without guarddigits we would obtain .1103 .0099 = 0.1004. Now the obtainedrelative error is |0.1003370.1004| 0.63 103 > .0.100337Thus, guard digits must be used to produce exact rounding.

NumCSE, Lecture 2, Sept 22, 2014

21/34

Numerical Methods for Computational Science and EngineeringRounding errors

Rounding error exampleFor t = 5 we have = 26 = 0.015625.fl() = fl(21 + 20 + 23 + 25 + ) = 21 + 20 + 23 = 1.10010 21|fl() | 0.0053fl()fl() = 10.01110001 22fl(fl()fl()) = 1.00111 23 = 9.75So,

NumCSE, Lecture 2, Sept 22, 2014

2 fl(fl()fl()) 0.12 2 fl(fl()fl()) 0.0122

22/34

Numerical Methods for Computational Science and EngineeringRounding errors

Note on machine epsilonI

For any number with || we havefl(1 + ) = 1.

I

The Matlab command eps returns = 252 , i.e., thesmallest positive number (for the data type double) for which1 + eps > 1.

I

eps can have a parameter, see help eps.

I

In the finite difference example of last week we had for verysmall h that fl(f (x + h)) = fl(f (x)). Therefore,

fl(f (x0 + h)) fl(f (x0 ))0

fl(f (x0 )) = fl(f 0 (x0 )) .

h

NumCSE, Lecture 2, Sept 22, 2014

23/34

Numerical Methods for Computational Science and EngineeringRounding errors

Rounding errors summaryLemma1. With the machine precision we havefl(x) = x(1 + ) with || .2. If is an elementary operation thenfl(x y ) = (x y )(1 + ) with || .

Wilkinsons PrincipleThe result of a numerical computation on the computeris the exact result with slightly perturbed initial data.This also holds for good implementations of (library) functions!

NumCSE, Lecture 2, Sept 22, 2014

24/34

Numerical Methods for Computational Science and EngineeringCancellation

CancellationCancellation (dt. Ausloschung) is a special kind of rounding error.Consider the following two numbers with 5 decimal digits:1.2345 e01.2344 e00.0001 e0 = 1.0000 e4If the two numbers were exact, the result delivered by thecomputer would also be exact. But if the first two numbers hadbeen obtained by previous calculations and were affected byrounding errors, then the result would at best be 1.xxxx e4,where the digits denoted by x are unknown.

NumCSE, Lecture 2, Sept 22, 2014

25/34

Numerical Methods for Computational Science and EngineeringCancellation

Cancellation (cont.)Suppose z = x y , where x y . Then|z fl(z)| |x fl(x)| + |y fl(y )|,from which it follows that the relative error satisfies|x fl(x)| + |y fl(y )||z fl(z)|,|z||x y |Numerator: OK.If the denominator is very close to zero, x y , then the relativeerror in z can become large.

NumCSE, Lecture 2, Sept 22, 2014

26/34

Numerical Methods for Computational Science and EngineeringAvoiding cancellation

Subtracting two almost equal numbersLet a = 1.2 and b = 1.1. We compare two algorithms to computea2 b 2 = (a + b)(a b) in decimal arithmetic with two essentialdigits.a2 = 1.44 = fl(a2 ) = 1.4b 2 = 1.21 = fl(b 2 ) = 1.2fl(a2 ) fl(b 2 ) = 0.2

(13% error)

a + b = fl(a + b) = 2.3a b = fl(a b) = 0.10

NumCSE, Lecture 2, Sept 22, 2014

(a + b)(a b) = 0.23

(0% error)

27/34

Numerical Methods for Computational Science and EngineeringAvoiding cancellation

Subtracting two almost equal numbers (cont.)9970 2 = 9801 9800 = The 3 formula give the following# positions 99 70 221.040.0200060.00530000100.005050660000

1= 0.0050506338833466 .9801 + 9800

results9801 98000.00.010000.005100000.005050640000

19801+ 9800

0.00500.0050510.005050630.005050633884

The reason for the numerical instability of the first two algorithmsis the subtraction of almost equal numbers (indicated by appendedzeros).NumCSE, Lecture 2, Sept 22, 2014

28/34

Numerical Methods for Computational Science and EngineeringAvoiding cancellation

Library function for sinhThe sinus hyperbolicus is defined asy = sinh(x)

1 xe e x .2

If x 0 we have to expect cancellation in the two terms.However, using the Taylor expansion of sinh:sinh(x) = x +

5x3+,6120

|| < x.

For small enough x we can expect very good approximations.

NumCSE, Lecture 2, Sept 22, 2014

29/34

Numerical Methods for Computational Science and EngineeringRemark on overflow

Remark on overflowAn overflow is obtained when a number is too large to fit into thefloating point system in use.qExample: kxk2 = x12 + x22 + + xn2To make things simple, let n = 2.If x = (1060 , 1)T , then the exact result is kxk2 = 1060 .But in the course of the computation we have to form x12 = 10120 .To avoid overflow, we scale with a positive scalar s, e.g.,s = max(|xi |)r x 2x1 2 x2 2nkxk2 = s++ +sss

NumCSE, Lecture 2, Sept 22, 2014

30/34

Numerical Methods for Computational Science and EngineeringQuadrature of a circle revisited

Quadrature of a circle revisitedIn our initial example to compute by the inscribed regularpolynoms we experienced severe cancelation.To compute sin(/2) from sin , we used the recurrence:spn1 1 sin2 nsin=.22Since sin n 0, the numerator on the rightp1 1 2 with small = sin n ,suffers from severe cancellation.Therefore the algorithm performed so badly, although theory andprogram are both correct.

NumCSE, Lecture 2, Sept 22, 2014

31/34

Numerical Methods for Computational Science and EngineeringQuadrature of a circle revisited

Quadrature of a circle revisited (cont.)It is possible to rearrange the computation and avoid cancellation:spn1 1 sin2 nsin=22vppuu 1 1 sin2 n 1 + 1 sin2 np=t21 + 1 sin2 ns1 (1 sin2 n )p=2(1 + 1 sin2 n )sin n=r .p2 1 + (1 sin n )(1 + sin n )NumCSE, Lecture 2, Sept 22, 2014

32/34

Numerical Methods for Computational Science and EngineeringQuadrature of a circle revisited

Quadrature of a circle revisited (cont.)oldA=0; s=sqrt(3)/2; A=3*s; n=6;% initializationz=[A-pi n A s];% store the resultswhile A>oldA% terminate if area doesoldA=A;%not increases=s/sqrt(2*(1+sqrt((1-s)*(1+s)))); % new sin() valuen=2*n; A=n/2*s;% A = new polygonal areaz=[z; A-pi n A s];endfor i=1:length(z)fprintf(%10d %20.15f %20.15f\n,z(i,2),z(i,3),z(i,1))end

Stopping criterion exploits fact that A6 < < An < A2n < NumCSE, Lecture 2, Sept 22, 2014

33/34

Numerical Methods for Computational Science and EngineeringQuadrature of a circle revisited

Paper on high precision computationDavid H. Bailey, Roberto Barrio, and Jonathan M. Borwein, Highprecision computation: Mathematical physics and dynamics,Applied Mathematics and Computation, vol. 218 (2012),pp. 1010610121.http://dx.doi.org/10.1016/j.amc.2012.03.087Gist of the paper:In many very large scale problems it is difficult to achieve sufficientaccuracy: for a rapidly growing body of important scientific computingapplications, a higher level of numeric precision is required.This is facilitated by high-precision software packages.Software available, but awfully slow.

NumCSE, Lecture 2, Sept 22, 2014

34/34

Numerical Methods for Computational Science and EngineeringIntroduction

GoalsI

Want to compute zeros of nonlinear scalar equations.

I

Want to develop useful methods for a basic, simply statedproblem, including such favourites as fixed point iteration andthe Newtons method.

I

Want to develop and assess several algorithmic concepts thatare prevalent throughout the field of numerical computing.

I

Want to study basic algorithms for minimizing a function inone variable.

NumCSE, Lecture 3, Sept 25, 2014

2/34

Numerical Methods for Computational Science and EngineeringIntroduction

Topics of todayI

Bisection algorithm

I

Fixed point iterations

ReferenceI

AscherGreif, Chapter 3, Sections 3.13.3.

Next timeI

Newton and other higher order methods

I

Minimizing a function in one variable

I

AscherGreif, Chapter 3, Sections 3.43.5.

I

Matlabs fzero, a combination of some basic methods.

NumCSE, Lecture 3, Sept 25, 2014

3/34

Numerical Methods for Computational Science and EngineeringThe problem

The problemWant to find solutions of the scalar nonlinear equationf (x) = 0

with continuous f : [a, b] R 7 R

We denote a solution of the equation (called root, or zero) by x .In contrast to scalar linear equationsax b = 0

=a6=0

x =

ba

nonlinear equations have an undetermined number of zeros.We denote the set of all continuous functions on the interval [a, b] by C [a, b].So, above, we require f C [a, b].

NumCSE, Lecture 3, Sept 25, 2014

4/34

Numerical Methods for Computational Science and EngineeringThe problem

Examples1. f (x) = x 1 on [a, b] = [0, 2].2. f (x) = sin(x)On [a, b] = [ 2 , 32 ] there is one root x = .

On [a, b] = [0, 4] there are five roots, cf. Fig. on next page.3. f (x) = x 3 30x 2 + 2552 on [0, 20].4. f (x) = 10 cosh(x/4) on < x < cosh(t) =

NumCSE, Lecture 3, Sept 25, 2014

12

(e t + e t )

5/34

Numerical Methods for Computational Science and EngineeringThe problem

Examples (cont.)

NumCSE, Lecture 3, Sept 25, 2014

6/34

Numerical Methods for Computational Science and EngineeringThe problem

Why study a nonlinear problem before addressing linearones?I

One linear equation, e.g. ax = b, is too simple, whereas asystem of linear equations can have many complications.

I

Several important general methods can be described in asimple context.

I

Several important algorithm properties can be defined andused in a simple context.

NumCSE, Lecture 3, Sept 25, 2014

7/34

Numerical Methods for Computational Science and EngineeringThe problem

Iterative methods for finding rootsI

Roots that can be expressed analytically are known for veryfew special nonlinear functions.

I

Even for polynomials this holds only for very low orders.

I

We have to resort to iterative methods:Starting with an initial guess/iterate x0 we generate asequence of iterates x1 , x2 , . . . that (hopefully) converges to aroot of the function.

I

A rough knowledge of the roots location is required.

I

(Plot)

I

Could probe the function and try to find two arguments a, bs.t. f (a) f (b) < 0.Intermediate value theorem = x in interval (a, b).

NumCSE, Lecture 3, Sept 25, 2014

8/34

Numerical Methods for Computational Science and EngineeringThe problem

Desirable algorithm propertiesIterative method: starting with initial iterate (guess) x0 , generatesequence of iterates x1 , x2 , . . . , xk , . . ., that hopefully converge to aroot x .I

Efficient: requires a small number of function evaluations.

I

Robust: fails rarely, if ever. Announces failure if it does fail.

I

Requires a minimal amount of additional information such asthe derivative of f .

I

Requires f to satisfy only minimal smoothness properties.

I

Generalizes easily and naturally to many equations in manyunknowns.

NumCSE, Lecture 3, Sept 25, 2014

9/34

Numerical Methods for Computational Science and EngineeringBisection

BisectionI

Method for finding a root of scalar equation f (x) = 0 in aninterval [a, b].

I

Assumption: f (a)f (b) < 0 .

I

Since f is continuous there must be a zero x [a, b].(This follows from intermediate value theorem.)

I

Compute midpoint m of the interval and check the value f (m).

I

Depending on the sign of f (m), we can decide if x [a, m] orx [m, b].(Of course, if f (m) = 0 then we are done.)

I

Robust code on next page from book by Gander et al.

NumCSE, Lecture 3, Sept 25, 2014

10/34

Numerical Methods for Computational Science and EngineeringBisection

Code bisectfunction [x,fx] = bisect(f,a,b,tol)%BISECT [x,y] = bisect(f,a,b) computes a zero of f(x) = 0% in interval [a,b] assuming that f(a)f(b) < 0.fa = f(a);fb = f(b);

if fa == 0, x = a; fx = fa; return; endif fb == 0, x = b; fx = fb; return; end

if fa*fb > 0, error(f(a)*f(b) > 0), endx = (a+b)/2; fx = f(x);if nargintol) & ((a < x) &if fx == 0, break, endif fa*fx < 0,b = x; fb = fx;elsea = x; fa = fx;endx = (a+b)/2; fx = f(x);end

(x < b)),

try withf=@(x) cos(x)*cosh(x)+1[x,fx] = bisect(f,0,3,1e-9)[x,fx] = bisect(f,0,3)NumCSE, Lecture 3, Sept 25, 2014

12/34

Numerical Methods for Computational Science and EngineeringBisection

Bisection (cont)I

I

If L0 = b a then Lk := L0 /2k is length of search interval instep k of the procedure.aError estimate: |xk x | b k+12with xk = midpoint of the k-th interval.

I

If we want the error to satisfy |xk x | , then it suffices tohave (b a)/2k+1 , so that

baba= log/ log 2.k + 1 > log2

I

Note the machine independent stopping criterion in bisect!

NumCSE, Lecture 3, Sept 25, 2014

13/34

Numerical Methods for Computational Science and EngineeringBisection

Properties of bisectionI

Simple

I

Safe, robust (even foolproof)

I

Requires only that f be continuous

I

Slow

I

Hard to generalize to systems

NumCSE, Lecture 3, Sept 25, 2014

14/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Fixed point iteration

I

An intuitively appealing approach which often leads to simplealgorithms for complicated problems.The methods discussed now have direct extensions to morecomplicated problems, e.g., to systems of nonlinear equationsand to more complex functional equations.

I

Rewrite the given problem

I

f (x) = 0asx = g (x).

I

()

so that f (x ) = 0 g (x ) = x .x is a fixed point of the mapping g .

NumCSE, Lecture 3, Sept 25, 2014

15/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Algorithm: Fixed point iterationGiven a scalar function f (x). Select a function g (x) such thatf (x) = 0

g (x) = x.

Then:1. Start from an initial guess x0 .2. For k = 0, 1, 2, . . . setxk+1 = g (xk ),

k = 0, 1, . . .

until xk+1 satisfies some termination criterionIt is all in the choice of the function g .

NumCSE, Lecture 3, Sept 25, 2014

16/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Examples of fixed point iterationsNote: there are many ways to transform f (x) = 0 into fixed pointform! Not all of them are good in terms of convergence.We are looking for a zero off (x) = xe x 1,

x [0, 1]

Plot function: fplot(@(x) x*exp(x)-1, [0 1], 1e-3)Different fixed point forms are possible, e.g.:g1 (x) = e x ,1+xg2 (x) =,1 + exg3 (x) = x + 1 xe x .NumCSE, Lecture 3, Sept 25, 2014

17/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Examples of fixed point iterations (cont.)k012345678910

xk+1 := g1 (xk )0.5000000000000000.6065306597126330.5452392118926050.5797030948780680.5600646279389020.5711721489772150.5648629469803230.5684380475700660.5664094527469210.5675596342622420.566907212935471

NumCSE, Lecture 3, Sept 25, 2014

xk+1 := g2 (xk )0.5000000000000000.5663110031972180.5671431650348620.5671432904097810.5671432904097840.5671432904097840.5671432904097840.5671432904097840.5671432904097840.5671432904097840.567143290409784

xk+1 := g3 (xk )0.5000000000000000.6756393646499360.3478126785112020.855321409174107-0.1565059553831690.977326422747719-0.6197642518955800.7137130874161460.2566266491298470.924920676910549-0.407422405542253

18/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Examples of fixed point iterations (cont.)k012345678910

|xk x |0.0671432904097840.0393873693028490.0219040785171790.0125598044682840.0070786624708820.0040288585674310.0022803434294600.0012947571602820.0007338376628630.0004163438524580.000236077474313

NumCSE, Lecture 3, Sept 25, 2014

|xk x |0.0671432904097840.0008322872125660.0000001253749220.0000000000000030.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.000000000000000

|xk x |0.0671432904097840.1084960742401520.2193306118985820.2881781187643230.7236492457929530.4101831323379351.1869075423053640.1465697970063620.3105166412799370.3577773865007650.974565695952037

19/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Some questions regarding the fixed point iterationSuppose that we have somehow determined the continuousfunction g C [a, b].Obvious questions arise about fixed point iteration xk+1 = g (xk ):1. Is there a fixed point x in [a, b]?2. If yes, is it unique?3. Does the sequence of iterates converge to a root x ?4. If yes, how fast?5. If not, does this mean that no root exists?

NumCSE, Lecture 3, Sept 25, 2014

20/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Fixed point theoremIf g C [a, b] and a g (x) b for all x [a, b], then there is afixed point x in the interval [a, b].If, in addition, the derivative g 0 exists and there is a constant < 1 such that the derivative satisfies|g 0 (x)|

x (a, b),

then the fixed point x is unique in this interval.See AscherGreif book for a proof.(This answers questions 1 and 2.)

NumCSE, Lecture 3, Sept 25, 2014

21/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Convergence of the fixed point iteration3. Does the sequence of iterates converge to a root x ?|xk+1 x | = |g (xk ) g (x )| = |g 0 ()| |xk x | |xk x |with [xk , x ].This is a contraction if the factor < 1. Thus,|xk+1 x | |xk x | 2 |xk1 x | k+1 |x0 x |.Since < 1 then k 0 as k .We have convergence.

NumCSE, Lecture 3, Sept 25, 2014

22/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Convergence of fixed point iterations in 1DVastly different behavior of different fixed point iterations:

g1 : linear convergence?

NumCSE, Lecture 3, Sept 25, 2014

g2 : quadratic convergence?

g3 : no convergence?

23/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Geometric interpretation of fixed point iterationx0g (x0 )x1 = g (x0 )g (x1 )etc.

start with x0 on the x-axisgo parallel to the y -axis to the graph of gmove parallel to the x-axis to the graph y = xgo parallel to the y -axis to the graph of g

NumCSE, Lecture 3, Sept 25, 2014

24/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

left: 1 < g 0 (x ) 0at least linear convergence

right: g 0 (x ) < 1divergence

Note: these are local scenarios.

NumCSE, Lecture 3, Sept 25, 2014

25/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

left: 0 g 0 (x ) < 1at least linear convergence

right: 1 < g 0 (x )divergence

Note: these are local scenarios.

NumCSE, Lecture 3, Sept 25, 2014

26/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Rate of convergenceLet x be a fixed point of the iteration xk+1 = g (xk ) and = |g 0 (x )|

with

0 < < 1.

x0 sufficiently close to x = xk x g 0 (x )(xk1 x ).|xk x | |xk1 x | k |x0 x |. is called contraction factor.Definition: The rate of convergence is defined as rate = log10 .The rate is higher (faster convergence), the smaller .

NumCSE, Lecture 3, Sept 25, 2014

27/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Rate of convergence (cont.)About k = d1/ratee iteration steps are needed to reduce the errorby one order of magnitude: k < 0.1.

0.9990.990.900.500.10

NumCSE, Lecture 3, Sept 25, 2014

rate0.000440.00440.0460.301

k23022302241

28/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Rate of convergence (for earlier numerical example)The rate of convergence for xk+1 = gi (xk ) is in the previousscenarios for solving f (x) = xe x 1, with x [0, 1].Fixed point is x = 0.5671432904.g10 (x ) = g1 (x ) = x = 0.5671432904.0 < |g10 (x )| < 1 convergence.

g30 (x ) = 1 x e x e x = 1.76 . . . divergence. xf (x )g20 (x ) = 1 x e 2 = 2 = 0,(e x + 1)(e x + 1)g200 (x ) 6= 0 quadratic convergence (see next time: order ofconvergence)

NumCSE, Lecture 3, Sept 25, 2014

29/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Termination criteriaResidual based termination:|f (xk )|

STOP convergent iteration {xk }, if

= prescribed tolerance > 0

= no guaranteed accuracyf

f

x

|f (xk )| small 6 |x x | smallNumCSE, Lecture 3, Sept 25, 2014

x

|f (xk )| small |x x | small

30/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Termination criteria (cont.)Correction based termination: STOP convergent iteration {xk }, ifabs|xk+1 xk | orrel |xk |abs /rel are prescribed absolute / relative tolerances > 0Usually (but not always) the relative criterion is more robust thanthe absolute one.A combination of the first two is

NumCSE, Lecture 3, Sept 25, 2014

|xk+1 xk | < (1 + |xk |)

31/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

A posteriori termination criterionA posteriori termination criterion for linearly convergent iterationswith rate of convergence 0 < < 1:|xk+1 x | 1 |xk+1 xk |That is: is we know (or can estimate) then the formula tells ushow close we are to the solution x .Proof.Idea of the proof: Bound |xk+m xk+1 |, m > 1, independent of mand let m .|xk+2 xk+1 | = |g (xk+1 ) g (xk )| |xk+1 xk |NumCSE, Lecture 3, Sept 25, 2014

32/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

A posteriori termination criterion (cont.)|xk+m xk+1 | = |xk+m xk+m1 + xk+m1 xk+m2 + + xk+3 xk+2 + xk+2 xk+1 | |xk+m xk+m1 | + |xk+m1 xk+m2 | + + |xk+3 xk+2 | + |xk+2 xk+1 |m1

(

+

m2

(

m2

+ + )|xk+1 xk |

m3

+

+ + 1)|xk+1 xk |

1 m1|xk+1 xk | 2

Let m then xk+m x .NumCSE, Lecture 3, Sept 25, 2014

33/34

Numerical Methods for Computational Science and EngineeringFixed point iteration

Useful results from analysisI

Intermediate value theorem. If f C [a, b] and s is a valuesuch that f (a) s f (b), then there exists a real numberc [a, b] for which f (c) = s.

I

Mean value theorem. If f C [a, b] and f is differentiable onthe open interval (a, b), then there exists a real number(a).c (a, b) for which f 0 (c) = f (b)fba

I

Rolles theorem. If f C [a, b] and f is differentiable on (a, b),and in addition f (a) = f (b) = 0, then there is a real numberc (a, b) for which f 0 (c) = 0.

NumCSE, Lecture 3, Sept 25, 2014

34/34

Numerical Methods for Computational Science and EngineeringIntroduction

Topics of todayI

Newtons algorithm and variants

I

The secant method and inverse interpolation

I

Zeroin (fzero)

I

Multiple zeros

I

Scalar optimization

ReferencesI

AscherGreif, Chapter 3, Sections 3.43.5.

I

C. Moler: Numerical Computing with Matlab. TheMathWorks. 2004. (Chapter 4: Zeros and roots.)

NumCSE, Lecture 4, Sept 29, 2014

2/40

Numerical Methods for Computational Science and EngineeringThe problem

The problem revisitedWant to find solutions of the scalar nonlinear equationf (x) = 0

with continuous f : [a, b] R 7 R

We denote a solution of the equation (called root, or zero) by x .We looked at bisection and general fixed point iterations todetermine solutions x of f (x) = 0.

NumCSE, Lecture 4, Sept 29, 2014

3/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Newton iterationI

Let the function f for which we look for a zero x bedifferentiable.

I

In practice, the derivative f 0 should be cheap to compute(comparable with the computation of f ).

I

In Newtons method the function f is linearized at someapproximate value xk x .

I

Define function t(x) that has at x = xk same function valueand derivative as f(Taylor polynomial of degree one)t(x) = f (xk ) + f 0 (xk )(x xk ).

I

(1)

The single zero of t(x) = 0 yields xk+1 .

NumCSE, Lecture 4, Sept 29, 2014

4/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Newton iteration (cont.)I

The single zero of t(x) = 0 yields xk+1 :xk+1 = xk f (xk )/f 0 (xk ),

I

k = 0, 1, . . .

(2)

Newtons method is a fixed point iteration with iterationfunctiong (x) = x f (x)/f 0 (x).Clearly, g (x ) = x .

I

Since we have neglected second (and higher) terms of theTaylor expansion of f at x we can expect that t(x) is a verygood approximation of f (x) if xk x .

NumCSE, Lecture 4, Sept 29, 2014

5/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Newtons method: geometric interpretation

NumCSE, Lecture 4, Sept 29, 2014

6/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Algorithm: Newtons iterationGiven a scalar differentiable function f (x).1. Start from an initial guess x0 .2. For k = 0, 1, 2, . . . setxk+1 = xk

f (xk )f 0 (xk )

until xk+1 satisfies some termination criterion

NumCSE, Lecture 4, Sept 29, 2014

7/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Newtons method: codefunction [x,it]=newton(f,df,x)%NEWTON Newton iteration for scalar eq f(x)=0.%x=newton(f,df,x0)%x0: initial approx, df = fdx = f(x)/df(x); it=0;while abs(dx) > 1e-10,x = x - dx;dx = f(x)/df(x);it = it+1;end

NumCSE, Lecture 4, Sept 29, 2014

8/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Example (Convergence of Newton iteration)Iteration for computing

a, a > 0:

f (x) = x 2 a,f 0 (x) = 2x,x2 ax2 + a1a== (x + ),2x2x2x2+ax1ag 0 (x) = 1 = 2,g 0 ( a) = 0,22x2 2xa1g 00 (x) = 3 ,g 00 ( a) = .xag (x) = x

a1xk+1 = (xk + )2xkNumCSE, Lecture 4, Sept 29, 2014

|xk+1

a| =

1|xk a|2 .2|xk |

9/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Example (Convergence of Newton iteration (cont.))Numerical experiment: iterates for a = 2: k012345

xk2.000000000000000001.500000000000000001.416666666666666521.414215686274509661.414213562374689871.41421356237309492

ek := xk 20.585786437626904850.085786437626904850.002453104293571370.000002123901414520.000000000001594720.00000000000000022

log

|ek ||ek1 |

: log

|ek1 ||ek2 |

1.8501.9842.0000.630

The number of significant digits in xk essentially doubles at eachiteration.When roundoff level is reached, no meaningful improvement can beobtained any further. The improvement from the 4th to the 5thiteration (in this example) is minimal.

NumCSE, Lecture 4, Sept 29, 2014

10/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Example (Choice of initial guess)Consider f (x) = 2 cosh(x/4) x;

g (x) = 2 cosh(x/4).

f has 2 roots: x1 2.35755106, x2 8.50719958.NumCSE, Lecture 4, Sept 29, 2014

11/40

Numerical Methods for Computational Science and EngineeringNewton iteration

The Newton iteration here isxk+1 = xk

2 cosh(xk /4) xk.0.5 sinh(xk /4) 1

Iterating until |f (xk+1 )| < 108 we observe:I

Starting from x0 = 2 requires 4 iterations to reach x1 towithin the given tolerance.

I

Starting from x0 = 4 requires 5 iterations to reach x1 .

I

Starting from x0 = 8 requires 5 iterations to reach x2 .

I

Starting from x0 = 10 requires 6 iterations to reach x2 .

NumCSE, Lecture 4, Sept 29, 2014

12/40

Numerical Methods for Computational Science and EngineeringNewton iteration

The values of f (xk ), starting from x0 = 8, are:kf (xk )

0-4.76e-1

18.43e-2

21.56e-3

35.65e-7

47.28e-14

51.78e-15

Number of significant digits essentially doubles at each iteration.When roundoff level is reached, no meaningful improvement can beobtained upon heaping more floating point operations, and theimprovement from the 4th to the 5th iteration in this example ismarginal.

NumCSE, Lecture 4, Sept 29, 2014

13/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Order of convergenceThe method is said to beI

linearly convergent if there is a constant < 1 such that|xk+1 x | |xk x |,

I

quadratically convergent if there is a constant M such that|xk+1 x | M|xk x |2 ,

I

for k sufficiently large;

for k sufficiently large;

superlinearly convergent if there is a sequence of constantsk 0 such that|xk+1 x | k |xk x |

for k sufficiently large;

The quadratic case is superlinear with k = M|xk x | 0

NumCSE, Lecture 4, Sept 29, 2014

14/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Convergence of Newtons methodTheorem (Convergence of Newton iteration)If f C 2 [a, b] and there is a root x in [a, b] such that f (x ) = 0,f 0 (x ) 6= 0, then there is a number > 0 such that, starting withx0 from anywhere in the neighborhood [x , x + ], Newtonsmethod converges quadratically.Note that f 0 (x ) 6= 0 means that x is a simple root of f .

Sketch of a proof.(i) It is easy to verify thatg 0 (x) =

f (x)f 00 (x).(f 0 (x))2

So, g 0 (x ) = 0.

NumCSE, Lecture 4, Sept 29, 2014

15/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Convergence of Newtons method (cont.)(ii) Since g is continuously differentiable there is someneighborhood [x , x + ] of x in which |g 0 | < 1.(iii) For convergence order use the Taylor expansion of g at x :x xk+1 = g (x ) g (xk )= g (x ) g (x ) + g 0 (x )(xk x )

+ g 00 ()(xk x )2= g 00 ()(xk x )2with between x and xk .

NumCSE, Lecture 4, Sept 29, 2014

16/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Another exampleThe equation f (x) = x 3 3x + 2 = (x + 2)(x 1)2 has two zeros:2 and 1. The Newton iteration is2xk2+33(xk + 1)2x(x + 2)22g 0 (x) == 3(x + 1)23 3(x + 1)2xk+1 = g (xk ) =

g 0 (2) = 01g 0 (1) =2The double zero makes troubles.

NumCSE, Lecture 4, Sept 29, 2014

17/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Newton iteration for multiple rootsLet x be a multiple root of f : f (x) = (x x )m h(x) with hdifferentiable and h(x ) 6= 0. Then,g (x) = x

f (x)(x x )m h(x)=xf 0 (x)m(x x )m1 h(x) + (x x )m h0 (x)(x x )h(x)=xm h(x) + (x x )h0 (x)

h(x) + (x x )h0 (x)(m h(x) + (x x )h0 (x))(x x )h(x)(m h0 (x) + h0 (x) + (x x )h00 (x)),(m h(x) + (x x )h0 (x))21g 0 (x ) = 1 .mg 0 (x) = 1

NumCSE, Lecture 4, Sept 29, 2014

18/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Newton iteration for multiple roots (cont.)Therefore, Newton iteration converges only linearly to multipleroots.For large m the convergence is very slow.For a double root (m = 2) the contraction factor is = 21 .Remedy: We have to extend the step length in accordance withthe multiplicity of the zero of f (x),xk+1 = xk m f (xk )/f 0 (xk )Note: Often we do not know the multiplicity of a root.Remark: One may try to apply Newton to the function f (x)/f 0 (x)that has only simple roots.NumCSE, Lecture 4, Sept 29, 2014

19/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Simplified Newton iterationxk+1 = xk f (xk )/f 0 (x0 ),

k = 0, 1, . . .

Linear convergence, convergence factor := |g 0 (x )| = |1 f 0 (x )/f 0 (x0 )|.Simplified Newton iteration can be very effective if, x0 is a goodapproximation of x . Then,1 f (x )/f 0 (x0 ) 1 f (x )/f 0 (x ) = 0such that the convergence factor is small.We may update x0 to improve speed of convergence.

NumCSE, Lecture 4, Sept 29, 2014

20/40

Numerical Methods for Computational Science and EngineeringNewton iteration

Damped NewtonTo avoid overshooting one can damp (shorten) the Newton stepxk+1 = xk k f (xk )/f 0 (xk ),

k = 0, 1, . . .

k is chosen such that |f (xk+1 )| < |f (xk )|x = f (x)/f 0 (x);while (|f (x x)| > |f (x)|) = /2;endClose to convergence we should let k 1 to have the full steplength and quadratic convergence.Before each iteration step: = min(1, 2);.

NumCSE, Lecture 4, Sept 29, 2014

21/40

Numerical Methods for Computational Science and EngineeringSecant method

Secant methodxk+1 = xk f (xk )

xk xk1f (xk ) f (xk1 )

secant method.

Notice that the secant method is obtained by approximating thederivative in Newtons method by a finite difference,f 0 (xk )

f (xk ) f (xk1 ).xk xk1

The secant method is not a fixed point method but a multi-pointmethod. It can interpreted as follows. The next iterate xk+1 is thezero of the degree-1 polynomial that interpolates f at xk and xk1 .Convergence order: 1.618, i.e., superlinear! No derivatives needed!!

NumCSE, Lecture 4, Sept 29, 2014

22/40

Numerical Methods for Computational Science and EngineeringSecant method

Secant method (cont.)function [x,i] = secant(x0,x1,f,tol,maxit)% secant methodf0 = f(x0);for i=1:maxitf1 = f(x1);s = f1*(x1-x0)/(f1-f0);x0=x1;x1=x1-s;if(abs(s) 0 such that, starting withx0 , x1 from anywhere in the neighborhood [x , x + ], thesecant method converges superlinearly.Without proof.The convergence order is in fact 1.618 . . ., i.e., assymptotically,|x xk+1 | = c |x xk |1.618...for some constant c.

NumCSE, Lecture 4, Sept 29, 2014

24/40

Numerical Methods for Computational Science and EngineeringSecant method

Example: Choice of initial guessConsider again f (x) = 2 cosh(x/4) x = 0.The secant method requires two initial iterates, x0 and x1 .I

Starting from x0 = 2, x1 = 4 requires 7 iterations to reach x1 .

I

Starting from x0 = 10, x1 = 8 requires 7 it.s to reach x2 .

The values of f (xk ), starting from x0 = 10, x1 = 8, are:kf (xk )

02.26

1-4.76e-1

2-1.64e-1

32.45e-2

4-9.93e-4

5-5.62e-6

61.30e-9

Number of significant digits increases more rapidly as we getnearer to the root (i.e., better than linear order), but not as fast aswith Newtons method. Thus we observe indeed a demonstrationof superlinear convergence.NumCSE, Lecture 4, Sept 29, 2014

25/40

Numerical Methods for Computational Science and EngineeringSecant method

Higher order methodsHigher order methods could be obtained by (e.g.) determining aquadratic polynomial q(t) such thatf (t) = q(t),

f 0 (t) = q 0 (t),

f 00 (t) = q 00 (t),

or (for three pairwise different ti )f (t1 ) = q(t1 ),

f (t2 ) = q(t2 ),

f (t3 ) = q(t3 ),

and then compute its roots.Questions ariseI

Which of the two roots?

I

What if the roots are complex?

NumCSE, Lecture 4, Sept 29, 2014

26/40

Numerical Methods for Computational Science and EngineeringInverse interpolation

Inverse interpolationGiven a data set(xi , yi = f (xi )),

i = 0, 1, . . . , n.

In inverse interpolation we want to find a position x such that, fora given y , f (x ) = y .If the given function f is monotone in the interval, then for each ythere is only one x for which f (x) = y . In this situation it makessense to interpolate the points (yi , xi = f 1 (yi )).Here: we are looking for x such that x = f 1 (0) f (x ) = 0.

NumCSE, Lecture 4, Sept 29, 2014

27/40

Numerical Methods for Computational Science and EngineeringInverse interpolation

Inverse linear interpolationThe secant method can be derived as inverse linear interpolation:The function that linearly interpolates (yk , f 1 (yk )) = (yk , xk ) and(yk1 , f 1 (yk1 )) = (yk1 , xk1 ) isf 1 (yk )

y yk1y yk f 1 (yk1 )yk yk1yk yk1

The value of this function at y = 0 gives the new approximatexk+1 :xk yk1 + xk1 ykxk+1 =yk yk1xk xk1= xk yk,yk = f (xk ).yk yk1NumCSE, Lecture 4, Sept 29, 2014

28/40

Numerical Methods for Computational Science and EngineeringInverse interpolation

Inverse quadratic interpolationfk2 fk1xk(fk fk2 )(fk fk1 )fk2 fk+xk1(fk1 fk2 )(fk1 fk )fk1 fk+xk2(fk2 fk1 )(fk2 fk )

xk+1 =

Code on next page from Moler, Numerical Computing withMatlab, SIAM, 2004.Convergence order: 1.839. No derivatives needed!!But: IQI needs good starting value!

NumCSE, Lecture 4, Sept 29, 2014

29/40

Numerical Methods for Computational Science and EngineeringInverse interpolation

Inverse quadratic interpolation (cont.)

function [b,k] = iqi(f,x)%x = iqi(f,x) tries to find a zero of f(x) near the initial%guesses x=[x0,x1,x2]. If successful, iqi returns the%inverse quadratic interpolation approximation to a root of f%[x,iter] = iqi(f,x) returns also the number of iterations%required to find the root.a = x(1); b = x(2); c = x(3);k = 0;while abs(c-b) > eps*abs(c)x = polyinterp([f(a),f(b),f(c)],[a,b,c],0);a = b; b = c; c = x;k = k + 1;disp(sprintf(x%d = %16.14f,k,c))end

NumCSE, Lecture 4, Sept 29, 2014

30/40

Numerical Methods for Computational Science and EngineeringInverse interpolation

Example for IQIFind zero of f (x) = xe x 1, x0 = 0, x (1) = 2.5, x (2) = 5.k12345678

xk0.085203900581750.160092526225860.798793818163900.630946367528430.561077509910280.567069410331070.567143317070920.56714329040980

NumCSE, Lecture 4, Sept 29, 2014

f (xk )-0.90721814294134-0.812112296373540.775605340679460.18579323999999-0.01667806436181-0.000204134767660.000000073670670.00000000000003

ek := xk x -0.48193938982803-0.407050764183920.231650527754110.06380307711864-0.00606578049951-0.000073880078720.000000026661140.00000000000001

log |ek+1 |log |ek |log |ek |log |ek1 |

3.337911543788392.287404889122081.824946672897151.873232642142171.798329369804541.84841261527097

31/40

Numerical Methods for Computational Science and EngineeringMatlabs fzero

Matlabs fzeroCombine the reliability of bisection with the convergence speed ofsecant and inverse quadratic interpolation (IQI). Requires onlyfunction evaluation.Outline:I

Start with a and b s.t. f (a)f (b) < 0.

I

Use a secant step to get c between a and b.

I

Repeat the following steps until |b a| < |b| or f (b) = 0.Arrange a, b, and c so that

I

III

f (a)f (b) < 0|f (b)| |f (a)|c is the previous value of b.

I

If c 6= a, consider an IQI step.

I

If c = a, consider a secant step.

NumCSE, Lecture 4, Sept 29, 2014

32/40

Numerical Methods for Computational Science and EngineeringMatlabs fzero

Matlabs fzero (cont.)I

If the IQI or secant step is in the interval [a, b], take it.

I

If the step is not in the interval, bisect.

This algorithm is foolproof: It never loses track of the zero trappedin a shrinking interval.It uses rapidly convergent methods when they are reliable.It uses a slow, but sure, method when it is necessary.It only uses function values, no derivatives.

NumCSE, Lecture 4, Sept 29, 2014

33/40

Numerical Methods for Computational Science and EngineeringComparision of methods

Comparision of methodsComparison of some methods for computing the smallest zero off (x) = cos(x) cosh(x) + 1 = 0. (x 1.8751, tol = 109 )methodbisectionsecant methodsecant methodNewtonfzerofzero

start[0, 3][1.5, 3][0, 3]x0 = 1.5[0, 3][1.5, 3]

# steps32815587

# function evals3491611109

Note: (1) These numbers depend on the function f !Numbers from SchwarzKockler, except for fzero.NumCSE, Lecture 4, Sept 29, 2014

34/40

Numerical Methods for Computational Science and EngineeringComputing multiple zeros

Computing multiple zerosIf we have found a zero z of f (x) = 0 and want to computeanother one, we want to avoid recomputing the already found z.We can explicitely deflate the zero by defining a new functionf1 (x) :=

f (x),x z

and apply our method of choice to f1 . This procedure can inparticular be done with polynomials which can be error prone if zis not accurate.We can proceed similarly for multiple zeros z1 , . . . , zm .

NumCSE, Lecture 4, Sept 29, 2014

35/40

Numerical Methods for Computational Science and EngineeringComputing multiple zeros

Computing multiple zeros (cont.)For the reciprocal Newton correction for f1 we getf10 (x)=f1 (x)

f 0 (x)xz

f (x)(xz)2f (x)xz

=

1f 0 (x).f (x)x z

Then a Newton correction becomesx (k+1) = xk

1f 0 (xk ) 1f (xk ) xk z

and similarly for multiple zeros z1 , . . . , zm .The above procedure is called implicit deflation. f is not modified.In this way errors in z are not propagated to f1NumCSE, Lecture 4, Sept 29, 2014

36/40

Numerical Methods for Computational Science and EngineeringMinimizing a function of one variable

Minimizing a function of one variableI

A major source of applications giving rise to root finding isoptimization.

I

One-variable version: find an argument x = x that minimizesa given objective function (x).

I

Example from earlier: Find the minimum of the functionx(x) = 10 cosh( ) x4over the real line.

I

Note: maximize function (x) minimize (x) = (x).

NumCSE, Lecture 4, Sept 29, 2014

37/40

Numerical Methods for Computational Science and EngineeringMinimizing a function of one variable

Conditions for a minimum pointAssume that C 2 [a, b]. Denotef (x) = 0 (x).An argument x satisfying a < x < b is called a critical point iff (x ) = 0.For parameter h small enough so that x + h [a, b] we canexpand in a Taylors series(x + h) = (x ) + h0 (x ) += (x ) +NumCSE, Lecture 4, Sept 29, 2014

h2 00 (x ) + 2

h2 00 [ (x ) + O(h)].2

38/40

Numerical Methods for Computational Science and EngineeringMinimizing a function of one variable

Conditions for a minimum point (cont.)Since |h| can be taken arbitrarily small, it is now clear that at acritical point:I

If 00 (x ) > 0, then x = x is a local minimizer of (x). Thismeans that attains a minimum at x = x in someneighborhood which includes x .

I

If 00 (x ) < 0, then x = x is a local maximizer of (x). Thismeans that attains a minimum at x = x in someneighborhood which includes x .

I

If 00 (x ) = 0, then a further investigation at x is required.

NumCSE, Lecture 4, Sept 29, 2014

39/40

Numerical Methods for Computational Science and EngineeringMinimizing a function of one variable

Computation of minimia of functionsIf (x) attains a minimum (or maximum) at a point x, then thispoint must be critical, i.e., f (x ) = 0.We can apply any of the zero finders to find x with f (x ) = 0.Example. For the function (x) = 10 cosh(x/4) x we have0 (x) = f (x) =

x10sinh( ) 1,44

00 (x) = f 0 (x) =

10xcosh( ).164

Note that for quadratic convergence of Newtons method, (x)must have three continuous derivatives.Note: the problem of finding all minima of a given function (x) can besolved by finding all the critical roots and then checking for each if it is aminimum by examining the sign of the second derivative of .

NumCSE, Lecture 4, Sept 29, 2014

40/40

Numerical Methods for Computational Science and EngineeringIntroduction

Topics of todayI

Some examples to review some basic concepts of linearalgebra.

ReferenceI

AscherGreif, Section 4.

I

The basic concepts are found in any book on numerical linearalgebra.

NumCSE, Lecture 5, Oct 2, 2014

2/35

Numerical Methods for Computational Science and EngineeringData fitting

Example: Data fittingI

Given measurements, or observations, {(ti , bi )}mi=1 .

I

Want to fit to this data a function v (t) that is a linearcombination of n known, linearly independent basis functionsin t, 1 (t), 2 (t), . . . , n (t),v (t) =

nX

xi i (t).

i=1

by interpolation at the distinct points t1 < t2 < . . . < tm .I

So, we seek coefficients x1 , . . . , xn such thatv (ti ) = bi ,

NumCSE, Lecture 5, Oct 2, 2014

i = 1, . . . , m.

(1)

3/35

Numerical Methods for Computational Science and EngineeringData fitting

Example: Data fitting (cont.)Setting aij := j (ti ) the interpolation requirements becomea11 x1 + a12 x2 + + a1n xn = b1a21 x1 + a22 x2 + + a2n xn = b2...am1 x1 + am2 x2 + + amn xn = bmLet us assume that m = n (i.e., the same number of data andunknown coefficients). Then we can write the above equations asA x = b,

A Rnn

If the j are linearly independent then ... (be careful!) A isnonsingular.NumCSE, Lecture 5, Oct 2, 2014

4/35

Numerical Methods for Computational Science and EngineeringData fitting

Example: Data fitting (cont.)Special choice of basis functions: j (t) = t j1 (monomials).Then v (t) is in the set Pn of polynomials of degree < n and1 t1 t12 t1n11 t2 t 2 t n1 22A = . . . ... ... .. .. ... 1 tn tn2

tnn1

This form of a matrix is called Vandermonde matrix. If the ti aredistinct then A is nonsingular, i.e., the polynomial interpolationproblem always has a unique solution.In fact, det(A) =

Q

1i n, i.e., the system is overdetermined.Imagine values such as n = 1, 2, or 3 and m = 100.

NumCSE, Lecture 5, Oct 2, 2014

7/35

Numerical Methods for Computational Science and EngineeringData fitting

Overdetermined data fitting problem (cont.)Obvious question: in what sense do we want to approximate.Often natural (or at least convenient) to use the 2-norm:minx

mX

2

[bi v (ti ; x)]

i=1

= minx

mXi=1

"bi

nX

#2xj j (ti )

i=1

Reusing rectangular matrix A with elements aij = j (ti ) the linearleast squares problem can be written asminkb Axk2 ,x

NumCSE, Lecture 5, Oct 2, 2014

A Rmn , b Rm , x Rn , m n.

8/35

Numerical Methods for Computational Science and EngineeringData fitting

Numerical example continuedtAAx

====

[0,0.1,0.8,1]; b = [1,-0.9,10,9];fliplr(vander(t));A(:,1:3); % only quadratic polynomialA \ b;% This solves the system Ax = b

NumCSE, Lecture 5, Oct 2, 2014

9/35

Numerical Methods for Computational Science and EngineeringOrdinary differential equation

Example: Ordinary differential equationI

Consider the recovery of a function v (t) from its given secondderivative g (t) on the interval [0, 1].

I

Origin: heat equation. Given heat sources/sinks. Recoversteady-state temperature distribution. Here, homogeneousmaterials.

I

Process is not unique: If v1 solves the problem,v 00 (t) = g (t), then v2 = v1 (t) + + t solves it as well.Boundary conditions are required to fix , .Consider two cases:

I

(1) v (0) = v (1) = 0,(2) v (0) = v 0 (1) = 0.

NumCSE, Lecture 5, Oct 2, 2014

10/35

Numerical Methods for Computational Science and EngineeringOrdinary differential equation

Example: Ordinary differential equation (cont.)I

We subdivide interval [0, 1] into N subintervals of sizeh = 1/N. Let ti = ih, i = 0, . . . , N and approximate v 00 = gby (vj v (tj ))

vi1 2vi + vi+1= g (ti ),h2

i = 1, . . . , N 1

or 1 2 1v0g (t1 ) v1 g (t2 ) 1 2 1 1 .. ..1 2 1=. .h2 ...... vN1 g (tN2 )...1 2 1vNg (tN1 )NumCSE, Lecture 5, Oct 2, 2014

11/35

Numerical Methods for Computational Science and EngineeringOrdinary differential equation

Example: Ordinary differential equation (cont.)I

Insert boundary conditions:(1) v (0) = v (1) = 0,Set v0 = vN = 0. Then the above system becomesN 1 N 1 and is spd.(2) v (0) = v 0 (1) = 0. We set v0 = 0.At the right end of the interval we add two equations:

vN1 2vN + vN+1= g (tN )h2

is the differential equation at t = 1.vN+1 vN1=02his a second-order approximation of the bc v 0 (1) = 0.NumCSE, Lecture 5, Oct 2, 2014

12/35

Numerical Methods for Computational Science and EngineeringOrdinary differential equation

Example: Ordinary differential equation (cont.)The system Av = g becomes square:

211 h2

12 112 1......1

v1v2v3...

g (t1 )g (t2 )g (t3 )...

=... . 2 1 vN1 g (tN1 )22vNg (tN )

The matrix A is not symmetric. It could be symmetrized bymultiplying the last row by 12 .The last row of the system would then read1 vN1 + 1 vN =

NumCSE, Lecture 5, Oct 2, 2014

1g (tN ).2

13/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System

A Global Positioning System (GPS)is a device that uses satellitereadings to determine a receiverssgeocentric coordinates (x, y , z), i.e.,coordinates relative to center ofearth (0, 0, 0).These measurements are accurate toat least 80 meters, usually better.

Picture athttp://www.geologie.ens.fr/~vigny/images/gps-cours1.jpgNumCSE, Lecture 5, Oct 2, 2014

14/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System (cont.)GPS device reads information from anumber of satellites (at least foursatellites are necessary).The ith satellite will send itsgeocentric coordinates (xi , yi , zi ) andthe time of transmission ti .These values are highly accurate.

Picture fromhttp://en.wikipedia.org/wiki/Global_Positioning_System

NumCSE, Lecture 5, Oct 2, 2014

Distances xi , yi , zi are measured inmeters [m] from the center of theearth, and the time ti is measured innanoseconds [nsec]. In GPS, time tiis the fourth dimension.

15/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System (cont.)The receiver gets information of satelite i at time Ti , measured onthe receivers clock. Distance between receiver and satellite can becomputed by i = c(Ti ti ) meters, where c is speed of light:c = 299792458 m/s = 0.299792458 m/nsec.An error of one nanosecond yields a 30 cm error in distance. Thei are called pseudorange.As receiver is a cheap device its clock is out of sync. Since thesatellites clocks are in sync the receiver clock will be out of syncwith each satelite by the same amount.Let t be the error in the receivers clock. Then the error in thepseudorange is b := ct.NumCSE, Lecture 5, Oct 2, 2014

16/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System (cont.) xxiy yi u=si = z zi ,bi| {z }{z}|data from satellite iunknown data about receiverActual distance of satelite i and receiver:q(xi x)2 + (yi y )2 + (zi z)2 + b = i .Thus,(xi x)2 + (yi y )2 + (zi z)2 = (i b)2 ,

i

This is a nonlinear least squares problem.NumCSE, Lecture 5, Oct 2, 2014

17/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System (cont.)Nonlinear least squares problem could be solved (and actually is).Now we derive an algorithm that is usually much faster since it justrequires a linear least squares solution.We rearrange terms to getxi2 + yi2 + zi2 2i 2(xi x + yi y + zi z i b) + (x 2 + y 2 + z 2 b 2 ) = 0.We define a modified dot product, called Minkovski metric,hu, vi = u1 v1 + u2 v2 + u3 v3 u4 v4 .Then the above equation becomes (after division by 2)

NumCSE, Lecture 5, Oct 2, 2014

11hsi , si i hsi , ui + hu, ui = 0,22

i.

(2)

18/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System (cont.)If we have n satellites, then we can write the collection of theseequations as 1x1 y1 z1 1hs1 , s1 ix2 y2 z2 2 11 hs2 , s2 i1B=., a = . , e = . , = hu, ui..........2 . 2..... 1xn yn zn nhsn , sn iB is an n 4 matrix, a and e are n-vectors, and is a scalar. Wenow can simultaneously write the n equations from (2) asa Bu + e = 0

NumCSE, Lecture 5, Oct 2, 2014

Bu = (a + e)

19/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System (cont.)The solution can be written asu = B + (a + e)

(3)

where B + = (B T B)1 B T is the pseudo-inverse of B.However, solution u involves which involves unknown u .We work around this problem with some clever algebraicmanipulations. Substituting u into the definition of gives11 = hu , u i = hB + (a + e), B + (a + e)i22or2 hB + e, B + ei + 2(hB + a, B + ei 1) + hB + a, B + ai = 0.NumCSE, Lecture 5, Oct 2, 2014

(Q)

20/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System (cont.)This is a quadratic equation in that has two solutions 1 and 2that in turn yieldu1 = B + (a + 1 e).

u2 = B + (a + 2 e).

(4)

It turns out that only one of these two solutions is useful, namelythe one that leads to receiver coordinates that are on the surfaceof the earth.To check this, the first three components of u are used.

NumCSE, Lecture 5, Oct 2, 2014

21/35

Numerical Methods for Computational Science and EngineeringGlobal Positioning System

Example: Global Positioning System (cont.)To summarize, here is the Bancroft algorithm for finding u :1. Organize your data into B, a, and e.2. Solve the quadratic equation (Q) for 1 and 2 .3. Solve for the least squares solutions u1 and u2 and pick theone that makes sense.In point 3, we look for that solution that is on the earths surface.

NumCSE, Lecture 5, Oct 2, 2014

22/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Example: Principal Component AnalysisLet x1 , . . . , xN Rp represent N observations, measurements, orother data. The sample mean of these observation vectors ism=

1(x1 + + xN )N

Letxk = xk m,

k = 1, . . . , N.

The columns of the p N matrixB = [x1 , x2 , . . . , xN ]have a zero sample mean. B is said to be in mean-deviation form.

NumCSE, Lecture 5, Oct 2, 2014

23/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Example: Principal Component Analysis (cont.)The (sample) covariance matrix isS=

1BB T Rpp .N 1

Clearly, S is symmetric positive semidefinite.

NumCSE, Lecture 5, Oct 2, 2014

24/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Example: Principal Component Analysis (cont.)Numerical example: 14785x1 = 2 , x2 = 2 , x3 = 8 , x4 = 4 , m = 4 .1131554B = [x1 , x2 , x3 , x4 ] = 241068S = 60 8

NumCSE, Lecture 5, Oct 2, 2014

12 324 0 .8 4 00832

25/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Example: Principal Component Analysis (cont.)Let now x = [x1 , x2 , . . . , xp ]T be a vector that varies over the set ofobservation vectors. The first component x1 , e.g., varies over thefirst components of the observations xi , 1 i N.The diagonal entry sjj is called the variance of xj , j = 1, . . . , p.The variance of xj measures the spread of the values of xj .In the above example, the variance of x1 is 10, while the varianceof x3 is 32. So, the values in x3 are spread wider than those in x1 .The total variance of the data is the sum of all variances, i.e. thetrace tr(S) of S.The entry sij of S for i 6= j is calles the covariance of xi and xj . Ifsij = 0 then xi and xj are uncorrelated.

NumCSE, Lecture 5, Oct 2, 2014

26/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Example: Principal Component Analysis (cont.)Analysis of the multivariate data in x1 , . . . , xN is greatly simplifiedif the covariance matrix is diagonal or nearly so.The goal of the principal component analysis is to find anorthogonal p p matrix P that determines a change of variablesC = PBwith the property that the new variables y1 , . . . , yp areuncorrelated and arranged in order of decreasing variance.

NumCSE, Lecture 5, Oct 2, 2014

27/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Example: Principal Component Analysis (cont.)A=

1BTN 1

=

AT A = S.

LetUV T = Abe the economic singular value decomposition of A. ThenAT A = V U T UV T = S

=

V T SV = 2 = diagonal.

The eigenvalues of S are the squares of the singular values1 2 p 0 of A.

NumCSE, Lecture 5, Oct 2, 2014

total variance = tr(S) =

pX

j2 .

j=1

28/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Example: Principal Component Analysis (cont.)Let V = [v1 , v2 , . . . , vp ]. The vectorsAv1 , Av2 , . . . , Avpare the principal components of B arranged according todescending variance. Note that Avi = ui i .Principal component analysis is valuable for applications in whichmost of the variation, or dynamic range, in the data is due tovariations in only a few of the transformed variables.Numerical example above:12 = 34.5513,

22 = 13.8430,

32 = 1.6057,

tr(S) = 50.

Since 34.5513+ 13.8430= 48.3943= 0.9679 the first two component505050contain 97% of the total variation (information).NumCSE, Lecture 5, Oct 2, 2014

29/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Application: Image processing

From David Lay: Linear Algebra and Its Applications, p. 394.NumCSE, Lecture 5, Oct 2, 2014

30/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Application: point cloud

NumCSE, Lecture 5, Oct 2, 2014

I

Graphics objects in R3 areoften represented by pointclouds (scattered data set ofpoints that together form acoherent surface).

I

Laser scanning, medicalimaging, visualization,...

I

Relevant tasks: clean dataset from noise and outliers,and display the surface thatgave rise to this cloud in apleasing way.

31/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Application: point cloud (cont.)I

For such applications we need the normal directions at thepoints with respect to the implied surface.

I

Finding normals (without forming the surface first!) can bedone using PCA.

I

For fixed point p in cloud, we define neighborhood Np ofnearby points, e.g., points whose distance in 2-norm from p isbelow some value.

I

Let us assume there are n = np such neighbors.

I

We calculate the mean of these neighbors to find the centroidof Np , call it p. Our 3 np data matrix A therefore has pip pfor its ith column, where pip is the ith neighbor of p.

NumCSE, Lecture 5, Oct 2, 2014

32/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Application: point cloud (cont.)I

We find the three singular vectors of A (eigenvectors of thecovariance matrix). The first two principal vectors span thetangent plane at p. The third one, orthogonal to the first twoand hence to the tangent plane, is our desired unsignednormal direction.

I

An additional, global, pass over the points is required todetermine what sign our normal vectors should have.

NumCSE, Lecture 5, Oct 2, 2014

33/35

Numerical Methods for Computational Science and EngineeringPrincipal Component Analysis

Application: point cloud (cont.)I

This procedure to consolidate the point cloud can becomplemented by a procedure to represent the surface byanother set of point, called particles. This is acomplementary set of points that enables the more economicrepresentation of the point cloud.Its determinati