numerical methods for computational science and · pdf filenumerical methods for computational...

48
Numerical Methods for Computational Science and Engineering Numerical Methods for Computational Science and Engineering Lecture 1, Sept 18, 2014: Introduction Peter Arbenz Computer Science Department, ETH Z¨ urich E-mail: [email protected] NumCSE, Lecture 1, Sept 18, 2014 1/48

Upload: trananh

Post on 07-Feb-2018

231 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical Methods for Computational Scienceand Engineering

Lecture 1, Sept 18, 2014: Introduction

Peter ArbenzComputer Science Department, ETH Zurich

E-mail: [email protected]

NumCSE, Lecture 1, Sept 18, 2014 1/48

Page 2: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Introduction

Outline of today’s lecture

I What is numerical methods for CSE

I Survey of the lecture

I Organization of the lecture (exercises/examination)

I References

I Start of the lecture

NumCSE, Lecture 1, Sept 18, 2014 2/48

Page 3: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Introduction

Scientific Computing

NumCSE, Lecture 1, Sept 18, 2014 3/48

Page 4: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

A CSE example

A CSE example: particle accelerators

Observed phenomenon

I Charged particles are accelerated in electromagnetic fields.I Cathode ray tubes: beam of electrons deflected by e-field to

create image (classic TV set).

I Particle accelerator is a device to propel charged particles tohigh speeds by applying electromagnetic fields.

I Paul Scherrer Institut: Numerous accelerator to investigatematerial properties, or for cancer treatment.

I CERN: Large hadron collider (LHC) to generate newelementary particles.

I Linear accelerators vs. cyclotrons. Demo: cyclo.m

NumCSE, Lecture 1, Sept 18, 2014 4/48

Page 5: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

A CSE example

A CSE example: particle accelerators (cont.)

NumCSE, Lecture 1, Sept 18, 2014 5/48

Page 6: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

A CSE example

A CSE example: particle accelerators (cont.)Model

1. Electric fields are modeled by the time-harmonic Maxwellequations (periodic solution (E(x, t) = e(x)e iωt), magneticfield eliminated, all pysical quantities = 1)

curl curl e(x) = λ e(x), div e(x) = 0, x ∈ Ω,

n× e = 0, x ∈ ∂Ω.

Ω is accelerator cavity.

2. Particles move according to Newton’s law of motion

dx(t)

dt= v,

dv(t)

dt=

q

m0(E + v × B) .

where E = Eext + Eself . Eext is E-field from 1. (q charge, m0

rest mass.)

NumCSE, Lecture 1, Sept 18, 2014 6/48

Page 7: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

A CSE example

A CSE example: particle accelerators (cont.)Discretization

1. e-field in the Maxwell equation is a continuous function.Approximate it by a finite element function.

This leads to a large sparsegeneralized eigenvalue problem

Ax = λMx.

Size can be in millions.

2. Particles are particles.

Big issue is the computation of the self-field, i.e.,particle-particle interaction.

NumCSE, Lecture 1, Sept 18, 2014 7/48

Page 8: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

A CSE example

A CSE example: particle accelerators (cont.)Discretization (cont.)

I Computing particle-particle interactions costs O(n2p)operations in the number of particles np.

I Remedy: Approximately compute the electric potential ϕ(x)that is due to the charged particles. To that end we have tosolve the Poisson equation

−∆ϕ(x) =1

ε0ρ(x)

with some boundary conditions.The quantity ρ(x) on the right is the ‘charge density’ which isin fact approximated on a regular grid.

NumCSE, Lecture 1, Sept 18, 2014 8/48

Page 9: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

A CSE example

A CSE example: particle accelerators (cont.)Also the solution ϕ(x) is obtained on this regular grid.

The gradient of ϕ(x) provides the force that acts on theparticle at position x,

Eself = −grad ϕ(x).

NumCSE, Lecture 1, Sept 18, 2014 9/48

Page 10: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

A CSE example

A CSE example: particle accelerators (cont.)Solution algorithms

I We need to solve systems of linear equations

Ax = b,

where A is large and sparse, maybe positive definite.

I In a time-dependent problem we may have to solve such asystem in every time step.

I For the Poisson problem on regular grids there are fastPoisson solvers that are related to the fast Fourier transform.

I We need to solve eigenvalue problems

Ax = λMx,

where A and M are large and sparse to get the driving e-field.

NumCSE, Lecture 1, Sept 18, 2014 10/48

Page 11: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

A CSE example

A CSE example: particle accelerators (cont.)Efficiency, accuracy

I These problems are to be solved efficiently w.r.t. time andmemory.

I Linear vs. nonlinear models.

I Direct vs. iterative solution of linear systems.

I High vs. relaxed accuracy.

I Repeated solution of equal/similar problems may amortizeexpensive solution procedures.

NumCSE, Lecture 1, Sept 18, 2014 11/48

Page 12: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Survey of the lecture

Survey of the lecture

1. Introduction

2. Roundoff errors

3. Nonlinear equations in one variable (2 lectures)

4. Linear algebra basics (Read sections 4.1–4.4 in Ascher–Greif)

5. Direct methods for linear system, pivoting strategies, sparsematrices (2)

6. Linear least squares problems (2)

7. Iterative methods for linear system (2)

8. Eigenvalues and singular values (2)

9. Nonlinear systems and optimization (3)

10. (Piecewise) polynomial interpolation (3)

NumCSE, Lecture 1, Sept 18, 2014 12/48

Page 13: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Survey of the lecture

Survey of the lecture (cont.)11. Best approximation

12. Filtering algorithms, Fourier transform

13. Numerical differentiation

14. Numerical integration (2)

15. Ordinary differential equations, initial value problems (3)

NumCSE, Lecture 1, Sept 18, 2014 13/48

Page 14: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

About this course

About this course

Focus

I on algorithms (principles, scope, and limitations),

I on (efficient, stable) implementations in Matlab,

I on numerical experiments (design and interpretation).

No emphasis on

I theory and proofs (unless essential for understanding ofalgorithms)

I hardware-related issues (e.g. parallelization, vectorization,memory access)

(These aspects will be covered in the course“High Performance Computing for Science and Engineering”offered by D-INFK)

NumCSE, Lecture 1, Sept 18, 2014 14/48

Page 15: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

About this course

Goals

• Knowledge of the fundamental algorithms in numericalmathematics

• Knowledge of the essential terms in numerical mathematicsand the techniques used for the analysis of numericalalgorithms

• Ability to choose the appropriate numerical method forconcrete problems

• Ability to interpret numerical results

• Ability to implement numerical algorithms efficiently inMatlab

Indispensable: Learning by doing (Ô exercises)

NumCSE, Lecture 1, Sept 18, 2014 15/48

Page 16: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

About this course

Literature

Uri Ascher & Chen Greif: A First Course in NumericalMethods. SIAM, 2011.http://www.siam.org/books/cs07/

I Excellent reference.

I Main reference for large parts of thiscourse.

I Target audience: undergraduate studentsin computer science.

I I will follow this book quite closely.

NumCSE, Lecture 1, Sept 18, 2014 16/48

Page 17: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

About this course

Literature (cont.)

W. Dahmen & A. Reusken: Numerik fur Ingenieure undNaturwissenschaftler, Springer, 2006.A lot of simple examples and good explanations, but also rigorous mathematical treatment. Target

audience: undergraduate students in science and engineering.

H.-R. Schwarz & N. Kockler: Numerische Mathematik.Teubner, 2006. 6. Auflage.Easy to read. Target audience: undergraduate students in science and engineering.

C. Moler: Numerical Computing with Matlab. SIAM 2004.Good reference for some parts of this course; Target audience: Matlab users and programmers. See

http://www.mathworks.ch/moler/.

W.Gander, M.J. Gander, & F. Kwok: Scientific Computing.An introduction using Maple and Matlab. Springer 2014.

NumCSE, Lecture 1, Sept 18, 2014 17/48

Page 18: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

About this course

Prerequisites

Essential prerequisite for this course is a solid knowledge in linearalgebra and calculus. Familiarity with the topics covered in thefirst semester courses is taken for granted, see

• K. Nipp and D. Stoffer, Lineare Algebra, vdfHochschulverlag, Zurich, 5 ed., 2002.

• M. Gutknecht, Lineare algebra, lecture notes, SAM, ETHZurich, 2009.http://www.sam.math.ethz.ch/~mhg/unt/LA/HS07/.

• M. Struwe, Analysis fur Informatiker. Lecture notes, ETHZurich, 2009.

NumCSE, Lecture 1, Sept 18, 2014 18/48

Page 19: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Organization

OrganizationLecturer:Prof. Peter Arbenz [email protected]

Assistants:Daniel Hupp [email protected] Schuller [email protected] Lobbe [email protected] Jagathrakashakan [email protected] Bohn [email protected] Moser [email protected] Thuring [email protected] Welti [email protected]

NumCSE, Lecture 1, Sept 18, 2014 19/48

Page 20: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Organization

VenueClasses: Mon 10.15-12.00 (CAB G11); Thu 10.15-12.00 (HG G5)Tutorials: Mon 13.15-15.00

Thu 8.15-10.00

Please register (on course website) for tutorial groups untilSeptember 23th:http://www.math.ethz.ch/education/bachelor/lectures/

hs2014/math/nummath_cse

Consulting hours: if needed, see the course website.

NumCSE, Lecture 1, Sept 18, 2014 20/48

Page 21: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Organization

Assignments

• The assignment sheets will be uploaded on the coursewebpage on Monday every week the latest.

• The exercise should be solved until the following tutorial class.(Hand them in to the assistant.)

NumCSE, Lecture 1, Sept 18, 2014 21/48

Page 22: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Organization

Examination

• Three-hour written examination involving coding problems tobe done at the computer on

TBA

• Dry-run for computer based examination:Does not exist anymore.Try out a computer in the student labs in HG.

• Pre-exam question session:TBA

NumCSE, Lecture 1, Sept 18, 2014 22/48

Page 23: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Organization

Examination (cont.)• Topics of examination:

I All topics, that have been addressed in class or in ahomework assignment.

I One exam question will be one of the homeworkassignment.

• Lecture slides will be available as (a single) PDF file duringthe examination.

• The Ascher-Greif book will be made available, too.

• The exam questions will be asked in English.

NumCSE, Lecture 1, Sept 18, 2014 23/48

Page 24: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Computing environment: Matlab

Problem solving environment: Matlab

I We use Matlab for the exercises.I Although most of the algorithm we are dealing with have been

implemented in Matlab, it is useful when you program themyourselves.

I These (little) programs will be building blocks when you willsolve more complex problems in your future.

I Matlab helpI Matlab commands help/docI Matlab online documentation, e.g.,

http://www.mathworks.nl/help/pdf_doc/allpdf.html

I Numerous introductory textbooks / user guides / primersI My own Matlab introduction:

http://people.inf.ethz.ch/arbenz/MatlabKurs/

matlabintro.pdf

NumCSE, Lecture 1, Sept 18, 2014 24/48

Page 25: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Numerical algorithms and errors

I The most fundamental feature of numerical computing is theinevitable presence of errors.

I The result of any interesting computation (and of manyuninteresting ones) is typically only approximate, and our goalis to ensure that the resulting error is tolerably small.

I Example: How many loop iterations are there in this littleMatlab program?

x = 0; h = 1/10;

while x<1,

x=x+h;

% do something depending on x

end

NumCSE, Lecture 1, Sept 18, 2014 25/48

Page 26: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

How to measure errors

I Can measure errors as absolute or relative, or a combinationof both.

I The absolute error in v approximating given scalar quantity uis |u − v |.

I The relative error (assuming u 6= 0) is|u − v ||u| .

u v Absolute RelativeError Error

1 0.99 0.01 0.011 1.01 0.01 0.01

–1.5 –1.2 0.3 0.2100 99.99 0.01 0.0001100 99 1 0.01

NumCSE, Lecture 1, Sept 18, 2014 26/48

Page 27: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Approximation example

The Stirling approximation

v = Sn =√

2πn ·(ne

)nis used to approximate u = n! = 1 · 2 · · · n for large n.

The formula involves the constant e = exp(1) = 2.7182818 . . ..

We use the Matlab program Example1 1.m fromhttp://www.siam.org/books/cs07/programs.zip

to compute the values of u and v for n = 1, . . . , 10.

Notice that the quality of v approximating u increases althoughthe absolute error increases rapidly!

NumCSE, Lecture 1, Sept 18, 2014 27/48

Page 28: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Types of errors

I Errors in the formulation of the problem to be solved.I Errors in the mathematical model. Simplifications.I Error in input data. Measurements.

I Approximation errorsI Discretization error.I Convergence error in iterative methods.I Discretization/convergence errors may be assessed by an

analysis of the method used.

I Roundoff errorsI Roundoff errors arise everywhere in numerical computation

because of the finite precision arithmetic.I Roundoff errors behave quite erratic.

NumCSE, Lecture 1, Sept 18, 2014 28/48

Page 29: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Discretization errors in action

Problem: want to approximate the derivative f ′(x0) of a givensmooth function f (x) at the point x = x0.

Example: Let f (x) = sin(x), −∞ < x <∞, and set x0 = 1.2.Thus, f (x0) = sin(1.2) ≈ 0.932 . . ..

Discretization: Function values f (x) are available only at a discretenumber of points, e.g. at grid points xj = x0 + jh, j ∈ Z.

Want to approximate f ′(x0) by values f (xj).

NumCSE, Lecture 1, Sept 18, 2014 29/48

Page 30: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Discretization errors in action (cont.)Taylor’s series gives us an algorithm to approximate f ′(x0):

f ′(x0) ≈ f (x0 + h)− f (x0)

h

NumCSE, Lecture 1, Sept 18, 2014 30/48

Page 31: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Discretization errors in action (cont.)Expanding f (x) by a Taylor series around x = x0 gives

f (x0 + h)− f (x0)

h= f ′(x0)− h

2f ′′(ξ), x0 < ξ < x0 + h.

So, we expect the error to decrease linearly with h.

∣∣∣∣f ′(x0)− f (x0 + h)− f (x0)

h

∣∣∣∣ =h

2

∣∣f ′′(ξ)∣∣ ≈ h

2

∣∣f ′′(x0)∣∣

Or, using the big-O notation:∣∣f ′(x0)− Dx0,h(f )∣∣ = O(h).

NumCSE, Lecture 1, Sept 18, 2014 31/48

Page 32: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Results

Try for f (x) = sin(x) at x0 = 1.2.

(So, we are approximating cos(1.2) = 0.362357754476674 . . .)

h Absolute error

0.1 4.71667 · 10−2

0.01 4.666196 · 10−3

0.001 4.660799 · 10−4

10−4 4.660256 · 10−5

10−7 4.619326 · 10−8

These results reflect the discretization error as expected.

Note that f ′′(x0)/2 = − sin(1.2)/2 ≈ −0.466.

NumCSE, Lecture 1, Sept 18, 2014 32/48

Page 33: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Results for smaller h

The above results indicate that we can compute the derivative asaccurate as we like, provided that we take h small enough.

If we wanted∣∣∣∣cos(1.2)− sin(1.2 + h)− sin(1.2)

h

∣∣∣∣ < 10−10.

We have to set h ≤ 10−10/0.466.

The following numbers and plot are generated by.../Greif/programs/chap01/Example1 3Figure1 3.m

NumCSE, Lecture 1, Sept 18, 2014 33/48

Page 34: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Results for smaller h

h Absolute error

10−8 4.36105 · 10−10

10−9 5.594726 · 10−8

10−10 1.669696 · 10−7

10−11 7.938531 · 10−6

10−13 6.851746 · 10−4

10−15 8.173146 · 10−2

10−16 3.623578 · 10−1

These results reflect both discretization and roundoff errors.

NumCSE, Lecture 1, Sept 18, 2014 34/48

Page 35: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Numerical algorithms and errors

Results for all h

The solid curve interpolates the computed values of|f ′(x0)− f (x0+h)−f (x0)

h | for f (x) = sin(x) at x0 = 1.2.

The dash-dotted straight line depicts the discretization errorwithout roundoff error.

NumCSE, Lecture 1, Sept 18, 2014 35/48

Page 36: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Algorithm properties

Algorithm properties

Performance features that may be expected from a good numericalalgorithm.

I AccuracyRelates to errors. How accurate is the result going to be whena numerical algorithm is run with some particular input data.

I EfficiencyI How fast can we solve a certain problem?

Rate of convergence. Floating point operations (flops).I How much memory space do we need?I These issues may affect each other.

I Robustness(Numerical) software should run under all circumstances.Should yield correct results to within an acceptable error orshould fail gracefully if not successful.

NumCSE, Lecture 1, Sept 18, 2014 36/48

Page 37: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Algorithm properties

Complexity I

Complexity/computational cost of an algorithm :⇐⇒ number ofelementary operatorsAsymptotic complexity = “leading order term” of complexity w.r.t.large problem size parametersThe usual choice of problem size parameters in numerical linearalgebra is the number of independent real variables needed todescribe the input data (vector length, matrix sizes).

operation description #mul/div #add/sub

inner product (x ∈ Rn, y ∈ Rn) 7→ xHy n n − 1 O(n)outer product (x ∈ Rm, y ∈ Rn) 7→ xyH nm 0 O(mn)tensor product

matrix product (A ∈ Rm×n,B ∈ Rn×k) 7→ AB mnk mk(n − 1) O(mnk)

NumCSE, Lecture 1, Sept 18, 2014 37/48

Page 38: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Algorithm properties

Big-O and Θ notation

For an error e depending on h we denote

e = O(hq)

if there are two positive constants q and C such that

|e| ≤ C hq ∀h > 0 small enough.

Similarly, for w = w(n) the expression

w = O(n log n)

means that there is a constant C > 0 such that

|w | ≤ Cn log n as n→∞.

NumCSE, Lecture 1, Sept 18, 2014 38/48

Page 39: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Algorithm properties

Big-O and Θ notation

More abstract:Class O(f ) of functions is defined as

O(f ) = g | ∃c1, c2 > 0 : ∀N ∈ Z+ : g(N) ≤ c1f (N) + c2

The Θ notation signifies a stronger relation than the O notation:a function φ(h) for small h (resp., φ(n) for large n) is Θ(ψ(h))(resp., Θ(ψ(n))) if φ is asymptotically bounded both above andbelow by ψ.

Example:O(h2) means at least “quadratic convergence” (see later). Θ(h2)is exact quadratic convergence.

NumCSE, Lecture 1, Sept 18, 2014 39/48

Page 40: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Algorithm properties

Complexity II

To a certain extent, the asymptotic complexity allows to predictthe dependence of the runtime of a particular implementation ofan algorithm on the problem size (for large problems). Forinstance, an algorithm with asymptotic complexity O(n2) is likelyto take 4× as much time when the problem size is doubled.

One may argue that the memory accesses are more decisive for runtimes than floating point operations. Often there is a lineardependence among the two. So, there is no difference in the Onotation.

NumCSE, Lecture 1, Sept 18, 2014 40/48

Page 41: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Elementary operations

Scaling

Scaling ≡ multiplication with diagonal matrices (with non-zerodiagonal entries) from left and/or right.

It is important to know the different effects of multiplying with adiagonal matrix from left or right:

DA vs. AD with Rn×n 3 A,D = diag(d1, . . . , dn)

Scaling with D = diag(d1, . . . , dn)in Matlab:

y = diag(d)*x;

or

y = d.*x;

NumCSE, Lecture 1, Sept 18, 2014 41/48

Page 42: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Elementary operations

Elementary matrices

Matrices of the form A = I + αuvT are called elementary.

Again we can apply A to a vector x in a straightforward and amore clever way:

Ax = (I + αuvT )x

orAx = x + αu(vTx)

Cf. exercises.

NumCSE, Lecture 1, Sept 18, 2014 42/48

Page 43: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Elementary operations

Problem conditioning and algorithm stability

Qualitatively speaking:

I The problem is ill-conditioned if a small perturbation in thedata may produce a large difference in the result.The problem is well-conditioned otherwise.

I The algorithm is stable if its output is the exact result of aslightly perturbed input.

NumCSE, Lecture 1, Sept 18, 2014 43/48

Page 44: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Elementary operations

An unstable algorithm

Ill-conditioned problem of computing output values y from inputvalues x by y = g(x): when x is slightly perturbed to x , the resulty = g(x) is far from y .

NumCSE, Lecture 1, Sept 18, 2014 44/48

Page 45: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Elementary operations

A stable algorithm

An instance of a stable algorithm for computing y = g(x): theoutput y is the exact result, y = g(x), for a slightly perturbedinput, i.e., x which is close to the input x . Thus, if the algorithm isstable and the problem is well-conditioned, then the computedresult y is close to the exact y .

NumCSE, Lecture 1, Sept 18, 2014 45/48

Page 46: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Elementary operations

Unstable algorithm

Problem statement: evaluate the integrals

yn =

∫ 1

0

xn

x + 10dx , for n = 0, 1, 2, . . . , 30.

Algorithm development: observe that analytically, for n > 0,

yn + 10yn−1 =

∫ 1

0

xn + 10xn−1

x + 10dx =

∫ 1

0xn−1 dx =

1

n.

Also,

y0 =

∫ 1

0

1

x + 10dx = log(11)− log(10).

Algorithm:

I Evaluate y0 = log(1.1).I For n = 1, 2, . . . , 30, evaluate yn = 1

n − 10yn−1.NumCSE, Lecture 1, Sept 18, 2014 46/48

Page 47: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Elementary operations

Unstable algorithm (cont.)Run the Matlab program Example1 6.m by Ascher and Greif tosee the catastrophic amplification of roundoff errors.This code is available fromhttp://www.siam.org/books/cs07/programs.zip.

NumCSE, Lecture 1, Sept 18, 2014 47/48

Page 48: Numerical Methods for Computational Science and · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What is numerical

Numerical Methods for Computational Science and Engineering

Elementary operations

Unstable algorithm (cont.)

Roundoff error accumulation

I In general, if En is error after n elementary operations, cannotavoid linear roundoff error accumulation

En ' c0nE0.

I Will not tolerate an exponential error growth such as

En ' cn1E0, for some constant c1 > 1.

This is an unstable algorithm.

NumCSE, Lecture 1, Sept 18, 2014 48/48