A History of Numerical Analysis Ideas

Alan Kaylor ClineDepartment of Computer SciencesThe University of Texas at Austin

Prepared for CS 378History of Computing

October 14, 2003

What is Different in Numerical Computing?

What is Different in Numerical Computing?

Well, it’s numbers

Scientific Computing vs.

Numerical Analysis

Your Original CS Department

1966

Your Original CS Department

196610 faculty with 2 numerical analysts

Is there anything special with numerical computing?

A Small Example

A computation of π

Simple iteration:

PP

nn n

n FHGIKJ

FHGG

IKJJ1

2

2 2 1 12

n 2 3, ,...

2 2 2P

2 2.828427124746190 3 3.061467458920719 4 3.121445152258053 5 3.136548490545941 6 3.140331156954739 7 3.141277250932757 8 3.141513801144145 9 3.141572940367883 10 3.141587725279961 11 3.141591421504635 12 3.141592345611077 13 3.141592576545004 14 3.141592633463248 15 3.141592654807589 16 3.141592645321215 17 3.141592607375720 18 3.141592910939673 19 3.141594125195191 20 3.141596553704820 21 3.141596553704820 22 3.141674265021758 23 3.141829681889202 24 3.142451272494134 25 3.142451272494134 26 3.162277660168380 27 3.162277660168380 28 3.464101615137754 29 4.000000000000000 30 0.000000000000000 31 0.000000000000000

Result of 15 digit computation

Red digits are correct

White and green digits are incorrect

2 2.828427124746190 3 3.061467458920719 4 3.121445152258053 5 3.136548490545941 6 3.140331156954739 7 3.141277250932757 8 3.141513801144145 9 3.141572940367883 10 3.141587725279961 11 3.141591421504635 12 3.141592345611077 13 3.141592576545004 14 3.141592633463248 15 3.141592654807589 16 3.141592645321215 17 3.141592607375720 18 3.141592910939673 19 3.141594125195191 20 3.141596553704820 21 3.141596553704820 22 3.141674265021758 23 3.141829681889202 24 3.142451272494134 25 3.142451272494134 26 3.162277660168380 27 3.162277660168380 28 3.464101615137754 29 4.000000000000000 30 0.000000000000000 31 0.000000000000000 . . .

Result of 15 digit computation

Red digits are correct

White and green digits are incorrect

π = 0 ?

Where’s the problem?

Where’s the problem?

1 1 small

PP

nn n

n FHGIKJ

FHGG

IKJJ1

2

2 2 1 12

is calculated as zero

1 1 small

Let’s replace

with the algebraically identical expression

1 1

small

small

New iteration:

n 2 3, ,...

2 2 2P

12

2

1 1 ( )2

n n

nn

P PP

results in …

2 2.828427124746190 3 3.061467458920719 4 3.121445152258053 5 3.136548490545941 6 3.140331156954739 7 3.141277250932757 8 3.141513801144145 9 3.141572940367883 10 3.141587725279961 11 3.141591421504635 12 3.141592345611077 13 3.141592576545004 14 3.141592633463248 15 3.141592654807589 16 3.141592645321215 17 3.141592607375720 18 3.141592910939673 19 3.141594125195191 20 3.141596553704820 21 3.141596553704820 22 3.141674265021758 23 3.141829681889202 24 3.142451272494134 25 3.142451272494134 26 3.162277660168380 27 3.162277660168380 28 3.464101615137754 29 4.000000000000000 30 0.000000000000000 31 0.000000000000000

2 2.828427124746190 3 3.061467458920719 4 3.121445152258053 5 3.136548490545940 6 3.140331156954753 7 3.141277250932773 8 3.141513801144301 9 3.141572940367091 10 3.141587725277160 11 3.141591421511200 12 3.141592345570118 13 3.141592576584872 14 3.141592634338563 15 3.141592648776985 16 3.141592652386591 17 3.141592653288992 18 3.141592653514593 19 3.141592653570993 20 3.141592653585093 21 3.141592653588618 22 3.141592653589499 23 3.141592653589719 24 3.141592653589774 25 3.141592653589788 26 3.141592653589792 27 3.141592653589793 28 3.141592653589793 29 3.141592653589793 30 3.141592653589793 31 3.141592653589793

π correct to all digits

Boring…

Is that all there isto numerical analysis?

Not so boring if the result of this

computation affects

Not so boring if the result of this

computation affects• The ability of the next plane you fly to stay in the air

Not so boring if the result of this

computation affects• The ability of the next plane you fly to stay in the air

• The integrity of the next bridge you cross

Not so boring if the result of this

computation affects• The ability of the next plane you fly to stay in the air

• The integrity of the next bridge you cross

• The state of the economy on which you live

Not so boring if the result of this

computation affects• The ability of the next plane you fly to stay in the air

• The integrity of the next bridge you cross

• The state of the economy on which you live

• The path of a missile that isn’t intended to strike you

So what are the common problems of numerical

analysis?

So what are the common problems of numerical

analysis?Application areas:

So what are the common problems of numerical

analysis?Application areas:

• Petroleum modeling

So what are the common problems of numerical

analysis?Application areas:

• Petroleum modeling• Atomic energy – including weapons

So what are the common problems of numerical

analysis?Application areas:

• Petroleum modeling• Atomic energy – including weapons • Weather modeling

So what are the common problems of numerical

analysis?Application areas:

• Petroleum modeling• Atomic energy – including weapons • Weather modeling • Other modeling such as aircraft and automobile

So what are the common problems of numerical

analysis?Algorithm areas:

So what are the common problems of numerical

analysis?Algorithm areas:

• Linear Equations

So what are the common problems of numerical

analysis?Algorithm areas:

• Linear Equations • Nonlinear equations - single and systems

So what are the common problems of numerical

analysis?Algorithm areas:

• Linear Equations • Nonlinear equations - single and systems• Optimization

So what are the common problems of numerical

analysis?Algorithm areas:

• Linear Equations • Nonlinear equations - single and systems• Optimization • Data Fitting - interpolation and approximation

So what are the common problems of numerical

analysis?Algorithm areas:

• Linear Equations • Nonlinear equations - single and systems• Optimization • Data Fitting - interpolation and approximation• Integration

So what are the common problems of numerical

analysis?Algorithm areas:

• Linear Equations • Nonlinear equations - single and systems• Optimization • Data Fitting - interpolation and approximation• Integration • Differential Equations - ordinary and partial

Didn’t we study that stuff in math

classes?

Didn’t we study that stuff in math

classes?Yes, but as the Pi Example shows, math classes are

just the beginning

Let’s get back to history

Why were computers used primarily for numerical problems

initially?

Why were computers used primarily for numerical problems

initially?

• Mathematicians and engineers designed them

Why were computers used primarily for numerical problems

initially?

• Mathematicians and engineers designed them• A history of algorithms in that area

Why were computers used primarily for numerical problems

initially?

• Mathematicians and engineers designed them• A history of algorithms in that area• Immediate war-time and post-war-time applications

Why were computers used primarily for numerical problems

initially?

• Mathematicians and engineers designed them• A history of algorithms in that area• Immediate war-time and post-war-time applications• Applications did not depend upon having a large number of computers

Why were computers used primarily for numerical problems

initially?

• Mathematicians and engineers designed them• A history of algorithms in that area• Immediate war-time and post-war-time applications• Applications did not depend upon having a large number of computers• However, there were non-numerical examples ENIGMA

What were the major computing ideas that arose in

numerical analysis?

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis

input output

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis

input output

true operation

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis

input output

true operation

approximate operation

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis

input output

true operation

approximate operation

error

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis

input output

true operation

approximate operation

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis

input output

true operation

approximate operation

backward error

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN

•Mathematical based•Computationally Efficient•Portable•Standardized – 3 times

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN • Mathematical software packages

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN • Mathematical software packages•IMSL

•Eispack •Linpack

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN • Mathematical software packages • NANET

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN • Mathematical software packages • NANET•Weekly information about people, problems, and papers•Software repository

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN • Mathematical software packages • NANET• Matlab, Mathematica

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN • Mathematical software packages • NANET• Matlab, MathematicaScientific computing environments

What were the major computing ideas that arose in

numerical analysis?

• Backward error analysis • FORTRAN • Mathematical software packages • NANET• Matlab, Mathematica•Super computers - Parallelism