engineering computation lecture 3 iterative solution of simultaneous equations...

1

Engineering Computation ECL3-1

ENGINEERING COMPUTATION Lecture 3 Stephen Roberts Michaelmas Term

Iterative Solution of Simultaneous Equations

Topics covered in this section:

1. Iterative algorithms and the need to test their convergence. 2. The Jacobi method of solution to solve Ax=b 3. The Gauss-Seidel algorithm. 4. Awareness of other numerical approached to solving Ax=b


Introduction So far we have discussed the solution of the simultaneous linear equation set bAx = , and the conditions for ill-conditioning. The underlying assumption was that we would go ahead and solve the set of equations by a direct method, such as Gaussian elimination. How many calculations would this take?

Consider N equations in N variables, where A is an N×N matrix. Assume additions are quick, and multiplications cost time.

2


Example: Consider using the echelon method to solve

=

4

3

1

643

432

321

z

y

x

Process the augmented matrix as follows:

−

−−

→

−−

−−

→

1

1

1

100

210

321

1

1

1

320

210

321

4

3

1

643

432

321

2row2 - row33row1 - row3

2row1 - row2

Then back-substitute: 21321121 =⇒=++=⇒=−= xzyxyz-y-z . .

.

The reduction to echelon form took 4×2 + 3×1 = 11 multiplications. The back substitution took a further 1 + 2 + 3 multiplications or divisions, making a total of 17.


Introduction General case:

Consider N equations in N variables, where A is an N×N matrix. Assume additions are quick, and multiplications cost time.

Reduction to echelon form will take ~ N(N-1) + (N-1)(N-2) + ..... ~ N

3 multiplications. For large systems (say 10

3 equations) this can be

a very large number (~ 109), and solutions may take a long time, even

with modern fast computers. Can we use a better approach?

3


Iterative Methods and Convergence


Iterative Methods are useful where successive approximations generate solutions which converge on the exact solutions. They are a viable alternative to direct methods and can provide solutions of sufficient accuracy with far fewer calculations. Iterative Methods are efficient when:

• An approximate solution close to the exact solution is known. Only a few iterations are required to get to an acceptable solution.

• The matrix A is of high dimension.

• The matrix A is sparse, with terms mainly near the diagonal. Many engineering problems satisfy this criterion, as the physical interactions between elements may only be local (eg circuit analysis, boundary value probs., PDEs)

• The matrix A is diagonally dominated (the largest elements are along the diagonal), and the iterative method converges quickly.

4


Revision: Iterative solutions of equations in one variable

We attempt to solve the equation ( ) 0=xf . e.g. ( ) 0cos =−= xxxf .

Rearrange the equation into the form ( )xgx = e.g. xx cos= .

Choose a starting value x0, and repeat the iterative calculation

( ) LL,2,1,0 ,1 ==+ nxgx nnuntil xn converges on a solution.


In our example, using nn

xx cos1 =+ , we get the following results, starting with x0

= 0.7000: n xn Error en 0 0.7000 -0.0391 1 0.7648 0.0258 2 0.7215 -0.0176 3 0.7508 0.0117 ... .......... 10 0.7383 -0.007

This converges (somewhat slowly! Try it on your calculator) to the solution 0.7391 accurate to 4 figures. But there is no guarantee that the method will converge! Consider an alternative formulation nn xx

1

1 cos−

+ = .

n xn 0 0.7000 1 0.7954 2 0.6511 3 0.8617 This is diverging!

=> need a method to show will converge

5


Test for Convergence

Let the true solution of ( )xgx = be sx = , so that ( )sgs = .

The error en+1 after n + 1 iterations is given by

( ) ( )sgxgsxe nnn −=−= ++ 11

Expanding the first two terms of a Taylor series about sxn = ,

( ) ( ) ( ) ( ) ( ) ( ) LL +′+=+′−+= sgesgsgsxsgxg nnn

Thus ( )sgee nn′≈+1 . (1)

The iteration scheme will converge if ∞→→ nen as 0 .


Thus ( )sgeenn

′≈+1

.

The iteration scheme will converge if ∞→→ ne

n as 0 .

From ((1), previous slide) The condition for convergence is ( ) 1<′ sg for a

starting point x0 sufficiently close to the solution s .

In the example ( )nn

xxg cos= , ( ) ( ) 6736.0sin =−=′ 0.73910.7391g →

converges.

But for ( )nn

xxg 1cos−= , ( ) ( ) 4845.11 2

12 =−−=′

−

xg 0.7391 → diverges!

It is necessary to test any iterative scheme for convergence.

6


Iterative Methods for solving Ax=b


Iterative Methods for solving Ax=b Let’s now consider how a related idea can be applied to develop an iterative technique for solving Ax=b. As with the case of the one dimensional case, let’s translate Ax=b into an equivalent system of the form x=Tx+c. Then having selected an initial guess x0 we can apply a similar method to that outlined above.

7


Jacobi Algorithm for solving bAx = . Divide A into a diagonal matrix D and two triangular matrices L and U .

For a 3×3 matrix,

ULDA ++=

+

+

=

=

000

00

0

0

00

000

00

00

00

23

1312

3231

21

33

22

11

333231

232221

131211

A

AA

AA

A

A

A

A

AAA

AAA

AAA

Rewrite bAx = as ( ) bxULD =++ and re-arrange to give

( )xULbDx +−= .

Now, the inverse D-1 of D is simply the diagonal matrix of the inverse of the diagonal elements of A, so write

( ) cBxbDxULDx +=++−= −− 11

This is in the required form Tx+c and suggests the Jacobi iterative scheme:

( ) cBxbDxULDx +=++−= −−

+ nnn

11

1


Example: Jacobi solution of weighted chain.

Consider a hanging chain of m + 1 light links with fixed ends at height x0 = xm+1 = 0. It could be the supporting chain for the Clifton

suspension bridge in Bristol, shown ⇒. There is unit tension in the chain, the links are of unit length. A downward force bj is applied on the jth hinge. If we make the approximation that the links make small angles to the horizontal, resolving forces vertically at each hinge gives:

mjbxxx jjjj ≤≤=+− +− 1 , 2 11

8


This is a matrix equation of the form bAx = . In the case m = 5, there are 5 links

−

−

−

−

−

=

21000

12100

01210

00121

00012

A

Dividing the matrix A into D + L+ U , and taking the case where

[ ]T1,1,1,11,=b gives the Jacobi iteration algorithm in the form

( ) cBxbDxULDx +=++−= −−

+ nnn

11

1 :

−

=+

21

21

21

21

21

21

21

21

21

21

21

21

21

1

0000

000

000

000

0000

nn xx


this is easy to code up in MATLAB.

% Jchain.m

% solves weighted chain with m+1 links with n iterations

function jchain(m,n)

b = ones(m+2,1); % array of 1's - downward forces

x = zeros(m+2,1); % array of 0's - initial guess of position

xnew = x;

fprintf('Iteration 0: ');

fprintf('%.4f ',x);

fprintf('\n');

for i = 1:n % n iterations

for j = 2:m+1 % miss out end points

xnew(j) = 0.5*(x(j-1) + x(j+1) - b(j));

end

x = xnew;

fprintf('Iteration %2d: ', i);

fprintf('%.4f ',x);

fprintf('\n');

end

9


Running this, with m = 5 gives, for the first 10 iterations:

» jchain(5,10)

Iteration 0: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

Iteration 1: 0.0000 -0.5000 -0.5000 -0.5000 -0.5000 -0.5000 0.0000

Iteration 2: 0.0000 -0.7500 -1.0000 -1.0000 -1.0000 -0.7500 0.0000

Iteration 3: 0.0000 -1.0000 -1.3750 -1.5000 -1.3750 -1.0000 0.0000

Iteration 4: 0.0000 -1.1875 -1.7500 -1.8750 -1.7500 -1.1875 0.0000

Iteration 5: 0.0000 -1.3750 -2.0313 -2.2500 -2.0313 -1.3750 0.0000

Iteration 6: 0.0000 -1.5156 -2.3125 -2.5313 -2.3125 -1.5156 0.0000

Iteration 7: 0.0000 -1.6563 -2.5234 -2.8125 -2.5234 -1.6563 0.0000

Iteration 8: 0.0000 -1.7617 -2.7344 -3.0234 -2.7344 -1.7617 0.0000

Iteration 9: 0.0000 -1.8672 -2.8926 -3.2344 -2.8926 -1.8672 0.0000

Iteration 10: 0.0000 -1.9463 -3.0508 -3.3926 -3.0508 -1.9463 0.0000

»

It eventually converges to 4 significant figures in 80 iterations:

Iteration 76: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000

Iteration 77: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000

Iteration 78: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000

Iteration 79: 0.0000 -2.5000 -4.0000 -4.4999 -4.0000 -2.5000 0.0000

Iteration 80: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000

Iteration 81: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000

You can substitute back into the equation Ax = b to check it is the solution!


To really go to town, we ran a 50 link chain in the figure below. As you can see it took about 4000 iterations to converge. Note similarity of n = 2000 and n = 4000 curves. We’re getting close!

10


Convergence of Matrix Iterations We need a test for convergence for matrix iteration analogous to the one used for equations in one variable earlier in this lecture. Since we have a structure Tx+c we can use a similar argument: For an iteration algorithm of the form cBxx +=

+ nn 1 , let the exact solution

be the vector s. Then cBss += .

Define the error vector sxe −=nn

.

Then ( ) ( ) ( )

nnnnnBecxBcBscBxsxe =−=+−+=−=

++ 11

Using the compatibility between matrix and vector norms,

nneBe ≤

+1

Thus the iteration cBxx +=

+ nn 1 converges if ∞→→ n

n as e 0 if there is

a matrix norm such that 1<B . (key result)


Looking at our hanging chain example,

=

0000

000

000

000

0000

21

21

21

21

21

21

21

21

B ,

note that the simplest norms 11

=B and 1=∞

B , and so the case for

convergence is not proved. But there is another norm, calculated by the MATLAB function norm(B,2),

which gives the 2-norm 8660.02

=B and thus guarantees convergence.

Note: 2

B is not the same as the Frobenius norm 2

1

2

Fro

= ∑∑

i j

ijAA in the

last lecture, but is more complex, and is described in the MATLAB manuals.

11


An alternative representation of the An alternative representation of the An alternative representation of the An alternative representation of the JacobiJacobiJacobiJacobi AlgorithmAlgorithmAlgorithmAlgorithm

As outlined above we have written the Jacobi algorithm in matrix form. This was convenient in order to understand the convergence properties of the method. Alternatively, we could write the technique in its matrix element form as follows:

for j=1,2,….N, and n indicates the iteration index.(If you have problems in seeing that this is so, compare with the Matlab example above.)This is a useful representation of the algorithm, both from the viewpoint of implementation, but also because it suggests an improvement to the method that we discuss next.

It also highlights the fact that we cannot have the condition What could we do in this case?

,1

)(

)1(

ii

iN

jii

n

jijn

ia

b

a

xax +∑

−=

=

+

0=ii

a


GaussGaussGaussGauss----Seidel AlgorithmSeidel AlgorithmSeidel AlgorithmSeidel Algorithm

the Jacobi algorithm ( ) ccccBxBxBxBxbbbbDDDDxxxxUUUULLLLDDDDxxxx +=++−= −−

+ nnn11

1 updates the whole of the newvector xxxxn+1 from the old vector xxxxn every iteration. But if we look at the MATLABalgorithm,

for j = 2:m+1 % miss out end points

xnew(j) = 0.5*(x(j-1) + x(j+1) - b(j));

end

we see that xnew(j) depends on x(j-1).

But xnew(j-1) was calculated last time round the loop, so is now available in shiny newupdated form! so let’s use it and modify the code to

. for j = 2:m+1 % miss out end points

xnew(j) = 0.5*(xnew(j-1) + x(j+1) - b(j));

end

12


» gschain(5,50)

Iteration 0: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

Iteration 1: 0.0000 -0.5000 -0.7500 -0.8750 -0.9375 -0.9688 0.0000

Iteration 2: 0.0000 -0.8750 -1.3750 -1.6563 -1.8125 -1.4063 0.0000

Iteration 3: 0.0000 -1.1875 -1.9219 -2.3672 -2.3867 -1.6934 0.0000

.................................................

Iteration 40: 0.0000 -2.5000 -3.9999 -4.4999 -4.0000 -2.5000 0.0000

Iteration 41: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000

This method, Known as the Gauss-Seidel method, converges more rapidly than the Jacobi

method.


The general formulation of the Gauss-Seidel method is

bDUxDLxDx11

1

1

1

−−

+

−

++−+−=

nnn

It looks strange to see the term

1

1

+

−

nLxD on the right hand side, but remember

that, since the matrix multiplications are carried out row by row, and L is lower triangular, each line of

1

1

+

−

nLxD contains only references to elements of

1+nx which are already known

13


Elemental FormElemental FormElemental FormElemental Form

The elemental version of the Gauss-Seidel method also clearly shows why the method is an improvement on the Jacobi method.This takes the form

where j=1,2,…..N, and n indicates the iteration index as before.

ii

i

j

i

N

ij

n

jij

n

jij

n

ia

bxaxa

x

∑ ∑−

= +=

+

+

+−−

=

1

1 1

)()1(

)1(

)()(


GaussGaussGaussGauss----Seidel Algorithm Seidel Algorithm Seidel Algorithm Seidel Algorithm –––– convergence propertiesconvergence propertiesconvergence propertiesconvergence properties

Let us start from the elemental form of the Gauss-Seidel iteration equation where j=1,2,…..N, and n indicates the iteration index as before.

Multiply by and collect the terms on the left hand side gives for i=1,2,…N

Hence there are N equations

ii

i

ji

N

ij

n

jij

n

jijn

ia

bxaxa

x∑ +∑−−

=

−

= +=

+

+

1

1 1

)()1(

)1(

)()(

iia 1+nx

ibnNxNia

Ni

xiianixiia

nxia

nxia +−

++−=+++++,.........

11,1...........1

221

11

NbnNxNNa

nsxNa

nxNa

bn

Nx

Nanxanxanxa

bnNxNa

nxa

nxa

nxa

+=+

++

++

+−−=+++

+−−−=+

1...........

12

111

22323

1

222

1

121

113132121

111

M

M

KK

KK

14


Hence

or

This is in the standard form Tx+cTx+cTx+cTx+c,,,, hence the convergence condition is

NbnNxNNa

nsxNa

nxNa

bn

Nx

Nanxanxanxa

bnNxNa

nxa

nxa

nxa

+=+

++

++

+−−=+++

+−−−=+

1...........

12

111

22323

1

222

1

121

113132121

111

M

M

KK

KK

bUxL)xDn1n +−=+ +(

bL)(DUxL)(Dx1n11n −−+ +++−=

1<−+ U1

L)(D


SummarySummarySummarySummary

• For an iterative scheme of the form Bx+cBx+cBx+cBx+c we have shown that convergence occurs when

1<B

• For the JacobiJacobiJacobiJacobi method this means

1<+−U)(LD

1

• For the GaussGaussGaussGauss----SeidelSeidelSeidelSeidel method this means

1<+ −UL)(D

1

•It can be shown (Q5 tute sheet) that if A is strictly diagonally dominantA is strictly diagonally dominantA is strictly diagonally dominantA is strictly diagonally dominant then the Jacobiand Gauss-Seidel methods converge.

15


More sophisticated Iterative Algorithms See (Johnson and Reiss) or (Faires and Burden) for clever modifications, using overrelaxation methods which can be tuned to give faster convergence. The chain example is, of course, fairly simple, and so could have been easily solved by Gaussian elimination. However, many problems can get a lot more complex, and iterative methods are often faster.

engineering computation lecture 3 iterative solution of simultaneous equations...

Documents