-
1
Engineering Computation ECL3-1
ENGINEERING COMPUTATION Lecture 3 Stephen Roberts Michaelmas Term
Iterative Solution of Simultaneous Equations
Topics covered in this section:
1. Iterative algorithms and the need to test their convergence. 2. The Jacobi method of solution to solve Ax=b 3. The Gauss-Seidel algorithm. 4. Awareness of other numerical approached to solving Ax=b
Engineering Computation ECL3-2
Introduction So far we have discussed the solution of the simultaneous linear equation set bAx = , and the conditions for ill-conditioning. The underlying assumption was that we would go ahead and solve the set of equations by a direct method, such as Gaussian elimination. How many calculations would this take?
Consider N equations in N variables, where A is an N×N matrix. Assume additions are quick, and multiplications cost time.
-
2
Engineering Computation ECL3-3
Example: Consider using the echelon method to solve
=
4
3
1
643
432
321
z
y
x
Process the augmented matrix as follows:
−
−−
→
−−
−−
→
1
1
1
100
210
321
1
1
1
320
210
321
4
3
1
643
432
321
2row2 - row33row1 - row3
2row1 - row2
Then back-substitute: 21321121 =⇒=++=⇒=−= xzyxyz-y-z . . .
The reduction to echelon form took 4×2 + 3×1 = 11 multiplications. The back substitution took a further 1 + 2 + 3 multiplications or divisions, making a total of 17.
Engineering Computation ECL3-4
Introduction General case:
Consider N equations in N variables, where A is an N×N matrix. Assume additions are quick, and multiplications cost time.
Reduction to echelon form will take ~ N(N-1) + (N-1)(N-2) + ..... ~ N
3 multiplications. For large systems (say 10
3 equations) this can be
a very large number (~ 109), and solutions may take a long time, even
with modern fast computers. Can we use a better approach?
-
3
Engineering Computation ECL3-5
Iterative Methods and Convergence
Engineering Computation ECL3-6
Iterative Methods are useful where successive approximations generate solutions which converge on the exact solutions. They are a viable alternative to direct methods and can provide solutions of sufficient accuracy with far fewer calculations. Iterative Methods are efficient when:
• An approximate solution close to the exact solution is known. Only a few iterations are required to get to an acceptable solution.
• The matrix A is of high dimension.
• The matrix A is sparse, with terms mainly near the diagonal. Many engineering problems satisfy this criterion, as the physical interactions between elements may only be local (eg circuit analysis, boundary value probs., PDEs)
• The matrix A is diagonally dominated (the largest elements are along the diagonal), and the iterative method converges quickly.
-
4
Engineering Computation ECL3-7
Revision: Iterative solutions of equations in one variable
We attempt to solve the equation ( ) 0=xf . e.g. ( ) 0cos =−= xxxf .
Rearrange the equation into the form ( )xgx = e.g. xx cos= . Choose a starting value x0, and repeat the iterative calculation
( ) LL,2,1,0 ,1 ==+ nxgx nn until xn converges on a solution.
Engineering Computation ECL3-8
In our example, using nn
xx cos1 =+ , we get the following results, starting with x0
= 0.7000: n xn Error en 0 0.7000 -0.0391 1 0.7648 0.0258 2 0.7215 -0.0176 3 0.7508 0.0117 ... .......... 10 0.7383 -0.007
This converges (somewhat slowly! Try it on your calculator) to the solution 0.7391 accurate to 4 figures. But there is no guarantee that the method will converge! Consider an alternative formulation nn xx
1
1 cos−
+ = .
n xn 0 0.7000 1 0.7954 2 0.6511 3 0.8617 This is diverging!
=> need a method to show will converge
-
5
Engineering Computation ECL3-9
Test for Convergence
Let the true solution of ( )xgx = be sx = , so that ( )sgs = . The error en+1 after n + 1 iterations is given by
( ) ( )sgxgsxe nnn −=−= ++ 11
Expanding the first two terms of a Taylor series about sxn = ,
( ) ( ) ( ) ( ) ( ) ( ) LL +′+=+′−+= sgesgsgsxsgxg nnn
Thus ( )sgee nn ′≈+1 . (1)
The iteration scheme will converge if ∞→→ nen as 0 .
Engineering Computation ECL3-10
Thus ( )sgeenn
′≈+1
.
The iteration scheme will converge if ∞→→ ne
n as 0 .
From ((1), previous slide) The condition for convergence is ( ) 1
-
6
Engineering Computation ECL3-11
Iterative Methods for solving Ax=b
Engineering Computation ECL3-12
Iterative Methods for solving Ax=b Let’s now consider how a related idea can be applied to develop an iterative technique for solving Ax=b. As with the case of the one dimensional case, let’s translate Ax=b into an equivalent system of the form x=Tx+c. Then having selected an initial guess x0 we can apply a similar method to that outlined above.
-
7
Engineering Computation ECL3-13
Jacobi Algorithm for solving bAx = . Divide A into a diagonal matrix D and two triangular matrices L and U .
For a 3×3 matrix,
ULDA ++=
+
+
=
=
000
00
0
0
00
000
00
00
00
23
1312
3231
21
33
22
11
333231
232221
131211
A
AA
AA
A
A
A
A
AAA
AAA
AAA
Rewrite bAx = as ( ) bxULD =++ and re-arrange to give
( )xULbDx +−= . Now, the inverse D-1 of D is simply the diagonal matrix of the inverse of the diagonal elements of A, so write
( ) cBxbDxULDx +=++−= −− 11 This is in the required form Tx+c and suggests the Jacobi iterative scheme:
( ) cBxbDxULDx +=++−= −−+ nnn
11
1
Engineering Computation ECL3-14
Example: Jacobi solution of weighted chain.
Consider a hanging chain of m + 1 light links with fixed ends at height x0 = xm+1 = 0. It could be the supporting chain for the Clifton
suspension bridge in Bristol, shown ⇒. There is unit tension in the chain, the links are of unit length. A downward force bj is applied on the jth hinge. If we make the approximation that the links make small angles to the horizontal, resolving forces vertically at each hinge gives:
mjbxxx jjjj ≤≤=+− +− 1 , 2 11
-
8
Engineering Computation ECL3-15
This is a matrix equation of the form bAx = . In the case m = 5, there are 5 links
−
−
−
−
−
=
21000
12100
01210
00121
00012
A
Dividing the matrix A into D + L+ U , and taking the case where
[ ]T1,1,1,11,=b gives the Jacobi iteration algorithm in the form ( ) cBxbDxULDx +=++−= −−+ nnn
11
1 :
−
=+
21
21
21
21
21
21
21
21
21
21
21
21
21
1
0000
000
000
000
0000
nn xx
Engineering Computation ECL3-16
this is easy to code up in MATLAB.
% Jchain.m
% solves weighted chain with m+1 links with n iterations
function jchain(m,n)
b = ones(m+2,1); % array of 1's - downward forces
x = zeros(m+2,1); % array of 0's - initial guess of position
xnew = x;
fprintf('Iteration 0: ');
fprintf('%.4f ',x);
fprintf('\n');
for i = 1:n % n iterations
for j = 2:m+1 % miss out end points
xnew(j) = 0.5*(x(j-1) + x(j+1) - b(j));
end
x = xnew;
fprintf('Iteration %2d: ', i);
fprintf('%.4f ',x);
fprintf('\n');
end
-
9
Engineering Computation ECL3-17
Running this, with m = 5 gives, for the first 10 iterations:
» jchain(5,10)
Iteration 0: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
Iteration 1: 0.0000 -0.5000 -0.5000 -0.5000 -0.5000 -0.5000 0.0000
Iteration 2: 0.0000 -0.7500 -1.0000 -1.0000 -1.0000 -0.7500 0.0000
Iteration 3: 0.0000 -1.0000 -1.3750 -1.5000 -1.3750 -1.0000 0.0000
Iteration 4: 0.0000 -1.1875 -1.7500 -1.8750 -1.7500 -1.1875 0.0000
Iteration 5: 0.0000 -1.3750 -2.0313 -2.2500 -2.0313 -1.3750 0.0000
Iteration 6: 0.0000 -1.5156 -2.3125 -2.5313 -2.3125 -1.5156 0.0000
Iteration 7: 0.0000 -1.6563 -2.5234 -2.8125 -2.5234 -1.6563 0.0000
Iteration 8: 0.0000 -1.7617 -2.7344 -3.0234 -2.7344 -1.7617 0.0000
Iteration 9: 0.0000 -1.8672 -2.8926 -3.2344 -2.8926 -1.8672 0.0000
Iteration 10: 0.0000 -1.9463 -3.0508 -3.3926 -3.0508 -1.9463 0.0000
»
It eventually converges to 4 significant figures in 80 iterations:
Iteration 76: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000
Iteration 77: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000
Iteration 78: 0.0000 -2.5000 -3.9999 -4.4999 -3.9999 -2.5000 0.0000
Iteration 79: 0.0000 -2.5000 -4.0000 -4.4999 -4.0000 -2.5000 0.0000
Iteration 80: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000
Iteration 81: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000
You can substitute back into the equation Ax = b to check it is the solution!
Engineering Computation ECL3-18
To really go to town, we ran a 50 link chain in the figure below. As you can see it took about 4000 iterations to converge. Note similarity of n = 2000 and n = 4000 curves. We’re getting close!
-
10
Engineering Computation ECL3-19
Convergence of Matrix Iterations We need a test for convergence for matrix iteration analogous to the one used for equations in one variable earlier in this lecture. Since we have a structure Tx+c we can use a similar argument: For an iteration algorithm of the form cBxx +=
+ nn 1 , let the exact solution
be the vector s. Then cBss += .
Define the error vector sxe −=nn
.
Then ( ) ( ) ( )
nnnnnBecxBcBscBxsxe =−=+−+=−=
++ 11
Using the compatibility between matrix and vector norms,
nneBe ≤
+1
Thus the iteration cBxx +=
+ nn 1 converges if ∞→→ n
n as e 0 if there is
a matrix norm such that 1
-
11
Engineering Computation ECL3-21
An alternative representation of the An alternative representation of the An alternative representation of the An alternative representation of the JacobiJacobiJacobiJacobi AlgorithmAlgorithmAlgorithmAlgorithm
As outlined above we have written the Jacobi algorithm in matrix form. This was convenient in order to understand the convergence properties of the method. Alternatively, we could write the technique in its matrix element form as follows:
for j=1,2,….N, and n indicates the iteration index.(If you have problems in seeing that this is so, compare with the Matlab example above.)This is a useful representation of the algorithm, both from the viewpoint of implementation, but also because it suggests an improvement to the method that we discuss next.
It also highlights the fact that we cannot have the condition What could we do in this case?
,1
)(
)1(
ii
iN
jii
n
jijn
ia
b
a
xax +∑
−=
=
+
0=ii
a
Engineering Computation ECL3-22
GaussGaussGaussGauss----Seidel AlgorithmSeidel AlgorithmSeidel AlgorithmSeidel Algorithm
the Jacobi algorithm ( ) ccccBxBxBxBxbbbbDDDDxxxxUUUULLLLDDDDxxxx +=++−= −−+ nnn 111 updates the whole of the newvector xxxxn+1 from the old vector xxxxn every iteration. But if we look at the MATLABalgorithm,
for j = 2:m+1 % miss out end points
xnew(j) = 0.5*(x(j-1) + x(j+1) - b(j));
end
we see that xnew(j) depends on x(j-1).
But xnew(j-1) was calculated last time round the loop, so is now available in shiny newupdated form! so let’s use it and modify the code to
. for j = 2:m+1 % miss out end pointsxnew(j) = 0.5*(xnew(j-1) + x(j+1) - b(j));
end
-
12
Engineering Computation ECL3-23
» gschain(5,50)
Iteration 0: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
Iteration 1: 0.0000 -0.5000 -0.7500 -0.8750 -0.9375 -0.9688 0.0000
Iteration 2: 0.0000 -0.8750 -1.3750 -1.6563 -1.8125 -1.4063 0.0000
Iteration 3: 0.0000 -1.1875 -1.9219 -2.3672 -2.3867 -1.6934 0.0000
.................................................
Iteration 40: 0.0000 -2.5000 -3.9999 -4.4999 -4.0000 -2.5000 0.0000
Iteration 41: 0.0000 -2.5000 -4.0000 -4.5000 -4.0000 -2.5000 0.0000
This method, Known as the Gauss-Seidel method, converges more rapidly than the Jacobi
method.
Engineering Computation ECL3-24
The general formulation of the Gauss-Seidel method is
bDUxDLxDx11
1
1
1
−−
+
−
++−+−=
nnn
It looks strange to see the term
1
1
+
−
nLxD on the right hand side, but remember
that, since the matrix multiplications are carried out row by row, and L is lower triangular, each line of
1
1
+
−
nLxD contains only references to elements of
1+nx which are already known
-
13
Engineering Computation ECL3-25
Elemental FormElemental FormElemental FormElemental Form
The elemental version of the Gauss-Seidel method also clearly shows why the method is an improvement on the Jacobi method.This takes the form
where j=1,2,…..N, and n indicates the iteration index as before.
ii
i
j
i
N
ij
n
jij
n
jij
n
ia
bxaxa
x
∑ ∑−
= +=
+
+
+−−
=
1
1 1
)()1(
)1(
)()(
Engineering Computation ECL3-26
GaussGaussGaussGauss----Seidel Algorithm Seidel Algorithm Seidel Algorithm Seidel Algorithm –––– convergence propertiesconvergence propertiesconvergence propertiesconvergence properties
Let us start from the elemental form of the Gauss-Seidel iteration equation where j=1,2,…..N, and n indicates the iteration index as before.
Multiply by and collect the terms on the left hand side gives for i=1,2,…N
Hence there are N equations
ii
i
ji
N
ij
n
jij
n
jijn
ia
bxaxa
x∑ +∑−−
=
−
= +=
+
+
1
1 1
)()1(
)1(
)()(
iia1+nx
ibnNxNia
Ni
xiianixiia
nxia
nxia +−++
−=+++++ ,.........11,1...........1
221
11
NbnNxNNa
nsxNa
nxNa
bn
Nx
Nanxanxanxa
bnNxNa
nxa
nxa
nxa
+=+
++
++
+−−=+++
+−−−=+
1...........
12
111
22323
1
222
1
121
113132121
111
M
M
KK
KK
-
14
Engineering Computation ECL3-27
Hence
or
This is in the standard form Tx+cTx+cTx+cTx+c,,,, hence the convergence condition is
NbnNxNNa
nsxNa
nxNa
bn
Nx
Nanxanxanxa
bnNxNa
nxa
nxa
nxa
+=+
++
++
+−−=+++
+−−−=+
1...........
12
111
22323
1
222
1
121
113132121
111
M
M
KK
KK
bUxL)xDn1n +−=+ +(
bL)(DUxL)(Dx1n11n −−+ +++−=
1
-
15
Engineering Computation ECL3-29
More sophisticated Iterative Algorithms See (Johnson and Reiss) or (Faires and Burden) for clever modifications, using overrelaxation methods which can be tuned to give faster convergence. The chain example is, of course, fairly simple, and so could have been easily solved by Gaussian elimination. However, many problems can get a lot more complex, and iterative methods are often faster.