1Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Systems of Linear Equations
x bA
Daniel Baur
ETH Zurich, Institut für Chemie- und Bioingenieurwissenschaften
ETH Hönggerberg / HCI F128 – Zürich
E-Mail: [email protected]
http://www.morbidelli-group.ethz.ch/education/index
2Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Problem Definition
We want to solve the problem
where x is the vector of unknowns, while A and b are given
Assumptions1. The number of equations is equal to the number of unknowns,
therefore A is a square matrix2. All components of A, b and x are real3. A solution exists and it is unique
( , ) ( ,1) ( ,1)n n x n b n A
• A-1 exists
• A is not singular
• A’s columns are linearly independent
• A’s rows are linearly independent
• det(A) is non-zero
• rank(A) is equal to n
• Ax = 0 only if x is a null vector
3Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Analytical Approach
Cramer’s rule (1750):The solution is given by
where Ai is defined as follows
det( )
det( )i
ix A
A
1,1 1,2 1, 1 1 1, 1 1,
2,1 2,2 2, 1 2 2, 1 2,
,1 ,2 , 1 , 1 ,
i i n
i i ni
n n n i n n i n n
a a a b a a
a a a b a a
a a a b a a
A
b replaces the ith column
4Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Calculation of the Determinant
The Laplace formula (1772) allows the computation of the determinant of a square matrix:
where Ci,j is the determinant of the sub-matrix obtained by removing the ith row and the jth column of the matrix, multiplied by (-1)i+j:
1, 1,1
det( )N
i ii
a C
A
, ,( 1) det( )i ji j i jC M 1,1 1, 1 1, 1 1,
1,1 1, 1 1, 1 1,
,1,1 1, 1 1, 1 1,
,1 , 1 , 1 ,
j j n
i i j i j i n
i ji i j i j i n
n n j n j n n
a a a a
a a a a
a a a a
a a a a
M
no jth column
no ith row
5Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
A First Numerical Approach: Gauss Elimination Method
The following operations do not change the result:1. Multiply a line by a constant2. Substitute a line with a linear combination of multiple lines3. Permute the order of lines
This can be used to produce a triangular matrix, which allows the solution to be found easily by substitution
Example system
1
2
3
3 2 1 1
1 2 2 5
1 1 1 2
x
x
x
6Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Gauss Elimination Example
• Multiply by -3
• Sum it to 1st line• Multiply by -3
• Sum it to 1st line
• Multiply by -4
• Sum it to 2nd line
Triangular System
7Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Generalization of the Gauss Elimination Method
We want to find a general procedure to replace one entry in the matrix with a zero; for this we define a multiplier l21
Note that
11 12 13
21 22 23
31 32 33
a a a
a a a
a a a
A
( ) ( )21 21 21 11( ) ( )21
21 22 22 21 1211 ( ) ( )
23 23 21 13
0new old
new old
new old
a a l aa
l a a l aa
a a l a
11 12 13 11 12 13( ) ( )
21 21 22 23 22 23
31 32 33 31 32 33
1 0 0
1 0 0
0 0 1
new new
a a a a a a
l a a a a a
a a a a a a
8Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
The Gauss Elimination in Matrix Notation
If we put all multipliers for one column in one matrix, we get
where
This way, a triangular matrix is easily obtained
(0) (0) (0) (0) (0) (0)11 12 13 11 12 13
(0) (0) (0) (0) (0) (0) (1) (1) (1)21 21 22 23 22 23(0) (0) (0) (0) (1) (1)31 31 32 33 32 33
1 0 0
1 0 0
0 1 0
a a a a a a
l a a a a a
l a a a a a
M A A
( )( )
( )
kijk
ij kjj
al
a
( 1) ( 2) (1) (0) (0)( 1) ( 1)
( 1) ( 2) (1) (0) (0)
n nn n
n nx b
b b
A M M M AA
M M M
9Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Example: Gauss Elimination in Matrix Notation
n(n-1) operations (flops)(n-1)(n-2) operations (flops)
Total number of operations required31 1
2
2 2( 1)
3
n n
j j
nj j j
10Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Gauss Elimination / Transformation Method
The Gauss elimination method is relatively easy to implement (even by hand), but has some distinct disadvantages; namely it Changes the matrix A Requires and changes the coefficient vector b Must be rerun if the vector b changes
If we consider the Gauss method in matrix form on the other hand, we can see that we can use
to transform A and b; M is therefore called the Gauss transformation matrix
( 2) (1) (0)nM M M M
11Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
The Gauss Transformation Method
We have so far The final matrix A, which is an right triangular matrix The matrix M, which is a left triangular matrix
The inverse of M is also a left triangular matrix
1 (0)21(0) (1)31 31
1 0 0
1 0
1
l
l l
L M
12Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
LR (LU) Factorization
We define our (right triangular) solution matrix as follows
If we multiply with L = M-1 from the left on both sides
( 2) (0) (0)n A MA R MA
1 (0) (0) LR M MA A
13Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
LR Factorization (Continued)
The starting matrix A is transformed (factorized) as
If we apply this to a linear equation system, we get
This approach rids us of the disadvantages discussed earlier, because For every vector b, two simple triangular systems must be solved
without factorizing again The matrices L and R can be stored using the elements of A If A is modified, it is often possible to modify L and R accordingly
without re-factorizing Note that Gauss elimination is still needed once to compute L and R
A LR
y bx x b
x y
LA LR
R
14Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Pivoting It is easy to see from the definition of the multiplication
factors, that the diagonal elements at each step (the pivot values) cannot be equal to zero
This is circumvented by reordering the rows of the matrix A by multiplication with a permutation matrix P
This approach is referred to as LRP-factorization(or LR-factorization with partial pivoting)
Example:
a11 = 0 switch the lines x1 = x2 = 1
1 PA PM R LR
2
1 2
1
0.4 0.3 0.1
x
x x
15Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Pivoting (Continued)
Pivoting must also take into account scaling problems;Let us consider a similar example
Pivot elements should have large absolute values
201 2
1 2
2 10 1
0.4 0.3 0.1
x x
x x
16Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
How does Matlab do it?
When using left divide \, Matlab chooses a procedure depending on the properties of the problem, i.e. if A is1. Diagonal, x is computed directly by division2. Sparse, square and banded, then banded solvers are
used; Either Gauss elimination without pivoting or LR factorization3. Left or right triangular, then backsubstitution is used4. A permutation of a triangular matrix, it is permuted and 3.
applies5. Symmetric or Hermitian, then a Cholesky factorization is
attempted (A = RR*, where R* is the conjugate transpose of R);
If it fails, another indefinite symmetric factorization is attempted
6. Square but 1 through 5 do not apply, then LR factorization with
partial pivoting is applied7. Not square, then Householder reflections are used to
compute afactorization which leads to a least-squares solution, i.e. a
vector xwhich minimizes the length of Ax – b
17Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Assignment 11. Find online the template for an LR-factorization of a matrix
with partial pivoting. Make the function operational by adding the lines that calculate the M matrix of the current step and the new A matrix. Why does the transformation matrix T appear in the formula for M?
Explain by comparing to the definition of M in the non-pivoting case.
2. Use the function to factorize the following matrix
Test if the factorization worked, i.e. if LR = A. Is L in the form you would expect it to be? What implications does
this have for its application in solving a linear system (see slide 13) and how could you correct for it?
1 2 3
9 8 1
5 0 1
A
18Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Hints
19Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Assignment 21. Create a random matrix A with dimensions 4x4 and a
random column vector b with size 4x1. Solve the system Ax = b.
2. Create a function that computes the determinant of square matrices using the Laplace formula. Use a recursive approach (see hints).
3. Use this function to compute the solution of the linear system above using Cramer’s rule.
4. Do the same for linear equation systems with sizes ranging from 5 to 9.
5. Read out the CPU time required to solve all these systems with both methods (Cramer’s method and A\b) and compare them.
20Daniel Baur / Numerical Methods for Chemical Engineers / Linear Equation Systems
Hints
If you remember the Laplace formula as a sum
you can see that the calculation of the determinant requires the calculation of a determinant. You could let the function call itself to do that (recursive function). Remember that the determinant of a 1x1 matrix is equal to its only element.
There are several Matlab commands to read out timings, however the most reliable one is t_start = tic;statements;t_elapsed = toc(t_start);
Where the time elapsed (in second) is stored in t_elapsed
11, 1, 1, 1,
1 1
det( ) ( 1) det( )N N
ii i i i
i i
a C a
A M