chapter 11 gaussian elimination (i) speaker: lung-sheng chien reference book: david kincaid,...
TRANSCRIPT
Chapter 11 Gaussian Elimination (I)
Speaker: Lung-Sheng Chien
Reference book: David Kincaid, Numerical Analysis
OutLine
• Basic operation of matrix- representation- three elementary matrices
• Example of Gaussian Elimination (GE)
• Formal description of GE
• MATLAB usage
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
A
Matrix notation in MATLAB
1,: 1,1: 4 first row of 6 2 2 4A A A
2
8:, 2 1: 4,2 second column of
13
4
A A A
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
x1
x2
x3
x4
6 x1 + (-2) x2 + 2 x3 + 4 x4
12 x1 + (-8) x2 + 6 x3 + 10 x4
3 x1 + (-13) x2 + 9 x3 + 3 x4
(-6) x1 + 4 x2 + 1 x3 + (-18) x4
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
x1
x2
x3
x4
6
12
3
-6
x1 -2
-8
-13
4
x2 2
6
9
1
x3
4
10
3
-18
x4
Matrix-vector product
Inner-product based
outer-product based
橫 直
直 橫
1: 4
1: 4
, *
for i
for j
b i A i j x j
end
end
Inner-product based
1: 4
1: 4
, *
for j
for i
b i A i j x j
end
end
outer-product based
Matrix-vector product: MATLAB implementation
compute b Ax matvec.m
Question: which one is better
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
6
12
3
-6
-2
-8
-13
4
2
6
9
1
4
10
3
-18
Logical index : 2D
physical index : 1D
Column-major nature in MATLAB
Question: how does column-major affect inner-product based matrix-vector product and outer-product based matrix-vector product?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
11 1 1 12 1 2 13 1 311 12 13 3
21 22 23 21 2 1 22 2 2 23 2 3, 1
31 32 33 31 3 1 32 3 2 33 3 3
T T T
T T T Tij i j
i j T T T
a e e a e e a e ea a a
a a a a e e a e e a e e a e e
a a a a e e a e e a e e
3 2
0 0 0 0
0 0 1 0 0 0 0
1 0 1 0
Te e
is outer-product representationwhere
11 12 13 1 3 3 3
21 22 23 2, 1 1 1
31 32 33 3
1,:
2,:
3,:
inner product basedTij i j i ij j
i j i j
a a a x A x
a a a x a e e x e a x A x
a a a x A x
Matrix representation: outer-product
11 12 13 11 12 13
21 22 23 31 32 33
31 32 33 21 22 23
1 0 0
0 0 1
0 1 0
a a a a a a
a a a a a a
a a a a a a
Elementary matrix [1]
(1) The interchange of two rows in A: ,: ,:A s A t
1
3
2
1 0 0
1,3,2 0 0 1
0 1 0
T
T
T
e
P e
e
Define permutation matrix
1 1 1
3 3 3
2 2 2
T T
inner product basedT T
T T
e e x x
Px e x e x x
e e x x
1
1 0 0
0 0 1
0 1 0
TP P
Question 1: why Question 2: how to easily obtain 1P
How to explain?
1
1 3
2
1 0 0
1,3,2 0 0 1
0 1 0
T
T
T
e
P e
e
3
2 2
1
0 0 1
3,2,1 0 1 0
1 0 0
T
T
T
e
P e
e
2 1 3,2,1 1,3,2 ?P P Question:
such that
1
1 3
2
x
Px x
x
such that
3
2 2
1
x
P x x
x
1 2
2 1 2 3 3
2 1
x x
P Px P x x
x x
1 1
2 3
3 2
x x
x x
x x
1 3
2 2
3 1
x x
x x
x x
implies 2 1 2,3,1P P
2 1
0 0 1 1 0 0 0 1 0
0 1 0 0 0 1 0 0 1
1 0 0 0 1 0 1 0 0
P P
Direct calculation
Concatenation of permutation matrices
11 12 13 11 12 13
21 22 23 21 22 23
31 32 33 31 32 33
1 0 0
0 0
0 0 1
a a a a a a
a a a a a a
a a a a a a
(2) Multiplying one row by a nonzero constant: ,: ,:A s A s
Elementary matrix [2]
Define scaling matrix 2
1 0 0
0 0
0 0 1
S
1
2 2
3
inner product based
x
S x x
x
How to explain?
1 12 2
1 0 0
10 0
0 0 1
S S
2 2
11 1
1/2 2 2
3 33
1S S
xx x
x x x
x xx
since
11 12 13 11 12 13
21 22 23 21 22 23
31 32 33 21 31 22 32 23 33
1 0 0
0 1 0
0 1
a a a a a a
a a a a a a
a a a a a a a a a
(3) Adding to one row a multiple of another: ,: ,: ,:A s A t A s
Elementary matrix [3]
Define GE (Gaussian Elimination) matrix 32
1 0 0
0 1 0
0 1
L
1 1
32 2 2
3 2 3
1 0 0
0 1 0
0 1
inner product based
x x
L x x x
x x x
How to explain?
32 3 2
1 0 0 0 0 0
0 1 0 0 0 0
0 0 1 0 0
TL I e e
outer-product representation
1
32 3 2 3 2 2 3 2
3 2
0
0T T
x
L x I e e x x e e x x x e x
x x
32 2 2 3: multiply by and add to L x x x x
32 : multiply 2,: by and add 2,: to 3,:L A A A A
: multiply by and add to ij j j iL x x x x
: multiply ,: by and add ,: to ,:ijL A A j A j A i
for i j
Use MATLAB notation
32
1,:
2,:
2,: 3,:
A
L A A
A A
1,: 1,:
2,: 2,:
3,: 2,: 3,:
A A
A A
A A A
21
1,:
1,: 2,:
3,:
A
L A A A
A
31
1,:
2,:
1,: 3,:
A
L A A
A A
31 21 31
1,: 1,:
1,: 2,: 1,: 2,:
3,: 1,: 3,:
A A
L L A L A A A A
A A A
21
1
1
1
L
such that
31
1
1
1
L
such that
31 21 21 31
1
1
1
L L L L
Concatenation of GE matrices
Suppose
31 21 ?L L Question:
OutLine
• Basic operation of matrix• Example of Gaussian Elimination (GE)
- forward elimination to upper triangle form- backward substitution
• Formal description of GE
• MATLAB usage
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
x1
x2
x3
x4
12
34
27
-38
12
6
6 -2 2 4
0 -4 2 2
3 -13 9 3
-6 4 1 -18
x1
x2
x3
x4
12
10
27
-38
12 -8 6 10
6 -2 2 412
6
)0 -4 2 2
3
6
3 -13 9 3
6 -2 2 43
6
)0 -12 8 1
12
34
10
12
21
27 6 -2 2 4
0 -4 2 2
0 -12 8 1
-6 4 1 -18
x1
x2
x3
x4
12
10
21
-38
6
6
6 -2 2 46
6
)0 2 3 -14
12
-26
-38-6 4 1 -18 6 -2 2 4
0 -4 2 2
0 -12 8 1
0 2 3 -14
x1
x2
x3
x4
12
10
21
-26
12
4
-12 8 1 21
-4 2 2 10
6 -2 2 4
0 -4 2 2
0 -12 8 1
0 2 3 -14
x1
x2
x3
x4
12
10
21
-26
12
4
First row does not change thereafter
12
4
)0 2 -5 -9
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 2 3 -14
x1
x2
x3
x4
12
10
-9
-26
2
4
2 3 -14 -26
-4 2 2 102
4
)0 4 -13 -21
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 4 -13
x1
x2
x3
x4
12
10
-9
-21
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 4 -13
x1
x2
x3
x4
12
10
-9
-21
4
2
4 -13 -21
2 -5 -94
2
)0 -3 -3
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 0 -3
x1
x2
x3
x4
12
10
-9
-3
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 0 -3
x1
x2
x3
x4
12
10
-9
-3
4 43 3 1x x
6 -2 2 4
0 -4 2 2
0 0 2 -5
x1
x2
x3
x4
12
10
-9
3 4 32 5 9 2x x x
6 -2 2 4
0 -4 2 2
x1
x2
x3
x4
12
10
2 3 4 24 2 2 10 3x x x x
6 -2 2 4 x1
x2
x3
x4
12
1 2 3 4 16 2 2 4 12 1x x x x x
Backward substitution: inner-product-based
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 0 -3
x1
x2
x3
x4
12
10
-9
-3
4 43 3 1x x
6 -2 2 4
0 -4 2 2
0 0 2 -5
x1
x2
x3
12
10
-9
x4 8
8
-4
3 32 4 2x x
6 -2 2
0 -4 2
x1
x2x3 8
8 12
122 24 12 3x x
6 -2x1 x2 12 6 1 16 6 1x x
Backward substitution: outer-product-based
OutLine
• Basic operation of matrix
• Example of Gaussian Elimination (GE)• Formal description of GE
- component-wise and column-wise representation- recursive structure
• MATLAB usage
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
x1
x2
x3
x4
12
34
27
-38
12
6
6 -2 2 4
0 -4 2 2
3 -13 9 3
-6 4 1 -18
x1
x2
x3
x4
12
10
27
-38
21 21
12 12
6 6L Ax L b
3
6
6 -2 2 4
12 -8 6 10
0 -12 8 1
-6 4 1 -18
x1
x2
x3
x4
12
34
21
-38
31 31
3 3
6 6L Ax L b
6
6
6 -2 2 4
12 -8 6 10
3 -13 9 3
0 2 3 -14
x1
x2
x3
x4
12
34
27
-26
41 41
6 6
6 6L Ax L b
41 31 21 41 31 211 0.5 2 1 0.5 2L L L Ax L L L b
6 -2 2 4
12 -8 6 10
3 -13 9 3
-6 4 1 -18
x1
x2
x3
x4
12
34
27
-38
6 -2 2 4
0 -4 2 2
0 -12 8 1
0 2 3 -14
x1
x2
x3
x4
12
10
21
-26
41 31 21
1
2 11 0.5 2
0.5 1
1 1
L L L
Exercise: check
1 6 2 2 4 6 2 2 4
2 1 12 8 6 10 0 4 2 2
0.5 1 3 13 9 3 0 12 8 1
1 1 6 4 1 18 0 2 3 14
by MATLAB
Eliminate first column: component-wise
where
1 11
6 2 2 4
12 8 6 10
3 13 9 3
6 4 1 18
Ta BA A
C A
1
11 3 3
1
1 02 1
0.5 1
1 1
LC a I
11
1 11
3 311 11
6 2 2 41 0
0 4 2 2
0 12 8 10
0 2 3 14
TT
T
a Ba B
L A C CBI AC Aa a
Exercise: check 2
11
4 2 2
12 8 1
2 3 14
TCBA A
a
by MATLAB
Eliminate first column: column-wise
define and
then
1
1 1 1 1 1 1 1 111 12 13 14 11 12 13 14
1 1 1 1 2 2 2 11 121 22 23 24 22 23 24 11
1 1 1 1 2 2 2 231 32 33 34 32 33 34
1 1 1 1 2 2 241 42 43 44 42 43 44
0
0 0
0
L
a a a a a a a a
a a a a a a a aA L A
a a a a a a a A
a a a a a a a
Notation:
6 -2 2 4
0 -4 2 2
0 -12 8 1
0 2 3 -14
x1
x2
x3
x4
12
10
21
-26
12
4
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 2 3 -14
x1
x2
x3
x4
12
10
-9
-26
2
4
1 132
12
4L L A L b
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 4 -13
x1
x2
x3
x4
12
10
-9
-21
1 142 32
2 12
4 4L L L A L b
Eliminate second column: component-wise
242 32
1
10.5 3
3 1
0.5 1
L L L
define
2 1
1 6 2 2 4 6 2 2 4
1 0 4 2 2 0 4 2 2
3 1 0 12 8 1 0 0 2 5
0.5 1 0 2 3 14 0 0 4 13
L L A
Exercise: check
Exercise: check 2 1 1 2
1
2 1
0.5 3 1
1 0.5 1
L L L L
, why? Give a simple explanation.
2
1 1 1 1 1 1 1 1111 12 13 14 11 12 13 14
112 2 2 2 2 21 2 1 222 23 24 22 23 24
222 2 2 3 3332 33 34 33 34
2 2 2 3 342 43 44 43 44
0 00
0 0 00 0
0 0 0
L
a a a a a a a aa
a a a a a aL A L L A a
a a a a aA
a a a a a
Notation:
Eliminate second column: column-wise
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 4 -13
x1
x2
x3
x4
12
10
-9
-21
4
2
6 -2 2 4
0 -4 2 2
0 0 2 -5
0 0 0 -3
x1
x2
x3
x4
12
10
-9
-3
2 1 2 143
4
2L L L A L L b
Eliminate third column: component-wise
343
1
12
1
2 1
L L
define
then 3 2 1 2 3 1 2 1 3
1
2 1
0.5 3 1
1 0.5 2 1
L L L L L L L L L
3
1 1 1 1 1 1 1 1 111 12 13 14 11 12 13 14 11
2 2 2 2 2 2 22 1 3 2 122 23 24 22 23 24 22
3 3 3 3 333 34 33 34 33
3 3 4 443 44 44
0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0 0
L
a a a a a a a a a
a a a a a a aL L A L L L A
a a a a a
a a a A
Notation:
3 2 1
6 2 2 4
4 2 2
2 5
3
L L L A U
1 1 11 2 3
1
2 1
0.5 3 1
1 0.5 2 1
L L L L
13 2 1A L L L U LU
Exercise: check
LU-decomposition
Question: Do you have any effective method to write down matrix L
LU-decomposition
Question: why don’t we care about right hand side vector b
A LU
6 2 2 4 1 6 2 2 4
12 8 6 10 2 1 4 2 2
3 13 9 3 0.5 3 1 2 5
6 4 1 18 1 0.5 2 1 3
Question 3: How to measure “goodness of LU-decomposition“?
Question 2: does any invertible matrix has LU-decomposition?
1 0 0
0 0 1
0 1 0
A
?A LU
Question 4: what is order of performance of LU-decomposition (operation count)?
Question 5: How to parallelize LU-decomposition?
Question 6: How to implement LU-decomposition with help of GPU?
Problems about LU-decomposition
Question 1: what condition does LU-decomposition fail?
OutLine
• Basic operation of matrix
• Example of Gaussian Elimination (GE)
• Formal description of GE• MATLAB usage
- website resource- “help” command- M-file
http://www.mathworks.com/access/helpdesk/help/techdoc/matlab.html
MATLAB website
MATLAB: create matrix
MATLAB: LU-factorization
Start MATLAB 2008
Command “help” : documentation
!= in C
Loop in MATLAB
Decision-making in MATLAB
IO in MATLAB
LU-factorization in MATLAB
A LU
PA LU
PAQ LU
:P permutation
, :P Q permutation
1: 4
1: 4
, *
for i
for j
b i A i j x j
end
end
Inner-product based
compute b Ax
matvec.m
M-file in MATLAB
Description of function matvec, write specificaiton of input parameter
Consistency check
Move to working directory
working directory
M-file
Execute M-file
Construct matrix
Execute M-file, like function call in C-language
11 12 13 14 11 11 12 13 14
21 22 23 24 21 22 22 23 24
31 32 33 34 31 32 33 33 34
41 42 43 44 41 42 43 44 44
a a a a l u u u u
a a a a l l u u uA LU
a a a a l l l u u
a a a a l l l l u
Ax b LUx b1
1
y L b
x U y
Forward substitution 1y L b 1 2 3 4y y y y since
backward substitution 1x U y 4 3 2 1x x x x since
Write MATLAB code to do forward substitution and backward substitution
Exercise: forward / backward substitution
3 2 1
6 2 2 4
4 2 2
2 5
3
L L L A U
13 2 1A L L L U LU
LU-decomposition
3
1
1
1
2 1
L
1
1
2 1
0.5 1
1 1
L
2
1
1
3 1
0.5 1
L
6 2 2 4
12 8 6 10
3 13 9 3
6 4 1 18
A
Exercise: LU-decomposition
You should find recursive structure of the decomposition