System of Linear Algebraic Equations

- - Basic Concepts Basic Concepts

- - Gaussian Gaussian Elimination Elimination
- - Backward Substitution Backward Substitution
- - LU LU Decomposition Decomposition
Numerical Methods for Numerical Methods for Civil Civil Engineers Engineers
Lecture 6
Mongkol JIRAVACHARADET
S U R A N A R E E INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING
Linear Algebraic Equations Linear Algebraic Equations
a
11
x
1
+ a
12
x
2
+ . . . a
1n
x
n
= b
1
a
21
x
1
+ a
22
x
2
+ . . . a
2n
x
n
= b
2
a
n1
x
1
+ a
n2
x
2
+ . . . a
nn
x
n
= b
n
.
.
.
.
.
.
nequations
nunknowns
A system of nlinear equations in nunknowns,
Example:
9 4 7 5
2 3 8 2
7 5 3
3 2 1
3 2 1
3 2 1
= +
= +
= +
c c c
c c c
c c c
Matrix Form:

9
2
7
4 7 5
3 8 2
5 3 1
3
2
1
c
c
c
A x = b Symbolic Notation
Augment Form:

9 4 7 5
2 3 8 2
7 5 3 1
System of linear algebraic equations
A x = b

m m mn m m
n
n
b
b
b
x
x
x
a a a
a a a
a a a
M M
L
M O M M
L
L
2
1
2
1
2 1
2 22 21
1 12 11
where a
ij
and b
i
are constants, i = 1,2,,m, j = 1,2,,n
m= number of rows
n = number of columns
Solution : Solution : A x = b
A
-1
A x = A
-1
b
x = A
-1
b
MATLAB : MATLAB :
>> x = inv(A)*b
- Use matrix inversion,
>> x = A\b
- Use backslash operator,
Requirements for a Solution,A
mn
m rows and n columns
m = n
Solving Ax = bfor n unknowns from n equations
m > n
Overdetermined systems
(e.g., least-squares problems)
m < n
Underdetermined systems
(e.g., optimization)
rank(A) = number of linearly independent columns in A
For matrix A with m rows and n columns, and Ax = b has a solution
- if rank(A) < n , infinite number of solutions
- if rank(A) = n , unique solution
For a system of n equations in n unknowns written in the form Ax = b,
the solution x exists and is unique for any b if and only if rank(A) = n.
Matrix Rank Matrix Rank
>> A = [3 -4 1;6 10 2;9 -7 3];
>> rank(A)

ans =

2
Consistency Consistency Test Test
11 12 1 1
21 22 2 2
1 2
1 2
n
n
n
m m mn m
a a a b
a a a b
Ax b x x x
a a a b



= + + + =




L
M M M M
Element of vector x are coefficients in the linear equation
Augmented matrix:
11 12 13 1 1
11 11 11 11 2
11 11 11 11 3
11 11 11 11
n
m
a a a a b
a a a a b
A a a a a b
a a a a b



=




L
L
%
L
M M M O M M
L
Consistent when and have the same rank A A
%
= [Ab]
Practical Model: y = x +
Example: An Inconsistent System from Data Fitting
Experiment data:
x
y
1
2
2
1
3
0.5
0 1 2 3 4
-2
-1
0
1
2
3
1 1 2
2 1 1
3 1 0.5

=





Matrix Form:
>> A=[1 1; 2 1; 3 1]; b=[2; 1; 0.5];
>> rank(A)
>> rank([A b])
Consistency test :
5 . 0 3
1 2
2
= +
= +
= +



Data Model:
2nd Experiment:

0
1
2
1 3
1 2
1 1

Consistency test:
>> A = [1 1; 2 1; 3 1]; b = [2; 1; 0];
>> rank(A)
>> rank([A b])
0 1 2 3 4
-2
-1
0
1
2
3
Exact solution

0
1
2
1
1
1
) 3 (
3
2
1
) 1 (
[ ] [ ]
T T
3 1 =
Nonsingular Case Nonsingular Case
A x = b
x = A
-1
b
If the coefficient matrix A is nonsingular, then it is invertible and we can
solve Ax = b as follows:
This solution is therefore unique. Also, if b = 0, it follows that the unique
solution to Ax = 0 is x = A
-1
0 = 0.
Thus if A is nonsingular, then the only solution to Ax = 0 is the trivial
solution x = 0.
Matrix
elimination
row operation
Upper triangular
backward
substitution
Solution
Naive Gauss Elimination Naive Gauss Elimination
The most basic systematic scheme for solving system of linear equations.
The procedure consisted of two steps:
1) Forward elimination of the linear equation matrix using row operation
to obtain an upper triangular matrix.
2) Backward substitution to solve for the unknowns
Forward elimination Forward elimination

3 33
2 23 22
1 13 12 11
3 33 32 31
2 23 22 21
1 13 12 11
b a
b a a
b a a a
b a a a
b a a a
b a a a
Backward substitution Backward substitution
11 2 12 3 13 1 1
22 3 23 2 2
33 3 3
/ ) (
/ ) (
/
a x a x a b x
a x a b x
a b x
=
=
=
Substitutions Substitutions
Backward substitution: Backward substitution:
Forward substitution: Forward substitution:
Upper triangular matrix
nn
n
n
a
b
x =
1 , 2 , , 2 , 1 ), (
1
1
L = =

+ =
n n i x a b
a
x
n
i j
j ij i
ii
i
Lower triangular matrix
11
1
1
a
b
x =
n i x a b
a
x
i
j
j ij i
ii
i
, , 3 , 2 , 1 ), (
1
1
1
L = =

=
Pivot row
Pivot element
EXAMPLE : Naive Gauss Elimination
6 4 4 3
7 7 6 6
1 2 3
3 2 1
3 2 1
3 2 1
= +
= +
= +
x x x
x x x
x x x

6 4 4 3
7 7 6 6
1 1 2 3
R1
R2
R3
Forward elimination (Row operation):
R2 + 2R1, R3+R1:

7 3 2 0
9 5 2 0
1 1 2 3
R3 - R2:

2 2 0 0
9 5 2 0
1 1 2 3
Backward substitution:
2 2
9 5 2
1 2 3
2 2 0 0
9 5 2 0
1 1 2 3
3
3 2
3 2 1
=
= +
= +

x
x x
x x x
2 2
9 5 2
1 2 3
3
3 2
3 2 1
=
= +
= +
x
x x
x x x
1
2
2
3
=

= x
2 ) 5 9 (
2
1
3 2
=

= x x
2 ) 2 1 (
3
1
3 2 1
= +

= x x x
>> A = [-3 2 -1;6 -6 7;3 -4 4]
>> b = [-1;-7;-6]
>> x=A\b
MATLAB:
next elimination fail
Gaussian Gaussian Elimination with Pivoting Elimination with Pivoting
To prevent failure from zero diagonal elements
Augmented matrix:
[ ]
2 4 2 2 4
1 2 4 3 5
3 3 8 2 7
1 1 6 3 7
A A b

= =




%
R1
R2
R3
R4
2 4 2 2 4
0 0 5 2 7
0 3 5 5 1
0 3 5 4 5

R2 R1/2,
R3 + 3R1/2,
R4 + R1/2
First row operation:
P I V O T I N G P I V O T I N G
Exchange row to avoid zero pivot element
2 4 2 2 4
0 0 5 2 7
0 3 5 5 1
0 3 5 4 5

2 4 2 2 4
0 3 5 4 5
0 3 5 5 1
0 0 5 2 7

R3=R3-R2
2 4 2 2 4
0 3 5 4 5
0 0 0 1 4
0 0 5 2 7

2 4 2 2 4
0 3 5 4 5
0 0 5 2 7
0 0 0 1 4

Solving Systems with the Backslash Operator Solving Systems with the Backslash Operator
A x = b
x = A
-1
b x = A \ b
MATLAB
A \ . . . means multiply on the left by the inverse of A
>> A = [2 4 -2 -2;1 2 4 -3;-3 -3 8 -2;-1 1 6 -3]
>> b = [-4;5;7;7]
>> x = A\b
x =
1.0000
2.0000
3.0000
4.0000

10 2 0
6 1 0
3 2 1
LU LU Decomposition Decomposition
To solve several Ax =b systems with the same A but different b
LU decomposition avoids repeating steps of Gaussian elimination on A
Main idea is to record the steps used in Gaussian elimination.
Consider the matrix:

=
10 2 0
12 5 2
3 2 1
A
Gaussian elimination:
R2 2R1
(2)
For recording
A
Gaussian elimination (cont):

2 ) 2 ( ) 0 (
6 1 ) 2 (
3 2 1
R3 0R1
R3 + 2R1

=
1 2 0
0 1 2
0 0 1
L
Lower triangular matrix:
Mysterious coincidence !!!
A LU =

=
10 2 0
12 5 2
3 2 1
2 0 0
6 1 0
3 2 1
1 2 0
0 1 2
0 0 1
MATLAB: >> L = [1 0 0;2 1 0;0 -2 1]
>> U = [1 -2 3;0 -1 6;0 0 2]
>> A = L*U
A =
1 -2 3
2 -5 12
0 2 -10
L
U
Using Using LU LU to solve equations to solve equations
Ax = b LUx = b
Ly = b : Forward substitution
Ux = y : Backward substitution
For Gauss elimination with pivoting:
PA = LU
P = identity matrix with same rows
switched as A in pivoting.
For example PA : (switch R2 R3)

12 5 2
10 2 0
3 2 1
10 2 0
12 5 2
3 2 1
0 1 0
1 0 0
0 0 1
PAx = Pb b LUx = b
Ly = b
Ux = y
Gauss elimination as Gauss elimination as LU LU decomposition decomposition
Gauss elimination can be used to decompose A into L and U as illustrated
for a three-equation system,

3
2
1
3
2
1
33 32 31
23 22 21
13 12 11
b
b
b
x
x
x
a a a
a a a
a a a
Forward
elimination
U =

33
23 22
13 12 11
0 0
0
a
a a
a a a
Store row operation @ zeros position:

33 32 31
23 22 21
13 12 11
) ( ) (
) (
a f f
a a f
a a a
,
11
21
21
a
a
f = where ,
11
31
31
a
a
f =
22
32
32
and
a
a
f

=
[A] [L][U]

=
33
23 22
13 12 11
0 0
0
a
a a
a a a
U

=
1
0 1
0 0 1
32 31
21
f f
f L
EXAMPLE : LU Decomposition with Gauss Elimination

=
4 4 3
7 6 6
1 2 3
A
From previous example,

=
2 0 0
5 2 0
1 2 3
U
After forward elimination,

=
33 32 31
23 22 21
13 12 11
a a a
a a a
a a a

=
33
23 22
13 12 11
0 0
0
a
a a
a a a
, 2
3
6
11
21
21
=

= =
a
a
f
, 1
3
3
11
31
31
=

= =
a
a
f
1
2
2
22
32
32
=

=
a
a
f
Lower triangular matrix,

=
1 1 1
0 1 2
0 0 1
1
0 1
0 0 1
32 31
21
f f
f L
Check Check LU LU = = A A or not ? or not ?
After the second step of forward elimination

3 2 0
5 2 0
1 2 3

=
33 32
23 22
13 12 11
0
0
a a
a a
a a a
First step: R2+2R1
Second step: R3+R1
Third step: R3-R2
Using Using LU LU to solve equations ( to solve equations (con con t t) )
Ax = b LUx = b
Ly = b
Ux = y

6
7
1
4 4 3
7 6 6
1 2 3
2
2
1
x
x
x

3
2
1
3
2
1
32 31
21
1
0 1
0 0 1
b
b
b
y
y
y
f f
f
Ly = b

6
7
1
1 1 1
0 1 2
0 0 1
3
2
1
y
y
y
1 1
b y =
21 1 2 2
f y b y =
32 2 31 1 3 3
f y f y b y =
1
1
= y
9 ) 2 )( 1 ( 7
2
= = y
2 1 ) 9 ( ) 1 )( 1 ( 6
3
= = y
Ux = y

3
2
1
3
2
1
33
23 22
13 12 11
0 0
0
y
y
y
x
x
x
a
a a
a a a

2
9
1
2 0 0
5 2 0
1 2 3
3
2
1
x
x
x
33
3
3
a
y
x

=
22
23 3 2
2
a
a x y
x

=
11
13 3 12 2 1
1
a
a x a x y
x

=
1
2
2
3
=

= x
2
2
) 5 )( 1 ( 9
2
=

= x
2
3
) 1 )( 1 ( 2 2 1
1
=

= x
MATLAB Function: MATLAB Function: lu lu
>> [L, U] = lu(A)

6
7
1
4 4 3
7 6 6
1 2 3
2
2
1
x
x
x
>> A = [-3 2 -1;6 -6 7;3 -4 4];
>> b = [-1; -7; -6];
>> [L,U] = lu(A)
LU Decomposition: LU Decomposition:
>> y = L\b
>> x = U\y
Solve equations: Solve equations:
Various Various LU LU decompositions decompositions
Doolittle Doolittle s method s method (L has ones on the diagonal)

33 32 31
23 22 21
13 12 11
33
23 22
13 12 11
32 31
21
0 0
0
1
0 1
0 0 1
a a a
a a a
a a a
u
u u
u u u
l l
l
Crout Crout s s reduction reduction (U has ones on the diagonal)

33 32 31
23 22 21
13 12 11
23
13 12
33 32 31
22 21
11
1 0 0
1 0
1
0
0 0
a a a
a a a
a a a
u
u u
l l l
l l
l
Cholesky Cholesky s s method method (The diagonal terms are the same value for
the L and U matrices)
where, l
ii
= u
ii

33 32 31
23 22 21
13 12 11
33
23 22
13 12 11
33 32 31
22 21
11
0 0
0 0
0 0
a a a
a a a
a a a
u
u u
u u u
l l l
l l
l
Cholesky Cholesky Decompositions Decompositions
One of the most popular approaches based on the fact that a symmetric
matrix can be decomposed as
[A] = [U]
T
[U]
- [A] is a symmetric matrix
- Computational advantages:
- Half storage
- Half computation time
Decomposition by recurrence relations. For the i th row:

=
=
1
1
2
i
k
ki ii ii
u a u
n i j
u
u u a
u
ii
i
k
kj ki ij
ij
, , 1 for
1
1
K + =

=
EXAMPLE : Cholesky Decomposition

=
494 , 1 216 54
216 32 8
54 8 4
A
For the first row (i = 1):
2 4
11 11
= = = a u
4
2
8
11
12
12
= = =
u
a
u
27
2
54
11
13
13
= = =
u
a
u For the second row (i = 2):
4 4 32
2 2
12 22 22
= = = u a u
27
4
27 4 216
22
13 12 23
23
=

=

=
u
u u a
u
For the third row (i = 3):
6 27 27 494 , 1
2 2 2
23
2
13 33 33
= = = u u a u

=
6 0 0
27 4 0
27 4 2
U
Thus, Cholesky yields
A U U =

=
494 , 1 216 27
216 32 8
27 8 4
T

=
33
23 22
13 12 11
0 0
0
u
u u
u u u
U
MATLAB Function: MATLAB Function: chol chol
>> U = chol(A)
>> A = [4 8 54;8 32 216;54 216 1494];
>> U = chol(A)
Cholesky Cholesky Decomposition: Decomposition:
>> U*U
Test: Test:

=
494 , 1 216 54
216 32 8
54 8 4
A
Example : Member forces in truss
[M
2
= 0] V
3
(2) 1000(1.5) = 0 V
3
= 750 N
[F
x
= 0] H
2
= 0
[F
y
= 0] V
2
= 250 N
@ node 1 :
[F
x
= 0] F
1
cos30
o
F
3
cos60
o
= 0
[F
y
= 0] F
1
sin30
o
+ F
3
sin60
o
= 1000
@ node 2 :
[F
x
= 0] F
1
cos30
o
+ F
2
= 0
[F
y
= 0] F
1
sin30
o
= 250
F
2
90
o
1
1000 N
30
o
60
o
F
3
F
1
3
2 H
2
V
2
V
3
0
.
8
6
6

m
0.5 m 1.5 m
@ node 3 :
[F
x
= 0] F
2
+ F
3
cos60
o
= 0
[F
y
= 0] F
3
= 750/sin60
o
= 866 N
F
1
= 250/sin30
o
= 500 N
F
2
= 866cos60
o
= 433 N
F
1
sin30
o
+ F
3
sin60
o
= 1000
F
1
cos30
o
+ F
2
= 0
F
2
+ F
3
cos60
o
= 0

0
0
000 , 1
60 cos 1 0
0 1 30 cos
60 sin 0 30 sin
3
2
1
F
F
F
>> A=[sin(pi/6) 0 sin(pi/3);-cos(pi/6) 1 0;0 -1 cos(pi/3)]
>> b=[1000;0;0]
>> F=A\b
F =

500.0000
433.0127
866.0254