Sei sulla pagina 1di 15

Chapter 2

Linear Algebraic Systems


2.1
2.1.1

Motivation and Objectives

ref. 2.1

Robotic Arm

Consider the problem of having the two-axis planar robotic arm (Fig. 2.1.1). A linear
algebraic system which relates the induced torques to the end-of-arm force F can be
obtained (see eq. 2.1.1). We want to find the unknowns F with the given .

2.1.2

Converter Circuit

Consider the problem of finding the voltages V and currents I in an electrical circuit
(see Fig. 2.1.2). Applying Kirchhoffs voltage law, one can obtain the system of linear
algebraic equations which describe the relation of V = IR (R =resistance) (see eq. 2.1.3).

2.1.3

Objective

Be able to efficiently solve linear algebraic systems of equations.


Be able to compute the determinant and the inverse of a matrix.
Learn how to represent linear systems of equations using compact vector notation,
then solve them using direct methods based on elementary row operations.
Be able to compare the computational effect required for the direct methods including Gauss-Jordan, Gaussian elimination, and LU decomposition.

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

Know when a system of equations is difficult to accurately solve by using the condition number of the coefficient matrix.
Be able to efficiently solve very large sparse systems of equations using iterative
methods including the Jacobi, Gauss-Seidel, and successive relaxation methods.

2.2

Gauss-Jordan Elimination

ref. 2.2

The linear system of equations has


A11 x1 + A12 x2 + + A1n xn = b1
A21 x1 + A22 x2 + + A2n xn = b2
..
..
.
.
An1 x1 + An2 x2 + + Ann xn = bn
Or rewrite as the matrix form

Ax = b,

A=

A11 A12 A1n


A21 A22 A2n
..
..
..
.
.
.
An1 An2 ann

x=

x1
x2
..
.
xn

b=

b1
b2
..
.
bn

Using matrix multiplication, the kth equation in the linear algebraic system can be expressed as follows
n
X

Akj xj = bk ,

1kn

(2.1)

j=1

Matrix inverse: brute way to solve the system of equations


x = A1 b,

A1 = inverse of A

There are more efficient methods that find x without first finding A1 . The basic idea
is to reformulate the equations, using elementary operations, so as to make the solution
transparent. Three elementary row operations are list below
Operation
Description
1
Interchange rows k and j
2
Multiply row k by 6= 0
3
Add times row j to row k
Terminology

Matrix
E1 (k, j)
E2 (k, )
E3 (k, j, )

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

Premultiplication (multiplication on the left BA), postmultiplication (multiplication on the left AB),
Augmented matrix: C = [A, b], C is the (n n + 1) augmented matrix for the
linear system Ax = b, (A is n) matrix.
Pivot element: the magnitude is largest in a row or column in a matrix.
Algorithm of Gauss-Jordan Elimination
1. Set E = 1 and form the augmented n (n + 1) matrix C = [A, b].
2. For j = 1 to n do
(a) Compute the pivot index j p n such that
n

|Cpj | = max{|Cij |}
i=j

(b) If Cpj = 0, set E = 0 and exit.


(c) If p > j, interchange rows p and j.
(d) Divide row j by the pivot Cjj .
(e) For each i 6= j, subtract Cij times row j from row i.
3. Partition C as C = [I, x].
Error code E = 1 =unique solution, E = 0 =singular matrix (no unique solution).
Steps (2a) to (2c) of above algorithm implement the pivoting operation that can improve
the numerical accuracy of the solution. Significant round-off error can occur that might
otherwise be avoided by using pivoting.
Operation Count
The operation could can be used to examine the computational effort for any numerical
method. Consider the number of floating point multiplication and divisions (FLOPs) of
the Gauss-Jordan elimination. The normalization step (2d) requires n + 1 j divisions
because Cij = 0 for i < j and Cjj = 1. The change of variable is i = n + 1 j (integer
P
summation ni=1 i = n(n + 1)/2).
division d = n + 1 j,
j =n+1i

variable change i = n + 1 j

d = n + 1 j = n + 1 (n + 1 i) = i

The total no. of division operations for the normalization step (2d) for 1 j n is
cnorm (n) =

n
X
i=1

d=

n
X
i=1

i=

n(n + 1)
2

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

Elimination process step (2e) requires (n1)(n+1j) multiplication because Cjk = 0 for
k < j and Cij = 0 for i 6= j. The total no. of multiplication operations for the elimination
process for 1 j n is
celim (n) = (n 1)cnorm
Combining computational effort of cnorm + celim is
cGJ =

2.2.1

n2 (n + 1)
n3
n(n + 1) (n 1)n(n + 1)
+
=

FLOPs n  1
2
2
2
2

(2.2)

Matrix Inversion

From system Ax = b
A1 Ax = A1 b x = A1 b
The solution x can be obtained with the matrix inversion.
A more effective alternative is to modify the algorithm of Gauss-Jordan elimination. From
the augmented n 2n matrix C = [A, I], applying elementary row operations
C = [A, I] [I, A1 ]
Algorithm of Matrix Inverse
1. Set E = 1 and form the augmented n 2n matrix C = [A, I].
2. For j = 1 to n do
(a) Compute the pivot index j p n such that
n

|Cpj | = max{|Cij |}
i=j

(b) If Cpj = 0, set E = 0 and exit.


(c) If p > j, interchange rows p and j.
(d) Divide row j by the pivot Cjj .
(e) For each i 6= j, subtract Cij times row j from row i.
3. Partition C as C = [I, A1 ].
Operation Count
The normalization step (2d) requires 2n j division because Cij = 0 for i < j, Cjj = 1.
The change of variable is i = n j
division d = 2n j,
j =ni

variable change i = n j

d = 2n j = 2n (n i) = n + i

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

10

The total no. of division operations for the normalization step (2d) for 1 j n is
cnorm (n) =

n
X

d=

i=1

n
X

(n + i) =

i=1

n(3n 1)
2

Elimination process step (2e) requires (n 1)(2n j) multiplication because Cjk = 0 for
k < j and Cij = 0 for i 6= j. The total no. of multiplication operations for the elimination
process for 1 j n is
celim (n) = (n 1)cnorm
Combining computational effort of cnorm + celim is
cinv =

n(3n 1) (n 1)n(3n 1)
n2 (3n 1)
3n3
+
=

FLOPs n  1
2
2
2
2

(2.3)

cinv > cGJ , matrix inversion method is more expensive than the Gauss-Jordan elimination
method.

2.3

Gaussian Elimination

ref. 2.3

Gauss elimination is a modification of the Gauss-Jordan elimination method that reduces


the no. of operations. This method solves the system Ax = b using only partial elimination, thereby reducing the first n column of C to an upper-triangular matrix D (a matrix
with zeros below the main diagonal).
C = [A, b] [D, e] then solve Dx = e
Since D is an upper-triangular matrix, only small computational effort is needed to solve
Dx = e.
Algorithm of Gaussian Elimination
1. Set E = 1 and form the augmented n (n + 1) matrix C = [A, b].
2. For j = 1 to n do
(a) Compute the pivot index j p n such that
n

|Cpj | = max{|Cij |}
i=j

(b) If Cpj = 0, set E = 0 and exit.


(c) If p > j, interchange rows p and j.
(d) For each i > j, subtract Cij /Cjj times row j from row i.

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

11

3. Partition C as C = [D, e where D is n n and e is n 1.


4. For j = n down to 1 compute
n
X
1
xj =
(ej
Dji xi )
Djj
i=j+1

The above algorithm contains two-pass algorithm: forward elimination (step (1) to (3)),
back substitution (step (4)).
Operation Count:
cGE

n3
FLOPs,
3

n1

cGE < cGJ : Gaussian elimination method reduce the computational cost comparing
with the Gaussian Jordan elimination method.

2.3.1

Determinant

Determinants are very useful for developing theoretical properties of matrices. However
they are used less often in numerical work because their computation is expensive and
susceptible to accumulated round-off error.
Useful properties of determinant: see Table 2.3.1.
The Gaussian elimination method transforms a matrix into upper-triangular form using
elementary row operations.
U = Er E2 E1 A,

r = no. row interchanges

det(Er E2 E1 ) = (1)r ,

det(U ) = U11 U22 Unn

det(A) = (1)r U11 U22 Unn


Algorithm of Matrix Determinant
1. Set r = 0.
2. For j = 1 to n do
(a) Compute the pivot index j p n such that
n

|Apj | = max{|Aij |}
i=j

(b) If Apj = 0, set = 0 and exit.

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

12

(c) If p > j, interchange rows p and j and set r = r + 1.


(d) For each i > j, subtract Aij /Ajj times row j from row i.
3. Set = (1)r A11 A22 Ann
= det(A)
Operation Count:
cdet

2.4
2.4.1

n3
FLOPs,
3

LU Decomposition

n1

ref. 2.4

LU Factorization

The matrix A can be decomposed into an LU factorization (L =lower-triangular matrix,


U = upper-triangular matrix with the diagonal terms set to unity).

A = LU

L11

L21
L=
.
..

Ln1

L22
.. . .
.
.

U =

0
..

Lnn

U11 U12 U1n


.
..
0 U22 ..
.
..
..
.. ..
.
.
.
.
0 0 Unn

We often set Ukk = 1, 1 k n in order to uniquely define the elements of L, U from A.


The Crouts method for computing the element of L and U .
Lkj = Akj

j1
X

Lki Uij ,

jkn

i=1

Ujk =

j1
X
1
(Ajk
Lji Uik ),
Ljj
i=1

jkn

Since there are a total of n2 variable elements inL and U , these two triangular matrices
can be stored in a single compact n n storage matrix Q = [L\U ] (see eq. 2.4.13).
Algorithm of LU Factorization
1. Set = 1 and form the augmented n 2n matrix D = [A, I].
2. For j = 1 to n do

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

13

(a) For k = j to n components


Dkj = Dkj

j1
X

Dki Dij

i=1

(b) Compute the pivot index j q n such that


n

|Dqj | = max{|Dij |}
i=j

(c) If Dqj = 0, set = 0 and exit.


(d) If q > j, interchange rows q and j of D and set = .
(e) For k = j + 1 to n components
Dkj

j1
X
1
(Djk
Dji Dik )
=
Djj
i=1

(f) Set = Djj


3. Partition D as D = [Q, P ] where Q and P are n n.
P =row permutation matrix due to pivoting, LU = P A.

2.4.2

Forward and Back Substitution

Once the coefficient matrix is factored into lower- and upper-triangular parts, a linear
algebraic system can be solved very efficiently for a variety of RHS vectors.
Consider the system Ax = b,
Rewrite as P Ax = P b since LU = P A LU x = P b
Solution procedure
d = Pb
Ly = dforward substitution
U x = yback substitution
Algorithm of LU Decomposition
1. Apply algorithm of LU Factorization to compute , Q, and P . If = 0, exit.
2. Compute d = P b.

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

14

3. For k = 1 to n compute
yk =

k1
X
1
(dk
Qki yi )
Qkk
i=1

4. For k = n down to 1 compute


n
X

xk = yk

Qki xi

i=k+1

Operation Count:
cLU

2.4.3

n3
FLOPs,
3

n1

Tridiagonal Systems

In engineering applications, there are many instances where the coefficient matrix A is
sparse in the sense that it has a large no. of zero elements arranged in some patterns. The
tridiagonal matrix is a banded matrix in which nonzero elements on the main-, upperand lower-diagonal.

b 1 a1 0
0

..

c 1 b 2 a2
0

..
...
...
0 ... ...
.

A= . .

.
.
.
.

.. ..
..
..
.

..

. 0 cn2 bn1 an1

cn1

bn

The LU decomposition method simplifies considerably when the coefficient matrix A is


tridiagonal. In this case the lower- and upper-triangular factors are also tridiagonal. The
matrices L and U have the following structure:
The element of L and U can be obtained as follows:
L11 = b1
ak
, 1kn
Uk,k+1 =
Lkk
Lk+1,k+1 = bk+1 ck Uk,k+1 , 1 k n
Operation Count:
ctri = 5n 4 FLOPs
This results in a dramatic reduction in computational effort when n is large. The storage
requirements for L and U can also be reduced significantly because there is no need to
store the zero elements. As we will see tridiagonal systems arise repeatedly in the solution
of linear PDE by the finite-difference method.

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

2.5

Ill-Conditioned Systems

15

ref. 2.5

Some linear algorithm system are more sensitive to round-off error than others. Indeed,
for some systems a small change in one of the values of the coefficient matrix or the RHS
vector can give rise to a large change in the solution vector.
When the solution is highly sensitive to the values of the coefficient matrix A or the RHS
vector b, the eqs. are said to be ill-conditioned. This can be a serious numerical problem
for two reasons:
Often the eqs. are generated from a mathematical model of an underlying physical
systems, and only estimates of the values of A and b are known. If the system
Ax = b is ill-conditioned, then the solution is very sensitive to the accuracy of these
estimates.
Even when exact values of A and b are known, we can not represent these values
exactly using a computer with finite precision.

2.5.1

Vector and Matrix Norms

How do we know when a linear algorithm systems is ill-conditioned? To develop a concise


measure, it is useful to first consider the notation of vector and matrix norm.
Norm a measure of the length or magnitude of the vector.
n

||x|| = max{|xk |},


k=1

infinity norm of x

Property of norms: see Table 2.5.1.


To illustrate the use of vector norm, let us define the residual error vector as
r(x) = b Ax
If r(x) = 0 x is a solution of Ax = b, and round-off error will typically cause r(x) 6= 0.
Thus, we can use ||r(x)|| to measure the accuracy of the solution.
Matrix norm it can be defined in terms of a vector norm as
||Ax||
}, sup = supremum or smallest upper bound
||A|| = sup{
||x||
x6=0
Note that ||Ax|| ||A|| ||x||.
The induced matrix norm can be expressed directly in terms of the components of the
matrix
n
n

||A|| = max{
k=1

X
j=1

|Akj |}

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

16

The expression for ||A|| involves summing the absolute values of elements in the rows of
A. The matrix norms satisfies all the same properties as the vector norm listed in Table
2.5.1.

2.5.2

Condition Number

We can perturb the eqs. and then examine the size of resulting change in the solution.
Consider the effect of changes as
b b + b,

then solutionx x + x

Then A(x + x) = b + b reduces to Ax = b. Note that x = A1 b, one gets


||x|| ||A1 || ||b||
This provides an upper bound on the variation in the solution x caused by the perturbation
b. One can further investigate the relative error caused by a relative change in b as
||b||
||x||
K(A)
||x||
||b||
Here the scalar K(A) is called th econdition number of the matrix A and is defined as
K(A) = ||A|| ||A1 ||,

A is nonsingular

It is also possible to express changes in the solution caused by changes in the coefficient
matrix A
||x||
||A||
K(A)
||x + x||
||A||
When the condition number K(A) becomes large, the system is regarded as being illconditioned.
How large does K(A) have to be before a system is regarded as ill-conditioned? The lower
and upper bound of K(A) is given as
1 K(A) <

2.5.3

Approximate Condition Number

The condition number of A can be approximated without going through all of the computations as K(A) = ||A|| ||A1 ||. To estimate K(A), note that if x = A1 b, then
||x|| ||A1 || ||. Thus

||x||
||A1 ||
||b||

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

17

Since the above eq. must hold for every b, the estimate can be improved by considering
several RHS vectors, solving the system for each, and then taking the largest ratio. In
particular, suppose {b1 , b2 , , bm } is a set of RHS vectors and let {x1 , x2 , , xm } be the
corresponding solutions of Ax = bk , 1 k m. The following can be use to approximate
the condition number
||xk ||
n
K(A) ||A|| max{ k }
k=1 ||b ||

2.5.4

Iterative Improvement

Two types of matrix solver


Direct method methods that takes a fixed number of steps to produce a solution.
Iterative method use a variable number of steps to produce a sequence of increasingly accurate estimates of the solution.
As a bridge between the direct methods and the iterative methods, we consider a hybrid
method called iterative improvement that can be applied as a final step to correct or
improve the numerical solution generated by a direct method. This method is particularly
useful for ill-conditioned systems.
Let xe denotes the exact solution and x denotes the numerical solution generated by a
direct method. Then
xe = x + x
where x =the error in the numerical solution. Note that neither xe nor x are known.
However, it is clear that if x can be determined, then xe can be recovered from x. The
solution of x can be found from the following auxiliary system
Ax = r(x)
Note that the residual error vector r(x) is often quite small. For this reason, the above
solution is often computed using high precision arithmetic. Once a solution of the above
system is obtained, it can be used to correct the original solution as follows:
xc = x + x
The solution xc is merely an improved approximation to the solution, since xc = xe only
if the exact solution to Ax = r(x) is obtained. Typically the iterative improvement step
is performed only once to tune the solution.

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

2.6

Iterative Methods

18

ref. 2.6

We can start with an arbitrary initial estimate and then proceed with as many iterative
steps as needed. Iterative methods are attractive for solving very large systems where
performing the direct methods (order O(n3 )) can be prohibitive. In practical applications, large systems have coefficient matrices that are sparse with relatively few nonzero
entries. Here iterative methods are particularly appealing because the sparse structure of
the coefficient matrix is preserved.

2.6.1

Jacobi Method

To apply any of the iterative techniques, the eqs. should first be preprocessed with a row
permutation matrix D, if needed, to ensure that the diagonal elements of the coefficient
matrix are nonzero. The coefficient matrix A is then decomposed into a sum of lowertriangular, diagonal, and upper-triangular terms as follows
A=L+D+U

(2.4)

where L =lower-triangular with zeros on the diagonal, D =diagonal with nonzero diagonal
elements, U =upper-triangular with zeros on the diagonal.
The L, D and U are obtained directly from inspection of A. Using eq. (2.4), we have
Ax = Dx + (L + U )x and Ax = b, then Dx = b (L + U )x. This is the basis for the
Jacobi method. Actual calculation procedure is obtained as follows
Dxk+1 = b (L + U )xk ,

k0

(2.5)

Note that D is diagonal, the Jacobi iteration in (2.5) can be written as


=
xk+1
i

X
1
(bi
Aij xkk ),
Aii
j6=i

1in

(2.6)

To start with the iterative scheme, an initial guess x0 must be chosen. Typically, the
iterations are repeated until the norm of the residual error vector is sufficiently small.
A simple sufficient condition for convergence is
xk x as k if

||D1 (L + U )|| < 1

(2.7)

] The above condition is equivalent to saying that for each row of A


|Akk | >

|Akj |,

1kn

j6=i

We say that the coefficient matrix A is strictly diagonal dominant in a row sense.

(2.8)

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

2.6.2

19

Gauss-Seidel Method

The Gauss-Seidel (GS) method is an improvement of the convergence rate of the Jacobi
method. The G-S method takes advantage of the most recent information when updating
an estimate of a solution as follows
xk+1
=
i

i1
n
X
X
1
(bi
Aij xk+1

Aij xkj ),
j
Aii
j=1
j=i+1

1in

(2.9)

The scalar version of (2.9) can be written in vector form as


xk+1
= D1 (b Lxk+1 U xk )
i
Or
(D + L)xk+1
= b U xk ,
i

k>0

(2.10)

The above eq. can be solved easily with forward substitution as eq. (2.10) because the
coefficient matrix (D + L) is lower-triangular.
The sufficient condition which guarantees that the G-S method will converge is
xk x as k if

||(D + L)1 U || < 1

(2.11)

Solving for xk+1 in eq. (2.11), the norm of the matrix coefficient of xk 1.

2.6.3

Relaxation Method

The GS method can be generalized by adding a new parameter. Begin from eq. (2.9), the
eq. can be rewritten by adding and subtracting xki
xk+1
= xki +
i
The term

1
( )
Aii

i1
n
X
X
1
(bi
Aij xk+1

Aij xkj ),
j
Aii
j=1
j=i

1in

(2.12)

represents the correction to xki needed to produce xk+1


.
i

The convergence rate can be increased by introducing a relaxation factor > 0.


xk+1
= xki +
i

i1
n
X
X

(bi
Aij xk+1

Aij xkj ),
j
Aii
j=1
j=i

1in

(2.13)

The above eq. is called the successive relaxation (SR).


< 1: under-relaxation, = 1: no-relaxation (recover the GS method), > 1: overrelaxation.
Rewrite eq. (2.13) as
xk+1
= (1 )xki +
i

i1
n
X
X

(bi
Aij xk+1

Aij xkj ),
j
Aii
j=1
j=i+1

1in

(2.14)

CHAPTER 2. LINEAR ALGEBRAIC SYSTEMS

20

The scalar eq. (1) can also be written in vector form as


xk+1
= (1 )xk + D1 (b Lxk+1 U xk )
i
Or
(D + L)xk+1
= b [(1 )D U ]xk ,
i

k0

(2.15)

The sufficient condition for convergence is


||(D + L)1 [(1 )D U ]|| < 1

(2.16)

A second sufficient condition for convergence is 0 < < 2.

2.6.4

Convergence

The three iterative methods that we have examined can all be formulated in the following
general way as
xk+1 = Bxk + c, k 0
where x0 =initial guess. The iteration matrix B, and offset vector c, corresponding to the
Jacobi method, the GS method, the SR method are summarized in Table 2.6.4.

2.7

Applications

ref. 2.7

Potrebbero piacerti anche