1

S YSTEMS

OF

L INEAR E QUATIONS

Introduction

Elimination Methods

Decomposition Methods

Matrix Inverse and Determinant

Errors, Residuals and Condition Number

Iteration Methods

Incomplete and Redundant Systems

Chapter 1

Systems of Linear Equations / 2

1.1 Introduction

The system of linear equations is formed by the addition of the products of

a variable with a coefficient, which is also a constant.

The system of linear equation can be solved via matrix approach.

The general form of a set of a linear equation having n linear equations and
n unknowns is

a11 x1 + a12 x 2 + L + a1n x n = b1

a 21 x1 + a 22 x 2 + L + a 2 n x n = b2

(1.1)

M
M
O
M
M
a n1 x1 + a n 2 x 2 + L + a nn x n = bn

where x1 , x2 ,K, xn are variables or unknowns, aij and bj are coefficient or

constant (real or complex).

Eq. (1.1) can be written in a more compact form:

[ aij ] { x j } = {bi }

Ax = b

(1.2)

where A is a matrix [aij] of size nn, x is a variable vector {xj} and b is a

right-hand side vector {bj}.

The process of solving Eq. (1.2) yield three possible solutions:

1.

Unique solution e.g.:

3 x1
x1
2.

3.

x2

= 1

+ 3 x2

= 1

x1 = x2 =

No solution e.g.:

x1

x2

x1

x2

= 1
= 1

x2

Infinite solutions e.g.:

x1
2 x1

+ 2 x2

= 2

1
4

Chapter 1

Systems of Linear Equations / 3

1.2 Elimination Methods

The most popular method is the Gauss elimination method, which

comprises of two steps:
1.
2.

Forward elimination to form an upper triangular system via rowbased transformation process,
Back substitution to produce the solution of xj.

Consider the following system:

a11 x1 + a12 x2 + L + a1n xn = b1

a21 x1 + a22 x2 + L + a2 n xn = b2
M

an1 x1 + an 2 x2 + L + ann xn = bn
If a11 0 , for i = 2,3,,n, substract the i-th equation with the product of
ai1 a11 with the first equation to produce the first transformed system:

a11 x1 +

a12 x2
(1)
a22
x2
M
(1)
a n 2 x2

+ L + a1n xn
+ L + a2(1n) xn
O
M
(1)
+ L + ann xn

= b1
= b2(1)
M
= bn(1)

where

aij(1) = aij

ai1
a1 j
a11

for i, j = 2,3,,n

bi(1) = bi

ai1
b1
a11

for i = 2,3,,n

The process can be repeated for (n1) times until the (n1)-th transformed
system is formed as followed, which completes the forward eliminations:

a11 x1 +

a12 x2
(1)
a22
x2

+
+

a13 x3
(1)
a23
x3
(2 )
a33
x3

+ L +
+ L +
+ L +
O

a1n xn
a2(1n) xn
a3(2n) xn
M

(n 1)

ann xn

=
=
=

b1
b2(1)
b2(2 )
M

(n 1)

= bn

(1.3)

Chapter 1

Systems of Linear Equations / 4

where
(k )

(k 1)

(k )

(k 1)

aij = aij

bi

= bi

aik(k 1) (k 1)
(k 1) akj
akk

aik(k 1) (k 1)
(k 1) bk
akk

for i, j = k+1,,n

(1.4a)

for i = k+1,,n

(1.4b)

Back substitutions can then be executed so that xj are solved:

bn(n1)
xn = (n1)
ann

(1.5a)

1 ( k 1)
xk = ( k 1) bk akj(k 1) x j for k = n1,,1
akk
j = k +1

(1.5b)

The above method can fail if akk 0, the row has to be interchanged,
which is referred to as pivoting:

x1

x2

= 2

x2

= 3

Pivoting

x1

x2

= 3

x2

= 2

where the new diagonal element akk is called a pivot, which can be selected
among the maximum absolute value of aik.

The pivotal Gauss elimination gives a more accurate solutions, e.g.

consider these systems (values to be rounded up to 3 significant figures):
Original Gauss elimination:
.34
(1)( 2 ) 0.00126
0.417 0.418 (2 ) 0.100126
0.417
0.418
0.00126

1.34

0.708 0.632
444.184 443.908

0 .0
x2 = 0.999
x1 = 1.125

Pivotal Gauss elimination:

(1)( 2 ) 1.34 0.708 0.632
0.708 0.632 (2 ) 0.100126
1.34
.34

0.00126
0.417 0.418
0.418 0.417
0.0

x2 = 0.998
x1 = 0.999

Exact solution:
x1 = 1

x2 = 1

Chapter 1

Systems of Linear Equations / 5

Example 1.1

Solve the following system using the Gauss elimination method:

2 x1
4 x1

+
+

x2
4 x2

+
+

3 x3
7 x3

=
=

2 x1

5x2

9 x3

= 3

1
1

Solution

The system can be rewritten in matrix form as:

2 1 3 x1 1
4 4 7 x = 1
2

2 5 9 x3 3

or

2 1 3 1
[A b] 4 4 7 1
2 5 9 3

First step of forward elimination:

1
2 1 3

0 2 1 1
0 4 6 2
( 2 ) 2 (1)( 2 )
( 3) (1)( 3)

Second step of forward elimination:

1
2 1 3
3) 2 ( 2 ) ( 3)
(
0 2 1 1
0 0 4 4
Hence, the transformed upper triangular system is:
2 x1

x2
2 x2

+
+

3 x3
x3

4 x3

1
= 1
4

Back substitutions are as follows

x3 = 4 4 = 1
1 x3
= 1
2
1 x2 3 x3
x1 =
= 12
2

x2 =

Chapter 1

Systems of Linear Equations / 6

Example 1.2

Perform the pivotal Gauss elimination to the system given in Example 1.1.
Solution

The pivotal Gauss elimination can be performed as followed:

( 2 ) 2 (1)( 2 )

4
7
4 4
2 1 3 1
4 4 7 1
2
(
3 ) (1)(3 )

4 4 7 1 (

1) ( 2 )
4
2 1 3 1

0 1 12

11
0 3
2 5 9 3
2 5 9 3
2

1
1
2
5
2

7 1
4 4 7 1
4 4
(3 ) 1(2 )(3 )

5
11
3
0 3 112 52

0 3
2
2
0 0 43 43
0 1 12 12
Hence, the upper triangular system is:
( 2 )(3 )

4 x1 + 4 x2

3 x2

7 x3
11
2
4
3

x3
x3

= 1
=
=

5
2
4
3

Then, back substitution can be performed:

x3 =
x2 =

4
3
5
2

4
3

= 1,

112 (1)

= 1 ,
3
1 4( 1) 7(1)
x1 =
= 12 .
4

Chapter 1

Systems of Linear Equations / 7

1.3 Decomposition Methods

In some cases, the left-hand side matrix A is frequently used while the
right-hand side vector b is changed depending on the case.

The overall system can be transformed to an upper triangular form so that it

can be used repeatedly for different b, thus matrix A has to be decomposed.

For a general non-symmetric system, the popular method is the Doolitle or

LU decomposition:

A = LU

(1.6)

where L and U are the lower and upper triangular matrices, respectively:
a11
a
21
a31

a12
a22
a32

a13 1 0
a23 = l21 1
a33 l31 l32
u11 u12
l21 u 22
l31 l32

u13
u 23
u33

0 u11 u12
0 0 u 22
1 0
0

u13
u 23 (in memory)
u33

The solution steps of the system are as followed:

A x = b LU x = b
By taking an intermediate vector y:

Ux = y

(1.7)

Ly = b

(1.8)

Hence,

The elements for L and U can be obtained from the Gauss elimination:
a11
U = 0
0

a12

(1)
a22
0

a13
(1)
a23

(2 )

a33

1
L = a21 a11
a31 a11

0
1
(1)
(1)
a32
a22

0
0
1

Chapter 1

Systems of Linear Equations / 8

Another variation of the LU decomposition is the Crout decomposition,

which maintains uii = 1 for i = 1,2,,n in U instead of L:
For the first row and column:

li1 = ai1
u1 j =

a1 j
l11

for i = 1,2,,n

(1.9a)

for j = 2,3,,n

(1.9b)

for i = j, j+1,,n

(1.9c)

for k = j+1, j+2,,n

(1.9d)

For j = 2,3,,n1:
j 1

lij = aij lik ukj

k =1

j 1

u jk =

a jk l ji uik
i =1

l jj

dan,
n 1

lnn = ann lnk u kn

(1.9e)

k =1

If the system is symmetric, the Cholesky decomposition can be used, where

matrix A can be decomposed such that:
A = LLT

(1.10)

For the k-th row:

i 1

lki =

aki lij lkj

j =1

lii

for i = 1,2,,k1

(1.11a)

k 1

lkk = akk lkj2

j =1

(1.11b)

This method optimises the use of computer memory in storing the

decomposed form of A.

Chapter 1

Systems of Linear Equations / 9

Example 1.5

Decompose the following matrix using the Doolittle LU decomposition:

2 1 3
A = 4 4 7
2 5 9
Solution

With reference to the matrix elements derived in Example 1.1:

2 1 3
U = 0 2 1 ,
0 0 4

0 0 1 0 0
1
L = 4 2 1 0 = 2 1 0 .
2 2 4 2 1 1 2 1
`

Example 1.7

Decompose the following matrix using the Cholesky decomposition:

2 1 3
A = 1 4 5
3 5 9
Solution

By using Eq. 1.11:

l11 = a11 = 2 ,

l21 = a21 l11 = 1

l31 = a31 l11 = 3 2 ,

a l l
l32 = 32 21 31 = 7 2 ,
l22

l22 = a22 l212 = 7 2 ,

2,

l33 = a33 l312 l322 = 1.

Maka,

L = 1 2
3 2

0
0 1.41421
0
0

7 2 0 = 0.70712 1.87083 0
7 2 1 2.12132 1.87083 1
`

Chapter 1

Systems of Linear Equations / 10

1.4 Matrix Inverse and Determinant

The Gauss elimination can be used to generate the inverse of a square

matrix A by replacing the left-hand side vector b with an identity matrix I.

By using the following identity:

A A 1 = I

(1.12)

(1) ( 2 )
(n )
If all columns of A1 are written as x , x ,K, x and the columns of the
(1) (2 )
(n )
I as e , e ,K, e , respectively, thus Eq. (1.12) can be rewritten as:

) (

A x(1) , x(2 ) ,K, x(n ) = e(1) , e(2 ) ,K, e(n )

Then, a set of n linear systems can be assembled:

A x (1) = e (1)
A x (2 ) = e (2 )
M

(1.13)

A x (n ) = e (n )

Consequently, the determinant of matrix A can simply be calculated using:

det (A ) A = ( 1) a11a22 a33 K ann

p

(1) ( 2 )

( n 1)

= ( 1)

a(
i =1

i 1)
ii

where p is the number of row interchange operation during pivoting.

(1.14)

Chapter 1

Systems of Linear Equations / 11

Example 1.8

Determine the inverse of the following matrix using the Gauss elimination:

4 2 1
A = 1 1 1
2 1 1
Solution

The combination of A and I can be represented in an augmented form:

4 2 1 1 0 0
1 1 1 0 1 0

2 1 1 0 0 1

4 2 1 1
0 12 54 14
0 0 92 32
Gauss
forward
elimination

0 0
1 0
4 1

Upon back substitution:

x (1)

0
= 13
13

x ( 2)

13
= 92
89

x ( 3)

13
= 59
92

Hence, the inverse of A is

A 1 = 13
13

1
3

92
8
9

95
2
9
1
3

`
Example 1.9

Calculate the determinant of the matrix given in Example 1.8.

Penyelesaian

In Example 1.0, there is no row interchange performed, thus p = 0. Hence,

det(A ) = 1

2
1

4 2 1

1 = ( 1) 0

2 1 1

1
2

5
4

0 0

9
2

= (4)( 12 )( 92 ) = 9
`

Chapter 1

Systems of Linear Equations / 12

1.5 Errors, Residuals and Condition Number

If x is an approximate solution of a linear system A x = b , then the

system error is defined as

e = x x

(1.16)

On the other hand, the system residue r is defined as

r = A e

(1.17)

r = A x A x = b A x

or,

For a well-conditioned system, the residue can represent the error.

Moreover, for comparison, a matrix or vector can be expressed in form of a

scalar known as norm.

For a vector x = (x1 , x2 ,K, xn ) , the p-norm is defined as

T

= x1 + x2

+ L + xn

p 1 p

n
p
= xi
i =1

1 p

(1.20)

If p = 1, it is known as 1-norm:
n

x 1 = x1 + x2 + L + xn = xi

(1.19)

i =1

If p = 2, it is known as Euclidean norm:

x e = x 2 = x + x +L+ x =
2
1

2
2

2
n

x
i =1

2
i

(1.18)

If p , it is known as a maximum norm:

= max{ x1 , x2 ,K, xn } = max xi

1in

1i n

(1.21)

For a matrix A = [ aij ] of size mn, the Frobenius norm, which is

equivalent to the Euclidean norm for vectors, is defined as

Ae=

a
i =1 j =1

2
ij

(1.22)

Chapter 1

Systems of Linear Equations / 13

and, the equivalent 1-norm and maximum norm for a matrix are defined as
n

A 1 = max aij = maximum sum of columns

1 j n

i =1

(1.23)

= max aij = maximum sum of rows

1i n

j =1

(1.24)

The properties of norms of a vector or matrix A are as followed:

1.

A 0

and A = 0 if, and only if, A = 0 .

2.

cA = c A

where c is a scalar quantity.

3.

A+B A + B

4.

AB A B

Triangular inequality, where B is a vector or

matrix of the same dimension of A.
Schwarz inequality, where B is a vector or
matrix which forms a valid product with A.

The concept of norms can be used to calculate the condition number

represents the health of a linear system, either ill- or well-conditioned.

If e is the error for the system A x = b , from the relations A e = r and

e = A 1 r , the following inequality can be established:

Ae r

and

e A 1 r

r
e A 1 r
A

b
x A 1 b
A

Also, from A x = b and x = A 1 b :

Ax b

and

x A 1 b

Thus, the combination of both inequality relations yields the range of the
relative error e x , i.e.

r
e
r
1
1

A
A
x
b
A A 1 b

Hence, the condition number is defined as

(A ) = A A 1

(1.25)

Chapter 1

Systems of Linear Equations / 14

where the range of the relative error is.

r
e
r
1

(A )
(A ) b
x
b

(1.26)

The characteristics of the condition number are that:

1.
2.

(A ) 1 the smaller the better, and otherwise.

If (A ) 1 , the relative residual r b can represent the relative
errors e x .

If the error is solely contributed by matrix A, the inequality becomes:

e
x

(A )

EA
A

On the other hand, if the error is solely contributed by vector b, the

inequality becomes:

e
e
(A ) b
b
x

(1.27)

(1.28)

Therefore, from Eqs. (1.26-8), it can be seen that the condition number can
determine the range of error and thus the health of a system.

Chapter 1

Systems of Linear Equations / 15

1.7 Iteration Methods

For large systems (size > 200), the elimination and decomposition methods
are not efficient due to increasing number of arithmetic operations.

The number of arithmetic operations can be reduced via iteration methods,

such as the Jacobi iteration and the Gauss-Seidel iteration methods.

In the Jacobi iteration, Eq. (1.1) can be written for xi from the i-th equation:

x1 =

1
(a12 x2 + a13 x3 + L + a1n xn b1 )
a11

x2 =

1
(a21 x1 + a23 x3 + L + a2 n xn b2 )
a22

(1.29)

M
xn =

1
(an1 x1 + an 2 x2 + L + an,n1 xn1 bn )
ann

(0)
( 0)
( 0)
(0)
Eq. (1.29) needs initial values x = (x1 , x 2 ,K, x n ) , which yield
T

x (1) = x1(1) , x2(1) ,K, xn(1) , and the computation continues as followed:

x1( k +1) =

1
a12 x2( k ) + a13 x3( k ) + L + a1n xn( k ) b1
a11

x2( k +1) =

1
a21 x1( k ) + a23 x3( k ) + L + a2 n xn( k ) b2
a22

)
(1.30)

M
xn( k +1) =

1
an1 x1( k ) + an 3 x3( k ) + L + an ,n 1 xn( k1) bn
ann

For k , vector x(k) converges to its exact solution if the diagonal

domain condition is followed, i.e.
n

aii > aij

j =1
j i

for i = 1, 2 , K , n

(1.31)

and the matrix which follows this condition is called a diagonal domain
matrix.

Chapter 1

Systems of Linear Equations / 16

To terminate the iteration process, a convergence or termination criterion

can be specified, i.e.

x (k +1) x (k ) <

(1.32)

The Gauss-Siedel iteration method uses the most current known solution
after each arithmetic operation in order to speed up convergence:

x1( k +1) =

1
a12 x2( k ) + a13 x3( k ) + L + a1n xn( k ) b1
a11

x2( k +1) =

1
a21 x1( k +1) + a23 x3( k ) + L + a2 n xn( k ) b2
a22

)
(1.33)

M
xn( k +1) =

1
an1 x1( k +1) + an 3 x3( k +1) + L + an ,n 1 xn( k1+1) bn
ann

As of the Jacobi method, the Gauss-Siedel method must also observe the
diagonal domain condition for convergence to be possible (see Fig. 1.1).
x1
2 x1

2 x2
+
x2
x2

= 2
=
2

2 x1
x1

+
x2
2 x2
x2

=
2
= 2

2x1 + x2 = 2
x1 2x2 = 2

( 52 , 65 )

( 52 , 65 )
x1
x1 2x2 = 2

2x1 + x2 = 2

(a) The off-diagonal domain system

FIG. 1.1

x1

(b) The diagonal domain system

Divergence and convergence in the Gauss-Seidel method

Chapter 1

Systems of Linear Equations / 17

Example 1.10

Use the Jacobi iteration method to solve the following system up to 5

decimal points:

64 x1
x1
2 x1

3 x2
+
x2
90 x2

x3
+ 40 x3
+
x3

= 14
= 20
= 5

Solution

First of all, form a diagonal domain system:

64 x1
2 x1
x1

3 x2
90 x2
+
x2

x3
+
x3
+ 40 x3

= 14
= 5
= 20

Then, rewrite the system according to Eq. (1.30):

x1(k +1) =

1
3x 2(k ) x3(k ) 14
64
1
x 2(k +1) = +
+ 2 x1(k ) + x3(k ) + 5
90
1
x3(k +1) =
+ x1(k ) + x 2(k ) 20
40

By taking an initial values x(0) = (0, 0, 0)T, thus the method converges
within 5 iterations:
Iteration no. 1:

x(1) = (0.21875, 0.05556, 0.50000)T,

Iteration no. 2:

x(2) = (0.22917, 0.06597, 0.49592)T,

Iteration no. 3:

x(3) = (0.22955, 0.06613, 0.49262)T,

Iteration no. 4:

x(4) = (0.22955, 0.06613, 0.49261)T,

Iteration no. 5:

x(5) = (0.22955, 0.06613, 0.49261)T.

`

Chapter 1

Systems of Linear Equations / 18

Example 1.11

Repeat problem given in Example 1.10 using the Gauss-Seidel iteration

method.
Solution

First of all, form a diagonal domain system:

64 x1
2 x1
x1

3 x2
90 x2
+
x2

x3
+
x3
+ 40 x3

= 14
= 5
= 20

By taking an initial values x(0) = (0, 0, 0)T, the first solution in the first
iteration:

x1(1) =

1
[ 3(0) 0 14] = 0.21875
64

(1)
(1)
Use x1 to calculate x2 and so on, i.e.

x2(1) =
x3(1) =

1
[2(0.21875) + 0 + 5] = 0.06042
90

1
(0.21875 + 0.06042 20) = 0.49302
40

Hence, the method converges within 4 iterations:

x(1) = (0.21875, 0.06042, 0.49302)T,
x(2) = (0.22929, 0.06613, 0.49262)T,
x(3) = (0.22955, 0.06613, 0.49261)T,
x(4) = (0.22955, 0.06613, 0.49261)T.
`

Chapter 1

Systems of Linear Equations / 19

1.8 Incomplete and Redundant Systems

If m n , there will be two situations:

1.
2.

m < n incomplete system.

m > n redundant system.

For incomplete system, no solution is possible since additional (n m)

equations from other independent sources are required until m = n.

For redundant system, a unique solution is not possible, and the system has
to be optimised via least square method (also known as linear regression):

S= e

2
e

= eTe ,

= (b A x ) (b A x ) ,
T

= b T b ( A x )T b b T ( A x ) + ( A x ) T ( A x ) ,
= (A x ) (A x ) ( A x ) b .
T

T
T T
Using the identity (AB ) = B A :

S = xT AT A x xT AT b

Minimising S:

S
x

= 0 = AT A x AT b

forms an approximate system of n equations, i.e.

ATA x = AT b

(1.34)

where the left-hand side matrix ATA is symmetry and the standard deviation
can be calculated from the Euclidean norm of e, i.e.:

S
=
mn

mn

(1.35)

Chapter 1

Systems of Linear Equations / 20

Example 1.12

Calculate the best approximate solution for the following system:

2 x1

x2

3 x3

4 x1
2 x1

+
+

4 x2
5x2

+
+

7 x3
9 x3

=
=

1
3

5 x1
7 x1

+ 5x2
+ 10 x 2

+ 9 x3
+ 15 x3

= 2
= 4

Also, calculate the resulting standard deviation.

Solution

The above system can be rewritted in form of A x = b as:

2 1 3
1
4 4 7 x

1 1
2 5 9 x 2 = 3

5 5 9 x 3 2
7 10 15
4

By using Eq. (1.34):

2 1 3

2 4 2 5 7 4 4 7 x1
1 4 5 5 10 2 5 9 x =
2

3 7 9 9 15 5 5 9 x 3

7 10 15

98
123

202

123
167
271

1

2 4 2 5 7 1
1 4 5 5 10 3

3 7 9 9 15 2

4

202 x1 50

271 x 2 = 70
445 x3 100

where its solutions are

x1 = 0.34930 ,

x 2 = 0.01996 ,

x 3 = 0.42914 .

Chapter 1

Systems of Linear Equations / 21

The standard deviation can be obtained from the Euclidean norm of the
error e:
1 2 1 3
1 4 4 7 0.34930

e = 3 2 5 9 0.01996 =
2 5 5 9 0.42914

4 7 10 15

0.43114
0.52695

0.06387
0.01597

0.20758

= 0.43114 2 + ( 0.52695 )2 + ( 0.06387 )2 + ( 0.01597 )2 + 0.20758 2 ,

= 0.71483 .

Therefore,

0.71483
53

= 0.50546

Chapter 1

Systems of Linear Equations / 22

Exercises
1.

Consider the following system:

1
0

1
a.
b.
c.
2.

2
1
1
1

4 8 x1 1.2
2 3 x2 0.2
=

4 12 x3 0.8

1 1 x4 1.5

Use the Gauss elimination method to obtain the solution of xi.

Calculate the determinant for the left-hand side matrix.
Generate the lower and upper triangular matrices using the Doolittle factorisation.

Consider the following system of 2 complex equations:

2 + 2i 1 + 2i z1 1 4i
=

3i
3 2i z 2 2 + 4i

By writing z k = x k + y k i , solve the equation using the Gauss-Siedel iteration method

using Microsoft Excel until it converges up to 5 decimal points.
3.

Consider the following set of redundant equations:

3x1

2 x2

x3

x1
x1

3x 2
+
x2

x3
x3

=
=

5
5

x3

= 2
=
2

2 x1
2 x1

a.
b.

x2
x2

Derive an approximate system of linear equations and solve it via the Gauss
elimination.
Calculate the corresponding standard deviation.