Math (P)refresher Lecture 6:

Linear Algebra I
September 2006

Todays Topics : Working with Vectors Linear Independence Matrix Algebra Square
Matrices Systems of Linear Equations Method of Substitution Gaussian Elimination GaussJordan Elimination

Working with Vectors

Vector: A vector in n-space is an ordered list of n numbers. These numbers can be represented as either a row vector or a column vector:

v1
v2

v = v1 v2 . . . vn , v = .
..
vn
We can also think of a vector as defining a point in n-dimensional space, usually Rn ; each
element of the vector defines the coordinate of the point in a particular direction.
Vector Addition: Vector addition is defined for two vectors u and v iff they have the same
number of elements:

u + v = u1 + v1 u2 + v2 uk + vn
Scalar Multiplication: The product of a scalar c and vector v is:

cv = cv1 cv2 . . . cvn

Vector Inner Product: The inner product (also called the dot product or scalar product)
of two vectors u and v is again defined iff they have the same number of elements
u v = u1 v1 + u2 v2 + + un vn =

n
X

ui vi

i=1

If u v = 0, the two vectors are orthogonal (or perpendicular).

Vector Norm: The norm of a vector is a measure of its length. There are many different
norms, the most common of which is the Euclidean norm (which corresponds to our usual
conception of distance in three-dimensional space):
||v|| =

v v = v1 v1 + v2 v2 + + vn vn

Much of the material and examples for this lecture are taken from Gill (2006) Essential Mathematics for Political
and Social Scientists, Simon & Blume (1994) Mathematics for Economists and Kolman (1993) Introductory Linear
Algebra with Applications.

Math (P)refresher: Linear Algebra I

Linear Dependence
Linear combinations: The vector u is a linear combination of the vectors v1 , v2 , , vk if
u = c1 v1 + c2 v2 + + ck vk
Linear independence: A set of vectors v1 , v2 , , vk is linearly independent if the only
solution to the equation
c1 v1 + c2 v2 + + ck vk = 0
is c1 = c2 = = ck = 0. If another solution exists, the set of vectors is linearly dependent.
A set S of vectors is linearly dependent iff at least one of the vectors in S can be written as
a linear combination of the other vectors in S.
Linear independence is only defined for sets of vectors with the same number of elements;
any linearly independent set of vectors in n-space contains at most n vectors.
Exercises: Are the following sets of vectors linearly independent?
1.

2.

1
1
1
v1 = 0 , v2 = 0 , v3 = 1
0
1
1

3
2
2
v1 = 2 , v2 = 2 , v3 = 3
1
4
1

Matrix Algebra
Matrix: A matrix is an array of mn real numbers arranged in m rows by n columns.

a11 a12 a1n

a21 a22 a2n

A= .
..
..
.
.
.
.
.
.
.
am1 am2

amn

Note that you can think of vectors as special cases of matrices; a column vector of length k
is a k 1 matrix, while a row vector of the same length is a 1 k matrix. You can also think
of larger matrices as being made up of a collection of row or column vectors. For example,

A = a1 a2

am

Matrix Addition: Let A and B be two m n matrices. Then

a11 + b11
a12 + b12 a1n + b1n
a21 + b21
a22 + b22 a2n + b2n

A+B=

..
..
..
.
.

.
.
.
.
am1 + bm1 am2 + bm2 amn + bmn

Math (P)refresher: Linear Algebra I

Note that matrices A and B must be the same size, in which case they are conformable for
addition.
Example:

1 2 3
A=
,
4 5 6

1 2 1
B=
2 1 2

2 4 4
A+B=
6 6 8

Scalar Multiplication: Given the scalar s, the scalar multiplication of sA is

sa11 sa12 sa1n

a11 a12 a1n
a21 a22 a2n sa21 sa22 sa2n

sA = s .
..
..
..
.. = ..
..
..

..
.
.
.
.
.
.
.
sam1 sam2 samn
am1 am2 amn
Example:

1 2 3
s = 2,
A=
4 5 6

2 4 6
sA =
8 10 12

Matrix Multiplication: If A is an mk matrix and B is a k n matrix, then their product

C = AB is the m n matrix where
cij = ai1 b1j + ai2 b2j + + aik bkj
Examples:

a b
aA + bC aB + bD
A B
1. c d
= cA + dC cB + dD
C D
e f
eA + f C eB + f D

2 5

1 2 1
1(2) + 2(4) 1(2) 1(5) + 2(3) 1(1)
4 2

4 3 =
2.
=
3 1 4
3(2) + 1(4) + 4(2) 3(5) + 1(3) + 4(1)
6 16
2
1
Note that the number of columns of the first matrix must equal the number of rows of the
second matrix, in which case they are conformable for multiplication. The sizes of the
matrices (including the resulting product) must be
(m k)(k n) = (m n)
Laws of Matrix Algebra:
1. Associative:

(A + B) + C = A + (B + C)
(AB)C = A(BC)

2. Commutative:

A+B=B+A

3. Distributive:

A(B + C) = AB + AC
(A + B)C = AC + BC

Math (P)refresher: Linear Algebra I

Commutative law for multiplication does not hold the order of multiplication matters:
AB 6= BA
Example:

1 2
A=
,
1 3

2 3
AB =
,
2 2

2 1
B=
0 1

1 7
BA =
1 3

Transpose: The transpose of the mn matrix A is the nm matrix AT (sometimes written

A0 ) obtained by interchanging the rows and columns of A.
Examples:

4
0
AT = 2 5
3 1

4 2 3
1. A =
,
0 5 1

2
2. B = 1 ,
3

BT = 2 1 3

The following rules apply for transposed matrices:

1. (A + B)T = AT + BT
2. (AT )T = A
3. (sA)T = sAT
4. (AB)T = BT AT
Example of (AB)T = BT AT :

1 3 2
A=
,
2 1 3

0 1
B = 2 2
3 1

0 1

1 3 2
12 7
T

2 2
(AB) =
=
2 1 3
5 3
3 1

1 2
12 7
0 2 3
T T

3 1 =
B A =
5 3
1 2 1
2 3

Square Matrices
Square matrices have the same number of rows and columns; a k k square matrix is referred
to as a matrix of order k.
The diagonal of a square matrix is the vector of matrix elements that have the same subscripts. If A is a square matrix of order k, then its diagonal is [a11 , a22 , . . . , akk ]0 .

Math (P)refresher: Linear Algebra I

Trace: The trace of a square matrix A is the sum of the diagonal elements:
tr(A) = a11 + a22 + + akk
Properties of the trace operator: If A and B are square matrices of order k, then
1.
2.
3.
4.

tr(A + B) = tr(A) + tr(B)

tr(AT ) = tr(A)
tr(sA) = str(A)
tr(AB) = tr(BA)

There are several important types of square matrix:

1. Symmetric Matrix: A matrix A is symmetric if A
for all i and j.
Examples:

4
1 2
A=
= A0 ,
B= 2
2 1
1

= A0 ; this implies that aij = aji

2. Diagonal Matrix: A matrix A is diagonal if all of

formally, if aij = 0 for all i 6= j
Examples:

4
1 0

A=
,
B= 0
0 2
0

its non-diagonal entries are zero;

2 1
1 3 = B0
3 1

0 0
1 0
0 1

3. Triangular Matrix: A matrix is triangular one of two cases. If all entries below the
diagonal are zero (aij = 0 for all i > j), it is upper triangular. Conversely, if all entries
above the diagonal are zero (aij = 0 for all i < j), it is lower triangular.
Examples:

1 0 0
1 7 4
ALT = 4 2 0 ,
AU T = 0 3 9
3 2 5
0 0 3
4. Identity Matrix: The n n identity matrix In is the matrix whose diagonal elements
are 1 and all off-diagonal elements are 0. Examples:

1 0 0
1 0
I2 =
,
I3 = 0 1 0
0 1
0 0 1

Linear Equations
Linear Equation: a1 x1 + a2 x2 + + an xn = b
ai are parameters or coefficients. xi are variables or unknowns.
Linear because only one variable per term and degree is at most 1.
1. R2 : line

x2 =

2. R3 : plane
3. Rn : hyperplane

x3 =

b
a2
b
a3

a1
a2 x 1
a1
a3 x 1

a2
a3 x 2

Math (P)refresher: Linear Algebra I

Systems of Linear Equations

Often interested in solving linear systems like
x 3y = 3
2x + y = 8
More generally, we might have a system of m equations in n unknowns
a11 x1
a21 x1
..
.

+
+

a12 x2
a22 x2

+
+
..
.

am1 x1 + am2 x2 +

+
+

a1n xn
a2n xn

=
=
..
.

b1
b2

+ amn xn = bm

A solution to a linear system of m equations in n unknowns is a set of n numbers x1 , x2 , , xn

that satisfy each of the m equations.
1. R2 : intersection of the lines.
2. R3 : intersection of the planes.
3. Rn : intersection of the hyperplanes.
Example: x = 3 and y = 2 is the solution to the above 2 2 linear system. Notice from the
graph that the two lines intersect at (3, 2).
Does a linear system have one, no, or multiple solutions?
For a system of 2 equations in 2 unknowns (i.e., two lines):
1. One solution:

The lines intersect at exactly one point.

2. No solution:

The lines are parallel.

3. Infinite solutions:

The lines coincide.

Methods to solve linear systems:

1. Substitution
2. Elimination of variables
3. Matrix methods

Method of Substitution
Procedure:
1. Solve one equation for one variable, say x1 , in terms of the other variables in the equation.
2. Substitute the expression for x1 into the other m1 equations, resulting in a new system
of m 1 equations in n 1 unknowns.
3. Repeat steps 1 and 2 until one equation in one unknown, say xn . We now have a value
for xn .

Math (P)refresher: Linear Algebra I

4. Backward substitution: Substitute xn into the previous equation (which should be a

function of only xn ). Repeat this, using the successive expressions of each variable in
terms of the other variables, to find the values of all xi s.
Exercises:
1. Using substitution, solve:
x 3y = 3
2x + y = 8
2. Using substitution, solve
x + 2y + 3z = 6
2x 3y + 2z = 14
3x + y z = 2

Elementary Equation Operations

Elementary equation operations are used to transform the equations of a linear system, while
maintaining an equivalent linear system equivalent in the sense that the same values of
xj solve both the original and transformed systems. These operations are
1. Interchanging two equations,
2. Multiplying two sides of an equation by a constant, and
3. Adding equations to each other
Interchanging Equations: Given the linear system
a11 x1 + a12 x2 = b1
a21 x1 + a22 x2 = b2
we can interchange its equations, resulting in the equivalent linear system
a21 x1 + a22 x2 = b2
a11 x1 + a12 x2 = b1
Multiplying by a Constant: Suppose we had the following equation:
2=2
If we multiply each side of the equation by some number, say 4, we still have an equality:
2(4) = 2(4)

8=8

More generally, we can multiply both sides of any equation by a constant and maintain an
equivalent equation. For example, the following two equations are equivalent:
a11 x1 + a12 x2 = b1
ca11 x1 + ca12 x2 = cb1

Math (P)refresher: Linear Algebra I

Adding Equations: Suppose we had the following two very simple equations:
3 = 3
7 = 7
If we add these two equations to each other, we get
7+3=7+3

10 = 10

Suppose we now have

a = b
c = d
If we add these two equations to each other, we get
a+c=b+d
Extending this, suppose we had the linear system
a11 x1 + a12 x2 = b1
a21 x1 + a22 x2 = b2
If we add these two equations to each other, we get
(a11 + a21 )x1 + (a12 + a22 )x2 = b1 + b2

Method of Gaussian Elimination

Gaussian Elimination is a method by which we start with some linear system of m equations
in n unknowns and use the elementary equation operations to eliminate variables, until we
arrive at an equivalent system of the form
a011 x1

a012 x2

a013 x3

a01n xn

b01

a022 x2

a023 x3

a02n xn

b02

a033 x3

+
..
.

a03n xn

= b03
..
.
= b0m

a0mn xn

where a0ij denotes the coefficient of the jth unknown in the ith equation after the above
transformation. Note that at each stage of the elimination process, we want to change some
coefficient of our system to 0 by adding a multiple of an earlier equation to the given equation.
The coefficients a011 , a022 , etc in boxes are referred to as pivots, since they are the terms
used to eliminate the variables in the rows below them in their respective columns. Once the
linear system is in the above reduced form, we then use back substitution to find the values
of the xj s.

As well see, pivots dont need to be on the ij, i = j diagonal. Additionally, sometimes when we pivot, we will
eliminate variables in rows above a pivot.

Math (P)refresher: Linear Algebra I

Exercises:
1. Using Gaussian elimination, solve
x 3y = 3
2x + y = 8

2. Using Gaussian elimination, solve

x + 2y + 3z = 6
2x 3y + 2z = 14
3x + y z = 2

10

Method of Gauss-Jordan Elimination

The method of Gauss-Jordan elimination takes the Gaussian elimination method one
step further. Once the linear system is in the reduced form shown in the preceding section,
elementary row operations and Gaussian elimination are used to
1. Change the coefficient of the pivot term in each equation to 1 and
2. Eliminate all terms above each pivot in its column,
resulting in a reduced, equivalent system. For a system of m equations in m unknowns, a
typical reduced system would be
x1

=
=
=
..
.

x2
x3
..

.
xn

b1
b2
b3

= bm

which needs no further work to solve for the xj s.

Exercises:
1. Using Gauss-Jordan elimination, solve
x 3y = 3
2x + y = 8

2. Using Gauss-Jordan elimination, solve

x + 2y + 3z = 6
2x 3y + 2z = 14
3x + y z = 2