NA1 Lecture Chap.01-06-Paged

Numerical Analysis using Maple and Matlab
Dr. Seongjai Kim

Professor of Mathematics
Department of Mathematics and Statistics

Mississippi State University
Mississippi State, MS 39762
skim@math.msstate.edu
1
Contents
MA-4313/6313: Numerical Analysis I
Ch.1: Mathematical Preliminaries

Ch.2: Solutions of Equations in One Variable
Ch.3: Interpolation and Polynomial Approximation
Ch.4: Numerical Differential and Integration
Ch.5: Initial-Value Problems for Ordinary Differential Equations
Ch.6: Direct Methods for Solving Linear Systems
MA-4323/6323: Numerical Analysis II
Ch.7: Iterative Alebraic Solvers

Ch.8: Approximation Theory
Ch.9: Approximating Eigenvalues
Ch.10: Numerical Solution of Nonlinear System of Equations
Ch.11: Boundary-Value Problems of One Variable
Ch.12: Numerical Solutions to Partial Differential Equations
2
1. Mathematical Preliminaries
In This Chapter:
Topics Applications/Properties
Review of Calculus Continuity & Differentiability
Intermediate Value Theorem
Mean Value Theorem
Taylor's theorem
Computer Arithmetic
Convergence
Order/rate of convergence and
Review of Linear Algebra Vectors and matrices
Norm
Determinant
Eigenvalues and eigenvectors
System of linear equations Matrix inversion
Elementary row operations LU factorization
System of tridiagonal matrices
Diagonally dominant matrices
Software Maple and Matlab
3
1.1. Review of Calculus
Continuity
Definition: A function is continuous at if
in other words, if for every there exists a such that

, for all x such that .
Examples and Discontinuities:
Definition: Let be an infinite sequence of real numbers. This

sequence has the limit (converges to ), if for every there exists a
positive integer such that whenever . The notation
or , as
means that the sequence converges to .
4
Theorem:
If is a function defined on a set of real numbers and , the following
are equivalent:
is continuous at .
If is any sequence in converging to , then
Differentiability
Definition: Let be a function defined on an open interval containing . The

function is differentiable at , if
exists. The number is called the derivative of at .
Important theorems for continuous/differentiable functions
Theorem:
If the function is differentiable at , then is continuous at .
Note: The converse is not true.
Example:
5
Intermediate Value Theorem (IVT):
Suppose and is a number between and . Then, there
exists a number for which .
Example: Show that has a solution in the interval .

Solution:
Define
Then, is continuous on . In addition,
=
=1
Thus the IVT implies that there is a number such that .
Rolle's Theorem:
Suppose and is differentiable on . If , then
there exists a number such that .
Example:
6
Mean Value Theorem (MVT):
Suppose and is differentiable on . Then, there exists a
number such that
,
which can be equivalently written as
.
Example: Let be defined on . Find which assigns the

average slope.
Solution, using Maple.
=
= 1.454648713
(7.2)
= 1.098818559
7
3
2
f(x)
L(x)
1 Average slope
0
0 1 2
x
Extreme Value Theorem:

If , then exist with for all
. In addition, if is differentiable on , then the numbers
and occur either at the endpoints of or where is zero.
Example: Find the absolute minimum and absolute maximum values of

on the interval .
Solution
=
=
8
Now, find the derivative of .
(9.1)
10
5
f(x)
f'(x)
0
2
x
1.358229874 (9.2)
(9.3)
=
=
9
The following theorem can be derived by applying Rolle's Theorem
successively to and finally to .
Generalized Rolle's Theorem:

Suppose is times differentiable on . If at the
distinct points , then there exists a
number such that .
Integration
Definition: The Riemann integral of a function on the interval is the

following limit, provided it exists:
where , with and x arbitrarily

chosen in the subinterval .
Continuous functions are Riemann integrable, which allows us to choose, for

computational convenience, the points to be equally spaced in and
choose , where . In this case,
10
Fundamental Theorem of Calculus:
Let be continuous on . Then,
Part I: .
Part II: , where is any antiderivative of , i.e., a
function such that .
Weighted Mean Value Theorem for Integrals

Suppose , the Riemann integral of exists on , and
does not change sign on . Then, there exists a number
such that
When , it becomes the usual Mean Value Theorem for Integrals,

which gives the average value of over the interval :
11
Taylor's Theorem
Taylor's Theorem with Lagrange Remainder:
Suppose , exists on , and . Then, for
every ,
where, for some between and ,
Note that is a polynomial of degree .
Example: Let and . Determine the second and third

Taylor polynomials for about .
Solution:
12
=
=
On the other hand, you can find the Taylor polynomials easily with Maple:
=
f(x)
0 1 2 p3(x)
x
Frequently used Taylor Series:
=
=
13
=
Note: When , and , the Taylor's Theorem reads

,
which is the Mean Value Theorem.
Taylor's Theorem with Integral Remainder:

Suppose and . Then, for every ,
where
14
Alternative Form of Taylor's Theorem:
Suppose and exists on Then, for every
,
where, for some between and ,
.
In detail,
Example: Determine Taylor's formula for and approximate .

Solution:
Let . Then, since , we have
Let . That is,
15
= 10.002302850208
= 10.002302850208247527
Taylor's Theorem for Two Variables:

Let . If and are points in
, then
,
where
in which lies between 0 and 1.
For , the Taylor's theorem for two variables reads

(1)
where . Equation (1), as a linear approximation or

tangent plane approximation, will be used for various applications.
16
Example: Find the tangent plane approximation of
at .
Solution:
3 (19.1)
2 (19.2)
(19.3)
Thus the tangent plane approximation at is
.
17
Empty Page
18
1.2. Computer Arithmetic and Convergence
Errors in Machine Numbers and Computational Results:

Numbers are saved with an approximation by either rounding or
chopping.
integer: in 4 bites (32 bits)
float: in 4 bites
double: in 8 bites (64 bits)
Computations can be carried out only for finite sizes of data points.
Example:
=
= 3.141592654
= 3.1415927
= 3.141592653589793
=
On the other hand, = 0.
Definition: Suppose that is an approximation to .

The absolute error is , and
the relative error is , provided that .
Definition: The number is said to approximate to -significant digits (or

figures) if is the largest nonnegative integer for which
19
Computational Algorithms
Definition: An algorithm is a procedure that describes, in an unambiguous

manner, a finite sequence of steps to be carried out in a specific order.
Algorithms consist of various steps for inputs, outputs, and functional

operations, which can be described effectively by a so-called pseudocode.
Definition: An algorithm is called stable, if small changes in the initial data

produce correspondingly small changes in the final results. Otherwise, it is
called unstable. Some algorithms are stable only for certain choices of initial
data/parameters, and are called conditionally stable.
Growth rates of the error:

Suppose that denotes an error introduced at some stage in the
computation and represents the magnitude of the error after subsequent
operations.
If , where is a constant independent of , then the growth
of error is said to be linear, for which the algorithm is stable.
If , for some , then the growth of error is exponential,
which turns out unstable.
20
Rates (Orders) of Convergence
Let be a sequence of real numbers tending to a limit .

Definition: The rate of convergence is at least linear if there are a constant
and an integer such that
, for all .
We say that the rate of convergence is at least superlinear if there exist a
sequence tending to 0 and an integer such that
, for all .
The rate of convergence is at least quadratic if there exist a constant (not
necessarily less than 1) and an integer such that
, for all .
In general, we say that the rate of convergence is of at least if there exist a
constant (not necessarily less than 1 for ) and an integer such that
, for all .
Example: Consider a sequence defined recursively as
(a) Find the limit of the sequence and (b) show that the convergence is
quadratic.
21
Big and Little Notation
Definition: A sequence is said to be in (big Oh) of if a
positive number exists for which
, for large .
In this case, we say is in , and denote or .
Definition: A sequence is said to be in (little oh) of if there

exists a sequence tending to 0 such that
, for large , (or equivalently, ).
In this case, we say is in , and denote or .
Example: Show that and
22
Definition: Suppose . A quantity is said to be in (big
Oh) of (h) if a positive number exists for which
, for sufficiently small.
In this case, we say is in , and denote .
Little oh of (h) can be defined the same way as for sequences.
Example:
which implies that is in
.
Note that , for
sufficiently small .
By the way,
.
Example: Choose the correct assertions (in each, )

a.
b. (n+1)/
c.
d.
e.
23
Example: Determine the best integer value of in the following equation
, as .
Ans:
Self study: Let . What are the limit and the rate of
convergence of as ?
Ans: and , i.e., as .
Self study: Show that these assertions are not true.

a. as
b. as
c. as
24
Example: Let and let . Show that as .
Hint
. Then, we have to show
as . For this, you can first get .
Since is bounded, if , then , which implies

.
25
Empty Page
26
1.3. Review of Linear Algebra
Vectors
A real -dimensional vector is an ordered set of real numbers and is

usually written in the coordinate form
Definitions: Let x and y be -dimensional vectors.

Liner combination:
Norm (length): , which is often referred

to as the Euclidean norm, or Euclidean norm.
Distance:
Dot product: . Thus .
Let be the angle between the vectors x and y. Then,

.
27
Matrices and Two-dimensional Arrays
A matrix is a rectangular array of numbers that is arranged systematically in

rows and columns. A matrix having rows and columns is called an
matrix, and is denoted as . When the capital letter
represents a matrix, the lowercase subscribed letter denotes the -th
entry of the matrix:
.
When matrices and have the same dimensions, their linear combination
can be defined as
.
Properties of Vectors and Matrices
Definition: If and are two matrices with the property

that has as many columns as has rows, the matrix product is defined
to be the matrix :
,
where is given as the dot product of the -th row of and the -th column of
:
28
Example: Find the matrix product of
and .
For the above example, may not be defined. (Why?)
If both and are square matrices of the same dimension, then and are
defined but they are in general not the same: . When it happens that
, we say that and commute.
Example: Compute and compare and .
Solution: Using Maple, we have
They are not the same, so the matrices do not commute.
29
Definition: determinant (det) of
If , we define .
For , let (minor of ) be the determinant of the
submatrix of obtained by deleting the -th row and the
-th column of . Define the cofactor of as . Then the
determinant of is given by
(he -th row cofactor expansion)

or
(he -th column cofactor expansion)
Example: Then, using the first row cofactor expansion,
Example: Let Find its determinant.
Solution:
Then, again using the first row cofactor expansion,
= 39 # which is determinant of the above

30
submatrix.
= 39
=
= 29
=
=
Thus, = = 77
You may get it directly using a Maple command:

= 77
31
Example: Find the determinant of the following matrices, if it exists.
a.
b.
c.
d.
32
Eigenvalues and Eigenvectors
Definition: An eigenvector of an matrix is a nonzero vector x such

that
,
eigenvalue of corresponding to x.
Hence, the eigenvector is a nontrivial solution of , which

implies that is singular. The eigenvalues of are solutions of
.
Example: Find eigenvalues and eigenvectors of
Ans:
33
Invertible (nonsingular) Matrices
Let .
Definition: The matrix is invertible if there is an matrix such that
. The matrix is called an inverse of , and is denoted as .
Definition: The transpose of is . The matrix is symmetric if
.
Theorem: A square matrix can possess at most one right inverse.
Theorem: Let and be invertible square matrices. Then,
Theorem: If and are square matrices such that , then .

Proof:
Let . Then
.
By the uniqueness of the right inverse, we can conclude , which
implies .
34
Invertible (Nonsingular) Matrix Theorem:
For , the following properties are equivalent:
1. The inverse of exists, i.e., is invertible.
2. The determinant of is nonzero.
3. The rows of form a basis for .
4. The columns of form a basis for .
5. As a map from to , is injective (one-to-one).
6. As a map from to , is surjective (onto).
7. The equation implies .
8. For each , there is exactly one such that .
9. is a product of elementary matrices.
10. 0 is not an eigenvalue of .
35
System of Linear Equations
Linear systems of equations of unknowns can be expressed as the

algebraic system:
,
where u denotes the vector of unknowns, b represents the source terms (the
right side), and is an matrix of real values, i.e.,
The above algebraic system can be solved by the elementary row operations
applied to the augmented system:
36
Elementary Row Operations:
1. (Replacement) Replace one row by the sum of itself and a multiple of
another row:
2. (Interchange) Interchange two rows:
3. (Scaling) Multiply all entries in a row by a nonzero constant:
Example: Solve the following system of equations:
Solution:
37
The following row operations (Gauss Elimination) solve the problem:
Forward elimination:
Back substitution:
Or, you may utilize the function "ReducedRowEchelonForm":
=
38
Thus the solution =
Note: Every elementary row operation can be achieved by multiplying the

augmented matrix on the left by an elementary matrix. For example, the
replacements and can be represented
respectively by and :
Then, the upper-triangular system in the last example can be obtained as
The last result is called an echelon form, and its diagonals are called pivots.
39
Remarks:
a. The elementary matrices for replacement row operations commute
= =
b. Their product is made by putting the entries below the main diagonal.
c. Their inverse can be obtained by negating the entries below the main
diagonal.
= = =
Self study: Find elementary matrices in for the row operations:

a.
b.
c.
d.
40
Example: Find the parabola that passes through (1,2), (2,
4), and (3,8).
Use the Gauss Elimination to solve the algebraic system for the unknowns
.
41
System of Tridiagonal Matrices
Such a matrix can be saved in an array:
and row operations can be applied correspondingly.
Example: Use Gauss Elimination to solve the tridiagonal system of 5

equations, , of which the coefficient matrix is given in a array for
nonzero entries.
42
Solution: (The underlined numbers are pivots.)
# The last column reads the unknown u.
43
LU Factorization (Triangular Factorization)
Definition: A nonsingular matrix has an LU factorization if it can be

expressed as the product of a lower-triangular matrix an upper-triangular
matrix :
.
In matrix form, this is written as
The condition that is nonsingular implies that .
With LU factorization, is written as
Thus, the system can be solved by a two-step procedure:
When the matrix is LU-factorizable, the factorization can be achieved by a

repetitive application of replacement row operations, as in the procedure of
echelon form. Here the difference is to place inverses of the elementary
matrices on the left.
Let be elementary matrices (for replacement row operations)
such that
is an echelon form of . Then,
where is a lower-triangular matrix and

is upper-triangular.
44
Thus, the LU factorization can be carried out step-by-step as in
Example: Find the LU factorization of
Solution:
Let
Then,
Let Then, = and
45
which completes the LU factorization.
Using Maple:
= =
= # a permutation matrix, implying no pivoting
required.
Here, .
Alternatively, you may use:

=
46
Example: Use replacement row operations to find the LU factorization.
a.
b.
47
Diagonally Dominant Matrices
Definition: A matrix is diagonally dominant if
, for all .
The matrix is strictly diagonally dominant, if the above inequalities are strict
for all .
Theorem on Preserving Strict Diagonal Dominance:

Theorem: Let be strictly diagonally dominant. Then, Gauss
Elimination without pivoting preserves the strict diagonal dominance of the
matric.
Corollary: Thus, the work of finding pivots is not required for the LU
factorization of a strictly diagonally dominant matrix, provided that the
rows of the matrix is linearly independent (i.e., invertible).
Corollary: Every strictly diagonally dominant matrix is nonsingular and
has an LU factorization.
Example: Verify the preservation of diagonal dominance of
Solution:
> for k from 1 by 1 to (n-1) do

48
>
m:=B(k+1,k)/B(k,k);
B:=RowOperation(B,[k+1,k],-m);
end do;
49
(12.1)
> All intermediate matrices are diagonally dominant!
50
Norms and Error Analysis
Definition: On a vector space , a norm is a function from to the set of

nonnegative real numbers that obeys the following three postulates:
, if
, if
, if (triangle inequality)
Examples of norms:
Example: Let .
Euclidean -norm:
The -norm:
The -norm:
Definition: If a vector norm has been specified, the matrix norm

subordinate to (associated with) it is defined by
It is equivalent to
51
Matrix norms:
1.
2.
3.
4. , where denotes the spectral radius of .
5.
6.
Definition: A condition number of a matrix is the number
Example: Let , for which and
1. Find , , and .
2. Find the -condition number.
52
Theorem on Neumann Series: If is an matrix such that for
any subordinate matrix norm, then is invertible and
53
Empty Page
54
Homework
1. Review of Calculus, Convergence, and Linear Algebra
#1. Prove that the following equations have at least one solution in the
given intervals.
a.
b.
c.
d.
#2. Let and .

a. Find the third Taylor polynomial about , , and use it to
approximate .
b. Use the Taylor Theorem to find an upper bound for the error
. Compare it with the actual error.
c. Find the fifth Taylor polynomial about , , and use it to
approximate .
d. Use the Taylor Theorem to find an upper bound for the error
. Compare it with the actual error.
#3. For the fair is it true that as ?
a.
b.
c.
d.
55
#4. Let a sequence be defined recursively by , where is
continuously differentiable. Suppose that as and .

Show that
Hint: Begin with
and use the Mean Value Theorem and the fact that is continuously
differentiable, to show that the quotient converges to zero.
#5. A square matrix is said to be skew-symmetric if .

Prove that if is skew-symmetric, then for all . (Hint: The
quantity is scalar so that .)
#6. Suppose that and are square matrices and that is invertible.
Show that each of , and is invertible.
#7. Find the determinant and eigenvalues of the following matrices, if it

exists. Compare the determinant with the product of eigenvalues.
a.
b.
c.
56
d.
#8. Find the LU factorization of the matrices:
a.
b.
#9. Show that satisfies the following three conditions:

, if
, if
, if (triangle inequality)
Hint: For the last condition, you may begin with
.
#10. Show that for all .
57
Empty Page
58
2. Solutions of Equations in One Variable
Objective:
For equations of the form
,
find solutions, that are real numbers such that .
Note: Various nonlinear systems of equations and discretization of

nonlinear PDEs can be expressed by
,
which is equivalently written as
.
In This Chapter:
Bisection method
Fixed-point iteration
Newton's method
Secant method Variant of Newton's method
Method of false position
Zeros of polynomials Application of Newton's method
Horner's method Effective evaluation of polynomials
Bairstow's method Quadratic factors
59
2.1. The Bisection (Binary-Search, Interval Halving) Method
Assumptions:
is continuous in .
0 By IVT, there must be a solution.
There is a single solution in .
Bisection: a pseudocode
p=pi, and stop;

else
if( f(ai)*f(pi)<0 ) then
a(i+1)=ai;
b(i+1)=pi;
else
a(i+1)=pi;
b(i+1)=bi;
end if
p(i+1)=( a(i+1)+b(i+1) )/2;
end if
60
Example: Find the solution of the equation in .
Solution: Using Maple
(3.1)
4 iteration(s) of the bisection method applied to

3 2
f x = x C 4 x K 10
with initial points a = 1.25 and b = 1.5
a p b
x
f x
61
(3.2)
62
Bisection: Maple code
>
>
>
k= 1: a= 1.000000 b= 2.000000 p= 1.500000 f(p)= 2.375000
k= 2: a= 1.000000 b= 1.500000 p= 1.250000 f(p)=-1.796875
63
k= 3: a= 1.250000 b= 1.500000 p= 1.375000 f(p)= 0.162109
k= 4: a= 1.250000 b= 1.375000 p= 1.312500 f(p)=-0.848389
k= 5: a= 1.312500 b= 1.375000 p= 1.343750 f(p)=-0.350983
k= 6: a= 1.343750 b= 1.375000 p= 1.359375 f(p)=-0.096409
k= 7: a= 1.359375 b= 1.375000 p= 1.367188 f(p)= 0.032356
p_7 = 1.367187500
dp = +- 0.007812 = (b0-a0)/2^k= 0.007812
f(p) = 0.032355785
175
(1)
128
64
Error Analysis:
Theorem: Suppose that and . Then, the
Bisection method generates a sequence approximating a zero of with
Proof. For every ,

and .
It follows from that
Example: Determine the number of iterations necessary to solve

with accuracy using and .
Solution:
We have to find the iteration count such the error bound is not larger than
. That is,
.
Thus, , which implies that = 9.965784285

and therefore .
65
Note: is the midpoint of and is the midpoint of either
of . So, . In other
words,
,
which implies that

.
The approximate solution carried out with the absolute difference

for the stopping criterion guarantees the actual error not greater
than the given tolerance.
Example: Suppose that the bisection method begins with the interval
. How many steps should be taken to compute a root with a relative
error not larger than ?
Solution:
. Thus,
66
Bisection: MATLAB code
M-file: bisect.m
function [c,err,fc]=bisect(f,a,b,TOL)
%Input - f is the function input as a string 'f'

% - a and b are the left and right endpoints
% - TOL is the tolerance
%Output - c is the zero
% - fc= f(c)
% - err is the error estimate for c
fa=feval(f,a);
fb=feval(f,b);
if fa*fb > 0,return,end
max1=1+round((log(b-a)-log(TOL))/log(2));
for k=1:max1
c=(a+b)/2;
fc=feval(f,c);
if fc==0
a=c;
b=c;
elseif fb*fc>0
b=c;
fb=fc;
else
a=c;
fa=fc;
end
if b-a < TOL, break,end
end
c=(a+b)/2;
err=abs(b-a);
fc=feval(f,c);
67
You can call the above algorithm with varying function, by
>> f = @(x) x.^3+4*x.^2-10;
>> [c,err,fc]=bisect(f,1,2,0.005)
c=
1.3652
err =
0.0039
fc =
7.2025e-005
Example: Consider the bisection method applied to find the zero of the
function with . What are ? What are
?
Answer:
k= 1: a= 0.000000 b= 1.000000 p= 0.500000 f(p)= 0.625000

k= 2: a= 0.500000 b= 1.000000 p= 0.750000 f(p)=-0.328125
k= 3: a= 0.500000 b= 0.750000 p= 0.625000 f(p)= 0.119141
68
Example: In the bisection method, does exist?
69
Empty Page
70
2.2. Fixed-Point Iteration
Definition: A number is a fixed point for a given function if .
Note: Given a root-finding problem , let

.
Then, since , the above defines a fixed-point problem.
Example: Find any fixed points of .

Answer:
(1.1)
(1.2)
=
Note:
(1.3)
(1.4)
=
(1.5)
71
The Fixed Point
y=g(x)
2 y=x
0 1 2 3
x
72
Theorem:
If and for all , then has at least
one fixed point in .
If, in addition, is differentiable in and there exists a positive
constant such that
for all ,
then there is a unique fixed point in .
Notes:
Example: Show that has a unique fixed point on .
73
Proof of the Theorem:
If g(a)=a or g(b)=b 0 g has a fixed point at an endpoint. If not, 0 g(a)>a
and g(b)<b. Define h(x)=g(x)-x. Then, h(a)>0 and h(b)<0. Thus, by the
IVT, there is p2(a,b) such that h(p)=0, which implies that g(p)=p.
In addition, suppose that for all Let p and q are
two fixed points;
, for some between p and q.
Thus
,
which is a contradiction.
74
Fixed-Point Iteration
Definition: A fixed-point iteration is an iterative procedure of the form: For a

given ,
for .
If the sequence converges to , since is continuous, we have

.
This implies that the limit of the sequence is a fixed point of , i.e., the
iteration converges to a fixed point.
Example: The equation has a unique root in . There

are many ways to change the equation to the fixed-point form :
a.
b. *
c.
d. *
e. *
f.
The associated (fixed-point) iteration may not converge for some choices of .
The real root of is .
75
Evaluation of and FPI
= 27
(4.1)
=3
(4.2)
(4.3)
76
= = 2.121320343
(4.4)
= = 0.1414213562
(4.5)
5
=
14
70
=
121
(4.6)
77
Fixed-Point Theorem:
Let be such that for all . Suppose
that is differentiable in and there exists a positive constant
such that
for all
Then, for any number , the sequence defined by
converges to the unique fixed point .
Proof:
It follows from the previous theorem that there exists a unique fixed point
, i.e., . Since for all ,
we have for all and, by the MVT,
,
for some . Therefore,
as .
78
Notes:
.
(Here we have used the MVT, for the last inequality.)
Thus,
That is,
defined on any
closed subset of =. By a contractive mapping, we mean a function that
satisfies for some
for all
Note: If a contractive mapping is differentiable, then the above implies

that
, for all .
79
In practice: is not known.
Consider the following:
Thus, we have
,
which is useful for stopping of the iteration.
80
Example: For each of the following equations, Determine an interval
on which the fixed-point iteration will converge. Estimate the number of
iterations necessary to obtain approximations accurate to within .
a.
b.
c.
d.
Solution:
Plots
2 4
1
3
0 1 2
x 2
2 3 4
x
y=g1(x) y=x y=g2(x) y=x
1
=
3
5
=
4
81
3 2
2
1
1
0 1 2
0 x
0 1 2 3
x
y=g3(x) y=x y=g4(x) y=x
82
Example: Prove that the sequence defined recursively as follows is
convergent.
Solution
Begin with setting , then show is a contractive mapping
on
83
Empty Page
84
2.3. Newton's (Newton-Raphson) Method
and Its Variants
Let be a zero of and an approximation of and

(1)
Our momentary concern is how to find the correction .
If exists and is continues, then by Taylor's Theorem
where lies between and . If is small, it is reasonable to ignore

the last term and solve for :
.
Then,
may be a better approximation of than .
This has motivated the Newton's Method:
85
Graphical Interpretation:
Consider the tangent line passing :
.
Let . Then,
which is , the -intercept of the tangent line .
Newton's method applied to

2
f x =x , with initial point p = 1.
0
p
x
f x Tangent lines
Example of Nonconvergence:
86
3 iterations of Newton's method applied to
f x = arctan x , with initial point
p = p/2
0
p
x
f x Tangent lines
Notes:
for some . As a matter of fact,

Newton's method is most effective when is bounded away from zero near
.
87
Convergence Analysis:
Let . Then,
By Taylor's Theorem, we have
.
Thus,
Theorem: Convergence of Newton's Method

Let and is such that and . Then,
there is a neighborhood of such that if Newton's method is started in
that neighborhood, it generates a convergent sequence satisfying
,
for a positive constant
88
Example:
(4.1)
Since , and
,
which is an occasional super-convergence.
Theorem on Newton's Method for a Convex Function:

Let be increasing, convex, and of a zero. Then, the zero is
unique and the Newton iteration will converge to it from any starting point.
Example: Use Newton's method to find the square root of a positive number
.
Solution:
Let . Then is a root of .
Set and .
The Newton's method reads
89
(6.1)
(6.2)
90
Implicit Functions
For a function implicitly defined as

,
if is prescribed, then the equation can be solved for using Newton's
method:
Example: Produce a table of , where is defined implicitly as a function

of . Use and start , proceeding in
steps of 0.1 to .
Solution:
91
x y F(x,y)
0.000000 1.000000 0
0.100000 0.997760 -9.434e-10
0.200000 0.991250 1.3577e-09
0.300000 0.980657 7.977e-10
0.400000 0.966019 -6.568e-10
0.500000 0.947227 3.970e-10
0.600000 0.924004 1.519e-10
0.700000 0.895854 2.77e-11
0.800000 0.861955 -1.774e-10
0.900000 0.820939 -5.029e-10
1.000000 0.770398 -2.217e-10
92
Systems of Nonlinear Equations:
Newton's method for systems of nonlinear equations follows the same strategy
that was used for single equation. That is, we linearized, solve for
corrections, and update the solution, repeating the steps as often as
necessary. For an illustration, we begin with a pair of equations involving two
variables:
Supposing that is an approximate solution of the system, let us

computer corrections so that will be a better
approximate solution.
The coefficient matrix combining linearly on the right of the above

is the Jacobian of at :
Hence, Newton's method for two nonlinear equations in two variables is
where
93
In general, the system of nonlinear equations,
,
can be expressed as
,
where and . Then
,
where and is the Jacobian of at :
The correction vector is obtained as

.
Hence, Newton's method for nonlinear equations in variables is given by
,
where is the solution of the linear system:
.
94
Example: Starting with , carry out 6 iterations of Newton's method
to find a root of the nonlinear system
Solution:
95
1 2.18932610 1.59847516 1.39390063 1.19 0.598 0.394
2 1.85058965 1.44425142 1.27822400 -0.339 -0.154 -0.116
3 1.78016120 1.42443598 1.23929244 -0.0704 -0.0198 -0.0389
4 1.77767471 1.42396093 1.23747382 -0.00249 -0.000475 -0.00182
5 1.77767192 1.42396060 1.23747112 -2.79e-006 -3.28e-007 -2.7e-006
6 1.77767192 1.42396060 1.23747112 -3.14e-012 -4.22e-014 -4.41e-012
96
The Secant Method
Newton's method, defined as
is a powerful technique; however it has a major drawback: the need to know

the value of derivative of at each iteration. Frequently, is far more
difficult to calculate than .
To overcome the disadvantage, a number of methods have been proposed.

One of most popular variants is the secant method, which replaces
by a difference quotient:
Thus, the resulting algorithm reads
Notes:
Two initial values must be given.
It requires only one new evaluation of per step.
The graphical interpretation of the secant method is similar to that of
Newton's method.
Convergence:
= 1.618033988
Graphical interpretation:
97
3 iteration(s) of the secant method applied
to
3
f x =x K1
b a
x
f x
(9.1.1)
Here, is the -intercept of the secant line joining
and .
Example: Apply one iteration of the secant method to find if

.
Solution:
= = 5.000000000
98
The Method of False Position:
It generates approximations in a similar manner as the secant method;

however, it includes a test to ensure that the root is always bracketed between
successive iterations.
Select and such that .

Compute = the -intercept of the line joining and
.
If ( ), then ( and bracket the root)
Choose = the -intercept of the line joining and
.
else
Choose = the -intercept of the line joining and
.
end if
99
3 iteration(s) of the method of false
position applied to
3
f x =x K1
b a
x
f x
(10.1)
Here, the root is bracketed in all iterations.
100
Comparison: Convergence Speed
Find a root for , starting with or .
(11.1)
(11.2)
(11.3)
101
n Newton Secant False Position
0 0.7853981635 0.5000000000 0.5000000000
1 0.7395361335 0.7853981635 0.7363841388
2 0.7390851781 0.7363841388 0.7390581392
3 0.7390851332 0.7390581392 0.7390848638
4 0.7390851332 0.7390851493 0.7390851305
5 0.7390851332 0.7390851332 0.7390851332
6 0.7390851332 0.7390851332 0.7390851332
7 0.7390851332 0.7390851332 0.7390851332
8 0.7390851332 0.7390851332 0.7390851332
102
2.4. Zeros of Polynomials
A polynomial of degree has a form
where 's are called the coefficients of and .
Theorem on Polynomials
Fundamental Theorem of Algebra: Every nonconstant polynomial has
at least one root (possibly, in the complex field).
Complex Roots of Polynomials: A polynomial of degree has exactly
roots in the complex plane, it being agreed that each root shall be
counted a number of times equal to its multiplicity. That is, there are
unique (complex) constants and unique integers
such that
Localization of Roots: All roots of the polynomial lie in the open disk
centered at the origin and of radius of
Uniqueness of Polynomials: Let and be polynomials of

degree . If , with , are distinct numbers with
for , then for all values of .
Particularly, two polynomials of degree are the same if they agree at
values.
103
Horner's Method
Known as nested multiplication and also as synthetic division, Horner's

method can evaluate polynomials very efficiently; it requires multiplications
and additions to evaluate an arbitrary th-degree polynomial.
Let us try to evaluate at . Then, utilizing the Remainder Theorem,

we first can rewrite the polynomial as
(1)
where is a polynomial of degree , say

.
Substituting the above into (1) and setting equal the coefficients of like powers
of on the two sides of the resulting equation, we have
which can be written
If the calculation of Horner's algorithm is to be carried out with pencil and

paper, the following arrangement is often used (known as synthetic division):
104
an an K1 an K2 ,,, a0
x
0
x0 bn x0 bn K1 ,,, x0 b1
bn bn K1 bn K2 ,,, P x0 = b0
Example: Use Horner's algorithm to evaluate , where
Solution
We arrange the calculation as mentioned above.
1 K4 7 K5 K2
3 3 K3 12 21
1 K1 4 7 19 = P 3
Thus, , and
Recall the Newton's method applied for finding an approximate zero of :
When the method is being used to find an approximate zero of a polynomial ,

both and must be evaluated at the same point in each iteration. The
derivative can be evaluated by using the Horner's method with the same
efficiency. Indeed, differentiating (1) reads
.
Thus
.
105
Example: Evaluate for considered in the previous example.
Solution
As in the previous example, we arrange the calculation and carry out the
synthetic division one more time:
1 K4 7 K5 K2
3 3 K3 12 21
1 K1 4 7 19 = P 3
3 3 6 30
1 2 10 37 = Q 3 = P' 3
Thus, .
Example: Implement the Horner's algorithm to evaluate and ,

where
.
Solution
# This algorithm is equivalent to the one on p.95.
106
= P(3)=19, P'(3)=37
The Maple command coeff can be used like
Horner's method will be presented in a more systematic fashion when we deal

with Polynomial Interpolation.
107
Complex Zeros: Finding Quadratic Factors
Quadratic Factors of Real-coefficient Polynomials:
Let .
Theorem on Real Quadratic Factor: If is a polynomial whose
coefficients are all real, and if is a nonreal root of , then z is also a
root and is a real quadratic factor of .
Polynomial Factorization: If is a nonconstant polynomial of real
coefficients, then it can be factorized as a multiple of linear and quadratic
polynomials of which coefficients are all real.
Theorem on Quotient and Remainder: If the polynomial is divided
by the quadratic polynomial , then the quotient and
remainder
can be computed recursively by setting and then using
108
Bairstow's Method
Bairstow's method seeks a real quadratic factor of of the form .

For simplicity, all the coefficients 's are real so that both and will be real.
In order for the quadratic polynomial to be a factor of , the remainder
must be zero. That is, the process seeks
Note that and must be functions of ( , which is clear from the last
theorem.
An outline of the process is as follows: Starting values are assigned to .

We seek corrections so that
Linearization of these equations reads
Thus, the corrections can be found by solving the linear system
109
Now, the question is how to compute the Jacobian matrix.
As first appeared in the appendix of the 1920 book "Applied Aerodynamics"
by Leonard Bairstow, we consider the partial derivatives
Differentiating the recurrence relation (the shaded, boldface equation in the

last theorem) results in the following pair of additional recurrences:
Note that these recurrence relations obviously generate the same two sequence
( ); we need only the first. The Jacobian explicitly reads
and therefore
We summarize the above procedure as in the following code:
110
Bairstow's algorithm:
111
1 2.2000000 -2.7000000 -0.8 1.3
2 2.2727075 -3.9509822 0.07271 -1.251
3 2.2720737 -3.6475280 -0.0006338 0.3035
4 2.2756100 -3.6274260 0.003536 0.0201
5 2.2756822 -3.6273651 7.215e-05 6.090e-05
6 2.2756822 -3.6273651 6.316e-09 -9.138e-09
7 2.2756822 -3.6273651 -1.083e-17 -5.260e-17
Q(x) = (1)x^2 + (-1.72432)x^1 + (-0.551364)
Remainder: -2.66446e-18 (x - (2.27568)) + (-2.47514e-16)
Quadratic Factor: x^2 - (2.27568)x - (-3.62737)
Zeros: 1.137841102 +- (1.527312251) i
112
Deflation
Given a polynomial of degree , , if the Newton's method finds a zero

(say, ), it will be written as
.
Then, we can find a second approximate zero (or, a quadratic factor) of
by applying Newton's method to the reduced polynomial ; the
computation continues up to the point that is factorized by linear and
quadratic factors. The procedure is called deflation.
The accuracy difficulty with deflation is due to the fact that when we obtain
the approximate zeros of , Newton's method is used on the reduced
polynomials . An approximate zero of will generally not
approximate a root of as well as a root of the reduced polynomial
, and inaccuracy increases as increases. One way to overcome this
difficulty is to (a) use the method of reduced equations to find approximate
zeros and then (b) improve these zeros by applying Newton's method to the
original polynomial .
113
Empty Page
114
Homework
2. Solutions of Equations in One Variable
#1. Let the bisection method be applied to a continuous function, resulting in intervals
. Let and . Which of these
statements can be false?
a.
b.
c.
d.
e. as
#2. Modify the provided Matlab code for the bisection method to incorporate
Consider the following equations defined on the given intervals:

I.
II.
For each of the above equations,

a. Use Maple to find the analytic solution on the interval.
b. Find the approximate root by using your Matlab with .
c. Report , for , in a table format.
115
#3. Let us try to find by sung fixed-point iterations. Use the fact that the result must
be the positive solution of to solve the following:
a. Introduce three different fixed-point forms of which at least one is convergent.
b. Rank the associated iterations based on their apparent speed of convergence for
.
c. Perform three iterations, if possible, on each of the iterations with , and measure
.
#4. Kepler's equation in astronomy reads

, with .
a. Show that for each , there exists an satisfying the equation.
b. Interpret this as a fixed-point problem.
c. Find 's for using the fixed-point iteration. Set .
(Hint: For (a), you may have to use the IVT for defined on , while for
(b) you should rearrange the equation in the form of . For (c), you may use any
source of program which utilizes the fixed-point iteration.)
#5. Consider a variation of Newton's Method in which only one derivative is needed; that
is,
Find and such that
(Hint: You may have to use )
#6. Starting with , carry out two iterations of Newton's method on the system:
116
#7. Consider the polynomial
a. Use Horner's algorithm to find .

b. Use the Newton's method to find a real-valued root, starting with and applying
Horner's algorithm for the evaluation of and
c. Apply Bairstow's method, with the initial point , to find a pair of
complex-valued zeros.
d. Find a disk centered at the origin that contains all the roots.
117
Empty Page
118
3. Curve Fitting:
Interpolation and Approximation
In This Chapter:
Polynomial interpolation The first step toward
approximation theory
Newton form
Lagrange form Basis functions for various
applications including FEM
Chebyshev polynomial Optimized interpolation
Divided differences
Neville's method Evaluation of interpolating
polynomials
Hermite interpolation Requires and
FEM for 4th-order PDEs
Spline interpolation Less oscillatory interpolation
B-splines
Parametric curves Curves in the plane or space
Rational interpolation Interpolation of rough data
with minimum oscillation
Research project
119
3.1. Polynomial Interpolation
Each continuous function can be approximated (arbitrarily close) by a
polynomial, and polynomials of degree interpolating values at distinct
points are all the same polynomial, as shown in the following theorems.
Weierstrass Approximation Theorem

Suppose . Then, for each , there exists a polynomial
such that
, for all .
Example:
120
20
18
16
14
12 f(x)
10 p0
p2
8
p4
6 p6
4
2
0 1 2 3
x
Theorem on Polynomial Interpolation

If are distinct real numbers, then for arbitrary values
, there is a unique polynomial of degree at most such that
.
Proof:
(Uniqueness). Suppose there were two such polynomials, and . Then
would have the property for . Since the
degree of is at most , the polynomial can have at most zeros
unless it is a zero polynomial. Since are distinct, has zeros
and therefore it must be 0. Hence, .
(Existence). For the existence part, we proceed inductively through
construction. For , the existence is obvious since we may choose the
constant function
.
Now suppose that we have obtained a polynomial of degree
121
with
for .
We try to construct in the form
for some constant . Note that this is unquestionably a polynomial of

degree . Furthermore, interpolates the data that interpolates:
.
Now we determine the constant to satisfy the condition ,
which leads to
This equation can certainly be solved for :
because the denominator is not zero. (Why?)
122
Newton Form of the Interpolating Polynomials
As in the proof of the previous theorem, each is obtained by

adding a single term to . Thus, at the end of the process, will be a sum
of terms and will be easily visible in the expression of .
Each has the form
The compact form of this is
(1)
(Here the convention has been adopted that when .)

The first few cases of (1) are
These polynomials are called the interpolation polynomials in Newton's

form.
Illustration of Newton's interpolating polynomials:
1 (5.1)
123
(5.2)
(5.3)
(5.4)
(5.5)
124
3
2
f(x)
p0
p1
p2
1 p3
p4
0 1 2
x
Evaluation of , assuming that are known:

We may use an efficient method called nested multiplication or Horner's
algorithm. This can be explained most easily for an arbitrary expression of
the form
(6.1)
The idea is to write it in the form
125
Thus the algorithm for the evaluation of can be written as
126
Now we can write an algorithm for computing the coefficient in Equation
(1):
The computation of , using Horner's algorithm:
A more efficient procedure exists that achieves the same result. The
alternative method uses divided differences to compute the coefficients . The
method will be presented later.
127
Example: For
(2)
Four values of this function are given as
Construct the Newton form of the polynomial from the data.
Newton form of the polynomial:
=
# Since , the coefficients are
#
(8.1)
# which is the same as the one in (2).
=
128
Polynomial interpolation
0 2 4
x
data points
interpolating polynomial -
newton
given function
(8.2)
129
Example: Find the Newton's form of the interpolating polynomial of the data
Answer:
130
Lagrange Form of the Interpolating Polynomials
Let data points be given, where abscissas are

distinct. The interpolating polynomial will be sought in the form
,
where are polynomials that depend on the nodes , but not
on the ordinates .
How to determine the basis :

Let all the ordinates be 0 except for a 1 occupying -th position, that is,
and other ordinates are all zero. Then,
On the other hand, the polynomial interpolation the data must satisfy
, where is the Kronecker delta which is 1 if and 0 if
. Thus all the basis polynomials must satisfy
for all .
Polynomials satisfying such a property are known as the cardinal
functions.
Now, let us try to construct . It is to be an th-degree polynomial

that takes the value 0 at and the value 1 at . Clearly, it must
be of the form
The constant is obtained by putting :
131
and
Hence, we have
Each cardinal function is obtained by similar reasoning; the general formula

is then
Example: Find an interpolation formula for the two-point table
Solution:
132
Example: Determine the Lagrange interpolating polynomial that passes
through and .
Answer: .
Maple plot:
Polynomial interpolation
1
2 3 4 5
x
data points
interpolating polynomial -
lagrange
133
Example: Use to find the second Lagrange interpolating
polynomial for . Use to approximate .
Solution:
Maple:
(11.1)
134
(11.2)
(11.3)
f(x)
p2(x)
2 3 4 5 6
x
= 0.3500000000
Polynomial Interpolation Error Theorem:

Let , and let be the polynomial of degree that
interpolates at distinct points in the interval .
Then, for each , there exists a number between ,
hence in the interval , such that
135
Example: For the previous example, determine the error bound in .
Solution
(13.1)
3
=
8
(13.2)
Thus,
= 0.1320382370
Example: If the function is approximated by a polynomial of

degree 5 that interpolates at six equally distributed points in
including end points, how large is the error on this interval?
Solution
The nodes are -1, -0.6, -0.2, 0.2, 0.6, and 1.
It is easy to see that .
= 0.06922606316
Thus,
0.00008090517158 (14.1)
136
Interpolation Error for Equally Spaced Nodes:
Polynomial Interpolation Error Theorem for Equally Spaced Nodes:

Let , and let be the polynomial of degree that
interpolates at
Then, for each ,
where
Here, as a proof, we consider bounding
Start by picking an . We can assume that is not one of the nodes, because
otherwise the product in question is zero. Let , for some . Then
we have
Now note that
Thus
Since , we can reach the following bound
137
(3)
The result of the theorem follows from the above bound.
Example: How many equally spaced nodes are required to interpolate

to within on the interval ?
138
Chebyshev Polynomials
In the Polynomial Interpolation Error Theorem, there is a term that can be

optimized by choosing the nodes in a special way. An analysis of this problem
was first given by a great mathematician Chebyshev (1821-1894). The
optimization process leads naturally to a system of polynomials called
Chebyshev polynomials.
The Chebyshev polynomials (of the first kind) are defined recursively as
follows:
The explicit forms of the next few are readily calculated:
(17.1)
(17.2)
(17.3)
(17.4)
(17.5)
139
1
T0
T1
T2
0 1
T3
x
T4
Theorem on Chebyshev Polynomials: For , the Chebyshev

polynomials have this closed-form expression:
It has been verified that if the nodes ,
and its minimum value will be attained if
(4)
The nodes then must be the roots of T , which are, for ,
(5)
140
Theorem on Interpolation Error, Chebyshev Nodes:
If the nodes are the roots of the Chebyshev polynomial , as in (5),
then the error bound for the th-degree interpolating polynomial reads
Example: If the function is approximated by a polynomial of

degree 5 that interpolates at roots of the Chebyshev polynomial in
, how large is the error on this interval?
Solution
It is easy to see that .
Thus,
0.00003652217816 (19.1)
It is an optimal upper bound of the error and smaller than the one in
Equation (14.1).
141
Accuracy comparison between Uniform nodes and Chebyshev nodes.
(20.1)
142
2
0 1
x
Uniform nodes
Chebyshev nodes
More details for Chebyshev polynomials is treated in Chapter 8, which will be

covered in Numerical Analysis II.
143
Empty Page
144
3.2. Divided Differences
It turns out that the coefficients for the interpolating polynomials in
Newton's form can be calculated relatively easily by using divided differences.
Recall: For , the th-degree Newton interpolating

polynomials are of the form
for which . The first few cases are
The coefficient is determined to satisfy

.
Thus, we have
(1.1)
and therefore
(1.2)
Now, since
,
it follows from the above and (1.1) and (1.2) that
145
We know that for distinct real numbers (nodes), , there is a
unique polynomial of degree at most that interpolates at the nodes:
We now introduce the divided differences.
Definition:
The zeroth divided difference of the function with respect to , denoted
, is the value of at :
The remaining divided differences are defined recursively; the first divided
difference of with respect to and is defined as
The second divided difference relative to is defined as
In general, the th divided difference relative to is

defined as
Note: For the th-degree Newton interpolating polynomials, we can see
In general,
146
DD1 ( DD2 DD3
Newton's Divided Difference Formula:

Input: and , saved as
Output:
Step 1: For
For
Step 2: Return
147
Example:
In detail:
Thus,
148
Example: Determine the Newton interpolating polynomial for the data:
Example: Prove that if is a polynomial of degree , then

for all .
149
Properties of Divided Differences:
Permutations in Divided Differences:

The divided difference is a symmetric function of its arguments. That is,
if is a permutation of , then
Error in Newton Interpolation:

Let be the polynomial of degree that interpolates at
distinct nodes, . If is a point different from the nodes, then
Proof: Let be the polynomial of degree at most that interpolates at

nodes, . Then, we know that is obtained from by adding
one term. In fact,
Since , the result follows.
Derivatives and Divided Differences:

If and if are distinct points in , then there
exists a point such that
.
Proof: Let be the polynomial of degree at most that interpolates

at . By the Polynomial Interpolation Error Theorem, there
150
exists a point such that
On the other hand, by the previous theorem, we have
.
The theorem follows from the comparison of above two equations.
Example: Prove that for ,

,
for some .
(Hint: Use the last theorem; employ the divided difference formula to find
.)
151
Empty Page
152
3.3. Data Approximation and Neville's Method
We have studied how to construct interpolating polynomials. A
frequent use of these polynomials involves the interpolation of
tabulated data. In this case, an explicit representation of the
polynomial might not be needed, only the values of the
polynomial at specified points. In this situation the function
underlying the data might be unknown so the explicit form of
the error cannot be used to assure the accuracy of the
interpolation. Neville's Method provides an adaptive mechanism
for the evaluation of accurate interpolating values.
Definition: Let be defined at , and suppose that

are distinct integers, with for each .
The polynomial that agrees with at the points
is denoted by .
153
Example: Suppose that and
. Determine the interpolating polynomial and

use this polynomial to approximate .
Solution: It can be the Lagrange polynomial that agrees with

at :
Thus
On the other hand,
154
Theorem: Let be defined at . Then, for each
,
which is the polynomial interpolating at .
Note: The above theorem implies that the interpolating

polynomial can be generated recursively. For example,
and so on. They are generated in the manner shown in the

following table, where each row is completed before the
succeeding rows are begun.
155
For simplicity in computation, we may try to avoid multiple
subscripts by defining the new variable
Then the above table can be expressed as
156
Example: Let . Use
Neville's method to approximate in a four-digit
accuracy.
Solution:
(2.1)
(2.2)
(2.3)
Note:
=
0.0000724766
=
0.0000050845
Thus is already in a four-digit
accuracy.
The real value is = 0.7419373447

The absolute error:
157
Neville's Iterated Interpolation:
Input: the nodes ; the evaluation point ; the
tolerance ; and values saved in the first
column of .
Output:
Step 1: For
For
If
}
Step 2: Return
158
Example: Neville's method is used to approximate , giving
the following table.
Determine .
159
Empty Page
160
3.4. Hermite Interpolation
The Hermite interpolation refers to the interpolation of a function and some of
its derivatives at a set of nodes. When a distinction is being made between this
type of interpolation and its simpler type (in which no derivatives are
interpolated), the latter is often called Lagrange interpolation.
Basic Concepts of Hermite Interpolation:

For example, we require a polynomial of least degree that interpolates a
function and its derivative at two distinct points, say and . The
polynomial sought will satisfy these four conditions:
Since there are four conditions, it seems reasonable to look for a solution in
, the space of all polynomials of degree at most 3. Rather than writing
in terms of , let us write it as
,
because this will simplify the work. This leads to
The four conditions on can now be written in the form
Thus, the coefficients can be obtained easily.
161
Hermite Interpolation Theorem:
If and are distinct, then the unique
polynomial of least degree agreeing with and at is the
Hermite polynomial of degree at most given by
,
where
Here, is the th Lagrange polynomial of degree . Moreover, if

, then
Construction of Hermite Polynomials using Divided Differences
Recall: The polynomial that interpolates at is given
162
Construction of Hermite Polynomials:
Define a new sequence by
Then the Newton form of the Hermite polynomial is given by
with
being replaced by .
Note: For each ,
The extended Newton divided difference table:

First divided Higher divided
differences differences
as usual
163
Example: Use the extended Newton divided difference method to obtain a
cubic polynomial that takes these values:
Example (continuation): Find a quartic polynomial that takes values given

in the preceding example and, in addition, satisfies .
164
3.5. Spline Interpolation
Runge's phenomenon:
The section begins with the so-called Runge's phenomenon. Recall:

Weierstrass Approximation Theorem
Suppose . Then, for each , there exists a polynomial
such that
, for all .
Interpolation at equidistant points is a natural and well-known approach to

construct approximating polynomials. Runge's phenomenon demonstrates,
however, that interpolation can easily result in divergent approximations.
Consider the function:
Runge found that if this function is interpolated at equidistant points

,
the resulting interpolation oscillates toward the end of the interval, i.e. close
to -1 and 1. It can even be proven that the interpolation error tends toward
infinity when the degree of the polynomial increases:
165
1 f(x)
P7
P10
0 1
x
Mitigations to the problem:
Change of interpolation points: e.g., Chebyshev nodes

Use of piecewise polynomials: e.g., Spline interpolation
Constrained minimization: e.g., Hermite-like higher-order polynomial
interpolation, whose first (or second) derivative has minimal norm.
166
Spline Interpolation
Definition: A partition of the interval is an ordered sequence

such that
The numbers are known as knots or nodes.
Definition: A function is a spline of degree on if

1. The domain of is .
2. There exits a partition of such that on each ,
.
3. are continuous on .
Linear Splines:
A linear spline is a continuous function which is linear on each subinterval.

Thus it is defined entirely by its values at the nodes. That is, given
the linear polynomial on each subinterval is defined as
Example: Find the linear spline for
167
Solution: The linear spline can be easily computed as
Its graph is as shown below.
L(x)
0 1
x
168
First-Degree Spline Accuracy
To find the error bound, we will consider the error on a single subinterval of
the partition, and apply a little calculus. Let be the linear polynomial
interpolating at the endpoints of . Then,
for some . Thus
where
169
Quadratic (Second Degree) Splines:
A quadratic spline is a piecewise quadratic function, of which the derivative is

continuous on . Typically, a quadratic spline is defined by its
piecewise polynomials,
Thus there are parameters to define .
For each of the subintervals, the data , gives two

equations regarding
This is equations. The continuity condition on gives a single equation

for each of the internal nodes:
This totals equations, but unknowns. Thus some additional user-

chosen condition is required, for example,
170
Computing quadratic splines:
Let and suppose that the additional condition is given by

specifying . Because
we have, for ,
By integrating it and using ,
In order to determine , we use the above at :
which implies
Thus we have
171
Example: Find the quadratic spline for
Solution: are computed as

z[0]=0
z[1]=17
z[2]=-23.6667
z[3]=24.3333
z[4]=-20.3333
The graph of is superposed over the graph of the linear spline
Q(x)
L(x)
0 1
x
172
Cubic Splines:
From the definition, a cubic spline is a continuous piecewise cubic polynomial

whose first and second derivatives are again continuous. I guess that at this
moment, your brain may have a clear blue print for how to construct such a
cubic spline. Anyway, let us first consider the constructability of cubic splines.
On each subinterval , we have to determine coefficients
of a cubic polynomial;
the numer of unknowns
On the other hand, the number of equations we can get is
interpolation and continuity of
continuity of
continuity of
Thus are equations; two degrees of freedom remain, and there have
been various ways of choosing them to advantage.
173
Construction of Cubic Splines:
Now we derive the equation for on the interval .

Similarly as for quadratic splines, we define
.
Then, is a linear function satisfying
and therefore is given by the straight line between and :
(1)
where
.
If (1) is integrated twice, the result reads
(2)
The interpolation conditions and can now be

imposed on in order to determine and . That is,
174
Thus the result is
(3)
Equation (3) is easily verified; simply let and to see that the
interpolation conditions are fulfilled. Once the values of have
been determined, Equation (3) can be used to evaluate for .
The values of can be determined from the continuity conditions

for :
Equation (3) gives us by differentiating.
(4)
Then substitution of and simplification lead to
Analogously, using (3) to obtain , we have
When the right sides of the last two equations are set equal to each other, the
result can be written as
175
(5)
for .
Note:
There are equations in (5), while we must determine unknowns,
.
There are two popular approaches for the choice of the two additional
conditions.
Natural Cubic Spline
Clamped Cubic Spline
Natural Cubic Splines:

Let . The system of linear equations in (5) can be written as
where
176
and
Since the matrix is strictly diagonally dominant, the system can be solved
by Gaussian elimination without pivoting.
Clamped Cubic Splines:

Let and be prescribed. The two extra conditions read
, which are equivalently expressed as
and .
Utilizing Equation (4), the conditions read
Equation (5) with the above two quations clearly make conditions for
unknowns, . It is a good exercise to compose an
algebraic system for the computation of clamped cubic splines.
177
Example: Find the natural cubic spline for
Solution: The graph of is superposed over graphs of the linear spline

and the quadratic spline .
S(x)
Q(x)
L(x)
0 1
x
178
Example: Find a natural cubic spline that interpolates the data
5
S(x)
4
Q(x)
L(x)
3
2
0 1 2 3
x
179
Optimality Theorem for Natural Cubic Splines
We now present a theorem to the effect that the natural cubic spline produces
the smoothest interpolating function. The word smooth is given a technical
meaning in the theorem.
Theorem: Let be continuous in and . If is

the natural cubic spline interpolating at the nodes for , then
180
3.6. Parametric Curves
Consider the data of the form:
of which the point plot is given

1
0 1
Then none of the interpolation methods we have learnt so far can be used to
generate an interpolating curve for this data, because the curve cannot be
expressed as a function of one coordinate variable to the other. In this section
we will see how to represent general curves by using a parameter to express
both the - and -coordinate variables.
181
Example: Construct a pair of interpolating polynomials, as a function of , for
the data:
Solution
0 1
182
Applications in Computer Graphics:
Required: Rapid generation of smooth curves that can be quickly and
easily modified.
Preferred: Change of one portion of a curve should have little or no
effect on other portions of the curve.
piecewise cubic Hermite
polynomial.
Note:
For data , the piecewise cubic
Hermite polynomial can be generated independently in each portion
. (Why?)
183
Piecewise cubic Hermite polynomial for General Curve Fitting:
Let us focus on the first portion of the piecewise cubic Hermite polynomial
interpolating between
and For the first portion, the given data are
Only six conditions are specified, while the cubic polynomials and
each have four parameters, for a total of eight. This provides flexibility in
choosing the pair of cubic polynomials to specify the conditions. Notice that
the natural form for determining and requires that we specify
and . Since
the slopes at the endpoints can be expressed using the so-called guidepoints
which are to be chosen from the desired tangent line:
guidepoint for
guidepoint for
Thus,
and therefore we may specify
184
The cubic Hermite polynomial , y(t)) on [0,1]:
The unique cubic Hermite polynomial satisfying
can be computed as
Similarly, the cubic Hermite polynomial satisfying
is
185
Example: Determine the parametric curve when
Solution:
Let and . Then,

: a guidepoint of
: a guidepoint of
The cubic Hermite polynomial on that satisfies
is
t (5.1)
The cubic Hermite polynomial on that satisfies
(5.2)
# On the other hand
(5.3)
186
(5.4)
0
0 1
t
187
Empty Page
188
Homework
3. Curve Fitting: Interpolation and Approximation
#1. For the given functions , let . Construct interpolation

polynomials of degree at most one and at most two to approximate , and find the
absolute error.
a.
b.
#2. Use the Polynomial Interpolation Error Theorem to find an error bound for the
approximations in Problem #1.
#3. The polynomial interpolates the

first four points in the table:
By adding one additional term to , find a polynomial that interpolates the whole table.
#4. Determine the Newton interpolating polynomial for the data:
#5. Neville's method is used to approximate , giving the following table.
Determine .
189
#6. Use the extended Newton divided difference method to obtain a quintic polynomial
that takes these values:
#7. Compose an algebraic system, of the form , explicitly for the computation of
clamped cubic splines.
#8. Find a natural cubic spline for the data.
#9. Construct the piecewise cubic Hermite interpolating polynomial for
190
#10. Let be the unit circle of radius 1: . Find a piecewise cubic parametric
curve that interpolates the circle at
Hint: For the first portion, you may set
0
1
Now, you should find parametric curves for the other two portions.
191
Empty Page
192
4. Numerical Differentiation and Integration
In This Chapter:
Numerical Differentiation
Three-point rules
Five-point rules
Richardson extrapolation Combination of low-order differences
Numerical Integration
Trapezoid rule Newton-Cotes formulas

Simpson's rule
Simpson's Three-Eights rule
Romberg integration
Gaussian Quadrature Method of undetermined coefficients
or orthogonal polynomials
Legendre polynomials
193
4.1. Numerical Differentiation
Note:
This formula gives an obvious way to generate an approximation of :
Let and be the first Lagrange polynomial interpolating .

Then
Differentiating gives
Thus
194
Definition: For ,
Example: Use the forward-difference formula to approximate at

using .
Solution:
= 1.331
3.310000000 (2.1)
= 1.157625
3.152500000 (2.2)
= 1.076890625
3.075625000 (2.3)
The error becomes half ?
195
In general:
Let be distinct points in some interval and
. Then
Its derivative reads
Hence,
Definition: An -point difference formula to approximate
196
Three-Point Formulas ( : For convenience, let
Recall:
Thus, the three-point endpoint formulas and the three-point midpoint

formula read
197
Summary: Numerical Differentiation, the -point formula
1.
2.
Five-Point Formulas: Let
Second-Derivative Midpoint Formula
We may derive the formula by using Taylor expansion. The Taylor expansion
can be used for the derivation of the first-derivative difference formulas.
Derivation:
198
Example: Use the second-derivative midpoint formula to approximate
for using ,0.05.
Solution
14.40000000 (5.1)
14.10000000 (5.2)
14.02500000 (5.3)
14 (5.4)
199
Empty Page
200
4.2. Richardson's Extrapolation
Richardson's extrapolation is used to generate high-accuracy difference results
while using low-order formulas.
Let us exemplify the three-point midpoint formula
Note that in this infinite series, the error series is evaluated at the same point,
.
Derivation using the Taylor's Series Expansion:
201
The last equation can be written as
where , an unknown constant, and is an approximation of

using the parameter .
Write out the above equation with replaced by
The leading term in the error series, , can be eliminated as follows
Thus, we have
The above equation embodies the first step in Richardson extrapolation. It

show that a simple combination of two second-order approximations,
and , furnishes an estimate of with accuracy .
We rewrite the above equation as
Then, similarly,
Subtracting (M41) from 16 times (M42) to produce the new formula:
202
The above idea can be applied recursively. The complete algorithm, allowing
for steps of Richardson extrapolation algorithm, is given next:
1. Select a convenient and compute
2. Compute additional quantities by the formula

Table 1: Richardson Extrapolation
Note: One can prove
The second step in the algorithm can be rewritten for a column-wise

computation:
203
Example: Let . Use the Richardson extrapolation to estimate
using
Solution
1.013662770
1.003353478 (2.2)
1.000834586 (2.3)
0.9999170470
0.9999949557 (2.5)
1.000000150 (2.6)
Error:
= 0.0000829530
= 0.0000050443
The Ratio: = 16.44489820
The error: =
= 1.013662770
= 1.003353478 = 0.9999170470
= 1.000834586 = 0.9999949557 = 1.000000150
204
Using the Taylor's Series Expansion, we can reach at
Example: Produce a Richardson extrapolation table for the approximation of

, for the problem considered in the previous example.
Solution:
(3.1)
(3.2)
(3.3)
(3.4)
(3.5)
(3.6)
Error:
= 0.0001385003
= 0.0000084127
The Ratio:
= 16.46324010
205
The error: =
= =
= = =
206
4.3. Numerical Integration
Numerical integration can be performed by
(1) approximation the function by a th-degree polynomial and
(2) integrating the polynomial over the prescribed interval.
What a simple task it is!
Let be distinct points (nodes) in . Then the Lagrange

interpolating polynomial reads
which interpolates the function . Then, as just mentioned, we simply

approximate
In this way, we obtain a formula that can be used on any . It reads as follows:
(1)
where
The formula of the form in (1) is called a Newton-Cotes formula if the nodes
are equally spaced.
207
The Trapezoid Rule:
The simplest case results if and the nodes are and . In this
case,
Since
and
(Here we have used the Mean Value Theorem on Integral because

does not change the sign over .) The corresponding
quadrature formula is
This is known as the trapezoid rule.
208
0
1
x
An animated approximation of
1
f x dx using trapezoid rule, where
0
f x = x3 C 2 C sin 2 p x and the
partition is uniform. The approximate
value of the integral is 2.500000000.
Number of partitions used: 1.
209
Composite Trapezoid Rule:
If the interval is partitioned like this:
then the trapezoid rule can be applied to each subinterval. Here the nodes are
not necessarily uniformly spaced. Thus, we obtain the composite trapezoid
rule:
With uniform spacing
the composite trapezoid rule takes the form
for which the composite error becomes
210
Example:
211
Simpson's Rule
Simpson's rule results from integrating over the second Lagrange

polynomial with equally spaced nodes:
The elementary Simpson's rule reads
which is reduced to
212
213
Error analysis for the elementary Simpson's rule:
The error for the Simpson's rule (Simpson) can be computed from
which must be . However, by approximating the problem in another

way, a higher-order term involving can be derived.
For each in interval , there is a number such that
Thus
214
However, by Mean Value Theorem,
Thus
We summarize the error analysis by
215
Composite Simpson's Rule:
A composite Simpson's rule using an even number of subintervals is often

adopted. Let be even, and set
Then
The error term for this formula is clearly
216
Simpson's Three-Eights Rule
We have developed quadrature rules when the function is approximated by

piecewise Lagrange polynomials of degrees 1 and 2. Such integration formulas
are called the closed Newton-Cotes formulas, and the idea can be extended
for any degrees. The word "closed" is used, because the formulas include
endpoints of the interval as nodes.
When three equal subintervals are combined, the resulting integration formula
is called the Simpson's three-eights rule:
Example: Let be a multiple of 3. For the nodes,
derive the error term for the composite Simpson's three-eights rule.
Solution:
Note: When is not even, you may approximate the integration by a

combination of the Simpson's rule and the Simpson's three-eights rule. For
example, let . Then you may apply the Simpson's rule for and
the Simpson's three-eights rule for the last three subintervals .
217
Self Study: Consider
Use 5 subintervals ( ) to approximate the integral using Trapezoid rule

and Simpson's rules.
(The true value is = 0.47775572251371818070.)
218
4.4. Romberg Integration
In the previous section, we have found that the Composite Trapezoid rule has
a truncation error of order . Specifically, we showed that for
and
we have
where is the Trapezoid quadrature obtained over equal subintervals:
(1)
Recursive Trapezoid Rule:

Let us begin with an effective computation technique for the Composite
Trapezoid rule when .
Example: What is the explicit formula for and in the

case in which the interval is ?
Solution:
Using Equation (1), we have
219
It is clear that if is to be computed, then we can take advantage of the
work already done in the computation of . For example, from the
preceding example, we see that
With , the general formula pertaining to any interval

is as follows:
or
(2)
Now, it there are uniform subintervals, Equation (2) provides a recursive

Trapezoid rule:
(3)
where
220
Romberg Algorithm:
By an alternative method (Taylor series method), it can be shown that if

, the Composite Trapezoid rule (1) can also be written with an
error term in the form
where is a constant that depends only on and .

Since
as for Richardson extrapolation, we have
Summary of the Romberg algorithm:

The computation of which is the trapezoid estimate with
subintervals obtained using the formula (3):
Then, evaluate higher-order approximations recursively using
221
Example: Use the Composite Trapezoid rule to find approximations to
with , and 8. Then perform Romberg extrapolation on
the results.
Solution:
# Trapezoid estimates
#----------------------------------
=0
# Now, perform Romberg Extrapolation:

# -------------------------------------------------
222
=0
= =
1.570796327 2.094395103
= = =
1.896118898 2.004559755 1.998570731
= = = =
1.974231602 2.000269171 1.999983131 2.000005551
=2
#
#-----------------------------
= 20.70179275
= 16.93999354 # these are 16 in theory.
#
#-----------------------------
= 84.72754757 # this is 64 in theory.
# The final error:

#-----------------------------
= 0.000005551
223
Empty Page
224
4.5. Gaussian Quadrature
In preceding section, we saw how to create quadrature formulas of the type
(1)
that are exact for polynomials of degree , which is the case if and only if
In those formulas, the choice of nodes was made a priori. Once

the nodes were fixed, the coefficients were determined uniquely form the
requirement that Formula (1) must be an equality for .
225
Example (Method of Undetermined Coefficients): Find with
which the following formula is exact for all polynomials of degree .
Solution:
By using as trial functions the polynomials in order, we get
The solution of this system of three simultaneous equations is
Thus the formula can be written as
which will produce exact values of integrals for any quadratic polynomial,
.
It must be noticed that the above formula is the elementary Simpson's rule
with .
226
Gaussian quadrature chooses the points for evaluation in an optimal, rather
than equally-spaced, way. The nodes in the interval and
the weights are chosen to minimize the expected error obtained
in the approximation
To measure this accuracy, we assume that the best choice of these values
produces the exact result for the largest class of polynomials, that is, the
choice that gives the greatest degree of precision.
The above formula gives parameters ( and ) to

choose. Since the class of polynomials of degree at most is -
dimensional (containing parameters), one may try to decide the parameters
with which the quadrature formula is exact for all polynomials in .
227
Example: Determine and so that the integration formula
gives the exact result whenever .
Solution:
As in the previous example, we may apply the method of undetermined
coefficients. By using as trial functions the polynomials
in order, we get
A little algebra shows that this system of equations has the unique solution
which gives the approximation formula
(2.1)
This formula has degree of precision 3, that is, it produces the exact result
for every polynomial in .
The method of undetermined coefficients can be used to determine the
nodes and weights for formulas that give exact results for high-order
polynomials, but an alternative method obtained them mote easily. The
alternative is related Legendre orthogonal polynomials.
228
Legendre Polynomials:
Legendre polynomials are a collection of orthogonal polynomials

satisfying
A few types of orthogonal polynomials will be treated in Chapter 8

(Numerical Analysis II). Here we will consider only the properties useful for
Gauss quadrature.
The Legendre polynomials obey the three term recurrence relation known as
(2)
beginning with . A few first Legendre polynomials are
Relevant properties are
229
Theorem (Gauss Integration):
Suppose that are the roots of the th Legendre polynomial
and obtained by
Then,
Note: Once the nodes are determined, the weights can also
be found by using the method of undetermined coefficients, that is, the
weights are the solution of the linear system
230
x
(5.1)
231
k=1
k=2
k=3
232
k=4
k=5
233
(5.2)
234
Theorem (Gauss-Lobatto Integration):
Let and and are the roots of the first-
derivative of the th Legendre polynomial, . Let be
obtained by
Then,
The Gauss-Lobatto integration is a closed formula for numerical integrations,

which is more popular in real-world applications than open formulas such as
Gauss integration.
Note: Once the nodes are determined, the weights can also
be found by using the method of undetermined coefficients, as for Gauss
Integration; the weights are the solution of the linear system
235
Gaussian Quadrature on Arbitrary Intervals:
An integral over an interval can be transformed into an
integral over by using the change of variables:
Using it, we have
236
Example: Find the Gaussian Quadrature for . Choose .
Solution:
(7.1)
2 (7.2)
n=2 GI=1.93582 error=0.0641804

n=3 GI=2.00139 error=0.00138891
n=4 GI=1.99998 error=1.5768e-05
237
Empty Page
238
Homework:
4. Numerical Differentiation and Integration
#1. Use the most accurate three-point formulas to determine the missing entries.
#2. Use your results in the above table to approximate and with -
accuracy. Make a conclusion by comparing all results (obtained here and from Problem
1) with the exact values:
#3. Let a numerical process be described by
Explain how Richardson extrapolation will work in this case. (Try to introduce a formula
described as in (Table 1).)
#4. Approximate using . Use
(a) Trapezoid rule

(b) Simpson's rules.
239
#5. A car laps a race track in 65 seconds. The speed of the car at each 5 second interval is
determined by using a radar gun and is given from the beginning of the lap, in
feet/second, by the entries in the following table:
Time
Speed
How long is the track?
#6. Use the Composite Trapezoid rule to find approximations and then perform Romberg
extrapolation on the results to find .
(a) (b)
#7. Find the Gaussian Quadrature for with
240
5. Numerical Solution
of Ordinary Differential Equations
In This Chapter:
Elementary Theory of IVPs Existence and uniqueness of
solution
Taylor-series methods
Euler's Method
Higher-Order Taylor Methods
Runge-Kutta (RK) Methods
Second-order RK (Heun's method) Modified Euler's method
Fourth-order RK
Runge-Kutta-Fehlberg method Variable step-size
(adaptive method)
Multistep Methods
Adams-Bashforth-Moulton method
Higher-Order Equations &
Systems of Differential Equations
241
5.1. Elementary Theory of Initial-Value Problems
Our model is a first-order initial-value problem (IVP) written in the
form
(IVP)
Here is a function of that we hope to construct form the information given

in the equations.
Theorem (Existence and Uniqueness of Solution):

Suppose that , is continuous on ,
and . If satisfies a Lipschitz condition on in the variable , i.
e., there is a constant such that
for all ,
then the initial value problem (IVP) has a unique solution for
.
Theorem: Suppose that is defined on . If a constant

exists with
,
then satisfies a Lipschitz condition on in the variable with Lipschitz
constant .
242
Example: Prove that the initial-value problem
has a unique solution on the interval .
Solution:
243
Example: Show that each of the initial-value problems has a unique solution
and find the solution.
a.
b.
Solution:
(Existence and uniqueness):
(Find the solution):

Here, we will try to find the solution for (b), using Maple.
(2.1)
(2.2)
(2.3)
244
5.2. Taylor-Series Methods
Here we rewrite the initial-value problem (IVP):
(IVP)
For the problem, a continuous approximation to the solution will not be

obtained; instead, approximations to will be generated at various points,
called mesh points, in the interval . Once the approximate solution is
obtained at the mesh points, the approximate solution at other points in the
interval can be found by interpolation.
Let the mesh points be equally spaced:
We denote an approximate value of by , that is,
Then, common numerical methods for the numerical solution of (IVP)

compute the approximate solution at a mesh point at a time, moving forward,
estimating an accurate average of and/or utilizing a few of solution
values at previous mesh points. Thus such procedures are step-by-step
methods.
245
Euler's Method
Let us try to find an approximation of , marching through the first

subinterval and using a Taylor-series involving only up to the first-
derivative of . Consider the Taylor series expansion,
Then, utilizing and the value can be

approximated by
(1)
Such an idea can be applied recursively for the computation of solution on

later subintervals. Indeed, since
by replacing and respectively with and , we obtain

(2)
which is an approximation of .
Summarizing the above, the Euler's method solving the first-order IVP is
formulated as
(3)
246
Notes:
The computed quantity is an approximation of .
In each subinterval, the method involves a local truncation error
Since the number of subintervals is , the global

truncation error becomes , which implies that
decreases linearly on , as approaches 0.
The global truncation error (error bound):

Let and . Then,
247
Example: Consider
As the step lengths become smaller, , the numerical

solutions represent the exact solution better, as shown in the following figures:
8 8 8
7 7 7
6 6 6
5 5 5
4 4 4
3 3 3
2 2 2
1 1 1
0 1 2 3 0 1 2 3 0 1 2 3
Exact Exact Exact

Solution Solution Solution
Euler Euler Euler
248
Example: Use Euler's method to solve
, with
Solution:
249
=
0.39169859 (2.1)
Note: You may solve the above problem by using a built-in command. For
example, after defining the ODE, odesys, apply the command
dsolve( , numeric, method=foreuler)
250
Higher-Order Taylor Methods
These methods are based on Taylor series expansion.
If we expand the solution , in terms of its th-order Taylor polynomial

about and evaluated at , we obtain
Successive differentiation of the solution, , gives

and generally,
.
Thus we have
The Taylor method of order corresponding the above equation is obtained

by deleting the remainder term involving .
(4)
where
251
Notes:
: The Euler's Method
Achieve higher-order accuracy as increases, but it requires the

computation of derivatives of .
Example:
Consider the initial-value problem:
.5.
(a) Find .
(b) Perform two iterations to find , with .
Solution: (a).
= =
= =
Thus,
(4.1)
(b).
252
155
=
96
16217
=
4608
3.519314236 (4.2)
3.486013602 (4.3)
The absolute error= = 0.033300634
253
Empty Page
254
5.3. Runge-Kutta Methods
The Taylor-series method of the preceding section has the drawback of
requiring the computation of derivatives of . This is a complicated and
time-consuming procedure for most cases, which makes the Taylor methods
seldom used in practice.
Runge-Kutta methods have high-order local truncation error of the Taylor

methods but eliminate the need to compute and evaluate the derivatives of
. That is, the Runge-Kutta Methods are formulated, incorporating a
weighted average of slopes, as follows:
,
where
are (recursive) evaluations of the slope
Nee to determine and other parameters to satisfy
255
Second-order Runge-Kutta Method (RK2)
Formulation:
(1)
where
Requirement: Determine such that
with error no greater than
Derivation: For the left-hand side of (1), the Taylor series reads
Since and , we have
(2)
On the other hand, the right-side of (1) can be reformulated as
Thus we obtain
(3)
The comparison of Equations (2) and (3) drives the following result, for the
second-order Runge-Kutta methods.
256
The result:
(4)
Choices:
RK2, which is also known as
Heun's method
Modified Euler method
RK2 (Heun's method):
(1.1)
where
Modified Euler method:

(1.2)
257
Fourth-order Runge-Kutta method (RK4)
Formulation:
(5)
where
Requirement: Determine such that
with error no greater than
The Choice: The most commonly used set of parameter values yields
(6)
where
The local truncation error

258
Maple code:
> f := proc(x,w)
w-x^3+x+1
end proc:
> RK4 := proc(x0,xt,nt,y0,y)

local h,x,w,n,K1,K2,K3,K4;
h:=(xt-x0)/nt:
x:=x0;
w:=y0;
y[0]:=w;
for n from 1 by 1 to nt do
K1:=f(x,w);
K2:=f(x+h/2,w+(h/2)*K1);
K3:=f(x+h/2,w+(h/2)*K2);
x:=x+h;
K4:=f(x,w+h*K3);
w:=w+(h/6.)*(K1+2*K2+2*K3+K4);
y[n]:=w;
end do
end proc:
>
>
>
> RK4(x0,xt,nt,y0,yRK4):
>
> for n from 0 by 1 to nt do
maxerr:=max(maxerr,abs(exacty(n*h)-yRK4[n]));
end do:
259
= 0.00000184873274
max_error Error_Rario
= 15.73378840
= 15.83783784
= 18.31683168
Thus, the global truncation error
260
Adaptive Methods
Accuracy of numerical methods can be improved by decreasing the step

size.
There may be subintervals where a relatively large step size suffices and
other subintervals where a small step is necessary to keep the truncation
error within a desired limit.
An adaptive method is a numerical method which uses a variable step size.
Example: Runge-Kutta-Fehlberg method (RKF45) which uses RK5 to
estimate local truncation error of RK4.
dsolve, with RKF45
261
=
= 8.13831077273522
= 0.0000184489183565617
8
7
6
5
4
3
2
1
0 1 2 3
x
262
5.4. Multistep Methods
The problem: first-order initial value problem (IVP)
Numerical Methods:
Single-step/Starting methods: Euler's method, Modified
Euler's, Runge-Kutta methods
Multi-step/Continuing methods: Adams-Bashforth-
Moulton
Definition: An -step method, , for solving the IVP, is a

difference equation for finding the approximation at
by solving
263
Fourth-order multistep methods:
Let .
Adams-Bashforth method (explicit):
Adams-Moulton method (implicit):
Adams-Bashforth-Moulton method (predictor-corrector):
where
Notes for fourth-order multistep methods:

can be computed by RK4.
Multistep methods may save evaluations of such
that in each step, they require only one or two evaluations
of to fulfill the step.
RK methods are accurate enough and easy to implement,
so that multistep methods are rarely applied in practice.
264
## Maple code: Adams-Bashforth-Moulton (ABM) Method
## Model Problem:
> f := proc(x,w)
w-x^3+x+1
end proc:
> RK4 := proc(x0,xt,nt,y0,y)

local h,x,w,n,K1,K2,K3,K4;
h:=(xt-x0)/nt:
x:=x0;
w:=y0;
y[0]:=w;
K1:=f(x,w);
K2:=f(x+h/2,w+(h/2)*K1);
K3:=f(x+h/2,w+(h/2)*K2);
x:=x+h;
K4:=f(x,w+h*K3);
w:=w+(h/6.)*(K1+2*K2+2*K3+K4);
y[n]:=w;
end do
end proc:
>
> ABM:= proc(x0,xt,nt,y0,y)

local h,x,w,n,ystar;
h:=(xt-x0)/nt:
### Initialization with RK4
RK4(x0,x0+3*h,3,y0,y);
w:=y[3];
### Now, ABM steps
x:=x0+n*h;
ystar:=w 265
+(h/24)*(55*f(x-h,y[n-1])-59*f(x-2*h,y[n-2])
+37*f(x-3*h,y[n-3])-9*f(x-4*h,y[n-4]))
;
w:=w
+(h/24)*(9*f(x,ystar)+19*f(x-h,y[n-1])
-5*f(x-2*h,y[n-2])+f(x-3*h,y[n-3]));
y[n]:=w;
end do;
end proc:
>
>
>
> ABM(x0,xt,nt,y0,yABM):
>
maxerr:=max(maxerr,abs(exacty(n*h)-yABM[n]));
end do:
= 0.00005294884316
Note: The maximum error for RK4
266
5.5. High-Order Equations &
Systems of Differential Equations
The problem: 2nd-order initial value problem (IVP)
Let
.
Then,
.
That is, the above IVP can be equivalently written as the following system of
first-order DEs:
267
Example: Write the following DEs as a system of first-order differential
equations.
(a) .
(b)
Solution:
(Hint: For (b), you should first rewrite it as and introduce
and .)
268
The -th order system of first-order IVPs (IVP_m):
Algorithm: RK4 for solving IVP_m:

Input:
Output:
Step 1: Set /
For , set
OUT
Step 2: for , do
For , set
For , set
For , set
For , set
For , set
Set
OUT
Step 3: Stop
269
## RK4SYS
##------------------------------
## Ex) IVP of 2 equations:
## x' = 2x+4y, x(0)= -1
## y' = -x+6y, y(0)= 6, 0<= t <= 1
> ef := proc(t,w,f)
f(1):=2*w(1)+4*w(2);
f(2):=-w(1)+6*w(2);
end proc:
> RK4SYS := proc(t0,tt,nt,m,x0,x)

local h,t,w,n,j,K1,K2,K3,K4;
#### initial setting
w:=Vector(m):
K1:=Vector(m):
K2:=Vector(m):
K3:=Vector(m):
K4:=Vector(m):
h:=(tt-t0)/nt:
t:=t0;
w:=x0;
for j from 1 by 1 to m do
x[0,j]:=x0(j);
end do;
#### RK4 marching
ef(t,w,K1);
ef(t+h/2,w+(h/2)*K1,K2);
ef(t+h/2,w+(h/2)*K2,K3);
ef(t+h,w+h*K3,K4);
w:=w+(h/6.)*(K1+2*K2+2*K3+K4);
x[n,j]:=w(j);
270
end do
end do
end proc:
>
>
>
>
>
>
>
>
> RK4SYS(t0,tt,nt,m,x0,xRK4):
>
>
>
>
(1)
>
>
>
>
if n=0 then
printf(" \t n x(n) y(n) error(x) error(y)\n");
printf(" \t -----------------------------------------------------\n");
end if;
printf(" \t %5d %10.3f %10.3f %-10.3g %-10.3g\n",
n, xRK4[n,1], xRK4[n,2], abs(xRK4[n,1]-ex(n*h)), abs(xRK4[n,
2]-ey(n*h)) );
end do;
271
\011 n x(n) y(n) error(x) error(y)
\011 -----------------------------------------------------
\011 0 -1.000 6.000 0 0
\011 2 0.366 8.122 6.04e-006 4.24e-006
\011 4 2.387 10.890 1.54e-005 1.07e-005
\011 6 5.284 14.486 2.92e-005 2.01e-005
\011 8 9.347 19.140 4.94e-005 3.35e-005
\011 10 14.950 25.144 7.81e-005 5.26e-005
\011 12 22.577 32.869 0.000118 7.91e-005
\011 14 32.847 42.782 0.000174 0.000115
\011 16 46.558 55.474 0.000251 0.000165
\011 18 64.731 71.688 0.000356 0.000232
\011 20 88.668 92.363 0.000498 0.000323
\011 22 120.032 118.678 0.000689 0.000443
\011 24 160.937 152.119 0.000944 0.000604
\011 26 214.072 194.550 0.00128 0.000817
\011 28 282.846 248.313 0.00174 0.0011
\011 30 371.580 316.346 0.00233 0.00147
\011 32 485.741 402.332 0.00312 0.00195
\011 34 632.238 510.885 0.00414 0.00258
\011 36 819.795 647.785 0.00549 0.0034
\011 38 1059.411 820.262 0.00725 0.00447
\011 40 1364.944 1037.359 0.00954 0.00586
272
## RK4SYSTEM
##------------------------------
## Ex)
> RK4SYSTEM := proc(a,b,nt,X,F,x0,xn)

local h,hh,t,m,n,j,w,K1,K2,K3,K4;
#### initial setting
with(LinearAlgebra):
m := Dimension(Vector(F));
w :=Vector(m);
K1:=Vector(m);
K2:=Vector(m);
K3:=Vector(m);
K4:=Vector(m);
h:=(b-a)/nt; hh:=h/2;
t :=a;
w:=x0;
xn[0,j]:=x0[j];
end do;
#### RK4 marching
K1:=Vector(eval(F, [x=t,seq(X[i+1] = xn[n-1,i], i = 1 .. m)]));
K2:=Vector(eval(F, [x=t+hh,seq(X[i+1] = xn[n-1,i]+hh*K1[i], i = 1 .. m)]));
K3:=Vector(eval(F, [x=t+hh,seq(X[i+1] = xn[n-1,i]+hh*K2[i], i = 1 .. m)]));
t:=t+h;
K4:=Vector(eval(F, [x=t,seq(X[i+1] = xn[n-1,i]+h*K3[i], i = 1 .. m)]));
w:=w+(h/6)*(K1+2*K2+2*K3+K4);
xn[n,j]:=evalf(w[j]);
end do
end do
end proc:
273
(2)
(3)
274
n y_n y(x_n) y'_n y'(x_n) err(y) err(y')
0 -0.40000000 -0.40000000 -0.60000000 -0.60000000 0 0
1 -0.46173334 -0.46173297 -0.63163124 -0.63163105 3.72e-07 1.91e-07
2 -0.52555988 -0.52555905 -0.64014895 -0.64014866 8.36e-07 2.84e-07
3 -0.58860144 -0.58860005 -0.61366381 -0.61366361 1.39e-06 1.99e-07
4 -0.64661231 -0.64661028 -0.53658203 -0.53658220 2.02e-06 1.68e-07
5 -0.69356666 -0.69356395 -0.38873810 -0.38873905 2.71e-06 9.58e-07
6 -0.72115190 -0.72114849 -0.14438087 -0.14438322 3.41e-06 2.35e-06
7 -0.71815295 -0.71814890 0.22899702 0.22899243 4.06e-06 4.59e-06
8 -0.66971133 -0.66970677 0.77199180 0.77198383 4.55e-06 7.97e-06
9 -0.55644290 -0.55643814 1.53478148 1.53476862 4.77e-06 1.29e-05
10 -0.35339886 -0.35339436 2.57876634 2.57874662 4.50e-06 1.97e-05
275
Empty Page
276
Homework:
5. Numerical Solution of Ordinary Differential Equations
#1. Show that the initial-value problem
has a unique solution in the interval . Can you find the solution, by
guessing?
You do not have to implement any code for Problems 2 and 3 below. You may
solve them by using your cute calculator and math formulas.
#2. Use Taylor's method of order to approximate the solution of the

following initial-value problems.
a. with and
b. with and
c. with and
#3. Consider the initial-value problem:
Its actual solution is . Use and a calculator to get the

approximate solution at by applying
a. Euler's method
b. RK2
c. Modified Euler method
d. RK4
Then, compare their results with the actual value .
277
#4. Apply the codes presented in this lecture note for solving the problem as in
the preceding exercise by
a. RK4
b. Adams-Bashforth-Moulton method
Use and compare the accuracy. If you need me to send you the Maple
codes, please let me know.
#5. Consider the following system of first-order differential equations:
The actual solution is
Use "RK4SYSTEM" to approximate the solution with and

compare the errors to see if you can conclude that RK4SYSTEM is a fourth-
order method for systems of differential equations.
278
6. Direct Methods for Solving Linear Systems
In This Chapter:
Gaussian Elimination Three elementary row operations
Replacement
Interchange
Scaling
with partial pivoting
with scaled partial pivoting
Matrix Factorization
factorization
Symmetric Positive Definite
Matrices
Cholesky factorization
factorization
279
6.1. Gaussian Elimination
Elementary Row Operations:

1. (Replacement) Replace one row by the sum of itself and a multiple of
another row:
2. (Interchange) Interchange two rows:
3. (Scaling) Multiply all entries in a row by a nonzero constant:
(1)
(2)
(3)
(4)
# Maple built-in function: GaussianElimination
280
Gaussian Elimination:
281
Forward Elimination:
R_2 <---- R_2 - (7) R_1
R_3 <---- R_3 - (2) R_1
R_3 <---- R_3 - (0.25) R_2
Backward Substitution:
R_2 <---- R_2 - (-8) R_3
R_1 <---- R_1 - (1) R_3
282
R_2 <---- (0.0833333) R_2
R_1 <---- R_1 - (-1) R_2
(2.1)
283
Gaussian Elimination with Partial Pivoting:
284
R_1 <---> R_2
R_2 <---- R_2 - (0.142857) R_1
R_3 <---- R_3 - (0.285714) R_1
285
R_3 <---- R_3 - (0.25) R_2
R_2 <---- R_2 - (1.14286) R_3
R_1 <---- R_1 - (-1) R_3
R_2 <---- (-0.583333) R_2
R_1 <---- R_1 - (5) R_2
286
R_1 <---- (0.142857) R_1
(3.1)
287
Gaussian Elimination with Scaled Partial Pivoting:
(Partial Pivoting for the scaled system: the max norm of rows is all 1.)
288
289
R_2 <---- R_2 - (7) R_1
R_3 <---- R_3 - (2) R_1
R_3 <---- R_3 - (0.25) R_2
R_2 <---- R_2 - (-8) R_3
R_1 <---- R_1 - (1) R_3
R_2 <---- (0.0833333) R_2
290
R_1 <---- R_1 - (-1) R_2
(4.1)
291
Example: Apply the three different Gaussian elimination methods to
(5)
Solution:
R_2 <---- R_2 - (0.176367) R_1
R_2 <---- (-9.58687e-06) R_2
R_1 <---- R_1 - (591400) R_2
R_1 <---- (0.0333333) R_1
(6)
R_2 <---- R_2 - (0.176367) R_1
292
R_2 <---- (-9.58687e-06) R_2
R_1 <---- R_1 - (591400) R_2
R_1 <---- (0.0333333) R_1
(7)
R_1 <---> R_2
R_2 <---- R_2 - (5.67001) R_1
R_2 <---- (1.69080e-06) R_2
293
R_1 <---- R_1 - (-6.13) R_2
R_1 <---- (0.189) R_1
(8)
294
Example: Solve the following system of linear equations
Solution:
Error, (in GaussElimination) numeric exception: division by zero
R_1 <---> R_3
295
R_2 <---- R_2 - (0.5) R_1
R_3 <---- R_3 - (0) R_1
R_2 <---> R_3
R_3 <---- R_3 - (0.0416667) R_2
R_3 <---- (-0.685714) R_3
R_2 <---- R_2 - (-1) R_3
296
R_1 <---- R_1 - (1) R_3
R_2 <---- (0.0833333) R_2
R_1 <---- R_1 - (1) R_2
R_1 <---- (0.5) R_1
(5.1)
R_1 <---> R_2
297
R_2 <---- R_2 - (0) R_1
R_3 <---- R_3 - (2) R_1
R_3 <---- R_3 - (-0.0833333) R_2
R_3 <---- (0.342857) R_3
R_2 <---- R_2 - (-1) R_3
R_1 <---- R_1 - (-1) R_3
298
R_2 <---- (0.0833333) R_2
R_1 <---- R_1 - (1) R_2
(5.2)
299
Example: Count the number of operations required for the Gaussian
Elimination.
300
6.2. LU Factorization
Definition: A nonsingular matrix has an LU factorization if it can be
expressed as the product of a lower-triangular matrix an upper-triangular
matrix :
.
In matrix form, this is written as
The condition that is nonsingular implies that .
With LU factorization, is written as
Thus, the system can be solved by a two-step procedure:
When the matrix is LU-factorizable, the factorization can be achieved by a

repetitive application of replacement row operations, as in the procedure of
echelon form. Here the difference is to place inverses of the elementary
matrices on the left.
301
Let be elementary matrices (for replacement row operations)
such that
is an echelon form of . Then,
where is a lower-triangular matrix and

is upper-triangular.
Thus, the LU factorization can be carried out step-by-step as in
302
LU Factorization:
The LU factorization can be carried out by the Gaussian Elimination
procedure. Define and
Then the Gaussian Elimination procedure can be expressed as
where
303
which is called the th Gaussian transformation matrix. Let be its
inverse:
Since
we have
304
Theorem: If Gaussian elimination can be performed on the linear system
without row interchanges, then the matrix can be factorized into the
product of a lower-triangular matrix and an upper-triangular matrix , that
is, , where .
305
Permutation Matrices:
Definition: An permutation matrix is a matrix obtained by rearranging

the rows of , the identity matrix. Note that .
Example:
The matrix
is a permutation matrix. Let
Then
306
We have seen that for a nonsingular matrix the linear system can be
solved by Gaussian elimination, with the possibility of row interchanges.
If we knew the row interchanges that were required to solve the system by
Gaussian elimination, we could rearrange the original equations so that no
further row interchanges are needed during Gaussian elimination. Hence there
is a rearrangement of the equations in the system that permits Gaussian
elimination to proceed without row interchanges. This implies that for any
nonsingular matrix , a permutation matrix exists for which the system
(1)
can be solved without row interchanges. Once the matrix is LU-factorized,
i.e.,
since we have , this produces the factorization.
307
Example: Determine a factorization in the form for the matrix
Solution:
Forward Elimination: (3.1)

R_1 <---> R_2
R_2 <---- R_2 - (0) R_1
R_3 <---- R_3 - (-1) R_1
R_4 <---- R_4 - (1) R_1
308
R_2 <---> R_4
R_3 <---- R_3 - (0) R_2
R_4 <---- R_4 - (0) R_2
R_4 <---- R_4 - (-1) R_3
(3.1)
The permutation matrix associated with the row interchanges (R_1

R_2) and (R_2 R_4) is
309
Thus,
Forward Elimination: (3.2)

R_2 <---- R_2 - (1) R_1
R_3 <---- R_3 - (-1) R_1
R_4 <---- R_4 - (0) R_1
310
R_3 <---- R_3 - (0) R_2
R_4 <---- R_4 - (0) R_2
R_4 <---- R_4 - (-1) R_3
(3.2)
311
Thus
(3.3)
and therefore
Maple built-in command:
(3.4)
= = =
312
Homework:
6. Direct Methods for Solving Linear Systems
#1. Use Gaussian Elimination with partial pivoting to solve the linear system
where
#2. Let
Use Gaussian Elimination (with partial pivoting, if necessary) to solve the

augmented system
#3. For the matrix considered in Problem 2, determine a factorization of the

form .
Did it require row interchanges?
313

NA1 Lecture Chap.01-06-Paged

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

NA1 Lecture Chap.01-06-Paged

Caricato da

Copyright:

Formati disponibili

Numerical Analysis using Maple and Matlab

Dr. Seongjai Kim

Department of Mathematics and Statistics

MA-4313/6313: Numerical Analysis I

Ch.1: Mathematical Preliminaries

MA-4323/6323: Numerical Analysis II

Ch.7: Iterative Alebraic Solvers

Definition: A function is continuous at if

in other words, if for every there exists a such that

Examples and Discontinuities:

Definition: Let be an infinite sequence of real numbers. This

Definition: Let be a function defined on an open interval containing . The

exists. The number is called the derivative of at .

Important theorems for continuous/differentiable functions

Example: Show that has a solution in the interval .

Example: Let be defined on . Find which assigns the

Extreme Value Theorem:

Example: Find the absolute minimum and absolute maximum values of

Generalized Rolle's Theorem:

Definition: The Riemann integral of a function on the interval is the

where , with and x arbitrarily

Continuous functions are Riemann integrable, which allows us to choose, for

choose , where . In this case,

Part II: , where is any antiderivative of , i.e., a

function such that .

Weighted Mean Value Theorem for Integrals

When , it becomes the usual Mean Value Theorem for Integrals,

where, for some between and ,

Note that is a polynomial of degree .

Example: Let and . Determine the second and third

Frequently used Taylor Series:

Note: When , and , the Taylor's Theorem reads

Taylor's Theorem with Integral Remainder:

where, for some between and ,

Example: Determine Taylor's formula for and approximate .

Let . That is,

Taylor's Theorem for Two Variables:

in which lies between 0 and 1.

For , the Taylor's theorem for two variables reads

where . Equation (1), as a linear approximation or

Errors in Machine Numbers and Computational Results:

Definition: Suppose that is an approximation to .

the relative error is , provided that .

Definition: The number is said to approximate to -significant digits (or

Definition: An algorithm is a procedure that describes, in an unambiguous

Algorithms consist of various steps for inputs, outputs, and functional

Definition: An algorithm is called stable, if small changes in the initial data

Growth rates of the error:

Let be a sequence of real numbers tending to a limit .

Example: Consider a sequence defined recursively as

Definition: A sequence is said to be in (little oh) of if there

, for large , (or equivalently, ).

In this case, we say is in , and denote or .

Example: Show that and

which implies that is in

Example: Choose the correct assertions (in each, )

Ans: and , i.e., as .

Self study: Show that these assertions are not true.

. Then, we have to show

as . For this, you can first get .

Since is bounded, if , then , which implies

A real -dimensional vector is an ordered set of real numbers and is

Definitions: Let x and y be -dimensional vectors.