Lecture notes for Math 61CM, Analysis, Version 2018

Lenya Ryzhik∗

December 6, 2018

Nothing found here is original except for a few mistakes and misprints here and there. These
notes are simply a record of what I cover in class, to spare the students some of the necessity of
taking the lecture notes. The readers should consult the original books for a better presentation and
context. The material from the following books will be likely used: L. Simon ”An Introduction to
Multivariable Mathematics”, V. Zorich ”Mathematical Analysis I”, and maybe W. Rudin ”Principle
of Mathematical Analysis”.

1 The real numbers

In this section, we will introduce the set of real numbers R via a collection of several groups of
axioms: we will postulate that the real numbers (1) form what is known as a field, (2) they are
ordered, and (3) satisfy the completeness axiom. The latter may be less intuitive at the first sight
but, as we will soon see, is absolutely crucial for the subject known as ”mathematical analysis”. The
intuition behind the axioms is that they should formalize what we are know as ”real numbers” from
”past experience”.

1.1 Axioms of addition and multiplication

First, we need to take care of the arithmetic of real numbers. We will assume that the operations of
addition and multiplication are defined for real numbers, and satisfy the following properties:
(A1) The addition operation is associative:

(a + b) + c = a + (b + c) for all a, b, c ∈ R,

(A2) There exists a zero element denoted by 0, so that

x + 0 = 0 + x = x for all x ∈ R.

(A3) For each x ∈ R there exists an element (−x) ∈ R such that

x + (−x) = 0.

(A4) The addition operation is commutative:

a + b = b + a for all a, b ∈ R.

Axioms (A1)-(A3) say that R is a group with respect to addition, and (A4) says that this group is
commutative. We will go over the general notion of a group again in the linear algebra portion of
the class.
∗
Department of Mathematics, Stanford University, Stanford, CA 94305, USA; ryzhik@math.stanford.edu

1
 The next group of axioms concerns multiplication. We will sometimes write the product of two
numbers a, b ∈ R as ab and sometimes as a · b, hoping this causes no confusion.
(M1) The multiplication operation is associative:

(ab)c = a(bc) for all a, b, c ∈ R,

(M2) There exists an identity element denoted by 1, so that 1 6= 0, and

x · 1 = 1 · x = x for all x ∈ R.

(M3) For each x ∈ R such that x 6= 0, there exists an element x−1 ∈ R such that

x · x−1 = 1.

(M4) The multiplication operation is commutative:

ab = ba for all a, b ∈ R.

These axioms mean that the set R \ {0} is a commutative group with respect to multiplication. The
last algebraic axiom, when taken together with the previous axioms, says that R is a field, and is
simply the distributive law with respect to addition and multiplication:

(F1) a(b + c) = ab + ac for all a, b, c ∈ R.

The familiar arithmetic properties of real numbers follow from these axioms relatively easily. For
example, to show that there is only one zero in the set of real numbers, let us assume that 01 and 02
are both zeros and write:
01 = 01 + 02 = 02 + 01 = 02 .
Make sure that you understand which axioms were used in each identity above! To show that each
element x ∈ R has a unique negative element, let us assume that for some x ∈ R there exist x1
and x2 such that
0 = x + x1 , 0 = x + x2 .
Then we can write:

x1 = x1 + 0 = x1 + (x + x2 ) = (x1 + x) + x2 = 0 + x2 = x2 .

Again, make sure that you understand which axioms were used in each identity above. The next
exercise lists some of the other elementary arithmetic properties of real numbers.

Exercise 1.1 Show using the above axioms that the following algebraic properties hold:
(1) Given any a, b ∈ R, equation
a+x=b
has a unique solution x ∈ R.
(2) For each x ∈ R there exists a unique y such that xy = 1.
(3) For each a 6= 0 and each b ∈ R there exists a unique solution to the equation a · x = b.
(4) For any x ∈ R we have x · 0 = 0.
(5) If xy = 0 then either x = 0 or y = 0.
(6) For any x ∈ R we have −x = (−1) · x.
(7) For any x ∈ R we have (−1) · (−x) = x.
(8) For any x ∈ R we have (−x) · (−x) = x · x.

2
1.2 The order axioms
The next set of axioms has little to do with arithmetic but rather ordering: it says that the real
numbers form an ordered set. That is, there is a relation ≤ between the real numbers such that
(O1) For each x ∈ R we have x ≤ x.
(O2) If x ≤ y and y ≤ x then x = y.
(O3) If x ≤ y and y ≤ z then x ≤ z.
These axioms mean that R is a (partially) ordered set. We postulate it is totally ordered:
(O4) For each x, y ∈ R either x ≤ y or y ≤ x.
To connect the axioms of addition, multiplication to the order axioms, we add the following two
axioms.
(OA) If x ≤ y and z ∈ R then x + z ≤ y + z.
(OM) If 0 ≤ x and 0 ≤ y then 0 ≤ xy.
If x ≤ y and x 6= y then we use the notation x < y. The relations ≥ and > are defined in the
obvious way.
Note that we have 0 < 1. Indeed, the multiplication axioms imply that 0 6= 1. If we have 1 < 0,
then adding (−1) to both sides gives 0 < −1, and (OM) implies that 0 < (−1)(−1) = 1, which is a
contradiction to the assumption 1 < 0. We used claim (8) of Exercise 1.1 in the last step.
The next exercise shows that ordering is actually a non-trivial structure – not every field can be
ordered.

Exercise 1.2 Show that not every set satisfying the addition and multiplication axioms can be
ordered. Hint: consider the set {0, 1} with 0 + 0 = 0, 0 + 1 = 1, 1 + 1 = 0 and 0 · 0 = 1 · 0 = 0 · 1 = 0,
and 1 · 1 = 1. Show that it satisfies the addition and multiplication axioms but can not be ordered.

1.3 The completeness axiom

As you can readily check, the axioms of addition, multiplication and order do not yet narrow down
the intuition we have for the set of real numbers that should, at the very least, include what we
know as rational numbers and irrational numbers. Indeed, these axioms are satisfied by the set of
rational numbers alone and hence are not yet sufficient. As a side note, strictly speaking, to define
the rational numbers we need to know what integers are but let us disregard this for the moment.
What is clear is that we need another axiom that would not be satisfied by what we think of as
”rational numbers”. This is achieved by the completeness axiom.
The completeness axiom. If X and Y are non-empty sets of real numbers such that for any x ∈ X
and y ∈ Y we have x ≤ y, then there exists c ∈ R such that x ≤ c ≤ y for all x ∈ X and y ∈ Y .
In a sense, this axiom says that R has no holes. On other hand, the set Q of rational numbers
does have holes: if we take

X = {r ∈ Q : r2 < 2}, Y = {r ∈ Q : r2 > 2 and r > 0} ,

√
then there is no c ∈ Q such that for any x ∈ X and y ∈ Y we have x ≤ c ≤ y – this is because 2 is
irrational. No rational number squared can equal to 2 and if a rational c separating the sets X and Y
were to exist, it is not hard to see that it would have to satisfy c2 = 2 – we will come back to this
a little later. Thus, the completeness axiom really distinguishes the reals from the set of rational
numbers. To see how this axiom can be used, let us give some definitions.

Definition 1.3 (1) A set S ⊆ R is bounded from above if there exists a number K ∈ R such that
any x ∈ S satisfies x ≤ K. Such K is called an upper bound for S.
(2) A set S ⊆ R is bounded from below if there exists a number K ∈ R such that any x ∈ S

3
satisfies K ≤ x. Such K is called a lower bound for S.
(3) A set S ⊆ R is bounded if it is bounded both from above and from below.

Exercise 1.4 Let us define |x| as |x| = x if 0 ≤ x and |x| = −x if 0 ≤ −x. Show that a set S is
bounded if and only if there exists M ∈ R such that |x| ≤ M for all x ∈ S.

Definition 1.5 We say that x ∈ S is the maximum of a set S if for any y ∈ S we have y ≤ x.
Similarly, we say that x ∈ S is the minimum of a set S if for any y ∈ S we have x ≤ y.

Note that the maximum and minimum of a set belong to the set, by definition. However, not every
set has a maximum or a minimum.

Exercise 1.6 Show that the set S = {x : 0 < x < 1} has neither a minimum nor a maximum.

Definition 1.7 (1) A number K is the least upper bound for a set S ⊆ R, denoted as K = sup S,
if K is an upper bound for S and any upper bound M for S satisfies K ≤ M .
(2) A number K is the greatest lower bound for a set S ⊆ R, denoted as K = inf S, if K is a lower
bound for S and any lower bound M for S satisfies M ≤ K.

In other words, sup S is the minimum of the set of the upper bounds for S, and inf S is the maximum
of the set of the lower bounds for S. As we have seen, not every set has a maximum or a minimum.
The completeness axiom for the real numbers has the following consequence showing that the set of
upper bounds for any set has a minimum.

Lemma 1.8 (The least upper bound principle). Every non-empty set of real numbers that is bounded
from above has a unique least upper bound.

Proof. Uniqueness of the least upper bound follows from the fact that (if it exists) it is the minimum
of the set of all upper bounds, and every set can have have at most one minimum. Thus, we only
need to show that a least upper bound exists. Let Y be the set of all upper bounds for a non-empty
set X. Since X is bounded, the set Y is not empty, and by the definition of an upper bound, we
know that for any x ∈ X and y ∈ Y we have x ≤ y. The completeness axiom then implies that there
exists c ∈ R such that for any x ∈ X and y ∈ Y we have x ≤ c ≤ y. The first inequality implies
that c is an upper bound for X and the second implies that it is the least upper bound. 

Exercise 1.9 Show that the conclusion of Lemma 1.8 does not hold for the set Q of rational
numbers. That is, exhibit a set S of rational numbers such that there is no smallest rational upper
bound for S.

One may wonder if one can construct a version of real numbers explicitly and verify that it
satisfies the axioms. A standard way is to start with the rational numbers and use what is known
as the Dedekind cuts. We will not do this here for lack of time, and simply note that an interested
reader should have no difficulty finding it in a variety of books or on the Internet.

1.4 The natural numbers and the principle of mathematical induction

We understand commonly that the natural numbers are 1, 1 + 1, 1 + 1 + 1, . . . In order to make this
more formal, we give the following definition.

Definition 1.10 A set X ⊆ R is inductive if for each x ∈ X, the set X also contains x + 1.

Definition 1.11 The set N of natural numbers is the intersection of all inductive sets that contain 1.

4
Exercise 1.12 Show that N is an inductive set.

Lemma 1.13 The sum of two natural numbers is a natural number.

Proof. Let S be the set of all natural numbers n such that for any m ∈ N we have n + m ∈ N.
Then S is not empty because 1 ∈ S (this, in turn, is because N is an inductive set). Moreover,
if n ∈ S then n + 1 is also in S because for any m ∈ N we have

(n + 1) + m = n + (m + 1) ∈ N,

because n ∈ S and m + 1 ∈ N. Thus, S is an inductive set, and, as it is a subset of N, it has to be

equal to N. 

Exercise 1.14 Show that the product of two natural numbers is a natural number.

Exercise 1.15 Show that if y ∈ N and y 6= 1 then y − 1 ∈ N. Hint: let E be the set of all real
numbers of the form n − 1 where n ∈ N and n > 1. Show that E = N.

Lemma 1.16 If n ∈ N then there are no natural numbers x such that n < x < n + 1.

Proof. We will show that the set S of all natural numbers for which this assertion holds contains 1
and is an inductive set. To show that 1 ∈ S, let

M = {x ∈ N : x = 1 or x ≥ 2}.

Then, obviously 1 ∈ M and if x ∈ M then either x = 1 so that x + 1 = 2 ∈ M or x ≥ 2 and then

x + 1 ≥ 2, thus x + 1 ∈ M also in that case. Thus, M is a subset of N that is an inductive set
containing 1, hence M = N. This means that the assertion of the lemma holds for 1 and 1 ∈ S.
Next, assume that x ∈ S – we will show that x + 1 ∈ S. Assume that there is y ∈ N such that
n + 1 < y < n + 2. Then y > 1, thus y − 1 is in N and n < y − 1 < n + 1, meaning that n ∈
/ S, which
is a contradiction. 
The set Z of integers consists of numbers x ∈ R such that either x = 0, or x ∈ N, or −x ∈ N.
Now, we can define the rational numbers as usual: they have the form mn−1 with m, n ∈ Z, n 6= 0.
The irrational numbers are those that are not rational. In order to show that irrational numbers
exist, we make the following observation.

Exercise 1.17 (1) Show that if s > 0 and s2 < 2 then there exists δ > 0 so that (s + δ)2 < 2. Hint:
you should be able find an explicit δ > 0 in terms of s.
(2) Show that if s > 0 and s2 > 2 then there exists δ > 0 so that (s − δ)2 < 2. Hint: again, pick an
explicit δ > 0 in terms of s.
(3) Let X = {x > 0 : x2 < 2}. Show that X is bounded from above and set s̄ = sup S. Show
that s̄2 = 2 and that s̄ is irrational.

1.5 The Archimedes principle

The Archimedes principle is a key to the approximation theory, and says the following.

Lemma 1.18 (The Archimedes principle) For any fixed h > 0 and any x ∈ R there exists an
integer k ∈ Z so that
(k − 1)h ≤ x < kh.

5
Proof. We first note that N is not bounded from above. Indeed, if it is bounded, set s = sup N.
If s ∈/ N, then, by the definition of supremum, there must be n ∈ N such that n > s − 1, but
then s < n + 1 and thus s is not an upper bound for N. However, if s ∈ N then s + 1 ∈ N
and s + 1 > s meaning that s is again not an upper bound for N. This shows that N is not bounded
from above, and thus Z is not bounded from above or from below. A similar argument shows that
if S is a set of integers bounded from above, it must contain a maximal element, and if it is a set
of integers bounded from below, it must contain a minimal element. Now, turning to the proof of
the Archimedes principle, let S = {n ∈ Z : x/h < n}. This set is bounded from below and thus
contains a minimal element k. Then we have
x
k−1≤ < k,
h
and the conclusion of Lemma 1.18 follows. 
Here are some important corollaries of the Archimedes principle.

Corollary 1.19 (i) For any ε > 0 there exists n ∈ N such that 0 < 1/n < ε.
(ii) For any a, b ∈ R such that a < b there exists r ∈ Q such that a < r < b.

Exercise 1.20 Prove the assertions of Corollary 1.19.

2 The nested intervals lemma and the Heine-Borel theorem

In this section, we will introduce the first ideas related to the notion of compactness.

2.1 The nested interval lemma

Definition 2.1 An infinite sequence of sets X1 , X2 . . . , Xn , . . . is nested if for each n ∈ N we have
Xn+1 ⊆ Xn .

A closed interval [a, b] is simply the set [a, b] = {x : a ≤ x ≤ b}, and an open interval is the set
(a, b) = {x : a < x < b}.

Lemma 2.2 (The nested intervals lemma). Let I1 ⊇ I2 ⊇ · · · ⊇ In ⊇ . . . be a nested sequence of

closed intervals. Then there exists c ∈ R that belongs to all Ik , k ∈ N. If, in addition, for any ε > 0
there exists k such that |Ik | < ε, then there is exactly one point c common to all intervals.

Proof. Since Ik is a nested sequence of intervals, we have ak ≤ am ≤ bm ≤ bk for all m ≥ k. Thus,

the sets A = {an : n ∈ N} and B = {bn : n ∈ N} satisfy the assumptions of the completeness
axiom. Hence, there exists c ∈ R such that am ≤ c ≤ bk for all m, k ∈ N, and, in particular, we
have an ≤ c ≤ bn for all n ∈ N, thus c ∈ In for all n ∈ N.
If there are two points c1 6= c2 that belong to all Ik , say, with c1 < c2 , then we have

an ≤ c1 < c2 ≤ bn for all n,

hence |bn − an | > c2 − c1 , and the length of all intervals In is larger than c2 − c1 . 

Exercise 2.3 Where did we use the fact that the intervals In are closed in the above proof? Give
an example of a nested sequence of open intervals In = (an , bn ) that has an empty intersection.

6
2.2 The Heine-Borel lemma
Definition 2.4 A collection F of sets is said to cover a set Y if for every element y ∈ Y there
exists a set X ∈ F such that y ∈ X.

A collection of sets G is a subcollection of F if every set X ∈ G is also in the collection F.

Lemma 2.5 (The Heine-Borel lemma) Every collection F of open intervals covering a closed in-
terval [a, b] contains a finite subcollection that also covers [a, b].

Proof. Let F be a collection of open intervals covering a closed interval [a, b] and let S be the set
of points y in [a, b] with the following property: the interval [a, y] can be covered by finitely many
intervals from F. As the point a itself is covered by some interval (α, β) ∈ F, with α < a < β,
we know that S is not empty: it contains a, as well as all points z ∈ [a, b] such that a ≤ z < β.
Moreover, the set S is bounded from above: any element y ∈ S satisfies y ≤ b. It is easy to see that
if y ∈ S then any x such that a ≤ x ≤ y is also in S. Let s̄ = sup S, and note that s̄ ∈ [a, b]. We
claim that s̄ ∈ S and s̄ = b. To see that s̄ ∈ S, assume that s ∈ / S̄, and observe that the point s̄ is
0 0 0 0
covered by some open interval (α , β ) ∈ F, with α < s̄ < β . Since s̄ is the least upper bound for S,
and s̄ ∈/ S, there must be a point x ∈ S such that α0 < x ≤ s̄. Since x ∈ S, we can cover the interval
[a, x] by finitely many intervals from F. Adding the interval (α0 , β 0 ) ∈ F to this collection gives us a
finite collection covering the interval [a, s̄], contradicting the assumption that s̄ ∈/ S. Therefore, we
have s̄ ∈ S. However, this also shows that unless s̄ = b, the points in the interval [s̄, min(β 0 , b)) are
also in S, thus s̄ is not an upper bound for S, which is a contradiction. Thus, we have b = s̄. As we
have shown that s̄ ∈ S, we know that b ∈ S, and we are done, by the definition of the set S. 

Exercise 2.6 Where did we use the fact that the intervals in the covering collection F are open?
Where did we use the fact that the interval [a, b] is closed? Construct a collection F of closed
intervals [a, b] that covers the closed interval [0, 1] but no finite sub-collection of F covers [0, 1].
Next, construct a collection G of open intervals (a, b) that covers the open interval (0, 1) but no
finite sub-collection of G covers (0, 1).

Exercise 2.7 Devise an alternative proof of the Heine-Borel lemma as follows. Assume that there
is no finite sub-collection of open intervals from F that covers I0 = [a, b]. Then either there is no
finite sub-collection that covers [a, a1 ] or there is no finite sub-collection that covers [a1 , b], where
a1 = (a + b)/2. Choose the interval I2 out of these two that can not be covered, split it into two
sub-intervals, choose a sub-interval I3 that can not be covered by a finite sub-collection and repeat
this procedure. This will give you a sequence of nested closed intervals I1 ⊇ I2 ⊇ I3 ⊇ · · · ⊇ In ⊇ . . . .
The Nested Intervals Lemma implies that there is a point c that belongs to all of these intervals.
As c ∈ [a, b], it is covered by some open interval I from the collection F. Use this to arrive at a
contradiction to the fact that none of Ik can be covered by finitely many intervals from F.

2.3 Limit points and the Bolzano-Weierstrass lemma

Let us recall that a neighborhood of a point x is an open interval (a, b) that contains x.

Definition 2.8 A point z ∈ R is a limit point of a set X if any neighborhood of z contains infinitely
many elements of X.

Exercise 2.9 Show that z is a limit point of a set X if and only if any neighborhood of z contains
at least one element of X different from z itself.

7
As an example, part (i) of the Corollary 1.19 of the Archimedes principle shows that z = 0 is a limit
point of the set {1/n : n ∈ N}.
Exercise 2.10 Use the Archimedes principle to show that every x ∈ R is a limit point of the set Q
of rational numbers.
The following lemma is a cornerstone of many things to come.
Lemma 2.11 (The Bolzano-Weierstrass lemma) Every bounded set of real numbers has at least one
limit point.
Proof. Let X be an infinite bounded subset of R. As X is bounded, there exists M > 0 so that X
is contained in the interval I = [−M, M ]. Assume that no point in I is a limit point of X. Then
for each x ∈ I we can find an open interval Ix = (ax , bx ) that contains x such that there are only
finitely many points of X inside Ix . The open intervals Ix form an open cover of the closed interval I,
and the Heine-Borel lemma implies that there exists a sub-cover of I by a finite sub-collection of
intervals Ixk = (axk , bxk ), k = 1, . . . , N with some N ∈ N. Each interval Ixk contains only finitely
many elements of X, hence there are only finitely many elements of X in the union of all these
intervals. As no elements of X can lie outside of this union, there are only finitely many elements
in X which is a contradiction. 
Another property we will use often is the following.
Lemma 2.12 Let S be a bounded from above set that does not have a maximum. Then sup S is a
limit point of S.

Exercise 2.13 Prove Lemma 2.12.

3 Limits
3.1 Definition of the limit of a sequence
We first define a limit point of a sequence. One may informally think of them as points where the
sequence ”bunches up”. Here is a way to formalize this.
Definition 3.1 We say that z ∈ R is a limit point of a sequence an if for any ε > 0 there exist
infinitely many k ∈ N such that |ak − z| < ε.
Note a subtle difference between a limit point of a sequence and a limit point of the set of its values.
For instance, for a constant sequence 1, 1, 1, 1, . . . , that is, an = 1 for all n ∈ N, the set of its values
is {1} – it consists of one point and has no limit points. However, x = 1 is a limit point of this
sequence. This small difference will be of no ”serious” importance for us but one should keep it in
mind.
Exercise 3.2 Show that z ∈ R is a limit point of a sequence an if and only if any open interval (c, d)
that contains z, also contains infinitely many elements of the sequence an : there exist infinitely
many n so that an ∈ (c, d).
A sequence may have more than one limit point. For instance, the sequence 1, −1, 1, −1, 1, . . . , that
is, an = (−1)n+1 has exactly two limit points z = 1 and z = −1.
Exercise 3.3 (1) Construct a sequence an that has no limit points. (2) Construct a sequence bk
that has infinitely many limit points. (3) Construct a sequence ck such that any point z ∈ R is a
limit point of bk .

8
 Next, we introduce the notion of the limit of a sequence.

Definition 3.4 A point z ∈ R is the limit of a sequence an if for any open interval (c, d) that
contains z there exists N so that all ak with k ≥ N lie inside (c, d). We write this as an → z as
n → +∞, or as lim an = z.
n→∞

The following exercise gives a slightly more ”practical” definition of the limit.

Exercise 3.5 Show that a point z is the limit of a sequence an if and only if for any ε > 0 there
exists N (ε) so that |ak − z| < ε for all k ≥ N (ε).

A small advantage of Definition 3.4 over the one in Exercise 3.5, and the reason to introduce it first,
is that it will be easy to generalize to spaces other than R where the notion of a distance may be not
defined but the notion of a neighborhood may be. However, the statement in Exercise 3.5 is much
more ”practical” and we will mostly use it rather than Definition 3.4 directly.
Important: unless otherwise specified, we will usually refer to Exercise 3.5 as the definition of
the limit of a sequence. For the first reading, the reader should really think of Exercise 3.5 as the
definition of the limit of a sequence.
The next important exercise gives another informal way to think of the limit of a sequence: this
is the only point where the sequence ”bunches up”.

Exercise 3.6 Show that a point z is the limit of a bounded sequence an if and only if z is the only
limit point of an .

Here are some examples of limits.

Exercise 3.7 Show that

1
(a) lim =0
n→∞ n
sin n
(b) lim =0
n→∞ n
1
(c) lim n = 0 for any q > 1.
n→∞ q

Hints: for part (a) – use part (i) in Corollary 1.19; for part (c) – write q = 1 + δ with δ > 0 and use
induction to show that q n ≥ 1 + nδ. Then apply part (a).

3.2 Basic properties of converging sequences

Definition 3.8 (1) A sequence an is bounded if there exists M ∈ R so that |ak | ≤ M for all k ∈ N.
(2) A sequence an is bounded from below if there exists M ∈ R so that ak ≥ M for all k ∈ N.
(3) A sequence an is bounded from above if there exists M ∈ R so that ak ≤ M for all k ∈ N.
(4) A sequence an is increasing if ak+1 ≥ ak for all k ∈ N.
(5) A sequence an is strictly increasing if ak+1 > ak for all k ∈ N.
(6) A sequence an is decreasing if ak+1 ≤ ak for all k ∈ N.
(7) A sequence an is strictly decreasing if ak+1 < ak for all k ∈ N.
(8) A sequence an is monotone if it is either decreasing or increasing.
(9) A sequence is converging if it has a limit.

Theorem 3.9 A convergent sequence is bounded.

9
Proof. Let an be a convergent sequence, with lim an = `. Take ε = 1 in the definition of the limit
n→∞
of a sequence (again, in Exercise 3.5!) – this implies existence of N so that |ak − `| < 1 for all k ≥ N .
It follows that all ak with k ≥ N satisfy ` − 1 ≤ ak ≤ ` + 1. There are only finitely many elements
aj with j < N . Thus, if we set

m = min(a1 , . . . , aN −1 , ` − 1), M = max(a1 , . . . , aN −1 , ` + 1),

then all ak satisfy m ≤ ak ≤ M , hence the sequence an is bounded. 

An immediate consequence is that any unbounded sequence can not have a limit.

Exercise 3.10 Not all bounded sequences converge: give an example of a bounded sequence that
has no limit.

The next theorem gives a condition for convergence that is actually very useful in many appli-
cations as monotone sequences arise very often.

Theorem 3.11 If an is monotone and bounded, then it converges. Moreover, if S = {a1 , a2 , . . . , ak , . . . },

then (1) If an is increasing and bounded from above then lim an = sup S.
n→∞
(2) If an is decreasing and bounded from below, the lim an = inf S.
n→∞

Proof. We will only prove (1). Let us assume that an is increasing and bounded from above. Then
the set S is bounded from above, thus ` = sup S exists. Let ε > 0, then, as ` − ε is not an upper
bound for S (this follows from the definition of sup S), there exists N (ε) so that aN (ε) > ` − ε. As
the sequence an is monotonically increasing, it follows that for all k ≥ N (ε) we have ak ≥ ` − ε as
well. However, as ` is an upper bound for the sequence an , we also know that ak ≤ `. Thus, we
have |ak − `| < ε for all k ≥ N (ε) and we are done. 

Theorem 3.12 Let an and bn be convergent sequences with lim an = A and lim bn = B.
n→∞ n→∞
(1) The sequence an + bn also converges and lim (an + bn ) = A + B.
n→∞
(2) The sequence an bn also converges and lim (an bn ) = AB.
n→∞
(3) If, in addition, B 6= 0, then the sequence an /bn also converges and lim (an /bn ) = A/B.
n→∞

Proof. We will only prove (2). Let us write

an bn − AB = an bn − an B + an B − AB = an (bn − B) + (an − A)B.

The triangle inequality implies that

|an bn − AB| ≤ |an ||bn − B| + |an − A||B|.

Since an is a convergent sequence, there exists M so that |an | ≤ M for all n ∈ N, which leads to

|an bn − AB| ≤ M |bn − B| + |an − A||B|.

Now, given ε > 0 find N1 so that |an −A| ≤ ε/(2|B|) for all n ≥ N1 , and N2 so that |an −A| ≤ ε/(2M )
for all n ≥ N1 . Then, for all n ≥ N = max(N1 , N2 ), we have
ε ε
|an bn − AB| ≤ M |bn − B| + |an − A||B| ≤ M + |B| = ε.
2M 2|B|

Hence, the sequence an bn converges to AB. 

10
Exercise 3.13 (1) Prove assertion (1) in Theorem 3.12.
(2) Prove assertion (3) in Theorem 3.12. Hint: it is easier to first show that the sequence 1/bn
converges to 1/B.
(3) Let an = 1/n, find a sequence bn such that the limit of an /bn exists, and a sequence dn such that
the limit of an /bn does not exist.
(4) Assume that lim an = A > 0, and lim bn = 0. Show that the sequence an /bn does not converge.
n→∞ n→∞

Theorem 3.14 Assume that an and bn are two convergent sequences with

lim an = A and lim bn = B.

n→∞ n→∞

If A < B then there exists N so that an < bn for all n ≥ N .

Exercise 3.15 Prove this theorem.

3.3 The number e

 1 n
Theorem 3.16 The sequence xn = 1 + converges. Its limit is denoted as e:
n
 1 n
e = lim 1 + (3.1)
n→∞ n
Proof. We will look instead at the sequence
 1 n+1
yn = 1 +
n
and show that it is decreasing. For that, we need the inequality

(1 + α)n ≥ 1 + nα (3.2)

that holds for all α ≥ 0 and all n ∈ N. Recall that we have seen this inequality in Exercise 3.7. Now,
we write
1 n
yn−1 (1 + n−1 ) nn nn+1 n2n n  n 2 n n
= = = =
yn (1 + n1 )n+1 (n − 1)n (n + 1)n+1 (n2 − 1)n n + 1 n2 − 1 n + 1
 1 n n
= 1+ 2 .
n −1 n+1

Now, we use (3.2) with α = 1/(n2 − 1), to get

yn−1  n  n  1 n
≥ 1+ 2 ≥ 1+ = 1.
yn n −1 n+1 n n+1
Thus, the sequence yn is a positive decreasing sequence, and Theorem 3.11 implies that limn→∞ yn
exists. Thus, the sequence xn can be written as a product of two converging sequences:
 1 n  1 n+1 1
xn = 1 + = 1+ ,
n n 1 + 1/n

hence xn is itself converging. 

11
 Let us explain informally why (3.1) agrees with what we may know from a standard calculus
course. It follows from (3.1) that we can write
 1 n
e= 1+ + αn , (3.3)
n
with a sequence αn that goes to zero as n → +∞. Taking the logarithm with respect to base e, that
we will denote simply by log, gives
 1
1 = n log 1 + + βn , (3.4)
n
with h  1 −n i
βn = log 1 + αn 1 + .
n
Exercise 3.17 Show that lim βn = 0. Hint: use the fact that αn → 0.
n→∞

Using the result of Exercise 3.17 in (3.4) gives

 1  1 βn
log 1 + = + . (3.5)
n n n
Exercise 3.18 Recalling the calculus definition of the derivative (that we officially do not know
yet), show that (3.5) implies that if the derivative of f (x) = log x exists at x = 1, then f 0 (x) = 1.
Show that e is the unique number a such that
d 
(loga x) = 1.

dx x=1

3.4 The Cauchy criterion

Definition 3.19 We say that an is a Cauchy sequence if for any ε > 0 there exists m ∈ N so that
|xn − xm | < ε for all n, m ≥ N .

Theorem 3.20 A sequence an converges if and only if an is a Cauchy sequence.

Proof. One direction is easy. Assume that an converges and A = lim an . Given ε > 0 we can find
n→∞
N ∈ N so that |an − A| < ε/2 for all n ≥ N . Then, for all n, m ≥ N we have
ε ε
|an − am | ≤ |an − A| + |A − am | < + = ε,
2 2
hence an is a Cauchy sequence.
Next, assume that an is a Cauchy sequence. First, we claim that an is bounded. Indeed,
taking ε = 1, we can find N such that |an − am | < 1 for all n, m ≥ N . In particular, we have
aN − 1 < xm < aN + 1 for all m ≥ N . In addition, there are only finitely many elements of the
sequence an with n < N , so the set {a1 , a2 , . . . , aN −1 } is a bounded set. Hence, {an } is a union of
two bounded sets, hence it is also bounded, and the sequence an is bounded as well. Thus, for each
n ∈ N we can define
xn = inf ak , yn = sup ak .
k≥n k≥n

It is clear from the definition that xn ≤ xn+1 ≤ yn+1 ≤ yn for all n ∈ N, so that the sequence xn is
increasing and the sequence yn is decreasing. By the nested intervals theorem there exists a point A
common to all intervals [xn , yn ]:
xn ≤ A ≤ yn for all n ∈ N.

12
In addition, we have
xn ≤ an ≤ yn for all n ∈ N,
and it follows that
|A − an | ≤ |xn − yn |. (3.6)
However, given any ε > 0 we can find N so that for all m, N ≥ N we have
ε
|an − am | < ,
10
and in particular,
ε
|an − aN | < .
10
Now, it follows from the definition of xn and yn that for all n ≥ N we have
ε ε
|xn − aN | ≤ , |yn − aN | ≤ ,
10 10
hence
2ε
|xn − yn | ≤
< ε.
10
We conclude from this and (3.6) that |A − an | < ε for all n ≥ N , thus an converges to A as
n → +∞. 

Exercise 3.21 Use the Cauchy criterion to show that the sequence
1 1 1
an = 1 + + + ··· +
2 3 n
does not converge. Hint: show that a2n − an ≥ 1/2 for all n.

Exercise 3.22 (i) Show that the claim of Theorem 3.20 does not hold for the set Q of rational
numbers. In other words, show that a Cauchy sequence of rational numbers does not necessarily
converge to a rational number. This is another crucial difference between Q and R. (ii) However,
show, without using the results about real numbers, that if a sequence ak of rational numbers is
Cauchy and a subsequence ank converges to a rational number ` ∈ Q, then the sequence an converges
to `. In other words, a Cauchy sequence that has a converging subsequence must converge.

3.5 The Bolzano-Weierstrass theorem, lim sup and lim inf

Definition 3.23 If xn is a sequence, and nk ∈ N is an increasing sequence: n1 < n2 < n3 · · · <
nk < . . . , then the sequence yk = xnk is called a subsequence of xn .

Theorem 3.24 Every bounded sequence of real number contains a convergent subsequence.

Proof. Let E be the set of values of a bounded sequence xn . If E is finite, then there exists a ∈ R so
that xn = a for infinitely many n. It is then an easy exercise to see that there exists a subsequence
xnk of xn such that xnk = a for all k, hence limk→∞ xnk = a and we are done. If the set E is infinite,
then, as xn is a bounded sequence, E is an infinite bounded set, hence by the Bolzano-Weierstrass
Lemma 2.11, it has a limit point A. Then one can choose n1 so that |xn1 − A| < 1. Next, note that,
since A is a limit point of xn , the interval (A − 1/2, A + 1/2) contains infinitely many elements of the
sequence, hence one can choose n2 > n1 so that |xn2 − A| < 1/2, and so on, at each step choosing
nk+1 > nk so that |xnk+1 − A| < 1/(k + 1). As lim 1/(k + 1) = 0, the sequence xnk converges to A,
k→∞
and we are done. 

13
Definition 3.25 If there exists a subsequence xnk such that ` = limk→∞ xnk , then we say ` is a
limit of xn along a subsequence.

We now define lim sup and lim inf of a sequence xn . As we have done in the proof of the Cauchy
criterion for convergence, let us set

an = inf xk , bn = sup xk .
k≥n k≥n

As we have observed in that proof, if xn is bounded from below, then an are well-defined, and the
sequence an is increasing, while if xn is bounded from, then bn are well-defined, and the sequence bn
is decreasing. We denote the corresponding limits by



lim sup xn = lim sup xk , lim inf xn = lim inf xk .
n→∞ n→∞ k≥n n→∞ n→∞ k≥n

Proposition 3.26 Let xn be a bounded sequence, then lim inf xn and lim sup xn are the largest and
the smallest limits of xn along a subsequence.

Exercise 3.27 Prove Proposition 3.26.

4 Continuity and limits of a function

4.1 Limit of a function at a point
Let f (x) be a function f : [c, d] → R and a ∈ [c, d].

Definition 4.1 We say that

lim f (x) = A,
x→a

in the sense of Cauchy if for every ε > 0 there exists δ > 0 so that for any x 6= a such that |x−a| < δ
and x ∈ [c, d] we have |f (x) − A| < ε.

Note that we exclude x = a in the above definition. The reason is that we do not want the value
of lim f (x) to depend on the value of f (a). For instance, if f (x) = 1 for x 6= 0 and f (0) = 0, we
x→a
would like to have lim f (x) = 1.
x→0
An alternative definition is as follows.

Definition 4.2 We say that A is a sequential limit of f (x) as x → a if for any sequence xn → a,
with xn 6= a and xn ∈ [c, d], we have
lim f (xn ) = A.
n→∞

Theorem 4.3 The two definitions of the limit of f (x) at a point a are equivalent.

Proof. Let us first assume that

lim f (x) = A,
x→a

in the sense of Cauchy, and let xn be a sequence such that xn → a. Given ε > 0 there exists δ > 0
so that for any x 6= a such that |x − a| < δ we have |f (x) − A| < ε. As xn → a, given this δ > 0, we
can find N so that |xn − a| < δ for all n ≥ N , and then |f (xn ) − A| < ε for all n ≥ N , by the above,
hence f (xn ) → A.

14
 Next, assume that f (x) converges in the sequential sense to A as x → a but that A is not the
limit of f (x) in the Cauchy sense as x → a. This means that there exists ε0 > 0 such that for
any δ > 0 there exists x 6= a such that |x − a| < δ and |f (x) − A| > ε0 . Let us take δn = 1/n –
this will generate the corresponding sequence xn . It is easy to see that xn → a but f (xn ) does not
converge to A, which is a contradiction. 

Exercise 4.4 Verify that the usual arithmetic and inequality properties hold for the limit of a
function at a point.

4.2 Continuous functions and their basic properties

Definition 4.5 (1) A function f (x) : [c, d] → R is continuous at a point a ∈ [c, d] if

lim f (x) = f (a).

x→a

(2) A function f : [c, d] → R is continuous on the interval [c, d] if it is continuous at every point
of [c, d].

We summarize some basic properties of continuous functions in the following proposition.

Proposition 4.6 (i) Let f (x) defined on an interval [c, d] be continuous at a point z ∈ [c, d]
and f (z) 6= 0, then there is a neighborhood U of z such that f (x) has the same sign as f (z) for
all x ∈ U such that x ∈ [c, d].
(ii) If the functions f and g defined on [c.d] are continuous at z ∈ [c, d], then so are the functions
f + g and f g. Also, the function f /g is continuous at z provided that g(z) 6= 0.
(iii) Let f be a continuous function defined on an interval [c1 , d1 ] and g be a continuous function
defined on an interval [c2 , d2 ], such that g(x) ∈ [c1 , d1 ] for all x ∈ [c2 , d2 ], so that the composi-
tion (f ◦ g)(x) = f (g(x)) is defined for all x ∈ [c2 , d2 ]. Show that if f is continuous on [c1 , d1 ]
and g is continuous on [c2 , d2 ], then f ◦ g is continuous on [c2 , d2 ]. be two functions such that the
composition

Exercise 4.7 Prove Proposition 4.6, pay special attention to (i) and (iii).

The next theorem, and Corollary 4.9 show that a continuous function ”can not skip values”.

Theorem 4.8 (Intermediate Value Theorem) Let f be a continuous function on a closed inter-
val [a, b] such that f (a) and f (b) have different signs, that is, f (a)f (b) ≤ 0. Then there exists c ∈ [a, b]
such that f (c) = 0.

Proof. If either f (a) = 0 or f (b) = 0, we are done, so let us assume without loss of generality
that f (a) < 0 and f (b) > 0. Let us denote I1 = [a, b] and divide I1 in half. If the value of f at
the mid-point is zero, we are done, otherwise, one of the two closed intervals has the same property:
the signs of f at the two end-points are different. Let us call that interval I2 = [a1 , b1 ], divide it at
its half-point, and continue. The process terminates if the value of f at one of the mid-points will
be zero, which means we are done – we have found a point on [a, b] where f vanishes. Otherwise,
we get an infinite sequence of nested intervals Ik = [ak , bk ] such that f (ak ) and f (bk ) have different
signs, and Ik+1 ⊂ Ik . We also know that |Ik | = |a − b|/2k−1 goes to zero as k → +∞. The Nested
Intervals Lemma implies that there is a unique point c ∈ [a, b] in the intersection of all Ik . Let a0k
be the endpoint of Ik such that f (a0k ) < 0 and b0k be the endpoint of Ik such that f (b0k ) > 0. Note

15
that a0k → c and b0k → c as k → +∞. As c ∈ [a, b], the function f is continuous at c. Continuity of f
at c implies that
f (c) = lim f (a0k ), f (c) = lim f (b0k ).
k→∞ k→∞

The first equality above implies that f (x) ≤ 0 and the second implies that f (c) ≥ 0. It follows
that f (c) = 0. 

Corollary 4.9 Let f be a continuous function on a closed interval [a, b] such that f (a) < A
and f (b) > A, or the other way around. Then there exists c ∈ [a, b] such that f (c) = A.

Exercise 4.10 Prove Corollary 4.9. Hint: apply the Intermediate Value Theorem to the func-
tion g(x) = f (x) − A.

Theorem 4.11 A continuous function on a closed interval [a, b] is bounded and attains both its
maximum and its minimum on [a, b].

Proof. Since f is continuous on [a, b], for every point x ∈ [a, b] there exists an open interval Ix
containing x such that

f (x) − 1 ≤ f (y) ≤ f (x) + 1 for all y ∈ Ix such that y ∈ [a, b].

The open intervals Ix form a cover of the closed interval [a, b], hence the Heine-Borel lemma implies
that there exists a finite sub-collection Ix1 , . . . IxN that also covers [a, b]. Let

M = 1 + max(f (x1 ), . . . f (xN )),

and
m = −1 + min(f (x1 ), . . . f (xN )).
As the intervals Ix1 , . . . IxN cover [a, b], we know that every point z ∈ [a, b] belongs to some Ixk . It
follows that
m − 1 ≤ f (z) ≤ M + 1,
for all z ∈ [a, b], hence the function f is bounded on [a, b].
To see that the maximum and minimum are attained, let

M = sup f (x) (4.1)

x∈[a,b]

and assume that f (x) 6= M for all x ∈ [a, b]. Then the function g(x) = 1/(M − f (x)) is continuous
on [a, b], thus, by what we have just proved, it is bounded: there exists K so that

g(x) ≤ K for all x ∈ [a, b].

But then we have

1
M − f (x) ≥ , for all x ∈ [a, b],
K
so that
1
f (x) ≤ M − , for all x ∈ [a, b],
K
which contradicts the definition of M in (4.1). Thus, there has to exist c ∈ [a, b] such that f (c) = M ,
hence f attains its maximum on [a, b]. The proof that f attains its minimum on [a, b] is almost
verbatim the same. 

16
Exercise 4.12 Prove that a continuous function on a closed interval [a, b] is bounded and attains
its minimum on [a, b].

Exercise 4.13 Give an alternative proof of Theorem 4.11 with the following outline that relies on
the sequential definition of continuity: assume that M = sup f (x) but f (x) 6= M for any x ∈ [a, b].
x∈[a,b]
Show that then there exists a sequence xn such that all xn ∈ [a, b] and f (xn ) → M as n → +∞.
Use the Bolzano-Weierstrass lemma to show that xn has a subsequence xnk that converges to some
point z ∈ [a, b]. Show that f (x) = M .

Exercise 4.14 (i) Give an example of a continuous function on the open interval (0, 1) that is
unbounded. (ii) Give an example of a continuous function on the open interval (0, 1) that is bounded
but does not attain its maximum or minimum on (0, 1). (iii) Give an example of a discontinuous
function on the closed interval [0, 1] that is unbounded. (iv) Give an example a discontinuous
function on the closed interval [0, 1] that is bounded but does not attain its maximum or minimum
on [0, 1].

4.3 Uniform continuity

Definition 4.15 A function f is uniformly continuous on a set E if for every ε > 0 there exists δ > 0
so that |f (x) − f (y)| < ε for all x, y ∈ E such that |x − y| < δ.

Exercise 4.16 Show that the function f (x) = sin(1/x) is continuous but not uniformly continuous
on the open interval (0, 1). Hint: consider the points xk = 1/(2πk + π/2) and yk = 1/(2πk − π/2)
with k ∈ N.

Exercise 4.17 (i) Let a function f be continuous on the open interval (a, b) and assume that f is
bounded on (a, b): there exists M so that |f (x)| ≤ M for all x ∈ (a, b) and also that f takes each
value y ∈ [−M, M ] at most finitely many times. Show that then lim f (x) exists.
x→a
(ii) Let a function f be uniformly continuous on the open interval (a, b). Show that then lim f (x)
x→a
exists.

Proposition 4.18 A uniformly continuous function defined on a bounded set E is bounded.

Proof. Assume that f is unbounded on (a, b). Then there exists a sequence of points xk ∈ (a, b)
such that
|f (xk+1 )| ≥ |f (xk )| + k.
It follows, in particular, that

|f (xm ) − f (xk )| > k for any k and m > k. (4.2)

Uniform continuity of f implies that there exists δ0 > 0 so that |f (x) − f (y)| < 1 for all x, y ∈ E
such that |x − y| < δ0 . The sequence xk is bounded, hence it has a convergent subsequence xnk .
Therefore, the sequence xnk is Cauchy. In particular, there exists N such that |xnk − xnm | < δ0 for
all k, m ≥ N . But then |f (xnk ) − f (xnm )| < 1 for all k, m ≥ N , which contradicts (4.2). Thus, the
function f has to be bounded on E. 

Theorem 4.19 A function that is continuous on a closed interval [a, b] is uniformly continuous on
that interval.

17
Proof. Let f be continuous on [a, b]. Then, given ε > 0 for each x ∈ [a, b] there exists δx > 0 such
that |f (x) − f (y)| < ε/10 for all y such that |y − x| < δx . Let us consider the family of smaller
open intervals Ix = (x − δx /2, x + δx /2). As they cover the closed interval [a, b], it follows from the
Heine-Borel lemma that there exists a finite sub-collection Ix1 , . . . , IxN of such intervals that also
covers [a, b]. Let us set δ = min(δx1 /2, . . . , δxN /2). We will show that for any x, y ∈ [a, b] such
that |x − y| < δ we have |f (x) − f (y)| < ε – this will prove the uniform continuity of f . Given such x
and y, since the intervals Ix1 , . . . , IxN cover [a, b], there exists k ∈ {1, . . . , N } such that x ∈ Ixk ,
and |xk − x| < δxk /2. Then we also have

δxk δx δx
|y − xk | ≤ |y − x| + |x − xk | < δ + ≤ k + k = δxk .
2 2 2
Hence, both x and y belong to the interval (xk − δxk , xk + δxk ). It follows that
ε ε
|f (x) − f (xk )| < , |f (y) − f (xk )| < .
10 10
The triangle inequality now implies that
ε ε
|f (x) − f (y)| ≤ |f (x) − f (xk )| + |f (y) − f (xk )| < + < ε.
10 10
Thus, we have shown that for any ε > 0 we can find δ > 0 so that for any x, y ∈ [a, b] such
that |x − y| < δ we have |f (x) − f (y)| < ε, which means that the function f is uniformly continuous
on [a, b]. 

5 Open, closed and compact sets in Rn

5.1 Open and closed sets
Let us recall that the distance between two points x = (x1 , . . . , xn ) ∈ Rn and y = (y1 , . . . , yn ) ∈ Rn
is
d(x, y) = [(x1 − y1 )2 + · · · + (xn − yn )2 ]1/2 ,
also often denoted as
kx − yk = [(x1 − y1 )2 + · · · + (xn − yn )2 ]1/2 .
We will use the notation B(x, r) for a ball centered at a point x ∈ Rn of radius r:

B(x, r) = {y ∈ Rn : d(x, y) < r},

and the closed ball as

B(x, r) = {y ∈ Rn : d(x, y) ≤ r}.

Definition 5.1 A set U ⊂ Rn is open if for every x ∈ U there is an open ball B(x, r) that is
contained in U .

Definition 5.2 A point x ∈ Rn is a limit point of a set G if for any open ball B(x, r) with r > 0
there exists a point y ∈ B(x, r) such that y ∈ G and y 6= x.

Definition 5.3 A set C ⊂ Rn is closed if all limit points of C are contained in C.

Definition 5.4 The complement of a set S ⊂ Rn is the set S c = Rn \ S.

18
 c = c
S T
Exercise 5.5 (i) Let F be a collection of sets and let U = S∈F S. Show that U S∈F S .
(ii) Let F be a collection of sets and let G = S∈F S. Show that Gc = S∈F S c .
T S

Theorem 5.6 A set C ⊂ Rn is closed if and only if its complement U = Rn \ C is open.

Proof. Let C be a closed set and x ∈ U = Rn \ C. As C is closed, we know that x is not a limit
point of C. Thus, there exists a ball B(x, r) such that there is no point from C, except, possibly, x,
in B(x, r). As x ∈ U = Rn \ C, we know that x 6∈ C, hence the ball B(x, r) contains no points
from C, hence B(x, r) ⊂ U . This shows that U is an open set.
Next, assume a set C is such that U = Rn \ C be an open set, and let x be a limit point of C.
We will show that x ∈ C. Indeed, if x 6∈ C, then x ∈ U and, as U is open, there exists a ball B(x, r)
that is contained in U . Hence, no points in B(x, r) are in C. This contradicts the assumption that x
is a limit point of C. Thus, C contains all its limit points and is closed. 

Theorem 5.7 (i) The union of any collection of open sets is itself an open set.
(ii) The intersection of any collection of closed sets is a closed set.
N
\
(iii) The intersection Ui of a finite collection of open sets U1 , . . . , UN is an open et.
i=1
N
[
(iv) The union Fi of a finite collection of closed sets F1 , . . . , FN is a closed set.
i=1

Proof. Let us prove (i). Let Uα , α ∈ A (here,

[ A is just some set of indices, finite or infinite) be
a collection of open sets and let x ∈ U = Uα . Then there exists some Uα with α ∈ A such
α∈A
that x ∈ Uα . As Uα is an open set, there exists r > 0 such that the ball B(x, r) is contained
in Uα . But then B(x, r) ⊆ U , hence U is an open set. Note that (i) is equivalent to (ii) because of
Theorem 5.6 and Exercise 5.5, hence (ii) is also proven.
N
\
To prove (iii), let the sets U1 , . . . , UN be open, and take x ∈ U = Ui . As all Ui are open sets,
i=1
there exist r1 , . . . , rN > 0 such that B(x, rk ) ∈ Uk for all k ∈ {1, . . . , N }. Let r = min(r1 , . . . , rN ),
then B(x, r) ∈ U , hence U is an open set. Note that (iii) is equivalent to (iv) because of Theorem 5.6
and Exercise 5.5, hence (iv) is also proven. 
Here is a bunch of related definitions.

Definition 5.8 (i) A point x is an interior point of a set U ⊂ Rn if there is a ball B(x, r) that is
contained in U .
(ii) A point x is a boundary point of a set U ⊂ Rn if for any r > 0 the ball B(x, r) contains both
points in U and not in U . The boundary of U is denoted as ∂U .
(iii) The closure Ū of a set U is the union of U and the set of its limit points.

Exercise 5.9 Show that Ū is the union of U and the boundary ∂U .

Exercise 5.10 Show that the closure S̄ of any set S is a closed set.

Exercise 5.11 Show that a set F is closed if and only F = F̄ .

19
5.2 Compact sets in Rn
Definition 5.12 A set K ⊂ Rn is compact if from every covering of K by sets that are open in Rn ,
one can extract a finite sub-covering of K.

The Heine-Borel lemma implies that a closed interval [a, b] is a compact set in R. Here is a general-
ization of this result to Rn . A closed n-dimensional interval is a set of the form

I¯ab = {x ∈ Rn : ai ≤ xi ≤ bi , i = 1, . . . , n},

with some fixed numbers ai < bi . We may also define the open n-dimensional interval as
o
Iab = {x ∈ Rn : ai < xi < bi , i = 1, . . . , n},

Exercise 5.13 A closed n-dimensional interval is a closed set in Rn .

Exercise 5.14 An open n-dimensional interval is an open set in Rn .

Proposition 5.15 A closed n-dimensional interval is a compact set in Rn .

Proof. Let us fix ai < bi , i = 1, . . . , N , and consider the corresponding closed n-dimensional
interval I¯ab . Assume that there is a covering G of I¯ab by open sets such that no finite sub-collection
of G covers I¯ab . Let us bisect each interval ai ≤ xi ≤ bi in half – this partitions I¯ab into 2n
closed n-dimensional sub-intervals. Note that least one of these n-dimensional sub-intervals can not
be covered by a finite sub-collection of G. Let us call this n-dimensional sub-interval I¯(1) . Continuing
this process, we obtain a nested sequence of closed n-dimensional intervals I¯(k) such that I¯(k+1) is
contained in I¯(k) , and none of which admit a finite sub-covering. Each I¯(k) has the form
(k) (k)
I¯(k) = {x ∈ Rn : ai ≤ xi ≤ bi , i = 1, . . . , n},
(k) (k) (k) (k)
with some ai < bi , i ∈ {1, . . . , n}. By construction, for each i ∈ {1, . . . , n}, the intervals [ai , bi ]
form a nested collection of closed intervals whose lengths goes to zero as k → +∞. Hence, by
the Nested Intervals Lemma, for each i there exists a unique ci ∈ R such that ci belongs to all
(k) (k)
intervals [ai , bi ], for all k ≥ 1. Then, the point c = (c1 , . . . , cn ) ∈ I¯ab belongs to all n-dimensional
intervals I¯(k) , for all k ≥ 1. Since c ∈ I¯ab , there exists an open set U ∈ G so that c ∈ U . As U is
an open set, there exists r > 0 so that the ball B(c, r) is contained in U . However, by construction,
then there exists N ∈ N so that all I¯(n) with n ≥ N are contained in U .
(k) (k)
Exercise 5.16 Prove this last assertion. Hint: first, show that ai → ci and bi → ci as k → +∞
for each i ∈ {1, . . . , n}.

This contradicts the fact that none of I¯(k) can be covered by finitely many sets from G. Hence, a
finite sub-cover of I¯ab by sets from G has to exist, thus the set I¯ab is compact. 

Proposition 5.17 Any compact set K in Rn is closed.

Proof. Let K be a compact set and z be not in K. For each x ∈ K, we take rx = kx − zk/3
and consider the open ball B(x, rx ). Note that z 6∈ B(x, rx ) for any x ∈ K and, moreover, the
balls B(x, rx ) and B(z, rx ) do not intersect for any x ∈ K. On the other hand, the open balls B(x, rx )
form a cover G of K by open sets. As the set K is compact, there exists a finite sub-cover
SN of K by the
sets in G. That is, there exist x1 , . . . , xN ∈ K such that K is contained in the union i=1 B(xi , rxi ).

20
Let R = min(r1 , . . . , rN ), then, by construction, the ball B(z, R) does not intersect any of the
balls B(xk , rxk ), k = 1, . . . , N . As these balls cover K, it follows that the ball B(z, R) contains no
points from K, hence z is not a limit point of K. Thus, K is a closed set. 
It is not true that all closed sets are compact in Rn . For example, the whole space Rn is a
closed set that is not compact, as is the closed half-space {x = (x1 , . . . , xn ) ∈ Rn : x1 ≥ 0}, or the
complement of the open unit ball {x ∈ Rn : |x| ≥ 1}. The next proposition shows that a closed set
that is contained in some compact set is itself compact.
Proposition 5.18 Let a set K ⊂ Rn be compact and K1 be a subset of K that is a closed set.
Then K1 is a compact set.
Proof. Let K1 ⊂ Rn be a closed set that is a subset of a compact set K ⊂ Rn , and let G1 be
a collection of open sets that covers K1 . Since K1 is a closed set, the set U1 = K1c = Rn \ K1 is
open. Let us add this set to the collection G1 – the resulting collection G covers not just K1 but
also the set K. Since K is compact, there is a finite sub-collection O1 , . . . , ON of sets in G that
covers K. If none of the sets Ok is U1 , then this is a sub-collection not only of G but also of the
original collection G1 . Moreover, as this sub-collection covers K, it also covers K1 – thus we have
found a finite sub-collection of sets in G1 that covers K1 . On the other hand, if one of the sets Ok ,
with k = 1, . . . , N , say, ON is U1 , then we can remove it and consider the collection O1 , . . . , ON −1 .
Note that, since the collection O1 , . . . , ON covers K, it also covers K1 . Therefore, for any x ∈ K1
we can find k = 1, . . . , N such that x ∈ Ok . However, since U1 does not intersect K1 and we assume
that ON = U1 , it is impossible that k = N for any x ∈ K1 . Therefore, the collection O1 , . . . , ON −1
covers K1 . However, all sets Ok , k = 1, . . . , N − 1 are actually not only in G but also in the original
collection G1 . Thus, we have found a finite sub-collection of G1 that covers K1 . Since G1 is an
arbitrary collection of open sets that covers K1 , it follows that K1 is a compact set. 
Definition 5.19 A set S ⊂ Rn is bounded if there exists R > 0 so that S is contained in the
ball B(0, R).
In other words, a set S is bounded if there exists R > 0 so that for any x ∈ S we have |x| < R.
Proposition 5.20 If a set K ⊂ Rn is compact then K is bounded.
Proof. Let K be a compact set, and consider the cover of K by open balls of the form B(x, 1), for
all x ∈ K. As the set K is compact, there exists a finite sub-cover of K by such open balls, that is,
a finite collection of open balls B(x1 , 1), B(x2 , 1), . . . , B(xN , 1) with xk ∈ K, that covers K. Let us
set
R = 2 + max |xi |.
1≤i≤N
As the balls B(xk , 1), k = 1, . . . , N , cover K, for any x ∈ K we can find k such that x ∈ B(xk , 1).
Then we have, by the triangle inequality,
|x| ≤ |x − xk | + |xk | ≤ 1 + max |xi | < R,
1≤i≤N

hence K is a bounded set. 

Theorem 5.21 A set K ⊂ Rn is compact if and only if it is closed and bounded.
Proof. Propositions 5.17 and 5.20 show that if K ⊂ Rn is a compact set then it is closed and
bounded.
Let us now assume that a set K ⊂ Rn is closed and bounded. Since K is bounded, it is contained
¯ Proposition 5.15 shows that I¯ is a compact set. Thus, K
in some closed n-dimensional interval I.
is a closed (by assumption) subset of a compact set. Now, Proposition 5.18 implies that K is a
compact set and we are done. 

21
6 Continuous mappings in Rn
6.1 Convergence of sequences in Rm
(n) (n) (n)
Let x(n) ∈ Rm be a sequence with values in Rm . In other words, x(n) = (x1 , x2 , . . . , xm ) is
(n) (n) (n)
an m-tuple of real-valued sequences x1 , x2 , . . . , xm .

Definition 6.1 Let ` ∈ Rm . We say that lim x(n) = ` if for every ε > 0 there exists N so that for
n→∞
all n ≥ N we have kx(n) − `k < ε.

Note that this definition is equivalent to the following: lim x(n) = ` if

n→∞

lim kxn − `k = 0,
n→∞

with the limit understood simply in the sense of convergence of the sequence kxn −`k of real numbers
to zero.

Exercise 6.2 Show that a sequence {x(1) , x(2) , . . . , x(n) , . . . } ∈ Rm converges to ` = (`1 , . . . , `m ) ∈
Rm if and only if
(n)
lim xk = `k , for all 1 ≤ k ≤ m.
n→∞

In other words, convergence of sequences with values in Rm is equivalent to the convergence of all m
components of the sequence. The key to this exercise is the following pair of inequalities that holds
for all x, y ∈ Rm :
|xi − yi | ≤ kx − yk for all 1 ≤ i ≤ m,
√ (6.1)
kx − yk ≤ m max |xi − yi |.
1≤i≤m

The first inequality in (6.1) is obvious from the definition of kx − yk and the second also follows
immediately from this definition:

kx − yk2 = (x1 − y1 )2 + · · · + (xm − ym )2 ≤ m max |xi − yi |2 .

1≤i≤n

Taking the square root in both sides gives the second inequality in (6.1).

Definition 6.3 Let x(n) be a sequence in Rm . We say that x(n) is Cauchy if for every ε > 0 there
exists N ∈ N so that for all j, k ≥ N we have kx(j) − x(k) k < ε.

Proposition 6.4 A sequence {xn } ∈ Rm is Cauchy if and only if it is convergent.

Exercise 6.5 Prove this proposition, using inequalities (6.1) and the fact that sequences of real
numbers are Cauchy if and only if they are convergent.

6.2 Continuity of maps from Rn to Rm

We will now consider maps f : Rn → Rm and generalize the notions of continuity and convergence
that we have defined for real-valued functions on Rn and sequences of real numbers. Such map is
simply a collection of m real-valued functions f1 (x), . . . , fm (x) that each maps Rn to R. For example,

f (x1 , x2 ) = (x21 + x22 , x2 )

22
is a map from R2 to R2 with f1 (x1 , x2 ) = x21 + x22 and f2 (x1 , x2 ) = x2 , while

g(x1 , x2 , x3 ) = (x1 + x2 , cos(x1 + x3 ), ex1 +x2 +x3 , sin(x3 ))

is a map from R3 to R4 with

g1 (x1 , x2 , x3 ) = x1 +x2 , g2 (x1 , x2 , x3 ) = cos(x1 +x3 ), g3 (x1 , x2 , x3 ) = ex1 +x2 +x3 , g4 (x1 , x2 , x3 ) = sin(x3 ).

We define the limit of a mapping f : Rn → Rm similarly to what we did for real-valued functions.
Definition 6.6 Let f : Rn → Rm be a mapping, a ∈ Rn , and ` ∈ Rm . We say that lim f (x) = ` if
x→a
for every ε > 0 there exists δ > 0 so that for all x ∈ B(a, δ) such that x 6= a we have f (x) ∈ B(`, ε).
Note that we use a slightly different notation than in R: instead of writing kx − ak < δ we write x ∈
B(a, δ), and instead of saying kf (x) − `k < δ we say f (x) ∈ B(`, ε) but the meaning is literally the
same. The reason for this choice is to emphasize the geometry of what is going on.
Definition 6.7 We say that a mapping f : U → Rm is continuous at x ∈ U ⊂ Rn if

lim f (x0 ) = f (x).

x0 →x

This definition is, of course, exactly the same as in R. The same proofs as in R show that if a
map f : U → Rm is continuous at x ∈ U and λ ∈ R then λf is continuous at x, and that if two
maps f : U → Rm and g : U → Rm are both continuous at x ∈ U then so is f + g. The same is true
for composition of continuous maps.
Proposition 6.8 Let f : E → Rm be a map defined on an open set E ∈ Rn , and g : U → Rk be a
map defined on an open set U ⊂ Rm . Assume that f is continuous at x ∈ E, that y = f (x) is in U ,
and that g is continuous at y = f (x). Then the composition g ◦ f is defined in a ball around x and
is continuous at x.
Proof. Given ε > 0, since g is continuous at y we can find r > 0 so that kg(y 0 ) − g(y)k < ε for
all y 0 ∈ B(y, r). Moreover, as f is continuous at x, we can find δ > 0 so that kf (x0 ) − yk < r for
all x0 ∈ B(x, δ). It follows that kg(f (x0 )) − g(f (x))k < ε for all x0 ∈ B(x, r) and we are done. 

6.3 Successive limits vs. the limit

One should not think that the limit of a function of several variables can be computed by taking
successively the limits in each variable. The standard example when this fails is

 x1 x2 , if (x1 , x2 ) 6= (0, 0),



f (x1 , x2 ) = x21 + x22

 0, if (x , x ) 6= (0, 0).
1 2

Then we have
lim f (x1 , x2 ) = 0 for any x1 6= 0,
x2 →0

and
lim f (x1 , x2 ) = 0 for any x2 6= 0,
x1 →0

so that
lim ( lim f (x1 , x2 )) = 0,
x2 →0 x1 →0
lim ( lim f (x1 , x2 )) = 0,
x1 →0 x2 →0

23
but f (x, x) = 1/2, so that the limit lim f (x1 , x2 ) does not exist.
(x1 ,x2 )→(0,0)
Here are some other variations on this theme. Consider
 2 2
 x1 − x2
2 2 , if (x1 , x2 ) 6= (0, 0),
f (x1 , x2 ) = x1 + x2
0, if (x1 , x2 ) 6= (0, 0).


Then each of the successive limits exists but they are different:

x21
lim f (x1 , x2 ) = = 1 for any x1 6= 0,
x2 →0 x21

and
x22
lim f (x1 , x2 ) = − = −1 for any x2 6= 0,
x1 →0 x22
so that
lim ( lim f (x1 , x2 )) = −1,
x2 →0 x1 →0
lim ( lim f (x1 , x2 )) = 1.
x1 →0 x2 →0

It is easy to see that lim(x1 ,x2 )→(0,0) f (x1 , x2 ) does not exist.
The next interesting example is
 2
 x1 x2
, if (x1 , x2 ) 6= (0, 0),
f (x1 , x2 ) = x41 + x22
0, if (x1 , x2 ) 6= (0, 0).


Then the limit of f (x1 , x2 ) is zero along any ray x1 = λt, x2 = µt, in the sense that for any λ, µ ∈ R
fixed, with at least one of λ, µ not equal to zero, we have

λ2 µt3
lim f (λt, µt) = lim = 0.
t→0 t→0 λ4 t4 + µ2 t2

However, if we take x1 = t, x2 = t2 , then f (t, t2 ) = 1/2, hence, again, lim(x1 ,x2 )→(0,0) f (x1 , x2 ) does
not exist.
Finally, consider the function
 1
x1 + x2 sin , if x1 6= 0,

f (x1 , x2 ) = x1
 0, if x = 0. 1

Then we have |f (x1 , x2 )| ≤ |x1 |+|x2 |, which immediately implies that the limit lim(x1 ,x2 )→(0,0) f (x1 , x2 )
exists and equals to zero. However, the successive limit

lim ( lim f (x1 , x2 ))

x2 →0 x1 →0

does not exist at all simply because the limit

lim f (x1 , x2 )
x1 →0

does not exist for any x2 6= 0.

24
6.4 Continuous functions on compact sets
Definition 6.9 We say that a function f : E → Rm is uniformly continuous on a set E ⊂ Rn if for
every ε > 0 there exists δ > 0 so that d(f (x1 ), f (x2 )) < ε for all x1 , x−2 ∈ E such that d(x1 , x2 ) < δ.

Proposition 6.10 If a mapping f : K → Rm is continuous on a compact set K ⊂ Rn then it is

uniformly continuous on K.

Proposition 6.11 If a mapping f : K → Rm is continuous on a compact set K ⊂ Rn then it is

bounded on K.

Proposition 6.12 If a mapping f : K → Rm is continuous on a compact set K ⊂ Rn then it attains

its maximum and minimum on K.

The proofs of all of the above propositions are exactly the same as for continuous real-valued functions
on R defined on a closed interval: these proofs used nothing but the Heine-Borel lemma for closed
intervals, and that is exactly the definition of a compact set. Check this!

6.5 Connected sets

In order to generalize the intermediate value theorem, we will need the notion of a connected set.

Definition 6.13 A continuous path is a continuous function Γ : [α, β] → Rm . We say that a

set E ⊂ Rm is pathwise connected if for every x, y ∈ E there exists a continuous path Γ : [a, b] → Rm
such that Γ(α) = x, Γ(β) = y and Γ(t) ∈ E for all α ≤ t ≤ β. A domain U ⊆ Rm is a pathwise
connected set that is open.

Note that by re-defining Γ(t) as Γ̃(t) = Γ((1 − t)a + tb) for 0 ≤ t ≤ 1, we can always assume
that α = 0 and β = 1 in the definition of the path.
An open ball B(a, r) and a closed ball B̄(a, r) are both pathwise connected sets in Rm , for
any a ∈ Rm . Indeed, given any two points x, y ∈ B(a, r) we can define the path s(t) = (1 − t)x + ty,
with 0 ≤ t ≤ 1. Then s(0) = x, s(1) = y and for each t ∈ [0, 1] we have

ks(t) − ak2 = (s1 (t) − a1 )2 + · · · + (sm (t) − am )2

= ((1 − t)x1 + ty1 − a1 )2 + · · · + ((1 − t)xm + tym − am )2
= ((1 − t)(x1 − a1 ) + t(y1 − a1 ))2 + · · · + ((1 − t)(xm − am ) + t(ym − am ))2 .

We now use the inequality

((1 − t)a + tb)2 ≤ (1 − t)a2 + tb2 , (6.2)
that holds for all a, b ∈ R, and 0 ≤ t ≤ 1, to get

ks(t) − ak2 ≤ (1 − t)(x1 − a1 )2 + t(y1 − a1 )2 + · · · + (1 − t)(xm − am )2 + t(ym − am )2

= (1 − t)kx − ak2 + tky − ak2 ≤ (1 − t)r2 + tr2 = r2 .

It follows that s(t) ∈ B(a, r) for all t, which proves that B(a, r) is a connected set. The proof for
the closed ball B̄(a, r) is identical.

Exercise 6.14 Prove the inequality (6.2). Hint: use the fact that y = x2 is a convex function
for x ∈ R.

25
The following version of the intermediate value theorem holds for continuous functions defined on
pathwise connected sets.

Proposition 6.15 Let E ⊂ Rn be a connected set, and f : E → R be a function that is continuous

on E. Assume that there exists x ∈ E and y ∈ E such that f (x) = A and f (x) = B. Then, for
any C ∈ R that is between A and B there exists z ∈ E such that f (z) = C.

Proof. Since E is a pathwise connected set and x, y ∈ E, there exists a path Γ : [0, 1] → E such
that Γ(0) = x and Γ(1) = y. The function g(t) = f (Γ(t)), defined for 0 ≤ t ≤ 1, is continuous,
as a composition of two continuous functions, g : [01, ] → E and f : E → R. In addition, we
have g(0) = f (x) = A and g(1) = f (y) = B. The intermediate value theorem for real valued
functions on [0, 1] then implies that there exists t0 ∈ [0, 1] such that g(t0 ) = C. This means
that f (z) = C, with z = Γ(t0 ). 

7 Basic properties of metric spaces

We now take a detour to discuss some very basic properties of metric spaces, generalizing what we
know about R and Rn .

7.1 Convergence in metric spaces

If you go back to what we have really used about kxk n Rn , we will discover that what we really
needed in most situations was that the function d(x, y) = kx − yk was non-negative, equal to 0 only
if x = y, symmetric, so that d(x, y) = d(y, x), and satisfied the triangle inequality. We thus make
the following definition.

Definition 7.1 A metric space (X, d) is a set X together with a map d : X × X → R (called a
distance function) such that

1. d(x, y) ≥ 0 for all x, y ∈ X, and d(x, y) = 0 if and only if x = y.

2. d(x, y) = d(y, x) for all x, y ∈ X,

3. (Triangle inequality) d(x, z) ≤ d(x, y) + d(y, z) for all x, y, z ∈ X.

Recall that a norm k.k : V → R on a vector space V over R (or C) is a map that is

1. positive definite, i.e. kxk ≥ 0 for all x ∈ V with equality if and only if x = 0,

2. absolutely homogeneous, i.e. kλxk = |λ| kxk for λ ∈ R (or C) and x ∈ V ,

3. satisfies the triangle inequality, i.e. kx + yk ≤ kxk + kyk for all x, y ∈ V .

Then one easily checks that every normed vector space is a metric space with the induced metric
d(x, y) = kx − yk; for instance the triangle inequality for the metric follows from

d(x, z) = kx − zk = k(x − y) + (y − z)k ≤ kx − yk + ky − zk = d(x, y) + d(y, z),

where the inequality in the middle is the triangle inequality for norms.
There are many interesting metric spaces that are not normed vector spaces, for the simple reason
that the distance function does not require that X is a vector space. For instance, for any set X we
may define a metric on it setting d(x, y) = 0 if x = y, d(x, y) = 1 if x 6= y.

26
 Another example of a metric space we will use often is the space C(K) of continuous real-valued
functions f : K → R, where K is a compact subset of Rn . The distance is defined as

d(f, g) = sup |f (x) − g(x)|.

x∈K

We will often simply look at K = [a, b], a closed interval on R.

Convergence in metric spaces is defined exactly as in Rn , except we use the metric d(x, y) rather
than kx − yk.

Definition 7.2 A sequence xn of points in a metric space (X, d) converges to x ∈ X, denoted

as lim xn = x, if for all ε > 0 there exists N ∈ N such that for all n ≥ N we have d(xn , x) < ε.

Exercise 7.3 Show that the limit of a sequence xn is unique (if it exists).

Exercise
  that if X is a normed space and xn → x in X, then kxn k → kxk. Hint: show
7.4 Show
that kxn k − kxk ≤ kx − xn k.
 

7.2 Continuous mappings between metric spaces

We now turn to continuity of functions f : X → Y where (X, dX ), (Y, dY ) are metric spaces,
generalizing what we have done for maps from Rn to Rm .

Definition 7.5 Suppose (X, dX ), (Y, dY ) are metric spaces. A function f : X → Y is continuous
at a point a ∈ X if for all ε > 0 there exists δ > 0 such for all x ∈ X such that dX (x, a) < δ we
have dY (f (x), f (a)) < ε. A function is called continuous on X if it is continuous at all a ∈ X.

As you recall, for functions on R we have shown that f is continuous at a ∈ R if for any sequence
xn → a we have f (xn ) → f (a). In general metric spaces, we have the same property.

Definition 7.6 Suppose (X, dX ), (Y, dY ) are metric spaces. A function f : X → Y is sequen-
tially continuous at the point a ∈ X if for every sequence xn in X which converges to a, we have
limn→∞ f (xn ) = f (a).

We then have

Lemma 7.7 Suppose (X, dX ), (Y, dY ) are metric spaces. A function f : X → Y is continuous
at a ∈ X if and only if it is sequentially continuous at a ∈ X.

Proof. The proof is verbatim as in R, check this! 

Let us give some examples. Let X = C[0, 1] and consider F : X → R defined as F (f ) = f (3/4).
A sequence of functions fn ∈ C[0, 1] converges to f ∈ C[0, 1] if

sup |fn (x) − f (x)| → 0 as n → +∞.

x∈R

In other words, for any ε > 0 there exists N so that

sup |fn (x) − f (x)| < ε for all n ≥ N .

x∈R

Rephrasing this again, this is equivalent to saying that for any ε > 0 there exists N so that

|fn (x) − f (x)| < ε for all n ≥ N and all x ∈ [0, 1].

27
It follows, in particular, that for any ε > 0 there exists N so that

|fn (3/4) − f (3/4)| < ε for all n ≥ N ,

which is nothing but

|F (fn ) − F (f )| < ε for all n ≥ N ,
which means that F (fn ) converges to F (f ), thus F is continuous at all f ∈ C[0, 1].
The Bolzano-Weierstrass theorem on R stated that bounded sequences had convergent subse-
quences. In particular, if xn is a sequence of points in a closed interval [a, b] ⊂ R, then it has a
subsequence xnk converging to some x ∈ [a, b]. This motivates the following generalization.

Definition 7.8 A metric space (X, dX ) is sequentially compact if every sequence xn of points
in (X, dX ) has a convergent subsequence that converges to a point in X.

Here the word ‘sequentially’ refers to the fact that there is in fact an equivalent (in the setting of
metric spaces) definition, in terms of the Heine-Borel property, which is usually called compactness.
However, sequential compactness, which is, again, equivalent in the setting of metric spaces, is
equally convenient.

Definition 7.9 A subset K of a metric space (X, dX ) is sequentially compact if every sequence xn
of points in K has a convergent subsequence that converges to a point in K.

Recall that in Rn a set is compact if and only if it is closed and bounded. In general metric spaces
only one implication is true.

Proposition 7.10 Let K be a sequentially compact subset of a metric space (X, d), then K is closed
and bounded.

Proof. First, let us show that K is closed. Consider a limit point y ∈ X of the set K. Then there
is a sequence xn ∈ K such that xn → y. As K is sequentially compact, there is a subsequence xnk
that converges to a point in K. Since it is a subsequence of a convergent sequence, it must converge
to the same limit, hence y ∈ K which shows that K is closed. To show that K is bounded, assume
it is not. Then for any z ∈ X and a sequence xk ∈ K such that d(xn , z) ≥ n. Let us then fix
some z ∈ X. As K is compact, xn should have a convergent subsequence xnk → y ∈ K. However,
we have, by the triangle inequality

d(xnk , z) − d(y, xnk ) ≤ d(y, z) ≤ d(y, xnk ) + d(xnk , z).

Hence, there exists N so that for all k ≥ N we have

d(y, z) ≥ d(xnk , z) − 1 ≥ nk − 1,

which is a contradiction. Thus, K must be bounded. 

The next proposition shows that the other direction fails in general.

Proposition 7.11 The closed unit ball B̄(0, 1) ⊂ C[0, 1] is a closed and bounded set that is not
sequentially compact.

Proof. The closed unit ball in C[0, 1] is the set

B̄(0, 1) = {f ∈ C[0, 1] : sup |f (x)| ≤ 1},

x∈[0,1]

28
that is, the set of all f ∈ C[0, 1] such that −1 ≤ f (x) ≤ 1 for all x ∈ [0, 1]. Consider the following
sequence of functions

 1, 0 ≤ x ≤ 1/2 − 1/n,
fn (x) = 2n(1/2 − x + 1/(2n)), 1/2 − 1/n ≤ x ≤ 1/2 − 1/(2n),
0, , 1/2 − 1/(2n) ≤ x ≤ 1.


Then all fn lie in B̄(0, 1) ⊂ C[0, 1] but the sequence fn is not Cauchy. This is because for m > 2n + 1
we have fm (1/2 − 1/(2n)) = 1 but fn (1/2 − 1/(2n)) = 0, so that kfn − fm k = 1. Therefore, the
sequence fn can not have a convergent subsequence, and B̄(0, 1) is not a compact set.
Theorem 7.12 Suppose (X, d) is a sequentially compact metric space, and f : X → R is continuous.
Then f is bounded, and it attains its maximum and minimum, i.e. there exist points a, b ∈ X such
that f (a) = sup{f (x) : x ∈ X}, f (b) = inf{f (x) : x ∈ X}.
Proof. The proof is verbatim as in Rn except in Rn we relied on the Heine-Borel definition
of compactness, so let show how this is done using the sequential compactness. Let us show first
that f (x) is uniformly bounded from above: there exists M > 0 so that f (x) ≤ M for all x ∈ X.
Indeed, suppose it is not, i.e. there is no upper bound for f (X). Then, for any n ∈ N, there
exists xn ∈ X such that f (xn ) > n. By the sequential compactness of X, the sequence xn has a
convergent subsequence, xnk . Let us say x = limk→∞ xnk ∈ X. Due to its continuity, f is sequentially
continuous at x, so
lim f (xnk ) = f (x).
k→∞
In particular, applying the definition of convergence with ε = 1, there exists N such that k ≥ N
implies |f (xnk ) − f (x)| < 1. But then
|f (xnk )| = |(f (xnk ) − f (x)) + f (x)| ≤ |f (xnk ) − f (x)| + |f (x)| < |f (x)| + 1
for all k ≥ N . Since f (xnk ) > nk ≥ k by the very choice of xnk (note nk ≥ k is true for any subse-
quence), this is a contradiction: choose any k > max(N, |f (x)|+1), and then the two inequalities are
contradictory. This completes the boundedness from above claim; a completely analogous argument
shows boundedness from below.
Let us show that the maximum is actually attained, the case of the minimum is identical. The
proof that we used in R works verbatim but let us give a slightly different proof. Now that we
know that f (x) is bounded from above, let M = supx∈X f (x). Then for all n ∈ N, M − n1 is not an
upper bound for f (x), so there exists xn ∈ X such that f (xn ) > M − 1/n. The sequence xn has a
convergent subsequence, by the sequential compactness of X, and we let
a = lim xnk ∈ X.
k→∞

We claim that f (a) = M . Indeed, due to its continuity, f is sequentially continuous at a, so

lim f (xnk ) = f (a).
k→∞

On the other hand, since

1 1
M ≥ f (xnk ) > M − ≥M− ,
nk k
by the very choice of the xn , we have by the sandwich theorem that
lim f (xnk ) = M.
k→∞

In combination with the just observed sequential continuity, this gives f (a) = M , as desired. 
A very similar proof gives

29
Theorem 7.13 Suppose (X, dX ), (Y, dY ) are metric spaces, X is sequentially compact, and f :
X → Y is continuous. Then (f (X), dY ) is sequentially compact.
Proof. Suppose that yn is a sequence in f (X), that is, yn = f (xn ) for some xn . We need to find a
subsequence of yn which converges to a point in f (X) in the metric dY .
Since X is compact, xn has a convergent subsequence xnk . We let

x = lim xnk ∈ X.
k→∞

We claim that ynk converges to y = f (x) ∈ f (X) in the metric dY . Indeed, by the continuity of f ,
we know that f is sequentially continuous at x, so

lim f (xnk ) = f (x).

k→∞

As f (xnk ) = ynk , this proves the claim, and thus the theorem. 

8 Basic facts about series

8.1 Convergence of a series
Given real numbers an , we will use the following definition to make sense of an infinite sum

a1 + a2 + · · · + an + . . . ,

that we will denote as

∞
X
ak ,
k=1
P
or sometimes simply as k ak . We call aj the individual terms of the series. Note that there is no
meaning yet Passigned to the sum above, at the moment it is just a notation. The n-th partial sum
of the series ∞k=1 ak is
n
X
Sn = a1 + · · · + an = ak .
k=1
P∞
Definition 8.1 We say that a series k=1 ak converges if the sequence
P∞ Sn of its partial sums
converges, and call the limit of the sequence Sn the sum of the series k=1 ak .
An immediate consequence of the definition is the Cauchy criterion for convergence that simply
restates the Cauchy criterion for the sequence Sn of the partial sums, using the fact that for all
m > n we have
Xn
Sm − Sn = Sk .
k=m+1

Theorem 8.2 (The Cauchy criterion for convergence) A series ∞

P
k=1 ak is convergent if and only
if for every ε > 0 there exists N so that for all m > n ≥ N we have

|an+1 + · · · + am | < ε.

Here are two important immediate consequence of the Cauchy criterion for convergence of a series.
Corollary 8.3 If only finitely many terms of a series are changed then the new series is convergent
if and only if the original series is convergent.

30
The proof is a simple exercise: be careful about how you choose N given ε > 0 in the definition of
the Cauchy property.
∞
X
Corollary 8.4 If a series an converges then lim an = 0.
n→∞
k=1

The proof is, again, an exercise in the application of the Cauchy property: take m = n + 1 and look
at what Sm − Sn is.
Th geometric series ∞ k
P
k=1 r plays an incredibly important role in analysis, and especially in
complex analysis that is outside the scope of this class.
∞
X
Exercise 8.5 (i) Show that the series rk converges if |r| < 1 and diverges if |r| > 1. Hint: recall
k=0
that
n
X 1 − rk+1
rk = .
1−r
k=0
∞
X
(ii) Show that the series rk diverges both when r = −1 and r = 1. Now, consider the function
k=0
∞
X
S(r) = rk on the open interval r ∈ (−1, 1). Show that lim S(r) does not exist but lim S(r)
r→1 r→−1
k=1
∞
X
exists. Why does it not contradict the divergence of the series (−1)k ?
k=1
∞
X 1
(iii) Show that the series diverges. Hint: note that we have
k
k=1

1 1 1 1 n 1
+ ··· + ≥ + ··· + = = ,
n+1 2n 2n 2n 2n 2
and use the Cauchy criterion.

8.2 Absolute convergence of a series

∞
X ∞
X
Definition 8.6 We say that a series ak converges absolutely if the series |ak | converges.
k=1 k=1

Note that since

|an + · · · + am | ≤ |an | + · · · + |am |,
the Cauchy criterion implies that if a series converges absolutely, then it converges. The converse is
not true. As an example, consider the series
1 1 1 1
1−1+ − + − + ...
2 2 3 3
that has partial sums equal either to zero or 1/n. Thus, it converges. However, part (iii) of Exer-
cise 8.5 shows that it does not converge absolutely.

31
 ∞
X
Theorem 8.7 Let an ≥ 0, then the series an converges if and only if the sequence of its partial
k=1
n
X
sums Sn = an is bounded.
k=1

Proof. The sequence Sn is increasing because an ≥ 0. Thus, Sn converges if and only if it is

bounded. 
∞
X
Theorem 8.8 (Comparison theorem) Assume that 0 ≤ an ≤ bn , then (i) if the series bn con-
k=1
∞
X ∞
X
verges then so does the series an , and (ii) if the series an diverges then so does the se-
k=1 k=1
∞
X
ries bn .
k=1

Exercise 8.9 Prove this theorem in two ways: (i) using the Cauchy criterion for convergence,
and (ii) using Theorem 8.7.
∞
X
Theorem 8.10 (The Weierstrass test) Assume that 0 ≤ |an | ≤ bn , then if the series bn converges
k=1
∞
X
then the series an converges absolutely.
k=1

This is an immediate consequence of Theorem 8.8.

Criteria for series convergence

We now discuss some criteria for theP convergence of a series, based on the Weierstrass and the
convergence of the geometric series rn with |r| < 1.
P
Theorem 8.11 (The Cauchy test) Let an be a series and set

α = lim sup(|an |)1/n .

n→∞
P
Then (i) if α < 1 then the series an converges absolutely, and (ii) if α > 1 then the series diverges.
Note that if α = 1 then the Cauchy criterion
P says nothing about the convergence. As an example of
a divergent series with α = 1, consider n (1/n), then
1
α = lim sup = 1,
n→∞ n1/n
2)
P
but the series diverges. On the other hand, the series n (1/n has
1
α = lim sup = 1,
n→∞ n2/n
but the series converges. This is because for n ≥ 2 we have
1 1 1 1
≤ = − .
n2 n(n − 1) n−1 n

32
and the series
∞ 
X 1 1
−
n−1 n
n=1

converges because its partial sums are explicit:

 1 1 1 1 1  1 1 1
Sn = 1 − + − + − + ··· + − =1− ,
2 2 3 3 4 n−1 n n
and Sn → 1 as n → +∞.
Proof of Theorem 8.11. Assume that α < 1, then there exists N so that for all n > N we
have
1−α 1+α
|an |1/n ≤ β = α + = < 1,
2 2
so that for all n > N we have |an | ≤ β n . P As 0 < β < 1, the series n β n converges, hence the
P
Weierstrass criterion implies that the series an converges absolutely.
On the other hand, if α > 1 then there exists N so that for all n > N we have
α−1 1+α
|an |1/n ≥ γ = α − = > 1,
2 2
hence
P |an | > γ n . As γ > 1, it follows that it is not true that an → 0 as n → +∞, hence the series
n an diverges. 
Here is another way to compare with a geometric series.

Theorem 8.12 (The d’Alembert test) Suppose that the limit

a 
 n+1 
α = lim 
an

n→+∞
P
exists for a series n an . Then (i) if α < 1, the series converges absolutely, and (ii) if α > 1, the
series diverges.

Proof. (i) If α < 1 then there exists N so that for all n ≥ N we have

|an+1 | ≤ r|an |,

where r = (1 + α)/2 < 1. Hence, for all n ≥ N we have |an | ≤ aN rn−N . As r < 1, the series an
converges absolutely.
(ii) If α > 1, then there exists N so that for all n ≥ N we have

|an+1 | ≥ r|an |,

where r = (1 + α)/2 > 1. Hence, for all n > N we have |an | ≥ |aN |rn−N with r > 1, hence it is
impossible that an → 0 as n → +∞, because |an | ≥ |aN |. 
The next proposition is both important, and its proof uses a very useful tool.

Theorem 8.13 The series n (1/np ) converges for all p > 1.

P

Proof. According to Theorem 8.7, it suffices to prove that the sequence of partial sumes
n
X 1
Sn =
kp
k=1

33
is bounded. As an ≥ 0, it is actually sufficient to show that the sequence S2n is bounded. Note that
an = 1/np is a decreasing sequence. Hence we may write

S2n = a1 + a2 + a3 + a4 + · · · + a2n ≤ a1 + a2 + a2 + a4 + a4 + a4 + a4 + a8 + · · · + a2n

= a1 + 2a2 + 4a4 + 8a8 + · · · + 2n a2n .

Recalling that an = 1/np , we have

n n n
X 1 X 1 X 1 1 − rn+1 1
S2n ≤ 2k k p
= k p−1
= p−1 k
= ≤ ,
(2 ) (2 ) (2 ) 1−r 1−r
k=1 k=1 k=1

with r = 1/2p−1 < 1 – here we use the fact that p > 1. Therefore, the partial sums S2n are bounded,
and hence so are the partial sums Sn , and the series converges. 

8.3 Uniform convergence of functions and series

We now discuss the notion of uniform convergence of functions.
Definition 8.14 A sequence of functions fn : E → R converges to a function f : E → R uniformly
on E if for any ε > 0 there exists N so that |f (x) − fn (x)| < ε for all n ≥ N and all x ∈ E. We
sometimes use the notation fn ⇒ f on E.
As an example of a uniformly convergent sequence, consider fn (x) ≡ 1/n on R, or any other set
E ⊂ R. The sequence fn (x) = x/n converges uniformly on [0, 1] to f ≡ 0 but is not uniformly
convergent on R, thus the notion of uniform convergence very much depends on the set E on which
we consider the convergence.

Definition 8.15 A sequence of functions fn : E → R is uniformly Cauchy on E if for any ε > 0

there exists N so that |fm (x) − fn (x)| < ε for all n, m ≥ N and all x ∈ E.

Here is a useful criterion for the uniform convergence.

Theorem 8.16 A sequence of functions fn : E → R is uniformly convergent on f : E → R if and
only if fn is uniformly Cauchy on E.
Proof. First, let fn ⇒ f on a set E. Then for any ε > 0 there exists N so that |f (x) − fn (x)| < ε/10
for all n ≥ n and all x ∈ E. The triangle inequality implies that for any n, m ≥ N and all x ∈ E we
have
ε ε
|fn (x) − fm (x)| ≤ |fn (x) − f (x)| + |f (x) − fm (x)| ≤ + < ε,
10 1)
hence fn is a uniformly Cauchy sequence on E.
Next, assume that fn is a uniformly Cauchy sequence on E. Then, for each x ∈ E fixed, the
sequence of numbers fn (x) is Cauchy, thus it converges to some limit that we denote by f (x). As
the sequence fn is uniformly Cauchy on E, given ε > 0 there exists N so that for all n, m ≥ n and
all x ∈ E we have
ε
|fn (x) − fm (x)| < .
10
Fixing m > N and letting n → ∞ above, we deduce that
ε
|f (x) − fm (x)| < , for all m ≥ n and all x ∈ E,
10
which means that fn ⇒ f on E. 

34
Theorem 8.17 Let fn : E → R be continuous functions on E. Assume that fn converge uniformly
to f on E, then f is continuous.

Proof. This is a homework problem, will be filled in later.

In the same way, we can talk about uniform convergence of a series of functions.
P
Definition 8.18 Let an : E → R be functions from a set E to R. The P series n an (x) converges
uniformly on E if the corresponding sequence of partial sums Sn (x) = nk=1 ak (x) converges uni-
formly on E.

Restating Theorem 8.16 for the uniform convergence of a series gives the following.
P
Theorem 8.19 A series n anP (x) converges uniformly on E if and only if the corresponding se-
quence of partial sums Sn (x) = nk=1 ak (x) is uniformly Cauchy on E.

Restating Theorem 8.17 for the uniform convergence of series of continuous functions gives the
next theorem.
P
Theorem 8.20 Let an : E → R be continuous
P∞ functions from a set E to R. If a series n an (x)
converges uniformly on E to S(x) = n=1 an (x), then the function S(x) is continuous.

Here is a very useful criterion for the uniform convergence of a series.

Theorem 8.21 (The Weierstrass test) Let an : E → R be functions from aPset E to R. If there
exists a sequence
P of numbers bn ≥ 0 such that |an (x)| < bn , and the series n bn converges, then
the series n an (x) converges uniformly on E. It also converges absolutely for each x ∈ E. If,
in addition, each function an (x) is continuous then the sum S(x) = ∞
P
n=1 an (x) is a continuous
function.

Proof. The absolute convergence follows simply from Theorem 8.10 (another Weierstrass test). For
the uniform convergence, note that the partial sums
n
X
Sn (x) = ak (x)
k=1

satisfy
n
 X n
X n
X
|Sn (x) − Sm (x)| =  ak (x)| ≤ |ak (x)| ≤ bk ,

k=m+1 k=m+1 k=m+1
P
for all x ∈ E. As the series bk is convergent, it is Cauchy, thus given any ε > 0 we can find N so
that for all n, m ≥ n we have
Xn
bk < ε.
k=m+1

It follows that then

|Sn (x) − Sm (x)| < ε
P
for all x ∈ E. Thus, the series an (x) is uniformly Cauchy, hence it is uniformly
P convergent.
Finally, the continuity of S(x) follows from the uniform convergence of the series n an (x) on the
set E that we have just proved, and Theorem 8.20. 

35
8.4 Convergence of power series
A series of the form

a0 + a1 (x − x0 ) + a2 (x − x0 )2 + · · · + an (x − x0 )n + . . .

is called a power series around x0 , or simply a power series. We are interested in two questions:
first, if the series converges and second, if the sum is a continuous function. The is answered by the
following theorem.

Theorem 8.22 Let

R−1 = lim sup |an |1/n
n→∞

if the lim sup in the right side is different from zero, otherwise set R = +∞. Then the series
∞
X
an (x − x0 )n
n=1

converges for all x such that |x − x0 | < R and diverges for each x such that |x − x0 | > R. Moreover,
convergence is uniform on any interval |x − x0 | ≤ r with 0 < r < R, and the sum
∞
X
S(x) = an (x − x0 )n
n=1

is continuous on the open interval |x − x0 | < R.

Proof. We will set x0 = 0 without loss of generality. Note that the individual terms of the series
are cn (x) = an xn , and
|cn |1/n = |x|(|an |)1/n ,
so that
|x|
lim sup |cn |1/n = |x| lim sup |an |1/n = .
n→∞ n→∞ R
Now, the Cauchy criterion, Theorem 8.11 tells us that the series converges if |x| < R and diverges in
|x| > R, the first claim of the theorem. To show that the series converges uniformly on any interval
[−r, r] with r < R, note that we can find N so that for all n > N we have
1
|an |1/n ≤ ,
(R + r)/2

so that  2 n
|an | ≤ .
R+r
Then, for all n ≥ n and all x ∈ [−r, r] we have
 2 n n  2r n 2r
|an ||x|n ≤ |x| ≤ = sn , s = .
R+r R+r R+r
P n
As r < R, we know that 0 < s < 1, so that the series s converges. Now, the Weistress test,
Theorem 8.21, implies that the series converges uniformly and the sum is a continuous function on
[−r, r]. As this is true for all 0 < r < R, the sum is continuous on the whole open interval (−R, R). 

36
Exercise 8.23 (i) Show that the power series
1 + x + x2 + · · · + xn + . . .
converges for |x| < 1.
(ii) Show that the power series
x2 x3 xn
1+x+ + + ··· + + ...
2! 3! n!
converges for all x ∈ R.
(iii) Show that the power series
x2 xn
1+x+ + ··· + + ...
2 n
converges for |x| < 1.

9 Differentiability in Rn
9.1 Differentiability in R
In this section, we deal with differentiability of functions defined on the real line, so x ∈ R throughout.
Definition 9.1 Let f be a function from an open set U to R, and a ∈ U If the limit
f (x) − f (a)
f 0 (a) = lim
x→a x−a
exists then f 0 (a) is called the derivative of f at a.
Equivalently, we f (x) is differentiable at a if
f (x) − f (a)
= f 0 (a) + α(x), (9.1)
x−a
and α(x) → 0 as x → a. The notation is α(x) = o(1) as x → a. We say that α(x) = o(|x − a|m ) as
x → a if
α(x)
lim = 0.
x→a |x − a|m

We say that α(x) = O(|x − a|m ) as x → a if

|α(x)|
lim sup < +∞.
x→a |x − a|m
In other words, α(x) = O(|x − a|m ) as x → a if there exists r0 > 0 and C > 0 so that
|α(x)|
< C for all x such that |x − a| < r0 .
|x − a|m
In other words, α(x) = o(|x − a|m ) if α(x) → 0 ”faster” than |x − a|m and α(x) = O(|x − a|m ) if
α(x) → 0 ”at least as fast” as |x − a|m .
Now, we can restate (9.1) as
f (x) = f (a) + f 0 (a)(x − a) + o(|x − a|), (9.2)
that is, f (x) is approximated, to the leading order by the linear function f (a) + f 0 (a)(x − a). We
often write this as
f (a + h) = f (a) + f 0 (a)h + o(|h|). (9.3)

37
Exercise 9.2 An important observation is that if α(h) = o(|h|) and β(h) is bounded, then (αβ)(h) =
o(|h|) as well. Check this!

Theorem 9.3 Let U be an open set. If f : U → R is differentiable at x ∈ U then f is continuous

at a.

Proof. This follows immediately from (9.2). 

Theorem 9.4 Let U be an open set. If f : U → R and g : U → R are differentiable at x ∈ U then

f + g and f g are differentiable at a, with

(f + g)0 (a) = f 0 (a) + g 0 (a), (f g)0 (a) = f (a)g 0 (a) + f 0 (a)g(a).

If, in addition, g(a) 6= 0, then f /g is differentiable at a and

 f 0 f 0 (a)g(a) − f (a)g 0 (a)
= .
g g 2 (a)

Proof. We will only prove this for f g, and use the o(h) notation, to get the reader used to it:

(f g)(a + h) − (f g)(a) = (f (a) + f 0 (a)h + o(|h|))(g(a) + g 0 (a)h + o(|h|) − f (a)g(a)

= (f 0 (a)g(a) + f (a)g 0 (a))h + o(|h|).

For 1/g we note that for α 6= 0 we have

1
= 1 − βh + o(|h|).
1 + βh + o(|h|)
Warning: the o(|h|) in the left and the right sides above are not the same functions. This identity
means that if α(h)/h → 0 as h → 0, then
1
− (1 − βh) = hγ(h),
1 + βh + α(h)
where γ(h) → 0 as h → 0. Make sure you understand this! Now, we can write
1 1 1 1 1 1 1
− = 0
− = 0
−
g(a + h) g(a) g(a) + g (a)h + o(|h|) g(a) g(a) 1 + (g (a)/g(a))h + o(|h|) g(a)
1  0
g (a)  1 g 0 (a)
= 1− h + o(|h|) − = − 2 h + o(|h|),
g(a) g(a) g(a) g (a)
so that  1 0 1
(a) = − .
g g 2 (a)

Theorem 9.5 Let X and Y be open subsets of R. If f : X → R is differentiable at a point x ∈ X,

and f (x) ∈ Y , and g : Y → R is differentiable at y = f (x), then the composition g ◦f is differentiable
at x and (g ◦ f )0 (x) = g 0 (f (x))f 0 (x).

Proof. We have
f (x + h) = f (x) + f 0 (x)h + o(|h|) as h → 0,
and
g(y + t) = g(y) + g 0 (y)t + o(|t|) as t → 0. (9.4)

38
Then we have
(g ◦ f )(x + h) = g(f (x) + f 0 (x)h + o(|h|)) = g(y + t(h)), (9.5)
where t(h) = f 0 (x)h+o(|h|). Note that t(h) → 0 as h → 0 and there exists r0 so that |t(h)| ≤ 2|f 0 (x)h
for |h| < r0 . If we write out the meaning of (9.4), it says that for any ε > 0 there exists δ > 0 so
that if |t| < δ, then
|g(y + t) − (g(y) + g 0 (y)t)|
< ε. (9.6)
|t|
Choosing |h| sufficiently small, we may ensure that |t(h)| < δ, so that

|g(y + t(h)) − (g(y) + g 0 (y)t(h))|

< ε, (9.7)
|t(h)|
and then
|g(y + t(h)) − (g(y) + g 0 (y)t(h))| < ε|t(h)| < 2ε|f 0 (x)|h. (9.8)
Going back to (9.5) it follows that

|(g ◦ f )(x + h) − g(y) − g 0 (y)t(h)| < 2ε|f 0 (x)|h. (9.9)

Now, we recall that y = f (x) and t(h) = f 0 (x)h + o(|h|), which gives

|(g ◦ f )(x + h) − g(f (x)) − g 0 (f (x))f 0 (x)h − g 0 (f (x))o(|h|)| < 2ε|f 0 (x)|h. (9.10)

It follows that
(g ◦ f )(x + h) − g(f (x)) − g 0 (f (x))f 0 (x)h = o(|h|), (9.11)
and we are done. 

Exercise 9.6 Go over the proof and make sure you understand at each step how the notation o(|h|)
is used and why it makers sense and is all rigorous!

Theorem 9.7 (Rolle’s theorem) If f : [a, b] → R is continuous, f (a) = f (b) and f is differentiable
at each point x ∈ (a, b), then there is c ∈ (a, b) such that f 0 (c) = 0.

Proof. If f (x) ≡ f (a) on [a, b] then f 0 (x) = 0 for all x ∈ (a, b). Let us assume that f is not
equal identically to a constant on [a, b]. Then it either attains its maximum, or a minimum at some
c ∈ (a, b). Then the ratios
f (c + h) − f (c)
h
have different signs for h > 0 and h < 0. It follows that f 0 (x) = 0. 

Theorem 9.8 (Mean Value theorem) If f : [a, b] → R is continuous, and f is differentiable at each
point x ∈ (a, b), then there is c ∈ (a, b) such that

f (b) − f (a)
f 0 (c) = .
b−a
Proof. Apply Rolle’s theorem to the function
f (b) − f (a)
g(x) = f (x) − (x − a).
b−a
Note that g(a) = g(b) = f (a). 

39
9.2 The Taylor series representation
We now ask the question of when we can approximate a function to a higher order than linearly. So
far, we have seen that if f 0 (x0 ) exists then we can approximate f (x + h) by a linear function, up to
an error of the order o(h). Here is a more general result.
Theorem 9.9 Let f be a function differentiable up to order m + 1 on an interval (x0 − r, x0 + r).
Then for any x such that |x − x0 | < r, there exists c between x0 and x such that
m
X f (n) (x0 ) f (m+1) (c)
f (x) = (x − x0 )n + (x − x0 )m+1 .
n! (m + 1)!
n=0

Proof. Fix some x such that |x − x0 | < r and consider the function g(t) defined on |t − x0 | < r by
m
X f (n) (x0 )
g(t) = f (t) − (t − x0 )n − M (t − x0 )m+1 ,
n!
n=0

with the constant M chosen so that g(x) = 0 (keep in mind that both x and x0 are fixed, the
independent variable is t):
m
X f (n) (x0 )
1 
n

M= f (x) − (x − x 0 ) . (9.12)
(x − x0 )m+1 n!
n=0

As g(x0 ) = 0 and g(x) = 0, we deduce that there exists c1 between x0 and x such that g 0 (c1 ) = 0.
However, we also g 0 (x0 ) = 0 (check this!) – hence there exists c2 between x0 and c1 (and still between
x0 and x) such that g 00 (c2 ) = 0. But we also have g 00 (x0 ) = 0 (check this!), so there exists c3 between
x0 and c2 (and still between x0 and x) such that g 000 (c3 ) = 0 Note that

g (k) (x0 ) = 0, for all 0 ≤ k ≤ m,

hence we can continue this argument until step m + 1, funding a point cm+1 between x0 and x so
that g (m+1) (cm+1 ) = 0. Note that for any |t − x0 | < r we have

g (m+1) (t) = f (m+1) (t) − M (m + 1)!,

hence
f (m+1) (c)
M= .
(m + 1)!
Using this in (9.12) says that
m
X f (n) (x0 ) f (m+1) (cm+1 )
f (x) = (x − x0 )n + (x − x0 )m+1 ,
n! (m + 1)!
n=0

and we are done. 

Theorem 9.10 Let f (x) be differentiable to all orders in an interval |x − x0 | < r and assume that
there exists a constant C > 0 so that
 f (n) (x) 
 n
r ≤ C, for all n ∈ N and all x such that |x − x0 | < r,

n!


then we have
∞
X f (n) (x0 )
f (x) = (x − x0 )n , for all x such that |x − x0 | < r.
n!
n=0

40
Proof. By assumptions of this theorem and by Theorem 9.9, for any m ∈ N and any x ∈ (x0 −
r, x0 + r), we can find cm (x) between x0 and x so that
m
X f (n) (x0 ) f (m+1) (cm (x))
f (x) − (x − x0 )n = (x − x0 )m+1 .
n! (m + 1)!
n=0
Using the assumptions of the present theorem, we see that
m
 X f (n) (x0 )  |f (m+1) (c (x))|
m |f (m+1) (cm (x))| m+1 |x − x0 |m+1
f (x) − (x − x0 )n  ≤ |x − x0 |m+1 = r
 
n! (m + 1)! (m + 1)! rm+1
n=0
 x − x m+1
0
≤ C → 0 as m → +∞,

r

and we are done. 

9.3 Differentiability in Rn
Recall that for a scalar function f : R → R we say that a function f is differentiable at x if there
exists a number a ∈ R so that
f (x + h) = f (x) + ah + o(|h|), as h → 0,
and then we say that a = f 0 (x). For a scalar-valued function f : Rn → R the definition is very
similar. It is differentiable at a point x ∈ Rn if there exists a linear function L : Rn → R such that
f (x + h) = f (x) + L(h) + o(khk), as h → 0,
in the sense that
kf (x + h) − (f (x) + L(h))k
lim = 0.
h→0 khk
A linear function L : Rn → R must have the form L(h) = (v · h) with some v ∈ Rn . We call this
vector the gradient of f at x:
v = ∇f (x).
In other words, a scalar-valued function f : Rn → R is differentiable at a point x ∈ Rn if
f (x + h) = f (x) + (∇f (x) · h) + o(khk), as h → 0.
This definition can be extended to mappings f : Rn → Rm easily: a mapping f : Rn → Rm is
differentiable at a point x ∈ Rn if there exists a linear map L : Rn → Rm such that
f (x + h) = f (x) + L(h) + o(khk), as h → 0,
in the sense that
kf (x + h) − (f (x) + L(h))k
lim = 0.
h→0 khk
Recall that a linear map L : Rn → R must have the form L(h) = Ah with some m × n matrix A.
We call this matrix the derivative matrix of f (or the Jacobian matrix):
A = Df (x).
Lemma 9.11 If U ⊂ Rn and f : U → Rm is differentiable at a point x0 ∈ U , then f is continuous
at x0 .
Exercise 9.12 Adapt the proof of this fact for scalar valued functions of a real variable to prove
Lemma 9.11.
Definition 9.13 For a scalar-valued function f (x), the gradient of f at x0 is the vector Df (x0 ) if
it exists.

41
9.4 Directional derivatives
We now interpret the derivative matrix Df (x) in more concrete terms. Let U ⊂ Rn be an open set,
and f a map from U to Rm . Then, for each x ∈ U we can find a ball B(x, r) that is contained in U .
This means that for any v ∈ Rn the point x + tv s in U as long as |t| < r/|v| is sufficiently small, so
that the function
g(t) = f (x + tv)
of a real variable t ∈ R is defined for |t| < r/v.
Definition 9.14 Given a vector v ∈ Rn , v 6= 0, the directional derivative Dv f (x0 ) is
1
Dv f (x0 ) = lim (f (x0 + tv) − f (x0 )),
t→0 t

whenever that limit exists.

Note that if f maps U to Rm then Dv f (x0 ) is a (row) vector in Rm as well.
Exercise 9.15 Fix some λ ∈ R and v ∈ Rn , and let w = λv. Express Dw f (x0 ) in terms of Dv f (x0 ).
In the special case when v = ej , the j-th vector of the standard basis, we denote Dej f (x0 ) as

∂f (x0 )
Dej f (x0 ) = .
∂xj

Let us show that if the gradient matrix Df (x0 ) exists then directional derivatives can all be computed
from it.
Theorem 9.16 Let U ⊂ Rn be an open set and f : U → Rm be a map that is differentiable at a
point x0 ∈ U . Then the directional derivative in a direction v ∈ Rn , v 6= 0, has the form

Dv f (x0 ) = (Df (x0 ))v,

or, written component-wise, we have

n
X ∂fk (x0 )
Dv fk (x) = vj .
∂xj
j=1

Proof. First, as U is an open set, there exists r so that x0 + tv s in U for |t| < r, and the function

g(t) = f (x0 + tv)

is defined, as a mapping from (−r, r) to Rm . Also, as f is differentiable at x0 , we have

f (x0 + h) − f (x) = (Df (x0 ))h + o(khk), as h → 0,

with h ∈ Rn . Note that if some function α(h) = o(kh)k as h → 0, then α(tv) = o(t) for any v ∈ Rn
fixed (check this!), hence using h = tv above, we have

f (x0 + tv) − f (x) = (Df (x0 ))(tv) + o(t), as t → 0, (9.13)

with t ∈ (−r, r). This means exactly that

f (x0 + tv) − f (x0 )
lim = (Df (x0 ))v,
t→0 t

42
hence Dv f (x0 ) = (Df (x0 ))v, as claimed. Going back to (9.13) and writing it separately for each
coordinate fk , k = 1, . . . , m, gives
fk (x0 + tv) − fk (x) = (Dfk (x0 ))(tv) + o(t), as t → 0, (9.14)
However, Dfk is simply a row vector in Rm and (Dfk )(v) is the dot-product hDfk (x0 ), vi. Taking
v = ej shows that
∂fk (x0 )
(Dfk (x0 ))j = .
∂xj
It follows then that
n n
X X ∂fk (x0 )
Dv fk (x0 ) = (Dfk (x0 ))v = vj (Dfk (x0 ))j = vj ,
∂xj
j=1 j=1

and we are done. 

Note that existence of partial derivatives of f at x0 does not imply that the derivative matrix
Df (x0 ) exists. To see that, we may borrow an example from Section 6.3. Consider a function
f : R2 → R defined by  1
x1 + x2 sin , if x1 6= 0,

f (x1 , x2 ) = x1
 0, if x1 = 0.
Then we have f (x1 , 0) = x1 and f (0, x2 ) = 0, which means that
∂f ∂f
(0, 0) = 1, (0, 0) = 0,
∂x1 ∂x2
so if Df (0, 0) exists then Df = (1, 0), but if we look at x1 = x2 = h, then
1
f (h, h) − f (0, 0) = h + h sin ,
h
and the right side is not of the form h(1, 0), (h, h)i+o(khk).
Here is an extra condition that ensures that existence of partial derivatives implies existence of
the derivative matrix.
Theorem 9.17 Let f : U → Rn , x0 ∈ U ⊂ Rn and assume that the partial derivatives Dk f (x)
exists for all x in a ball B(x0 , ρ) with some ρ > 0 and are continuous at x0 . Then f is differentiable
at x0 .
Proof. We will assume without loss of generality that m = 1, so that f is real-valued and the
derivative ”matrix” is simply the gradient row-vector (if it exists). For general m > 1 one argue
simply component by component. Take h = (h, . . . , hn ) with |h| < ρ and write
f (x0 + h) − f (x0 ) = (f (x0 + h1 e1 ) − f (x0 )) + (f (x0 + h1 e1 + h2 e2 ) − f (x0 + h1 e1 )) + . . .
(9.15)
+ (f (x0 + h1 e1 + · · · + hn en ) − f (x0 + h1 e1 + · · · + hn−1 en−1 )).
The intermediate value theorem for functions of one variable implies that there exist ξ1 ∈ B(x0 , khk), ξ2 ∈
B(x0 , khk), . . . , ξn ∈ B(x0 , khk), such that
∂f (ξ1 )
f (x0 + h1 e1 ) − f (x0 ) = h1 ,
∂x1
∂f (ξ2 )
f (x0 + h1 e1 + h2 e2 ) − f (x0 + h1 e1 ) = h2 ,
∂x2
...,
∂f (ξn )
f (x0 + h1 e1 + · · · + hn en ) − f (x0 + h1 e1 + · · · + hn−1 en−1 ) = hn .
∂xn

43
Using this in (9.15) gives
∂f (ξ1 ) ∂f (ξ2 ) ∂f (ξn )
f (x0 + h) − f (x0 ) = h1 + h2 + · · · + hn . (9.16)
∂x1 ∂x2 ∂xn
∂f
However, as each ξk is contained in the ball B(x0 , khk) and each partial derivative ∂xk is continuous,
we know that
∂f (ξk ) ∂f (x0 )
= + o(1), as h → 0,
∂xk ∂xk
so that, as hk o(1) = o(khk) for any 1 ≤ k ≤ n fixed (check this!), we have
∂f (x0 ) ∂f (x0 ) ∂f (x0 )
f (x0 + h) − f (x0 ) = h1 + h2 + · · · + hn + o(khk), as h → 0. (9.17)
∂x1 ∂x2 ∂xn
This means precisely that f is differentiable at x0 and
 ∂f (x ) ∂f (x0 ) 
0
Df (x0 ) = ,..., ,
∂x1 ∂xn
and we are done. 

9.5 The chain rule

We now consider the chain rule for maps. Let U ⊂ Rn and V ⊂ Rm be two open sets, f be a
map from U to V and g a map from V to Rk . Assume that f is differentiable at x0 ∈ U and g
is differentiable at y0 = f (x0 ). Then the composition g ◦ f is a map from U to Rk . We will show
that (g ◦ f ) is differentiable at x0 and that the derivative matrix of (g ◦ f ) at x0 is

D(g ◦ f )(x0 ) = Dg(y0 )Df (x0 ). (9.18)

Note that Df (x0 ) is an m × n matrix, while Dg(y0 ) is a k × m matrix, so that the product
Dg(y0 )Df (x0 ) makes sense and is a k × n matrix, as it should be for a derivative matrix of a
map from Rn to Rk . The proof is quite similar to that for real-valued functions. First, note that
the continuity of f and g and the fact that V is open imply that there exists r > 0 small so that
f (x) ∈ V for all x ∈ B(x0 , r), so that g ◦ f is defined for all x ∈ B(x0 , r). Next, we write

f (x0 + h) = f (x0 ) + Df (x0 )h + o(khk), as h → 0, (9.19)

with h ∈ Rn , and
g(y0 + t) = g(y0 ) + Dg(y0 )t + o(ktk), as t → 0, (9.20)
with t ∈ Rm . Given h, let us take t = f (x0 + h) − f (x0 ), then, as y = f (x0 ), (9.19), together
with (9.20) say that

g(f (x0 ) + h) = g(f (x0 )) + Dg(y0 )(f (x0 + h) − f (x0 )) + r(h), (9.21)

with a function r(h) such that

r(h)
→ 0 as h → 0. (9.22)
kf (x0 + h) − f (x0 )k
It follows from (9.22) and (9.19) that actually
r(h)
→ 0 as h → 0, (9.23)
khk

44
that is, r(h) = o(h). Thus, (9.21) says that

g(f (x0 ) + h) = g(f (x0 )) + Dg(y0 )(f (x0 + h) − f (x0 )) + o(h). (9.24)

Next, use (9.19) for the difference f (x0 + h) − f (x0 ) in the right side above:

g(f (x0 ) + h) = g(f (x0 )) + Dg(y0 )(Df (x0 )h + o(|hk)) + o(h). (9.25)

Exercise 9.18 Go over the details on how we obtained (9.25), in particular, how (9.19) and (9.22)
imply (9.23).

It remains to observe that

Dg(y0 )o(khk) = o(khk),
to conclude that
g(f (x0 ) + h) = g(f (x0 )) + Dg(y0 )Df (x0 )h + o(khk), (9.26)
from which we conclude that g ◦ f is differentiable at x0 and

D(g ◦ f )(x0 ) = Dg(y0 )Df (x0 ), (9.27)

which is the chain rule.

An important special case is when g : Rm → R is a real-valued function. Let us define

p(x) = g(f (x)) = g(f1 (x), f2 (x), . . . , fm (x)).

Then the chain rule says that

Dp(x) = D(g ◦ f ) = Dg(f (x))Df (x),

where Dg is a row-vector of length m, and Df is an m × n matrix, so the product Dp is a row vector

of length n, with the entries
m
∂p(x) X ∂g(f (x)) ∂fl (x)
= . (9.28)
∂xj ∂yl ∂xj
l=1

9.6 Higher order derivatives in Rn

In this section, we will look at higher order derivatives of a map f : Rn → Rm and show that ”the
order of differentiation does not matter”. As by now we know that the derivative matrix Df has
rows that are the gradient vectors of each component fk : Rn → R, and that differentiability of f
is equivalent to differentiability of all components of f , we will assume that f : Rn → R is a scalar
valued function. We say that a function f is twice differentiable at a point x0 if the gradient vector
Df , viewed as map from Rn → Rn is a differentiable map. We denote the second order partial
derivatives of f as
∂2f
,
∂xi ∂xj
or as Di Dj f , and, more generally, k-th order partial derivatives of f as

∂kf
,
∂xi1 ∂xi2 . . . ∂xik

or as Di1 Di2 . . . Dik f , with some collection of indices 1 ≤ ik ≤ n.

45
Theorem 9.19 Let U ⊂ Rn be an open set, with x0 ∈ U . Assume that a function f is twice
differentiable at x0 , and that the partial derivatives Di Dj f and Dj Di f are continuous at x0 . Then
Di Dj f (x0 ) = Dj Di f (x0 ).

Proof. Let ei and ej be the standard basis vectors, and for h, t ∈ R sufficiently small, consider the
four points x0 , x0 + tej , x0 + hei , and x0 + tej + hei . Then the ”second order difference”

φ(h, t) = f (x0 + hei + tej ) + f (x0 ) − f (x0 + hei ) − f (x0 + tej )

can be written in two ways: first, as

φ(h, t) = [f (x0 + hei + tej ) − f (x0 + hei )] − [f (x0 + tej ) − f (x0 )]. (9.29)

but also as
φ(h, t) = [f (x0 + hei + tej ) − f (x0 + tej )] − [f (x0 + hei ) − f (x0 )]. (9.30)
Looking at (9.30), let us define the function g(x), with h fixed, as

g(x) = f (x + hei ) − f (x), (9.31)

so that (9.30) says that

φ(h, t) = g(x0 + tej ) − g(x0 ). (9.32)
The intermediate value theorem applied to (9.32) implies that there exists t1 between 0 and t so
that
φ(h, t) = (Dj g)(x0 + t1 ej )t. (9.33)
Note that, differentiating (9.31) gives

Dj g(x) = Dj f (x + hei ) − Dj f (x), (9.34)

Again, as Dj f is differentiable, the intermediate value theorem applied to Dj f implies that the
difference in the right side of (9.34) can be written as

Dj g(x) = hDi Dj f (x + h1 ei ), (9.35)

with some h1 between 0 and h. Using this in (9.33) with x = x0 + t1 ej gives

φ(h, t) = ht(Di Dj f )(x0 + t1 ej + h1 ei ). (9.36)

Next, let us use (9.29) rather than (9.30). Define the function p(x), with t fixed, as

p(x) = f (x + tej ) − f (x), (9.37)

so that (9.29) says that

φ(h, t) = p(x0 + hei ) − p(x0 ). (9.38)
The intermediate value theorem implies that there exists h2 between 0 and h so that

φ(h, t) = (Di p)(x0 + h2 ei )h. (9.39)

Note that, differentiating (9.37) gives

Di p(x) = Di f (x + tej ) − Di f (x). (9.40)

46
Again, as Di f is differentiable, the intermediate value theorem implies that the difference in the
right side can be written as
Di p(x) = tDj Di f (x + t2 ej ), (9.41)
with some t2 between 0 and t. Using this in (9.39) with x = x0 + h2 ei gives

φ(h, t) = ht(Dj Di f )(x0 + t2 ej + h2 ei ). (9.42)

Now, comparing (9.36) and (9.42) we deduce that

(Di Dj f )(x0 + t1 ej + h1 ei ) = (Dj Di f )(x0 + t2 ej + h2 ei ). (9.43)

Note that t1 , h1 , t2 , h2 in (9.43) all depend on t and h but tend to zero as t, h → 0. Moreover, both
Di Dj f and Dj Di f are continuous at x0 . Passing to the limit in (9.43) gives, therefore:

(Di Dj f )(x0 ) = (Dj Di f )(x0 ), (9.44)

and we are done. 

As a corollary, we deduce the following: if f is k times differentiable at a point x0 , and all of its
derivatives of order k are continuous, then the partial derivative

Di1 Di2 . . . Dik f

does not depend on the order in which these derivatives are taken.

9.7 Critical points in R

Let us now investigate the critical points of a function f : R → R, a real-valued function of a real
variable. We say that x0 is a local maximum of a function f (x) if there exists R0 > 0 so that we
have f (x) ≤ f (x0 ) for all x ∈ (x0 −R0 , x0 +R0 ). Similarly, x0 is a local minimum of a function f (x) if
there exists R0 > 0 so that we have f (x) ≥ f (x0 ) for all x ∈ (x0 − R0 , x0 + R0 ). In other words, there
exists an interval I around x0 so that f attains its maximum (minimum) on I at the point x0 . Recall
that we have already proved that if f attains it maximum or minimum over an open interval I at a
point x0 ∈ I, then f 0 (x0 ) = 0. In order to be able to say whether x0 is a maximum or a minimum, we
use the second derivative test. In order to make the presentation very easily adaptable to functions
on Rn , we begin with the following lemma.

Lemma 9.20 Let f be twice continuously differentiable on an interval (a, b) and x0 ∈ (a, b). Assume
that f 0 (x0 ) = f 00 (x0 ) = 0, then
f (x0 + h) − f (x0 ) = o(h2 ).

Proof. The intermediate value theorem implies that, given h, we can find ξ1 (h) that is between x0
and x0 + h so that
f (x0 + h) − f (x0 ) = f 0 (ξ1 (h))h. (9.45)
Next, the intermediate value theorem applied to the function f 0 (x), implies that there exists ξ2 (h)
that is between x0 and ξ1 (h) so that

f 0 (ξ1 (h)) − f 0 (x0 ) = f 00 (ξ2 (h))(ξ1 (h) − x0 ). (9.46)

As f 0 (x0 ) = 0, this is simply

f 0 (ξ2 (h)) = f 00 (ξ2 (h))(ξ1 (h) − x0 ). (9.47)

47
Using this in (9.45) gives

f (x0 + h) − f (x0 ) = f 00 (ξ2 (h))h(ξ1 (h) − x0 ). (9.48)

As |ξ1 (h) − x0 | ≤ h, we deduce that

|f (x0 + h) − f (x0 )| ≤ |f 00 (ξ2 (h))|h2 , (9.49)

so that
|f (x0 + h) − f (x0 )|
≤ |f 00 (ξ2 (h))|. (9.50)
h2
Now, as f 00 (x) is continuous at x0 , and ξ2 (h) → x0 as h → 0, we know that f 00 (ξ2 (h)) → f 00 (x0 ) = 0
as h → 0. Thus, we have
f (x0 + h) − f (x0 ) = o(h2 ),
and we are done. 

Theorem 9.21 Let f be twice continuously differentiable at a point x0 . Assume that f 0 (x0 ) = 0,
then (i) if f 0 (x0 ) > 0 then x0 is a local minimum of f and (ii) if f 00 (x0 ) < 0 then x0 is a local
maximum of f .

Proof. Let us assume that f 00 (x0 ) > 0. Consider the function

1
g(x) = f (x0 ) + f 00 (x0 )(x − x0 )2 ,
2
and set
p(x) = f (x) − g(x).
Note that g 0 (x0 ) = 0 and g 00 (x0 ) = f 0 (x0 ), so that

p(x0 ) = 0, p0 (x0 ) = 0, p00 (x0 ) = 0.

It follows from Lemma 9.20 that p(x0 + h) = o(h2 ), so that

f (x0 + h) = g(x0 + h) + o(h2 ),

so that
1
r(h) = f (x0 + h) − f (x0 ) − f 00 (x0 )h2 = o(h2 ). (9.51)
2
Thus, we can choose R0 so that for all h ∈ (−R0 , +R0 ) we have
1 00
|r(h)| ≤ f (x0 )h2 .
10
Using this in (9.51) gives
 1  1
f (x0 + h) − f (x0 ) − f 00 (x0 )h2  ≤ f 00 (x0 )h2 , for all |h| < R0 ,
 
2 10
and thus
1 1
f (x0 + h) − f (x0 ) ≥ f 00 (x0 )h2 − f 00 (x0 )h2 > 0, for all |h| < R0 ,
2 10
thus x0 is a local minimum of f if f (x0 ) > 0. The case f 0 (x0 ) < 0 is essentially identical. 
00

48
9.8 A digression on the quadratic forms
A quadratic form is an expression of the form
n
X
F (ξ) = qij ξi ξj .
i,j=1

Here, ξ ∈ Rn is a vector that we think of as an ”independent variable”, and the n × n symmetric

matrix Q with the entries qij is fixed.

Exercise 9.22 Why can we assume that qij = qji so that the matrix Q is symmetric?

This is a natural generalization of the quadratic functions in R that do not have a linear part: in
that case, F (ξ) = qξ 2 , where q ∈ Rn is a number, and ξ ∈ R is an independent variable. Notice
that, depending on the sign of the number q, we either have F (ξ) ≥ 0 for all ξ ∈ R or F (ξ) ≤ 0 for
all ξ ∈ R. In higher dimensions n > 1, it is also possible that a quadratic form may be positive for
all ξ ∈ Rn , for instance for
F (ξ) = a1 ξ12 + a2 ξ22 + · · · + an ξn2 ,
with all aj ≥ 0, 1 ≤ j ≤ n. It may also happen that a quadratic form may be negative for all ξ ∈ Rn ,
for instance for
F (ξ) = a1 ξ12 + a2 ξ22 + · · · + an ξn2 ,
with all aj ≤ 0, 1 ≤ j ≤ n. It is also possible that a quadratic form may take both positive and
negative values: take n = 2 and consider

F (ξ) = ξ12 − ξ22 , ξ = (ξ1 , ξ2 ) ∈ R2 .

Then, for instance F (1, 0) = 1 > 0 and F (0, 1) = −1 < 0.

In order to understand what is happening in general, let us write
n
X n X
X n  n
X
F (ξ) = qij ξi ξj = qij ξj ξi = v i ξi , (9.52)
i,j=1 j=1 i=1 i=1

with the vector v = (v1 , . . . , vn ) ∈ Rn having the components

n
X
vi = qij ξj , i = 1, . . . , n.
j=1

Note that v = Qξ so that we can re-write (9.52) as

F (ξ) = (Qξ · ξ). (9.53)

The matrix Q is an n × n symmetric matrix. The spectral theorem tells us that it has n eigenvalues
λ1 , . . . , λn taken with multiplicities. We denote the corresponding eigenvectors v1 , . . . , vn :

Qvk = λk vk . (9.54)

Note that if λk 6= λj then the eigenvectors vk and vj are orthogonal. To see that, write

Qvk = λk vk
(9.55)
Qvj = λj vj ,

49
and take the dot product of the first equation with vj and of the second with vk . This gives

(Qvk · vj ) = λk (vk · vj )
(9.56)
(Qvj · vk ) = λj (vj · vk ).

Note that as the matrix Q is symmetric, we have QT = Q, so that

(Qvk · vj ) = (vk · (QT vj )) = (vk · Qvj ) = (Qvj · vk ).

With this identity in hand, subtracting the second equation in (9.56) from the first gives

(λk − λj )(vk · vj ) = 0.

As λk 6= λj , it follows that (vk · vj ) = 0.

Exercise 9.23 Use the above to show that if Q is symmetric then there exists an orthonormal basis
{v1 , . . . , vn } of Rn so that each vj is an eigenvector of Q.

This basis is extremely useful for the quadratic form F (ξ) = (Qξ · ξ). Given a vector ξ, let
c1 (ξ), . . . cn (ξ) be its coordinates in that basis:

ξ = c1 (ξ)v1 + · · · + cn (ξ)vn .

As the basis vk is orthonormal, we have (vk · vj ) = 0 if k 6= j and (vk · vk ) = 1 for all 1 ≤ k ≤ n, so

that
|ξ|2 = (ξ · ξ) = |c1 (ξ)|2 + · · · + |cn (ξ)|2 . (9.57)
Furthermore, we may write Qξ as

Qξ = c1 (ξ)Qv1 + · · · + cn (ξ)Qvn = c1 (ξ)λ1 v1 + · · · + cn (ξ)λn vn ,

thus
F (ξ) = (Qξ · ξ) = (c1 (ξ)λ1 v1 + · · · + cn (ξ)λn vn ) · (c1 (ξ)v1 + · · · + cn (ξ)vn ). (9.58)
As the basis vk is orthonormal, we have (vk · vj ) = 0 if k 6= j and (vk · vk ) = 1 for all 1 ≤ k ≤ n.
Using this in (9.58) gives

F (ξ) = (Qξ · ξ) = λ1 |c1 (ξ)|2 + · · · + λn |cn (ξ)|2 . (9.59)

It follows that if we set

m = min λk , M = max λk , (9.60)
1≤k≤n 1≤k≤n

then
m(|c1 (ξ)|2 + · · · + |cn (ξ)|2 ) ≤ F (ξ) = (Qξ · ξ) ≤ M (|c1 (ξ)|2 + · · · + |cn (ξ)|2 ). (9.61)
With the help of (9.57), this becomes

mkξk2 ≤ F (ξ) = (Qξ · ξ) ≤ M kξk2 , (9.62)

with m and M as in (9.60). Let us summarize this as the following theorem.

50
Theorem 9.24 Let Q be an n × n symmetric matrix with eigenvalues λ1 , . . . , λn (taken with mul-
tiplicities), and set
m = min λk , M = max λk , (9.63)
1≤k≤n 1≤k≤n

then for any ξ ∈ Rn we have

mkξk2 ≤ (Qξ · ξ) ≤ M kξk2 . (9.64)
Moreover, there exists ξ ∈ Rn , ξ 6= 0, so that (Qξ · ξ) = mkξk2 , and η ∈ Rn , η 6= 0, so that
(Qξ · ξ) = M kξk2 .

Definition 9.25 We say that a quadratic form F (ξ) on Rn is positive definite if there exists m > 0
so that for any ξ ∈ Rn we have
m|ξ|2 ≤ F (ξ). (9.65)
We say that a quadratic form F (ξ) on Rn is negative definite if there exists m < 0 so that for any
ξ ∈ Rn we have
m|ξ|2 ≥ F (ξ). (9.66)

Corollary 9.26 (i) A quadratic form F (ξ) defined on Rn is positive definite if all eigenvalues of
the corresponding symmetric positive definite n × n matrix Q are positive.
(ii) A quadratic form F (ξ) defined on Rn is negative definite if all eigenvalues of the corresponding
symmetric positive definite n × n matrix Q are negative.

Another function one can associate to a symmetric n× matrix Q is a bilinear form G(ξ, η) defined
for a pair of vectors ξ ∈ Rn and η ∈ Rn as
n
X n X
X n 
G(ξ, η) = qij ξi ηj = qij ξj ηi = (Qξ · η). (9.67)
i,j=1 i=1 j=1

Since Q is symmetric, we have the identity

1
(Qξ · η) = [(Q(ξ + η) · (ξ + η)) − (Q(ξ − η) · (ξ − η))],
4
which allows us to write
1
G(ξ, η) = [F (ξ + η) − F (ξ − η)],
4
hence the values of the bilinear form are determined by the values of the quadratic form. Note that
the corresponding quadratic form is simply F (ξ) = G(ξ, ξ).

Exercise 9.27 Show that the entries of the matrix Q can be expressed in terms of the bilinear form
as
1
qij = [F (ei + ej ) − F (ei − ej )].
4

9.9 The second order Taylor polynimial in Rn

Let us now consider the analog of the Taylor series in Rn . Recall that for an (m + 1)-differentiable
function f (x) of a single variable we have the Taylor formula

f (m+1) (ξ) m+1

f (x0 + h) = Pm (h) + h ,
(m + 1)!

51
where ξ is a point between x0 and x0 + h, and the Taylor polynomial is
m
X f (k) (x0 )
Pm (h) = hk .
k!
k=0

The polynomial Pm (h) was constructed so that for all 0 ≤ k ≤ m we have

dk  dk 
f (x + h) = P (h) .
 
k 0 k m
dh dh

h=0 h=0

We can do a similar procedure for functions on Rn but we will only do this in detail for the quadratic
Taylor polynomials. We are given a function f : B(x0 , r) → R that is twice differentiable in a ball
B(x0 , r) and such that its second derivative matrix D2 f is continuous in B(x0 , r). Our goal is to
find a vector b and a matrix Q with entries qij such that we have

1
f (x0 + h) = f (x0 ) + (b · h) + (Qh · h) + o(khk2 ). (9.68)
2
Note that
1
(Qh · h) + o(khk2 ) = o(khk),
2
hence we must have b = Df (x0 ), so that (9.68) becomes
1
f (x0 + h) = f (x0 ) + (Df (x0 ) · h) + (Qh · h) + o(khk2 ). (9.69)
2
If we blindly differentiate both sides of (9.68) twice in the directions ei and ej and manage to argue
that we may pass to the limit h → 0, disregarding the derivative of the o(khk2 )-term in (9.68), we
would get that
∂ 2 f (x0 )
= qij , (9.70)
∂xi ∂xj
that is, the matrix Q is Q = D2 f (x0 ). Now, the question is whether (9.68) does, indeed, hold with
b = Df (x0 ) and Q = D2 f (x0 ). Let us start with the following lemma.

Lemma 9.28 Let f be twice differentiable in a ball B(x0 , r), and D2 f (x) be continuous in B(x0 , r).
Assume that Df (x0 ) = 0 and D2 f (x0 ) = 0, then

f (x0 + h) − f (x0 ) = o(khk2 ), as h → 0. (9.71)

Proof. Let us fix h ∈ B(x0 , r) and consider the function

g(t) = f (x0 + th) − f (x0 ), t ∈ [−1, 1],

of a scalar variable t. Note that g(0) = 0 and, using the definition of the partial derivatives of f and
also the partial derivatives of ∂f /∂xj , we can compute
n
X ∂f (x0 + th)
g 0 (t) = (h · Df (x0 + th)) = hj ,
∂xj
j=1
n n n n (9.72)
00
X  ∂f (x0 + th)  X X ∂ 2 f (x0 + th) X ∂ 2 f (x0 + th)
g (t) = hj h · D = hj hi = hi hj .
∂xj ∂xi ∂xj ∂xi ∂xj
j=1 j=1 i=1 i,j=1

52
The intermediate value theorem tells us that

g(1) − g(0) = g 0 (c1 ), (9.73)

with some c1 ∈ (0, 1). Applying the intermediate value theorem again we conclude that

g 0 (c1 ) = g 00 (c2 )c1 , (9.74)

with some c2 ∈ (0, c1 ). Using this in (9.73) gives

g(1) − g(0) = g 00 (c2 )c2 . (9.75)

Let us use the definition of g(t) and the fact that g(0) = 0 in (9.75):

f (x0 + h) = g 00 (c2 )c2 . (9.76)

Now, let us replace g 00 (c2 ) by the corresponding expression in (9.72):

n
X ∂ 2 f (x0 + c2 h)
f (x0 + h) = hi hj c2 . (9.77)
∂xi ∂xj
i,j=1

As D2 f (x0 ) = 0 and D2 f (x) is continuous at x0 , given any ε > 0 we can find δ > 0 so that
 ∂ 2 f (x + z) 
0
 < ε for all z ∈ Rn such that kz − x0 k < δ. (9.78)
 
∂xi ∂xj


As |c2 | < 1, as soon as |h| < δ, we have

n  ∂ 2 f (x + c h)  n
0 2
X X
|f (x0 + h)| ≤ |hi hj | ≤ khk2 ε ≤ n2 khk2 ε. (9.79)
 
∂xi ∂xj

i,j=1 i,j=1

It follows that kf (x0 + h)| = o(khk2 ), and we are done. 

Now we can prove the second order approximation theorem.

Theorem 9.29 Let f be twice differentiable in a ball B(x0 , r), and D2 f (x) be continuous in B(x0 , r).
Assume that Df (x0 ) = 0 and D2 f (x0 ) = 0, then
1
f (x0 + h) = f (x0 ) + (Df (x0 ) · h) + (D2 f (x0 )h · h) + o(khk2 ), as h → 0. (9.80)
2
Proof. This is a simple consequence of Lemma 9.28. Define the function
1
f˜(x) = f (x) − f (x0 ) − (Df (x0 ) · (x − x0 )) − (D2 f (x0 )(x − x0 ) · (x − x0 )).
2

Note that the f˜(x0 ) = 0, Df˜(x0 ) = 0 and D2 f˜(x0 ) = 0. Moreover, D2 f˜(x) is continuous at x0 .
It follows from Lemma 9.28 that f˜(x) = o(kx − x0 k2 ), which is exactly the claim of the present
theorem. 

53
9.10 The critical points of a function and the second derivative test
Now, we are ready to discuss conditions for a function f : Rn → R to attain a local maximum or
a local minimum at a point x0 . Recall that a function f defined on a open set U attains a local
maximum at a point x0 ∈ U if there is exists r > 0 so that

f (x) ≤ f (x0 ) for all x ∈ U such that |x − x0 | < r. (9.81)

Similarly, f attains local minimum at a point x0 ∈ U if there is exists r > 0 so that

f (x) ≥ f (x0 ) for all x ∈ U such that |x − x0 | < r. (9.82)

Note that if f is differentiable at a point x0 and attains a local maximum or a local minimum at x0
then we must have Dk f = 0 for all 1 ≤ k ≤ n, which means that Df (x0 ) = 0. To see that, assume
that r attains a local maximum at x0 , and for a given k define the function

g(t) = f (x + tej ),

of a real variable t ∈ (−r, r). Then g(t) attains its maximum on the interval (−r, r) at the point
t = 0, hence g 0 (0) = 0, which exactly means that Dk f (x0 ) = 0. The question is whether we can
determine if a given x0 such that Df (x0 ) = 0 is a local maximum, a local minimum or neither. Note
that any of the three possibilities are possible: simply look at f (x) = x21 + x22 , g(x) = −x21 − x22 , and
p(x) = x21 − x22 at x0 = (0, 0) ∈ R2 . All three functions have a gradient equal to zero at x0 , and f
attains a local minimum at x0 , g attains a local maximum at x0 but for p(x) the point x0 is neither
a local maximum nor a local minimum.
For the rest of this discussion, see Section 7 in Chapter 2 of the Simon book, and Sections
8.4.3.-8.4.5 in Zorich.

10 A crash course on integrals

10.1 Extension of a continuous function from a dense subset
Before we turn to the integral, let us consider the following question. Let X be a metric space,
and S be a dense sub-set of X. Suppose we have a real-valued continuous function f that is defined
on S, f : S → R, the question is if we can extend f to all of X so that the extension is continuous.
In other words, can we construct a function f¯ : X → R such that f¯ is continuous on X and for s ∈ S
we have f¯(s) = f (s)?
First, let us find a candidate for what f¯ should be. Given x ∈ X, since S is a dense subset
of X, there exists a sequence sn → x. If f¯ is continuous on X, then we must have f¯(sn ) → f¯(x)
as n → ∞. Since f¯ = f on S, we should have f¯(sn ) = f (sn ). This brings the following possibility:
let us define f¯(x) for x 6∈ S as follows – take a sequence sn → x and set

f¯(x) = lim f (sn ). (10.1)

n→∞

This brings two immediate questions: (i) does the limit in (10.1) exist, and (ii) if the limit exists,
will it be the same for all sequences sn that converge to a given point x 6∈ S?
Let us make a stronger assumption than that f is continuous on S: let us assume that f is
uniformly continuous on S, so that for any ε > 0 there exists δε > 0 such that for any s1 , s2 ∈ S
with d(s1 , s2 ) < δε we have |f (s1 )−f (s2 )| < ε. Now, as the sequence sn converges to x, this sequence
is Cauchy, hence for any ε > 0 there exists Nε so that d(sn , sm ) < δε , with δε as above. Then we

54
have |f (sn ) − f (sm )| < ε for all n > Nε . It follows that the sequence f (sn ) is Cauchy, hence the
limit in (10.1) exists. This answers (i). Furthermore, (ii) has also been answered – indeed, take
another sequence s0n that converges to x and consider the alternating sequence s1 , s01 , s2 , s02 , . . . , that
is s00k = sk if k is even and s00k = s0k if k is odd. The sequence s00k also converges to x, hence, by what
we have just shown, the limit f (s00k ) exists. But then the limits of f (sk ) and f (s0k ) must coincide.
Exercise 10.1 Fill in the details in proof of the claim that
lim f (sn ) = lim f (s0n )
n→∞ n→∞
above.
Now we know that f¯(x) is well-defined for x 6∈ S. It remains to show that f¯(x) is a continuous on X.
Since f is uniformly continuous on S, given ε > 0, we can choose δ > 0 so that for all s1 , s2 ∈ S
such that d(s1 , s2 ) < δ we have |f (s1 ) − f (s2 )| < ε/4. Let us take x, y ∈ X such that d(x, y) < δ/4.
As S is dense in X, we can find two sequences sn , s0n ∈ S such that
lim sn = x, lim s0 = y,
n→∞ n→∞ n
and then
f¯(x) = lim f (sn ), f¯(y) = lim f (s0n ).
n→∞ n→∞
Let us choose N sufficiently large, so that all of the following hold:
d(sN , x) < δ/4, d(s0N , y) < δ/4, |f¯(x) − f (sN )| < ε/4, |f¯(y) − f (s0N )| < ε/4. (10.2)
The first two inequalities in (10.2) imply that
δ δ δ
d(sN , s0N ) < d(sN , x) + d(x, y) + d(y, s0N ) < + + < δ,
4 4 4
thus the way we have chosen δ implies that |f (sN ) − f (s0N )| < ε/4. Now, the last two inequalities
in (10.2) imply that
ε ε ε
|f¯(x) − f¯(y)| ≤ |f¯(x) − f (sN )| + |f (sN ) − f (s0N )| + |f¯(y) − f (s0N )| < + + < ε.
4 4 4
¯
It follows that the function f is uniformly continuous on X. Thus, we have proved the following
theorem.
Theorem 10.2 Let X be a metric space and S its dense subset. Suppose that a function f : S → R
is uniformly continuous on S. Then there exists a function f¯ : X → R that is uniformly continuous
on X and such that f¯(s) = f (s) for all s ∈ S.

10.2 The integral of a continuous function

We now give a brief and short-cutty way to define the integral of a continuous function on an
interval [a, b] using the above strategy of extension. There is a class of functions on which one
”knows” what the integral should be if we think of the integral as the area under the graph of a
function. Namely, if f : [a, b] → R is an affine function, that is,
f (t) = αt + β
with some α, β ∈ R, then by the area of a triangle formula we should have
Z b  a+b 
f (x)dx = α + β (b − a). (10.3)
a 2
Here, (b − a)β is the are of the rectangle under the graph, and α(b − a)(a + b)/2 is the area of the
triangle – draw a picture to clarify this!

55
The integral of a piecewise affine continuous function
The next clear property the integral should have is that the integral is additive with respect to
”interval splitting”: for a < b < c we should have
Z c Z b Z c
f (x)dx = f (x)dx + f (x)dx. (10.4)
a a b

This tells us what the integral must be for piecewise affine functions. A function is piecewise affine
if there exists a partition a = t0 < t1 < . . . < tn = b of the interval [a, b] so that f (t) = αi t + βi on
each interval t ∈ [ti−1 , ti ], with αi , βi ∈ R. For piecewise affine functions we should have
Z b n 
X ti + ti−1 
f (x)dx = αi + βi (ti − ti−1 ). (10.5)
a 2
i=1

Note that we require that the piecewise affine functions are continuous: the values at ti on the left
and on the right must match. Our goal is to extend the notion of the integral from piecewise affine
to all continuous functions.
It is not completely obvious that the above definition of the integral makes sense even for piecewise
affine functions. The issue is that if f is piecewise affine, there may be many ways of choosing the
partition points ti . For instance, if one such partition works, one could always refine it by adding
more division points. One then has to show that the result is independent of the particular choice
of division points, subject to f being affine on each subinterval.
Exercise 10.3 Show that if f is a piecewise affine function then the integral defined by the right
side of (10.5) does not depend on the choice of the partition. Hint: given two such partitions,
take their common refinement, consisting of all the division points in either, and then show that
the integrals defined using the original two partitions are equal to that defined using their common
refinement, and thus to each other.
Once one knows that the integral (10.5) is well-defined on the set D = D([a, b]) of piecewise affine
functions, one can show the following basic properties of the integral for piecewise affine functions.
Proposition 10.4 (i) Linearity: let f1 , f2 ∈ D([a, b]) and c1 , c2 ∈ R, then
Z b Z b Z b
(c1 f1 (x) + c2 f2 (x))dx = c1 f1 (x)dx + c2 f2 (x)dx.
a a a

(ii) Positivity: if f ∈ D([a, b]), and f (x) ≥ 0 for all x ∈ [a, b], then
Z b
f (x)dx ≥ 0.
a

(iii) Boundedness: if f ∈ D([a, b]) then

Z b  Z b
f (x)dx ≤ |f (x)|dx ≤ (b − a)kf k, (10.6)
 

a a

where kf k is the norm on C(a, b]): kf k = sup{|f (x)| : x ∈ [a, b]}.

(iv) Additivity: if a < b < c and f ∈ D([a, c]) then
Z b Z c Z c
f (x)dx + f (x)dx = f (x)dx.
a b a

56
Exercise 10.5 Prove Proposition 10.4. Hint: for (i) you may notice that c1 f1 + c2 f2 is an affine
function with respect to the partition that is a refinement both of the partition for f1 and f2 and
compute the integral with respect to that partition – recall that the integral does not depend on the
partition. In parts (ii)-(iv) you should use the definition of the integral directly.

Positivity plus linearity imply that if f, g ∈ D and f (x) ≥ g(x) for all x ∈ [a, b], then
Z b Z b
f (x)dx ≥ g(x)dx.
a a

Indeed, we have
Z b Z b Z b
f (x)dx − g(x)dx = (f (x) − g(x))dx ≥ 0
a a a
where the first equality uses the linearity of the integral and the second the positivity of the integral
of f − g ≥ 0.

Approximation of continuous functions by piecewise affine functions

Next, we will extend the definition of the integral from piecewise affine functions on [a, b] to all
continuous functions on [a, b]. First, we need to show that that a continuous function can be
approximated by a sequence of piecewise affine functions.

Lemma 10.6 Let f ∈ C[a, b] be a continuous function. There is a sequence of piecewise affine
functions fn ∈ C[a, b] such that fn → f in C[a, b].

Proof. As f is continuous on a closed interval [a, b], f is uniformly continuous on that interval.
Therefore, given n ∈ N we can find δn so that |f (x) − f (y)| < 1/n for all x, y such that |x − y| < δn .
Consider now a partition a = t0 < t1 · · · < tk = b of the interval [a, b] such that |tj − tj−1 | < δn for
all 1 ≤ j ≤ k. Consider a piecewise affine function fn so that fn (t) is linear on each interval [tj−1 , tj ]
and fn (tj ) = f (tj ) for all 0 ≤ j ≤ k – draw a picture to visualize fn ! Note that fn depends on n
through the partition. Then, for each t ∈ [a, b], we can find 1 ≤ j ≤ k so that t ∈ [tj−1 , tj ]. Then,
we have
|fn (t) − f (t)| ≤ |fn (t) − fn (tj )| + |fn (tj ) − f (tj )| + |f (tj ) − f (t)|. (10.7)
The basic properties of linear functions imply that the first term in (10.7) can be estimated as
1
|fn (t) − fn (tj )| ≤ |fn (tj−1 − fn (tj )| = |f (tj−1 ) − f (tj )| ≤ , (10.8)
n
since |tj−1 −tj | < δn for all j. The second term in the right side of (10.7) vanishes since fn (tj ) = f (tj ),
and the last one can be bounded by
1
|f (tj ) − f (t)| < , (10.9)
n
because |tj − t| < δn . It follows that for all t ∈ [a, b] we have

2
|fn (t) − f (t)| ≤ , (10.10)
n
hence kfn − f k → 0 as n → +∞, thus fn → f in C[a, b]. 

57
Defining the integral of a continuous function
Suppose f ∈ C([a, b]), and let fn ∈ D[a, b] be a sequence such that fn → f , in C[a, b]. Note that by
Lemma 10.6 at least one such sequence exists but of course it is not unique. We would like to show,
first, that
Z b
lim fn (x)dx
n→∞ a

exists and, second, that it is independent of the choice of the particular sequence in D[a, b] that
converges to f . If we show both of these properties, then we can define
Z b Z b
f (x)dx = lim fn (x)dx. (10.11)
a n→∞ a

Lemma 10.7 Let fn ∈ D[a, b] be a sequence of piecewise affine functions that converges to a func-
tion f ∈ C[a, b], then the limit
Z b
lim fn (x)dx
n→∞ a

exists.
Proof. As the sequence fn converges in C[a, b], it is a Cauchy sequence in C[a, b]. Thus, for all ε > 0
there is N such that n, m ≥ N imply kfn − fm k < ε/(b − a). The linearity property (i) and the
boundedness property (iii) in Proposition 10.4 imply that
Z b Z b  Z b 
f − f = (f − fm  ≤ (b − a)kfn − fm k < ε.
)
   
 n m  n
a a a

It follows that the sequence of numbers

Z b
In = fn (x)dx
a

is a Cauchy sequence in R, thus it converges. 

Lemma 10.8 Let fn ∈ D[a, b] and gn ∈ D[a, b] be two sequences of piecewise affine functions
in D[a, b] that both converge to the same function f ∈ C[a, b], then
Z b Z b
lim fn (x)dx = lim gn (x)dx. (10.12)
n→∞ a n→∞ a

Proof. Consider the alternating sequence f1 , g1 , f2 , g2 , f3 , g3 . . ., that is, the sequence hn with

h2k−1 = fk , h2k = gk .

This sequence also converges to f , as follows easily from the definition of convergence. Thus,
Lemma 10.7 implies that
Z b
lim hn (x)dx
n→∞ a

exists, but then all subsequences of

Z b
hn (x)dx
a
converge to this very same limit, and (10.12) follows. 

58
 Putting these together, we can define
Z b
f
a
for f ∈ C([a, b]) as follows: take any sequence fn in D[a, b] such that fn → f in C[a, b], and set
Z b Z b
f = lim fn . (10.13)
a n→∞ a

This argument remains valid if the the target space R of f is replaced by any complete normed vector
space V , such as Rm . Complete normed vector spaces are called Banach spaces.
It is straightforward to check that the integral inherits the properties from D[a, b] in Proposi-
tion 10.4.
Proposition 10.9 (i) Linearity: let f1 , f2 ∈ C([a, b]) and c1 , c2 ∈ R, then
Z b Z b Z b
(c1 f1 (x) + c2 f2 (x))dx = c1 f1 (x)dx + c2 f2 (x)dx. (10.14)
a a a

(ii) Positivity: if f ∈ C([a, b]), and f (x) ≥ 0 for all x ∈ [a, b], then
Z b
f (x)dx ≥ 0. (10.15)
a

(iii) Boundedness: if f ∈ C([a, b]) then

Z b  Z b
f (x)dx ≤ |f (x)|dx ≤ (b − a)kf k, (10.16)
 

a a

where kf k is the norm on C(a, b]): kf k = sup{|f (x)| : x ∈ [a, b]}.

(iv) Additivity: if a < b < c and f ∈ C([a, c]) then
Z b Z c Z c
f (x)dx + f (x)dx = f (x)dx. (10.17)
a b a

Proof. Linearity follows from the linearity of the limit and of the integral on D[a, b]: if hn → f1
and gn → f2 , with hn , gn ∈ D[a, b], then
Z b Z b Z b Z b
c1 f1 (x)dx + c2 f2 (x)dx = c1 lim hn (x)dx + c2 lim gn (x)dx
a a n→∞ a n→∞ a
 Z b Z b  Z b
= lim c1 hn (x)dx + c2 gn (x)dx = lim (c1 hn (x) + c2 gn (x))dx
n→∞ a a n→∞ a
Z b
= (c1 f1 (x) + c2 f2 (x))dx.
a

The first equality uses the definition of the integral, the second is the linearity of the limit in R,
the third is the linearity of the integral on D[a, b], and the fourth uses the linearity of the limit in
the vector space C([a, b]), so that that c1 hn + c2 gn converges to c1 f1 + c2 f2 in C[a, b]. The proof of
additivity is completely similar.
To show the boundedness property, note that if fn → f in C[a, b] with fn ∈ D[a, b], then
also |fn | → |f | in C[a, b]. This is because
 
|fn (x)| − |f (x)| ≤ |fn (x) − f (x)|, (10.18)
so that if the right side converges uniformly on [a, b] to zero as n → ∞, then so does the left side.

59
Exercise 10.10 Check this!

Now, the first inequality in (10.16) is easy: by boundedness in D, we have

Z b  Z b
fn (x)dx ≤ |fn (x)|dx, (10.19)
 

a a

and the left side in (10.19) converges to

Z b  Z b 
f (x)dx → f (x)dx (10.20)
   
 n   
a a

by the definition of the integral. As |fn | → |f | in C[a, b] and |fn | ∈ D[a, b], the right side of (10.19)
converges to
Z b
|f (x)|dx, (10.21)
a
and the first inequality in (10.16) now follows from (10.16), (10.19) and (10.20).
To show the second inequality in (10.16), let ε > 0. Then there exists N such that

sup |fn (x)| ≤ sup |f (x)| + ε,

x∈[a,b] x∈[a,b]

for all n ≥ N , and |fn | ∈ D[a, b], so the second inequality in (10.6), the boundedness on D[a, b],
shows that Z b
|fn (x)|dx ≤ (b − a) sup |fn (x)| ≤ (b − a)(sup |f | + ε)
a x∈[a,b]

for n ≥ N . The integral in the left side above converges to

Z b
|f (x)|dx,
a

while the right side does not depend on n, hence

Z b
|f (x)|dx ≤ (b − a)(sup |f | + ε). (10.22)
a

As ε > 0 was arbitrary, this shows that

Z b
|f (x)|dx ≤ (b − a) sup |f (x)|,
a x∈[a,b]

as claimed.
Finally positivity follows the same way as the bound we just observed: if f ≥ 0 and fn → f ,
then for ε > 0 there exists N such that for n ≥ N , fn > −ε, and thus
Z b Z b
fn (x)dx ≥ −ε 1 · dx = −(b − a)ε.
a a

Note that 1 ∈ D, and

Z b
1 · dx = b − a,
a

60
from the definition. As a consequence, we have
Z b
f (x)dx ≥ −(b − a)ε.
a

Since ε > 0 was arbitrary, we deduce that

Z b
f (x)dx ≥ 0,
a

finishing the proof. 

In summary, we have shown the existence part of the following theorem.

Theorem 10.11 Given a < b, there exists a unique linear map Iab : C[a, b] → R such that for
all f ∈ D[a, b] we have
Z b
Iab (f ) = f (x)dx, (10.23)
a
and such that there is a constant K0 such that

|Iab f | ≤ K0 kf k, f ∈ C([a, b]). (10.24)

Furthermore, this unique linear map satisfies properties (10.14)-(10.17).

One writes Z b
f (x)dx = Iab (f ).
a

Proof. As mentioned, we have already proved existence. To show uniqueness, assume that Iab is a
linear map from C[a, b] to R that satisfies (10.24), and has the property that for fn ∈ D we have
(10.23). Take any f ∈ C[a, b] and a sequence fn ∈ D[a, b] such that fn → f as n → ∞. Then, by
linearity of Iab , we have
Iab (f ) − Iab (fn ) = Iab (f − fn ).
The estimate (10.24) implies then that

|Iab (f ) − Iab (fn )| ≤ K0 kf − fn k → 0,

as n → +∞. We conclude that Z b

Iab f = lim fn ,
a

thus Iab is uniquely determined by its values on D[a, b]. 

10.3 The fundamental theorem of calculus

We now prove the Newton-Leibniz theorem, also known as the fundamental theorem of calculus.
For this purpose it is convenient to define
Z a
f = 0.
a

The first part of the theorem says that the integral gives an antiderivative.

61
Theorem 10.12 (The fundamental theorem of calculus, part I) Suppose f : [a, b] → R is continuous,
and let Z t
F (t) = f (x)dx, t ∈ [a, b].
a
Then the function F (x) is continuously differentiable on (a, b), and F 0 (x) = f (x) for all x ∈ (a, b). In
addition, the right derivative of F at x = a and the left derivative of F at b exist and are Fr0 (a) = f (a)
and Fl0 (b) = f (b).

Proof: It suffices to show that F is differentiable with F 0 = f , since the continuity of F 0 then
follows from that of f . Let t ∈ [a, b) be fixed and consider h > 0, the case of t ∈ (a, b] and h < 0
being identical similar. Then, for h < b − t, we have, using the basic properties of the integral:
Z t+h Z t Z t+h
F (t + h) − F (h) = f (x)dx − f (x)dx = f (x)dx
a a t
Z t+h Z t+h Z t+h
(10.25)
= f (t)dx + (f (x) − f (t))dx = hf (t) + (f (x) − f (t))dx,
t t t

so that Z t+h
F (t + h) − F (h) − hf (t) = (f (x) − f (t))dx. (10.26)
t

Exercise 10.13 Check carefully which properties of the integral we are using in each step in (10.25).

Using the boundedness property (10.16) of the integral, we deduce from (10.26) that

|F (t + h) − F (t) − hf (t)| ≤ h sup {|f (x) − f (t)|}.

t≤x≤t+h

By the continuity of f at t, given ε > 0 there exists δ > 0 such that |x−t| < δ implies |f (x)−f (t)| < ε.
Thus, for 0 < h < min(δ, b − t) we have

|F (t + h) − F (t) − hf (t)| ≤ εh.

Together with the analogous argument for h < 0, this is exactly the definition of differentiability
at t, with F 0 (t) = f (t), proving the theorem. 
In order to prove the second part of the fundamental theorem of calculus, we need an observation.

Proposition 10.14 If f ∈ C([a, b]) is differentiable on (a, b) and f 0 (x) = 0 for all x ∈ (a, b) then f
is constant: f (x) = f (a) for all x ∈ [a, b].

Proof. Let a < x1 < x2 < b, then by the mean value theorem on [x1 , x2 ], there is c ∈ (x1 , x2 ) such
that
f (x2 ) − f (x1 )
f 0 (c) = .
x2 − x1
As, by assumption, we have f 0 (c) = 0, we conclude that f (x1 ) = f (x2 ). Thus, for all x ∈ [a, b], we
have f (x) = f (a), so that f is a constant function. 

Theorem 10.15 (Fundamental theorem of calculus, part II) If f ∈ C 1 ([a, b]) then
Z b
f (b) − f (a) = f 0 (x)dx.
a

62
This is the part of the theorem that is actually used to evaluate integrals explicitly: to find the
integral
Z b
g(x)dx,
a

for given a function g ∈ C([a, b]), one looks for f ∈ C 1 ([a, b]) such that f 0 = g, and then
Z b
g(x)dx = f (b) − f (a).
a

Note that part I of the fundamental theorem says that the indefinite integral of g is such a function,
but does not give an explicit evaluation – you need a different method of finding the antiderivative
for the explicit calculation. For instance, if g(t) = tn , then you may check that f (t) = tn+1 /(n + 1)
satisfies f 0 (t) = g(t) using the product rule for differentiation, and then apply the second part of the
fundamental theorem of calculus to find the integral of g(x).
Proof. Let us set Z t
F (t) = f 0 (x)dx.
a
By the first part of the fundamental theorem of calculus proven above, F is continuously differentiable
with F 0 (t) = f 0 (t) for all t ∈ [a, b]. Thus, the function g = f − F satisfies g ∈ C 1 ([a, b]) and g 0 (t) = 0
for all t ∈ [a, b]. Proposition 10.14 implies that g(b) = g(a), so that

f (b) − F (b) = f (a) − F (a),

which is exactly
Z b
f (b) − f (a) = F (b) − F (a) = f 0 (x0dx,
a
completing the proof. 

A comment on the integration by parts

Note that integration by parts is a simple consequence of the fundamental theorem of calculus and
the product rule for differentiation:
(f g)0 = f 0 g + f g 0 .
Indeed, for f, g ∈ C 1 ([a, b]), we have
Z b Z b
0
f (b)g(b) − f (a)g(a) = (f g)(b) − (f g)(a) = (f (x)g(x)) dx = (f 0 (x)g(x) + f (x)g 0 (x))dx
a a
Z b Z b
0
= f (x)g(x)dx + f (x)g 0 (x)dx,
a a

and now a simple rearrangement gives the integration by parts formula:

Z b b Z b b
0
f (x) g(x) = f g  − f (x)g 0 (x)dx, f g  = f (b)g(b) − f (a)g(a).
 
a a a a

Consider, for example, the integral

Z b
xex dx.
a

63
If we know that (ex )0 = ex , but can not guess the anti-derivative of xex , we can integrate it by parts
as follows: Z b Z b b Z b
xex dx = x(ex )0 dx = xex  − 1 · ex dx = beb − aea − eb + ea .

a a a a
Essentially, this discovers the fact that
dh i
xex = (x − 1)ex .
dx
Or we can use the fact that (log x)0 = 1/x, to compute, for 0 < a < b:
Z b Z b  x 2 0 Z b
b2 a2 1 x2
x log xdx = (log x) dx = log b − log a − · dx
a a 2 2 2 a x 2
b2 a 2 b2 a 2
= log b − log a − + .
2 2 4 4
It may also help to understand convergence of indefinite integrals. Recall that if a function f (x)
is continuous on [a, +∞), then we set
Z ∞ Z R
f (x)dx = lim f (x)dx, (10.27)
a R→∞ a

provided that the limit in the right side exists. Let us look at
Z ∞
sin x
dx.
1 xp
Note that if p > 1, then we have
 Z R2 sin x  Z R2 | sin x| Z R2
1 1  1 1  1 1
dx ≤ dx ≤ dx = − ≤ , (10.28)
 
x p x p p p+1 p+1 p+1
R1 x p + 1 R1 p + 1 R1
 
R1 R1 R1
thus the limit Z R
sin x
lim dx
R→∞ 1 xp
exists, and the integral Z ∞
sin x
dx
1 xp
is well-defined.
Exercise 10.16 Make the above argument precise: use (10.28) to show that if p > 1 then for any
sequence Rn → +∞ the limit Z Rn
sin x
lim dx
n→+∞ 1 xp
exists and does not depend on the choice of the sequence Rn → +∞. Hint: show that the sequence
Z Rn
sin x
In = dx
1 xp
is Cauchy.
The question we would like to answer is whether the integral converges for p ∈ (0, 1). Note that the
integral has to diverge for p ≤ 0 since the integrand does not tend to zero as n → +∞.

64
Exercise 10.17 Let the function f be continuously differentiable on [1, +∞), and assume that the
integral Z ∞
f (x)dx
1
exists, that is, the limit in the right side of (10.27) exists with a = 1. Assume, in addition, that
there exists M > 0 so that |f 0 (x)| ≤ M for all x ≥ 1. (10.29)
(i) Show that then
lim f (x) = 0.
x→+∞

(ii) Show that this conclusion may be false without assumption (10.29).
Let us then consider the case 0 < p < 1. We have to use some cancellation because of the sign
changes in sin x, because of the following.
Exercise 10.18 Show that the integral
∞
| sin x|
Z
dx
1 xp
does not converge for p ∈ (0, 1).
Consider next Z R
sin x
I(R) = dx,
1 xp
with the idea to let R → +∞ eventually. Note that sin x = (− cos x)0 , (1/xp )0 = −p/xp+1 , hence we
can integrate by parts as follows:
(− cos x)0
Z R Z R Z R 
sin x cos R 1 0
I(R) = dx = dx = − + cos 1 − (− cos x)dx
1 xp 1 xp Rp 1 xp
Z R (10.30)
cos R cos x
= − p + cos 1 − p p+1
dx.
R 1 x

As in Exercise 10.16, the integral

Z ∞ Z R
cos x cos x
dx = lim dx
1 xm R→+∞ 1 xm
exists for all m > 1, thus the limit Z R
cos x
lim dx
R→+∞ 1 xp+1
exists for all p > 0 and equals Z ∞
cos x
dx.
1 xp+1
Going back to (10.30) and passing to the limit R → +∞, we conclude that the integral
Z ∞ Z ∞
sin x cos x
p
dx = cos 1 − p dx
1 x 1 xp+1
also exists. Note how integration by parts allowed us to increase the power of x in the denominator,
and reduce a problem that involves the integral of a function that is decays slowly so the integral is
not absolutely convergent (this terminology is similar to the absolute convergence to a series) to an
integrand that decays faster by one power of x and the integral converges absolutely.

65
11 The contraction mapping principle
The contraction mapping principle is the most basic tool that can be used to prove existence of
solutions in many situations in analysis. Many problems can be formulated in the form
f (x) = y0 , (11.1)
where y0 is an element of some metric space X, f is a mapping from X to X, and x is the unknown
that we need to find. We can reformulate it as
f (x) + x − y0 = x.
The advantage of the latter formulation is that now we have what is known as a fixed point problem.
These are equations of the form
F (x) = x, (11.2)
where F is a mapping from a metric space X to itself, and x is an unknown point x ∈ X. A solution
of (11.2) is known as a fixed point of the mapping F . In other words, (11.1) is equivalent to (11.2)
with
F (x) = f (x) + x − y0 . (11.3)
Of course, there is a serious difference: the notion of a fixed point as a solution to (11.2) requires
only that F maps X to itself, and the notion of a solution of (11.1) also requires only that f maps X
to itself and y0 ∈ X. However, to pass from (11.1) to (11.2) we need to introduce the mapping F
in (11.3) which requires an addition structure on X: we need X to be a vector space to be able to
do that. However, often X is a vector space, so that issue is not a problem.
An important class of mappings of a metric space X onto itself are contractions. We say that a
mapping f : X → X is a contraction if there exists a number q ∈ (0, 1) so that for an x1 , x2 ∈ X
we have
d(f (x1 ), f (x2 )) ≤ qd(x1 , x2 ). (11.4)
We also need the following definition: a metric space X is complete if any Cauchy sequence in X
converges. The spaces Rn are complete. Before we proceed further with the contraction mapping
theorem, let us show that the space C[0, 1] with the norm
kf k = sup |f (x)|
0≤x≤1

is a complete metric space.

Theorem 11.1 The metric space C[0, 1] is complete.
Proof. Let fn be a Cauchy sequence in C[0, 1]. This means that for any ε > 0 there exists N so
that for any n, m ≥ N we have
kfn − fm k < ε.
In other words, we have
sup |fn (x) − fm (x)| < ε,
0≤x≤1
so that
|fn (x) − fm (x)| < ε,
for all n, m ≥ N and all x ∈ [0, 1]. This means that the sequence fn is uniformly Cauchy on [0, 1].
Recall that Theorem 8.16 implies that then fn is uniformly convergent, and by Theorem 8.17 the
limit is a continuous function. Thus, the space C[0, 1] is a complete metric space. 
The contraction mapping principle says the following.

66
Theorem 11.2 Let X be a complete metric space, and f : X → X be a contraction, then f
has a unique fixed point a in X. Moreover, for any x0 ∈ X, the sequence defined recursively
by xk+1 = f (xk ), with x1 = f (x0 ), converges to a as n → +∞. The rate of convergence can be
estimated by
qn
d(xn , a) ≤ d(x1 , x0 ). (11.5)
1−q
Note that the theorem provides an algorithm to compute the unique fixed point, and that the rate
of convergence in (11.5) depends on how close q is to 0 or 1: it gets faster for q close to 0 and slower
for q close to 1.
Proof. We will show that the sequence xk is a Cauchy sequence. Note that

d(xn+1 , xn ) = d(f (xn ), f (xn−1 )) ≤ qd(xn , xn−1 ), (11.6)

so that an induction argument shows that

d(xn+1 , xn ) ≤ q n d(x1 , x0 ). (11.7)

Now, by the triangle inequality and (11.7), we have

d(xn+k , xn ) ≤ d(xn+k , xn+k−1 ) + d(xn+k−1 , xn+k−2 ) + · · · + d(xn+1 , xn )

qn (11.8)
≤ (q n+k−1 + q n+k−2 + · · · + q n )d(x1 , x0 ) ≤ d(x1 , x0 ).
1−q
It follows that if 0 < q < 1, then the sequence xn is Cauchy. Since the space X is complete, the
limit of xn exists, and we set
a = lim xn .
n→∞

Now, as f is a continuous map, passing to the limit n → ∞ in the recursion relation xn = f (xn−1 ),
we arrive at a = f (a), hence a is a fixed point of f .
The reason the fixed point is unique is that f is a contraction. Indeed, if a1 and a2 are two fixed
points, so that a1 = f (a1 ) and a2 = f (a2 ), then by the contraction property we have

d(f (a1 ), f (a2 )) ≤ qd(a1 , a2 ).

However, as both a1 and a2 are fixed points of f , the left side above equals d(a1 , a2 ). Since q ∈ (0, 1),
we deduce that d(a1 , a2 ) = 0 and a1 = a2 . 

Existence theorem for ordinary differential equations

Let us consider an ordinary differential equation (ODE) for an unknown function y(x)

y 0 = f (x, y) (11.9)

supplemented by the initial condition

y(x0 ) = y0 , (11.10)
with some given x0 ∈ R and y0 ∈ R. We assume that the function f (x, y) is continuous in (x, y) and
Lipschitz in y: there exists a constant M > 0 so that

|f (x, y1 ) − f (x, y2 )| ≤ M |y1 − y2 | for all x ∈ R and y1 , y2 ∈ R. (11.11)

Theorem 11.3 Under the above assumptions on f (x, y), there exists an interval (x0 − δ0 , x0 + δ0 ),
so that the problem (11.9)-(11.10) has a unique solution y(x) on the interval (x0 − δ0 , x0 + δ0 ).

67
Proof. We can write (11.9-(11.10) together, using the fundamental theorem of calculus as
Z x
y(x) = y0 + f (t, y(t))dt. (11.12)
x0

Let us define the map A that maps a function y(x) to a function A[y] via
Z x
(A[y])(x) = y0 + f (t, y(t))dt. (11.13)
x0

Then (11.12) can be written as

y(x) = A[y](x), (11.14)
so that y(x) is a solution to (11.12), or, equivalently, to (11.9)-(11.10) if and only if the function y
is a fixed point of the mapping A. The first step is to show that A maps the space C[x0 − δ, x0 + δ]
to itself.
Lemma 11.4 If y ∈ C[x0 − δ, x0 + δ] for some δ > 0, then A[y] is also in C[x0 − δ, x0 + δ].
Proof of Lemma. Let y(x) be a continuous function on the closed interval [x0 − δ, x0 + δ] for some
given δ > 0. Then for any x1 , x − 2 ∈ [x0 − δ, x0 + δ], we have
Z x1 Z x2 Z x1
A[y](x1 ) − A[y](x2 ) = y0 + f (t, y(t))dt − y0 − f (t, y(t))dt = f (t, y(t))dt. (11.15)
x0 x0 x2

The function y is continuous on [x0 − δ, x0 + δ], hence it is bounded on that interval: there exists K
such that |y(t)| ≤ K for all x0 − δ ≤ t ≤ x0 + δ. As the function f is continuous on [x0 − δ, x0 + δ] × R,
there exists M > 0 so that |f (x, y)| ≤ M for all x ∈ [x0 − δ, x0 + δ] and y ∈ [−K, K]. It follows that
|f (t, y(t))| ≤ M for all t ∈ [x0 − δ, x0 + δ]. Using this in (11.15) gives
Z x1
|A[y](x1 ) − A[y](x2 )| ≤ |f (t, y(t))|dt ≤ M |x1 − x2 |. (11.16)
x2

It follows that the function A[y] is continuous. 

We return to the proof of the theorem. Our goal is to show that if δ is sufficiently small, then A
is a contraction on C[x0 − δ, x0 + δ]. To this end, let us take two functions y1 , y2 ∈ C[x0 − δ, x0 + δ]
and write
Z x Z x Z x
A[y1 ](x) − A[y2 ](x) = y0 + f (t, y1 (t))dt − y0 − f (t, y2 (t))dt = [f (t, y1 (t)) − f (t, y2 (t))]dt.
x0 x0 x0
(11.17)
We will now use the Lipschitz property (11.11) of the function f (x, y):
Z x Z x
|A[y1 ](x) − A[y2 ](x)| ≤ |f (t, y1 (t)) − f (t, y2 (t))|dt ≤ M |y1 (t) − y2 (t)|dt. (11.18)
x0 x0

Note that for all t ∈ [x0 , x] we have

|y1 (t) − y2 (t)| ≤ sup |y1 (x) − y2 (x)| = ky1 − y2 k.

x0 −δ≤x≤x0 +δ

Using this in (11.18), we arrive at

Z x Z x
|A[y1 ](x)−A[y2 ](x)| ≤ M |y1 (t)−y2 (t)|dt ≤ M ky1 −y2 k|dt = M |x−x0 |ky1 −y2 k ≤ M δky1 −y2 k.
x0 x0
(11.19)

68
Taking supremum over all x ∈ [x0 − δ, x0 + δ] gives

kA[y1 ] − A[y2 ]k ≤ sup |A[y1 ](x) − A[y2 ](x)| ≤ M δky1 − y2 k. (11.20)

x∈[x0 −δ,x0 +δ]

Therefore, if M δ < 1 then A is a contraction on C[x0 − δ, x0 + δ]. It follows that A has a unique
fixed point y in C[x0 − δ, x0 + δ]. This means that the function y(x) satisfies (11.12):
Z x
y(x) = y0 + f (t, y(t))dt. (11.21)
x0

It follows immediately that y(x0 ) = y0 . Moreover, as the function y(t) is continuous, and f (t, y) is
continuous in both variables, it follows that p(t) = f (t, y(t)) is continuous in t. The fundamental
theorem of calculus implies then that y(x) is differentiable and

y 0 (x) = f (x, y(x)). (11.22)

This finishes the proof. 

Let us now combine the existence theorem for ODEs with the construction of the fixed point of
a contraction mapping in the proof of the existence theorem of a fixed point. Consider an ODE

y 0 (t) = y(t), y(0) = 1,

and write it, as in (11.12), in the form

Z t
y(t) = 1 + y(s)ds.
0

The mapping A is now defined via

Z t
A[y](t) = 1 + y(s)ds.
0

Consider the recursive sequence yn (t), with y0 (t) = 1, and

Z t
yn+1 (t) = 1 + yn (s)ds.
0

Exercise 11.5 Show by induction that

n
X tk
yn (t) = .
k!
k=1

We see that the unique solution is y(t) = et .

12 The implicit function theorem

The implicit function theorem addresses the question of when an equation of the form f (x, y) = 0
uniquely defines x in terms of y, or y in terms of x. The former question means that given y, we
trying to ”solve the equation f (x, y) = 0 for an unknown x”. The latter asks when an equation
of the form f (x, y) = 0 uniquely defines a function y(x) – hence, the name the implicit function
theorem. The two questions are totally equivalent but the points of view are slightly different.

69
 Let us consider a very simple example: the equation

x2 − y = 0. (12.1)
√
Then, for y > 0 this equation has two solutions x = ± y, for y = 0 it has one solution x = 0,
and for y < 0 it has no real solutions. Let, us change our perspective somewhat. Suppose we know
a particular solution (x0 , y0 ) – that is, x20 = y0 and we ask: given a y close to y0 , can we find a
unique x close to x0 so that x2 = y? That is, if we perturb y0 slightly, can we still find a unique
solution of (12.1) close to the original solution x0 ? Note that we have y0 ≥ 0 automatically, simply
because x20 = y0 . The answer to our question is that if y0 > 0, and, say, x0 > 0 then, indeed,
√
for y close to y0 we still have a solution to (12.1) that is close to x0 : x = y. Similarly, if we
√
have x0 < 0, then we still have a solution of (12.1) that is close to x0 : x = − y. On the other hand,
if x0 = 0 so that y0 = 0, then in any interval y ∈ (−δ, δ) around y0 = 0 and any interval (−δ 0 , δ 0 )
around x0 = 0 we can find y < 0 for which the equation x2 = y has no solutions and y > 0 for
which x2 = y has two solutions in the interval (−δ 0 , δ 0 ) – we just need to take y < (δ 0 )2 . Thus, there
is a difference between x0 = 0, y0 = 0 and other points on the graph of y = x2 – we can invert the
relationship around the latter but not the former. The implicit function theorem generalizes this
trivial observation.

12.1 The inverse function theorem on R

We begin with the inverse function theorem, that looks not at an ”implicit” equation f (x, y) = 0 but
at the simpler problem of solving an equation of the form f (x) = y. In one dimension the situation
is quite simple.

Proposition 12.1 Let f (x) be continuously differentiable on an interval [a, b], and set

m = inf f (x), M = sup f (x).

a≤x≤b a≤x≤b

(i) Then f is a one-to-one map from [a, b] to [m, M ] if and only if f is monotonic on [a, b]. (ii) In ad-
dition, if f is monotonic on [a, b] then the inverse function g = f −1 : [m, M ] → [a, b] is continuously
differentiable at y0 ∈ [m, M ] if and only if f 0 (g(y0 )) 6= 0. In that case, g 0 (y0 ) = 1/f 0 (g(y0 )).

Exercise 12.2 Prove the first statement (i) in the above proposition.

To prove (ii), assume first that x0 ∈ (a, b) and f 0 (x0 ) 6= 0. Without loss of generality, we may
assume that f 0 (x0 ) > 0. As the function f 0 is continuous at x0 , there exists δ > 0 so that

f 0 (x) > f 0 (x0 )/2 for all x ∈ (x0 − δ, x0 + δ), (12.2)

thus f is strictly increasing on that interval. Let us set α = f (x0 − δ), β = f (x0 + δ) and take
some y ∈ (α, β), so that y = f (x) for some x ∈ (x0 −δ, x0 +δ), that is, x = g(y) – recall that g = f −1 .
Our goal is to show that the function g(y) is differentiable at y0 and g 0 (y0 ) = 1/f 0 (g(y0 )). As f is
differentiable on (x0 − δ, x0 + δ), there exists c between x and x0 so that we have

f (x) − f (x0 ) = f 0 (c)(x − x0 ), (12.3)

and, because of (12.2), we know that f 0 (c) > f 0 (x0 )/2 > 0. Take now some y ∈ (f (x0 −δ), f (x0 +δ)),
and use (12.3) with x = g(y) ∈ (x0 − δ, x0 + δ). This gives

y − y0 = f 0 (c)(g(y) − g(y0 )), (12.4)

70
so that
g(y) − g(y0 ) 1
= 0 , (12.5)
y − y0 f (c)
with c between x = g(y) and x0 = g(y0 ). We now let y → y0 in (12.5). As c lies between g(y)
and g(y0 ), and the function g(y) is continuous, we know that c → g(y0 ) as y → y0 . Hence, the limit
of the right side of (12.5) exists, and thus so does the limit of the left side, and

g(y) − g(y0 ) 1
lim = 0 , (12.6)
y→y0 y − y0 f (x0 )

which means exactly that g 0 (y0 ) = 1/f 0 (g(y0 )), as claimed. 

12.2 The inverse function theorem for maps Rn → Rn : an outline

Before we proceed with the inverse function theorem for maps from Rn to Rn , that is, n × n systems
of equations
F1 (x1 , . . . , xn ) = y1 ,
...... (12.7)
Fn (x1 , . . . , xn ) = yn ,
let us first consider a special case when F : Rn → Rn is an affine map, that is, then there exists
an n × n matrix A and a vector q ∈ Rn , so that

F (x) = Ax + q. (12.8)

In that case, the equation F (x) = y takes the form

Ax + q = y, (12.9)

and has an explicit solution

x = A−1 (y − q),
provided that the matrix A is invertible. A general differentiable map F can be approximated in a
neighborhood of a point x0 as

F (x) ≈ F (x0 ) + [DF (x0 )](x − x0 ), (12.10)

in the sense that

F (x) − (F (x0 ) + [DF (x0 )](x − x0 )) = o(kx − x0 k). (12.11)
Here, DF (x0 ) is the derivative matrix of F at x0 . Now, for y close to y0 = F (x0 ), let us replace the
exact equation
F (x) = y (12.12)
by an approximate equation
F (x0 ) + [DF (x0 )](x − x0 ) = y. (12.13)
Warning: at the moment, we do not really know that solutions of (12.12) and (12.13) are close, we
are simply trying to understand what should be important for (12.12) to have a solution x close
to x0 . The question you may want to keep in the back of your mind is why the solution to an
approximate equation (12.13) is close to a solution of the true equation (12.12) – this should become

71
clear from the proof of Theorem 12.7. Note that the approximate equation (12.13) has the familiar
form (12.9), and its solution is explicit:

x = x0 + [DF (x0 )]−1 (y − y0 ). (12.14)

Recall that y0 = F (x0 ). For (12.14) to make sense, we must know that the matrix DF (x0 ) is
invertible. This is a generalization of the condition f 0 (f −1 (y0 )) 6= 0 in Proposition 12.1. A natural
guess then is that a map F : Rn → Rn is invertible in a neighborhood of a point y0 = F (x0 ) if the
derivative matrix DF (x0 ) is an invertible matrix. This will be our goal.

12.3 Some preliminaries

We will need the following auxiliary lemmas for the proof of the inverse function theorem.

Lemma 12.3 Let A be an n × n matrix, U ⊂ Rn be an open set, and a map f : U → Rn be

continuously differentiable on U . Set g(x) = Af (x), then g : U → Rn is a continuously differentiable
map with
Dg(x) = ADf (x). (12.15)

Proof. Recall that the entries of the matrix Dg are

∂gi (x)
[Dg(x)]ij = ,
∂xj

and
n
X
gi (x) = Aik fk (x).
k=1

It follows that
n n
∂gi (x) X ∂fk (x) X
= Aik = Aik [Df (x)]kj = (ADf (x))ij ,
∂xj ∂xj
k=1 k=1

thus Dg(x) = ADf (x), and we are done. 

Lemma 12.4 Let A be an n × n matrix with entries such that Aij ≤ ε for all 1 ≤ i, j ≤ n, then for
any vector v ∈ Rn we have kAvk ≤ nεkvk.

Proof. First, we recall the inequality

(x1 + · · · + xn )2 ≤ n(x21 + · · · + x2n ). (12.16)

To see that (12.16) holds, write

X X
(x1 + · · · + xn )2 = x21 + · · · + x2n + 2 xi xj ≤ x21 + · · · + x2n + (x2i + x2j )
1≤i<j≤n 1≤i<j≤n

= x21 + ··· + x2n + (n − 1)(x21 + ··· + x2n ) = n(x21 + · · · + x2n ).

Alternatively, one can use the Cauchy-Schwarz inequality: observe that

x1 + x2 + · · · + xn = (1, 1, , . . . , 1) · (x1 , x2 , . . . , xn ),

thus p √
|x1 + x2 + · · · + xn |2 ≤ k(1, 1, . . . , 1)kxk = nkxk,

72
which is (12.16). Note that for each 1 ≤ i ≤ n we have
Xn  X n n
X
|(Av)i | =  Aij vj  ≤ |Aij ||vj | ≤ ε |vj |,
 
j=1 j=1 j=1

so that, using (12.16), we get

n
X 2 n
X
2 2 2
|(Av)i | ≤ ε |vj | ≤ nε |vj |2 = nε2 kvk2 .
j=1 j=1

Next, summing over i, we get

n
X
kAvk2 = |(Av)i |2 ≤ n2 ε2 kvk2 ,
i=1

and the claim of the lemma follows. 

Lemma 12.5 Let A be an n× invertible matrix. Then there exist αA > 0 and βA > 0 so that

βA kxk ≤ kAxk ≤ αA kxk, for all x ∈ Rn . (12.17)

Proof. The function G(x) = kAxk is continuous on Rn , and the unit sphere S = {x ∈ Rn : kxk = 1}
is a compact subset of Rn . Hence, G attains its maximum αA and minimum βA on S. Moreover, as
the matrix A is invertible, G(x) 6= 0 for all x ∈ S, thus αA > 0 and βA > 0. However, each x ∈ Rn
can be written as x = ry, with kyk = 1, and r = kxk, so that y ∈ S and r ≥ 0. Then, we have

kAxk = kA(ry)k = krA(y)k = rkA(y)k = rG(y) = kxkG(y),

thus
kAxk ≤ kxkαA ,
and
kAxk ≥ kxkβA ,
finishing the proof. 

Corollary 12.6 Let A be an n × n invertible matrix, then the map F (x) = Ax maps any open
set U ∈ Rn to an open set.

Proof. Lemma 12.5 applied to the matrix A−1 implies that there exists α > 0 so that

kA−1 wk ≤ αkwk for all w ∈ Rn . (12.18)

Let now U be any open set and V = F [U ] be the image of U under F . Take any y0 ∈ F [U ], so
that y0 = Ax0 , with x0 ∈ U . As the set U is open, it contains a ball B(x0 , r) with some r > 0.
Consider any z ∈ B(y0 , ρ), with ρ < r/α. Then z = F (x), with x = A−1 (z), and we have,
using (12.18):

kx − x0 k = kA−1 (z) − A−1 (y0 )k = kA−1 (z − y0 )k ≤ αkz − y0 k ≤ αρ < r, (12.19)

so that x ∈ B(x0 , r), and thus x ∈ U . It follows that z ∈ F [U ], thus the ball B(y0 , ρ) is contained
in V and the set V is open. 

73
12.4 The inverse function theorem for maps Rn → Rn
We will now prove the inverse function theorem for maps from Rn to Rn .
Theorem 12.7 Let U ∈ Rn be an open set, and x0 ∈ U . Let f : U → Rn be a continuously
differentiable map, and set y0 = f (x0 ). Suppose that the derivative matrix Df (x0 ) is invertible.
Then there exist an open set V ⊂ U such that x0 ∈ V , and an open set W ⊂ Rn such that y0 ∈ W ,
so that f is a one-to-one map from V to W . Moreover, the inverse map g = f −1 : W → V is also
continuously differentiable and for y ∈ W we have Dg(y) = [Df (g(y))]−1 .
Proof. Step 1. Reduction to the case Df (x0 ) = I. We first note that it suffices to prove the
theorem under an additional assumption that

Df (x0 ) = In , (12.20)

the n × n identity matrix In . Indeed, if Df (x0 ) 6= In , we consider the map

f˜(x) = [Df (x0 )]−1 f (x).

Then the gradient matrix of f˜ at the point x0 is In :

Df˜(x0 ) = [Df (x0 )]−1 Df (x0 ) = In ,

as follows from Lemma 12.3. Since the matrix Df (x0 ) is invertible, the function f is one-to-one from
a neighborhood V of x0 to a neighborhood W of y0 = f (x0 ) if and only if the function f˜ is a one-
to-one map from V to W̃ = [Df (x0 )]−1 W , and W̃ is a neighborhood of the point ỹ0 = f˜(x0 ). This
is a consequence of Corollary 12.6. Again, as the matrix Df (x0 ) is invertible, and by Lemma 12.3
we have
Df˜(x) = [Df˜(x0 )]−1 Df (x),
the function f˜ is continuously differentiable if and only if f is continuously differentiable. Hence, we
may assume without any loss of generality that f satisfies (12.20), and this is what we will do for
the rest of the proof.
As the map f (x) is continuously differentiable in U , and f satisfies (12.20), for any ε > 0 there
exists r > 0 so that  ∂f 
 i
− δij  < ε (12.21)

∂xj


for all x ∈ B(x0 , r). Here, δij is the Kronecker delta: δij = 1 if i = j and δij = 0 if i 6= j. In other
words, for each x ∈ B(x0 , r), we have

Df (x) = I + E(x), (12.22)

with the matrix E(x) such that

|Eij (x)| ≤ ε for all 1 ≤ i, j ≤ n. (12.23)

It follows that for every v ∈ Rn and all x ∈ B(x0 , r) we have, by the triangle inequality

kDf (x)vk = k(I + E(x))vk = kv + E(x)vk ≥ kvk − kE(x)vk.

Now, Lemma 12.4, with A = E(x), implies that for every v ∈ Rn and all x ∈ B(x0 , r) we have
1
kDf (x)vk ≥ kvk − εnkvk ≥ kvk,
2

74
as long as ε < 1/(2n). In particular, it follows that the kernel of the matrix Df (x) is {0}, and the
matrix Df (x) is invertible for all x ∈ B(x0 , r).
Step 2. Reformulation as a fixed point problem for a contraction mapping. Now, we
turn to solving
f (x) = y (12.24)
for y close to y0 = f (x0 ), with the assumption Df (x0 ) = In . Note that, because of (12.20), the
approximate linear equation (12.13) in this case takes the particularly simple form

f (x0 ) + x − x0 = y, (12.25)

with y0 = f (x0 ), that, naturally, has a unique solution

x = x0 + y − y0 .

We assume that y ∈ B(y0 , ρ) and will later see how small ρ needs to be. In order to turn (12.24)
into a contraction mapping question, let us reformulate (12.24) as a fixed point problem by setting

F (x) = x − f (x) + y, (12.26)

again, with y ∈ B(y0 , ρ) fixed. The map F depends on y – we will drop it in the notation but you
should keep this in mind. Note that x satisfies (12.24) if and only if it satisfies

F (x) = x. (12.27)

This is a fixed point problem and we will address it using the contraction mapping principle. The
following lemma is a key part of the proof.

Lemma 12.8 There exist ρ > 0 and r1 > 0 so that for each y ∈ B(y0 , ρ) the map F maps the closed
ball B̄(x0 , r1 ) to itself and is a contraction on B̄(x0 , r1 ).

As for each y ∈ B(y0 , ρ), the map F is a contraction, the contraction mapping principle will then
imply that for each y ∈ B(x0 , ρ) there exists a unique x ∈ B(x0 , r1 ) such that F (x) = x, which
means that there is a unique solution to (12.24) in B(x0 , r1 ) for each y ∈ B(y0 , ρ). In other words,
we may define the inverse map f −1 that maps B(y0 , ρ) to B(x0 , r1 ).
Proof of Lemma 12.8. The proof requires us to verify two conditions: first, that F maps the
closed ball B̄(x0 , r1 ) to itself, and, second, that there exists q ∈ (0, 1) so that for any z, w ∈ B̄(x0 , r1 )
we have
kF (z) − F (w)k ≤ qkz − wk. (12.28)
We will start with (12.28). Note that, as Df (x0 ) = In , we have

DF (x0 ) = 0. (12.29)

To verify that F satisfies (12.28), let us take z, w ∈ B(x0 , r), with r determined by the condition
that (12.21) holds in B(x0 , r), and define

g(t) = F (z + t(w − z)),

for 0 ≤ t ≤ 1, so that F (z) = g(0), and F (w) = g(1). The fundamental theorem of calculus implies
that Z 1
Fk (w) − Fk (z) = gk0 (t)dt, 1 ≤ k ≤ n. (12.30)
0

75
We compute the derivative gk0 (t) using the chain rule:
n
dgk (t) X ∂Fk (z + t(w − z))
= (wj − zj ). (12.31)
dt ∂xj
j=1

The definition (12.26) of F (x) implies that

∂Fk (x) ∂fk (x)
= δkj − .
∂xj ∂xj
Recalling (12.22)-(12.23), we see that the derivative matrix DF (x) = E(x), and thus the entries of
the matrix DF (x) satisfy
 ∂F 
 i
 ≤ ε for all 1 ≤ i, j ≤ n. (12.32)
∂xj


This is why we modified f (x) to make sure that Df (x0 ) = I at the beginning of the proof. Now,
using (12.32) in (12.31) gives
 dg (t)  X n
 k 
≤ ε|wj − zj | ≤ nεkw − zk, (12.33)
dt
 
j=1

for all 1 ≤ k ≤ n. Using this in (12.30) gives

Z 1 Z 1
0
|Fk (w) − Fk (z| ≤ |gk (t)|dt ≤ nεkw − zkdt = nεkw − zk, (12.34)
0 0

for all 1 ≤ k ≤ n. It follows from (12.34) that

n
X 1/2
kF (w) − F (z)k ≤ |Fk (w) − Fk (z|2 ≤ (n(n2 ε2 kw − zk2 ))1/2 = εn3/2 kw − zk. (12.35)
k=1

Therefore, F satisfies (12.28) with q = 1/2 on B̄(x0 , r) if we take ε = 1/(2n3/2 ) and then take r1 > 0
so small that (12.21) holds in B̄(x0 , r1 ). Thus, we have established that
1
kF (w) − F (z)k ≤ kw − zk, for all z, w ∈ B(x0 , r1 ), (12.36)
4
if r1 is sufficiently small. Note that even though the map F depends on y, this estimate holds for
all y ∈ B(y0 , ρ): this will be important in Step 3 below.
Next, we need to verify that F maps B̄(x0 , r1 ) to itself if ρ is sufficiently small and y ∈ B(y0 , ρ).
Note that
F (x0 ) = x0 − f (x0 ) + y = x0 − y0 + y, (12.37)
hence we have
r1
kF (x0 ) − x0 k = ky − y0 k ≤ ρ < , (12.38)
2
provided we take ρ < r1 /2, thus, in particular, F (x0 ) ∈ B(x0 , r1 ). Next, we take x ∈ B(x0 , r1 ) and
write, using the triangle inequality
kF (x) − x0 k = kF (x) − F (x0 ) + F (x0 ) − x0 k ≤ kF (x) − F (x0 )k + kF (x0 ) − x0 k. (12.39)
The first term in the right can be estimated using (12.36), and the second using (12.38), to give
1 r1
kF (x) − x0 k ≤ kF (x) − F (x0 )k + kF (x0 ) − x0 k ≤ kx − x0 k + ≤ r1 , (12.40)
4 2
finishing the proof of Lemma 12.8. 

76
Exercise 12.9 Use the above argument to show that there exists r0 so that for any r < r0 the image
of the ball B(x0 , r) under f is an open set: it contains a ball B(y0 , ρ) centered around y0 = f (x0 ),
with ρ > 0 that depends on r.

Step 3: continuity of the inverse map. We now show that the inverse map g = f −1 is a
continuous map from B(y0 , ρ) to B(x0 , r1 ), provided that ρ and r1 are sufficiently small. Recall that
for a given y ∈ B(y0 , ρ), the point g(y) ∈ B(x0 , r1 ) is the unique fixed point in B(x0 , r1 ) of the map

F (x) = x − f (x) + y,

that depends on y, and that is how g(y) depends on y. To show that g(y) is continuous, we need to
show that the fixed point of F depends continuously on y. Let us take y, y 0 ∈ B(x0 , ρ) and consider
the corresponding maps

F (x) = x − f (x) + y, F 0 (x) = x − f (x) + y 0 .

Then, for any z, w ∈ B(x0 , r1 ), we have

F (z) − F 0 (w) = F (z) − F (w) + F (w) − F 0 (w) = F (z) − F (w) + w − f (w) + y − (w − f (w) + y 0 )
= F (z) − F (w) + y − y 0 ,
(12.41)
hence
kF (z) − F 0 (w)k ≤ kF (z) − F (w)k + ky − y 0 k. (12.42)
Using (12.36), we get
1
kF (z) − F 0 (w)k ≤ kz − wk + ky − y 0 k. (12.43)
4
Let us now take z = g(y) and w = g(y 0 ), so that F (z) = z and F 0 (w) = w. It follows from (12.43)
that
1
kz − wk ≤ kz − wk + ky − y 0 k, (12.44)
4
thus
3
kz − wk ≤ ky − y 0 k. (12.45)
4
In other words, we have shown that the map g satisfies
3
kg(y) − g(y 0 )k ≤ ky − y 0 k, for all y, y 0 ∈ B(y0 , ρ). (12.46)
4
It follows that g is a continuous map on B(y0 , ρ).
Step 4: computing the derivative of the inverse map. Now that we have shown that the
inverse map g = f −1 : B(y0 , ρ) → B(x0 , r1 ) is well defined and continuous, provided that ρ and r1
are sufficiently small, it remains to show that g is continuously differentiable and for y ∈ B(y0 , ρ)
we have
Dg(y) = [Df (g(y))]−1 . (12.47)
Note that it actually suffices to prove differentiability and relation (12.47) only for y = y0 = f (x0 )
since x0 was chosen arbitrarily, as an arbitrary point such that Df (x0 ) is an invertible matrix.
In this step of the proof we will no longer assume that Df (x0 ) is the n × n identity matrix In ,
and will denote A = Df (x0 ). For a given ε > 0 we can find δ > 0 such that δ < r0 , with r0 as in
Exercise 12.9, and if kx − x0 k < δ, then

kf (x) − f (x0 ) − A(x − x0 )k < εkx − x0 k. (12.48)

77
The result of Exercise 12.9 shows that the image of the ball B(x0 , δ) under f is an open set,
hence it contains a ball B(y0 , ρ). Let us take any y ∈ B(y0 , ρ). Then, we have y = f (x) with
some x ∈ B(x0 , δ), thus we may use (12.48) with y = f (x) and y0 = f (x0 ), and x = g(y), x0 = g(y0 ),
to get
ky − y0 − A(g(y) − g(y0 ))k < εkg(y) − g(y0 )k. (12.49)
Let us write
y − y0 − A(g(y) − g(y0 )) = A[A−1 (y − y0 ) − (g(y) − g(y0 ))], (12.50)
and recall that by Lemma 12.5 there exists β > 0 so that
kAwk ≥ βkwk for all w ∈ Rn . (12.51)
Using this in (12.50) gives
ky − y0 − A(g(y) − g(y0 ))k = kA[A−1 (y − y0 ) − (g(y) − g(y0 ))]k ≥ βkA−1 (y − y0 ) − g(y) − g(y0 )k
= βkg(y) − g(y0 ) − A−1 (y − y0 )k.
(12.52)
Inserting this into (12.49) gives
ε
kg(y) − g(y0 ) − A−1 (y − y0 )k ≤ kg(y) − g(y0 )k, (12.53)
β
for all y ∈ B(y0 , ρ). This is almost want we need to say that Dg(y0 ) = A−1 except in the right
side we have kg(y) − g(y0 )k rather than ky − y0 k that we need. However, we can bootstrap: (12.53)
implies that
ε
kg(y) − g(y0 )k − kA−1 (y − y0 )k ≤ kg(y) − g(y0 )k, (12.54)
β
so that  ε
1− kg(y) − g(y0 )k ≤ kA−1 (y − y0 )k. (12.55)
β
Lemma 12.5 tells us that there exists α > 0 so that
kA−1 wk ≤ αkwk for all w ∈ Rn . (12.56)
Using (12.56) in (12.55) gives us
 ε
1− kg(y) − g(y0 )k ≤ αk(y − y0 )k. (12.57)
β
Now, (12.54) and (12.57) together imply that
 ε −1 α
kg(y) − g(y0 ) − A−1 (y − y0 )k ≤ 1 − εky − y0 k, (12.58)
β β
for all y ∈ B(y0 , ρ). It follows that g is differentiable at y0 and Dg(y0 ) = A−1 .
The final step, continuity of Dg(y) is surprisingly easy: we know that Dg(y) = [Df −1 (g(y))]
and the matrix Df (x) is invertible at x = x0 . It follows that detDf (x) 6= 0 in a ball around x0 .
Then, the explicit formula for the inverse of a matrix in terms of its minors implies that the inverse
matrix [Df ]−1 (x) is a continuous function of x, hence its composition [Df ]−1 (g(y)) with a continuous
map g(y) is also continuous. This completes the proof of the inverse function theorem. 
Let us note that, once we know that the inverse map g = f −1 is differentiable, its derivative
matrix can be computed as follows. We start with the identity
f (g(y)) = y
and write it out in components as
fk (g1 (y), g2 (y), . . . , gn (y)) = yk , 1 ≤ k ≤ n.

78
12.5 The implicit function theorem
The implicit function theorem starts with a system of n equations for n unknowns x = (x1 , . . . , xn ),
parametrized by y ∈ Rm :
G1 (x1 , . . . , xn , y) = 0,
...... (12.59)
Gn (x1 , . . . , xn , y) = 0.
Let us assume that z = (z1 , . . . , zn ) is a solution of this system for some y0 ∈ Rm :

G(z, y0 ) = 0, (12.60)

or, in the system form,

G1 (z1 , . . . , zn , y0 ) = 0,
...... (12.61)
Gn (z1 , . . . , zn , y0 ) = 0.
The question is whether if we take y close to y0 , can we find a solution x of (12.59) that is close
to z. The system (12.7), addressed by the inverse function theorem, is a special case of this problem,
with y ∈ Rn and G(x, y) = G(x) − y. There is a nice way to understand the general case via an
application of the inverse function theorem. One problem to apply this theorem is that G(x, y)
maps Rn+m to Rn and not to Rn+m , and in the inverse function theorem we need the domain of the
map and the image of the map to have the same dimension. To fix this, consider the map

G̃(x, y) = (G(x, y), y), (12.62)

that maps Rn+m to Rn+m . This simply means that we re-write (12.59) by adding to it m equations
of the form yk = yk , and the system (12.59) is equivalent to

G̃(x, y) = (0, y). (12.63)

We know from (12.60) that the point (z, y0 ) satisfies

G̃(z, y0 ) = (0, y0 ). (12.64)

In addition, the derivative matrix DG̃(x, y) has the block form

 
Dx G(x, y) Dy G(x, y)
DG̃(x, y) = . (12.65)
0 Im×m

Thus, the matrix DG̃(x, y) is invertible if and only if the matrix Dx G(x, y) is invertible. Let us
assume that the derivative matrix DG(z, y0 ) is invertible and take y close to y0 . Now, the inverse
function theorem implies that for all y close to y0 the system (12.63) has a unique solution x close
to z – this is the implicit function theorem. Let us formulate it precisely.

Theorem 12.10 Let a map G : Rn+m → Rn satisfy G(x0 , y0 ) = 0 for some x0 ∈ Rn and y0 ∈ Rm .
Assume that G is continuously differentiable at (x0 , y0 ) and the n×n matrix Dx G(x0 , y0 ) is invertible.
Then there exist r > 0 and ρ > 0 so that for every y ∈ B(y0 , ρ) there exists a unique x ∈ B(x0 , r)
such that G(x, y) = 0.

79