Foundations of Applied Mathematics

Preface

Purpose. This book is intended primarily as a text for a single- or multi-semester course in applied mathematics for first-year graduate students or seniors in engineering. It grew out of a set of lecture notes for a two-semester course that I taught both at Cornell University and at the University of Delaware, a course taken primarily by first-year graduate students in applied mechanics, materials science, mechanical, chemical, civil, and aerospace engineering, as well as some undergraduates and a number of people from electrical engineering, oceanography, chemistry, and astronomy.

Prerequisites. Prerequisites consist of the usual undergraduate four-semester sequence in calculus and ordinary differential equations, together with the general maturity and background of a senior or beginning graduate student. Knowledge of computer programming is not required.

Content and Organization. In books of this type it is tempting to try to include something for everyone, that is, diverse physical applications from all branches of engineering, applied physics, and so on. Instead we have elected to concentrate on only a few areas of application, including fluid mechanics, heat conduction, and Newtonian mechanics. Discussion of such topics is self-contained; for instance, the governing equations of fluid mechanics and heat conduction are developed in the section on vector field theory before solutions are sought in the later sections on partial differential equations. As a result, these physical concepts weave through the mathematics and act as a unifying element. I try to emphasize not only how the mathematics permits us to clarify and understand the physics but also how our physical insight can support the mathematics and provide the key to finding the appropriate mathematical line of approach. Practical and numerical aspects are emphasized as well.

We begin in Part I with basic notions of functions and limit processes and proceed through various aspects of advanced calculus of real variables, with complex variables treated in Part II. In Part III we introduce linear vector space, operators, and the eigenvalue problem, with an emphasis on matrix operators, and an introduction to integral operators which is arranged in parallel so that it can be bypassed if desired. Ordinary and partial differential operators are so important that they are split off and treated separately in Parts IV and V.

Although the most natural sequence of topics is Parts I through V in that order, I have tried to maximize the book’s adaptability to other formats of presentation. For example, in Part V on Partial Differential Equations a brief summary of the Sturm-Liouville theory is included, in Section 26.2, so that the reader can cope with the necessary eigenfunction expansions without necessarily having read Part III, which deals with that subject in more detail. I’ve also tried to refer the reader back to any specific formulas or results that are used and that appeared in an earlier part of the book.

It should be possible to begin with any of Parts I, II, or III. For, Parts IV and V (not including the starred material), however, I suggest the following preliminary reading:

For Part IV: Section 4.2.

For Part V: Section 4.2, Chapters 5, 6, 7, 8, 9, Section 22.4.

In addition, some complex variable theory is used in Part V, and so it is recommended, although not essential, that Part II be read in advance as well. Perhaps it goes without saying that if the last two chapters of Part V (on Green’s functions, perturbation methods, and finite-difference methods) are included, then it would be helpful to the reader first to have read the corresponding sections on ordinary differential equations in Part IV.

Suggested Courses.

TWO OR THREE SEMESTERS: In the two-semester course that I teach we cover about 90 percent of the material. This means that the pace is rather fast and does not permit class time to be used for both lecture and the discussion of assigned homework. Instead, I use class time for lecture and simply make the Solutions Manual available to the students to assist them in working out the suggested Exercises. The grade is based solely on two or three examinations with at least one class before each exam reserved for questions and review. At a more leisurely pace, the entire book could be covered in three semesters.

ONE SEMESTER: A natural one-semester course might cover Part I only, and would amount to a course in Advanced Calculus for Scientists and Engineers.

Starred Material. A certain amount of starred material is included. It can be bypassed with no loss of continuity, and is either of a more advanced or supplementary nature.

Exercises. The Exercises at the end of each chapter cover the material in that chapter and introduce additional material as well. They are arranged in the same order as the topics within the chapter and are not ordered in terms of their degree of difficulty. Starred problems, however, are generally more difficult and/or are based on material from starred sections within the chapter.

References. Besides the references cited in footnotes, a list of Additional References appears at the end of each part. Undoubtedly numerous excellent sources have not been cited, and I apologize in advance for these oversights.

Acknowledgment. I would like to extend special thanks to my department chairperson, Professor Jack R. Vinson, who has encouraged me in this time-consuming endeavor, to my students who are a continuing source of inspiration and friendship, and to Miss Leslie A. Month in particular. Finally, I am also grateful to Professors A. Eshett, Peter J. Gingo, Jerry Kazdan, Ronald J. Lomax, Ivar Stakgold, Dale W. Thoe, and Erich Zauderer for offering helpful comments on the manuscript.

Closure. In spite of all precautions, errors will no doubt be uncovered in using this book, and I would like to encourage you to let me know about them, if you have the time, so that they can be corrected. Beyond that, any comments, suggestions, or ideas that you may wish to share would be most appreciated. Applied mathematics is a beautiful subject, and I certainly hope that some of this comes through, in a quiet way, within this book.

MICHAEL D. GREENBERG

PART I

REAL VARIABLE THEORY

The first chapter, on LIMIT PROCESSES, sets the stage for much of what follows in Part I: Chapters 2 and 3 deal with the related limits INFINITE SERIES and SINGULAR INTEGRALS; Chapter 4 takes time out to consider questions concerning the interchange of the order of multiple limit processes, and this discussion leads naturally to the delta function and generalized functions; Chapter 5 returns to series and considers the so-called Fourier series of a periodic function (with mean-square convergence and vector space notions treated separately in Part III). It ends with the passage to the Fourier integral as the period becomes infinite, thereby leading to the Fourier and Laplace transforms in Chapter 6. Besides discussing FUNCTIONS OF SEVERAL VARIABLES, VECTORS, SURFACES AND VOLUMES, VECTOR FIELD THEORY, and THE CALCULUS OF VARIATIONS, the final four chapters provide self-contained introductions to the physics and governing differential equations of heat transfer, fluid mechanics, strings and membranes, and the gravitational potential, so that these subject areas will be readily available for many of the applications that form an important part of the rest of the book.

Chapter 1

The Important Limit Processes

In this first chapter we review a number of fundamentals: functions, functionals, limits, continuity, and uniform continuity. Then we look at two important limit processes, differentiation and integration, the latter in some detail. Practical and computational aspects are emphasized as well.

1.1. FUNCTIONS AND FUNCTIONALS

Recall that a function f is a mapping from one set, called the domain of f, to another, called the range of f. Suppose, for instance, that f is defined by f (x) = 8x(x – 1)² + 2x over the domain D, which is the segment 0.1 < x ≤ 1 of a real axis (the "x axis"). Clearly, this f is real valued so that the range R will also be some portion of a real axis (the "f axis"), as sketched in Fig. 1.1. Although the symbols f and f (x) are generally used interchangeably without confusion, note that there is a difference and that occasions do arise in which it is important to respect this difference in the interest of clarity. Specifically, f (x) actually denotes the range point associated with the domain point x, whereas the single letter f denotes the entire mapping, as can be displayed graphically if, following Descartes (1596–1650), we arrange the x and f axes at right angles (hence Cartesian axes), as shown in Fig. 1.2. In this example we find that R is the segment 0.848 < f ≤ 2.

Figure 1.1. Domain and range.

Similarly, a real-valued function f (x, y) of two real variables x, y can be displayed (or at least visualized) as a surface, but when the number of dimensions of D plus the number of dimensions of R exceeds three, we’re in difficulty in regard to graphical display.

Figure 1.2. Graph of f (x) = 8x(x – 1)² + 2x.

We always require that f be single valued—that is, that each point in D be sent into a uniquely defined point in R. Yet a given point in R may correspond to more than one point in D. For instance, both x and x′ in Fig. 1.2 get sent into the same range point. That’s fine; but if each point in R were to correspond to a unique point in D, we would then say that f is one-to-one. Clearly, f must be one-to-one if an inverse function x = x(f) is to exist as well.

In formulating a water wave problem, for example, and designating the free surface elevation by y = η(x, t), where x is the horizontal coordinate and t is the time, we thereby make it impossible to consider waves that break, since η would then be multivalued. Of course, we can describe the profile of a breaking wave parametrically by x = x(s, t), y = y(s, t), where s is a convenient parameter, perhaps arc length along the free surface, and where x(s, t), y(s, t) are single-valued functions of s for each t.

A bit more exotic than a function, a functional has as its domain a set of functions and as its range a portion of the real axis; thus it is a function of a function. To illustrate, consider the functional F defined by

where the domain D of F might be defined to be the set of all (real-valued) functions u defined over 0 < x ≤ 1, subject only to the restriction that the integral (1.1) does in fact exist. For instance, if u(x) = 6x, then F(u) = 12. In this example it is clear that the range R of F is the positive real axis, 0 ≤ F(u) < ∞.

Functionals are the objects of interest in the so-called variational calculus, which is the subject of Chapter 10.

1.2. LIMITS, CONTINUITY, AND UNIFORM CONTINUITY

We say that f (x) has a limit L as x tends to x0 and write

if to each > 0 (no matter how small) there corresponds a δ( , x0) > 0 such that | f (x) – L| < whenever 0 < | x – x0 | < δ; in other words, f (x) will be close to L if x is close to x0.

Example 1.1. Consider f (x) = 1/x over 0.1 ≤ x ≤ 1. At the x0 shown in Fig. 1.3, we claim that the limit exists and is equal to the L shown—namely, 1/x0. To prove it, we will put forward a suitable δ( , x0). First, we draw an arbitrarily small band about f (x) = L and, where it intersects the graph (at A and B), drop verticals to the x axis. Noting that a < b, we choose δ = a; then | x – x0 | < δ is the centered interval denoted by the small parentheses. Surely, when x is closer to x0 than δ—that is, 0 < | x – x0 | < δ, f(x) will be closer to L than , as desired. To evaluate δ in terms of and x0, we write f (x0 – δ) – f (x0) = —that is, 1/(x0 – δ) – 1/x0 = . Solving,

Figure 1.3. lim f (x) for f (x) = 1/x.

Of course, any smaller (nonzero) value, say δ = [x0 – x0/( x0 + 1)]/5, would do just as well.    

Note carefully that because of the 0 < | x – x0 | < δ in our definition of , f (x0) need not equal L. For instance, if

then f (x) = 9, not 27! If in addition to having = L we also have f (x0) = L, we say that f (x) is continuous at x0. In , δ language, f (x) is continuous at x0 if to each > 0 there corresponds a δ( , x0) > 0 such that | f(x) – f(x0) | < whenever | x – x0 | < δ. [Recalling our definition of lim, we see that the foregoing definition of continuity is equivalent to the statement f (x + Δx) = f (x).] Clearly, the function given in Example 1.1 is continuous over its domain, x² and sin x are continuous for all x (i.e., for –∞ < x < ∞), and so on. On the other hand, the so-called Heaviside step function, defined by

and H(0) = , say, is discontinuous at x = 0; for any smaller than we see [from a graph of H(x)] that there is no δ such that H(x) remains within an band of H(0) whenever x is closer to 0 than δ.

A more heroic type of discontinuity is exhibited by the function f (x) = cos (1/x). To sketch its graph, note that f (x) = cos (1/x²)x ≡ cos ωx, where the frequency ω = 1/x² tends to infinity as x → 0 and to zero as x → ∞, as shown in Fig. 1.4. Actually, f (x) is not defined at x = 0, since cos ∞ is undefined. So in setting up this example, let us define f (x) = cos (1/x) for all x ≠ 0 and f (0) ≡ A, say. No matter what value is assigned to A, f (x) is discontinuous at x = 0 because f (0 + Δx) = cos (1/Δx) cannot equal A.

Figure 1.4. Graph of f (x) = cos (1/x).

Finally, we say that f (x) is uniformly continuous over D if to each > 0 there corresponds a δ( ) (i.e., independent of x0) such that | f(x) – f(x0) | < whenever | x – x0 | < δ for all x0’s in D. Consider Example 1.1 again. For a given it is clear that δ( , x0) will be smallest, over 0.1 ≤ x0 ≤ 1, when x0 = 0.1, since a is skinniest there. Thus δ( , 0.1) = 0.1 – 1/( + 10) ≡ δ( ) will suffice for all x0’s in D, and hence f (x) = 1/x is uniformly continuous over 0.1 ≤ x ≤ 1. On the other hand, suppose that D were 0 < x ≤ 1 instead. Then δ( , x0) tends to zero as x0 → 0, since a merely gets skinnier and skinnier. Consequently, we are unable to single out one value to serve as our δ( ) over all D, and f (x) = 1/x is not uniformly continuous over 0 < x ≤ 1.

Applying these ideas to functionals, what might be meant by a statement like "F(u) is continuous at u0"? Following the foregoing discussion on functions, we will mean that to each > 0 there corresponds a δ( , u0) such that | F(u) – F(u0) | < whenever u is closer to u0 than δ, as indicated in Fig. 1.5. Note carefully that whereas the range R is part of a real axis, the domain D is a set of functions, indicated schematically in the figure, with each point in D representing a function. The only catch is, what do we mean by "u is closer to u0 than δ" ? Well, we need to introduce a notion of distance for the set D by defining a so-called distance function, or metric, d(u, v). That is, for any points u, v in D we define a real-valued function d(u, v) called "the distance from u to v." It seems entirely reasonable that we require it to be nonnegative and the same as the distance from v to u. Also, the distance from u to itself should certainly be zero. So let us require of d(u, v) that it satisfy the following conditions.

Figure 1.5. Schematic representation of a functional.

(i) Symmetry:

(ii) Positiveness:

(iii) Triangle Inequality:

where the triangle inequality, not mentioned above, is motivated by the simple schematic picture in Fig. 1.6 and is the abstract analog of the Euclidean proposition that the length of any one side of a triangle cannot exceed the sum of the lengths of the other two sides.

One distance function that can be assigned to D is simply

Figure 1.6. Schematic motivation for triangle inequality.

but this is not very discriminating. A more reasonable choice might be

where the functions in D are defined over 0 ≤ x ≤ 1, say; both choices satisfy the requirements (1.6) to (1.8). To illustrate, d(5x, 4x²) is 1 according to (1.9) and 25/16 according to (1.10).

With this done, we are now in a position to talk about lim F(u), continuity of F, and so on. By F(u) = L, for example, we mean that to each > 0 (no matter how small) there corresponds a δ( , u0) such that | F(u) – L | < whenever 0 < d(u, u0) < δ. That is (Fig. 1.5), F(u) must be closer to L than if u is kept within a neighborhood of radius δ of u0.

1.3. DIFFERENTIATION

Recall from the calculus that the derivative df/dx, or f′(x), is defined as the limit of the difference quotient

when the limit exists. If f (x) = x² for instance, then

for all x.

It is easy to show that if f (x) is differentiable, then it must be continuous because (Exercise 1.6)

For example, since the Heaviside function H(x) is discontinuous at x = 0, it follows that H′(0) does not exist; that is, H(x) is not differentiable at x = 0. We say that it is singular there.

The converse of the preceding statement, however, is not true—continuity does not imply differentiability. To illustrate, the continuous function f(x) = | x | is not differentiable at x = 0 because its difference quotient, | Δx |/Δx ≡ sgn Δx (read as "signum Δx or the sign of Δx"), is discontinuous at Δx = 0 (Fig. 1.7), so that of the difference quotient does not exist. In fact, Weierstrass (1815–1897) constructed an example of a function that is continuous everywhere and yet differentiable nowhere!

Figure 1.7. | x | and its difference quotient at x = 0.

Differentiability can break down at various kinds of singularities: at a jump discontinuity [as in H(x)], at a kink (as in | x |), and at a point in the neighborhood of which the function wiggles too much (amplitudewise) and too fast (frequencywise) [as in g(x) of Exercise 1.15]. As a result of the standard graphical interpretation of f′(x) as the slope of f at x, it is clear that f′ (x) will also fail to exist at a point where the slope is infinite, as for f (x) = (defined over 0 ≤ x < ∞) at x = 0 (Fig. 1.8).

Figure 1.8. Singularity in at x = 0.

1.4. INTEGRATION

Consider next the integral of f (x), say from x = a to x = b, where both a and b are finite; the case where one or both limits are infinite will be considered later. First, we partition the interval a ≤ x ≤ b by choosing an integer n and designating a set of points x0 to xn, not necessarily evenly spaced, such that x0 = a < x1 < x2 < ··· < xn–1 < xn = b, as shown in Fig. 1.9. Second, we designate a ξk point in each subinterval such that xk–1 ≤ ξk ≤ xk and compute the Riemann sum

corresponding to the partition P. Then we choose a finer partition—that is, one with a smaller norm (defined as the size of the largest subinterval and denoted as | P |)—and recompute the Riemann sum that corresponds to the new partition. Doing so for a sequence of partitions whose norms tend to zero, we define the Riemann integral as the limit of the sequence of Riemann sums,

Figure 1.9. A typical partition P.

provided, of course, that the limit does exist¹ ; if so, we say that the integral exists or converges.

So far we’ve assumed that a < b. To complete our definition, we define the integral to be zero if a = b and the negative of the integral from b to a if b < a.

There are theorems which guarantee existence of the integral if f (x) is sufficiently well behaved—for instance, if it is continuous or of bounded variation² over a ≤ x ≤ b. Of more concern to us in applied mathematics, however, is the problem of actually evaluating integrals. First, however, let us consider a moment how we evaluate derivatives. We can compute the derivatives of a number of elementary functions (x², sin x, ex, etc.) directly from the limit definition (1.11) as we did in (1.12). Then we show that the derivative d/dx is a linear operator—that is,

for any scalars α, β and any (differentiable) functions u, v; we show that

and so on. With these operational formulas, as well as a fairly short list of derivatives like d(x²)/dx = 2x, d(sin x)/dx = cos x, …, we can then handle more complicated combinations, such as 3x² sin 4x/(x³ + 1).

The general procedure for evaluating integrals is about the same. We generate a short list of integrals of some simple functions and then add a few operational formulas, such as linearity,

integration by parts,

plus various tricks like partial fractions and clever substitutions that are undoubtedly familiar from calculus. Here and in later chapters other techniques will be developed, including numerical integration, Leibnitz differentiation, asymptotic expansions, contour integration, as well as some additional tricks.

Note, however, that we don’t evaluate the integral of even our simple functions (x², sin x, …) directly from the limit definition (1.14) because it is too unwieldy (exceptions like Exercise 1.16 notwithstanding). Instead we return to the (simpler) realm of differentiation by means of the Fundamental Theorem of the Integral Calculus :

If f(x) is continuous over a ≤ x ≤ b and

for a ≤ x ≤ b, then F′(x) = f (x) over a ≤ x ≤ b.³ Conversely, if f (x) is continuous for a ≤ x ≤ b and F′(x) = f (x), then f (ξ) dξ = F(x).

From this theorem and our list of derivatives it follows that the integral of sin x is – cos x, … and so on.

Consider the integral

It turns out that this integral is not expressible in terms of elementary functions. Yet it appears quite often in applications and so has been given its own name, the error function erf x, and has been tabulated. The is introduced by way of normalization so that erf ∞ = 1, and

is called the complementary error function erfc x. But tables, because of interpolation problems and storage, are not too convenient in computer applications, particularly if the domain is infinite as in (1.20). The trend in recent years has been away from tabulations and toward approximate expressions, not just for erf x but for virtually all of the important special functions. For instance, the expression

where

due to Hastings⁴ is uniformly accurate over 0 ≤ x ≤ ∞ to within ±1.5 × 10–7 ! A virtually indispensable reference for special functions, containing a wealth of computational formulas like (1.22) (as well as a variety of other information), is the Handbook of Mathematical Functions.⁵

Needless to say, we often encounter integrals that we cannot evaluate in terms of known functions—that is, elementary functions and/or special functions for which tables or approximate formulas are available. We may then wish to carry out the integration numerically. Roughly speaking, this process means going back to the Riemann sum definition (1.14) or some variation of it. First, however, let us introduce some important notation.

1.5. ASYMPTOTIC NOTATION AND THE BIG OH

By f (x) ~ g(x) as x → x0; we mean that [f (x)/g(x)] → 1 as x → x0; we say that f is asymptotic to g as x → x0. For instance, (3x⁴ + x + 8)/(x² + 2) is asymptotic to 3x² (i.e., 3x⁴/x²) as x → ∞, to 4 (i.e., 8/2) as x → 0, and to 4 as x → 1, say. As another illustration, recalling the Taylor expansion

it is apparent that sin x ~ x as x → 0 or, revealing a bit more⁶, sin x ~ x – x³/6. But for the limit x → ∞ note that the asymptotic notation is of no help, since there is no simpler function to which sin x is asymptotic; in other words, the best that we can manage is the trivial statement that sin x ~ sin x.

It is often more convenient to use the big oh Bachmann–Landau order of magnitude symbol. By f(x) = O[g(x)] as x → x0, we simply mean that f(x)/g(x) is bounded as x → x0. For example, (3x⁴ + x + 8)/(x² + 2) is O(x²) as x → ∞⁷ and O(1) as x → 0. There is no need to worry about multiplicative constants like 3 or 100, since we are not stating asymptotic behavior but only the order of magnitude. For instance, O(3x²) and O(x²) are entirely equivalent, and so we simply omit the 3. As final illustrations, observe that

Be sure to see Exercises 1.19 and 1.20 concerning ex, e–x and ln x.

1.6. NUMERICAL INTEGRATION

Returning to (1.14), suppose that we break the interval into n equal parts of length (b – a)/n ≡ h (i.e., xk = a + kh) and choose ξk = xk–1—that is, at the left endpoint of each interval. As n → ∞, | P | → 0 and the Riemann sum tends to I. But we’re not going to let n → ∞ (in fact, doing so would be a neat trick!); we’re going to choose a large n and accept whatever error results. Denoting f (ξk) = f (xk–1) ≡ fk–1, we have

This is known as the rectangular rule, and Rn corresponds to the cross-hatched area in Fig. 1.10, where we’ve set n = 4 for simplicity. If f′(x) is continuous over a ≤ x ≤ b, it can be shown⁸ that the discrepancy between I and Rn can be expressed as

for some ξ between a and b. Although there’s no way to determine ξ, (1.24) is still useful as an error estimate. Suppose, for instance, that b – a = 1 and f′(ξ) ≈ 1. If we desire 10–5 accuracy (i.e., to five decimal places), it is clear from (1.24) that we need h ≈ 2 × 10–5 or n ≈ 50,000. Not very encouraging!

Figure 1.10. Rectangular and trapezoidal rules, for n = 4.

Instead suppose that we had chosen ξk = xk—that is, at the right endpoint of each interval instead of at the left. Then we’d have I ≈ h(f1 + f2 + ··· + fn). Adding this result to (1.23) and dividing by 2, we obtain the trapezoidal rule

where Tn can be interpreted as the area under the dashed lines in Fig. 1.10. This time [assuming that f″(x) is continuous over a ≤ x ≤ b] it is found that

where ξ is some point between a and b, not the same as ξ of (1.24) except by coincidence. The important point to note is that whereas was O(h), is O(h²), so that for a given (small) h (1.25) should be appreciably more accurate than (1.23). Or if the rectangular rule needs n ≈ 50,000 to achieve a certain accuracy, the trapezoidal rule will need only n ≈ ≈ 224 for comparable accuracy!

Example 1.2. Suppose that we plan to use the trapezoidal rule to evaluate erf 1 and would like 10–6 accuracy. To determine how small h needs to be, note that

Not knowing ξ, let us be conservative and use ξ = 0, which gives the maximum value of e–ξ²(4ξ² – 2) over 0 ≤ ξ ≤ 1. Then . Equating this result to 10–6 gives h = 0.002306, or n = (1 – 0)/h = 433.7—say 434 to be safe. Using extended-precision arithmetic, n = 434 would probably suffice. Using singleprecision arithmetic, however, machine roundoff might become significant and we may be tempted to increase n somewhat to allow for that. Unfortunately, increasing n is likely to increase the effects of roundoff. In fact, from the calculations presented in Conte⁹ for this example, it appears that 10–6 accuracy simply cannot be achieved by using the trapezoidal rule and single-precision arithmetic, no matter how large we make n.    

But we can do much better than the trapezoidal rule. For instance, recall that I = Tn + O(h²). More precisely, it can be shown (e.g., Conte, p. 127) that

where A is an unknown constant. Suppose that we run two calculations,¹⁰ one with n divisions and a step size of 2h and one with 2n divisions and a step size of h. Then

Multiplying the second equation by 4 and subtracting give

Since Tn and T2n are already computed, (1.28) gives I to within an error that is now O(h⁴), compared with O(h²) for either Tn or T2n individually. Furthermore, we can repeat the process. For instance, if we suspect (or can prove) that (1.27) proceeds in even powers of h, then (1.28) will actually be

and by repeating the preceding process, we can eliminate the Bh⁴ term and be left with an error that is O(h⁶) (Exercise 1.23). This procedure is known as Romberg integration.

Romberg integration actually amounts to an extrapolation process. That is, knowing how the error decays with h in (1.27), basically as Ah¹, we are able to determine A empirically (or, what is equivalent, to eliminate it) by fitting two data points to (1.27)—by writing down (1.27) for two different n’s. This basic idea is important and will be employed again with great success in the discussion on summation of series.

Before continuing, it is interesting to write out (1.28). Recalling (1.25), we have

which is the well-known Simpson’s rule. Derivation of Simpson’s rule usually follows other lines; here it occurs as the first Romberg extrapolation of the trapezoidal rule.

The numerical integration formulas considered thus far have all involved partitions consisting of equal subdivisions and are known as Newton–Cotes-type formulas. They are of the form

where xk = a + kh and the Ak weights are selected (as in the discussion above) so that the resulting error is of reasonably high order in h. Gauss (1777–1855), on the other hand, found that, for a given n, greater accuracy can generally be achieved if the xk abscissas are not fixed a priori but rather are selected together with the Ak weights in an optimal way for a wide class of integrands. But optimal is a rather subjective term. What Gauss meant by optimal was that (1.30) should be exact for all polynomial f (x)’s of as high a degree as possible. Since we have 2n + 2 parameters at our disposal (A0, …, An, x0, …, xn) and an Nth-degree polynomial contains N + 1 parameters (i.e., the coefficients), we expect to be able to obtain a formula that is exact for all polynomials up to and including degree 2n + 1.

It is customary to first change the integration limits from a, b to – 1, 1. This step is easily accomplished by means of the linear change of variables

Then

where F(t) = f[x(t)],¹¹ and so it suffices to consider

For example, let n = 1. Then we want to choose A0, A1, t0, t1 so that (1.33) is exact for all polynomials of degree 3 or less or, equivalently, for F(t) = 1, t, t², and t³. Setting F equal to 1, t, t², t³ in (1.33) yields the algebraic equations

in A0, A1, t0, t1. Although these coupled equations are nonlinear, it’s not hard to show that A0 = A1 = 1 and t0 = – t1 = , so that (1.33) becomes

For n not small, the nonlinear equations for the Ak’s and tk’s are apparently quite unwieldy. However, it can be shown that the tk’s are the zeros of the so-called Legendre polynomials (Exercise 1.24). With the tk’s thus determined, the Ak’s are easily found, since the governing equations are linear in the Ak’s [see, for example, (1.34)]. We have listed the Ak’s and tk’s in Table 1.1 for n = 3 and 9; 15-decimal-place tabulations, for n’s as large as 95, can be found in Abramowitz and Stegun.¹²

Example 1.3. As in Example 1.2, let us compute erf 1.

Using extended-precision, four-point Gauss (n = 3), and ten-point Gauss (n = 9) integration yield I ≈ 0.84270117 and 0.84270079, respectively, compared with the known (tabulated) value I 0.84270079. By way of comparison, we find that the trapezoidal rule (1.25) gives

TABLE 1.1

Abscissas and Weight Factors for Gaussian Integration

Note: The missing values (i.e., k = 2, 3 for n = 3 and k = 5, …, 9 for n = 9) are given by An–k = Ak and tn–k = – tk.

and a double Romberg extrapolation of these values (Exercise 1.23) yields the value 0.84270079.    

Example 1.4. Consider

We know that N-point Gauss integration (i.e., N = n + 1) will be exact (except, of course, for roundoff error) if f (x) is a polynomial of degree 2N – 1 or less and is expected to be inexact but very accurate if f (x) can be closely approximated by such a polynomial. Since f (x) in (1.36) is continuous over 0 ≤ x ≤ 1 we know that it can be uniformly approximated to any desired accuracy by a sufficiently high degree polynomial.¹³ But as x → 0, observe that the integrand behaves like , which has an infinite slope at x = 0 (Fig. 1.8) and is not at all agreeable to polynomial approximation; thus a discouragingly large N will be needed in order to achieve good accuracy. One solution is to make a change of variables, say x = ξ², which stretches out the immediate neighborhood of the origin (e.g., x = 0.01 becomes ξ = 0.1) so that the slope at the origin is brought down to a finite value. In fact, (1.36) then becomes

and we see that the new integrand behaves like 2ξ² as ξ → 0 (since cos ξ⁴ ~ 1 and ξ² + 1 ~ 1), which is fine. Application of Gauss integration to (1.37) will undoubtedly be more successful than to (1.36).

Another interesting feature of (1.36) lies in the cos x² factor. Its frequency increases linearly with x (i.e., cos x² = cos ωx, where the frequency ω = x), so that if our upper limit were 20, say, instead of 1, the integrand would then suffer many oscillations. The integration methods discussed, all of which are fundamentally based on the notion of polynomial approximation, would not be able to cope and special techniques would be necessary.¹⁴    

1.7. DIFFERENTIATION OF INTEGRALS CONTAINING A PARAMETER; LEIBNITZ’S RULE

The integral

is a function of the parameter α. (It is not a function of x, which is only the dummy variable of integration.) As we shall see, it is important to know how to take the derivative

In principle, we can do so by evaluating the integral and then taking d/dα, of it; in practice, however, it may be advantageous somehow to invert the order of integration and differentiation. Forming the difference quotient,

since f is essentially constant (if it is continuous) over the infinitesimal intervals b ≤ x ≤ b + Δb and a ≤ x ≤ a + Δa. Formally letting Δα → 0, we obtain the Leibnitz rule,¹⁵

Suppose that α1 ≤ α ≤ α2, and that c1 ≤ a(α) ≤ c2 and c1 ≤ b(α) ≤ c2 for each α in α1 ≤ α ≤ α2. It can be shown that a sufficient condition for the validity of (1.41) is that ∂ f/∂α be continuous in the rectangular x, α domain c1 ≤ x ≤ c2, α1 ≤ α ≤ α2 (and, of course, that the derivatives ∂ f/∂α, db/dα, and da/dα all exist). For applications that we are likely to encounter, however, it is generally true that (1.41) is correct simply if the integral in (1.41) does converge.

For the special case where a and f are independent of α and b(α) = α, the Leibnitz rule reduces to the Fundamental Theorem of the Integral Calculus, (1.19).

Example 1.5. To illustrate the mechanics of (1.41), note that if

then

Example 1.6. To illustrate how (1.41) can be of help in evaluating integrals, consider

To evaluate it, consider instead

Then (if α ≠ – 1)

Integrating this simple differential equation for J,

To evaluate the constant of integration C, observe from (1.43) that J(0) = 0. Thus 0 = ln 1 + C, so C = 0. Finally,

A word of caution. To compute I′(x), where

it is a good idea first to reexpress

say, to emphasize that the ξ’s and the x are entirely distinct; ξ is the dummy variable of integration and x is the right endpoint of the interval of integration. Thus I’(x) = x².

EXERCISES

Exercises 1.1 through 1.7 are on the algebra of limits. It would be best to read through them even if you plan to work only one or two.

1.1. Prove the simple inequality | A + B | ≤ | A | + | B |, which will be useful in the following exercises. Show that it implies that

1.2. Prove that if f (x) = A and f(x) = B, then B must equal A; that is, the limit must be unique.

1.3. Is f (x) = A equivalent to the statement (f (x) – A) = 0? Prove or disprove.

1.4. Prove (a) . (Allow for the possibility that C = 0.)

(b)

.

I’ll do (b); you do (a). Let f (x) ≡ A and g(x) ≡ B. Then for any > 0 there is a δ1 such that | f(x) – A | < whenever 0 < | x – a | < δ1 and there is a δ2 such that | g(x) – B | < whenever 0 < | x – a | < δ2. Then

for any ′ > 0 whenever 0 < | x – a | < the smaller of δ1, δ2.

1.5. Using 1.4(a) and (b), show that is a linear operator—that is, that

for arbitrary scalars α, β, provided, of course, that the two limits on the right exist.

1.6. Does

Not always; for instance, if a = 0, f (x) = 1/x, and g(x) = x, then the left-hand side is clearly 1, whereas the right-hand side is undefined because 1/x does not exist! But it is true if both limits on the right-hand side exist. For suppose that f (x) → A and g(x) → B. Then for any > 0 there is a δ1 such that | f (x) – A | < whenever 0 < | x – a | < δ1 and there is a δ2 such that | g(x) – B | < whenever 0 < | x – a | < δ2. Let be < 1. Then | f (x)| = | A + [f (x) – A] | < | A | + < | A | + 1; so

Finish the story.

1.7. If f (x) = A ≠ 0, prove that 1/f (x) = 1/A. This time no hints.

1.8. Does x² + y² = 4 define a function y(x) over | x | ≤ 2 ? Discuss.

1.9. Is f (x) = x² continuous over 0 ≤ x ≤ ∞ ? Uniformly continuous? Why (not)? Interpret graphically.

1.10. Show that f (x) = sin x is uniformly continuous over 0 ≤ x ≤ π/2. Give a suitable δ( ) and interpret graphically.

1.11. What is a linear functional? Is F(u) = u(0) linear? F(u) = u²(0)?

*1.12. Show that the distance functions (1.9) and (1.10) satisfy the requirements (1.6) to (1.8).

*1.13. Consider the functional F(u) = u(0), whose domain is the set of all continuous functions u(x) defined over 0 ≤ x ≤ 1. Show whether F(u) is continuous with respect to the distance function (1.10).

1.14. Show that the integral of the Heaviside function is ∫ H(x) dx = xH(x) + C. Sketch H(x), xH(x), and H(x – a).

*1.15. We showed in (1.13) that if f′(x) exists at x, then f must be continuous there. But does the derivative f′(x) need to be continuous there? Well, if f′(x) had a jump discontinuity at x0, then its integral f (x) would have a kink there (e.g., recall the preceding exercise) and would not have been differentiable in the first place. But there are other kinds of discontinuities besides jump discontinuities. In fact, show that for

g′(x) exists at x = 0 but is not continuous there.

(You may be interested in the dictionary of exceptional functions and examples compiled by B. Gelbaum and J. Olmsted in Counterexamples in Analysis, Holden-Day, San Francisco, 1964.)

1.16. Show, directly from the definition (1.14), that

Hint: Choose equal subintervals and choose each at the center of the kth subinterval.

1.17. Show that if m ≤ f (x) ≤ M over a ≤ x ≤ b, then

1.18. (a) xe–x/(cos x + x³) is asymptotic to what, as x → 0?

(b) Give asymptotic expressions for as x → ∞ and x → 0.

1.19. (Important) Show that e–x = O(1) as x → ∞. As a more informative statement, show that e–x = O(x–n) for arbitrarily large n. Note that no matter how large we make n, e–x dies out faster than x–n! Since we can’t seem to get a handle on e–x in terms of powers of x (i.e., since n is arbitrarily large), we often don’t attempt to find a simpler form and simply leave the e–x’s intact. For instance, we might write x²e–x/(cos x + 3x) = O(xe–x) as x → ∞ rather than "O(x–n) for n arbitrarily large and positive." How about e+x as x → ∞ ?

1.20. (Important) Sketch the graph of ln x for 0 < x < ∞. Show that ln x = O(xα) as x → ∞, where α is an arbitrarily small positive number. Thus ln x has an extremely weak singularity at infinity. Show that it has an equally weak singularity at the origin—that is, that ln x = O(x–α) as x → 0, where α is again an arbitrarily small positive number. Since we can’t seem to get a handle on ln x as x → 0 or ∞ (i.e., since α is arbitrarily small), frequently we simply leave ln x as ln x. For example, we might write (3x² + sin x) ln x = O(x² ln x) as x → ∞ rather than "O(x²+α), where α is an arbitrarily small positive number."

1.21. Estimate n so that intigration of dx by the trapezoidal rule is accurate to within ±10–5.

1.22. Whereas the trapezoidal rule

is exact if f (x) is linear, show that Simpson’s rule

is exact if f (x) is a quadratic (i.e., a parabolic arc).

1.23. (Romberg integration) Starting with I = (4T2n – Tn)/3 + Bh⁴ + O(h⁶), obtain

1.24. The Legendre polynomials Pm(x) are defined by the recursion formula

for m = 1, 2, …, where P0(x) ≡ 1 and P1(x) ≡ x. Recall that the tk abscissas in (1.33) (i.e., for the n + 1-point Gauss) are the zeros of Pn+1(x). Compute the tk’s for the four-point Gauss scheme, say, to slide rule accuracy and compare with Table 1.1 (left half). Then with the tk’s known solve the pertinent simultaneous equations for the weights A0, A1, A2, A3, to slide rule accuracy, and compare with Table 1.1.

1.25. Apply the Leibnitz rule to

1.26. Verify that

satisfies the differential equation d²x/dt² + k²x = f(t), plus the initial conditions x(0) = (dx/dt)(0) = 0.

1.27. Knowing that

for a² < 1, show that

1.28. Use the known integral cos ax dx = (sin aπ)/a to evaluate x² cos x dx.

1.29. Evaluate x⁰.⁷ ln x dx.

Hint: Start with the known integral .

1.30. The Bessel function of integral order n may be defined as

Using this expression, show that

1.31. Make up two examination-type questions on the material in Chapter 1. (Potentially, this can be a valuable and nontrivial exercise.)

¹In , δ language, we say that to each > 0 there corresponds a δ( ) > 0 such that

whenever | P | ≤ δ.

²By the total variation of f (x) between two points a and b, denoted , we mean our total up-down travel as we move from x = a to x = b along the graph of f (x). For example, sin x = 2, H(x) = 1, and TV sin 1/x = ∞. If is finite, we say that f is of bounded variation over a ≤ x ≤ b.

³Indefinite Integrals. Sometimes we write ∫ f (x) dx (i.e., without limits) or sometimes ∫x f(x) dx. Called an indefinite integral of f, it denotes a function whose derivative is f. For example, ∫ 2x dx = x² or x² – 37 or, as generally as possible, x² + C, where C is an arbitrary constant.

⁴C. Hastings, Jr., Approximations for Digital Computers, Princeton University Press, Princeton, N.J., 1955.

Note: Besides the footnoted references, a list of Additional References is included at the end of each of Parts I to V.

⁵M. Abramowitz and I. Stegun (Eds.), National Bureau of Standards Applied Math Series, 1964. Updated material concerning the special functions can be found in Y. L. Luke, Mathematical Functions and their Approximation, Academic, New York, 1975.

⁶Actually, we could also argue that sin x ~ x + x³, since sin x/(x + x³) does → 1 as x → 0; but the understanding is that if we bother to include a third-order term, it should at least be correct! That is, transposing the x term, note that sin x – x ~ –x³/6 is true, but sin x – x ~ x³ is not.

⁷Actually, it is O(xp) for any p ≥ 2, but the limiting value p = 2 is the most informative and surely the most natural choice.

⁸See, for example, the very nice book by S. D. Conte, Elementary Numerical Analysis, McGraw-Hill, New York, 1965. Incidentally, you may wonder how f′(x) can exist and not be continuous. If so, see Exercise 1.15.

⁹Ibid., p. 124.

¹⁰Note that half the fk function values needed in T2n (i.e., every other one) have already been computed for Tn and need not be calculated again.

¹¹This is a point of nomenclature that sometimes causes confusion. Suppose, for example, that f (x) = x² and x = 2t + 1. Then f[x(t)] = (2t + 1)² = 4t² + 4t + 1 ≡ F(t). Thus f ( ) = ( )², whereas F( ) = 4( )² + 4( ) + 1; that is, f and F are different functions, which is why we use different names, f and F.

¹²Ibid., p. 916.

¹³ This statement is guaranteed by the famous Weierstrass approximation theorem, which states that if f (x) is real valued and continuous over a ≤ x ≤ b, then given any > 0 (no matter how small) there exists a polynomial p(x) such that | f(x) – p(x) | < for all x’s in the interval.

¹⁴See, for example, I. M. Longman, A Method for the Numerical Evaluation of Finite Integrals of Oscillatory Functions, Mathematics of Computation, Vol. 14, 1960, pp. 53–59.

¹⁵When a mathematician uses formally, what is really meant is "informally"—that is, without worrying about questions of rigor at each step.

*Asterisked material, both in text and exercises, is either a little more difficult or of a supplementary nature and can be bypassed with no loss of continuity.

Chapter 2

Infinite Series

Limit Processes: Differentiation, Integration, and Infinite Series might have been covered in Chapter 1, but the discussion of infinite series is long and best considered separately. As usual, we try to tie together the underlying principles and the practical and computational aspects. For instance, beyond determining if a given series is convergent, there is the crucial practical problem of actually summing it. Thus a convergent series may be so slowly convergent as to be essentially worthless insofar as computation is concerned, whereas merely the first few terms of certain divergent series may provide excellent results. Furthermore, we will see that it is often possible to take a given series and accelerate its convergence substantially by means of certain nonlinear transformations.

2.1. SEQUENCES AND SERIES; FUNDAMENTALS AND TESTS FOR CONVERGENCE

Whereas a finite sum

is well defined (thanks to the associative law of addition), an infinite sum or series

is not because it is clearly not possible actually to add up an infinite number of terms.

To give meaning to (2.2), we define

That is, we form the sequence of partial sums sn, where

and then define the sum of the infinite series as the limit of the sequence of partial sums.¹ If lim sn = s exists,² we say that the series is convergent or converges to s; if not, we say that the series is divergent or diverges.

In "ε language"

means that to each ε (no matter how small) there corresponds an N(ε) such that | sn — s | < ε for all n > N.

It is important to be aware that even though (2.3) is the standard definition, it is not the only one possible. Various summability methods are sometimes employed, of which (2.3) (often called ordinary convergence) is only one.³ For instance, according to Cesàro summability, which is especially useful in the theory of Fourier series,

that is, the limit of the arithmetic means of the partial sums. It can be shown that if a series converges to s (i.e., according to ordinary convergence), then it will also be Cesàro-summable, in fact, to the same value. Yet there are series that diverge in the ordinary sense, that are, nevertheless, Cesàro-summable.

Example 2.1. Consider the well-known geometric series 1 + x + x² + · · ·. Here the terms are functions of x, but we think of x as being fixed so that we actually have a series of constants. Now

and so

provided that x ≡ 1; if x = 1, then clearly sn = n. Letting n → ∞, xn → 0 or diverges, depending on whether |x| < 1 or |x| > 1, respectively. Treat |x| = 1 separately. If x = +1, then sn = n, which diverges (to infinity); and if x = — 1, then the sn sequence oscillates between 1 and 0 and hence again diverges (not to infinity this time but simply because it does not converge). So we can say that the geometric series 1 + x + x² + · · · converges to

and diverges if | x | ≥ 1.

Next, reconsider the geometric series from the Cesàro point of view. We find (Exercise 2.1) that

if x ≠ 1. As before, this result tends to 1/(1 — x) as n → ∞ for | x | < 1 and diverges for |x| > 1 (Exercise 2.2). And as before, we have divergence if x = + 1 (Exercise 2.3). The only difference is that, for x = —1, the limit of (2.8) does exist and equals ,⁴ whereas ordinary convergence tells us the series diverges. Which is correct? Is 1 — 1 + 1— 1 + ··· = , or does it diverge ? The answer depends on whether we define the series according to (2.5) or (2.3), and the definition adopted is up to the person who wrote the series down! From here on we adopt the notion of ordinary convergence (2.3).    

It should be clear that any finite number of (finite) terms of a series—for example, the first six million—in no way affects the convergence or divergence of the series. In other words, it is the tail of the series that counts. This idea is basic to the Cauchy convergence theorem. First,

DEFINITION. We say that sn is a Cauchy sequence⁵ if to each ε > 0 (no matter how small), there corresponds an N(ε) such that | sm — sn | < ε for all m and n > N.

Then we have

THEOREM 2.1. (Cauchy Convergence Theorem) A sequence sn is convergent if and only if it is a Cauchy sequence.

This is unquestionably one of the central concepts of analysis, as is more apparent in Part III. Proof of the theorem consists of two parts: first, we show that if sn is convergent, then it must be a Cauchy sequence; next, we show the converse—that if sn is a Cauchy sequence, it must be convergent.

Proof. Suppose that sn is convergent, say to s. Then for any ε/2 > 0 there must correspond an N such that

Adding,

But using the simple inequality of Exercise 1.1 (with A ≡ sm — s and B = s — sn), we have

for all m, n > N, and so sn must, in fact, be a Cauchy sequence.

Proof of the converse is contained in the * section below.

First some background. Consider a general set S—for example, the set of points 0