Linear_Systems.

html

Page 1 of 19

Systems of Linear Equations

>

restart;

Some Different Ways of Solving Linear Systems Using Maple's solve Command
In order to use solve we need to write the system of linear equations in equation form . Here are three linear equations in the three unknowns
>

, and

eqn1 := x1 + 2*x2 + 3*x3 = 1; eqn2 := 2*x1 + x2 + 4*x3 = 1; eqn3 := 3*x1 + 4*x2 + x3 = 1;

The solution obtained via solve:

>

solution := solve({eqn1, eqn2, eqn3}, {x1, x2, x3});

And verification of the answer:

>

subs(solution, eqn1); subs(solution, eqn2); subs(solution, eqn3);

The use of solve is not recommended , however, in general as the following example of five linear equations in five unknowns shows.
>

eqn1 := x1 + 2*x2 + 3*x3 - 4*x4 - 5*x5 = 1; eqn2 := 2*x1 + x2 + 4*x3 + 3*x4 - 2*x5 = 1; eqn3 := 3*x1 + 4*x2 + x3 + 2*x4 + 3*x5 = 1; eqn4 := 4*x1 - 2*x2 + 4*x3 + x4 + 3*x5 = 1; eqn5 := 5*x1 + 4*x2 - 2*x3 + 4*x4 + x5 = 1;

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 2 of 19

>

solution := solve({eqn1, eqn2, eqn3, eq4, eq5}, {x1, x2, x3, x4, x5});

A solution does indeed exist as we shall see below. Maple's solve just can't handle this problem.

Using Maple's linalg Package

A more natural approach is to formulate the problem in matrix-vector form . Maple has a large package containing all kinds of linear algebra routines. Let's load it.
>

with(linalg):

Warning, new definition for norm Warning, new definition for trace

Now we define the coefficient matrix

>

and right-hand side vector

of the 5x5 problem above:

A := matrix([[1, 2, 3, -4, -5], [2, 1, 4, 3, -2], [3, 4, 1, 2, 3], [4, -2, 4, 1, 3], [5, 4, -2, 4, 1]]); b := vector([1, 1, 1, 1, 1]);

And then we use the command linsolve to solve the system

>

x:=linsolve(A,b);

To verify this answer we need to explain Maple's somewhat awkward handling of vectors and matrices. 1) To multiply these kind of objects we need to use &* instead of *.

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 3 of 19

>

A&*x;

2) In order to actually see the result of this computation we need to apply evalm to it.
>

evalm(%);

We could also have done everything in one statement.

>

evalm(A&*x=b);

Calling Matlab from within Maple

Especially for large problems it will pay off to make use of Matlab's superior and faster matrix handling. Short of just dumping Maple and using Matlab instead, Maple also provides us with an easy to use interface to Matlab. To make use of this we load the Matlab package .
>

with(Matlab):

Warning, new definition for det Warning, new definition for transpose

We can then pass our matrix

>

and vector

defined in Maple to Matlab using the command setvar .

setvar("A", A); setvar("b", b);

Now we can enclose basically any Matlab statement within the Maple command evalM (note the difference between evalm and evalM ). Matlab's way of solving the system vector . Due to the special role of the backslash character this symbol has to be repeated in order to be effective in the following example. Also note that we are using ' and not inside Matlab since a standard Maple vector is passed to Matlab as a row is via the statement x = A\b , where is assumed to be a column

vector , and the symbol ' transposes a vector/matrix.

>

evalM(" x= A \\ b' ");

What remains to be done is to get the result of the Matlab computation back into Maple. This is accomplished by getvar .
>

x_Matlab := getvar("x");

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 4 of 19

In order to compare the Matlab answer to the Maple answer computed above we need to evalf Maple's answer (note the use of evalm ).
>

evalf(evalm(x));

Some Nasty Examples The Vandermonde Matrix

The matrix below is called the Vandermonde matrix . We have already seen it in our discussion of poynomial interpolation. We solve a linear system with a 4x4 Vandermonde matrix as system matrix below. At the end we print the difference between the value . This quantity is called the residual .
> A:=matrix([[1,2,4,8], [1,3,8,27], [1,4,16,64], [1,5,25,125]]); b:=vector([15,40,85,156]); x:=linsolve(A,b); residual=evalm(A&*x-b);

and a right-hand side

as defined

, where

is our computed solution, and the right-hand side

Let's solve some more systems involving Vandermonde matrices, and this time do things numerically in single precision. We print residuals. Note: since
>

, ideally the residual should be a zero vector .

Digits:=8: for n from 4 to 12 do A:=matrix([seq([seq(evalf((1+i)^(j-1)), j=1..n)], i=1..n)]);

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 5 of 19

b:=vector([seq(evalf(((1+i)^n-1)/i), i=1..n)]): x:=linsolve(A,b): print(residual=evalm(A&*x-b)); od:

As we can see, the residuals become increasingly worse as the matrix The explanation for this phenomenon is that is ill-conditioned .

grows.

Multiquadric Interpolation
As another example of an ill-conditioned matrix we look at interpolation using so-called multiquadrics . The basic idea is that, instead of interpolating to given data with a polynomial or spline, we use a function of the form ,

where the

are coefficients to be determined, and

is a parameter to be selected in advance by the user.

First we define some data, and plot this data along with the function that generated it. n:=6: x := vector([seq(i, i=0..n)]): y := vector([seq(evalf(exp(-(x[i]-n/2)^2)), i=1..n+1)]): data := [seq([x[i], y[i]], i=1..n+1)]: data_plot := plot(data, style=POINT, color=red): funct := plot(exp(-(t-n/2)^2), t=0..n, color=green): plots[display](data_plot, funct);
>

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 6 of 19

We set the parameter

>

(try different choices for

here, e.g.,

).

r:=.1;

The coefficients

will be determined by solving the system of linear equations for ,..., .

which we get by enforcing

the interpolation conditions First we construct .

As an indicator of problems to be expected we print the condition number of

>

A:=matrix(n+1,n+1,0): for i from 1 to n+1 do for j from 1 to n+1 do A[i,j] := evalf(sqrt((x[i]-x[j])^2+r^2)); od: od: A=evalm(A); condition_number_of_A = cond(A);

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 7 of 19

Now we find the coefficients

>

c:=linsolve(A,y);

And finally we compute the fit

>

using the formula stated above

f := t->sum('c[i]*sqrt((t-x[i])^2+r^2)', 'i'=1..n+1);

Here is a plot of the data (including the original function) along with the multiquadric interpolant we computed.
>

mq_plot := plot(f(x), x=0..n, color=blue): plots[display](data_plot, mq_plot, funct);

Note that for "interpolant"!

the matrix is so ill-conditioned that the interpolation conditions are no longer satisfied by the

The Benefits of Row Swapping

For this example we will simulate a computer with double precision.
>

Digits:=15:

We will use the following system matrix and right-hand side vector:
> original_A:=matrix([[6,2,2],[2,2/3,1/3],[1,2,-1]]); original_b:=vector([-2,1,0]);

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 8 of 19

For our experiment we want to do floating point computations , so

>

A_float := evalf(evalm(original_A)); b_float := evalf(evalm(original_b));

In order to simulate Gaussian elimination we first generate the augmented matrix which we will call command augment .
>

using Maple's

A:=augment(A_float,b_float);

We produce zeros in the first column below

>

with the help of addrow :

A:=addrow(A,1,2,-A[2,1]/A[1,1]); A:=addrow(A,1,3,-A[3,1]/A[1,1]);

The last step of the elimination phase is given by

>

A:=addrow(A,2,3,-A[3,2]/A[2,2]);

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 9 of 19

Now we can find the solution of

>

by back substitution . Maple offers the command backsub for this purpose.

x:=backsub(A);

Let's compute the residual Ax-b:

>

residual := evalm(A_float&*x-b_float);

This does not look good! What is the exact solution? Maple's exact arithmetic tells us
>

linsolve(original_A,original_b); evalf(%);

What went wrong above? Let's check the condition number of A:

>

cond(A_float);

There should not be a problem because of this. If we go back, we see that in the last elimination step centers around this number. Let's check whether things will improve if we switch rows 2 and 3 of A before we do the last elimination step. We start as above
>

was an extremely small number, and last computation

A:=augment(A_float,b_float);

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 10 of 19

The first elimination step is also the same:

>

A:=addrow(A,1,2,-A[2,1]/A[1,1]); A:=addrow(A,1,3,-A[3,1]/A[1,1]);

Here is the difference : we swap rows 2 and 3 using the command swaprow .
>

A:=swaprow(A,2,3);

and then complete the elimination

>

A:=addrow(A,2,3,-A[3,2]/A[2,2]);

Let's see what the solution is now.

>

backsub(A);

Obviously, much better! This example motivates the concept of partial pivoting .

LU-Factorization
LU factorization is an efficient matrix version of Gauss elimination .

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 11 of 19

As we will see below, it can be used beneficially for solving multiple linear systems of the form ...., all with the same system matrix .

How It Works
The result of LU-factorization is that we can represent , where is a permutation matrix taking recording all the row swaps performed during Gauss elimiation with partial pivoting (i.e., P is an identity matrix whose rows have been interchanged), is a lower triangular matrix (in fact the inverse of the lower triangular matrix containing all the multipliers used during Gauss elimination), is an upper triangular matrix (the endresult of Gauss elimination). We start our first example by defining a matrix
>

in the following way:

and right-hand side

restart; with(linalg): A := matrix([[1, 2, 3, -4, -5], [2, 1, 4, 3, -2], [3, 4, 1, 2, 3], [4, -2, 4, 1, 3], [5, 4, -2, 4, 1]]); b := vector([1, 1, 1, 1, 1]);

Warning, new definition for norm Warning, new definition for trace

Next we perform LU factorization (or decomposition) using Maple's LUdecomp command from the linalg package. Since Maple actually thinks of the LU decomposition of in the form , the returned by Maple is

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 12 of 19

actually our

(which is the same as

, the transpose of

, since

is a permutation matrix).

We print out

, and

, and also check whether

, as it is supposed to be.

>

U := LUdecomp(A, P='Pt', L='L'); L=evalm(L); Pt=evalm(Pt); P:=transpose(Pt); evalm(P&*A = L&*U);

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 13 of 19

Since we used exact arithmetic there was no need to do any pivoting. For the next example we use floating point arithmetic.
>

A:=matrix(evalf([[6,2,2],[2,2/3,1/3],[1,2,-1]])); b:=vector(evalf([-2,1,0]));

Once more we compute the LU factorization of

>

and check the endresult.

U := LUdecomp(A,P='Pt',L='L'); P := transpose(Pt); evalm(P&*A = L&*U);

If we take a look at

, we see that this time row swapping did occur.

In fact, rows 2 and 3 were interchanged, which is exactly what we did by hand in the previous section.

Why Is LU-Factorization Important?

We will now solve multiple linear systems. As an example, consider the MQ interpolation example again.

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 14 of 19

First we generate some nodes at which we wish to interpolate.

>

n:=10: x := vector([seq(i, i=0..n)]): different sets of data (or right-hand sides), and put them all in a matrix , instead of a

Next we generate vector as before.

Each column of

corresponds to a different set of data. (The first column is generated by the same function we used

earlier, the other columns by slightly different functions).

>

Y := matrix([seq([seq(evalf(exp(-(x[i]-n/2)^2)/j), i=1..n+1)], j=1..n+1)]): and set up the matrix , so . is the same for all interpolation problems .

Finally, we fix a value for Note that

depends only on the nodes

You can think of the problems we are solving as one big problem for which different types of data are given at a fixed set of measurement stations, and we want to fit all of these different sets of data. For example, the nodes could correspond to wheather stations, and the different columns of could be rainfall,

temperature, air pressure, etc. data at the respective locations.

>

r:=.1: A:=matrix(n+1,n+1,0): for i from 1 to n+1 do for j from 1 to n+1 do A[i,j] := evalf(sqrt((x[i]-x[j])^2+r^2)); od: od:

Now we will use three different methods to solve these n+1 interpolation problems simultaneously, and time how long each one of them takes. First we do Gauss elimination followed by back substitution one at a time on the augmented matrices [ where
>

],

is the

-th column of

start:=time(): U := gausselim(augment(A,col(Y,1))): X := backsub(U): for j from 2 to n+1 do U := gausselim(augment(A, col(Y,j))); X.j := backsub(U); X := augment(X,X.j); od: printf("Time to solve %d systems with Gaussian elimination: %f\n", n+1, time()-start);

Time to solve 11 systems with Gaussian elimination: 2.142000

Next we do the LU-factorization of

only once followed by a series of forward and back substitutions for each

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 15 of 19

column of

The reason why this works is that LU decomposition gives us

, or

Therefore,

becomes

or

We can now break the solution down into two steps: first we solve then we solve
>

by forward substitution (since by back substitution (since

is lower triangular), is upper triangular).

start:=time(): U := LUdecomp(A,P='Pt',L='L'): z := forwardsub(augment(L,transpose(Pt)&*col(Y,1))): X := backsub(augment(U,z)): for j from 2 to n+1 do z := forwardsub(augment(L,transpose(Pt)&*col(Y,j))); X.j := backsub(augment(U,z)); X:=augment(X,X.j); od: printf("Time to solve %d systems with LU-factorization: %f\n", n+1, time()-start);

Time to solve 11 systems with LU-factorization: 2.390000

As a third alternative we use Maple's linsolve , which can handle matrix right-hand sides directly:
>

start:=time(): X:=linsolve(A,Y): printf("Time to solve %d systems with linsolve: %f\n", n+1, time()-start);

Time to solve 11 systems with linsolve: 1.991000

In order to appreciate the benefits of LU decomposition you need to complete the assignment below .

Tridiagonal and Banded Systems

We explore the use of specialized procedures for solving linear systems with tridiagonal (and more generally, banded) system matrices . An application area in which these kind of systems frequently arise is in the approximate solution of differential equations . In particular, if we want to use the so-called finite difference method to solve the boundary value problem

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 16 of 19

on [

] using

discretization points, then a tridiagonal system arises whose system matrix is of the type

and whose right-hand side is of the form .

First we see how Maple performs on this problem. Maple has a command band in its linalg package which enables us to define a banded matrix. However, there is no specialized solution procedure for banded systems, so we must use linsolve to do this. In the experiment below, we solve a sequence of 7 finite difference approximations to the BVP given above, each time essentially doubling the number of discretization points . As a performance measure we print the time it takes Maple to solve the problem along with the number of points used and the ratio of the most recent time to the time for the previous solution. At the end we plot the approximate solution , which is simply a collection of points close to the solution curve.
>

with(linalg): n:=2: printf("#points time for band and linsolve rate\n\n"); for k from 1 to 7 do start:=time(): n:=2*n: h:=evalf(1/n): A:=evalf(band([-1,2+Pi^2*h^2,-1], n-1)): x:=evalf(vector([seq(1.*h*i,i=1..n-1)])): b:=evalf(vector([seq(2*h^2*Pi^2*sin(Pi*x[i]),i=1..n-1)])): u:=linsolve(A,b): new_time := time()-start: if k > 1 then rate := new_time/old_time; printf("%4d %15.3f %9.6f\n", n+1, new_time, rate); else printf("%4d %15.3f\n", n+1, new_time); fi; old_time:=new_time; od: plot([[0,0], seq([x[i],u[i]],i=1..n-1), [1,0]], color=red, style=point);
time for band and linsolve .111 .027 .243243 rate

#points 5 9

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 17 of 19

17 33 65 129 257

.084 .329 1.003 3.844 15.189

3.111111 3.916667 3.048632 3.832502 3.951353

As we can see, the solution process seems to perform on the order of

Thus, Maple seems to be using some kind of intelligence here since ordinary Gauss elimination or LU-factorization is an process.

>

restart;

Now we write our own specialized procedure for solving tridiagonal systems (see book p.251).
>

tri := proc(n, a, dd, c, bb) local i, xmult, b, d, x; b := bb; d := dd; x := vector([seq(0, i=1..n)]); for i from 2 to n do xmult := a[i-1]/d[i-1]; d[i] := d[i] - xmult*c[i-1]; b[i] := b[i] - xmult*b[i-1]; od; x[n] := b[n]/d[n]; for i from n-1 by -1 to 1 do x[i] := (b[i]-c[i]*x[i+1])/d[i]; od; RETURN(eval(x)); end:

The following lines repeat what we did previously using Maple's built-in functions, i.e., we solve 7 finite difference approximations (using our procedure tri ) on increasingly finer discretizations, and print the execution times and ratios.

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 18 of 19

n:=2: printf("#points time for tridiadonal solver rate\n\n"); for k from 1 to 7 do start:=time(): n:=2*n: h:=evalf(1/n): a:=evalf(vector([seq(-1, i=1..n-2)])): d:=evalf(vector([seq(2+Pi^2*h^2, i=1..n-1)])): x:=evalf(vector([seq(1.*h*i,i=1..n-1)])): b:=evalf(vector([seq(2*h^2*Pi^2*sin(Pi*x[i]), i=1..n-1)])): u:=tri(n-1,a,d,a,b): new_time := time()-start: if k > 1 then rate := new_time/old_time; printf("%4d %15.3f %9.6f\n", n+1, new_time, rate); else printf("%4d %15.3f\n", n+1, new_time); fi; old_time:=new_time; od: plot([[0,0], seq([x[i],u[i]],i=1..n-1), [1,0]], color=red, style=point);
> #points 5 9 17 33 65 129 257 time for tridiadonal solver .065 .013 .029 .054 .096 .262 .495 .200000 2.230769 1.862069 1.777778 2.729167 1.889313 rate

This process is clearly more efficient . In fact it seems to be about

Assignment

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008

Linear_Systems.html

Page 19 of 19

Repeat the last experiment in the section on LU-factorization with for the three methods.

and compute the ratios of execution times

What do you conclude?

http://math.iit.edu/courses/Linear_Systems1.html

1/14/2008