MACHINE PROBLEM 1

ES 204

Submitted
by:
Timothy John S. Acosta
Submitted on:
March 18, 2015

Executive Summary: Minimum Residual Norm Iteration

A. Define the Problem
A linear sparse matrix A with a corresponding right hand vector b
was given for this problem. To solve the system, the minimum residual
(minres) norm iteration was used. A program in C for the steepest
descent algorithm was given by Dr. Cortes. It was tasked to compare
the performance of the minres method when using the three different
norms (L1,L2,L). Reference for the accuracy of the solution is the
Gaussian elimination method. (60 pts).
The Minimum Residual Method uses the following algorithm.
Until Convergence
r=b Ax
=( Ar , r ) /( Ar , Ar)
x=x+ r

End
B. Problems Encountered
The first problem encountered was to understand the program
written by Dr. Cortes. After understanding the code written, certain
parts were altered to change the code for the minres algorithm. The
lattar part of the program was also altered to use the different residual
norms since the stopping criterion of the original code was for a L 2
norm.
C. References
Dr. Cortes Program for Steepests Descent
D. Results
Minimum Residual Norm Iteration
L
L2
L1
Gaussian
Norm
Norm
Norm
Elimination
Iterati
ons
Values
x[190]
x[191]

42535

57341

29189

0.8729
54
0.8839
26

0.8728
58
0.8838
39

0.8738
81
0.8847
69

n/a

0.8729
0.8838

0.8950
1
0.9062
07
0.9175
18
0.9289
46

x[192]
x[193]
x[194]
x[195]

0.8949
31
0.9061
37
0.9174
57
0.9288
93

0.8957
69
0.9068
82
0.9181
1
0.9294
53

0.8949
0.9061
0.9175
0.9289

Table 1. Summary of Results for Minres

Executive Summary: Successive over relaxation method
A. Define the problem
For the same given linear system of equations, the successive
over relaxation algorithm was to be used. The optimum omega
(relaxation parameter) was to be determined for the system. Again, the
results were to be compared with using the 3 different norms as
stopping criteria. (60 pts)
The SOR method is presented below:
x

i1

1
)
=
b i a ij x(k+1
aij x (k)
j
j
aii
j=1
j=i +1

(k+1 )
i

B. Problems encountered
Unlike the previous problem, there was no base code to work
from. For this problem, instead of considering the whole matrix A to be
used for computation, only the tri-diagonal band diagonal was stored
as three separate arrays.
0

e m f
0
0 e m

m
e

f 0
m f

The program then used the following algorithm:

Do Until Convergence
b1x 2 f 1
x 1=
m1
For i=2,3,n-1

x i=

bix i+1 f i xi1 e i

mi

x n=

bnx n f n
mn

The relaxation parameter was varied until the smallest iteration

was attained. One of the problems was that the solution did not
converge for relaxation parameters greater than 1.48.
C. References
Gerald and Wheatley.(2004).Applied Numerical Analysis. 7th Ed.
USA: Pearson Education, Inc..
D. Results

SOR Algorithm
70000
60000
50000
40000
30000
20000
10000
0
0.2

0.4

0.6

0.8

1.2

1.4

1.6

Figure 1. Iteration vs Relaxation Parameter using L Norm Stopping

Criteria

Alpha
Iterati
ons
Values
x[190]
x[191]
x[192]

SOR Algorithm
L2
L1
1.477
1.476

L
1.477

9357

12002

6972

0.8728
67
0.8838
46
0.8949

0.8728
52
0.8838
33
0.8949

0.8730
19
0.8839
85
0.8950

x[193]
x[194]
x[195]

38
0.9061
43
0.9174
63
0.9288
98

26
0.9061
32
0.9174
53
0.9288
89

63
0.9062
54
0.9175
6
0.9289
81

Table 2. Summary of Optimal Relaxation Parameters for different

Norms
The optimal relaxation parameter was estimated by running the
programs at different values. As shown in figure 1, the lowest iterations
can be seen at values of 1.4. This gives the general behavior for the
algorithm for the given system of equations. For values higher than
1.5, the solution does not converge using the algorithm. Table 2 shows
the optimal relaxation parameter to an accuracy of 1e-3.

Executive Summary: Conjugate-gradient-squared (CGS) method

A. Define the Problem
A linear sparse matrix B with a corresponding right hand side
vector b was given for this problem. Using a tolerance of 1e-6, the
linear system of equations is to be solved using the CGS method.
The Algorithm for the method is shown below:
1. Compute
2. Set

r o=b Ax ; r o

is arbitrary

po=u o=r o

3. For j=0,1,2 until convergence

j=(r j , r o )/( Ap j , r o )
q j=u j A p j
x j+1 =x j +(u j +q j )
r j +1=r j A (u j+ q j)
j=(r j+1 ,r o)/(r j , r o )
u j +1=r j+1 + j q j

pj +1=u j+1 + j (q j + j p j )
4. End
B. Problems Encountered
For the problem, the base code used was taken from the book
Numerical Recipes 2nd Edition. The first challenged encountered was
to write and understand the code in the book. The code was then
altered accordingly for the given problem. The program makes use
of a function that converts the matrix B into two vectors sa and ija
which only takes into account the non-zero values of the sparse
matrix. This was very confusing at first.

C. References
Press, Teukolsky et al. (1992). Numerical Recipes in C. 2nd Edition.
Cambridge University Press.
Saad, Y. (2000). Iterative Methods for Sparse Linear Systems. 2nd
Edition.

D. Results
CGS Method
Iteration
s
200
Est.
Error
5.31E-12
Values
x[190]
1.008166
x[191]
1.005189
x[192]
1.002212
x[193]
0.999234
x[194]
0.996256
x[195]
0.993278

Table 3. Summary of Results for CGS Method

Gaussian Elimination
Values
x[190]
1.008166

x[191]
x[192]
x[193]
x[194]
x[195]

1.005189
1.002212
0.999234
0.996256
0.993278

Table 4. Solution attained by Gaussian Elimination

The table 3 above shows the partial results for the solution
vector x to the system of equations given by the Matrix B and right
hand vector b while table 4 gives the partial results by Gaussian
elimination. As observed, the values are the same and we can
conclude that an accuracy up to 1e-6 significant figures is obtained.
Although the program computed that the estimated error was 5.32e12, checking the significant figures beyond 6 decimal places is beyond
the scope of this machine problem.

Executive Summary: Bi-conjugate gradient (BiCG) method

A. Define the Problem
The same sparse matrix B with corresponding vector b was
given for this problem. Instead of the CGS method, the BiCG method
is to be used which is a variant of the conjugate method wherein
the transpose of the original matrix is not needed. But instead it
makes use of the projection of two vectors po and uo.
The algorithm is shown below:
1. Compute

r o=b Ax ; ( r o ,r o ) 0

2. Set

po=r o , po =r o

3. For

j=0,1,2

until convergence

 j=(r j , r j)/( Ap j , pj)

x j+1 =x j + p j
r j +1=r j Ap j

r j +1=r j A p j

j=(r j+1 ,r j+1 )/(r j ,r j)

p j +1=r j+1 + j p j
pj +1=r j+1 + j p j
4. End

B. Problems Encountered
For the bi-conjugate method, there was not much of a
problem since the code in C was given in the book numerical
recipes 2nd edition. But the understanding and the encoding of the
functions was rather tedious.
C. References
Press, Teukolsky et al. (1992). Numerical Recipes in C. 2nd Edition.
Cambridge University Press.
Saad, Y. (2000). Iterative Methods for Sparse Linear Systems. 2nd
Edition.

D. Results
Bi-Conjugate
Iterations
Est. Error
Values
x[190]
x[191]
x[192]
x[193]

Method
200
2.33E-15
1.008166
1.005189
1.002212
0.999234

x[194]
x[195]

0.996256
0.993278

Table 6. Summary of Results for BiCG Method

As observed in the table above, it also gives the same values up
to 1e-6 decimal point as the Gaussian elimination solution. The
iterations also give the same value for the CGS method. But the
estimated error for the BiCG was computed to be 2.33e-15 which we
can infer that this would have a more accurate solution if the
significant figures were increased.

Appendix 1. List of Programs

MP1.c <Minres Algo>
#include<stdio.h>
#include<math.h>
#define size 200
double a[size][size];
int main(void)
{
double v[size],r[size],t[size];
double tv[size],x_old[size],x_new[size];
double b[size];

double eps,tol,temp,num,den,h,alpha,max,max2;
int i,j,k,count,n;
n=size; /*number of interior points or number of equations */
h=1.0/(double)(n+1);
tol=1.0e-6;
eps=1.0;
/*Routine for generating the A matrix and the b vector for the homework*/
n=200;
/* h = spacing between points*/
h=1.0/(float)(n+1);
/*initialize the A matrix*/
for(i=1;i<=n+1;i++)
for(j=1;j<=n;j++)
a[i][j]=0.0;
/*The matrix generating routine*/
a[1][1] = -(2.0+4.0*h*h);
a[1][2] = 1.0;
/*generate the main body*/
for(i=2;i<=n-1;i++)
{
a[i][i-1]=1.0;
a[i][i]=-(2.0+4.0*h*h);
a[i][i+1]=1.0;
}
a[n][n-1]=1.0;
a[n][n]=-(2.0+4.0*h*h);
/*now generate the right hand side vector of values*/
for(i=1;i<=n-1;i++)
b[i]=h*h;
b[n]=h*h-1.0;
/*now initialize the guesses with a one vector */
for(i=1;i<=n;i++)
x_new[i]=1.0;
/*initialize the counter*/
count=0;
/*get into the large iteration loop*/
max2=0;
for(i=1;i<=n;i++)
{
max2 += (b[i])*(b[i]);
}

max2=sqrt(max2);
while(eps>tol)
{
for(i=1;i<=n;i++)
{
tv[i]=0.0;
for(j=i-1;j<=i+1;j++)
{
tv[i]+=a[i][j]*x_new[j];
}
}
/*now we calculate the residuals*/
for(i=1;i<=n;i++)
r[i]=b[i]-tv[i];
for(i=1;i<=n;i++)
{
v[i]=0.0;
for(j=i-1;j<=i+1;j++)
{
v[i]+=a[i][j]*r[j];
}
}
num=0.0;
den=0.0;
for(i=1;i<=n;i++)
{
num+=r[i]*v[i];
den+=v[i]*v[i];
}
alpha=num/den;
/*now calculate the adjustments to the vector of guesses*/
for(i=1;i<=n;i++)
x_new[i]=x_new[i]+alpha*r[i];
/*search for the largest error */
max=0.0;
for(i=1;i<=n;i++)
{
max += (r[i])*(r[i]);
}
max=sqrt(max);
/*
for(i=1;i<=n;i++)
{
double diff = fabs(x_new[i]-x_old[i]);
if(diff>max) max=diff;
}

*/
eps=max/max2;
/*update the guesses*/

*/

/*for(i=1;i<=n;i++)
x_old[i]=x_new[i];
count +=1;
if(count%2500==0)
printf("iteration = %5d

tol= %10.8f \n",count,eps);

}
for(i=190;i<=195;i++)
printf("x[%5d] = %12.6f \n",i,x_new[i]);
printf("tol=%10.8f\n",eps);
printf("Total number of iterations = %5d \n",count);
return(0);
}
SOR <main.cpp>
#include<iostream>
#include<math.h>
#include "header.h"
#include "nrutil.h"
#define tiny 1.0e-15
#define SWAP(a,b) {temp=(a);(a)=(b);(b)=temp;}
using namespace std;
int main(){
int n=200,i,ctr=0,choice;
double mx,h,eps,tol=1e-6,alpha;
sarray
sarray
sarray
sarray
sarray
sarray

e=defsarray(n);
m=defsarray(n);
f=defsarray(n);
b=defsarray(n);
xnew=defsarray(n);
xold=defsarray(n);

h=1/(double)(n+1);
while(choice!=-1){
ctr=0;
//Initializing Tri-Band Matrix and RH
for(i=0;i<n;i++)
{
e.elems[i]=1.0;
m.elems[i]=-(2.0+4.0*h*h);
f.elems[i]=1.0;
b.elems[i]=h*h;
xold.elems[i]=1;

xnew.elems[i]=1;
}
b.elems[n-1]=h*h-1.0;
eps=1;
printf("\nEnter relaxation parameter\n");
scanf("%lf",&alpha);
//SOR Algo
while(eps>tol){
//First Part
xnew.elems[0]=(b.elems[0]-xold.elems[1]*f.elems[0])/m.elems[0];
xnew.elems[0]=(alpha*xnew.elems[0])+(1-alpha)*xold.elems[0];
//Interior Part
for(i=1;i<n-1;i++)
{
xnew.elems[i]=(b.elems[i]-(xold.elems[i+1]*f.elems[i])-(xnew.elems[i1]*e.elems[i]))/m.elems[i];
xnew.elems[i]=(alpha*xnew.elems[i])+(1-alpha)*xold.elems[i];
}
//End Part
xnew.elems[n-1]=(b.elems[n-1]-(xold.elems[n-2]*e.elems[n-1]))/m.elems[n-1];
xnew.elems[n-1]=(alpha*xnew.elems[n-1])+(1-alpha)*xold.elems[n-1];
//Using L2 Norm
mx=0;
for(i=0;i<n;i++){
double diff=xnew.elems[i]-xold.elems[i];
if(fabs(diff)>mx) mx=fabs(diff);
}
eps=mx;
ctr++;
//Update the Guess
for(i=0;i<n;i++)
xold.elems[i]=xnew.elems[i];
if(ctr%2500==0)
printf("iteration=%5d
}

tol=%10.8f \n",ctr,eps);

for(i=189;i<195;i++)
printf("%10.8lf\n",xnew.elems[i]);
printf("Total Iterations is = %d\n",ctr);
printf("Input -1 to exit\n");
scanf("%d",&choice);
}
return 0;
}

Bi-conjugate function
void linbcg(unsigned long n,double b[],double x[],int itol, double tol, int itmax,int
*iter, double *err)
{
void asolve(unsigned long n, double b[],double x[],int itrnsp);
void atimes(unsigned long n, double x[],double r[],int itrnsp);
double snrm(unsigned long n, double sx[],int itol);
unsigned long j;
double ak,akden,bk,bkden,bknum,bnrm,dxnrm,xnrm,zminrm,znrm;
double *p,*pp,*r,*rr,*z,*zz;
p=dvector(1,n);
pp=dvector(1,n);
r=dvector(1,n);
rr=dvector(1,n);
z=dvector(1,n);
zz=dvector(1,n);
*iter=0;
atimes(n,x,r,0);
for(j=1;j<=n;j++){
r[j]=b[j]-r[j];
rr[j]=r[j];
}
if(itol==1){
bnrm=snrm(n,b,itol);
asolve(n,r,z,0);
}
else if(itol==2){
asolve(n,b,z,0);
bnrm=snrm(n,z,itol);
asolve(n,r,z,0);
}
else if(itol==3 || itol ==4){
asolve(n,b,z,0);
bnrm=snrm(n,z,itol);
asolve(n,r,z,0);
znrm=snrm(n,z,itol);
} else printf("hello\n");
while(*iter<=itmax){
++(*iter);
asolve(n,rr,zz,1);
for(bknum=0.0,j=1;j<=n;j++) bknum+=z[j]*rr[j];
if(*iter==1){
for(j=1;j<=n;j++){
p[j]=z[j];
pp[j]=zz[j];
}
}
else{
bk=bknum/bkden;

for(j=1;j<=n;j++){
p[j]=bk*p[j]+z[j];
pp[j]=bk*pp[j]+zz[j];
}

}
bkden=bknum;
atimes(n,p,z,0);
for(akden=0.0,j=1;j<=n;j++) akden+=z[j]*pp[j];
ak=bknum/akden;
atimes(n,pp,zz,1);
for(j=1;j<=n;j++){
x[j]+=ak*p[j];
r[j]-=ak*z[j];
rr[j]-=ak*zz[j];
}
asolve(n,r,z,0);
if(itol==1)
*err=snrm(n,r,itol)/bnrm;
else if(itol==2)
*err=snrm(n,z,itol)/bnrm;
else if(itol==3 || itol ==4){
zminrm=znrm;
znrm=snrm(n,z,itol);
if(fabs(zminrm-znrm)>EPS*znrm){
dxnrm=fabs(ak)*snrm(n,p,itol);
*err=znrm/fabs(zminrm-znrm)*dxnrm;
} else{
*err=znrm/bnrm;
continue;
}
xnrm=snrm(n,x,itol);
if(*err<=0.5*xnrm) *err/=xnrm;
else{
*err=znrm/bnrm;
continue;
}
}
if(*iter%500==0) printf("iter=%4d err=%12.6f\n",*iter,*err);
}

if(*err<=tol) break;

free_dvector(p,1,n);
free_dvector(pp,1,n);
free_dvector(r,1,n);
free_dvector(rr,1,n);
free_dvector(z,1,n);
free_dvector(zz,1,n);

Conjugate Gradient Squared Function

void lincgs(unsigned long n,double b[],double x[],int itol, double tol, int itmax,int
*iter, double *err)

{
void asolve(unsigned long n, double b[],double x[],int itrnsp);
void atimes(unsigned long n, double x[],double r[],int itrnsp);
double snrm(unsigned long n, double sx[],int itol);
unsigned long j;
double ak,akden,bk,bkden,bknum,bnrm,dxnrm,xnrm,zminrm,znrm;
double *p,*r,*rr,*z,*zz,*u,*q,*ro,*uq;
p=dvector(1,n);
ro=dvector(1,n);
uq=dvector(1,n);
q=dvector(1,n);
r=dvector(1,n);
rr=dvector(1,n);
z=dvector(1,n);
zz=dvector(1,n);
u=dvector(1,n);
*iter=0;
atimes(n,x,r,0);
for(j=1;j<=n;j++){
r[j]=b[j]-r[j];
rr[j]=r[j];
}
if(itol==1){
bnrm=snrm(n,b,itol);
asolve(n,r,z,0);
}
else if(itol==2){
asolve(n,b,z,0);
bnrm=snrm(n,z,itol);
asolve(n,r,z,0);
}
else if(itol==3 || itol ==4){
asolve(n,b,z,0);
bnrm=snrm(n,z,itol);
asolve(n,r,z,0);
znrm=snrm(n,z,itol);
} else printf("hello\n");
for(j=1;j<=n;j++) ro[j]=rr[j];
while(*iter<=itmax){
++(*iter);
//asolve(n,rr,zz,1);
for(bknum=0.0,j=1;j<=n;j++) bknum+=r[j]*rr[j];
if(*iter==1){
for(j=1;j<=n;j++){
p[j]=r[j];
u[j]=r[j];
}
}
else{

bk=bknum/bkden;
for(j=1;j<=n;j++){
u[j]=r[j]+bk*q[j];
p[j]=bk*(bk*p[j]+q[j])+u[j];
//pp[j]=bk*pp[j]+zz[j];
}

}
bkden=bknum;
atimes(n,p,z,0);
for(akden=0.0,j=1;j<=n;j++) akden+=z[j]*rr[j];
ak=bknum/akden;
for(j=1;j<=n;j++) q[j]=u[j]-ak*z[j];
for(j=1;j<=n;j++) uq[j]=u[j]+q[j];
atimes(n,uq,zz,0);
for(j=1;j<=n;j++){
x[j]+=ak*(u[j]+q[j]);
r[j]-=ak*zz[j];
//rr[j]-=ak*zz[j];
}

asolve(n,r,z,0);
if(itol==1)
*err=snrm(n,r,itol)/bnrm;
else if(itol==2)
*err=snrm(n,z,itol)/bnrm;
else if(itol==3 || itol ==4){
zminrm=znrm;
znrm=snrm(n,z,itol);
if(fabs(zminrm-znrm)>EPS*znrm){
dxnrm=fabs(ak)*snrm(n,p,itol);
*err=znrm/fabs(zminrm-znrm)*dxnrm;
} else{
*err=znrm/bnrm;
continue;
}
xnrm=snrm(n,x,itol);
if(*err<=0.5*xnrm) *err/=xnrm;
else{
*err=znrm/bnrm;
continue;
}
}
printf("iter=%5d err=%12.6f\n",*iter,*err);
if(*err<=tol) break;

free_dvector(p,1,n);
free_dvector(u,1,n);
free_dvector(q,1,n);
free_dvector(uq,1,n);
free_dvector(r,1,n);
free_dvector(rr,1,n);
free_dvector(z,1,n);
free_dvector(zz,1,n);

Functions needed for CGS and BiCG

double snrm(unsigned long n, double sx[],int itol)
{
unsigned long i,isamax;
double ans;

if(itol<=3){
ans=0;
for(i=1;i<=n;i++) ans+=sx[i]*sx[i];
return sqrt(ans);
} else {
isamax=1;
for(i=1;i<=n;i++){
if(fabs(sx[i])>fabs(sx[isamax])) isamax=i;
}
return fabs(sx[isamax]);
}

void atimes(unsigned long n,double x[],double r[], int itrnsp)

{
void dsprsax(double sa[],unsigned long ija[],double x[],double b[],unsigned long
n);
void dsprstx(double sa[],unsigned long ija[],double x[],double b[],unsigned long n);
if(itrnsp) dsprstx(sa,ija,x,r,n);
else dsprsax(sa,ija,x,r,n);
}
void asolve(unsigned long n,double b[],double x[],int itrnsp)
{
unsigned long i;
for(i=1;i<=n;i++) x[i]=(sa[i] !=0.0 ? b[i]/sa[i]:b[i]);
}
void dsprsin(double **a,int n, double thresh, unsigned long nmax,double
sa[],unsigned long ija[])
{
void nrerror(char error_text[]);
int i,j;
unsigned long k;
for(j=1;j<=n;j++) sa[j]=a[j][j];
ija[1]=n+2;
k=n+1;
for(i=1;i<=n;i++){
for(j=1;j<=n;j++){
if(fabs(a[i][j]) >= thresh && i != j){
if(++k >nmax) nrerror("dsprsin: nmax too small");
sa[k]=a[i][j];
ija[k]=j;
}

}
ija[i+1]=k+1;

<main.cpp> for BiCG and CGS

#include<iostream>
#include<math.h>
#include "header.h"
#include "nrutil.h"
#define tiny 1.0e-15
#define SWAP(a,b) {temp=(a);(a)=(b);(b)=temp;}
#define EPS 1.0e-6
#define size 200
int main(){
int i,j,n=size;
int nmax=2*n*n+1;
double **a;
double *b;
double h=1/(double)(n+1);
double *x;
x=dvector(1,200);
a=dmatrix(1,200,1,200);
b=dvector(1,200);
ija=lvector(1,nmax);
sa=dvector(1,nmax);
//Initialize the A matrix
for(i=1;i<=n;i++)
for(j=1;j<=n;j++)
a[i][j]=0;
for(i=1;i<n;i++)
b[i]=0;
//Create the coefficient matrix
a[1][1]=-(2.0-4.0*h*h);
a[1][2]=1.0+5.0/2.0*h;
for(i=2;i<=n-1;i++)
{
a[i][i-1]=1.0-5.0/2.0*h;
a[i][i]=-(2.0-4.0*h*h);
a[i][i+1]=1.0+5.0/2.0*h;
}
a[n][n-1]=1.0-5.0/2.0*h;
a[n][n]=-(2.0-4.0*h*h);
//Now generate the right hand side
for(i=1;i<=n-1;i++)

b[i]=h*h;
b[n]=h*h-(1.0-5.0/2.0*h);
//Converting A Matrix
dsprsin(a,n,0,nmax,sa,ija);
//Initialize the guesses with a one vector
for(i=1;i<=n;i++) x[i]=1.0;
//Bi-Conjugate Method
double tol=1e-6;
int itol=1;
int itmax=50000;
int iter,ii;
double err;
lincgs(n,b,x,itol,tol,itmax,&iter,&err);
printf("%s %15e\n","Estimated error:",err);
printf("%s %6d\n","Iterations needed:",iter);
printf("\nSolution vector:\n");
for(ii=1;ii<=n/5;ii++){
for(i=5*(ii-1)*1;i<=5*ii;i++) printf("%12.6f",x[i]);
printf("\n");
}
printf("\n\n");
for(i=190;i<195;i++)
{
printf("%12.6f\n",x[i]);
}
return 0;
}

Appendix 2. Details of Cell Phone compiler

Compiler: Cppdroid App
Model: O+ Model 8.37z Android
OS: Android 4.2 (Jelly Bean)
Processor: Dual Core 1.2 GHz
SoC: Mediatek MTK6572

Memory: 512 MB RAM

*All computational time took less than 15 seconds to finish