Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Genetic Algorithms
and Neural Networks
William Sayers (05025397)
Supervised by: Colin W. Morris (BSc, MSc)
Backpropagation neural networks are usually trained using some
iterative method derived from a mathematical analysis of the operation of the
network. This technique is time consuming and requires understanding of the
mathematical principles involved.
This project will investigate the application of genetic algorithms to replace the
"normal" training of the network. This involves setting up some candidate
networks and then picking the best of these. The best of the population are
then combined in some way to produce a new population of candidate
networks. This procedure is continued until a satisfactory network is obtained.
Contents
Contents ............................................................................................................................ 2
1 - Introduction ............................................................................................................... 10
2 - Research .................................................................................................................... 10
2.3.1 - Advantages................................................................................................... 14
Page 2 of 225
William Sayers 05025397
3 - Design ....................................................................................................................... 38
Page 3 of 225
William Sayers 05025397
4 - Implementation ......................................................................................................... 46
Page 4 of 225
William Sayers 05025397
Figure 30 - Mean squared error upon last training test (XOR) ...................................... 61
Page 5 of 225
William Sayers 05025397
Figure 33- Mean squared error upon last training test (Iris data) ................................... 64
Figure 34- The mean squared error at testing (Iris data) ................................................ 65
6 - Comparisons .............................................................................................................. 68
Page 6 of 225
William Sayers 05025397
7 - Evaluation ................................................................................................................. 70
9 - Appendices ................................................................................................................ 74
11 - Bibliography.......................................................................................................... 214
Page 7 of 225
William Sayers 05025397
13 - Diary...................................................................................................................... 222
Page 8 of 225
William Sayers 05025397
STATEMENT OF ORIGINALITY
SCHOOL OF COMPUTING
This is to certify that, except where specific reference is made, the work described
within this project is the result of the investigation carried out by myself, and that neither this
project, nor any part of it, has been submitted in candidature for any other award other than
this being presently studied.
Any material taken from published texts or computerised sources have been fully
referenced, and I fully realise the consequences of plagiarising any of these sources.
Page 9 of 225
William Sayers 05025397
1 - Introduction
The purpose of this report is to bring to a conclusion my final year project on neural
networks and genetic algorithms, on the course “Computer Games Development”.
This report encompasses design, development and final prototyping of the program
developed for this project. In addition, the report will investigate, compare and contrast three
common learning algorithms that are applicable to neural networks.
2 - Research
2.1 - Neural Networks
Neural networks are networks of small-specialised units that sum the products of their
inputs and the weights on those inputs. Although they are simple in concept, they can
produce immensely complex results.
The human brain is composed of many millions of nerve cells called “Neurons”.
Neurons do not make decisions alone, but in combination with a network of other neurons
within the brain.
Page 10 of 225
William Sayers 05025397
A neuron is composed of a cell body, with extensions called “Dendrites” which are the
neuronal inputs and a single extension called an “Axon” which is the output for that neuron.
The axon and dendrites both split before terminating into “Axon branches” and “Synapses”.
Axon branches connect to synapses to form the neural networks within the brain. Axon
branches can connect to other types of cells also, in order to allow the brain to control cells
external to the brain.
Concerning inputs from the dendrites, the nucleus of the neuron cell processes them
and produces an output. If the inputs from the dendrites to the neuron nucleus when summed
are past a certain threshold, the nucleus will generate an “action potential”. This action
potential extends down the axon to the cells at the end of the axon branches (often synapses
connecting to other neuronal cells, as mentioned above). In the case of the axon branches
connecting to synapses that are part of a second neuron, a synaptic gap occurs between the
axon branch and the synapse, in this gap between neurons chemical reactions take place that
will inhibit or excite the signal.
(Bourg & Seeman, 2004), (Barber, AI: Neural Network for Beginners pt. 1, 2006)
In order to construct an artificial neuron we need a way of representing the same model
in data structures and algorithms. The usual method for approaching this is to have a number
of inputs, a number of weights associated with those inputs, a summing function, an
activation function and a method for representing an output – either to another artificial
neuron in the next layer, or to another part of the program.
Page 11 of 225
William Sayers 05025397
The system devised with these components is as follows. A specific input accepts a data
item (usually a binary digit, real number, or integer). The product of the data item and its
associated weighting is then stored before the summation of all the data items and associated
weights occurs. The activation function accepts this stored value.
The next stage depends on the activation function chosen. A stepping function can be
used which will output zero (or non-activated) if the summed products are below a certain
value, or one (activated) if the summed products are above a certain value. Alternatively, a
differentiable function of some kind (such as a sigmoid function) which will output values
between zero and one depending on the value of the inputs is another possibility.
Page 12 of 225
William Sayers 05025397
1.2
0.8
0.6
0.4
0.2
0
-6 -4 -2 0 2 4 6
1
=ݕ
1 + ݁ ି௧
Equation 2 - A sigmoid function (Bourg & Seeman, 2004). In this example “y” is the function output and “t” is
the function input, as defined in the graph above.
(Bourg & Seeman, 2004), (Barber, AI: Neural Network for Beginners pt. 1, 2006)
McCulloch and Pitts formulated the first neural network model which featured digital
neurons but the neurons had no capability to learn (McCulloch & Pitts, 1943).
Frank Rosenblatt developed the perceptron model then, in 1958, however he was
unable to come up with a reliable mathematically accurate mechanism for allowing multi-
layer perceptrons to learn (Rosenblatt, 1958).
Page 13 of 225
William Sayers 05025397
Two layer perceptrons were the subject of several experiments to determine their usage
and effectiveness. The next major advancement in neural networks was not until 1974, when
Werbos (Werbos, 1974) discovered the back-propagation algorithm (independently re-
discovered by Parker in 1982 (Parker, 1982)).
Werbos discovered the algorithm whilst working on his doctoral thesis in statistics and
called it “dynamic feedback”, Parker discovered it independently in 1982 whilst performing
graduate work at Stanford University in the United States of America and called it “learning
logic”.
(Blum, 1992).
Neural networks make possible or practical many things that with conventional
solutions would be extremely difficult.
One of the benefits of neural networks when compared with other problem solving
techniques in that they are inherently parallel and thus can run very effectively and efficiently
on parallel hardware, used on this architecture they become an extremely fast as well as
flexible tool.
The individual neurons in a neural network have no time dependencies on each other
and each can therefore be run on a separate processor (or as separate threads on a single
processor) if desired without causing the problems often associated with such parallelism.
A good example of this is that neural networks have been implemented to run on
architecture originally designed for processing three dimensional computer graphics, taking
Page 14 of 225
William Sayers 05025397
advantage of the massively parallel architecture of this hardware and the programmable
sections of the graphics pipeline (vector and fragment shaders) to produce extremely fast
neural networks (Davis, 2001).
Neural networks can excel at determining what data in a particular set is relevant, as
irrelevant data will simply end up weighted so that it has zero or close to zero actual effect on
the solution produced by the network.
When using a neural network there's no need to establish before attempting problem
solving which data is relevant. We can simply present the neural network with all the data as
opposed to following the circuitous and overly complex (by comparison) statistical approach
demonstrated in “Table 1 - The statistical approach to deciding relevant data”.
Another advantage of neural networks is that noisy data is not a real problem for them
to learn to interpret; in fact, noisy data is utilised during training upon occasion, to attempt to
provide a more robust post-training network.
2.3.2 - Disadvantages
One disadvantage to neural networks can be that it is very difficult for a human being to
analyse a trained neural network and describe why it may come up with one answer over
another. This is a large disadvantage in areas such as medical diagnosis, where explaining
why you have arrived at a particular diagnosis is an important part of the process.
Page 15 of 225
William Sayers 05025397
Once a neural network’s training is complete it is hard to tell why it is coming up with a
particular solution, it can also be hard to tell how well (or how badly) trained a neural
network is.
A classic example is that in the 1980s, the US military wanted to use a neural network
to analyse images and determine whether they contained a concealed tank, for obvious
purposes.
They collected one hundred images with a concealed tank, then a further one hundred
images without. Fifty pictures of each set were set aside and the neural network trained with
the remaining one hundred images with and without tanks. After the neural network’s
training was complete, they tested the network with the remaining fifty images that the neural
network had not seen before and the neural network correctly identified most of the tanks.
The US military then tested the neural network with a further set of one hundred
images, with and without tanks, only to find that on this round of testing, the network came
up with apparently random answers.
Eventually the original data set was re-examined. All the photographs with tanks were
from a cloudy day and all the images without, a sunny day. Therefore, asked to differentiate
between the two sets of pictures, the network had trained itself to recognise whether the sky
was cloudy or not.
When using neural networks is that, there is no formal methodology with which to
choose the architecture, train the neural network, or even to verify the quality of the training.
A tool used to solve heuristic problems, requires largely heuristic construction and
preparation.
The training times on neural networks can also be a disadvantage, although the length
of time the problem would take to solve without employing a neural network must be taken
into account when deciding how much of a disadvantage.
Page 16 of 225
William Sayers 05025397
Another possible criticism is that neural networks are extremely dependant on the
quality and content of the original training data set. The more time spent training the neural
network on quality data sets, the more accurate and in line with expectations and desires, the
eventual result will be.
This drawback is however one that is generally universal to computing (and many other
areas) often referred to as “GIGO” or “Garbage in, Garbage out”. Neural networks make it
more apparent however because (as in the previous US Military example) it can be difficult
sometimes to decide what constitutes good data.
Fortunately since neural networks are (as has been mentioned above) good at ignoring
irrelevant data, the usual best approach is to simply feed them as much data as can be
obtained, covering as many relevant situations as possible; this has to be done within reason
however, or it does exacerbate the problem with the time spent training the neural network.
The real strength of neural networks, in the opinion of the author, lies in their ability for
generalisation.
Data sets for training neural networks generally separate into two sub-sets, “training”
and “testing”. The network training first takes place using the appropriate “training” set.
Testing may then take place with the “testing” set to determine accuracy with unseen data.
This is generalisation and it is far more useful than it at first appears and similar from a
very high-level perspective to how the human brain and brains in general function.
There is a large amount of research now covering the applications of neural networks in
medicine. One of the main research areas is using neural networks to recognise
diseases/conditions via scans.
Page 17 of 225
William Sayers 05025397
This is particularly suitable work for neural networks because you do not need perfect
examples of how to recognise the diseases and conditions, merely a variety of scans covering
all possible permutations of the disease or condition.
You can also use the technology to model parts of the human biological system in order
to better understand it, for example the human cardio-vascular system.
The main reasons for using a neural network to model the human cardio-vascular
system are that a neural network is capable of adjusting itself to be relevant to a particular
individual and also to adapt by itself to changes in that individual.
In the 1980s, research on using neural networks to diagnose diseases took place.
However, a clear problem with this approach is that when using a neural network it would be
difficult to tell how a neural network reached its conclusion.
Neural networks, being good at analysing patterns and predicting future trends, can fit
very well into most business situations.
Businesses have used neural networks in the past for applications such as assisting the
marketing control of airline seat allocations in the AMT (Airline Marketing Tactician).
A back-propagation neural net is integrated with the airline marketing tactician which
monitored and recommended on booking for each flight, thus supplying information more or
less directly linked to the airlines main income.
Credit scoring, mortgage screening, and assessing borrowers have all incorporated
neural networks as an integral part of the system.
Page 18 of 225
William Sayers 05025397
Predicting the trajectory of an object in real time is another area where neural networks
have been utilised. The position, velocity and acceleration of the object being estimated in
those implementations by several neural networks using several of the most recent
measurements of the object coordinates.
When controlling manipulators a neural net must deal with four major problems (Robot
Control, 2008):
• Forward Kinematics
• Inverse Kinematics
• Dynamics
• Trajectory generation
The ultimate goal of course is to position the “effecter” (I.e. a robotic hand or any other
similar manipulator) in the appropriate position to grasp or otherwise manipulate an object or
part of an object. When a robot arm is one hundred percent accurate that is a simple task,
(relatively speaking) involving the following steps (Robot Control, 2008):
A neural network allows the robotic arms to be very flexible in their operation and
perform self-adjustments as time goes by, compensating automatically for wear and tear on
Page 19 of 225
William Sayers 05025397
themselves. This is far preferable to having to perform lengthy and expensive re-calibrations
on any regular basis (Robot Control, 2008).
2.5 - Back-propagation
2.5.1 - Back-propagation overview
The advantage of using a function like this is that it allows us to differentiate how close
we came to the correct result, allowing us to apply our learning algorithm; the disadvantage
of a differentiable function is that it is by the use of these functions that local optima become
apparent in the search space.
In a single layer neural network (or with a single node) the perceptron rule is
applicable. This rule essentially describes adjusting the weight using the difference between
the expected and actual outputs.
Page 20 of 225
William Sayers 05025397
Equation 3 – The delta rule (Matthews, Back-propagation for the Uninitiated, 2002)
However, this rule has no positive effect in a multi-layer neural network, as the effect
that the change in the weight will have on the rest of the network is a missing factor.
Provided you are using a differentiable function, such as the sigmoid function (see 2.1.2
-The Artificial Neuron) an alteration can be made to the perceptron learning rule allowing
this difficulty to be overcome. The sigmoid function differentiates very neatly.
Equation 4 - differentiated sigmoid function (Matthews, Back-propagation for the Uninitiated, 2002)
Therefore, the following alteration to the perceptron learning rule would serve our
purposes (Matthews, Back-propagation for the Uninitiated, 2002):
Equation 5 - Altered perceptron learning rule (Matthews, Back-propagation for the Uninitiated, 2002)
In this equation “w” and “x” are as before, “n” is the learning rate of the neural
network. The other two values “di and yi” are the actual and desired outputs, respectively.
Assuming the sigmoid activation function is in use, the output layer nodes training
function can be written as:
Equation 6 - Altered delta calculation rule for the output layer (Matthews, Back-propagation for the
Uninitiated, 2002)
Calculating the hidden layer deltas is a little more complicated. The effect that the
change will have on the following neurons in the network is an extra factor.
݊݅ ݍ ݊ݎݑ݁݊ ݎܨℎ݅݀݀݁݊ ݈ܽ ݎ݁ݕ: ߜ ሺݍሻ = ݔ ሺݍሻ[1 − ݔ ሺݍሻ] ݓାଵ ሺݍ, ݅ሻߜାଵ ሺ݅ሻ
Page 21 of 225
William Sayers 05025397
Equation 7 - Altered delta calculation rule for the hidden layer(s) (Matthews, Back-propagation for the
Uninitiated, 2002)
As demonstrated above, to calculate the delta for a hidden layer requires the deltas for
the following layers. Therefore, in order to apply this learning mechanism you start at the
output of the network and work your way towards the inputs.
The FANN library was decided upon for several reasons, it is fast, easy to implement,
open-source (allowing me to alter it if I needed too, although as it turned out I did not), in a
language that can be linked to C# fairly easily and it also supports cascade neural networks,
saving me finding a separate library.
Natural selection (Darwin, 1859) very neatly avoids one of the larger problems
involved in software design: how to specify in advance all the possible permutations of the
original problem and how the program should react to those permutations.
Most organisms evolve by means of sexual reproduction and natural selection (Darwin,
1859)
Page 22 of 225
William Sayers 05025397
Sexual reproduction ensures mixing and recombination of genes. During the process
chromosomes line up together and then cross over partway along their length, swapping
genetic material, this mixing leads to much faster evolution than if off-spring simply copied
the genetic material as it is in the parents.
Mutation plays a part in this as well, occasionally causing the genetic information,
during this process, to alter slightly in a random way. This could result in either an effective
or defective solution, but this variety helps to prevent stagnation in the gene pool.
Genetic algorithms first appeared on computers in the late 1950’s and early 1960’s,
mostly designed to help evolutionary biologists’ model aspects of natural evolution. By 1962,
many researchers had independently developed evolution-inspired algorithms for function
optimisation and machine learning but did not get much follow up to their work.
In 1965 one Ingo Rechenberg, then of the technical University of Berlin introduced a
technique he named evolution strategy, although this technique involved no crossover, or
indeed multiple genomes in a population.
In 1975, the publication of the book “Adaptation in Natural and Artificial Systems”
occurred, by the author John Holland, built on papers both by himself and by other
researchers. This book presents the concepts of adaptive digital systems using mutation,
selection and crossover, simulating processes of biological evolution.
By the early 1980s, genetic algorithms were becoming more widely used, across a
range of subjects. At first, this application was largely theoretical; however, genetic
Page 23 of 225
William Sayers 05025397
algorithms soon moved into commercial territory and nowadays help solve many problems in
many different areas.
(Marczyk, 2004)
The primary advantage of genetic algorithms is their inherent parallelism. Many other
algorithms are largely serial and can only explore a search-tree so far in any one direction
before abandoning all progress and starting again from the beginning, or further up the
hierarchy.
Genetic algorithms effectively explore many different branches of the tree at once and
when a certain branch turns out to be non-optimal, abandon that search, proceeding with
other more likely candidates.
This leads to the exploration of a large proportion of the search space, with processing
power exponentially dedicated to better areas as the exploration progresses.
The real advantage of parallelism however is that by evaluating the relatively small
search-space that it does, a genetic algorithm is implicitly evaluating a much larger group of
individuals, in the same way that the average response of a relatively small percentage of the
population of any given country, can be used to fairly accurately predict national trends.
This is “Schema Theorem” and allows a genetic algorithm to move towards the search-
space with the most promising individuals in a timely fashion and then select the best
member of that group (Marczyk, 2004).
Thanks to the parallelism that is a genetic algorithms main advantage, they are very
well suited as a means of exploring search spaces too large to be effectively searched in a
reasonable amount of time by other, more conventional, methods. These are “non-linear”
Page 24 of 225
William Sayers 05025397
problems. In a linear problem, each component’s fitness is individual and any improvement
to individual component's fitness is necessarily an improvement to the whole. Many real life
problems are not like this, they tend towards non-linearity where altering one component's
fitness positively may make other components less fit and cause a ripple effect across the
whole system. This non-linearity results in a huge increase in the search space, making
genetic algorithms an effective way to search them, due to their strength in navigating large
search spaces (Marczyk, 2004).
A third advantage of genetic algorithms is that they do not tend to be easily trapped by
local optima, due again to the parallelism of their approach, the random nature of the various
starting points of the initial population and the other methods they employ, such as crossover
and mutation.
In infinite or very large search spaces, it is hard to know whether we have reached the
global optimum, or merely very good local optima. However, genetic algorithms tend to give
good results compared to other search strategies.
One of the largest strengths of genetic algorithms can at first glance appear to be their
largest weakness. Genetic algorithms have no prior knowledge about the problem they are
trying to solve, they produce random changes to their candidates and then use the objective
function to determine whether those changes are positive or negative overall.
This allows them to discover solutions that other algorithms may have over-looked, or
never contained in their search space in the first place. A good example of this is the concept
Page 25 of 225
William Sayers 05025397
of negative feedback, rediscovered by genetic algorithms, but denied a patent for several
years because it ran counter to established beliefs (Marczyk, 2004).
2.7.2.2 - Disadvantages
Another method is to use genetic programming, where the actual code of the program
forms the genomes in the genetic algorithm and it can swap portions in or out, or adjust them.
A second disadvantage of genetic algorithms is that the fitness function is crucial to the
development of a suitable solution for the problem. If you have a poorly written fitness
function, the genetic algorithm may end up solving an entirely different problem from the
originally intended one.
A third issue is setting the correct mutation rate, population size, etc. If the mutation is
too high, the system will never converge towards a suitable solution, however if it is too low,
a comparatively smaller segment of the search space will be covered and the eventual
solution may take longer to reach or never be reached as a result. Likewise, if the size of the
population is too low, the investigation may cover too little of the search space to find the
optimum solution (Marczyk, 2004).
A further problem is that fitness functions can be deceptive, if the solutions to various
problems are in areas of the search space that the fitness function would not consider likely,
then genetic algorithms (and most other search techniques) are no better than a random search
for finding the solution.
For example in the diagram below, the global optima is at the far right, however that is
only one point, there is no slope leading to it, therefore no gradual increase can lead you to it,
Page 26 of 225
William Sayers 05025397
you either have the solution or you don’t. In this example a properly constructed genetic
algorithm would be likely to settle on the marginally less optimal, but altogether easier to
find, local optima in the centre of the diagram, unless by random chance it discovered the
point with the most optimum solution.
900
800
700
600
500
400
300
200
100
0
0 2 4 6 8 10 12
A further problem is premature convergence: this occurs when a mutation early in the
process produces a large leap in the fitness of that particular genome, which will then
reproduce abundantly, lowering the diversity of the population and resulting in genetic
algorithms possibly falling into the local optima that the mutation represents. There are
various methods for solving this, including sigma scaling and Boltzmann selection (Mitchell,
1996).
Finally, where analytical solutions exist they should be take precedence over genetic
algorithms. This is because analytical solutions usually produce more accurate results faster
than a genetic algorithm (or any heuristic method) is capable of (Marczyk, 2004).
Page 27 of 225
William Sayers 05025397
The error surfaces associated with these problems tend to be highly variable and
contain many local optima, however the method of choice for training neural networks is
usually back propagation (see 2.5 -Back-propagation) which is a gradient descent algorithm
and as such can be easily trapped in local optima (Branke, 1995). There are methods to try to
avoid this problem when using back propagation, such as adding a “momentum” to the
algorithm, to try to help it move beyond local optima; however, these are not a complete
solution (Blum, 1992).
Genetic algorithms (as described in 2.7 -Genetic Algorithms) are good at avoiding
being stuck on local optima due to their searching several regions of search space
simultaneously. Genetic algorithms also have the benefit that they do not place any
restrictions whatsoever on the architecture of the neural network, since you are merely
increasing the size of the genomes being worked with, not altering them besides that in any
way.
In addition, the only information genetic algorithms require to train a neural network is
an objective function (as described in 2.7 -Genetic Algorithms), to test the appropriateness of
a given genome.
Page 28 of 225
William Sayers 05025397
Crossover can disrupt genes far apart, but is less likely to do so for genes close
together. Because of this it is helpful to keep functional units close together (I.e. place a
neurons input and output weights, a nodes input weights and the nodes of a layer, side by
side).
One of the most important decisions when deciding how to represent a neural network
within a genome is whether to use binary strings or real number strings.
Standard genetic algorithms use binary strings, however representing a large number of
real valued numbers as a binary string, can lead to very large strings (thousands of bits)
(Branke, 1995).
As well as using a genetic algorithm to evolve the weight set for a fixed-structure
neural network, it is also possible to use a genetic algorithm to evolve a structure for a neural
network.
The average neural network has an input layer, one or two hidden layers of an
indeterminate number of nodes and one output layer. The problem at hand dictates the input
and output layer structures.
These are all connected, outputs to inputs, to form a feed forward network.
Page 29 of 225
William Sayers 05025397
This architecture is then altered (number of nodes in each hidden layer, number of
hidden layers) by trial and error and intuition to establish a neural network that performs well
for the problem at hand.
However, this is not necessarily the most efficient form for a network for any given
problem and the trial and error method is an inefficient method for determining the correct
architecture for an optimally performing neural network for the problem at hand.
If neural network architecture is too simplistic for the problem at hand, the neural
network will never learn to solve the problem (or come close enough to suggest a solution).
On the other hand, if the neural network architecture is too complex, the neural network
will learn fast but will generate too specific a solution and will not generalize between similar
inputs. As mentioned in section 1.0, a neural networks generalisation is its main strength and
therefore you do not wish to lose it.
Although network architecture plays such a large role in network performance, there is
currently no method by which to establish, given a specific problem, a neural network
topology to deal with that problem. There is also no method to check how optimal your
solution is (Branke, 1995).
Page 30 of 225
William Sayers 05025397
Ideally, a representation is required which can accommodate all networks that will
work for the problem but none that will not, thus eliminating the searching of meaningless
space and by definition, being able to contain the most optimum solution for the problem at
hand.
There are two methods for representing neural network topologies within a genetic
algorithm, the strong, direct, or low-level encoding and the weak, indirect, or high-level
encoding (Branke, 1995).
In direct encoding, the genetic algorithm represents the neural network by means of its
connections, thus meaning that in order to remove a node completely, you must remove all
connections to or from that node completely.
In cases where a genetic algorithm determines the number of hidden layers and the
number of nodes per hidden layer and the network itself is interconnected fully, an exception
is possible.
In one version of high level encoding of a neural network, the network is divided into
areas and for each area the number of nodes and the density of connections to other nodes is
determined (Branke, 1995).
Back propagation learning rate is also a parameter for one of these areas.
Page 31 of 225
William Sayers 05025397
One of the reasons why alternatives to low-level encoding arose is the exponential rise
in the number of connections, as the size of the neural network increases. These mapping
methods are therefore more suitable for small networks or networks with a small number of
connections.
This could be useful, by limiting the potential size of the networks evolved and
preventing the generation of overly large and complex networks but on the other hand, it is
impossible to know whether these large and complex networks may be the most suitable way
to solve the problem at hand.
As mentioned in section 5.2 neural networks that are too large will solve a problem fast
but will not generalize well and therefore their usefulness is limited severely. Neural
networks that are too small on the other hand, will not ever succeed in learning the problem.
Therefore, a balance between speed of learning and generalisation exhibited in the trained
network is necessary.
A problem with high level encoding is that regular networks are favoured but not every
network is equally probable. This can lead to human bias entering the system and affecting
the outcome, which is obviously undesirable.
This section largely drawn from information in: “Curved trajectory prediction using a
self-organizing neural network” (Branke, 1995).
Page 32 of 225
William Sayers 05025397
The system begins with a basic neural network of inputs and outputs (numbers of each
dictated by the task as usual) and no hidden layer.
The connection between each input and output is via an adjustable weight. There is also
a bias input, with a value of one. The output units may use either a linear or differentiable
activation function. It then trains the adjustable weights using the perceptron rule (as
described in 2.5.2 -Back-propagation in depth), or sometimes the quick prop rule (Fahlman,
An Empirical Study of Learning Speed in Back-Propagation Networks, 1988), with the aim
of getting as close to a solution as possible.
After a number of training cycles pass with no substantial improvement (the precise
number being determined by a user-supplied “patience” parameter); either an extra node is
added, or if the networks output is of sufficient quality the training is ended.
If a trigger occurs for the addition of a unit, the initiation of a number of “candidate”
units takes place and trained using either the perceptron rule (2.5.2 -Back-propagation in
depth) or the quick prop rule (Fahlman, An Empirical Study of Learning Speed in Back-
Propagation Networks, 1988).
The “candidate” nodes take the neural net inputs, plus the outputs of any previously
created units, and attempt to maximise “s” the sum overall output units “o” of the magnitude
of the correlation between “v”, the candidate unit’s value, and “Eo”, the residual output error
at “o” (Fahlman & Lebiere, The Cascade-Correlation Learning Architecture, 1991).
Equation 8 - The definition of 's' given that 'o' is the network output at which the error is measured, 'p' is the
training pattern, 'v' is the candidate unit's value, the quantities ‘`v ’ and ‘`Eo’ are the values of v and Eo averaged
over all patterns (Fahlman & Lebiere, The Cascade-Correlation Learning Architecture, 1991).
In order to maximise "s", the partial derivative of "s" with respect to each of the
candidate unit’s incoming weights "wi" is a necessity.
Page 33 of 225
William Sayers 05025397
ߜݏ
ߜݓ
Equation 9 - The partial derivative of ’s’ (Fahlman & Lebiere, The Cascade-Correlation Learning
Architecture, 1991).
ߜݏ
= ൫ܧ, − `ܧ ൯݂′ ܫ,
ߜݓ
,
Equation 10 - The partial derivative of ‘s’ with respect to each of the candidates incoming weights ‘wi’, where
‘oo’ is the sign of the correlation between the candidates value output ‘o’, ‘f'p’ is the derivative for pattern 'p' of the
candidate unit's activation function with respect to the sum of it’s inputs and ‘Ii,p’ is the input the candidate receives
from unit ‘i’ for pattern ‘p’.
After these computations are complete, a gradient descent to maximize "s" can take
place.
Once the correlation on a candidate has reached an acceptable level, or the completed
training epochs reaches a maximum level, The connection of the best of the candidates to the
neural network as it stands takes place, and the inputs for that node are frozen.
Adjustable weights link the outputs for this node and all the output nodes inputs. The
training iterations for these adjustable weights in the main network then continue.
Page 34 of 225
William Sayers 05025397
Figure 6 - The initial state of a cascade correlation neural network. The circular connections are adjustable
weights (Fahlman & Lebiere, The Cascade-Correlation
Cascade Correlation Learning Architecture, 1991).
1991)
Page 35 of 225
William Sayers 05025397
Figure 8 - The third state of a cascade correlation neural network, with two nodes added (Fahlman & Lebiere,
The Cascade-Correlation Learning Architecture, 1991).
Page 36 of 225
William Sayers 05025397
Another challenge is making the neural network training independent of the user
interface, in order to avoid the interface freezing whilst training a neural network. The
solution I plan for this is to use the background worker object provided with C#, which
allows you to assign a delegate function to be your background task, and will trigger events
upon completion, for which event handlers will be executed (Microsoft, BackgroundWorker
Class, 2009).
Making a C# user interface interact with C++/CLI dll’s simply involves adding a
reference to the dll in your project, via the Visual Studio® user interface.
The VC++ compiled can swap between managed and unmanaged code on the fly whilst
compiling and allows you to have both managed and unmanaged code (in C++) as part of the
same project. Thus, a managed class can communicate with the unmanaged class (performing
appropriate marshalling of the non-primitive data types (Microsoft, Using C++ Interop
(Implicit PInvoke), 2009)) and the C# code can then access it directly.
Page 37 of 225
William Sayers 05025397
Section 2.11 -Writing C++/CLI Wrappers for Unmanaged C++ code, covers the
combination of C# and C++ so I will not bother to cover it here, I plan to use the DLL
method, with one DLL combining managed and unmanaged C++ linked into my C #user
interface code.
The user interface will then have methods to adjust parameters such as the genetic
algorithms population size, number of generations, etc. In order to test and see what kind of
genetic algorithm learns and develops fastest. It will also allow you to specify the number of
back-propagation iterations.
The use of this application will be in evaluating the difference in performance between
a genetic algorithm, back-propagation algorithm and cascade correlation algorithm as neural
network training algorithms.
3 - Design
The purpose of this application will be to allow the user to run a back-propagation
trained neural network, or a genetic algorithm trained neural network and present the results
of the network in such a fashion as to allow analysis and comparison of the training methods
and their suitability in different situations.
Page 38 of 225
William Sayers 05025397
The two prior designs to this involved representing neural networks via objects and
compositions of objects in vectors, which lead to flexible, but slow and overly complex code
upon experimentation.
The final solution, the solution documented here, is a far simpler one (in terms of the
neural network representation, it is far more detailed in the other areas than the initial designs
were).
This solution loosely follows a model, view, controller architecture. The solution
enables easy replacement of sections of code as long as the interfaces and outputs remain the
same.
For more information on the class structure, see section 9.2 -UML Class Diagram.
Page 39 of 225
William Sayers 05025397
This is superior to the direct C# to unmanaged code linking that used in milestone two,
since it makes for cleaner and thus more easily maintainable code and the code is more robust
in this fashion.
The robustness comes from the fact that C++/CLI is far better at interacting with
unmanaged C++ code than C# is. Indeed – they can even be in the same source file and
compiled into one object. Therefore, by using a C++/CLI wrapper to interface to the
unmanaged C++ and then interacting with the C++/CLI wrapper/s errors are less likely to
arise.
I also used the ZedGraph control (ZedGraph) to display the graph of my results.
The main form is the crux of the application (as is default in C# Winforms
programming) and if the main form is closed, all other forms and classes are disposed and the
application terminates.
Page 40 of 225
William Sayers 05025397
This tab needs a method by which to initialise the adding of a new network (3.4.2 -New
Network Wizard) a method to train the current network and a method to test that training.
Three buttons will therefore be present, along with two displays. One display will show
network details and one display will show the training and testing output.
The mean squared error graph tab is a simple tab, which simply needs to display the
graph of the mean squared error over the current epoch/generation.
Page 41 of 225
William Sayers 05025397
This tab needs two display sections, to show the training data and to show the testing
data.
Page 42 of 225
William Sayers 05025397
The new network wizard is the form that appears when the user selects the creation of a
new network. It allows you to move in a logical progression through the set up process of a
new neural network and associated training algorithm. The user can progress through the
steps via the “previous” and “next” buttons or via the tabs at the top in whatever order you
choose. Verification of all data takes place before the form passes control back to the main
form and triggers a main form event with a parameter that takes a specially constructed class
to transfer the data from the wizard.
On the learning algorithm tab you select the learning algorithm for the neural network
you are creating, out of three choices previously mentioned (2.5 -Back-propagation, 2.7 -
Genetic Algorithms, 2.9 -Cascade Correlation).
This tab allows you to set appropriate settings for the network and training algorithm
you have selected in 3.4.2.1 -Learning Algorithm tab (if no algorithm has yet been selected,
the default is genetic algorithm).
Page 43 of 225
William Sayers 05025397
The dataset tab allows you either to select from two pre-setup datasets, or to choose
custom data sets from your hard disk for training and testing data (appropriately formatted in
the FANN style).
The working form is merely a small display shown as a dialog (lock out access to
displaying form) which shows whilst a neural network is training, both to protect the main
form from data inputs that may cause training problems and to provide a visual indicator of
work taking place.
Page 44 of 225
William Sayers 05025397
The about form displays information about the application (author, project supervisor,
project description), displays licensing information and displays information on the pre
selected datasets. Clicking the datasets button or the licenses button displays the appropriate
information, clicking the same button again hides that information.
Clicking the data sets button displays a tabbed interface, one tab with XOR information
and one tab with Fishers Iris Data information.
Page 45 of 225
William Sayers 05025397
Clicking the licenses button displays a small information panel with selection methods;
depending on your selection, it will display the appropriate license.
4 - Implementation
The implementation is largely in C#, with small portions in C++/CLI and the back-end
FANN (Nissen, FANN) library written in C and linked as a static library to the managed
FANN_Wrapper dynamic link library.
Page 46 of 225
William Sayers 05025397
Page 47 of 225
William Sayers 05025397
Page 48 of 225
William Sayers 05025397
Page 49 of 225
William Sayers 05025397
Page 50 of 225
William Sayers 05025397
Page 51 of 225
William Sayers 05025397
Page 52 of 225
William Sayers 05025397
Page 53 of 225
William Sayers 05025397
Passing the data between the wizard form and the main form takes place as previously
described (2.10 -C# User Interface Programming) via events and delegate functions.
Page 54 of 225
William Sayers 05025397
The declaration and association of a delegate and an event takes place in the form that
needs to generate an event (9.1.1.3 -“Wiz1.cs”, lines 29 – 31).
The “NetworkUpdatedEventArgs” class is a storage class for passing data (see 9.1.1.1
-“NetworkUpdatedEventHandler”).
The declaration of a new event handler in the main form takes place prior to displaying
the new neural network wizard form (see 9.1.1.2 -“FrmMain.cs” lines 120-129).
The trigger of a new event takes place in the Wizard form when you are on the last tab
and the user clicks “next” (see 9.1.1.3 -“Wiz1.cs” lines 145-163).
In the event handler in the main form, the processing of the class sent from the Wizard
form takes place (see 9.1.1.2 -“FrmMain.cs” lines 70-119).
The user interface delegates the processing to a background worker class. This leads to a user
interface that is much more accessible (Microsoft, BackgroundWorker Class, 2009).
Firstly, the background worker class is instantiated and supplied with the event handler
references functions (see 9.1.1.2 -“FrmMain.cs”, lines 29-44).
Then the runworkerasync function executes and then shows the working form. (See 9.1.1.2 -
“FrmMain.cs”, lines 266-281).
Page 55 of 225
William Sayers 05025397
Therefore, when the C++ implementation of genetic algorithms that I wrote functioned,
but had issues with memory leaks and was slow, I decided to try re-implementing it in C#.
Once this implementation was complete, it was both faster than the C++
implementation (due mostly to my use of build in .NET sort methods in the sorting part of the
genetic algorithm), and didn’t suffer from the memory leaks (being managed code) so that’s
the version that exists in the application today (see 9.1.2.5 -
“Algorithms\GeneticAlgorithm.cs” for the C# implementation)
The FANN library implements both cascade correlation and back propagation (Nissen,
FANN) which is a C programming language based neural network library.
The compilation of the FANN library is static, and the Dll linked to this to allow access
to the functionality. There is an unmanaged C++ wrapper (9.1.4.3 -
Page 56 of 225
William Sayers 05025397
C# accesses the C++/CLI class directly, which allows it to manipulate the FANN
functions in the static library through this class.
In the constructor (lines 41 – 77), the separation of the textual inputs takes place, via a
number of designated separator characters, into arrays of strings.
From there, these arrays form parameters for the other functions, which using the data
from the heading (number of epochs of data, number of inputs, number of outputs) separates
the data into two-dimensional arrays suitable to be passed to other functions.
Three jagged arrays are utilised, which store respectively, the inputs, the weights, and
the outputs. The inputs and weights are structures so that the coordinates for an input, will
also be the coordinates for its corresponding weight (lines 17-19). In the initialisation routines
(lines 28-124), the initialisation of all the arrays to their correct sizes takes place and then the
initialisation of the weights to a random double between zero and one.
Page 57 of 225
William Sayers 05025397
The main functionality of the neural network class can be found in the “internal” region
(lines 126-235), there are three functions, to run the output layer, input layer, and hidden
layer which are all called by the public “run()” function (lines 241-270). All nodes determine
their outputs using the sigmoid function (lines 130-133).
I tested the neural network implementation by creating a neural network with structure
2, 3, 1, and creating a neural network with a similar structure on an excel spreadsheet. Then
entered the weights from the neural network program (by means of using the visual studio
debugger to check variable values) into the spreadsheet and checked that the outputs
coincided.
5 - Testing Data
Testing took place several times for each algorithm and dataset (see 9.4 -Datasets) (ten
times for XOR, five times for Virus classification and Fishers iris data). As the program
presented the data in each case, recording took place for analysis.
The genetic algorithm and back-propagation neural networks each had four hidden
nodes for each dataset. The target error was 0.01 for XOR and 0.02 for Fishers iris data and
the virus classification data.
5.1 - XOR
In the XOR dataset, the dataset is too small to split, so repetition is instead used.
Page 58 of 225
William Sayers 05025397
5.1.2 - Back-propagation
In the back-propagation testing data, the mean squared error from the network result
and the testing mean squared error are different despite the data being the same, because of
floating point inaccuracies converting between double precision floating-point numbers in my
code and single precision in the FANN code.
Page 59 of 225
William Sayers 05025397
In the cascade correlation data, the testing result differs quite widely from the network
result. At each node addition, the call back function triggers, which is when the program
records the mean squared error, but in the case of cascade training, the network is trained
more after that point (output weight training).
2 0.008039565 1.43545E-13
Table 4 - Cascade Correlation testing data (XOR)
Page 60 of 225
William Sayers 05025397
5.1.4 - Graphs
1000
900
800
700
600
Iterations (GA)
500
Iterations (BP)
400
Nodes (CC)
300
200
100
0
0 5 10 15
0.012
0.01
0.008
0.002
0
0 5 10 15
Page 61 of 225
William Sayers 05025397
0.012
0.01
0.008
0.002
0
0 5 10 15
Page 62 of 225
William Sayers 05025397
5.2.2 - Back-propagation
Iterations
Mean squared error (BP) Testing mean squared error (BP)
(BP)
434 0.0199999 0.014144
529 0.0199719 0.0140669
474 0.0199892 0.0143778
391 0.0199916 0.0139982
527 0.0199972 0.0143806
2 0.01950238 0.0160925
Table 7 - Cascade correlation testing data (Iris data)
Page 63 of 225
William Sayers 05025397
5.2.4 - Graphs
10000
9000
8000
7000
6000
Iterations (GA)
5000
Iterations (BP)
4000
Nodes (CC)
3000
2000
1000
0
0 2 4 6
0.0202
0.02
0.0198
0.0186
0 2 4 6
Figure 33- Mean squared error upon last training test (Iris data)
Page 64 of 225
William Sayers 05025397
0.025
0.02
Testing mean
0.015 squared error (GA)
Testing mean
0.01 squared error (BP)
Testing mean
0.005 squared error (CC)
0
0 2 4 6
• Hordeviruses x3
• Tobraviruses x6
• Tobamoviruses x39
• Furoviruses x13
Page 65 of 225
William Sayers 05025397
5.3.2 - Back-propagation
Iterations (BP) Mean Squared Error training (BP) Mean Squared Error testing (BP)
3275 0.0197935 0.0391179
10922 0.0196959 0.0434423
2800 0.019743 0.0393106
3744 0.0199621 0.0350318
6295 0.0199916 0.0347797
Page 66 of 225
William Sayers 05025397
5.3.4 - Graphs
14000
12000
10000
2000
0
0 2 4 6
0.025
0.02
0
0 2 4 6
Page 67 of 225
William Sayers 05025397
0.05
0.045
0.04
0.035
Mean Squared Error
0.03 testing (GA)
0.025 Mean Squared Error
0.02 testing (BP)
0.015 Mean Squared Error
0.01 testing (CC)
0.005
0
0 2 4 6
6 - Comparisons
6.1 - Back-propagation and Genetic Algorithms
6.1.1 - XOR
As regards to the XOR problem, genetic algorithms and back-propagation offer similar
performance in solving the problem. The back-propagation implementation seems to be
slightly faster in real-time but this is possibly attributable to managed code inefficiencies
versus unmanaged code, or possible inefficiencies in my programming.
Page 68 of 225
William Sayers 05025397
Genetic algorithms clearly found this a difficult problem to solve as opposed to back-
propagation. With an average of 3318 generations for genetic algorithms versus 471 epochs
for back-propagation this became clear. The back-propagation generated networks were also
more accurate overall (in fact on this particular problem analysed the testing data more
effectively than the training data).
The solution of this problem was more effective by genetic algorithms than by back-
propagation, with an average of 3375.8 generations versus 5407.2 epochs for back-
propagation.
The genetic algorithm solutions also held their effectiveness through to the testing data
with extreme consistency, whereas the back-propagation solutions lost effectiveness on the
training data.
This problem was solved more effectively by the Cascade algorithm than by genetic
algorithms, genetic algorithms taking 550 generations on average to solve the problem to a
suitable accuracy, whilst the cascade algorithm used 2 nodes to solve the problems. At a
maximum of 150 epochs per node added (the default (Nissen, FANN)) this is around 300
epochs to solve the problem. The cascade algorithm also achieves a higher degree of
accuracy.
Again, on this data set cascade training came to a satisfactory solution faster and more
accurately than genetic algorithms.
Page 69 of 225
William Sayers 05025397
With this data set cascade training again came to a satisfactory solution faster and more
accurately than genetic algorithms did on the same data.
7 - Evaluation
Overall, I feel the project has been a success, although there are further investigations I
would like to continue with given more time (and may continue with in my own time).
Page 70 of 225
William Sayers 05025397
Build an application that is capable of training a neural network with a genetic algorithm and with a back-
propagation system, and that allows manual adjustment of relevant variables.
Build a user-interface for the aforementioned application allowing for easy alteration of appropriate
variables.
Using above application, as well as previous research, evaluate training methods for neural networks.
Identify the most effective ways of training neural networks using genetic algorithms, and in which cases
these methods are suitable.
The only objective that I feel could be improved upon is the final objective.
“Identify the most effective ways of training neural networks using genetic algorithms,
and in which cases these methods are suitable.”
With more time more advanced tests could be undertaken using larger data sets, leading
to a more solid conclusion and more information on the various strengths and weaknesses of
the various learning algorithms examined.
Page 71 of 225
William Sayers 05025397
8 - Possible Improvements
More advanced testing would be the biggest improvement that more time would make
possible. Some forms of testing that would be possible are:
Page 72 of 225
William Sayers 05025397
Page 73 of 225
William Sayers 05025397
9 - Appendices
9.1 - Source code
The source code in this section is organised first by which part of the application it
represents and then by what source file it originally resided in. The code contains line
numbering for pinpointing the precise lines in question within the body-text when cross-
referencing.
Although there was another section of the program compiled from source (the FANN
library), I have not included it in the source code listing, so that this listing contains only code
I have personally written for the project.
This section also contains only the final draft of the code. The earlier attempts at
genetic algorithm and neural network implementations are not included, and neither is the
code for the prototype constructed in milestone two, which this code is partially based upon.
Page 74 of 225
William Sayers 05025397
9.1.1 - View
9.1.1.1 - “NetworkUpdatedEventHandler”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.Linq;
00000004 using System.Text;
00000005
00000006 namespace _05025397.Controller.Classes
00000007 {
00000008 public class NetworkUpdateEventArgs : System.EventArgs
00000009 {
00000010 #region data
00000011 /********************\
00000012 |* DATA *|
00000013 \********************/
00000014
00000015 //Training Algorithm Identifier
00000016 private int trainingalg;
00000017
00000018 //Dataset identifier
00000019 private int _TrDataSet;
00000020
00000021 //Cascade Correlation
00000022 private double _CC_Terr;
00000023 private int _CC_MaxNeurons;
00000024 private int _CC_Reports;
00000025 private double _CC_LearningRate;
00000026 private bool _CC_ITerr;
Page 75 of 225
William Sayers 05025397
00000027
00000028 //Genetic Algorithm
00000029 private int _GA_PopSize;
00000030 private int _GA_GenLimit;
00000031 private int _GA_Crossover;
00000032 private int _GA_Mutation;
00000033 private double _GA_Terr;
00000034 private int _GA_Reports;
00000035 private int _GA_HiddenL;
00000036 private bool _GA_ITerr;
00000037
00000038 //BackProp
00000039 private int _BP_EpochLimit;
00000040 private int _BP_HiddenL;
00000041 private int _BP_Reports;
00000042 private double _BP_Terr;
00000043 private double _BP_LearningRate;
00000044 private bool _BP_ITerr;
00000045 #endregion
00000046
00000047 #region constructor
00000048 /********************\
00000049 |* CONSTRUCTOR *|
00000050 \********************/
00000051 //Constructor to accept all necessary data
00000052 public NetworkUpdateEventArgs
00000053 (int Algorithm,
00000054 double CCTerr, int CCMaxNeurons,
00000055 int CCReports, double CCLearningRate,
00000056 bool CCITerr, int GAPopSize,
Page 76 of 225
William Sayers 05025397
Page 77 of 225
William Sayers 05025397
Page 78 of 225
William Sayers 05025397
Page 79 of 225
William Sayers 05025397
00000147 #endregion
00000148
00000149 #region external
00000150 //General
00000151 public int TrDataSet
00000152 { get { return _TrDataSet; } }
00000153 public int TrainingAlgorithm
00000154 { get { return trainingalg; } }
00000155 #endregion
00000156 }
00000157 }
9.1.1.2 - “FrmMain.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.ComponentModel;
00000004 using System.Data;
00000005 using System.Drawing;
00000006 using System.Linq;
00000007 using System.Text;
00000008 using System.IO;
00000009 using System.Windows.Forms;
00000010
00000011 using ZedGraph;
00000012 using _05025397;
00000013
00000014
00000015 namespace _05025397.Controller
00000016 {
00000017 public partial class FrmMain : Form
Page 80 of 225
William Sayers 05025397
00000018 {
00000019 #region data
00000020 /********************\
00000021 |* DATA *|
00000022 \********************/
00000023 //The neuralnetwork
00000024 Controller.Algorithms.TrainingAlgorithm network;
00000025
00000026 //Working dialog
00000027 Controller.Working workingdialog = new Controller.Working();
00000028
00000029 //Background thread
00000030 static BackgroundWorker bw = new BackgroundWorker();
00000031 #endregion
00000032
00000033 #region constructor
00000034 /********************\
00000035 |* CONSTRUCTOR *|
00000036 \********************/
00000037 public FrmMain()
00000038 {
00000039 InitializeComponent();
00000040 this.Icon = Properties.Resources.icon;
00000041 bw.DoWork += new DoWorkEventHandler(bw_DoWork);
00000042 bw.RunWorkerCompleted += new RunWorkerCompletedEventHandler(bw_RunWorkerCompleted);
00000043 }
00000044 #endregion
00000045
00000046 #region internal_functions
00000047 /********************\
Page 81 of 225
William Sayers 05025397
00000048 |* INTERNAL *|
00000049 \********************/
00000050 private void plotgraphsMSE(PointPairList MSE)
00000051 {
00000052 this.zedGraphControl1.GraphPane.CurveList.Clear();
00000053
00000054 GraphPane MSEpane = this.zedGraphControl1.GraphPane;
00000055
00000056 //Titles
00000057 MSEpane.Title.Text = "Mean Squared Error";
00000058 MSEpane.YAxis.Title.Text = "MSE";
00000059 MSEpane.XAxis.Title.Text = "Iteration";
00000060
00000061 // Hide the legend
00000062 MSEpane.Legend.IsVisible = false;
00000063
00000064 MSEpane.AddCurve("", MSE, Color.Red, SymbolType.Circle);
00000065
00000066 this.zedGraphControl1.AxisChange();
00000067
00000068 }
00000069
00000070 private void updatenetwork(object sender, Classes.NetworkUpdateEventArgs e)
00000071 {
00000072 LoadDataset(e.TrDataSet);
00000073
00000074 try
00000075 {
00000076 if (e.TrainingAlgorithm == 0)
00000077 {
Page 82 of 225
William Sayers 05025397
00000078 network =
00000079 new Controller.Algorithms.GeneticAlgorithm
00000080 (e.GA_PopSize, e.GA_Crossover,
00000081 e.GA_Mutation, e.GA_HiddenL,
00000082 e.GA_ITerr, e.GA_Terr,
00000083 e.GA_Reports, e.GA_GenLimit,
00000084 txtTrainData, txtTestData);
00000085 }
00000086
00000087 if (e.TrainingAlgorithm == 1)
00000088 {
00000089 network =
00000090 new Controller.Algorithms.BackProp
00000091 (e.BP_HiddenL, e.BP_LearningRate,
00000092 e.BP_ITerr, e.BP_Terr,
00000093 e.BP_Reports, e.BP_EpochLimit,
00000094 txtTrainData, txtTestData);
00000095 }
00000096
00000097 if (e.TrainingAlgorithm == 2)
00000098 {
00000099 network =
00000100 new Controller.Algorithms.CascadeCorrelation
00000101 (e.CC_LearningRate, e.CC_Terr,
00000102 e.CC_Reports, e.CC_MaxNeurons, e.CC_ITerr,
00000103 txtTrainData, txtTestData);
00000104 }
00000105 }
00000106 catch
00000107 {
Page 83 of 225
William Sayers 05025397
Page 84 of 225
William Sayers 05025397
00000138 break;
00000139 case (int)Dataset.FISHER:
00000140 txtTrainData.Text = Properties.Resources.FISHERtrain;
00000141 txtTestData.Text = Properties.Resources.FISHERtest;
00000142 break;
00000143 case (int)Dataset.CUSTOM:
00000144 LoadCustomDataSet();
00000145 break;
00000146 }
00000147 }
00000148
00000149 private void LoadCustomDataSet()
00000150 {
00000151
00000152 string trainpath = SelectTextFile(".\\", "Select Training Data");
00000153 string testpath = SelectTextFile(".\\", "Select Testing Data");
00000154
00000155 string train = "";
00000156 string test = "";
00000157
00000158 try
00000159 {
00000160 train = LoadTxtFile(trainpath);
00000161 }
00000162 catch
00000163 {
00000164 MessageBox.Show("Error: Problem loading training data",
00000165 "Error!", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000166 }
00000167
Page 85 of 225
William Sayers 05025397
00000168 try
00000169 {
00000170 test = LoadTxtFile(testpath);
00000171 }
00000172 catch
00000173 {
00000174 MessageBox.Show("Error: Problem loading testing data",
00000175 "Error!", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000176 }
00000177
00000178 txtTestData.Text = test;
00000179 txtTrainData.Text = train;
00000180 }
00000181
00000182 private string LoadTxtFile(string path)
00000183 {
00000184 string data = "";
00000185 FileStream File = null;
00000186 StreamReader Reader = null;
00000187
00000188 try
00000189 {
00000190 File = new FileStream(path, FileMode.Open, FileAccess.Read);
00000191 Reader = new StreamReader(File);
00000192 data = Reader.ReadToEnd();
00000193 }
00000194 catch
00000195 {
00000196 MessageBox.Show("Error reading selected file: " + path,
00000197 "Error!", MessageBoxButtons.OK, MessageBoxIcon.Error);
Page 86 of 225
William Sayers 05025397
Page 87 of 225
William Sayers 05025397
Page 88 of 225
William Sayers 05025397
00000258 /********************\
00000259 |* BUTTONS *|
00000260 \********************/
00000261 private void btnNew_Click(object sender, EventArgs e)
00000262 {
00000263 new_network();
00000264 }
00000265
00000266 private void btnTrain_Click(object sender, EventArgs e)
00000267 {
00000268 //Shouldn't be busy if the user managed to click this
00000269 //but just make 100% certain.
00000270 if (!bw.IsBusy)
00000271 {
00000272 if (network != null)
00000273 {
00000274 if (!network.Trained)
00000275 {
00000276 bw.RunWorkerAsync();
00000277 workingdialog.ShowDialog();
00000278 }
00000279 }
00000280 }
00000281 }
00000282
00000283 private void btnTest_Click(object sender, EventArgs e)
00000284 {
00000285 if (network != null)
00000286 {
00000287 if (network.Trained)
Page 89 of 225
William Sayers 05025397
00000288 {
00000289 if (!network.Tested)
00000290 {
00000291 bool success = network.Test();
00000292 if (success)
00000293 {
00000294 txtOutput.Clear();
00000295 txtOutput.Text = network.ReportData;
00000296 txtOutput.SelectionStart = txtOutput.Text.Length;
00000297 txtOutput.ScrollToCaret();
00000298 txtOutput.Refresh();
00000299 }
00000300 }
00000301 }
00000302 }
00000303 }
00000304 #endregion
00000305
00000306 #region bg_worker
00000307 /********************\
00000308 |* BG WORKER *|
00000309 \********************/
00000310 private void bw_DoWork(object sender, DoWorkEventArgs e)
00000311 {
00000312 bool success = true;
00000313
00000314 if (network != null)
00000315 {
00000316 success = network.Train();
00000317 }
Page 90 of 225
William Sayers 05025397
00000318
00000319 e.Result = success;
00000320 }
00000321
00000322 private void bw_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
00000323 {
00000324 bool success;
00000325 //Check for errors
00000326 if (e.Error != null)
00000327 {
00000328 MessageBox.Show("Error during network train: " + e.Error.ToString(),
00000329 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000330
00000331 }
00000332
00000333 success = (bool) e.Result;
00000334
00000335 if (success)
00000336 {
00000337 txtOutput.Clear();
00000338 txtOutput.Text = network.ReportData;
00000339 txtOutput.SelectionStart = txtOutput.Text.Length;
00000340 txtOutput.ScrollToCaret();
00000341 plotgraphsMSE(network.getGraphData());
00000342 }
00000343 else
00000344 {
00000345 MessageBox.Show("Error training network!", "Error training network!",
00000346 MessageBoxButtons.OK, MessageBoxIcon.Error);
00000347 }
Page 91 of 225
William Sayers 05025397
00000348
00000349 //Remove the dialog locking out the main form
00000350 //and showing that we're currently working
00000351 workingdialog.Hide();
00000352 }
00000353 #endregion
00000354
00000355 #region misc
00000356 /********************\
00000357 |* MISC *|
00000358 \********************/
00000359 private void FrmMain_Shown(object sender, EventArgs e)
00000360 {
00000361 //Run the new neural net wizard
00000362 new_network();
00000363 }
00000364 #endregion
00000365 }
00000366 }
9.1.1.3 - “Wiz1.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.ComponentModel;
00000004 using System.Data;
00000005 using System.Drawing;
00000006 using System.Linq;
Page 92 of 225
William Sayers 05025397
Page 93 of 225
William Sayers 05025397
Page 94 of 225
William Sayers 05025397
00000067 {
00000068 if (rBtnBackprop.Checked == true)
00000069 {
00000070 picAlg.Image = _05025397.Properties.Resources.IMAGEBackProp;
00000071 lblGA.Visible = false;
00000072 lblBP.Visible = true;
00000073 lblCC.Visible = false;
00000074
00000075 panBackprop.Visible = true;
00000076 panGeneticAlgorithm.Visible = false;
00000077 panCC.Visible = false;
00000078
00000079 algorithm = (int)Training.BP;
00000080 }
00000081 }
00000082
00000083 private void rBtnCascadeCorrelation_CheckedChanged(object sender, EventArgs e)
00000084 {
00000085 if (rBtnCascadeCorrelation.Checked == true)
00000086 {
00000087 picAlg.Image = _05025397.Properties.Resources.IMAGECascadeCorrelation;
00000088 lblGA.Visible = false;
00000089 lblBP.Visible = false;
00000090 lblCC.Visible = true;
00000091
00000092 panBackprop.Visible = false;
00000093 panGeneticAlgorithm.Visible = false;
00000094 panCC.Visible = true;
00000095
00000096 algorithm = (int)Training.CC;
Page 95 of 225
William Sayers 05025397
00000097 }
00000098 }
00000099
00000100 private void rBtnGeneticAlgorithm_CheckedChanged(object sender, EventArgs e)
00000101 {
00000102 if (rBtnGeneticAlgorithm.Checked == true)
00000103 {
00000104 picAlg.Image = _05025397.Properties.Resources.IMAGEGeneticAlgorithm;
00000105 lblGA.Visible = true;
00000106 lblBP.Visible = false;
00000107 lblCC.Visible = false;
00000108
00000109 panBackprop.Visible = false;
00000110 panGeneticAlgorithm.Visible = true;
00000111 panCC.Visible = false;
00000112
00000113 algorithm = (int)Training.GA;
00000114 }
00000115 }
00000116
00000117 private void btnBack_Click(object sender, EventArgs e)
00000118 {
00000119 if (tabControl1.SelectedIndex > 0)
00000120 {
00000121 tabControl1.SelectTab(tabControl1.SelectedIndex - 1);
00000122 }
00000123 }
00000124
00000125 private void btnNext_Click(object sender, EventArgs e)
00000126 {
Page 96 of 225
William Sayers 05025397
Page 97 of 225
William Sayers 05025397
Page 98 of 225
William Sayers 05025397
00000187 {
00000188 if (chkDatasetXOR.Checked)
00000189 {
00000190 picDataset.Image = _05025397.Properties.Resources.IMAGEXOR;
00000191 Trainset = (int)Dataset.XOR;
00000192 }
00000193 }
00000194
00000195 private void chkDatasetCustom_CheckedChanged(object sender, EventArgs e)
00000196 {
00000197 if (chkDatasetCustom.Checked)
00000198 {
00000199 picDataset.Image = _05025397.Properties.Resources.IMAGEQuestionMark;
00000200 Trainset = (int)Dataset.CUSTOM;
00000201 }
00000202 }
00000203 #endregion
00000204
00000205 #region misc
00000206 /********************\
00000207 |* MISC *|
00000208 \********************/
00000209 private void tabControl1_SelectedIndexChanged(object sender, EventArgs e)
00000210 {
00000211 if (tabControl1.SelectedIndex == 1)
00000212 {
00000213 if (panCC.Visible == true)
00000214 {
00000215 txtCC_TargetError.Focus();
00000216 }
Page 99 of 225
William Sayers 05025397
9.1.1.4 - “FrmAbout.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.ComponentModel;
00000004 using System.Data;
00000005 using System.Drawing;
00000006 using System.Linq;
00000007 using System.Text;
00000008 using System.Windows.Forms;
00000009
00000010 namespace _05025397.Controller
00000011 {
00000012 public partial class FrmAbout : Form
00000013 {
00000014 #region constructor
00000015 /********************\
00000016 |* CONSTRUCTOR *|
00000017 \********************/
00000018 public FrmAbout()
00000019 {
00000020 InitializeComponent();
00000021 }
00000022 #endregion
00000023
00000024 #region form
00000025 /********************\
00000026 |* FORM *|
00000027 \********************/
00000028 private void FrmAbout_Load(object sender, EventArgs e)
00000029 {
00000060 {
00000061 try
00000062 {
00000063 System.Diagnostics.Process.Start
00000064 (linkLabel4.Text);
00000065 }
00000066 catch
00000067 {
00000068 MessageBox.Show("Error opening link.",
00000069 "Error", MessageBoxButtons.OK,
00000070 MessageBoxIcon.Error);
00000071 }
00000072 }
00000073
00000074 private void linkLabel5_LinkClicked(object sender, LinkLabelLinkClickedEventArgs e)
00000075 {
00000076 try
00000077 {
00000078 System.Diagnostics.Process.Start
00000079 (linkLabel5.Text);
00000080 }
00000081 catch
00000082 {
00000083 MessageBox.Show("Error opening link.",
00000084 "Error", MessageBoxButtons.OK,
00000085 MessageBoxIcon.Error);
00000086 }
00000087 }
00000088
00000089 private void linkLabel6_LinkClicked(object sender, LinkLabelLinkClickedEventArgs e)
00000090 {
00000091 try
00000092 {
00000093 System.Diagnostics.Process.Start
00000094 (linkLabel6.Text);
00000095 }
00000096 catch
00000097 {
00000098 MessageBox.Show("Error opening link.",
00000099 "Error", MessageBoxButtons.OK,
00000100 MessageBoxIcon.Error);
00000101 }
00000102 }
00000103
00000104 private void linkLabel2_LinkClicked(object sender, LinkLabelLinkClickedEventArgs e)
00000105 {
00000106 try
00000107 {
00000108 System.Diagnostics.Process.Start
00000109 ("http://www.snikks.co.uk");
00000110 }
00000111 catch
00000112 {
00000113 MessageBox.Show("Error opening link.",
00000114 "Error", MessageBoxButtons.OK,
00000115 MessageBoxIcon.Error);
00000116 }
00000117 }
00000118
00000119 private void linkLabel7_LinkClicked(object sender, LinkLabelLinkClickedEventArgs e)
00000120 {
00000121 try
00000122 {
00000123 System.Diagnostics.Process.Start
00000124 ("http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html");
00000125 }
00000126 catch
00000127 {
00000128 MessageBox.Show("Error opening link.",
00000129 "Error", MessageBoxButtons.OK,
00000130 MessageBoxIcon.Error);
00000131 }
00000132 }
00000133
00000134 #endregion
00000135
00000136 #region buttons
00000137 /********************\
00000138 |* BUTTONS *|
00000139 \********************/
00000140 private void btnDatasets_Click(object sender, EventArgs e)
00000141 {
00000142 if (tabControl1.Visible != true)
00000143 {
00000144 groupBox1.Visible = false;
00000145 groupBox2.Visible = false;
00000146 tabControl1.Visible = true;
00000147 pictureBox1.Visible = false;
00000148 }
00000149 else
00000150 {
00000151 groupBox1.Visible = true;
00000152 groupBox2.Visible = false;
00000153 tabControl1.Visible = false;
00000154 pictureBox1.Visible = true;
00000155 }
00000156 }
00000157
00000158 private void btnLicences_Click(object sender, EventArgs e)
00000159 {
00000160 if (groupBox2.Visible != true)
00000161 {
00000162 groupBox2.Visible = true;
00000163 groupBox1.Visible = false;
00000164 tabControl1.Visible = false;
00000165 pictureBox1.Visible = false;
00000166 }
00000167 else
00000168 {
00000169 groupBox1.Visible = true;
00000170 groupBox2.Visible = false;
00000171 tabControl1.Visible = false;
00000172 pictureBox1.Visible = true;
00000173 }
00000174 }
00000175
00000176 private void btnOkay_Click(object sender, EventArgs e)
00000177 {
00000178 this.Dispose();
00000179 }
00000180 #endregion
00000181
00000182 #region radiobuttons
00000183 /********************\
00000184 |* RADIOBUTTONS *|
00000185 \********************/
00000186 private void radioButton1_CheckedChanged(object sender, EventArgs e)
00000187 {
00000188 if (radioButton1.Checked == true)
00000189 textBox5.Text = Properties.Resources.LICENCE_LGPL21;
00000190 }
00000191
00000192 private void radioButton2_CheckedChanged(object sender, EventArgs e)
00000193 {
00000194 if (radioButton2.Checked == true)
00000195 textBox5.Text = Properties.Resources.LICENCE_LGPL30;
00000196 }
00000197
00000198 private void radioButton3_CheckedChanged(object sender, EventArgs e)
00000199 {
00000200 if (radioButton3.Checked == true)
00000201 textBox5.Text = Properties.Resources.LICENCE_GPL30;
00000202 }
00000203 #endregion
00000204 }
00000205 }
9.1.1.5 - “Working.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.ComponentModel;
00000004 using System.Data;
00000005 using System.Drawing;
00000006 using System.Linq;
00000007 using System.Text;
00000008 using System.Windows.Forms;
00000009
00000010 namespace _05025397.Controller
00000011 {
00000012 public partial class Working : Form
00000013 {
00000014 #region functions
00000015 /********************\
00000016 |* FUNCTIONS *|
00000017 \********************/
00000018 public Working()
00000019 {
00000020 InitializeComponent();
00000021 this.Icon = Properties.Resources.icon;
00000022 }
00000023
00000024 private void Working_Click(object sender, EventArgs e)
00000025 {
00000026 System.Media.SystemSounds.Beep.Play();
00000027 }
00000028 #endregion
00000029 }
00000030 }
9.1.2 - Controller
9.1.2.1 - “VerifyData.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.Linq;
00000004 using System.Text;
00000005 using System.Windows.Forms;
00000006
00000007 namespace _05025397.Controller
00000008 {
00000009 public class VerifyData
00000010 {
00000011 #region data
00000012 /********************\
00000013 |* DATA *|
00000014 \********************/
00000015 //Dataverified variable
00000016 private bool verified;
00000017
00000018 //Training Algorithm Identifier
00000019 //1 = genetic algorith
00000020 //2 = backprop
00000021 //3 = cascade correlation
00000022 private int trainingalg;
00000023
00000024 //Dataset identifer
00000025 //0 = XOR
00000056
00000057 #region constructor
00000058 /********************\
00000059 |* CONSTRUCTOR *|
00000060 \********************/
00000061 public VerifyData
00000062 (int Algorithm,
00000063 string CCTerr, string CCMaxNeurons,
00000064 string CCReports, string CCLearningRate,
00000065 bool CCITerr, string GAPopSize,
00000066 string GAGenLimit, string GACrossOver,
00000067 string GAMutation, string GATerr,
00000068 string GAReports, string GAHiddenL,
00000069 bool GAITerr, string BPEpochLimit,
00000070 string BPHiddenL, string BPReports,
00000071 string BPTerr, string BPLearningRate,
00000072 bool BPITerr, int TData_Set)
00000073 {
00000074 //Initially set to true, will be changed to
00000075 //false if any data verification errors occur
00000076 verified = true;
00000077
00000078 //Algorithm
00000079 trainingalg = Algorithm;
00000080
00000081 //DataSet
00000082 _TDataSet = TData_Set;
00000083
00000084 if (_TDataSet !=
00000085 (int)Dataset.XOR && _TDataSet !=
00000116 if (verified)
00000117 {
00000118 if (trainingalg == 0)
00000119 {
00000120 try
00000121 {
00000122 //Genetic algorithm
00000123 _GA_PopSize = Convert.ToInt32(GAPopSize);
00000124 _GA_GenLimit = Convert.ToInt32(GAGenLimit);
00000125 _GA_Crossover = Convert.ToInt32(GACrossOver);
00000126 _GA_Mutation = Convert.ToInt32(GAMutation);
00000127 _GA_Terr = Convert.ToSingle(GATerr);
00000128 _GA_Reports = Convert.ToInt32(GAReports);
00000129 _GA_HiddenL = Convert.ToInt32(GAHiddenL);
00000130 _GA_ITerr = GAITerr;
00000131 }
00000132 catch (FormatException)
00000133 {
00000134 MessageBox.Show
00000135 ("Parsing Error: Please check supplied"
00000136 +" genetic algorithm data values",
00000137 "Parsing Error",
00000138 MessageBoxButtons.OK,
00000139 MessageBoxIcon.Error);
00000140 verified = false;
00000141 }
00000142 }
00000143 }
00000144
00000145 if (verified)
00000146 {
00000147 if (trainingalg == 1)
00000148 {
00000149 try
00000150 {
00000151 //Back-propagation
00000152 _BP_EpochLimit = Convert.ToInt32(BPEpochLimit);
00000153 _BP_HiddenL = Convert.ToInt32(BPHiddenL);
00000154 _BP_Reports = Convert.ToInt32(BPReports);
00000155 _BP_Terr = Convert.ToDouble(BPTerr);
00000156 _BP_LearningRate = Convert.ToDouble(BPLearningRate);
00000157 _BP_ITerr = BPITerr;
00000158 }
00000159 catch (FormatException)
00000160 {
00000161 MessageBox.Show
00000162 ("Parsing Error: Please check " +
00000163 "supplied back propagation data values",
00000164 "Parsing Error",
00000165 MessageBoxButtons.OK,
00000166 MessageBoxIcon.Error);
00000167 verified = false;
00000168 }
00000169 }
00000170 }
00000171
00000172 if (verified)
00000173 {
00000174 verified = checkforsanity();
00000175 }
00000176 }
00000177
00000178 #endregion
00000179
00000180 #region internal
00000181 /********************\
00000182 |* INTERNAL *|
00000183 \********************/
00000184 private bool checkGAsanity()
00000185 {
00000186 bool sanity = true;
00000187
00000188 if (_GA_PopSize < 3)
00000189 sanity = false;
00000190
00000191 if (_GA_GenLimit < 1)
00000192 sanity = false;
00000193
00000194 if (_GA_Crossover < 0 || _GA_Crossover > 100)
00000195 {
00000196 sanity = false;
00000197 }
00000198
00000199 if (_GA_Mutation < 0 || _GA_Mutation > 100)
00000200 {
00000201 sanity = false;
00000202 }
00000203
00000204 if (_GA_Terr < 0 || _GA_Terr > 1)
00000205 sanity = false;
00000206
00000207 if (_GA_Reports < 1 || _GA_Reports > _GA_GenLimit)
00000208 sanity = false;
00000209
00000210 if (_GA_HiddenL < 1)
00000211 sanity = false;
00000212
00000213 if (!sanity)
00000214 MessageBox.Show
00000215 ("Sanity Check Fail: Please check supplied " +
00000216 "genetic algorithm data values",
00000217 "Sanity Check Fail",
00000218 MessageBoxButtons.OK,
00000219 MessageBoxIcon.Error);
00000220
00000221 return sanity;
00000222 }
00000223
00000224 private bool checkBPsanity()
00000225 {
00000226 bool sanity = true;
00000227
00000228 if (_BP_EpochLimit < 1)
00000229 sanity = false;
00000230
00000231 if (_BP_HiddenL < 1)
00000232 sanity = false;
00000233
00000234 if (_BP_Reports < 1 || _BP_Reports > _BP_EpochLimit)
00000235 sanity = false;
00000236
00000237 if (_BP_Terr < 0 || _BP_Terr > 1)
00000238 sanity = false;
00000239
00000240 if (_BP_LearningRate < 0 || _BP_LearningRate > 1)
00000241 sanity = false;
00000242
00000243 if (!sanity)
00000244 MessageBox.Show
00000245 ("Sanity Check Fail: Please check supplied " +
00000246 "back propagation data values",
00000247 "Sanity Check Fail",
00000248 MessageBoxButtons.OK,
00000249 MessageBoxIcon.Error);
00000250
00000251 return sanity;
00000252 }
00000253
00000254 private bool checkCCsanity()
00000255 {
00000256 bool sanity = true;
00000257
00000258 if (_CC_Terr < 0 || _CC_Terr > 1)
00000259 sanity = false;
00000260
00000261 if (_CC_MaxNeurons < 1)
00000262 sanity = false;
00000263
00000264 if (_CC_Reports < 1 || _CC_Reports > _CC_MaxNeurons)
00000265 sanity = false;
00000266
00000267 if (_CC_LearningRate < 0 || _CC_LearningRate > 1)
00000268 sanity = false;
00000269
00000270 if (!sanity)
00000271 MessageBox.Show
00000272 ("Sanity Check Fail: Please check supplied " +
00000273 "cascade correlation data values",
00000274 "Sanity Check Fail",
00000275 MessageBoxButtons.OK,
00000276 MessageBoxIcon.Error);
00000277
00000278 return sanity;
00000279 }
00000280
00000281 private bool checkforsanity()
00000282 {
00000283 //Some of the values that are allowed through here would not produce
00000284 //good results - however I don't want to prevent any flexibility, only
00000285 //halt any values from proceeding that are so wrong they could cause
00000286 //serious errors later down the line.
00000287
00000288 // - Basically we're making sure the data is clean before it goes
00000289 // - anywhere near a DLL.
00000290 // - We're also doing some very minor processing of the probability
00000291 // - values, if they're greater than 1 but less than 100 scale them
00000292 // - appropriately.
00000293 bool sanity = true;
00000294
00000295 if (trainingalg < 0 || trainingalg > 2)
00000296 {
00000297 sanity = false;
00000298 MessageBox.Show
00000299 ("Sanity Check Fail: Training Algorithm not " +
00000300 "selected or selected incorrectly",
00000301 "Sanity Check Fail",
00000302 MessageBoxButtons.OK,
00000303 MessageBoxIcon.Error);
00000304 }
00000305
00000306 if (sanity)
00000307 {
00000308 if (trainingalg == 0)
00000309 {
00000310 sanity = checkGAsanity();
00000311 }
00000312
00000313 if (trainingalg == 1)
00000314 {
00000315 sanity = checkBPsanity();
00000316 }
00000317
00000318 if (trainingalg == 2)
00000319 {
00000320 sanity = checkCCsanity();
00000321 }
00000322 }
00000323
00000324 return sanity;
00000325 }
00000326 #endregion
00000327
00000328 #region getters/setters
00000329 /********************\
00000330 |* EXTERNAL *|
00000331 \********************/
00000332
00000333 #region cascade_correlation
00000334 //Cascade Correlation
00000335 public double CC_Terr
00000336 { get { return _CC_Terr; } }
00000337 public int CC_MaxNeurons
00000338 { get { return _CC_MaxNeurons; } }
00000339 public int CC_Reports
00000340 { get { return _CC_Reports; } }
00000341 public double CC_LearningRate
00000342 { get { return _CC_LearningRate; } }
00000343 public bool CC_ITerr
00000344 { get { return _CC_ITerr; } }
00000345 #endregion
00000346
00000347 #region genetic_algorithm
00000348 //Genetic Algorithm
00000349 public int GA_PopSize
00000350 { get { return _GA_PopSize; } }
00000351 public int GA_GenLimit
00000352 { get { return _GA_GenLimit; } }
00000353 public int GA_Crossover
00000354 { get { return _GA_Crossover; } }
00000355 public int GA_Mutation
9.1.2.2 - “DatasetParser.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.Linq;
00000004 using System.Text;
00000005 using System.Windows.Forms;
00000006 using System.Collections;
00000007
00000008 namespace _05025397.Controller
00000009 {
00000010 public class DatasetParser
00000011 {
00000012 #region data
00000013 /********************\
00000014 |* DATA *|
00000015 \********************/
00000016 private bool _Verified;
00000017
00000018 private ArrayList trinputs;
00000019 private ArrayList troutputs;
00000020
00000021 private ArrayList teinputs;
00000022 private ArrayList teoutputs;
00000023
00000024 private ArrayList aldata;
00000025 private ArrayList altest;
00000026
00000027 private string[] data;
00000028 private string[] test;
00000029
00000060
00000061 //Split at the chars specified above
00000062 data = txtData.Text.Split(delimiterChars);
00000063 test = txtTest.Text.Split(delimiterChars);
00000064
00000065 //Get the number of iterations, number of inputs
00000066 //and number of outputs
00000067 Niterations = System.Convert.ToDouble(data[0]);
00000068 Ninputs = System.Convert.ToDouble(data[1]);
00000069 Noutputs = System.Convert.ToDouble(data[2]);
00000070
00000071 //parse the data into appropriately
00000072 //structured arrays
00000073 Verified = parseTrainingData();
00000074 Verified = parseTestingData();
00000075
00000076 _Verified = Verified;
00000077 }
00000078 #endregion
00000079
00000080 #region internal_functions
00000081 /********************\
00000082 |* INTERNAL *|
00000083 \********************/
00000084 private bool parseTrainingData()
00000085 {
00000086 bool success = true;
00000087 try
00000088 {
00000089 for (int i = 3; i < data.Length; i++)
00000090 {
00000091 aldata.Add(System.Convert.ToDouble(data[i]));
00000092 }
00000093
00000094 while (aldata.Count > 0)
00000095 {
00000096 for (int i = 0; i < Ninputs; i++)
00000097 {
00000098 trinputs.Add(aldata[0]);
00000099 aldata.RemoveAt(0);
00000100 }
00000101 for (int j = 0; j < Noutputs; j++)
00000102 {
00000103 troutputs.Add(aldata[0]);
00000104 aldata.RemoveAt(0);
00000105 }
00000106 }
00000107 }
00000108 catch
00000109 {
00000110 MessageBox.Show("Error parsing training data",
00000111 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000112 success = false;
00000113 }
00000114 return success;
00000115 }
00000116
00000117 private bool parseTestingData()
00000118 {
00000119 bool success = true;
00000120
00000121 try
00000122 {
00000123 for (int i = 3; i < data.Length; i++)
00000124 {
00000125 altest.Add(System.Convert.ToDouble(data[i]));
00000126 }
00000127
00000128 while (altest.Count > 0)
00000129 {
00000130 for (int i = 0; i < Ninputs; i++)
00000131 {
00000132 teinputs.Add(altest[0]);
00000133 altest.RemoveAt(0);
00000134 }
00000135 for (int j = 0; j < Noutputs; j++)
00000136 {
00000137 teoutputs.Add(altest[0]);
00000138 altest.RemoveAt(0);
00000139 }
00000140 }
00000141 }
00000142 catch
00000143 {
00000144 MessageBox.Show("Error parsing testing data",
00000145 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000146 success = false;
00000147 }
00000148 return success;
00000149 }
00000150 #endregion
00000151
00000152 #region external_functions
00000153 /********************\
00000154 |* EXTERNAL *|
00000155 \********************/
00000156 public void GetStructure(out double iter,
00000157 out double Ninp, out double Nout)
00000158 {
00000159 Ninp = Ninputs;
00000160 iter = Niterations;
00000161 Nout = Noutputs;
00000162 }
00000163
00000164 public void GetTrainingData
00000165 (out double[][] trinputdata, out double[][] troutputdata)
00000166 {
00000167 trinputdata = new double[trinputs.Count / (int) Ninputs][];
00000168 troutputdata = new double[troutputs.Count / (int) Noutputs][];
00000169
00000170 for (int i = 0; i < (trinputs.Count / (int)Ninputs); i++)
00000171 trinputdata[i] = new double[(int) Ninputs];
00000172
00000173 for (int i = 0; i < (troutputs.Count / (int)Noutputs); i++)
00000174 troutputdata[i] = new double[(int) Noutputs];
00000175
00000176 for (int j = 0, l = 0; j < trinputs.Count; j+=(int) Ninputs, l++)
00000177 {
00000178 for (int k = 0; k < Ninputs; k++)
00000179 {
00000210 }
00000211 }
00000212
00000213 for (int j = 0, l = 0; j < teoutputs.Count; j += (int)Noutputs, l++)
00000214 {
00000215 for (int k = 0; k < Noutputs; k++)
00000216 {
00000217 teoutputdata[l][k] = (double)teoutputs[j + k];
00000218 }
00000219 }
00000220 }
00000221
00000222 public void GetTestingDataAL
00000223 (out ArrayList teinputdata, out ArrayList teoutputdata)
00000224 {
00000225 teinputdata = teinputs;
00000226 teoutputdata = teoutputs;
00000227 }
00000228
00000229 public bool Verified
00000230 {
00000231 get { return _Verified; }
00000232 }
00000233
00000234 #endregion
00000235 }
00000236 }
9.1.2.3 - “Algorithms\TrainingAlgorithm.cs”
00000001 using System;
00000062
00000063 #region internal
00000064 /********************\
00000065 |* INTERNAL *|
00000066 \********************/
00000067 protected void Report(string output)
00000068 {
00000069 _ReportData += output;
00000070 }
00000071 #endregion
00000072
00000073 #region external
00000074 /********************\
00000075 |* EXTERNAL *|
00000076 \********************/
00000077 public abstract bool Train();
00000078 public abstract bool Test();
00000079 public abstract string network_details();
00000080
00000081 public PointPairList getGraphData()
00000082 {
00000083 return BestErrList;
00000084 }
00000085
00000086 public double TargetError
00000087 {
00000088 set { _TargetError = value; }
00000089 get { return _TargetError; }
00000090 }
00000091
9.1.2.4 - “Algorithms\CascadeCorrelation.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.Linq;
00000004 using System.Text;
00000005 using System.Windows.Forms;
00000006 using System.Collections;
00000007 using System.IO;
00000008 using ZedGraph;
00000009 using _05025397;
00000010 using FANN_Wrapper;
00000011
00000012
00000013 namespace _05025397.Controller.Algorithms
00000014 {
00000015 public class CascadeCorrelation : TrainingAlgorithm
00000016 {
00000017 #region data
00000018 /********************\
00000019 |* DATA *|
00000020 \********************/
00000021 private double _InputL;
00000022 private double _OutputL;
00000023 private double _LearnRate;
00000024
00000025 FANN_Cascade CCNet;
00000026
00000027 #endregion
00000028
00000029 #region constructor
00000030 /********************\
00000031 |* CONSTRUCTOR *|
00000032 \********************/
00000033 public CascadeCorrelation(double LearnRate,
00000034 double TargetError, int Reports,
00000035 int Iter_Max, bool ITerr,
00000036 RichTextBox txtData, RichTextBox txtTest)
00000037 : base(TargetError, Reports, Iter_Max,
00000038 txtData, txtTest, ITerr)
00000039 {
00000040 _LearnRate = LearnRate;
00000041
00000042 double iterations;
00000043 getdata.GetStructure(out iterations, out _InputL, out _OutputL);
00000044
00000045 CCNet =
00000046 new FANN_Cascade((int)_InputL, (int)_OutputL,
00000047 _LearnRate, base.TargetError, base.ReportInterval,
00000048 base.IterMax);
00000049
00000050 saveDllData(txtData, txtTest);
00000051 }
00000052 #endregion
00000053
00000054 #region internal_functions
00000055 /********************\
00000056 |* INTERNAL *|
00000057 \********************/
00000058 private void saveDllData(RichTextBox tr, RichTextBox te)
00000059 {
00000090 string.Format
00000091 ("\t\t\tTarget Error: {0:0.00000}",
00000092 base.TargetError));
00000093 output.Append(
00000094 string.Format
00000095 ("\t\tMaximum Nodes: {0:d}",
00000096 base.IterMax));
00000097 output.Append(
00000098 string.Format
00000099 ("\nReport every {0:d} generations.",
00000100 base.ReportInterval));
00000101 output.Append(
00000102 string.Format
00000103 ("\t\t\t\t\t\tIgnore target error " +
00000104 "(process all iterations): {0}",
00000105 _ITerr));
00000106
00000107 return output.ToString();
00000108 }
00000109
00000110 public override bool Train()
00000111 {
00000112 bool success = true;
00000113 try
00000114 {
00000115 success = CCNet.Train();
00000116 Report(CCNet._tr_output);
00000117 }
00000118 catch
00000119 {
00000150
00000151 return success;
00000152 }
00000153
00000154 public override bool Test()
00000155 {
00000156 bool success = true;
00000157 try
00000158 {
00000159 success = CCNet.Test();
00000160 Report(CCNet._te_output);
00000161 }
00000162 catch
00000163 {
00000164 MessageBox.Show("Error running Cascade testing!",
00000165 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000166 success = false;
00000167 }
00000168 base._Tested = success;
00000169 return success;
00000170 }
00000171 #endregion
00000172 }
00000173 }
9.1.2.5 - “Algorithms\GeneticAlgorithm.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.Linq;
00000004 using System.Text;
00000005 using System.Windows.Forms;
00000006 using System.Collections;
00000007 using ZedGraph;
00000008 using _05025397;
00000009
00000010 namespace _05025397.Controller.Algorithms
00000011 {
00000012 public class GeneticAlgorithm : TrainingAlgorithm
00000013 {
00000014
00000015 #region data
00000016 /********************\
00000017 |* DATA *|
00000018 \********************/
00000019 //GA Data
00000020 private int _PopSize;
00000021 private int _Crossover;
00000022 private int _Mutation;
00000023 private int _HiddenL;
00000024
00000025 //Network Data
00000026 private double _InputL;
00000027 private double _OutputL;
00000028
00000029 //Population
00000090
00000091 //Report if we're at a report iteration
00000092 //Also update the besterrlist (for graphing)
00000093 if (base.CurIt == ReportInterval)
00000094 {
00000095 base.BestErrList.Add
00000096 ((double)i, ((Model.GANetwork)Population[0]).MSqE);
00000097
00000098 base.Report("Best error at iteration [ "
00000099 + i + " ] was " + ((Model.GANetwork)Population[0]).MSqE + ".\n");
00000100
00000101 base.CurIt = 0;
00000102 }
00000103 base.CurIt++;
00000104
00000105 //Check for having reached the target
00000106 if (((Model.GANetwork)Population[0]).MSqE <= base.TargetError)
00000107 {
00000108 Report("\nNetwork matching or improving upon target error"+
00000109 "found at iteration [ " + i + " ].");
00000110 break;
00000111 }
00000112
00000113 success = build_generation();
00000114 }
00000115 return success;
00000116 }
00000117
00000118 private bool sort_fitnesses()
00000119 {
00000210 catch
00000211 {
00000212 MessageBox.Show("Error building new generation",
00000213 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000214 success = false;
00000215 }
00000216 }
00000217
00000218 return success;
00000219 }
00000220 #endregion
00000221
00000222 #region external_functions
00000223 /********************\
00000224 |* EXTERNAL *|
00000225 \********************/
00000226 public override string network_details()
00000227 {
00000228 //Add stringbuilder for efficiency here
00000229 StringBuilder output = new StringBuilder();
00000230 output.Append(
00000231 string.Format("Network Details: \n Population: {0:d}",
00000232 _PopSize));
00000233 output.Append(
00000234 string.Format("\t\t\tCrossover Probability: {0:d}",
00000235 _Crossover));
00000236 output.Append(
00000237 string.Format("\t\t\tMutation Probability: {0:d}",
00000238 _Mutation));
00000239 output.Append(
00000270 {
00000271 bool success = true;
00000272
00000273 StringBuilder output = new StringBuilder();
00000274
00000275 double[][] tstinput;
00000276 double[][] tstoutput;
00000277 double[] netoutput;
00000278
00000279 getdata.GetScaledTestingData(out tstinput, out tstoutput);
00000280
00000281 output.Append("\n\n\t\t\t~~Network Testing~~");
00000282
00000283 try
00000284 {
00000285 for (int i = 0; i < tstinput.Length; i++)
00000286 {
00000287 ((Model.GANetwork)Population[0]).Run(tstinput[i]);
00000288
00000289 output.Append("\nWith inputs:");
00000290 for (int j = 0; j < tstinput[i].Length; j++)
00000291 {
00000292 output.Append(string.Format(" [{0:g}] ", tstinput[i][j]));
00000293 }
00000294
00000295 output.Append("\nOutput achieved was:");
00000296 netoutput = ((Model.GANetwork)Population[0]).GetOutput();
00000297 for (int j = 0; j < netoutput.Length; j++)
00000298 {
00000299 output.Append(string.Format(" [{0:g}] ", netoutput[j]));
00000300 }
00000301
00000302 output.Append("\nOutput Desired was:");
00000303 for (int j = 0; j < tstoutput[i].Length; j++)
00000304 {
00000305 output.Append(string.Format(" [{0:g}] ", tstoutput[i][j]));
00000306 }
00000307 }
00000308 output.Append("\nMean Squared Error with these inputs and outputs is:");
00000309 ((Model.GANetwork)Population[0]).getMsQE(tstinput,tstoutput);
00000310 output.Append(
00000311 string.Format
00000312 ("{0:g}", (double) ((Model.GANetwork)Population[0]).MSqE));
00000313
00000314 base.Report(output.ToString());
00000315 }
00000316 catch
00000317 {
00000318 MessageBox.Show("Error Running Test.",
00000319 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000320 success = false;
00000321 }
00000322 base._Tested = success;
00000323 return success;
00000324 }
00000325 #endregion
00000326
00000327 }
00000328 }
9.1.2.6 - “Algorithms\BackProp.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.Linq;
00000004 using System.Text;
00000005 using System.Windows.Forms;
00000006 using System.Collections;
00000007 using System.IO;
00000008 using ZedGraph;
00000009 using _05025397;
00000010 using FANN_Wrapper;
00000011
00000012 namespace _05025397.Controller.Algorithms
00000013 {
00000014 public class BackProp : TrainingAlgorithm
00000015 {
00000016 #region data
00000017 /********************\
00000018 |* DATA *|
00000019 \********************/
00000020 private double _InputL;
00000021 private double _OutputL;
00000022 private int _HiddenL;
00000023 private double _LearnRate;
00000024
00000025 FANN_BackProp BPnet;
00000026
00000027 #endregion
00000028
00000029 #region constructor
00000030 /********************\
00000031 |* CONSTRUCTOR *|
00000032 \********************/
00000033 public BackProp
00000034 (int HiddenL, double LearnRate, bool ITerr,
00000035 double TargetError, int Reports,
00000036 int Iter_Max, RichTextBox txtData, RichTextBox txtTest)
00000037 : base(TargetError, Reports, Iter_Max,
00000038 txtData, txtTest, ITerr)
00000039 {
00000040 _HiddenL = HiddenL;
00000041 _LearnRate = LearnRate;
00000042
00000043 double iterations;
00000044 getdata.GetStructure(out iterations, out _InputL, out _OutputL);
00000045
00000046 BPnet =
00000047 new FANN_BackProp((int)_InputL, HiddenL, (int)_OutputL,
00000048 _LearnRate, base.TargetError, base.ReportInterval,
00000049 base.IterMax);
00000050
00000051 //Copy the training and testing data to the dat files
00000052 //ready for the DLL's to access.
00000053
00000054 saveDllData(txtData, txtTest);
00000055 }
00000056 #endregion
00000057
00000058 #region internal_functions
00000059 /********************\
00000060 |* INTERNAL *|
00000061 \********************/
00000062 private void saveDllData(RichTextBox tr, RichTextBox te)
00000063 {
00000064 TextWriter tw = new StreamWriter("dlltrdata.dat");
00000065 tw.Write(tr.Text);
00000066 tw.Close();
00000067
00000068 tw = new StreamWriter("dlltedata.dat");
00000069 tw.Write(te.Text);
00000070 tw.Close();
00000071 }
00000072 #endregion
00000073
00000074 #region external_functions
00000075 /********************\
00000076 |* EXTERNAL *|
00000077 \********************/
00000078 public override string network_details()
00000079 {
00000080 StringBuilder output = new StringBuilder();
00000081 output.Append(
00000082 string.Format
00000083 ("Network Details: \n Input Layer: {0:d}",
00000084 (int) _InputL));
00000085 output.Append(
00000086 string.Format
00000087 ("\t\t\tOutput Layer: {0:d}",
00000088 (int) _OutputL));
00000089 output.Append(
00000090 string.Format
00000091 ("\t\t\tHidden Layer: {0:d}",
00000092 (int) _HiddenL));
00000093 output.Append(
00000094 string.Format
00000095 ("\nLearning Rate: {0:0.00000}",
00000096 _LearnRate));
00000097 output.Append(
00000098 string.Format
00000099 ("\t\t\tTarget Error: {0:0.00000}",
00000100 base.TargetError));
00000101 output.Append(
00000102 string.Format
00000103 ("\t\tMaximum Epoch: {0:d}",
00000104 base.IterMax));
00000105 output.Append(
00000106 string.Format
00000107 ("\nReport every {0:d} generations.",
00000108 base.ReportInterval));
00000109 output.Append(
00000110 string.Format
00000111 ("\t\t\t\t\t\tIgnore target error " +
00000112 "(process all iterations): {0}",
00000113 _ITerr));
00000114
00000115 return output.ToString();
00000116 }
00000117
00000118 public override bool Train()
00000119 {
00000150 }
00000151 }
00000152 catch
00000153 {
00000154 MessageBox.Show("Error getting graph data from DLL",
00000155 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000156 success = false;
00000157 }
00000158
00000159 return success;
00000160 }
00000161
00000162 public override bool Test()
00000163 {
00000164 bool success = true;
00000165 try
00000166 {
00000167 success = BPnet.Test();
00000168 Report(BPnet._te_output);
00000169 }
00000170 catch
00000171 {
00000172 MessageBox.Show("Error running Backprop testing!",
00000173 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000174 success = false;
00000175 }
00000176 base._Tested = success;
00000177 return success;
00000178 }
00000179 #endregion
00000180 }
00000181 }
9.1.3 - Model
9.1.3.1 - “Network.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.Linq;
00000004 using System.Text;
00000005 using System.Windows.Forms;
00000006 using _05025397;
00000007
00000008 namespace _05025397.Model
00000009 {
00000010 public class Network
00000011 {
00000012 #region data
00000013 /********************\
00000014 |* DATA *|
00000015 \********************/
00000016 //[layer][node][input]
00000017 protected double[][][] _inputs;
00000018 protected double[][][] _weights;
00000019 protected double[][] _outputs;
00000020
00000021 protected int _numinputs;
00000022 protected int _hiddennodes;
00000023 protected int _numoutputs;
00000024
00000055 }
00000056 }
00000057
00000058 private void init_outputlayer()
00000059 {
00000060 _inputs[2] = new double[_numoutputs][];
00000061 _weights[2] = new double[_numoutputs][];
00000062 _outputs[2] = new double[_numoutputs];
00000063
00000064 for (int i = 0; i < _numoutputs; i++)
00000065 {
00000066 _inputs[2][i] = new double[_hiddennodes + 1];
00000067 _weights[2][i] = new double[_hiddennodes + 1];
00000068 }
00000069 }
00000070
00000071 private void init_bias()
00000072 {
00000073 //Input bias
00000074 for (int i = 0; i < _numinputs; i++)
00000075 {
00000076 _inputs[0][i][1] = 1.0;
00000077 _weights[0][i][1] = 1.0;
00000078 }
00000079
00000080 //Hidden layer bias
00000081 for (int i = 0; i < _hiddennodes; i++)
00000082 {
00000083 _inputs[1][i][_numinputs] = 1.0;
00000084 _weights[1][i][_numinputs] = 1.0;
00000085 }
00000086
00000087 //Output layer bias
00000088 for (int i = 0; i < _numoutputs; i++)
00000089 {
00000090 _inputs[2][i][_hiddennodes] = 1.0;
00000091 _weights[2][i][_hiddennodes] = 1.0;
00000092 }
00000093 }
00000094
00000095 private void init_inputweights()
00000096 {
00000097 for (int i = 0; i < _numinputs; i++)
00000098 {
00000099 _weights[0][i][0] = Rand.NextDouble();
00000100 }
00000101 }
00000102
00000103 private void init_hiddenweights()
00000104 {
00000105 for (int i = 0; i < _hiddennodes; i++)
00000106 {
00000107 for (int j = 0; j < _numinputs; j++)
00000108 {
00000109 _weights[1][i][j] = Rand.NextDouble();
00000110 }
00000111 }
00000112 }
00000113
00000114 private void init_outputweights()
00000115 {
00000116 for (int i = 0; i < _numoutputs; i++)
00000117 {
00000118 for (int j = 0; j < _hiddennodes; j++)
00000119 {
00000120 _weights[2][i][j] = Rand.NextDouble();
00000121 }
00000122 }
00000123 }
00000124 #endregion
00000125
00000126 #region internal
00000127 /********************\
00000128 |* INTERNAL *|
00000129 \********************/
00000130 private double sigmoid(double input)
00000131 {
00000132 return (1 / (1 + Math.Pow(Math.E, -input)));
00000133 }
00000134
00000135 private bool Runinput()
00000136 {
00000137 bool success = true;
00000138 double sum = 0.0;
00000139
00000140 try
00000141 {
00000142 //Calculate results for this layer
00000143 for (int i = 0; i < _numinputs; i++)
00000144 {
00000175 }
00000176 }
00000177
00000178 //Calculate results for this layer
00000179 for (int i = 0; i < _hiddennodes; i++)
00000180 {
00000181 for (int j = 0; j < _numinputs + 1; j++)
00000182 {
00000183 sum += (_inputs[1][i][j] * _weights[1][i][j]);
00000184 }
00000185 _outputs[1][i] = sigmoid(sum);
00000186 sum = 0.0;
00000187 }
00000188 }
00000189 catch
00000190 {
00000191 MessageBox.Show("Error processing hidden layer",
00000192 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000193 success = false;
00000194 }
00000195
00000196 return success;
00000197 }
00000198
00000199 private bool Runoutput()
00000200 {
00000201 bool success = true;
00000202 double sum = 0.0;
00000203
00000204 try
00000205 {
00000206 //Feed forward the results from hidden layer
00000207 for (int i = 0; i < _numoutputs; i++)
00000208 {
00000209 for (int j = 0; j < _hiddennodes; j++)
00000210 {
00000211 _inputs[2][i][j] = _outputs[1][j];
00000212 }
00000213 }
00000214
00000215 //Calculate results for this layer
00000216 for (int i = 0; i < _numoutputs; i++)
00000217 {
00000218 for (int j = 0; j < _hiddennodes + 1; j++)
00000219 {
00000220 sum += (_inputs[2][i][j] * _weights[2][i][j]);
00000221 }
00000222 _outputs[2][i] = sigmoid(sum);
00000223 sum = 0.0;
00000224 }
00000225 }
00000226 catch
00000227 {
00000228 MessageBox.Show("Error processing output layer",
00000229 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000230 success = false;
00000231 }
00000232
00000233 return success;
00000234 }
00000235 #endregion
00000236
00000237 #region external
00000238 /********************\
00000239 |* EXTERNAL *|
00000240 \********************/
00000241 public bool Run(double[] inputs)
00000242 {
00000243 bool success = true;
00000244
00000245 //The numbers of inputs must match up
00000246 if (inputs.Length != _numinputs)
00000247 {
00000248 MessageBox.Show("Error: Incorrect number of inputs supplied to NN",
00000249 "Incorrect number of inputs", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000250 success = false;
00000251 }
00000252
00000253 if (success)
00000254 for (int i = 0; i < _numinputs; i++)
00000255 //Each input node has only one real input
00000256 //and a bias - the number of nodes corresponds
00000257 //to the number of inputs accepted.
00000258 _inputs[0][i][0] = inputs[i];
00000259
00000260 if(success)
00000261 success = Runinput();
00000262
00000263 if(success)
00000264 success = Runhidden();
00000265
00000266 if(success)
00000267 success = Runoutput();
00000268
00000269 return success;
00000270 }
00000271
00000272 public double[] GetOutput()
00000273 {
00000274 //Return the outputs from the
00000275 //output layer
00000276 return _outputs[2];
00000277 }
00000278 #endregion
00000279
00000280 #region constructor
00000281 /********************\
00000282 |* CONSTRUCTOR *|
00000283 \********************/
00000284 public Network(int numinputs, int hiddennodes, int numoutputs, int SetRandom)
00000285 {
00000286 //Set random number generator
00000287 Rand = new Random(SetRandom);
00000288
00000289 //Set network structure descriptors
00000290 _numinputs = numinputs;
00000291 _hiddennodes = hiddennodes;
00000292 _numoutputs = numoutputs;
00000293
00000294 //We'll always have 3 layers
9.1.3.2 - “GANetwork.cs”
00000001 using System;
00000002 using System.Collections.Generic;
00000003 using System.Linq;
00000004 using System.Text;
00000005 using System.Windows.Forms;
00000006 using _05025397;
00000007 using System.Collections;
00000008
00000009 //This class contains Genetic Algorithm specific functions
00000010 //in addition to the basic feed foward neural net functionality.
00000011
00000012 namespace _05025397.Model
00000013 {
00000014 public class GANetwork : Network, IComparable
00000015 {
00000016 #region data
00000017 /********************\
00000018 |* DATA *|
00000019 \********************/
00000020 double _MSqE;
00000021 #endregion
00000022
00000023 #region .NET
00000024 /********************\
00000025 |* .NET *|
00000026 \********************/
00000027 //Interface_Implementation
00000028 //Return Value Meanings:
00000029 //-<Zero: x < y
00000030 //-Zero: x == y
00000031 //->Zero: x > y
00000032 public int CompareTo(object a)
00000033 {
00000034 GANetwork b;
00000035
00000036 if (a is GANetwork)
00000037 {
00000038 b = a as GANetwork;
00000039
00000040 return _MSqE.CompareTo(b.MSqE);
00000041 }
00000042
00000043 return 0;
00000044 }
00000045 #endregion
00000046
00000047 #region external
00000048 /********************\
00000049 |* EXTERNAL *|
00000050 \********************/
00000051 public void getMsQE(double[][] inputs, double[][] outputs)
00000052 {
00000053 double sum = 0.0;
00000054 int counter = 0;
00000055 double[] netoutputs = new double[_numoutputs];
00000056
00000057 for (int i = 0; i < inputs.Length; i++)
00000058 {
00000059 base.Run(inputs[i]);
00000060 netoutputs = base.GetOutput();
00000061
00000062 for (int j = 0; j < netoutputs.Length; j++)
00000063 {
00000064 sum += (outputs[i][j] - netoutputs[j]) *
00000065 (outputs[i][j] - netoutputs[j]);
00000066 counter++;
00000067 }
00000068 }
00000069
00000070 _MSqE = (sum / counter);
00000071 }
00000072
00000103 {
00000104 //Take a 1D array of doubles and apply
00000105 //to the correct positions in our weights
00000106 //array
00000107 try
00000108 {
00000109 for (int i = 0, wc = 0; i < _weights.Length; i++)
00000110 {
00000111 for (int j = 0; j < _weights[i].Length; j++)
00000112 {
00000113 for (int k = 0; k < _weights[i][j].Length; k++, wc++)
00000114 {
00000115 _weights[i][j][k] = weights[wc];
00000116 }
00000117 }
00000118 }
00000119 }
00000120 catch
00000121 {
00000122 MessageBox.Show("Fatal Error adding 1D weight array to 3D weight array",
00000123 "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
00000124 Application.Exit();
00000125 }
00000126 }
00000127
00000128 /********************\
00000129 |* GETTERS/SETTERS *|
00000130 \********************/
00000131 public double MSqE
00000132 {
00000057 {
00000058 int size;
00000059 const double *ar = UMWrapper->get_besterrs(size);
00000060
00000061 array<double>^ arout =
00000062 gcnew array<double>(size);
00000063
00000064
00000065 for(int i = 0; i < size; i++)
00000066 {
00000067 arout[i] = ar[i];
00000068 }
00000069
00000070 return arout;
00000071 }
00000072 }
00000073 };
00000074
00000075 public ref class FANN_BackProp
00000076 //Managed wrapper for UnManaged_FANN_Backprop
00000077 //which is a C++ interface to the C programmed
00000078 //FANN dll.
00000079 {
00000080 public:
00000081 /********************\
00000082 |* DATA *|
00000083 \********************/
00000084 UnManaged_FANN_Backprop* UMWrapper;
00000085
00000086 public:
00000087 /********************\
00000088 |* EXTERNAL *|
00000089 \********************/
00000090 FANN_BackProp(int InputL, int HiddenL, int OutputL,
00000091 double LearnRate, double TargetErr,
00000092 int ReportInterval, int MaximumIteration);
00000093 ~FANN_BackProp(void);
00000094 !FANN_BackProp(void);
00000095
00000096 bool Train(void);
00000097 bool Test(void);
00000098
00000099 public:
00000100 /********************\
00000101 |* GETTERS/SETTERS *|
00000102 \********************/
00000103 property String^ _te_output
00000104 {
00000105 String^ get()
00000106 {
00000107 return gcnew String((wchar_t *) UMWrapper->get_report_test());
00000108 }
00000109 }
00000110
00000111 property String^ _tr_output
00000112 {
00000113 String^ get()
00000114 {
00000115 return gcnew String((wchar_t *) UMWrapper->get_report_train());
00000116 }
00000117 }
00000118
00000119 property array<double>^ besterrorlist
00000120 {
00000121 array<double>^ get()
00000122 {
00000123 int size;
00000124 const double *ar = UMWrapper->get_besterrs(size);
00000125
00000126 array<double>^ arout =
00000127 gcnew array<double>(size);
00000128
00000129
00000130 for(int i = 0; i < size; i++)
00000131 {
00000132 arout[i] = ar[i];
00000133 }
00000134
00000135 return arout;
00000136 }
00000137 }
00000138 };
00000139 }
9.1.4.2 - “FANN_Wrapper.cpp”
00000001 #include "stdafx.h"
00000002 #include "FANN_Wrapper.h"
00000003
00000004 namespace FANN_Wrapper
00000005 {
00000006 /********************\
00000007 |* FANN_BackProp *|
00000008 \********************/
00000009
00000010 //Constructor
00000011 FANN_BackProp::FANN_BackProp(int InputL, int HiddenL, int OutputL,
00000012 double LearnRate, double TargetErr,
00000013 int ReportInterval, int MaximumIteration)
00000014 {
00000015 UMWrapper = new UnManaged_FANN_Backprop(
00000016 InputL, HiddenL, OutputL,
00000017 LearnRate, TargetErr,
00000018 ReportInterval, MaximumIteration);
00000019 }
00000020
00000021 //Destructor
00000022 FANN_BackProp::~FANN_BackProp(void)
00000023 {
00000024 if (UMWrapper)
00000025 {
00000026 delete UMWrapper;
00000027 UMWrapper = 0;
00000028 }
00000029 }
00000030
00000031 //Finalizer
00000032 FANN_BackProp::!FANN_BackProp(void)
00000033 {
00000034 if (UMWrapper)
00000035 {
00000036 delete UMWrapper;
00000037 UMWrapper = 0;
00000038 }
00000039 }
00000040
00000041
00000042 //Train
00000043 bool FANN_BackProp::Train(void)
00000044 {
00000045 return UMWrapper->Train();
00000046 }
00000047
00000048 //Test
00000049 bool FANN_BackProp::Test(void)
00000050 {
00000051 return UMWrapper->Test();
00000052 }
00000053
00000054 /********************\
00000055 |* FANN_Cascade *|
00000056 \********************/
00000057
00000058 //Constructor
00000059 FANN_Cascade::FANN_Cascade(int InputL, int OutputL,
9.1.4.3 - “UnManaged_FANN_BackProp.h”
00000001 #pragma once
00000002 #include <iostream>
00000003 #include <string>
00000004 #include <sstream>
00000005 #include <vector>
00000006 #include <doublefann.h>
00000007 #include <fann_cpp.h>
00000008
00000009 class UnManaged_FANN_Backprop
00000010 //An interface to the C programmed
00000011 //FANN dll.
00000012 {
00000013 private:
00000014 /********************\
00000015 |* DATA *|
00000016 \********************/
00000017 //Network Structure
00000018 int _InputL;
00000019 int _HiddenL;
00000020 int _OutputL;
00000021
00000022 //Training Parameters
00000023 double _LearnRate;
00000024 double _TargetErr;
00000025 int _ReportInterval;
00000026 int _MaximumIteration;
00000027 double _Momentum;
00000028
00000029 //Output for C#
00000060 |* GETTERS/SETTERS *|
00000061 \********************/
00000062 const wchar_t* get_report_train(void);
00000063 const wchar_t* get_report_test(void);
00000064
00000065 const double* get_besterrs(int &size);
00000066 };
9.1.4.4 - “UnManaged_FANN_BackProp.cpp”
00000001 #include "StdAfx.h"
00000002 #include "UnManaged_FANN_BackProp.h"
00000003
00000004 //Constructor
00000005 UnManaged_FANN_Backprop::UnManaged_FANN_Backprop
00000006 (int InputL, int HiddenL, int OutputL,
00000007 double LearnRate, double TargetErr,
00000008 int ReportInterval, int MaximumIteration)
00000009 {
00000010 _InputL = InputL;
00000011 _HiddenL = HiddenL;
00000012 _OutputL = OutputL;
00000013
00000014 _LearnRate = LearnRate;
00000015 _TargetErr = TargetErr;
00000016 _ReportInterval = ReportInterval;
00000017 _MaximumIteration = MaximumIteration;
00000018
00000019 wchoutput = NULL;
00000020 bsterr = NULL;
00000021 data = NULL;
00000022 ann = NULL;
00000023 }
00000024
00000025 //Destructor
00000026 UnManaged_FANN_Backprop::~UnManaged_FANN_Backprop()
00000027 {
00000028
00000029 fann_destroy_train(data);
00000030
00000031 delete bsterr;
00000032 bsterr = NULL;
00000033
00000034 delete wchoutput;
00000035 wchoutput = NULL;
00000036 }
00000037
00000038 //Train
00000039 bool UnManaged_FANN_Backprop::Train(void)
00000040 {
00000041 static int firstrun = false;
00000042
00000043 bool success = true;
00000044 int reportcounter = 0;
00000045 double error;
00000046
00000047 ann = fann_create_standard(3, _InputL, _HiddenL, _OutputL);
00000048
00000049 try
00000050 {
00000051 data = fann_read_train_from_file("dlltrdata.dat");
00000052 }
00000053 catch(...)
00000054 {
00000055 throw("");
00000056 success = false;
00000057 }
00000058
00000059 if (fann_num_input_train_data(data) != _InputL)
00000060 {
00000061 throw("");
00000062 success = false;
00000063 }
00000064
00000065
00000066 if (fann_num_output_train_data(data) != _OutputL)
00000067 {
00000068 throw("");
00000069 success = false;
00000070 }
00000071
00000072 fann_set_activation_steepness_hidden(ann, 1);
00000073 fann_set_activation_steepness_output(ann, 1);
00000074
00000075 //Sigmoid Activation Functions (the same one
00000076 //the GA uses).
00000077 fann_set_activation_function_hidden
00000078 (ann, FANN_SIGMOID);
00000079 fann_set_activation_function_output
00000080 (ann, FANN_SIGMOID);
00000081
00000082 //Standard backprop
00000083 fann_set_training_algorithm(ann, FANN_TRAIN_BATCH);
00000084
00000085 //Set the learning rate
00000086 fann_set_learning_rate(ann, (float) _LearnRate);
00000087
00000088 //Same range the GA's weights are
00000089 //initialised too
00000150 te_output << " [" << calc_out[j] << "] ";
00000151 }
00000152
00000153 te_output << "\nOutput Desired was";
00000154 for (int j = 0; j < _OutputL; j++)
00000155 {
00000156 te_output << " [" << data->output[i][j] << "] ";
00000157 }
00000158 }
00000159
00000160 te_output << "\nMean Squared Error with these inputs and outputs is:";
00000161 te_output << fann_test_data(ann, data);
00000162
00000163 delete error;
00000164 delete calc_out;
00000165
00000166 return true;
00000167 }
00000168
00000169 //get_report
00000170 const wchar_t* UnManaged_FANN_Backprop::get_report_train(void)
00000171 {
00000172 if (wchoutput != NULL)
00000173 {
00000174 delete wchoutput;
00000175 wchoutput = NULL;
00000176 }
00000177
00000178 wchoutput = new wchar_t[tr_output.str().length() + 1];
00000179 wcscpy(wchoutput, tr_output.str().c_str());
00000180
00000181 return wchoutput;
00000182 }
00000183
00000184 const wchar_t* UnManaged_FANN_Backprop::get_report_test(void)
00000185 {
00000186 if (wchoutput != NULL)
00000187 {
00000188 delete wchoutput;
00000189 wchoutput = NULL;
00000190 }
00000191
00000192 wchoutput = new wchar_t[te_output.str().length() + 1];
00000193 wcscpy(wchoutput, te_output.str().c_str());
00000194
00000195 return wchoutput;
00000196 }
00000197
00000198
00000199 //get_besterrs
00000200 const double* UnManaged_FANN_Backprop::get_besterrs(int &size)
00000201 {
00000202 if (bsterr == NULL)
00000203 {
00000204 delete bsterr;
00000205 bsterr = NULL;
00000206 }
00000207
00000208 bsterr = new double[vecbsterr.size()];
00000209 size = vecbsterr.size();
00000210
00000211 for (unsigned int i = 0; i < vecbsterr.size(); i++)
00000212 {
00000213 bsterr[i] = vecbsterr[i];
00000214 }
00000215
00000216 vecbsterr.clear();
00000217
00000218 return bsterr;
00000219 }
9.1.4.5 - “UnManaged_FANN_CasC.h”
00000001 #pragma once
00000002 #include <iostream>
00000003 #include <string>
00000004 #include <sstream>
00000005 #include <vector>
00000006 #include <doublefann.h>
00000007
00000008 class UnManaged_FANN_CasC
00000009 {
00000010 public:
00000011 /********************\
00000012 |* DATA *|
00000013 \********************/
00000014 //Network Structure
00000015 int _InputL;
00000016 int _OutputL;
00000017
00000018 //Training Parameters
00000049
00000050 //Interface functions
00000051 //accessed from C#
00000052 bool Train(void);
00000053 bool Test(void);
00000054
00000055 static int FANN_API Report_Callback
00000056 (struct fann *ann, struct fann_train_data *train,
00000057 unsigned int max_epochs, unsigned int epochs_between_reports,
00000058 float desired_error, unsigned int epochs);
00000059
00000060 public:
00000061 /********************\
00000062 |* GETTERS/SETTERS *|
00000063 \********************/
00000064 const wchar_t* get_report_train(void);
00000065 const wchar_t* get_report_test(void);
00000066
00000067 const double* get_besterrs(int &size);
00000068 };
9.1.4.6 - “UnManaged_FANN_CasC.cpp”
00000001 #include "StdAfx.h"
00000002 #include "UnManaged_FANN_CasC.h"
00000003
00000004 //Static variable declarations
00000005 std::wstringstream UnManaged_FANN_CasC::tr_output;
00000006 std::vector<double> UnManaged_FANN_CasC::vecbsterr;
00000007
00000008 //Constructor
00000009 UnManaged_FANN_CasC::UnManaged_FANN_CasC
00000010 (int InputL, int OutputL,
00000011 double LearnRate, double TargetErr,
00000012 int ReportInterval, int MaxNeurons)
00000013 {
00000014 _InputL = InputL;
00000015 _OutputL = OutputL;
00000016
00000017 _LearnRate = LearnRate;
00000018 _TargetErr = TargetErr;
00000019 _ReportInterval = ReportInterval;
00000020 _MaxNeurons = MaxNeurons;
00000021
00000022 wchoutput = NULL;
00000023 bsterr = NULL;
00000024 ann = NULL;
00000025 data = NULL;
00000026 }
00000027
00000028 //Destructor
00000029 UnManaged_FANN_CasC::~UnManaged_FANN_CasC(void)
00000030 {
00000031 delete bsterr;
00000032 bsterr = NULL;
00000033
00000034 fann_destroy_train(data);
00000035
00000036 delete wchoutput;
00000037 wchoutput = NULL;
00000038 }
00000039
00000040
00000041 //get_report
00000042 const wchar_t* UnManaged_FANN_CasC::get_report_train(void)
00000043 {
00000044 if (wchoutput != NULL)
00000045 {
00000046 delete wchoutput;
00000047 wchoutput = NULL;
00000048 }
00000049
00000050 wchoutput = new wchar_t[tr_output.str().length() + 1];
00000051 wcscpy(wchoutput, tr_output.str().c_str());
00000052 return wchoutput;
00000053 }
00000054
00000055 const wchar_t* UnManaged_FANN_CasC::get_report_test(void)
00000056 {
00000057 if (wchoutput != NULL)
00000058 {
00000059 delete wchoutput;
00000090 //Train
00000091 bool UnManaged_FANN_CasC::Train()
00000092 {
00000093 int reportcounter = 0;
00000094
00000095 ann = fann_create_shortcut(2, _InputL, _OutputL);
00000096
00000097 try
00000098 {
00000099 data = fann_read_train_from_file("dlltrdata.dat");
00000100 }
00000101 catch(...)
00000102 {
00000103 throw("");
00000104 }
00000105
00000106 if (fann_num_input_train_data(data) != _InputL)
00000107 throw("");
00000108
00000109 if (fann_num_output_train_data(data) != _OutputL)
00000110 throw("");
00000111
00000112 fann_set_learning_rate(ann, (float)_LearnRate);
00000113
00000114 //Some more network customisation here
00000115 //might be nice in future.
00000116
00000117 fann_set_quickprop_decay(ann, 0.0);
00000118
00000119 fann_set_quickprop_mu(ann, 2.0);
00000120
00000121 fann_set_cascade_weight_multiplier(ann, 1);
00000122
00000123 fann_set_cascade_max_out_epochs(ann, 150);
00000124
00000125 fann_set_activation_steepness_output(ann, 1);
00000126 fann_set_activation_steepness_hidden(ann, 1);
00000127
00000128 fann_set_learning_rate(ann, (float) _LearnRate);
00000129
00000130 fann_set_training_algorithm(ann, FANN_TRAIN_QUICKPROP);
00000131
00000132 fann_set_activation_function_hidden(ann, FANN_SIGMOID);
00000133
00000134 fann_set_activation_function_output(ann, FANN_SIGMOID);
00000135
00000136 fann_randomize_weights(ann, 0.0, 1.0);
00000137 fann_set_cascade_output_change_fraction(ann, 0.01f);
00000138 fann_set_cascade_candidate_change_fraction(ann, 0.01f);
00000139 fann_set_train_stop_function(ann, FANN_STOPFUNC_MSE);
00000140
00000141 fann_set_callback(ann, UnManaged_FANN_CasC::Report_Callback);
00000142
00000143 fann_cascadetrain_on_data(ann, data, _MaxNeurons, _ReportInterval, (float) _TargetErr);
00000144
00000145 return true;
00000146 }
00000147
00000148 //Callback for reporting
00000149 int FANN_API UnManaged_FANN_CasC::Report_Callback
00000210
00000211 delete error;
00000212 delete calc_out;
00000213
00000214 return true;
00000215 }
-BackProp
Previous Page
-CascadeCorrelation
1 1
FANN_Cascade FANN_BackProp
-UMWrapper -UMWrapper
+FANN_Cascade(in AllDataToFillAttributes) +FANN_BackProp(in AllDataToFillAttributes)
+Train() : bool +Train() : bool
+Test() : bool +Test() : bool
1 1
1
1
UnManaged_FANN_BackProp
-_InputL : int UnManaged_FANN_CasC
-_HiddenL : int -_InputL : int
-_OutputL : int -_OutputL : int
-_LearnRate : double -_LearnRate : double
-_TargetErr : double -_TargetErr : double
-_ReportInterval : int -_ReportInterval : int
-_MaximumIteration : int -_MaxNeurons : int
-_Momentum : double -tr_output
-tr_output -te_output
-te_output -wchoutput
-wchoutput -vecbsterr
-vecbsterr -bsterr
-bsterr 1 -ann
-ann -data
-data +UnManaged_FANN_CasC(in AllDataToAFillAttributes)
+UnManaged_FANN_Backprop(in AddDataToFillAttributes) 1 +~UnManaged_FANN_CasC()
+~UnManaged_FANN_Backprop() +Train() : bool
+Train() : bool +Test() : bool
+Test() : bool +Report_Callback(in Attributes) : int
1 1
FANN_Library
o Matthew Wall
o http://lancet.mit.edu/ga/
o Accessed: 31/10/2008
• Libneural
o D. Franklin
o http://ieee.uow.edu/au/~daniel/software/libneural/
o Accessed: 31/10/2008
• Libann
o Anonymous
o http://www.nongnu.org/libann/index.html
o Accessed: 31/10/2008
• Flood
o Roberto Lopez
o http://www.cimne.com/flood/
o Accessed: 31/10/2008
• Annie
o Asim Shankar
o http://annie.sourceforge.net/
o Accessed: 31/10/2008
o Steffen Nissen
o http://leenissen.dk/fann/index.php
o Accessed: 31/10/2008
9.4 - Datasets
• XOR
1 0 1
1 1
0
1 0
1
0 0
0
9.5 - Objectives
FINAL YEAR DEGREE PROJECT
• Build an application that is capable of training a neural network with a genetic algorithm and with a back-
propagation system, and that allows manual adjustment of relevant variables.
• Build a user-interface for the aforementioned application allowing for easy alteration of appropriate
variables.
• Using above application, as well as previous research, evaluate training methods for neural networks.
• Identify the most effective ways of training neural networks using genetic algorithms, and in which cases
these methods are suitable.
• Deliverable one: A research document describing and detailing the research done for objectives one and
two.
• Deliverable two: An application meeting the requirements of objectives three and four, and a document
describing its implementation and usage.
• Deliverable three: A document detailing and explaining the evaluation of different training techniques, and
the conclusions derived from that evaluation.
10 - Works Cited
Barber, S. (2006). AI: Neural Network for Beginners pt. 1. Retrieved April 13, 2009,
from Codeproject: www.codeproject.com/KB/recipes/NeuralNetwork_1.aspx
Bourg, D. M., & Seeman, G. (2004). Chapter 14: Neural Networks. In D. M. Bourg, &
G. Seeman, AI for Game Developers (pp. 356-358). O'Reilly.
Darwin, C. (1859). On the origin of the species by means of natural selection, or the
preservation of favoured races in the struggle for life. London: John Murray.
Lysle, S. (2007, January 16). Passing Data between Windows Forms. Retrieved April
16, 2009, from C# Corner: http://www.c-
sharpcorner.com/UploadFile/scottlysle/PassData01142007013005AM/PassData.aspx
Marshall, J. A., & Srikanth, V. (1999). Curved trajectory prediction using a self-
organizing neural network. Retrieved April 15, 2009, from Computer Science - UNC-Chapel
Hill: http://www.cs.unc.edu/~marshall/WWW/PAPERS/curve9912.ps.gz
Matthews, J. (2002). Back-propagation for the Uninitiated. Retrieved April 15, 2009,
from generation5: http://www.generation5.org/content/2002/bp.asp
McCulloch, W., & Pitts, W. (1943). A logical calculus of the ideas immanent in
nervous activity. Bulletin of Mathematical Biophysics , 5, 115-133.
Microsoft. (2009). BackgroundWorker Class. Retrieved April 16, 2009, from MSDN:
http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx
Microsoft. (2009). Using C++ Interop (Implicit PInvoke). Retrieved April 16, 2009,
from MSDN: http://msdn.microsoft.com/en-us/library/2x8kf7zx.aspx
Robot Control. (2008). Retrieved April 15, 2009, from Artificial Neural Networks:
http://www.learnartificialneuralnetworks.com/robotcontrol.htm
Stergiou, C., & Dimitrios, S. (1997). Neural Networks. Retrieved April 15, 2009, from
Imperial College London:
http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Contents
Werbos, P. (1974). Beyond Regression: New tools for prediction and analysis in the
behavioral sciences. PhD, Harvard University.
11 - Bibliography
Barber, S. (2006). AI: Neural Network for Beginners pt. 1. Retrieved April 13, 2009,
from Codeproject: www.codeproject.com/KB/recipes/NeuralNetwork_1.aspx
Barber, S. (2006). AI: Neural Network for Beginners pt. 2. Retrieved April 13, 2009,
from Codeproject: http://www.codeproject.com/KB/recipes/Backprop_ANN.aspx
Barber, S. (2006). AI: Neural Network for Beginners pt. 3. Retrieved April 13, 2009,
from Codeproject: http://www.codeproject.com/KB/cs/GA_ANN_XOR.aspx
Bourg, D. M., & Seeman, G. (2004). Chapter 14: Neural Networks. In D. M. Bourg, &
G. Seeman, AI for Game Developers (pp. 356-358). O'Reilly.
Darwin, C. (1859). On the origin of the species by means of natural selection, or the
preservation of favoured races in the struggle for life. London: John Murray.
DiLascia, P. (2002). Call Unmanaged DLLs from C#, Killing Processes Cleanly.
Retrieved April 13, 2009, from MSDN: http://msdn.microsoft.com/en-
us/magazine/cc301501.aspx
DLL Tutorial for Beginners. (2007, August 6). Retrieved April 13, 2009, from
Codeguru: http://www.codeguru.com/Cpp/Cpp/cpp_mfc/tutorials/article.php/c9855/
Lysle, S. (2007, January 16). Passing Data between Windows Forms. Retrieved April
16, 2009, from C# Corner: http://www.c-
sharpcorner.com/UploadFile/scottlysle/PassData01142007013005AM/PassData.aspx
Marshall, J. A., & Srikanth, V. (1999). Curved trajectory prediction using a self-
organizing neural network. Retrieved April 15, 2009, from Computer Science - UNC-Chapel
Hill: http://www.cs.unc.edu/~marshall/WWW/PAPERS/curve9912.ps.gz
Matthews, J. (2002). Back-propagation for the Uninitiated. Retrieved April 15, 2009,
from generation5: http://www.generation5.org/content/2002/bp.asp
McCulloch, W., & Pitts, W. (1943). A logical calculus of the ideas immanent in
nervous activity. Bulletin of Mathematical Biophysics , 5, 115-133.
Microsoft. (2009). BackgroundWorker Class. Retrieved April 16, 2009, from MSDN:
http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx
Microsoft. (2009). Using C++ Interop (Implicit PInvoke). Retrieved April 16, 2009,
from MSDN: http://msdn.microsoft.com/en-us/library/2x8kf7zx.aspx
Nguyen, D., & Widrow, B. (1990). Improving the learning speed of 2-layer neural
networks by choosinginitial values of the adaptive weights. Neural Networks, 1990., 1990
IJCNN International Joint Conference on , 21-26.
Nplot Charting Library for .NET. (n.d.). Retrieved 21 01, 2009, from
http://netcontrols.org/nplot/wiki/
Robot Control. (2008). Retrieved April 15, 2009, from Artificial Neural Networks:
http://www.learnartificialneuralnetworks.com/robotcontrol.htm
Stergiou, C., & Dimitrios, S. (1997). Neural Networks. Retrieved April 15, 2009, from
Imperial College London:
http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Contents
Werbos, P. (1974). Beyond Regression: New tools for prediction and analysis in the
behavioral sciences. PhD, Harvard University.
12 - Tables
12.1 - Table of Figures
Figure 1 - A human neuron ............................................................................................ 10
Figure 6 - The initial state of a cascade correlation neural network. The circular
connections are adjustable weights (Fahlman & Lebiere, The Cascade-Correlation Learning
Architecture, 1991). ................................................................................................................. 35
Figure 7 - The second state of a cascade correlation neural network. A first node has
been added. The square connections are locked weights(Fahlman & Lebiere, The Cascade-
Correlation Learning Architecture, 1991)................................................................................ 35
Figure 8 - The third state of a cascade correlation neural network, with two nodes added
(Fahlman & Lebiere, The Cascade-Correlation Learning Architecture, 1991). ...................... 36
Figure 30 - Mean squared error upon last training test (XOR) ...................................... 61
Figure 33- Mean squared error upon last training test (Iris data) ................................... 64
Figure 34- The mean squared error at testing (Iris data) ................................................ 65
Equation 2 - A sigmoid function(Bourg & Seeman, 2004). In this example “y” is the
function output and “t” is the function input, as defined in the graph above. ......................... 13
Equation 3 – The delta rule(Matthews, Back-propagation for the Uninitiated, 2002) ... 21
Equation 6 - Altered delta calculation rule for the output layer(Matthews, Back-
propagation for the Uninitiated, 2002)..................................................................................... 21
Equation 7 - Altered delta calculation rule for the hidden layer(s) (Matthews, Back-
propagation for the Uninitiated, 2002)..................................................................................... 22
Equation 8 - The definition of 's' given that 'o' is the network output at which the error
is measured, 'p' is the training pattern, 'v' is the candidate unit's value, the quantities ‘`v ’ and
‘`Eo’ are the values of v and Eo averaged over all patterns (Fahlman & Lebiere, The Cascade-
Correlation Learning Architecture, 1991)................................................................................ 33
Equation 9 - The partial derivative of ’s’ (Fahlman & Lebiere, The Cascade-Correlation
Learning Architecture, 1991). .................................................................................................. 34
Equation 10 - The partial derivative of ‘s’ with respect to each of the candidates
incoming weights ‘wi’, where ‘oo’ is the sign of the correlation between the candidates value
output ‘o’, ‘f'p’ is the derivative for pattern 'p' of the candidate unit's activation function with
respect to the sum of it’s inputs and ‘Ii,p’ is the input the candidate receives from unit ‘i’ for
pattern ‘p’. ................................................................................................................................ 34
13 - Diary
• 01/10/2008 – Research progress satisfactory.
o Decision made to adjust the weights of the neural network during
training as opposed to using a genetic algorithm to establish network
structure.
o Decided on XOR as a starting point.
• 10/10/2008
o GALib decided as a genetic algorithm starting point
o Further non-web references as a goal
• 16/10/2008
o Investigation into various neural network libraries taking place.
• 17/10/2008
o Investigating more deeply into journals based on research already
conducted.
o Submitted objectives.
• 24/10/2008
o University computers were down so code demo was impossible;
however, back-propagation network was running and learning XOR.
o Agreed on further investigation into GALib and genetic algorithms.
• 31/10/2008
o First presentation complete and ready
o References building up
o Project plan complete
• 10/11/2008
o Presentation went well
o Progress satisfactory
• 14/11/2008
o Discussed document formatting and layout
o Progress satisfactory.
• 19/11/2008
o Milestone one submitted successfully.
• 21/11/2008
o Progress satisfactory
• 28/11/2008
o Progress satisfactory
• 05/12/2008
o Progress satisfactory
• 12/12/2008
o Progress satisfactory
• 09/01/2009
o Progress satisfactory
• 16/01/2009
o Progress satisfactory
• 23/01/2009
o Progress satisfactory
• 30/01/2009
o Milestone two has been successfully submitted
o Started investigating other areas to explore outside of project
requirements.
• 06/02/2009
o Still exploring extra areas for project to cover.
• 13/02/2009
o Have decided to extend the project by including cascade correlation
learning algorithm and comparison to genetic algorithms.
o Progress satisfactory
• 20/02/2009
o Progress satisfactory
• 27/02/2009
o Progress satisfactory
• 06/03/2009
o Project progress delayed by other coursework
• 13/03/2009
o Project progress delayed by other coursework
• 20/03/2009
o Project progressing, catch-up time for previous two weeks will be used
over Easter break.
o Have started report
• 02/04/2009
o Main problems experienced with genetic algorithm seem related to
memory leaks, have decided to implement in managed language (C#) to
try to solve this
o Report continues satisfactorily
• 04/04/2009
o To do:
About box
Add dataset notes
Decided on virus classification as a third dataset via the custom
dataset functionality.
o Report satisfactory
• 05/04/2009
o Further work on user interface needed
o Finish C# genetic algorithm code (C++ code does work but is very slow)
o Report satisfactory
• 06/04/2009
o C# genetic algorithm code is complete, is much faster (fixed several
bugs along the way) and retains none of the problematic memory leaks.
o Report satisfactory
• 10/04/2009
o Fully functional project – commencing report.
o Report satisfactory.