Neural Network Topology Design through Evolution

Ivan Fuentes
UIN 425002273
jivfur@tamu.edu
April 2019

1 Introduction
Neural Networks appeared in the scene in 1943 with Warren McCulloch and Walter Pitts proposing a network
to simulate how to brain works, however, due to the computational power that this model required, they, the
neural networks, stayed in the cooler for a time. Currently, after over passing the computational problem
with modern supercomputers, Neural Networks appeared again stronger than ever. An artificial neural
network is a model inspired by the biological model of the brain; the synaptic dogma [1], is the dogma
of how we think the brain works, this is, the neurons, brain cells, connect to other neurons through their
dendrites and share electrical pulses to share “information” or knowledge, and that is how the brain learns.
An artificial
P neural, or perceptron, is a mathematical function, which takes a linear combination of inputs,
X = xi Wi , to obtain a predicted output, F (X). The perceptron has shown to be powerful enough for
some tasks, besides, as a colony of ants is capable of doing more than a single ant, a set of perceptrons can
achieve more than a single one. A drawback that I personally find in Neural Networks is the topology design,
What I meant for it is, tuning the hyperparameters of them, like number of layers, number of neuron in each
layer, number of epochs for training, batch size, momentum, among others. So far it looks like that neural
network experts’ task is to release network topologies for applications, like AlexNet that is used in ImageNet
Large Scale Visual Recognition Challenge (ILSVRC), or AlphaGo, a topology designed by Google researches
to play Go [3]. The Synaptics of the brain changes all the time, making the brain to adapt to new challenges
or to evolve with the environment. What if we can use evolution to find a neural network that best solves a
problem. In this project, I trained a Genetic Algorithm to evolve a chromosome of hyperparameters [4, 5]to
create neural networks for specific data sets.

2 Data Sets
I will use databases available in keras 2.2.4, like IMDB Movie reviews sentiment classification, Reuters
newswire topics classification, MNIST database of handwritten digits, Fashion-MNIST database of fashion
articles, Boston housing price regression dataset, CIFAR10 small image classification
Another Data set besides the one provided by keras will be: Parkinson’s Disease Classification Data Set.
The data used in this study were gathered from 188 patients with PD (107 men and 81 women) with ages
ranging from 33 to 87. The task is a classification task. [2]

3 Model
In this project I used a steady stated Genetic Algorithm. In a steady state GA, the concept of generations
is not clear defined, because instead of creating a whole new population in each time, the new offspring
replaces the worst individuals (the worst performance) in the population, in that sense, the population
remains constant during the whole evolutionary process.

1
 Figure 1: Chromosome Definition

The rule in machine learning is, the model is as good as the fitness function. The validation accuracy
is used as fitness function. I tried to use the accuracy as a fitness function but I decided that validation
accuracy might provide more of the sense of the test, which the validation set is a unseen set for the model.
The chromosome includes any combination of the following neural network hyperparameters: Activation
Function, Epochs, Hidden Layers, Neurons, Learning Rate, Batch Size, Momentum, Dropout. The user can
select any combination of the hyperparamenters to evolve, allowing to set any default value in any of them.
The size of the population is also defined by the user as long as the mutation probability which must be
small. In figure 1 there is a example of how the population and chromosome are defined.
The selection process is done using the roulette algorithm. The roulette crossover, as the name suggests,
there is a roulette with all the individuals to be selected, however the probability of being selected is related
to its individual fitness, in this way the best individuals have higher chance to send their genes into the next
generations while the bad ones, have a small chance to reproduce, but if the odds are in their favor, the bad
ones could be selected too for crossover.
The crossover operator is a sexual reproduction, this is, it is necessary to have two individuals to create
two new ones. In this version, there are a random number of crossover points between the second and one
previous of the last genes.
The mutation operator is done by gene,i.e., each gene individually is checked to mutate or not. The
probability of mutation is set by the user and it has to be small in order to keep all the earned by evolution.
In this version of the project, the neural network topologies to evolve belongs to a Feed Forward model.
In Keras is defined as a sequential model with Dense Layers.

4 Instruction on how to test the trained DNN and how to use the
GUI
4.1 Install Dependencies
To use this project, you need to hace installed the following python 2.7 libraries:
• numpy

• Keras
• Tkinter
• Pandas
• TensorFlow

• GraphViz
• pydot

2
4.2 Graphical User Interface
4.2.1 Main
The first window to interact with in the software is shown in figure 2. It includes Four buttons with the task
to perform. The first button is to evolve the neural network, the second button is to test using a whole data
set, and the third button is to test with at most 10 elements, while the last one is to visualize a model

Figure 2: Main Window

4.2.2 Run Genetic Algorithm

The GUI,shown in figure 3, is organized in three sections, the data set section, the hyperparameters to tune
and the genetic algorithm hyperparameters.
The first section of the GUI is where the user could add the data set. The data set is divided in two sets,
the features and labels, it is important to notice that the order has to be the same in both files, this is, the
first row in features corresponds to the first class in labels. After selecting the files, click on Load DataSet
button, which , loads both files and presents the columns names, as a user verification process, the number
of samples is shown and the number of features to use too. User should make sure that Number of Features
and Input Neurons have the same number and the number of classes correspond to the problem.
These data sets are comma separated value files, csv files, with a specific format. All the lines that begin
with # are ignored, this are considered comments from the data set and it could be used to write some meta
information that is not needed for the neural model, but for people to share some information for instance
who is the author of the data set, or where it was downloaded. The first valid line is reserved for the feature
names, if for a reason the feature names are not present or important for the problem, for example, when
an image is flatten, the pixels are organized in one array the features name are not necessary, this can be
set by unchecking the check box below the data set input. After selecting the file to loading the data set the
number of records (rows) and features (columns) will be shown.
In order to solve a problem, the user has to select the optimizer, the loss function and activation function
of the output layer. The optimizer is a list with all the optimizer available in keras. This set contains SGD,
RMSprop, Adagrad, Adadelta, Adam, Adamax and Nadam. The loss function, as well, are all the one pro-
vided by keras which includes, mean squared error, mean absolute error, mean absolute percentage error,
mean squared logarithmic error, squared hinge, hinge, categorical hinge, logcosh, categorical crossentropy,
sparse categorical crossentropy, binary crossentropy, kullback leibler divergence, poisson, cosine proximity.These
parameters are not allowed to evolve because they define the kind of problem to solve and the problem does
not change.
The second section is related to the parameters the user wants to evolve. The system allows the user to
select which neural network hyperparamenters to evolve by selecting the check box next to it. If the user
decide not to evolve one specific parameter, he/she will have to set the default value. The activation function
will be a list of all the one available in keras. This contains, softmax, elu, selu, softplus, softsign, relu, tanh,
sigmoid, hard sigmoid, exponential and linear.
The third GUI section contains the hyperparameters of the Genetic Algorithm. The hyperparameters
are three, the Population Size, number of generations to evolve and mutation probability.

3
 After filling all the values in the GUI, the button Run will be available to click. When the user is ready
to start the evolution process the system ask the project name, which is used to create a folder where the
user wants to save all the neural network population.
The system uses four types of files. The file .h5 is used to save the weights of the model. A json file
is used to save the topology recognized by keras. The .mod file is a defined for the system, which includes
information about how to test the model with the information required which is used in the testing routine.
The last file is the .gen file, this file save the individuals id and fitness per generation.
After the evolution does its job, the button Fitness Curve is enable. This functionality allows to see how
the fitness function behavior during the whole evolutionary process, i.e., the learning or adaptation curve.
This curve shows the population fitness through generations.

Figure 3: Window to Design the Neuro-Evolution Model

A learning curve is shown after clicking on the Fitness Curve Button, next to the Curve the best model
with its fitness is shown, allowing to open the two testing windows.

4.2.3 Testing a Data Set

After evolving the Neural Networks Topology a set of best Neural Networks will be saved in order to test
them. The window show in figure 5 allows to test using a whole data set and show the loss and accuracy
obtained in the testing set.

4
 Figure 4: Fitness Curve

Figure 5: Frame for testing a Data set

4.2.4 Testing a Small Data Set

If the user wants to introduce a particular sample or small data set, at most 10 elements. She can use the
window 6, which shows the predicted value and the expected value to compare the result.

4.2.5 Visualization of a Model

The frame 7 allows the user to visualize the model. It creates a ”png” file that is saved in the same folder
where the model is located.

5 Future Work
The future of this project is related to the GUI and the adding more functions. For the GUI I plan to
add a menu to organize the different functions to make easier for the user. In relation to the functions, I

5
 Figure 6: Frame for testing a small Data set

(a) Visualize Model

(b) Example Model

Figure 7: Visualization Tool

plan to add an automatic normalization process, to save the project with the parameters and to provide the
possibility to pause and restart it when needed. Although the system does not crash at this moment, I need
to create a way to catch all errors and show them to the user.

6 Supplementary Information
The project’s code is available in https://github.tamu.edu/jivfur/NeuroEvolution, and the dataset from
parkinson is available at https://archive.ics.uci.edu/ml/datasets/Parkinson%27s+Disease+Classification
A video demo from this GUI execution can be located at : https://youtu.be/ugJy1zUokKY

6
References
[1] Gul Muhammad Khan, Julian F. Miller, and David M. Halliday. Breaking the synaptic dogma: Evolving
a neuro-inspired developmental network. In Xiaodong Li, Michael Kirley, Mengjie Zhang, David Green,
Vic Ciesielski, Hussein Abbass, Zbigniew Michalewicz, Tim Hendtlass, Kalyanmoy Deb, Kay Chen Tan,
Jürgen Branke, and Yuhui Shi, editors, Simulated Evolution and Learning, pages 11–20, Berlin, Heidel-
berg, 2008. Springer Berlin Heidelberg.

[2] Serbes G. Gunduz A. Tunc H.C. Nizam H. Sakar B.E. Tutuncu M. Aydin T. Isenkul M.E. Sakar, C.O.
and H. Apaydin. A comparative analysis of speech signal processing algorithms for parkinson’s disease
classification and the use of the tunable q-factor wavelet transform. applied soft computing. https:
//archive.ics.uci.edu/ml/datasets/Parkinson%27s+Disease+Classification, 2018.
[3] Mohit Sewak, Md Rezaul Karim, and Pradeep Pujari. Practical Convolutional Neural Networks: Imple-
ment advanced deep learning models using Python. Packt Publishing Ltd, 2018.
[4] Kenneth O Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulainen. Designing neural networks through
neuroevolution.
[5] Honglei Zhang, Serkan Kiranyaz, and Moncef Gabbouj. Finding better topologies for deep convolutional
neural networks by evolution. arXiv preprint arXiv:1809.03242, 2018.