Using Artificial Neural Network with JOONE

Version 1.0

Prepared by, 21-Apr-2010

Vivek Sheel Gupta, Software Developer
Viveksheelg@yahoo.com
 JOONE Framework

1. Artificial Neural Network using JOONE Framework...........................................................................3

2. Joone?....................................................................................................................................................3
3. Using JOONE........................................................................................................................................4
a. Setting up the Neural Network..........................................................................................................5
a. Setting up the Neural Network..............................................................................................................5
b. Training the Neural Network.............................................................................................................8
b. Training the Neural Network.................................................................................................................8
c. Using the Neural Network.................................................................................................................9
c. Using the Neural Network.....................................................................................................................9
4. How to save the Neural Network?.......................................................................................................10
5. How to load the saved Neural Network?.............................................................................................11
6. How to enter new input layer to trained network in order to test Neural Network?...........................11
7. Using Plug-ins ....................................................................................................................................11
8. Conclusions.........................................................................................................................................13
1.Artificial Neural Network using JOONE Framework

JOONE Framework implements a new approach in the use of Artificial Neural Networks, in
other terms Artificial Intelligence. This will help us identifying the best architecture for
our Neural Network. The basic idea is to have an environment to easily train many neural
networks in parallel, initialized with different weights, parameters, or different
architectures, so the user can find the best Neural Net simply by selecting the fittest
neural network after the training process. Our basic goal is to build a flexible
environment programmable by the end user, so any existing or newly discovered global
optimization algorithm can be implemented. This is why a neural network built with
JOONE is serializable and remotely transportable using any wired or wireless protocol and
it is easily runnable using a simple, small and generalized program. Thus allowing us to
implement Artificial Intelligence in every Java Virtual Machine based technology such as
Mobiles phones, PDA. Making more economic efficient models, models that gets smarter
with time.
Hoping this guide helps you in meeting your requirements in implementing ANN (Artificial
Neural Net) and also finds this work interesting and useful.

2.Joone?

Joone is a Java framework to build and run AI applications based on artificial neural
networks. Joone applications can be built on a local machine, be trained on a distributed
environment and run on any device.
Joone consists of a modular architecture based on linkable components that can be
extended to build new learning algorithms and neural networks architectures.
All the components have some basic specific features, like persistence, multithreading,
serialization and parameterization. These features guarantee scalability, reliability and
expansibility, all mandatory features to reach the final goal to represent the future
standard on the AI world. JOONE applications are built out of components. These
components are pluggable, reusable and persistent code modules. These components
are written by developers.
Joone can be used to build Custom Systems, adopted in an embedded manner to
enhance an existing application, or employed to build applications on Mobile Devices,
Virtual Systems.

Custom systems
A great need of the industrial market is to have the possibility to resolve business
problems suitable with AI. Joone wants to represent the optimal solution to build
applications to satisfy such needs (Weather Forecasting etc.).
Its characteristics are optimal to build custom applications driven from the user’s needs,
where it’s important to have flexibility, scalability and portability.

Embedded systems
Into the core engine, the components are the bricks to build NN architecture. Their
purpose is to create AI applications writing Java code that uses the Joone’s API. The
license of the core engine is the Lesser General Public License (LGPL). Joone can be
embedded into various systems (i.e. data mining systems, automatic categorization
for search engines, customer classification, etc.)

3.Using JOONE
In this article, you will be shown various examples implementing Artificial Intelligence
using JOONE. The topic of neural networks is very broad and covers many different
applications like Weather forecasting. In this article, we will show you how to use JOONE
to solve few examples of a simple pattern recognition problem. Pattern recognition is a
very common use for neural networks.

Firstly, The NN (Neural Network) is trained with some specific pattern. Then this distorted
pattern is presented to network such that network recognizes that pattern to see
whether the neural network is able to recognize that pattern. The pattern should be
distorted in some way and the neural network still is able to recognize it. This is similar to
a human's ability to recognize and identify the pattern and will also help in obtaining the
output based on history or training. Thus as the program matures it makes even smarter
decisions.

JOONE basically consist of two types of objects

1. Neuron Layers 2. Synapses

Neuron layer represent the layer of one or more neuron that are similar in
characteristics. Network generally consist of 3 layers
1. Input Layer 2. Hidden Layer 3. Output Layer
These layers are generally connected together by Synapses. The synapses carry the
pattern, which is to be recognized from layer to layer.
Synapses do not just transmit the pattern from one neuron layer to the next. Synapses
develop biases towards elements of the pattern. As we know, in non Linear situation the
output will depend on the input variables but to the degree on which the output depends
on certain input variable will vary. These synapses will help to identify this bias behavior
and will be called biases. These biases will cause certain elements of the pattern to be
transmitted less effectively to the next layer than they would otherwise be and thus
control the flow of pattern in the network. These biases, which are usually called weights,
form the memory of the neural network. By adjusting the weights, which are stored in
synapses, the behavior of the neural network is altered.
Synapses also play another role in JOONE. In JOONE, it is useful to think of synapses as
data conduits. Synapse can carry both data in and out of Layer.

Will show a simple program that teaches JOONE to recognize the XOR operation and
produce the correct result. Will train the neural network & the process of training
involves presenting the XOR problem to the neural network and observing the result. If
the result is not what was expected, the training algorithm will adjust the weights, stored
in the synapses. The difference between the actual output of the neural network and the
anticipated output is called the error RMSE (Root Mean Square Error also known as
RMSE). Training will continue until the error falls below an acceptable level. This level is
generally a percent, such as 10%.
The training process begins by setting up the neural network. The input, hidden, and
output layers must all be created.

a. Setting up the Neural Network

Since it is example of XOR we are using sigmoid layer as input variable will be in range
from 0 to 1
input = new SigmoidLayer();
hidden = new SigmoidLayer();
output = new SigmoidLayer();

To connect the neuron layers, we must construct synapses. In this example we will have
two Full Synapses
//input -> hidden conn.
 FullSynapse synapse_IH = new FullSynapse();
// hidden -> output conn.
FullSynapse synapse_HO = new FullSynapse();

Connect the synapses to the appropriate neuron layers. The following lines of code do
this.

// Connect the input layer with the hidden layer

input.addOutputSynapse(synapse_IH); //Adding output synapse to carry out from input
hidden.addInputSynapse(synapse_IH); //Adding input synapse to carry in from input
// Connect the hidden layer with the output layer
hidden.addOutputSynapse(synapse_HO); //Adding output synapse to carry out from Hidden
output.addInputSynapse(synapse_HO); //Adding input synapse to carry in from Hidden

Set up the input synapse. There are various ways to provide the input to the network:-
1: FileInputSynapse
2: XLSInputSynapse
3: URLInputSynapse
4: MemoryInputSynapse
5: ImageInputSynapse
6: JDBCInputSynapse
7: YahooFinanceInputSynapse
As their names suggest, these are the various connectors which will provide the input to
the Neural Network In the following example we will use FileInputSynapse &
XLSInputSynapse.
fileInputSynapse= new FileInputSynapse();

We will import the required columns which are to be used. For XOR we will use first 2
columns as input. Code to set up the first 2 columns as input
fileInputSynapse.setFirstCol(1);
fileInputSynapse.setLastCol(2);
We can also use:
fileInputSynapse.setAdvancedColumnSelector (“1-2”);

In case of Excel sheet we will select all the columns which are present in the excel sheet
and use InputConnectors.
XLSInputSynapse= new XLSInputSynapse();
xlsInputSynapse.setFirstCol(1);
xlsInputSynapse.setLastCol(3);
//xlsInputSynapse.setAdvancedColumnSelector("1-3"); //Another way to set the number of Cols

Next, provide the name to the input file. The following lines of code set the filename for
the FileInputSynapse and XLSInputSynapse.
// This is the file that contains the input data
fileInputSynapse.setFileName(“input.txt”);
 Input.txt
xlsInputSynapse.setFileName(“input.xls”);

0 0 0
0 1 1
1 0 1
1 1 0

For XLSInputSynapse we will need to use the InputConnectors (InputConnectors are

used when we need to use other application from the JOONE Engine).

inputConnector= new InputConnector();

inputConnector.setAdvancedColumnSelector("1");
inputConnector.setFirstRow(1);
inputConnector.setLastRow(4);
inputConnector.setBuffered(false);

A synapse is just a conduit for data to travel between neuron layers. The
FileInputSynapse, XLSInputSynapse and InputConnectors are the conduit through which
data enters the neural network. To facilitate this, we must add the respective
InputSynapse to the input layer of the neural network.

input.addInputSynapse(fileInputSynapse) \\ input is the Input Layer of Neural Network

For Excel Sheets, firstly we will add the XLS to Connectors

inputConnector.setInputSynapse(xlsInputSynapse);

Then, add the Connector to Input layer

input.addInputSynapse(inputConnector);

Now in order to control the Neural Network and regulate, we use an object named
Monitor.
Monitor regulates the neural network. Using this we can set the learning rate of the
neural network. Always make sure you set an optimized value for this one as higher
learning rate will not let you reach the minimum RMSE and too small will slower your
learning itself. To optimize it one needs to check with different values and use the one
which has minimum RMSE.

monitor = new Monitor();

monitor.setLearningRate(0.8);// Setting the rate at which it learn
monitor.setMomentum(0.3); //Setting the rate at which it processes input

The monitor allows listeners to be added to it. As training progresses, it will notify the
listeners as to the progress of the training. For example:
monitor.addNeuralNetListener(this); // will implement event Listener during various phases of
Neural Network
 public void errorChanged(NeuralNetEvent e) {
Monitor mon = (Monitor)e.getSource();
/* We want print the results every 200 cycles */
if (mon.getCurrentCycle() % 200 == 0)
System.out.println(mon.getCurrentCycle() + " epochs remaining - RMSE = " +
mon.getGlobalError());
}

Set the other parameters for training the Monitor such as number of Iteration with
setTotCycles, Patterns. Total type of cases we are asking Neural Network to learn in this
XOR case is 4, so we specify pattern as 4. Along with that we need to specify the learning
to be true. If you set the learning parameter to false, the neural network would simply
process the input and not learn.

monitor.setPatterns(4);
monitor.setTotCycles(20000);
monitor.setLearning(true);

b. Training the Neural Network

The NN Setup is now complete. Now create a Teacher/Trainer and Monitor. The Teacher
is used to train the neural network because the monitor runs the neural network through
a number of training iterations. During training iteration, data is presented to the neural
network and the results are observed. The NN’s weights are stored in the synapse
connection that go between the neuron layers, which will be adjusted based on error. As
training progresses, this error/RMSE level will drop. We will repeat this procedure till the
acceptable value is reached.

teachingSynapse= new TeachingSynapse();

teachingSynapse.setMonitor(monitor);

Input file contains three columns. We have used the first two columns which specify the
input to the neural network. The third column contains the expected output when the
neural network is presented with the numbers in the first two columns. Train the teacher
with the 3rd column so that the error/RMSE can be determined. The error is the difference
between the actual output of the neural network and this expected output.

samples = new FileInputSynapse();

samples.setFileName(“input.txt”);
samples.setAdvancedColumnSelector (“3”);
teachingSynapse.setDesired(samples); //Adding samples to Teaching Synapse

For Excel Sheet:

desiredInputConnector= new InputConnector();
desiredInputConnector.setAdvancedColumnSelector("3");
desiredInputConnector.setFirstRow(1);
desiredInputConnector.setLastRow(4);
desiredInputConnector.setBuffered(false);
desiredInputConnector.setInputSynapse(xlsInputSynapse);
teachingSynapse.setDesired(desiredInputConnector); //Adding Desired Connector to Teaching
Synapse
Connect the Teacher to the output layer of the Neural Network. This will cause the trainer to receive
the output of the neural network.

// Connects the Teacher to the last layer of the net

output.addOutputSynapse(teachingSynapse);

Initialize the Neural Network which is to be trained and specify all the Layers of this network. We will
also add the teacher of the NeuralNet along with the Monitor, We can also add multiple layers to the
input layer but in order to simplify, we will be using only 1 layer for each input, hidden and output
layers. This will help the neural Network to distinguish each and every layer.

neuralNet = new NeuralNet();

neuralNet.addLayer(inputlayer,neuralNet.INPUT_LAYER);
neuralNet.addLayer(hiddenlayer,neuralNet.HIDDEN_LAYER);
neuralNet.addLayer(outputlayer,neuralNet.OUTPUT_LAYER);
neuralNet.setTeacher(teachingSynapse);
monitor= neuralNet.getMonitor();

Start all the Layers to begin the thread in each layer at background.

input.start();
hidden.start();
output.start();
teacher.start();

To execute the Neural Network use Go() function in the monitor.

monitor.Go(); //Monitor and NeuralNet have separate functions to run the Neural Network
monitor.Go(), will execute the Neural Network
NeuralNet.go() will also be used to execute the Neural Network

When program is executed monitor will learn from 4 different patterns and will revise
these patterns for 20000 times. During learning we will compare output of the neural
Network with the original output and obtain the RMSE. Our motive is bring down the
RMSE level to below 10% thus in acceptable range. In case we are not able to bring down
the RMSE level to acceptable range we need to tune up the network.

c. Using the Neural Network

After neural network has been trained, test it by presenting the input patterns to the
neural network and observing the results. To execute with Neural Network we need to
reset input as well as teacher for the Neural Network Currently, the neural network is in a
training mode. To begin with, we will remove the trainer from the output layer. We will
replace the trainer with FileOutputSynapse so that we can record the output from the
neural network. The following lines of code do this.

output.removeOutputSynapse(trainer);
FileOutputSynapse results = new FileOutputSynapse();
results.setFileName(“output.txt”); //Output of the Neural Network
fileInputSynapse.resetInput();
samples.resetInput(); //reset the teacher values
results.setMonitor(monitor);
 output.addOutputSynapse(results);

Next, we must restart all of the threads that correspond to the neural network.

input.start();
hidden.start();
output.start();
teacher.start();

Set the basic parameters for the monitor.

monitor.setPatterns(4);
monitor.setTotCycles(1);

This time we will setLearning as to be false as we just need to process the new input.

monitor.setLearning(false);

We will initiate the network by calling Go() method.

monitor.Go();

When the execution completes we will see the output.txt and we will observe the output
and the RMSE.

Output.txt:

0.012549763955262739
0.9854631848890223
0.9853159647305264
0.01783622084836082

Example Implementing XOR with XLS: Kindly find the AverageClass.java

Package

4.How to save the Neural Network?

Neural Networks are saved so we can transfer them to remote location and use it for processing. This is
one feature which makes the neural network highly portable.

FileOutputStream stream=new FileOutputStream(String networkname=”net.snet”);

ObjectOutputStream outputStream=new ObjectOutputStream(stream);
outputStream.writeObject(neuralNet); //Writing the Neural Net as object after training
outputStream.close();
5.How to load the saved Neural Network?
public NeuralNet restoreneuralnetwork(String fileName)
{
NeuralNetLoader netLoader=new NeuralNetLoader(fileName);
net= new NeuralNet();
net=netLoader.getNeuralNet();
}

6.How to enter new input layer to trained network in

order to test Neural Network?
NeuralNet xorNNet = this.restoreneuralnetwork("net.snet");
/**
* Getting the Input Layer of the loaded Network to the new Layer
* */
input = xorNNet.getInputLayer();
/**
* Removing the Input layer from the Network
* */
input.removeAllInputs();
/**
* Creating a New Input
* */
XLSInputSynapse xlsInputSynapse=new XLSInputSynapse();
xlsInputSynapse.setFileName("AvarageCheck.xls"); //Loading xls Sheet as new Input
xlsInputSynapse.setName("inputXLS");
xlsInputSynapse.setFirstCol(1);
xlsInputSynapse.setLastCol(2);
xlsInputSynapse.setFirstRow(1);
xlsInputSynapse.setLastRow(4);
xlsInputSynapse.setBuffered(true);
InputConnector inputConnector=new InputConnector();
inputConnector= new InputConnector();
inputConnector.setAdvancedColumnSelector("1");
inputConnector.setFirstRow(1);
inputConnector.setLastRow(4); //No. of Pattern user want to process through the trained Neural
Network
inputConnector.setBuffered(false);
inputConnector.setInputSynapse(xlsInputSynapse);
input.addInputSynapse(inputConnector);
/** Set this as the new Input layer in the network*/
xorNNet.setInputLayer(input);
/** Get the new Output*/
Layer output = xorNNet.getOutputLayer();
/***/
/**Start the network*/
xorNNet.go(true);

7.Using Plug-ins
To extend the capability of each layer, we can use Plug-ins. These plug-in will help us to
normalize the input for each specific layer, thus allowing us to feed input to the neural
network as per the real scenario. We can use a similar plug-in to extract the output from
the system in which ever format we require.

Implementation of Plug-ins in order to normalize the input variables.

/**
* Adding Plug-in to format the input in the format which is acceptable to the layer.
* */

NormalizerPlugIn normalizerPlugIn= new NormalizerPlugIn();

normalizerPlugIn.setAdvancedSeriesSelector("1");
normalizerPlugIn.setMax(1); //setting the max value as 1
normalizerPlugIn.setMin(0); //setting the min value as 0
normalizerPlugIn.setName("InputPlugin");
xlsInputSynapse.addPlugIn(normalizerPlugIn);

Implementation of Plug-ins helps us to predict the trend of the network.

Trend Prediction is not only one of the most famous but also toughest to implement,
where you want to predict the future trend of the output , whether it will be up or down,
To enable the prediction or forecasting we implement the 2 types of Plug-ins -

• MovingAveragePlugin
• MinMaxExtractorPlugin

For Example:

/**
*To Implement Traditional Trend we can implement the MovingAveragePlugIn -:
*Moving average Plug-in will monitor the output and the rate it is changing.
*this will help us to identify where the change of the output will take place.
**/
/**
* It finds the difference between the previous outcome and current outcome which makes it
*dependent on the past variable not only the current variable
* helps in identifying the pattern
* */

MovingAveragePlugIn averagePlugIn=new MovingAveragePlugIn();

averagePlugIn.setAdvancedMovAvgSpec("2");
averagePlugIn.setAdvancedSeriesSelector("1");
averagePlugIn.setName("Average Plugin");
normalizerPlugIn.addPlugIn(averagePlugIn);
/**
*Will set the input in the range to 0 to 1 as Sigmoid layer accepts only from 0 to 1
* */
/**
* MinMaxExtractorPlugIn is a plugin used to find out the variable at
* which level we are going to monitor the change
* Like in the below case, the input variable in the range
* 0
8
15
22
20
* the key point at which change takes places is "1" if the order was in format 0,2,4,6,8,10...
* In such case we can take MinMaxExtractorPlugIn.setMinChangePercentage(2);
* This will increase in the learning rate of the data especially in which case we want to predict
* trend of the output.
*/
MinMaxExtractorPlugIn extractorPlugIn=new MinMaxExtractorPlugIn();
extractorPlugIn.setMinChangePercentage(1);
extractorPlugIn.setAdvancedSerieSelector("2");
extractorPlugIn.setName("Check Points");

This plug-in will be added to the teacher layer.

normalizerPlugIndesired.addPlugIn(extractorPlugIn);

8.Conclusions
Artificial Intelligence is based on procedure of learning as much as data will be provided
during learning the better will be the results and these AI based network will help us to
replace the RULES Engine and make our technologies much more flexible and smarter.
This will also help us to make non linear operations; Predictions and Joone Framework
provide ideal platform for Implementing AI in any Java Virtual Machine based
Technologies.