Now a day, artificial neural network has been applied popularly in many fields of human

life. However, creating an efficient network for a large classifier like handwriting
recognition systems is still a big challenge to scientists. In my last article named
Library for online handwriting recognition system using UNIPEN database, I presented
an efficient library for a handwriting recognition system which can create, change a
neural network simply. The demo program showed good recognition results to digit set
(97%) and alphabet sets (93%).This article I will continue to present a solution for a
large patterns classification in general and handwriting recognition in particular.

Recognition rate significantly increase when using additional spell checker

module

Neural network for a recognition system

In the traditional model of pattern recognition, a hand-designed feature extractor
gathers relevant information from input and eliminates irrelevant variabilities. A trainer
classifier (normally, a standard, fully-connected multi-layer neural network can be used
as a classifier) then categorizes the resulting feature vectors into classes. However, it
could have some problems which should influent to the recognition results. The
convolution neural network (CNN) solves this shortcoming of traditional one to achieve
the best performance on pattern recognition task.

The CNNs is a special form of multi-layer neural network. Like other networks, CNNs are
trained by back propagation algorithms. The difference is inside their architecture. The
convolutional network combines three architectural ideas to ensure some degree of
shift, scale, and distortion invariance: local receptive field, shared weights (or weight
replication) spatial or temporal sub-sampling. They have been designed specially to
recognize patterns directly from digital images with the minimum of pre-processing
operations. The architecture details of CNN have been described comprehensively in
articles of Dr. Yahn LeCun and Dr. Patrice Simard (see my previous articles).

Figure 1: The Architecture of LeNET 5

Figure 2: An input image followed by a feature map performing a 5 5

convolution and a 2 x 2 sub-sampling map
The recognition results of the above networks are really high to small patterns collection
such as digit, capital letters or lower case letters etc. However, when we want to create
a larger neural network which can recognize a bigger collection like digit and English
letters (62 characters) for example, the problems begin appear. Finding an optimized

and large enough network becomes more difficult, training network by large input
patterns takes much longer time. Convergent speech of the network is slower and
especially, the accuracy rate is significant decrease because bigger bad written
characters, many similar and confusable characters etc. Furthermore, assuming we can
create a good enough network which can recognize accurately English characters but it
certainly cannot recognize properly a special character outsize its outputs set (a Russian
or Chinese character) because it does not have expansion capacity. Therefore, creating
a unique network for very large patterns classifier is very difficult and may be
impossible.
The proposed solution to the above problems is instead of using a unique big network
we can use multi smaller networks which have very high recognition rate to these own
output sets. Beside the official output sets (digit, letters) these networks have an
additional unknown output (unknown character). It means that if the input pattern is not
recognized as a character of official outputs it will be understand as an unknown
character. Then the input pattern will be transferred to the next network until the
system can recognize it correctly.

Figure 3: Convolution neural network with unknown output

Figure 4: Recognition System using multi neural networks

This solution overcomes almost limits of the traditional model. The new system includes
a several small networks which are simple for optimizing to get the best recognition
results. Training these small networks takes less time than a huge network. Especially,
the new model is really flexible and expandable. Depending on the requirement we can
load one or more networks; we can also add new networks to the system to recognize
new patterns without change or rebuilt the model. All these small networks have
reusable capacity to another multi neural networks system.

Experiment
The demo program is built to the purpose showing all stages of a recognition system
including: create a component network, train a network, test networks on UNIPEN
dataset and test networks on a mouse drawing control. It is tutorials which can help
everybody can understand to a recognition system. All functions can be implemented
on the program GUI. So you can create, train, and test your network on runtime without
change any code or restart the program.

Creating new neural network

Figure 6: Creating new neural network Interface

Creating new neural network completely bases on GUI. Creating a network depends on
the input pattern size, number of layers, data set. On the output layer we can choose
unknown output checkbox to create an additional unknown output to the network or
ignore it to create a normal network
Of course, we can still to create a network by code:

Training a network
After creating a neural network using "Create network" function, the network will be
trained using UNIPEN database.

Figure 7: Training network interface

Depending on the network size we can choose training set is 1a, 1b or 1c in the
UNIPENdata folder. Statistic of training process can show many useful information such
as: No. of epoch, MSE, training time per epoch, success rate

Figure 10: Loading trained network parameters files

In order to active the recognition function I simply load trained network parameters
files. Depending to my recognition requirement I can load one, two or all files. The
recognition results are really good (higher 90%) if I load only one network to recognize
its output characters. However, when I load multi network the systems accuracy rate
becomes lower. The main reasons are many confusable characters in cursive text; the
training sets are not large enough etc.
For a large pattern collection like handwritten characters, there are so many similar
characters which can make not only machine but also human confuse in some cases
such as: O, 0 and o; 9, 4, g,q etc. These characters can make networks misrecognize.
Hence the solution has been being upgraded which significant increate recognition rate
by using an additional spellchecker/voting module at the output of system. The input
pattern will be recognized by all component networks. These outputs (except unknown
outputs) then will be set as the inputs of the spellchecker/voting module. The module
will bases on previous recognized characters, internal dictionary and other factors to
decide which one will be the most accurate recognized character.

Figure 11: The new recognition system using Spell checker /voting module

The new recognition system using Spell checker /voting module (internal
dictionary)

The spellchecker module makes the system recognizes much better

Conclusion
The proposed recognition model has solved almost problems to a large recognition
system: the capacity of recognizing large pattern collection, flexible design and
deployment, expandable and reusable capacity...etc. Increasing accuracy rate to the
system also can do easier by increasing recognition rate of component networks, using
the spell checker /voting module etc. The demo program also proved the capacity of the
library which should be used in many other applications such as prediction application,
face recognition...
Future work and upgrade
Some features would be update to the library:
- Convolution and sampling layer of LeNET model.
- Spell checker / voting module
-character segmentation.