Chapter 2

Simple Neural Nets for Pattern

Classification

Architecture
z A simple neural network consists
of a layer of input units and a
single output unit.
z In pattern classification problems,
each input vector (pattern)
belongs, or does not belong, to a
particular class or category.
z For a neural net approach, we
assume we have a set of training
patterns for which the correct
classification is known.
z Eg. Pattern classification is used
to detect heart abnormalities
(1963-Donald Specht)

1
Biases and Thresholds

z Bias acts as a weight on a connection from a unit

whose activation is always 1.
Activation function :
1 if net 0;
f (net ) =
- 1 if net < 0;
where
net = b + xi wi
i

z A fixed threshold can be used instead of using

a bias. This is equivalent to an adjustable bias.
Activation function :
1 if net ;
f (net ) =
- 1 if net < ;
where
net = xi wi
i

Linear Separability
y _ in = b + xi wi
i
z The boundary between yes (y_in > 0) and no (y_in < 0) is called
the decision boundary and determined by
b + xi wi = 0 (bipolar signal)
i

Bipolar signal : " yes" = 1, " no" = - 1

z The region where y is positive is separated from the region where it
is negative by the line
w1 b
x2 = x1
w2 w2
z We can have many different lines and many choices of w1, w2 and
b. The choice of the sign of b determines which side of the
separating line corresponds to a +1 response and which side to a -1
response.

2
AND function
Input (x1, x2) Output (t)
(1,1) +1
(1,-1) -1
(-1,1) -1
(-1,-1) -1
A possible decision boundary:
x2=-x1+1
b=-1
w1=1
w2=1

OR function
Input (x1, x2) Output (t)
(1,1) +1
(1,-1) +1
(-1,1) +1
(-1,-1) -1
A possible decision boundary:
x2=-x1-1
b=1
w1=1
w2=1

3
XOR function

Input (x1, x2) Output (t)

(1,1) -1
(1,-1) +1
(-1,1) +1
(-1,-1) -1
Try to solve this!!

zIt is easy to observe that no single straight

line can separate the points for positive
and negative responses.
zXOR function cannot be solved by 2-input
single output mapping (single-layer net).

4
Hebb Networks
z Hebb proposed that learning occurs by modification of the
synapse strengths (weights) in a manner such that if two
neurons are both on at the same time, the weight
between those neurons should be increased. (Hebb rule)
z Extended Hebb rule: Also increase the weight if both
neurons are off at the same time.
z A single-layer neural net trained using Hebb rule is called
as a Hebb net.
wi (new) = wi (old ) + xi y
z There are some limitations of the Hebb rule for binary
data.

Hebb net for the AND function:

Binary inputs and targets
Input Target
(x1 x2 1)
(1 1 1) 1 w1 = x1t
(1 0 1) 0 w2 = x2t
(0 1 1) 0
b = t
(0 0 1) 0
Weight updates for the first input are:
Input Target Weight Changes Weights
(x1 x2 1) (w1 w2 b) (w1 w2 b)
(0 0 0)
(1 1 1) 1 (1 1 1) (1 1 1)

Separating line: x2 = -x1 -1

5
zNo learning occurs for second, third and
fourth training inputs.
Input Target Weight Changes Weights
(x1 x2 1) (w1 w2 b) (w1 w2 b)
(1 0 1) 0 (0 0 0) (1 1 1)
(0 1 1) 0 (0 0 0) (1 1 1)
(0 0 1) 0 (0 0 0) (1 1 1)

Thus, using binary target values prevents the net from

learning any pattern for which the target is off.

Hebb net for the AND function:

Binary inputs, bipolar targets
Input Target
(x1 x2 1)
(1 1 1) 1
(1 0 1) -1
(0 1 1) -1
(0 0 1) -1
Weight updates for the first input are:
Input Target Weight Changes Weights
(x1 x2 1) (w1 w2 b) (w1 w2 b)
(0 0 0)
(1 1 1) 1 (1 1 1) (1 1 1)

Separating line: x2 = -x1 -1

6
zLearning continues for second, third and
fourth training patterns.

Input Target Weight Changes Weights

(x1 x2 1) (w1 w2 b) (w1 w2 b)
(1 0 1) -1 (-1 0 -1) (0 1 0)
(0 1 1) -1 (0 -1 -1) (0 0 -1)
(0 0 1) -1 (0 0 -1) (0 0 -2)

These weights do not provide the correct response for the

input pattern. (Tutorial show it!!)

Hebb net for the AND function:

Bipolar inputs and targets
Input Target
(x1 x2 1)
(1 1 1) 1
(1 -1 1) -1
(-1 1 1) -1
(-1 -1 1) -1
Weight updates for the first input are:
Input Target Weight Changes Weights
(x1 x2 1) (w1 w2 b) (w1 w2 b)
(0 0 0)
(1 1 1) 1 (1 1 1) (1 1 1)

Separating line: x2 = -x1 -1

7
Weight updates for the second input are:
Input Target Weight Changes Weights
(x1 x2 1) (w1 w2 b) (w1 w2 b)
(1 1 1)
(1 -1 1) -1 (-1 1 -1) (0 2 0)

Separating line: x2 =0
Response of the net is now correct for the first two input
points.

Weight updates for the third input are:

Input Target Weight Changes Weights
(x1 x2 1) (w1 w2 b) (w1 w2 b)
(0 2 0)
(-1 1 1) -1 (1 -1 -1) (1 1 -1)

Separating line: x2 = -x1 +1

Response of the net is now correct for all input points.

8
Weight updates for the fourth input are:
Input Target Weight Changes Weights
(x1 x2 1) (w1 w2 b) (w1 w2 b)
(1 1 -1)
(-1 -1 1) -1 (1 1 -1) (2 2 -2)

Separating line: x2 = -x1 +1 (still the same)

Perceptron
z Originally developed by Rosenblatt.
z Have three layers of neurons sensory units,
associator units and response units.
z Sensory units are connected to the associator
units with fixed weights of +1, 0 or -1.
z Activation function for associator unit is binary
step function with a threshold.
z The output of perceptron is y=f(y_in).
1 if y_in >

f ( y _ in) = 0 if - y_in
1 if y_in < -

9
zWeights from the associator units to the
response (output) units are adjusted by
the perceptron learning rule.
zIf an error occurs, the weights would be
changed according to the formula
wi(new)=wi(old) + txi
t is target value (+1 or -1)
is the learning rate
zTraining would continue until no error
occurred.

Simple Perceptron For Pattern

Classification
z Only the weights from the
associator units to the output
units could be adjusted. We limit
out consideration to the single-
layer net show.
z The goal of the net is to classify
each input pattern. Output unit
gives a response of +1 if
classified as belonging, and -1 if
classified as not belonging.
z Have 2 separating lines:
w1x1+w2x2+b >
w1x1+w2x2+b < -

10
Perceptron for the AND Function: Binary
inputs, bipolar targets
We take =1, initial weights and bias = 0, threshold = 0.2.
w = t(x1, x2, 1) or 0 (no error)

INPUT NET OUT TARGET W WEIGHTS

(x1 x2 1) (w1 w2 b)
(0 0 0)
(1 1 1) 0 0 1 (1 1 1) (1 1 1)

Separating lines:
x1+x2+1 = 0.2 and x1+x2+1 = -0.2

INPUT NET OUT TARGET W WEIGHTS

(x1 x2 1) (w1 w2 b)
(1 1 1)
(1 0 1) 2 1 -1 (-1 0 -1) (0 1 0)

Separating lines: x2 = 0.2 and x2 = -0.2

INPUT NET OUT TARGET W WEIGHTS

(x1 x2 1) (w1 w2 b)
(0 1 0)
(0 1 1) 1 1 -1 (0 -1 -1) (0 0 -1)

11
INPUT NET OUT TARGET W WEIGHTS
(x1 x2 1) (w1 w2 b)
(0 0 -1)
(0 0 1) -1 -1 -1 (0 0 0) (0 0 -1)

Response for input (1,1) is not correct. We train with the second epoch.

INPUT NET OUT TARGET W WEIGHTS

(x1 x2 1) (w1 w2 b)
(0 0 -1)
(1 1 1) -1 -1 1 (1 1 1) (1 1 0)

Separating lines: x1+x2 = 0.2 and x1+x2 = -0.2

INPUT NET OUT TARGET W WEIGHTS

(x1 x2 1) (w1 w2 b)
(1 1 0)
(1 0 1) 1 1 -1 (-1 0 -1) (0 1 -1)

Separating lines: x2 -1 = 0.2 and x2 -1 = -0.2

INPUT NET OUT TARGET W WEIGHTS

(x1 x2 1) (w1 w2 b)
(0 1 -1)
(0 1 1) 0 0 -1 (0 -1 -1) (0 0 -2)

INPUT NET OUT TARGET W WEIGHTS

(x1 x2 1) (w1 w2 b)
(0 0 -2)
(0 0 1) -2 -1 -1 (0 0 0) (0 0 -2)

12
INPUT NET OUT TARGET W WEIGHTS
(x1 x2 1) (w1 w2 b)
(0 0 -2)
(1 1 1) -2 -1 1 (1 1 1) (1 1 -1)
(1 0 1) 0 0 -1 (-1 0 -1) (0 1 -2)
(0 1 1) -1 -1 -1 (0 0 0) (0 1 -2)
(0 0 1) -2 -1 -1 (0 0 0) (0 1 -2)

(1 1 1) -1 -1 1 (1 1 1) (1 2 -1)
(1 0 1) 0 0 -1 (-1 0 -1) (0 2 -2)
(0 1 1) 0 0 -1 (0 -1 -1) (0 1 -3)
(0 0 1) -3 -1 -1 (0 0 0) (0 1 -3)

(1 1 1) -2 -1 1 (1 1 1) (1 2 -2)
(1 0 1) -1 -1 -1 (0 0 0) (1 2 -2)
(0 1 1) 0 0 -1 (0 -1 -1) (1 1 -3)
(0 0 1) -3 -1 -1 (0 0 0) (1 1 -3)

INPUT NET OUT TARGET W WEIGHTS

(x1 x2 1) (w1 w2 b)
(1 1 1) -1 -1 1 (1 1 1) (2 2 -2)
(1 0 1) 0 0 -1 (-1 0 -1) (1 2 -3)
(0 1 1) -1 -1 -1 (0 0 0) (1 2 -3)
(0 0 1) -3 -1 -1 (0 0 0) (1 2 -3)

(1 1 1) 0 0 1 (1 1 1) (2 3 -2)
(1 0 1) 0 0 -1 (-1 0 -1) (1 3 -3)
(0 1 1) 0 0 -1 (0 -1 -1) (1 2 -4)
(0 0 1) -4 -1 -1 (0 0 0) (1 2 -4)

(1 1 1) -1 -1 1 (1 1 1) (2 3 -3)
(1 0 1) -1 -1 -1 (0 0 0) (2 3 -3)
(0 1 1) 0 0 -1 (0 -1 -1) (2 2 -4)
(0 0 1) -4 -1 -1 (0 0 0) (2 2 -4)

(1 1 1) 0 0 1 (1 1 1) (3 3 -3)
(1 0 1) 0 0 -1 (-1 0 -1) (2 3 -4)
(0 1 1) -1 -1 -1 (0 0 0) (2 3 -4)
(0 0 1) -4 -1 -1 (0 0 0) (2 3 -4)

13
INPUT NET OUT TARGET W WEIGHTS
(x1 x2 1) (w1 w2 b)

(1 1 1) 1 1 1 (0 0 0) (2 3 -4)
(1 0 1) -2 -1 -1 (0 0 0) (2 3 -4)
(0 1 1) -1 -1 -1 (0 0 0) (2 3 -4)
(0 0 1) -4 -1 -1 (0 0 0) (2 3 -4)

Positive response:
2x1 + 3x2 - 4 > 0.2
Boundary line x2 = -2/3x1 + 7/5

Negative response:
2x1 + 3x2 4 < -0.2
Boundary line x2 = -2/3x1 + 19/5

Perceptron for the AND Function: Bipolar

inputs and targets.
INPUT NET OUT TARGET W WEIGHTS
(x1 x2 1) (w1 w2 b)
(1 1 1) 0 0 1 (1 1 1) (1 1 1)
(1 -1 1) 1 1 -1 (-1 1 -1) (0 2 0)
(-1 1 1) 2 1 -1 (1 -1 -1) (1 1 -1)
(-1 -1 1) -3 -1 -1 (0 0 0) (1 1 -1)

(1 1 1) 1 1 1 (0 0 0) (1 1 -1)
(1 -1 1) -1 -1 -1 (0 0 0) (1 1 -1)
(-1 1 1) -1 -1 -1 (0 0 0) (1 1 -1)
(-1 -1 1) -3 -1 -1 (0 0 0) (1 1 -1)

Change from binary to bipolar representation improves the result.

14
ADALINE
zAdaptive Linear Neuron (ADALINE) uses
bipolar activations for its input signals and
target output.
zADALINE is trained using the delta rule
(also known as least mean squares (LMS)
or Widrow-Hoff rule).
zAn ADALINE is a single neuron that
receives input from several units. It has
only one output unit.

z If ADALINES are combined so that the output

from some of them becomes input for the
others, then the net becomes multilayer and is
known as MADALINE.
The squared error for a particular training pattern is
E = (t - y_in)2
Minimum squared error :
E y _ in
= 2(t y _ in) = 2(t y _ in) xi
w i wi
x i is the input unit
n
y_in is the net input to the output unit Y where y_in = x i w i
i =1

t is the target output

15