Neural Networks - I

[Artificial] Neural Networks

• Computation based on Biological Neural Net
– humans can generalize from experience
– Can ANNs do the same

• A class of powerful, general-purpose tools

– Prediction
– Classification
– Clustering

• Computerized Neural Nets attempt to bridge the gap

– Predicting time-series in financial world
– Diagnosing medical conditions
– Identifying clusters of valuable customers
– Fraud detection
– etc…
Neural Networks
• ‘When applied in well-defined domains, their ability to
generalize and learn from data “mimics” a human’s
ability to learn from experience.’
(!!!)
• Very useful in Data Mining…
better results are the hope
Adequately designed and trained NN can capture varied
patterns
• Drawback – models tend to be difficult to understand
(Is it necessary to ‘understand’?)
History
• Early years 1930-1950

• AI vs. NN
– ‘Perceptrons’ and the XOR problem

• 1980s
– Back-propagation, Kohonen nets
– Applications
– Availability of computing power

• More ‘intelligent’ NNs?

– “On Intelligence” –Jeff Hawkins
Real Estate Appraiser…

??

“…is not applying some set formula, but balancing her experience
and knowledge of sale prices of similar houses…her knowledge about
housing prices is not static...fine tuning her calculation to fit the latest data”
Loan Prospector – HNC/Fair Isaac

A Neural Network is like a black box that knows how to

process inputs to create a useful output. The calculation is
quite complex and difficult to understand, yet the results are
often useful
Is it a black box ?

Garbage in ? Garbage out ?

Neural Net Limitations
• Neural Nets are good for prediction and
estimation when:
– Inputs are well understood
– Output is well understood
– There are adequate examples to “train” the neural net

• Neural Nets are only as good as the training set

used to generate it. The resulting model is static
and must be updated with more recent
examples and retraining for it to stay relevant
Feed-Forward Neural Net Examples
• One-way flow through the network from inputs to outputs
Feed-Forward Neural Net Examples
Biological neuron
Artificial Neuron
The loan appraiser

Model defined by
interconnection weights
NN can have multiple output neurons
Neural Network Training
• Training - process of setting the best weights on the
edges connecting all the units in the network

• Use the training set to calculate weights such that NN

output is as close as possible to the desired output for as
many of the examples in the training set as possible

• Back propagation has been used since the 1980s to

adjust the weights (other methods are now available):
– Calculates the error by taking the difference between the
calculated result and the actual result
– The error is fed back through the network and the weights are
adjusted to minimize the error
How does it work ?
x1
w1
x2 w2
w3 y = f (Net)
x3
w4 Transfer function
x4 Net= ∑wi xi
i

1
y = −Net
1+e
Hard limiter Linear threshold Squashing function
(Sigmoid)
Common transfer functions
 Single ‘layer’ NN
W: weight matrix
w11 w11 w12 …w1n
x1 y1
w21 w22 …w2n
……
x2 y2 wm1 wm2 …wmn
x: input vector
Net = x.W (dot product)
xm wmn wk
yn
Net = |wk||x| (cos A)
Inputs Outputs x
m Weights n
mxn
Assume no transfer function
w1
Let only one neuron with
w2
highest output ‘fire’
yk = w1kx1 + w2kx2 +…+wmkxm
w5 w3 = x.wk

w4 x NN classifies inputs into one of 5 classes.

How do we get the weights? – NN Training
 Multi-layer NN
Consider no transfer function
x1
y1
Then Y = (x W1) W2
x2
y2 = x (W1W2)

= x ( W)
yp
xm ie. equivalent to single layer
Outputs
Weights Weights p
Inputs W1
m W2 Advantage arises from transfer
function (non-linearity)
Perceptrons and XOR
x1 Rosenblatt (1962): Perceptron can learn
w1 t
w2 anything that it can represent
w3
Representation – ability of a NN to simulate
wm
xm Net = ∑ wi xi a particular function
i
If Net >= threshold, output 1, else output 0

x2 (0,1) (1,1)
Can we linearly separate the red from yellow?

Linear separability limitation of single layer

(1,0) perceptron
x1
(0,0)
Overcoming linear seperability
s1 t In layer 1, each neuron implements
w11 0.5 a linear separator
x1 t=0.75
w21 w21 0.5 In layer-2, output is 1 only when
x2
w22 both s1=1 and s2=1 (AND)
s2

Output is 1
only is this
region
Training a NN
• Adjust weights such that the application of inputs produce desired
outputs (as close as possible)
• Input data is continuously applied, actual outputs calculated, and
weights are adjusted
• Weights should converge to some value after many rounds of training

• Supervised training
– Adjust weights such that differences between desired and actual outputs
are minimized
– Desired output: dependent variable in training data
– Each training example specifies
{independent variables, dependent variable}

• Unsupervised training
– No dependent variable specified in training data
– Train the NN such that similar input data should generate same output