Non Linear Systems

Materiale ad uso esclusivo degli studenti del corso. Vietata ogni forma di copia o distribuzione.
Per la preparazione allesame questo materiale va integrato con le spiegazioni del docente durante le lezioni, e con i libri di testo del corso.
NON-LINEAR SYSTEMS
Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS
Materiale ad uso esclusivo degli studenti del corso. Vietata ogni forma di copia o distribuzione. Per la preparazione allesame questo materiale va integrato con le spiegazioni del docente durante le lezioni, e con i libri di testo del corso.
INTRODUCTION
Data:
{u (1), u (2),K, u ( N )} {y (1), y (2),K y ( N )}
u (i)
Model f (u; )
y (i)
f (u; ) is a static function (not dynamic) of the input u. In general: y (i ) = f (u (i ); ) is the vector of the model parameters y (i ) only depends on u (i ) with the same index; it depends neither on u (i 1), u (i 2),K neither on y (i 1), y (i 2),K NB: index i and not necessarily t
Unlike what weve seen up to now, f (; ) is a non-linear function.
y (t ) = a1 y (t 1) + K + am y (t m) + b0u (t 1) + K + bpu (t p 1)
Linear/dynamic relationship Non-linear/static relationship
y (t ) = f (u (t ))
Since the relation between u and y is static, we can represent the points {u (1), u ( 2),K, u ( N )} {y (1), y (2),K y ( N )}, ignoring the index:
i
This representation gives information about the order of {u (i ), y (i )} pairs.
Example of static function (e.g. market basket)
y
i-th pair u-y {u (i ), y (i )}
u
In this representation we loose information about the temporal relationship among the different {u (i ), y (i )} pairs.
Its no longer necessary to explicitly consider the temporal order
of {u (i ), y (i )} pairs, since we want to identify a static model ( any information about a temporal order is not relevant)
On the other side we make use of i instead of t to underline that
the data order is not important (this is not necessarily a temporal index).
Dimensionality assumptions:
u (i ) p
u1 (i ) u (i ) 2 p u (i ) = M u i ( ) p
y (i ) 1 (mono-dimensional output)
Remark
These assumptions do not affect the generality of our models, since a system with p inputs and m outputs can be modeled with m parallel models, each with p inputs and one output.
Ex.
2 inputs 2 outputs
u1 (i ) u2 (i )
f (u (i ); )
equivalent
y1 (i ) y2 (i ) y1 (i )
2 inputs 1 output
u1 (i )
f 1 (u (i); 1 )
f 2 (u (i); 2 )
y2 (i )
u2 (i )
Remark
Its important to distinguish between two classes of problems.

Interpolation
We assume that data are not disturbed and we want that the
) passes through each {u (i ), y (i )} point. estimated function f (u ,
{u (i), y (i)}
) y (i ) = f (u (i );
A sufficient condition to solve the interpolation problem is that

N = n ( n = dim() ).
Fitting
We assume that data are noise-disturbed its useless to look
) passing through each {u (i ), y (i )} pair we want for a f (u ,

) to fit the data by canceling each meaningless noisef (u ,
generated variation.
y
) interpolated f (u (i );
) approximated f (u (i );
u
Unlike the interpolation problem, where N = n , the fitting problem intrinsically demands N >> n .
The fitting function is not perfect for the data we used for its identification, but owns generalization properties (suitable for other data too).
The interpolation function fits only to measured data (it makes no generalizations)
N.B.: If we can interpolate a data set using n < N , we solve a data-
compression problem.
Problem: how can we choose a class of non-liner parametric
functions f (; ) ?
The normally requested characteristic is: f must be a universal approximating function: provided that we use a sufficiently-large number of parameters, we can approximate any continuous function at whatever precision level, on a compact domain. If u U p , compact, and we need to fit the g : U function (with g of class C 0 ) with an f (u; ) class of parametric functions, we can state that f is an universal approximating function if: > 0, n that fulfils the following relation: g (u ) f (u; ) < so that
u U (the mistake made by f is uniformly lower than on all U).
The following function classes are universal approximating functions:

Polynomial Splines Multi-layer perceptron Neural Networks (NN) Radial Basis Functions (RBF) Wavelets
Example
2-dimensional polynomial
u1
y u2
y = a00 + a10 u1 + a01u 2 + a11u1u 2 + a20 u12 + a02 u 22 + K Provided that we use a sufficiently large number of parameters, the polynomial we obtain approximates any continuous function at any precision level, on a compact domain.
FEEDFORWARD NEURAL NETWORKS WITH MULTILAYER PERCEPTRONS (NN)
Example of Neural Network with:

1 input; 1 hidden layer with 1 neuron; 1 output.
u
Hidden layer (nonlinear layer)
()
Linear layer
: , , , are the Neural Network parameters:

= [ ] .
T
10
: is a static non-linear sigmoidal function. Examples:

x tanh( x)
( x )
1 1 0.5e x ( x )
1
0,5 0
x
Both these functions:

Regularity C ; Linearity around x = 0 ; They saturate asymptotically when x .
11
Analytical expression of the Neural Network:
y = + ( + u )
The structure:
is called neuron.
()
The parameters are:

: neuron weight;
: neuron bias (or offset) ;
12
Extend the example to a more general case:
Neural Network with:

2 inputs 1 hidden layer/non linear with 3 neurons; 1 output.
1
1
11
u1
()
1
2
12
1
1
()
21
22
u2
2 3
1
3
31
32
Non linear layer
()
Linear layer
13
To write the analytical expression, we define:

11 A = 21 31 1 B = 2 3 12 22 32
(weight matrix of the non-linear layer)
(bias vector of the non-linear layer)
1 C = 2 3 D = []
(weight vector of the linear layer).
(bias of the linear layer).
u U = 1 u 2
The I/O analytical expression is:
y = D + C T (B + AU )
= +
x1 ( x1 ) is applied element by element: x = ( x ) . 2 2

14
Its easy to extend this I/O relationship to a generic Neural Network with:

p inputs;
1 hidden layer with h neurons; 1 output.
y = D + C T ( B + AU )
1x1 hx1 hx1 hxp
15
(Graphic) example of Neural Network with:

1 input 2 non-linear layers with 3 and 2 neurons respectively; 1 output:
11
1
111
1
()
211
212
21
()
1
u1
12
213
1
y
121
1
()
221
22
()
13
222
()
223
131
1st non linear layer
2nd non linear layer
Linear layer
The 1st non-linear layer has: 1 x 3 weights + 3 bias The 2nd non-linear layer has: 3 x 2 weights + 2 bias. The linear layer has: 2 x 1 weights + 1 bias.
Totally we have 17 parameters.
16
Remark
A NN with one non-linear layer with h neurons and a linear layer is a universal approximating function (in theory - and in practice- its not necessary to use so many hidden layers).
Suppose we use Neural Networks with:

p inputs;
1 hidden layer with h neurons;
1 output layer (1 output).
Problem: how to find h optimal?
The easiest way is to use the cross-validation approach. Divide the data in two sets:

Training data / I; Validation data /V.
, using Find the optimal vector / I , and validate the model on /V.
) J (
, J /V
) )
, J /I
1 2 3 4
h h optimal
17
PROBLEM: IDENTIFICATION OF THE NN PARAMETERS

= arg min{J ()}
1 N 2 J () = ( y (i ) f (u (i ); )) N i =1
J () is the sample variance of the output error (MSE: Mean Square
Error). In case of Neural Networks, J () is a function:

0;
Not quadratic; Not convex.
we need iterative algorithms to minimize it (global minimum not
guaranteed). In case of Neural Networks, the iterative method for the minimum-
is known as Back-propagation algorithm. search
This algorithm is just the classical gradient method applied to Neural Networks:
( t +1) = ( t ) J () =
(t )
Where is the algorithm step and
J () is the gradient at step t. =

(t )
18
Example (1 input, 1 non-linear layer with 1 neuron, 1 output):

1 N J () = (i ) 2 N i =1 J () 2 N (i ) = (i ) N i =1
(i ) = y (i ) ( + ( + u (i ) ))
(i ) = 1 (i ) = ( + u (i ) ) (i ) & ( + u (i ) ) = (i ) & ( + u (i ) ) u (i ) = 2 N & ( ) + i u i u i ( ) ( ) ( ) N i =1 2 N & ( + u (i ) ) (i ) J () N i =1 = N 2 (i )( + u (i ) ) N i =1 2 N ( i ) N i =1
Remark: the choice of is made empirically:

If is small slow and safe optimization; If is big quick optimization but with risk of instability.
There are many variations of the basic Back-propagation method which differ in the choice of (and in the way it is made timevariant).
19
RADIAL BASIS NEURAL NETWORKS (RBF - RADIAL BASIS FUNCTIONS)
The RBF (Radial Basis Functions) networks are an alternative to the NN networks previously studied.
Example:
u 2 inputs u = 1 u 2

3 neurons (basis): 1 output

1
u1
1 2
1 y
u2
Analytic expression: y = + j j (u1 , u 2 )

j =1
20
The inner structure of j-th neuron (j-th basis) is:

j
u1 u2
(u
1j , 2j }) 2 = dist({u1 , u2 }, {
1 1j ) + (u 2 2j ) 2 2
exp(())
The dist function computes the Euclidean distance between {u1 , u 2 } and {1 j , 2 j }. (u ) overall works this way:
j u1 1 j
( ((
)2 + (u 2 2 j )2 ))
Notice that each j neuron is characterized by the following inner parameters:

1j
, 2 j } called neuron center
j called neuron width
21
Graphic representation (1 input)
1
for j decreasing
j
Graphic representation (2 inputs)
j
1 j
2 j
Vector {1 j , 2 j } defines the centre of each function j (bellshaped) j defines the bell width
22
Main characteristics of a RBF:

Its a linear function w.r.t. parameter vector:
y = + j j (u )
j =1
( and j are the optimization parameters)

Unlike the NN networks it always has only one non-linear layer; The non-linear function inside the neuron is no longer sigmoidal but bell-shaped:
j
( x )
The j of a RBF is a local function with compact support (in practice); instead works on the whole real domain.
23
General expression of a RBF:

u1 p inputs u = M ; u p

n neurons (basis) 1 output.

p 2 y = + j exp j uh h j j =1 h =1 n
Parameter vector :
= [
n ]
24
Problem: how to obtain the optimal parameter vector?
Given a measurement set:
{u (1), u ( 2),..., u ( N )} {y (1), y (2),..., y ( N )}

which minimizes we look for
1 J ( ) = N
n y ( i ) ( u ( i )) j j i =1 j =1 N 2
Problem of minimization of a semi-definite positive and quadratic function!
We have to solve this over-determined problem:

1 1 (u (1)) 2 (u (1)) 1 (u ( 2)) (u ( 2)) 1 2 M O M 1 1 (u ( N )) 2 (u ( N )) = Y c n (u (1)) y (1) 1 n (u (2)) y (2) 2 = O M M M L n (u ( N )) y N ( ) n L L
25
= Y is a system with N equations and n+1 unknown values (with N >> n + 1) T = T Y = ( T ) T Y

1
where ( T ) 1 T = + (pseudo-inverse of ).
To find the optimal parameter vector of a RBF, its enough to solve a problem of quadratic optimization:

1! solution; No local minima; Explicit solution.
26
Problem: the parameters hj and j were hidden in j ; actually we
assumed they were chosen a priori and that they were not the object of optimizations; how do we choose them? To choose the centers hj we normally use two approaches:
We randomly select n points {u (i )} among the N measured input
values;
Ex. (p=2; n=6)
u2
u (i ) measured
u1
6 centers of 6 neurons randomly chosen
27
We distribute evenly n points inside a rectangular domain of p .
Ex. (p=2; n=6)
u2 u2 max
Evenly-distributed 6 centers of 6 neurons
u2 min
u1 min
To choose j , we normally assume:
1 = 2 = K = n =
u1 max
u1
and then minimize with respect to (we design many RBF, using different values of )
28
) J (
4
optimal
The choice of is very critical as defines the smoothness of the function. (Graphical examples of chosen incorrectly )
29
Remark (comparison)
RBF Structure Training

NN Multy-layer

1 layer 1! solution no local minima explicit many
local minima iterative optimization few
Number of parameters
In general:

If u dimension is low ( p = 1,2,3,K5 ), we prefer to use RBF; If u dimension is high ( p = 10,K,100,K), we prefer to use the NN.
30
N-ARMAX MODELS
ARMAX Model:
y (t ) = a1 y (t 1) + K + a m y (t m) + b0 u (t 1) + K + b p u (t p 1) + + e(t ) + c1e(t 1) + K + c n e(t n) e ~ WN(, 2 )
N-ARMAX Model:
y (t ) = f ( y (t 1),K, y (t m), u (t 1),K, u (t p 1), e(t ),K, e(t n); )

f is a non-linear parametric and static function.
In practice N-ARX models are typically used:
y (t ) = f ( y (t 1),K , y (t m), u (t 1),K , u (t p 1); ) + e(t )

In this case the noise is linear and additive.
N-ARX: its easy to write its predictor:
(t / t 1) = f ( y (t 1),K, y (t m), u (t 1),K, u (t p 1); ) y

We obtain the optimal parameter vector , by minimizing:
J () =
1 N 2 (t / t 1) ) ( y (t ) y N t =1
31
The structure of N-ARX predictor is:
u (t )
z 1 z 1
u (t 1) u (t 2) f (; )
(e.g. NN or RBF)
z 1
y (t )
u (t p 1)
(t / t 1) y y (t 1) y (t 2)
z 1 z 1
z 1
y (t m)
The identification of a non-linear and dynamic model N-ARX (m,p+1) is practically reduced to the training of a static Neural Network (with m+p+1 inputs instead of a single one (delayed outputs and inputs).
32
Remark
When we identify NARX (or NARMAX) models its important to understand the difference between a prediction and simulation model. Assuming we collected the following measured data sets:
{u {y
M M
(1), u M (2),K, u M ( N )}
(1), y M ( 2),K, y M ( N )}
(M is used to stress the fact that these are Measured data)
The performance index is:

J N () = 1 N 2 (t , ) N t =1
Prediction error
p (t , ) = y M (t ) f (u M (t 1), K , u M (t p 1), y M (t 1), K

K , y M (t m); )
Simulation error
(t 1 / t 2), K s (t , ) = y M (t ) f (u M (t 1), K , u M (t p 1), y (t m / t m 1); ) K, y
33
Model used as a predictor
uM (t )
z 1
uM (t 1) uM (t 2)
z 1
z 1
yM (t )
z 1
uM (t p 1) f yM (t 1) yM (t 2) (t / t 1) y
z 1
z 1
yM (t m)
34
Model used as simulator:
uM (t )
z 1
uM (t 1)
z 1
uM (t 2)
z 1 z 1 z 1
uM (t p 1)
f
(t m / t m 1) y
(t / t 1) y
(t 2 / t 3) y
z 1
(t 1 / t 2) y
35
Problem: in the optimization phase its better to use:

J N () = 1 N 2 p (t , ) N t =1
or
1 N J N () = s (t , ) 2 ? N t =1
If the model target is not simply data-prediction, we would use

s (t , ).
Making use of s (t , ), J N () minimization is much more complex. For this reason, in practice, we optimize by p (t , ) and afterwards we use the final results for a simulation model.
36

Non Linear Systems

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Non Linear Systems

Caricato da

Copyright:

Formati disponibili

Materiale ad uso esclusivo degli studenti del corso. Vietata ogni forma di copia o distribuzione.

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

{u (1), u (2),K, u ( N )} {y (1), y (2),K y ( N )}

Unlike what weve seen up to now, f (; ) is a non-linear function.

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

Linear/dynamic relationship Non-linear/static relationship

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

Its important to distinguish between two classes of problems.

A sufficient condition to solve the interpolation problem is that

We assume that data are noise-disturbed its useless to look

) passing through each {u (i ), y (i )} pair we want for a f (u ,

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

N.B.: If we can interpolate a data set using n < N , we solve a data-

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

Problem: how can we choose a class of non-liner parametric

u U (the mistake made by f is uniformly lower than on all U).

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

The following function classes are universal approximating functions:

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

FEEDFORWARD NEURAL NETWORKS WITH MULTILAYER PERCEPTRONS (NN)

Example of Neural Network with:

1 input; 1 hidden layer with 1 neuron; 1 output.

: , , , are the Neural Network parameters:

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

: is a static non-linear sigmoidal function. Examples:

Both these functions:

Regularity C ; Linearity around x = 0 ; They saturate asymptotically when x .

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

Analytical expression of the Neural Network:

The parameters are:

: neuron bias (or offset) ;

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

Extend the example to a more general case:

Neural Network with:

2 inputs 1 hidden layer/non linear with 3 neurons; 1 output.

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

To write the analytical expression, we define:

(weight matrix of the non-linear layer)

(bias vector of the non-linear layer)

(weight vector of the linear layer).

(bias of the linear layer).

The I/O analytical expression is:

x1 ( x1 ) is applied element by element: x = ( x ) . 2 2

1 hidden layer with h neurons; 1 output.

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

(Graphic) example of Neural Network with:

1 input 2 non-linear layers with 3 and 2 neurons respectively; 1 output:

1st non linear layer

2nd non linear layer

Totally we have 17 parameters.

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

1 output layer (1 output).

Problem: how to find h optimal?

Training data / I; Validation data /V.

Identificazione dei Modelli ed Analisi dei Dati - NON-LINEAR SYSTEMS

PROBLEM: IDENTIFICATION OF THE NN PARAMETERS

J () is the sample variance of the output error (MSE: Mean Square

Error). In case of Neural Networks, J () is a function:

Not quadratic; Not convex.

we need iterative algorithms to minimize it (global minimum not

is known as Back-propagation algorithm. search

Where is the algorithm step and

J () is the gradient at step t. =