Wavenet-ARX Based Nonlinear System Identification

Selami Beyhan*, Musa Alcı**


* Electrical and Elecronics Engineering Department, Ege University,
İzmir,Turkey, (e-mail: selami.beyhan@ege.edu.tr)
** Electrical and Elecronics Engineering Department, Ege University,
İzmir,Turkey, (e-mail: musa.alci@ege.edu.tr)

Abstract: This article incorporates the wavelets functioned neural network (wavenet) and auto-regressive
with exogenous input (ARX) approaches in a parallel combination for nonlinear system identification.
Wavenet composition has recently used identification of nonlinear systems with agreeable results. ARX
structure also well known base for linear system identification in least squares sense. In this work; the
power of nonlinear identification ability of wavenet and the linear identification ability of ARX structure
are merged to have better results to govern identification problem. The new model is tested and compared
for some benchmark problems and then it seen that the resulting model approximates better than the other
well known nonlinear identification models.

optimization method and initial parameters (Haykin 1998). So
1. INTRODUCTION
there is no guarantee about the trained system that is the
System identification and modeling is an essential and correct system model.
important subject for controlling the systems without human
In this study, by considering these drawbacks of the neural
intervention. A mathematical model of the system or artificial
networks, first it is desired to improve the approximation
intelligent model which has same input-output characteristic
capability of neurons by using wavelet functions. Second
with model is necessary to analyze and control the system.
identification ability is also increased with parallel ARX
Modeling which is based on physical laws forms a
structure. Thus combination of the models may have better
mathematical model for the system. But for the identification,
identification ability.
there is no need to use previous knowledge and physical
structure of the system, so they are known as the black-box 2. WAVENET-ARX BASED SYSTEM IDENTIFICATION
identification process (Ljung 1999). If the behavior of the
system is much complicated, it is a big troublesome for the
identification. Because of wide variety of the systems, there is 2.1 ARX Model
not a universal solution to identification of systems. Due to
ARX is one of simpler time series models where the output is
nonlinear behavior, in the beginning of the identification task,
expressed as a linear function of past input and output values.
selection of the identification method is usually the most
difficult part. Appropriate methods can be chosen according
to system behavior and desired goal of identification. A( q ) y (t )  B (q )u (t  nk )  e(t ) (1)
Therefore, it is necessary to use enhanced identification
methods to handle the model that approximates the system Here e(t) is the ‘change’ component used to account for
correctly. uncertainty in the relationship and assumed to be identically
distributed with zero mean and finite variance,  2 [Thum
After the back propagation algorithm developed, neural M.T. 1999]. A and B are polynomials of order na and nb
networks are utilized in many problems as classification, respectively and nk is the time delay in the input.
function approximation, control etc. Even though its power
proved in many areas, neural networks are enhanced with new
theories everyday and compared with the other fields. The A(q)  1  a1q 1  ...  ana q  na
main advantage of the neural network is its capability for the (2)
B(q)  b1  b2 q 1  ...  bnb q  nb 1
self learning from the previously collected input-output data.
The capability of its neuron to make approximation has
In this approach, the model parameters are estimated by the
caused the neural networks used to provide mapping between
least-squares method (Ljung 1999).
input-output data (Nazaruddin et al. 2006). However
identification with neural networks has some difficulties. First
there is not an explicit mathematical model of the system 2.2 Wavelet Function
handled from trained neural network and it is difficult to
analyze. Secondly, in training part there can be many different
Wavelet functions are important for the modeling wavelet
resulting alterable neural network system models for the same
networks because of their different basis characteristics. A
data with same input-output characteristic according to
wavelet function is usually called mother wavelet and its
different dilation and translation parameterized functions are
called daughter wavelets (Burrus et al. 1998). By updating the
translation and scaling parameters different daughter wavelets
are obtained in the algorithms. Some of the wavelet functions
are Gaussian function derivatives. In this work Polynomials
WindOwed with Gaussian (Polywog) wavelet function is used
in wavenet as used before (Lekutai G. et al. 1997) (Edison et
al. 2004), (Nazaruddin et al. 2006) and its function is shown
below.
yˆ (k )  p(1) yˆ1 (k )  p(2) yˆ 2 (k ) (4)
The basic identification scheme with wavenet-ARX model is
shown in figure 3. In the algorithm the model parameters are
updated through minimisation of the mean square error
(MSE). Neurons (wavelet windows) with wavelet parameters
are gradually changed to cover the time-frequency region
occupied by the signal to be identified (Narendra et al. 1990).
Plant

h( )  (3 2   4 ) exp( 2 / 2) (3) u (k ) y (k )

+
yˆ ( k )
Wavenet –ARX
Model
2.3 Wavenet Model -
e(k)
Optimisation 0.5 e 2
Wavenet is the abbreviation of the adaptive neural networks Algorithm
where the wavelet functions are used for activation functions
of the neurons. It is an alternative linear feed-forward neural
network for approximating arbitrary nonlinear functions Figure 3. Wavenet-ARX identification scheme
(Edison et al. 2004). h( ) ’s the activation functions of
neurons and its one hidden layer figure is shown in figure 1.
2.5 Wavenet-ARX Algorithm
tb 
h 1 w
1 The all parameters of the wavenet-ARX model are optimized
 a 
 1  with Levenberg-Marquardth (LM) optimization method by
yˆ (t )
u(t) minimizing the cost function (Alcı, M. 1999),
. +
E  0.5 k 1 e 2 (k ) . So we need jacobians of the parameters.
N
.

Error term between the model output and desired system

t b 
h
 a
k 

w
k output is e(t )  y (t )  yˆ (t ) . For the system output formula is
 k 
given in Eq. 4. y1 is given as follows,
N
Figure 1. Wavenet structure yˆ1 ( k )  w
k 1
k hk ( ) (5)

where   (t  bk ) / ak . For the y2 the ARX input output orders

2.4 Proposed Model
are selected as two and the model is shown in below.
For the identification we proposed the following model that is
shown in figure 2. Here u  k  is the common inputs for both yˆ 2 (k )  c(1)u (k )  c (2)u (k -1)
(6)
models and yˆ(k ) is the final output of the proposed model. It  d (1) y ( k -1)  d (2) y (k - 2)  e(t )
can be thought that the wavenet model identification result is
enough for the identification task but combination with ARX To update the p k , wk , ak , bk , c, d parameters with the LM
model intuitively could have better results. Also this method their jacobians w.r.t cost function are needed. p(1) and
parallelism idea can reduce the number of the neurons used in p(2) parameters are seen in figure 2.
neural network effectively.
The gradients of the parameters w.r.t the cost function are
yˆ 1 ( k ) computed as follows,
Wavenet System p1
E N
e(k )
  e( k ) (7)
u (k ) p j k 1 p j
+
yˆ( k ) The jacobian at qth Iteration is computed as shown below.
yˆ 2 ( k )
ARX Model p2 N
e(k )
Jq  
k 1 p j
(8)

Figure 2. Proposed Model

The parameter update equation is p(k  1)  p(k )  p where
the parameter change is shown for the LM below.
 p  [ J (k )T J (k )   (k ) I n ]1 ( J ( k )T e(k )) (9) 1
Input-Output the Test Model

0.8
The jabobians of the all parameters are formulated in
0.6
equations 9-13 as follows.
0.4

J N
  yˆ1 (k )
0.2
(10)
p1 k 1 0

-0.2

J N
  yˆ 2 (k )
-0.4
(11)
p2 k 1 -0.6

-0.8

J N
  yˆ1 (k ) * h( ) (12) -1
0 50 100 150 200 250
w j k 1

Figure 4. First System input-output Data

J N
h( )
  e(k ) * yˆ1 (k ) * w j * (13)
b j k 1 b j
In this simulation first 100 samples are used for the training
J N
h( ) J and last 100 samples for the testing. In the original model
  yˆ1 (k ) * w j * *  * (14) output first 100 samples are triangular shaped and next 100
a j k 1 b bj
j samples are sinusoidal like shaped. Even so the proposed
model identifies the system well.
The jacobians of ARX model are computed also same
procedure.
Neural-Network Identification
J N
  p (2) * u (k  j  1)
5
(15)
c j k 1
0

J N
  p(2) * y (k  j ) (16) -5
d j k 1
0 20 40 60 80 100 120 140 160 180 200
Wavenet Identification
5
The parameters vector is
0
P  [ p1 , p2 , c1 , c2 , d1 , d 2 , w1 ,.., wn , a1 ..an , b1 ...bn ]T (17)
where n is the number of the neurons. -5
0 20 40 60 80 100 120 140 160 180 200
Wavenet-ARX Identification
5
3. SIMULATION RESULTS
0
In simulations one nonlinear model and two benchmark
problems are used to identify and compare proposed model -5
with other methods. In the simulations for the faith 0 20 40 60 80 100 120 140 160 180 200
comparison neuron numbers or similar structures are used. In
results wavenet model, neural network model, wavenet-ARX Figure 5. Output Comparison for the first test
model and fuzzy-ARMAX models are compared. This fuzzy-
ARMAX model also has parallel structure with proposed
model (Alcı et al. 2007). In the end, there is seen that the
proposed model has effective resulting MSE values. In all Table 1. The resulting MSE values for the fist test
simulations the models are constructed with two inputs one
output system. Inputs are original input signal and other input Model MSE (Test)
signal is the one delayed term of the output signal.
Neural Network
3.228
3.1 Simulation Results-1 (One-Hidden-Layer-5 Neurons)
Wavenet
In the first simulation a nonlinear static mapping test model is 0.455
(One hidden-layer -5 Neurons)
chosen [Alcı, 2008]. This system has a linear input and the
output is the triangular plus sine wave. The input-output data Wavenet-ARX
0.301
is shown in figure 4. (One hidden-layer -5 Neurons)
3.2 Simulation Results-2 y(k )  [ y(k  1) y(k  2) y(k  3)u (k  2)...
Here, Box-Jenkins furnace (Chung et al. 2006) identification *(( y (k  3)  1)  0.5)  u(k )]/(1  y(k  2)2  y(k  3)2 )
is used. First 200 input-output data is used for the training the (19)
system and last 90 input-output data are used for the testing In this simulation totally 1000 samples are used. First 600
and the resulting comparison is shown in Table 2. samples for training the system and last 400 samples are for
the testing. The resulting comparison is shown in Table 3 and
simulations are shown in Fig. 6.
ARX Model Estimation
5
0 ARX Model Estimation
2
-5
0 50 100 150 200 250 300 0
Neural-Network Identification
5 -2
0 100 200 300 400 500 600 700 800 900 1000
0 Neural-Network Identification
2
-5
0 50 100 150 200 250 300 0
Wavenet Identification
5 -2
0 100 200 300 400 500 600 700 800 900 1000
0 Wavenet Identification
2
-5
0 50 100 150 200 250 300 0
Wavenet-ARX Identification
5 -2
0 100 200 300 400 500 600 700 800 900 1000
0 Wavenet-ARX Identification
2
-5
0 50 100 150 200 250 300 0
-2
0 100 200 300 400 500 600 700 800 900 1000
Figure 5. Box-Jenkins Furnace Identification

Figure 6. Dynamic Model Identification

Table 2. Comparison for Box-Jenkins data
Model MSE (Test) Table 3. Results for dynamic model
Neural Network 5.137 Model MSE (Test)
(One-Hidden-Layer-5 Neurons)
Wavenet Neural Network
2.867 7.509
(One hidden-layer -5 Neurons) (One-Hidden-Layer-5 Neurons)
4.470 Wavenet
Fuzzy-ARMAX Model [9] 2.014
(One hidden-layer -5 Neurons)
Wavenet-ARX 0.926 Wavenet-ARX
(One hidden-layer -5 Neurons) 0.6357
(One hidden-layer -5 Neurons)

3.3 Simulation Results-3

To this end, a benchmark problem is employed, the
identification of a dynamical system, most frequently used in
the literature (Gonzalez et al. 2006) for comparing different
learning algorithms and model types.
This is also one of the well known benchmark problem and
similar to the first test example, the test input-output signals
differ than the training signals. In consequent the proposed
model identified better than other methods.

sin(2 k / 25), k  1: 250 4. CONCLUSION

1, k  250 : 500
 Our results indicate the combined use of ARX and wavenet
u(k )  1, k  500 : 750 (18)
modeling schemes can be a feasible alternative to existing
0.3sin(k / 25)  0.1sin(k / 32)
 nonlinear system identification methods. From the simulations
  0.6sin(k /10), k  750 :1000 study, for all tests the proposed model identified the systems
better than the other similar identification methods. In results,
especially the Box-Jenkins identification MSE (test) is
effectively better than other methods. The proposed method
uses the wavelet function approximation in neurons and
paralleled linear ARX model. In addition to these the
proposed model uses less number of neurons.
5. REFERENCES
Alci, M. (1999), “Gradient Based Fuzzy Logic Systems
Depending on Training”, PhD Thesis, SAU.
Alci, M., Asyali M.H. Beyhan S.(2007), “Fuzzy-ARMAX
modeling for nonlinear system identification” Proc. of
First International Conference on Soft Computing
Technologies in Economy, 34-40, November, 19–21
2007, Baku, Azerbaijan.
Alci, M. (2008),, "Fuzzy Rule-Base Driven Orthogonal
Approximation", Neural Computing & Applications,
Volume 17, 5-6:501–507, DOI: 10.1007s00521– 007–
0146–2
Burrus C.S., Gopinath R.A.,Guo H. (1998) Introduction
to Wavelets and Wavelet Transforms A Primer, Prentice
Hall, ISBN 0-13-489600-9
Chung, W., Kim, E.(2006), “A New Two-Phase Approach to
Fuzzy Modeling for Nonlinear Function
Approximation”, IEICE Trans. Inf. & Syst., Vol. E89–D,
No.9, September 2006.
Edison Righeto, Luiz Henrique M. Grassi, João Antonio
Pereira. (2004) “Nonlinear plant Identification by
Wavelets”, ABCM Symposium Series in Mechatronics,
Vol. 1 - pp.392-398
Haykin Simon, (1998) Neural Networks a Comprehensive
Foundation, Prentice Hall Int.
Lekutai G. (1997), “Adaptive Self-Tuning Neuro Wavelet
Network Controllers”, PhD. Thesis, Virginia Polytechnic
Institute and State University, Blacksburg, Virginia,
USA.
Ljung L. (1999), System Identification Theory for the User,
Prentice Hall PTR.
Marcos Angel Gonzalez-Olvera, Yu Tang, (2006), “A new
recurrent neurofuzzy network for identification of
dynamic systems” Science Direct, Fuzzy Sets and
Systems 158 (2007) 1023 – 1035
Narendra, K. S.; Parthasarathy, K., (1990), “Identification and
control of dynamical systems using neural network.” IEEE
Transactions on Neural Networks, v.1, n.1, New York,
USA, pp. 4-27.
Nazaruddin Y.Y. Yuliati. (2006), Wavenet Based Modeling of
Vehicle Suspension System, 1-4244-0136- 4/06/$20.00
'2006 IEEE
Tham M.T. (1996), “Dynamic models for the controller
design”, Newcastle upon Tyne, NE1 7RU, UK. Course
Notes,