MFx – Macroeconomic

Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models
Introduction
 Model Evaluation: Introduction
• The
Th plan
l ffor thi
this module:
d l
• Linear regression model, “important assumptions”, “BLUE” properties (session1: introduction – today!)
• Model properties when one or more of the important assumptions do not hold (session 2)
• Consequences of:
• error term irregularities
g for forecastingg (session 3)
• of breaks for forecasting (session 4)
• Testing model residuals:
• for autocorrelation (session 5)
• testing model residuals for heteroskedasticity and normality (session 6)
• Testing for
f structurall b
breaks
k (session
( 7))
• Testing coefficient restrictions (session 8)
• Practicalities of dealing with error term irregularities and breaks (sessions 9 and 10)
Stages of Econometric Analysis
Economic Theory

Econometric Model Data

Estimation

Model Evaluation/Diagnostic

Is the Model Adequate?

NO YES

Usin the Model

Using Model, e
e.g. for Fore
Forecasting
astin
and Policy Analysis

4
 Linear Regression Model
• Linear
Li regression
i model:
d l
yt  β0  β1 x1t  ...  βk xkt   t
• or in a matrix form
Y  Xβ  

where Y  ( y1 ,..., yT ))', β  ( β0 , β1 ,..., βk ))'

1 x11 ... xk 1 
 
X   ... ... ... ...    ( 1 ,...,  T ))'
1 x ... xkT 
 1T
5
 Linear Regression
g Model: Important
p
Assumptions
• A1: Linear Relationship between Y and X

• A2: E{ t | X }  0

• A3: X s are non-stochastic (e.g., intercept, trend) OR X s are stochastic and E{ X  }  0

/ / /

• A4: Var{ t | X }   2  const .

• A5: E{ t' t s | X }  0 for any t,

t s

• A6:  t | X ~ N 0, 
2

6
 Linear Regression Model: OLS
• Our
O ttask:
k estimate
ti t coefficients, fi d ˆ tto minimize
ffi i t ii.e. find i i i
T
SS 
SSE 
t 1
ˆt2  ˆ ' ˆ  (Y  X ˆ )' (Y  X ˆ ),
)

• “Work-horse”
“W k h ” – ordinary
di lleastt squares, OLS-estimator:
OLS ti t
• f.o.c.:
foc:
SSE
 2 X 'Y  2 X ' X ˆ  0
ˆ
• OLS
OLS-estimator
estimator of the parameters:

ˆ   X ' X 1 X ' Y

7
 Linear Regression Model: “BLUE”
BLUE
• If A1-A5
A1 A5 hold, and assuming X is stochastic, then ̂ is the…
V { | X }    X ' X 
ˆ 1
h minimum variance: Var
• Best, has 2

• Linear, because it is linear in Y : ˆ   X ' X 1 X ' Y

• Unbiased: E{ˆ}  

• Estimator

• If,, additionally,
y, A6 (errors
( y) holds,, then ˆ | X ~ N  , 2 ( X ' X ) 1 
normality)
• … and
d A6 helps
h l a llot when
h we evaluate
l the
h properties
i off an estimated
i d model
d l

8
 Linear Regression Model: “BLUE”
BLUE
(cont…)
• If A1-A6
A1 A6 hold:
hold
1
• We can estimate of the error variance using: ˆ 
T k
2
 t
t
ˆ 2

• ….and therefore the estimator of the std. error of model parameters ̂ :

s.e.{  }  ˆ  X ' X 
ˆ 1
2

• Reduced chances of negative consequences of autocorrelation or heteroskedasticity

• for forecasting performance and estimation of forecasting uncertainty (i.e. confidence bounds)

• discussed further in sessions 2 and 3

• Can proceed with hypothesis testing (e.g., coefficient restrictions or structural breaks)…

• … with better chances of test statistics have “correct” (and known) distributions
9
 Hypothesis Test Example
• Regression: yt  β0  β1 x1t  ...  βk xkt   t
• If  1  0 then x1t can be excluded from the regression

• t-test statistic:
ˆ 1   1 ˆ 1  0 ˆ 1
t ˆ   
s . e .{{ ˆ 1 } s . e .{{ ˆ 1 } s . e .{{ ˆ 1 }

• I.e. unbiased estimates of ˆ1 and s.e.{

{ ˆ1} are important for this test
• Verifying model assumptions/testing model residuals is a step towards:
• reducing the risk of having poor (e.g., biased) parameter estimates (sessions 2 and 3)…

• … and correct model evaluation!!!

10
 Summary
• Assumptions A1 - A6 are important:

• If hold, the OLS-estimator of model coefficients is “BLUE”

• In the
h next session:

• We
W check
h k the
th properties
ti off a regression
i model,
d l when
h some off A1
A1-A6
A6 do
d nott hold…
h ld

11
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models
Violation of Linear Regression
Assumptions
i
Part I
 Model Properties when Some
Assumptions Do Not Hold
• In session 1:
• Linear regression model
• “Important
Important assumptions”
assumptions and “BLUE”
BLUE properties of the OLS-estimator
OLS estimator

• This session
session:
• Model properties when some important assumptions do not hold in the data
• EViews examples

Please have “M7_Data.wf1” opened

Reminder: Linear Regression Model Assumptions
• A1: Linear Relationship between Y and X
• A2: E{ t | X }  0
are stochastic and E{ X  }  0
/ / /
• A3: X s are non-stochastic
non stochastic (e.g.,
( iintercept,
t d) OR
t ttrend) X s
• A4: Var{ t | X }    const .
2

A5 E{  | X }  0 for
• A5: '
t t s
f any t,
t s
• A6 
A6: t | X ~ N 0,  2


• N
Note:
t
• A1 holds by construction
• A2 is easy to ensure – including intercept in your regression will do
• What is some of A3-A6
A3 A6 do not hold?...
4
 What if A3
A3-A6
A6 Do Not Hold?
• A3. Some regressors X are correlated with errors: E{ X  }  0 /

• Example (2 regressors - xt, zt: E{ z t  t }  0 ):

yt   1 xt   2 zt  t

• ˆ1 , ˆ 2 are in general

are, general, BIASED: E { ˆ1 }   1 , E { ˆ 2 }   2

• ˆ1 is UNBIASED only if E { x ' z}  0

• In EViews: Open pagefile “m7

m7_s2_endogen_var_bias
s2 endogen var bias”,, equations “yy_dgp
dgp” and “yy_mod
mod”

5
 What if A3
A3-A6
A6 Do Not Hold (2)?
A4 Errors are autocorrelated: E {  t  s }  0 for some s
/
• A4. t

• Example 1 (regressors are exogenous & AR(1) errors):

y t   1 x t   t andd  t   t 1   t

• ˆ1 is still unbiased: E { ˆ1 }   1

• BUT!!! The estimate  ( X ' X ) 1

ˆ 2
is NOT a consistent estimate of s.e.{  }
1

Open
p p pagefile
g “m7_s2_autocorr_case01”,
_ _ _ equations
q “y_dgp”
y_ gp and “y_mod”
y_

6
 What if A3
A3-A6
A6 Do Not Hold (3)?
A4 Errors are autocorrelated: E{  t  s }  0 for some s
• A4. t
/

• Example 2,
2 lagged dependent variable:

y t   1 zt   2 y t 1   t and  t   t 1   t
• N
Note:
t yt is
i correlated
l t d with
ith εt (by
(b construction),
t ti ) th then yt-1 is
i also
l correlated
l t d with
ith εt-1, but
b t th
then yt-1 is
i
correlated with εt. This case is analogous to violation of A3

• ˆ is BIASED E { ˆ }  

Open pagefile “m7_s2_autocorr_case02”,

“m7 s2 autocorr case02” equations “y
“y_dgp”
dgp” and “y
“y_mod”
mod”

7
 In part II of this session

• What
h if…
f
• A5 - constant error variance (homoskedasticity)
• A6 - normality
li

• …do not hold?

8
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models
Violation of Linear Regression
A
Assumptions
ti
Part II
 What if A3-A6 Do Not Hold ?
Var { t }  const
• A5. Errors are heteroskedastic: Va

β t   t and
yt  βx V { t }  σ t2
d Var{ε

•  t
2
i time
is i d
dependent,
d b
but, in
i general,l off unknown
k ffunctional
i l fform

• ˆ given
i b
by OLS iis still bi d E { ˆ }  
till unbiased:
• BUT!!! ˆ is no longer efficient, i.e. has the smallest variance  ( X ' X ) 1
ˆ 2

• s.e.{{ˆ }  ˆ 2 ( X ' X ) 1 can no longer be used in inference (e

(e.g.,
g testing coefficients)

• since  2 ( X ' X ) 1 is no longer the true s.e.{  }

In EViews: open pagefile “m7_s2_garch”, equations “rb_ols” and “rb_garch”
3
 What if A3-A6 Do Not Hold? (cont
(cont…))
• A6.
A6 Errors are not normal
• Example:
yt  0  1x1t  ...  k xkt  t and
d t is
i nott ~ N( 0, ) 2

• Bad news: some tests statistics (e.g., t and F) do not have standard distributions in
small samples
• Good news:
1
• OLS-Eestimator ̂ 2
and ˆ 
T k  t
ˆt 2 are consistent (converge to their true values
in large samples)

• Can still implement inference: test statistics will have different (but known!) “asymptotic”
distributions (e.g., normal for t-stat.)

4
 Next Steps
Steps…

• Effect of error irregularities

g on forecastingg p
performance ((session #3))

• Structural breaks and their effect on forecasting performance

(session #4)

5
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models
Consequences of error irregularities
f forecasting
for f ti
Part I
Consequences of error irregularities for forecasting

• Consequences of error…
• … autocorrelation

• … heteroskedasticity

• We use examples with…

with
• … generated data (1000 obs.) for autocorrelation cases

• … actual data for heteroskedasticity case

Please have “M7_Data.wf1” opened

 F
Forecasting
ti
• We analyze:
l
• “Correct” (DGP) model – accounts for error irregularity
• “Incorrect”
Incorrect model – neglects error irregularity

• Compare models’ forecasting performance out-of-sample:

• Series of 1
1-period
period ahead forecast: RMSE

• Multi-period
M li i d ahead
h d fforecast: C
Confidence
fid b
bounds
d and
d mean

4
 Forecasting: 1 period ahead, repeated for 50
periods
• 1-period ahead forecasts (Expanding
( window strategy – see module 4))
• Split the sample:
• 950 observations to estimate both “incorrect” and “correct” models

• 50 observations for 1 period ahead out-of-sample forecasts

• Analogy: inflation forecast for 1 quarter ahead, repeated every quarter

“correct”
correct DGP model yt  yˆt [1,950
ytˆ][z1,t951ˆ][z1,t952] tz
t , ˆt[1t,,950ˆ][t1t,951
, 
tt  
ˆ[t11,952
]
11  1t  t1  t
] t 1

1 950 1000

.....
~ ~ ~
“incorrect”
incorrect model t []1z, 951
y t  t [1y, 950
y t t] zt t  t
t []1z, 952
5
 F
Forecasting:
ti multi-period
lti i d fforecastt
• Multi-period forecast
• Split the sample:
• 950 observations to estimate both “incorrect” and “correct” models

• produce mean forecast and confidence bounds over 50 remaining

• Analogy: inflation forecast over the next 8 quarters, starting from now

“correct”
correct DGP model yt  ˆ[1,950] zt  t , t  ˆ[1,950] t 11  t

1 950 1000

.....
~
“incorrect”
incorrect model y t   [1 , 950 ] z t   t
6
Autocorrelation of errors in a linear regression model:
case 1
• True data generating process (generate
( 1000 obs.)) yt :

yt  0 .5 z t   t  t  0.8 t 1   t ,  t ~ N (0,1)

• Assume we neglect
g residual autocorrelation…

7
Autocorrelation of errors
errors, case 1: “incorrect”
incorrect model
and DGP
• …and we estimate an “incorrect” model:

yt   zt   t  t ~ iid (0,   )
2

• … while
h l ““correct”” ((DGP)) model
d l is

yt   zt   t  t   t 11   t

In EViews:
Please, have pagefile “m7_s3_Autocorr_case01” opened
Open and run “m7
m7_s3_autocorr_case01_application.prg
s3 autocorr case01 application.prg”
8
 In the next part
part…

• In Part II of this session

• … One more case of autocorrelation

• … and heteroskedasticity case

 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models
Consequences of error irregularities
f forecasting
for f ti
Part II
Autocorrelation of errors in a linear regression model:
case 2

• True
T f yt (generate
DGP for ( t 1000 obs.):
b )

yt  0.5 yt 11   t  t  0.8 t 1   t ,  t ~ N (0,1)

• we have
h d yt 1 iin th
llagged the RHS and
d AR(1) errors…

• … which we neglect…

3
Autocorrelation of errors
errors, case 2: “incorrect”
incorrect model
andd DGP
• … and estimate the “incorrect” model…

y t   y t 1   t  t ~ iid (0,   )
2

• … … while “correct” ((DGP)) model is

y t   y t 1   t  t   t 1   t
In Eviews:
Please,, have p
pagefile
g “m7_s3_Autocorr_case02” opened
p
Open and run “m7
m7_s3_autocorr_case02_application.prg
s3 autocorr case02 application prg”

4
Heteroskedasticity of errors in a linear regression
model
Th process yt h
• The has h
heteroskedastic
t k d ti errors:

yt  1 yt 1   2 yt  2   t  t ~ N (0, t2 )

• … i.e. the variance is time-dependent:  t

2

5
Heteroskedasticity of errors: “incorrect”
incorrect model and
DGP
• … and estimate an “incorrect” OLS model, assuming constant variance…

yt  1 yt 1   2 yt  2   t  t ~ iid (0,   )
2

• … while “correct” model with GARCH errors is

yt  1 yt 1   2 yt  2   t   0   
t
2 2
1 t 1   2 2
t 1

6
 Heteroskedasticity of errors: forecasting
• Next example: actual data for monthly total return UK bond index

• 1979:1
1979 1 – 2006:12
2006 12

In EViews:

• Please,
Please have pagefile “m7
m7_s3_garch
s3 garch” opened

• Open and run “m7_s3_garch_application.prg”

7
 Consequences of Error Irregularities for
Forecasting: Summary
• Autocorrelation may cause:
• Worse forecasting performance in terms of the mean forecast
• Incorrectly estimated confidence bounds,…
• Biased up or down due to the biased mean forecast
• T narrow or too
Too t wide
id

• … i.e.
i e incorrectly estimated forecast uncertainty
• Heteroskedasticity may cause:
• No effect on the mean forecast
• Incorrectly estimated confidence bounds,…
• Too narrow or wide
• … i.e. incorrectly estimated forecast uncertainty
8
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models

Consequences of Breaks for Forecasting

 C
Consequences off Breaks
B k ffor Forecasting
F ti

• In
I this
thi session:
i
• Breaks in the intercept OR slope parameters of a linear regression model…

• … and consequences for forecasting

Please have “M7

M7_Data.wf1”
ata.wf opened
 Introduction of Breaks in Intercept and Slope
Parameters
• A model might break in the estimation sample [1, ...,T ] at point T *, 1  T *  T

• … in
i th
the iintercept:
t t
yt   0  d T  1 xt   t
 
* *
0, T T
• … in the slope: dT  
*

1, T  T
*

yt   0  (  1  d T ) z t   t
*

• i.e.,
i shift by  att T *
hift b
• and new parameter value is 0   (intercept) and 1   (slope)
4
 B k and
Breaks dF
Forecasting
ti
• We
W generate data b ) yt with
d (1000 obs.) i hab b T *  800
breakk at obs.
• Evaluate the effect of a break in the estimation sample
p on forecastingg
• We analyze:
• “Correct” (DGP) model – accounts for a break

• “Incorrect” model – neglects a break

• Compare models’
models forecasting performance (as in the previous session):
• Series of 1-period ahead forecast: RMSE

• Multi-period ahead forecast: Confidence bounds and mean

5
 Break in the Intercept: DGP
• Data
ata generating
ge e at g process ( 000 obs.) yt :
p ocess (1000
0, T  800
yt  1  1.5d T  0.5 yt 1   t
* dT  
1, T  800
*

 t ~ N (0,2.25 )

6
Break in the Intercept: “Incorrect” Model and DGP
• We do not realize there is a break …
• E.g.,
E g we forget to test for breaks (will do later!)

• …and
and estimate an “incorrect”
incorrect model:
yt   0  1 yt 1   t

• … while “correct” (i.e., DGP) that accounts for the break:

yt   0  d T  1 yt 1   t
*

In EViews:
Please have pagefile “M7
Please, M7_S4_str_break_interc
S4 str break interc” opened
Use “M7
M7_S4_break_intercept_application.prg
S4 break intercept application prg”
7
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models

Consequences of Breaks for Forecasting

Part II
 Break in the Slope Parameter of a Linear
Regression Model
• Again: we do not realize there is a break in the estimation sample
• E.g., could test for breaks, but we did not…

• “Incorrect”
Incorrect model:
yt   1 z t   t
• Compare forecasts from “incorrect” and “correct” (i.e. DGP), that accounts
for the break in the slope:

yt   zt  d T zt   t *

3
 Break in the Slope: two cases
• Data generating process (1000 obs.) yt , two cases:
• p
positive shift

y t  0 .5 z t  0 .4 d T z t   t
*

• negative shift
0, T  800
y t  0 .5 z t  0 .4 d T z t   t  t ~ N (0,1) dT  
1, T  800
* *

• In EViews:
Please, have p
pagefile
g “M7_S4_str_break_slope”
_ _ _ _ p opened
p
Use “M7_SS4_b
break
ea _sslope
ope_app
application.prg”
cat o p g
4
 Consequences of Breaks for Forecasting:
Summary
• Breaks may cause:
• Worse forecasting performance in terms of the mean forecast

• Incorrectly estimated confidence bounds,…

• Biased up or down due to the biased mean forecast

• Too narrow or too wide

• … i.e.
i iincorrectly
tl estimated
ti t d fforecastt uncertainty
t i t
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models
Testingg Residuals for Autocorrelation
 Testing Residuals for Autocorrelation
• Portmanteau Q-test of no autocorrelation

• LM-test
LM test of no autocorrelation

• Quick Reminder: consequences of autocorrelation (see sessions 2and 3 for details)

Please, have “M7_Data.wf1”pagefile “M7_S5_Exmpl_Belarus_infl” opened

 Q Tests of Autocorrelation: Ingredients
Q-Tests
• Regression ode yt  β0  β1 x1t  ...  βk xkt   t
eg ess o model

• Residuals ˆt  yt  βˆ0  βˆ1 x1t  ...  βˆk xkt

• Residual autocorrelation coeff s: r (i )  corr (ˆt , ˆt i ) , i  1,..., m

coeff’s:

• Portmanteau Q-test:

• check if autocorrelation coeffs are all 0 (jointly not significant)

• i.e. H0: r (1)  0, r (2)  0,..., r (m)  0 no autocorrelation up to order m

4
 Q Tests of Autocorrelation: Intuition
Q-Tests
• Test statistic
T (T  2 ) m 2
Q (m )  
T  k i 1
r (i ) ~  ( m )
2

• Intuition:
• if ˆt are autocorrelated,  r (i )' s should be “large”
•  and Q(m) is “large”
• If Q((m) is “large enough” – larger than a “critical value” Q ( m)  QCV ( m)
•  H0: r (1)  0, r ( 2)  0,..., r ( m )  0 is rejected
j
So  t are autocorrelated indeed!
• So,

EViews example: pagefile “M7_S5_Exmpl_Belarus_infl”

5
 LM-Tests of Autocorrelation: Ingredients
• Residuals: ˆt  yt  βˆ0  βˆ1 x1t  ...  βˆ k xkt

• Regress residuals on xit and lagged residuals ˆt i (auxiliary regression):

ˆt   1   1 x1t  ...   k xkt  1ˆt 1  ...   mˆt  m   t (*)

• If no systematic f i iin ˆt  no autocorrelation

i iinformation l i  regression
i coefficients
ffi i are
insignificant
• i.e., we test H0:
1  0, 2  0,...,  m  0

6
 LM-Tests
LM Tests of Autocorrelation: Intuition
• Get
G the 2
h R off our auxiliary
ili regression
i
ˆt   1   1 x1t  ...   k xkt  1ˆt 1  ...   mˆt  m   t
• Construct a statistic: T  R ~  (m
2 2
(m) ,
• T – number of observations in the sample,
p , R2 is a coefficient of determination
• Intuition:
• if some of the ˆt  i in regression (*) are correlated with ˆt
•  then R is “large”  T  R is “large”
2 2

• if T  R iis llarger th
2
than a ““critical”
iti l” value
l
•  i.e.,
i e regression captures systematic information in residuals!
•  H0: 1  0, 2  0,...,  m  0 is rejected
•  t are autocorrelated up to m lags

EViews example: pagefile “M7

M7_S5_Exmpl_Belarus_infl
S5 Exmpl Belarus infl”
7
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models
Testing Residuals for
Heteroskedasticity
k d i i and d Normality
li
 Testing Residuals for Heteroskedasticity and
Normality
• Q-test
• A family of LM-tests
LM tests
• Jarque-Bera
J B test off normality
li

• Quick Reminder: consequences of heteroskedasticity (see sessions 2 and 3 for

details)

• Please have “M7_Data.wf1” pagefile “M7_S6_Interbank_Ukraine” opened

 Q Tests of Heteroskedasticity: Ingredients
Q-Tests
• Regression ode yt  β0  β1 x1t  ...  βk xkt   t
eg ess o model

• Residuals Squared residuals

ˆt  yt  βˆ0  βˆ1 x1t  ...  βˆk xkt ˆt
2

• Q-test: similar to autocorrelation test

• Calculate r (i )  corr ( t ,  t i ) , for i  1,..., m

ˆ 2
ˆ 2

• Check if H0: r (1)  0, r ( 2)  0,..., r ( m )  0 , i.e. no heteroskedastisity up to order m

4
 Q Tests of Heteroskedasticity: Intuition
Q-Tests
• Familiar Test statistic
T (T  2 ) m 2
Q (m )  
T  k i 1
r (i ) ~  ( m )
2

• Intuition:
• if ˆt2 are autocorrelated, i.e. residuals are heteroskedastic
•  r (i )' s should be “large”
•  and Q(m) is “large”
• If Q((m) is “large
g enough” g than a “critical value” Q ( m)  QCV ( m)
g – larger
•  H0: r (1)  r ( 2)  ...  r ( m )  0 is rejected
So squares of residuals are autocorrelated 
• So, t are heteroskedastic indeed
EViews example: pagefile “M7
M7_S6_Interbank_Ukraine
S6 Interbank Ukraine”
5
 LM Tests of Heteroskedasticity: Ingredients
LM-Tests
• LM-style
y tests:
• investigate an auxiliary regression for  t
ˆ 2

ˆ 2
• if t change
h systematically
i ll (d(depend i bl )   t are
d on some variables)
heteroskedastic

• Example:
E l ARCH LM
LM-test
• Regress  t on  t 1 :
ˆ 2
ˆ 2

ˆ   0   ˆ  ...   mˆ
t
2 2
1 t 1
1
2
t m
 t
• H0 of homoskedasticity:
1  0,  2  0,...,  m  0
6
 LM Tests of Heteroskedasticity: Intuition
LM-Tests
• Get the of the regression
R 2

ˆ  1   ˆ  ...   mˆ
t
2 2
1 t 1
1
2
t m
 t

• Test statistic: T  R ~  (m) , T is the sample size

2 2

• Intuition:
• if ˆt in regression are significantly related to
2
 t i
ˆ 2

“large””  T  R2 is
2
•  R is
i “l i “l
“large””
• if T  R2 is also larger than a “critical”
critical value
•  H0: 1  0,  2  0,...,  m  0 rejected!
•   t are heteroskedastic up to m lags

EViews example: pagefile “M7_S6_Interbank_Ukraine” 7

 Test for Normality
• Residuals: ˆt  yt  βˆ0  βˆ1 x1t  ...  βˆ k xkt


4
1  
Tˆ 
3
ˆ 1  ˆt 
T
• skewness
k ˆ t
t i K   
S     , kkurtosis
T t 1  ˆ  T t 1  ˆ 
• Jarque-Bera-test
q statistic:
JB 
T  ˆ 2
S 
Kˆ  3  ~ 
2
2
2 ,
6  4 

• ˆt are normal if not skewed (symmetric, Ŝˆ ~ 0 ) and not leptokurtic ( K̂ˆ ~ 3 )  JB ~ 0

8
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models
Testingg for Structural Breaks
 Testing for Structural Breaks
• The main question: Have the true parameters of the model changed during
p p
estimation sample period?
• This session:
• Recursive coefficients
• Chow
Ch b breakpoint
k i t ttestt
• Quandt-Andrews
Q dt A d test
t t

• Quick Reminder: consequences of breaks for forecasting (see session 4 for details)

Please have “M7_Data.wf1” pagefile “M7_S7_Exmpl_Russia_infl” opened

 Recursive Coefficients
• Regression ode yt  β0  β1 x1t  ...  βk xkt   t
eg ess o model

• Idea: if there is a break in the true parameters, then estimated coefficients

change “wildly” over the estimation sample

• Ingredients: recursively estimated coefficients

y t  ytˆ0,[1yˆ ˆ ˆ  ˆ 
t x ˆ ...
 yx t 

 ˆˆ  ...

...
x  ˆ x  ˆˆ xt xxt  ... t  ˆk ,[1,T ] x k   t
,T 0],[1,T 0,1[]11,T
0 0 ,[1,T2 ]]1,[11,T 11,[]1,T1  2 ] 1k0,[,[11,T,T ]k ,[1k,T k1,][11,[,T1k,T 2] ] 1k
0 0 0 0 0 0 0

1 T0 T0+1 T0+2 T
.......................

EViews example: pagefile “M7_S7_Exmpl_Russia_infl”

4
 Chow Breakpoint Test: Ingredients
• Restricted regression - no breaks
yt  β0  β1 x1t  ...  β k xkt   t
• Unrestricted regression – breaks at T1,…,Tm
y t   00   01 x1t  ...   0 k x ktk   0 t , t  1,..., T1
y t   10   11 x1t  ...   1 k x ktk   1t , t  T1  1,..., T2
...
y t   m 0   m 1 x1t  ...   mk x kt   mt , t  Tm  1,..., T
• The test compares:
• how well restricted (no breaks) and unrestricted (breaks) models fit the data

• Caveat: we need to specify dates T1,…,TTm that we suspect

5
 Chow Breakpoint Test: Intuition
• H0: no breaks
• F-statistic:
F
ˆ 'ˆ
 ˆU 'ˆU  / mk
R R
~ F mk , T  m  1k 
ˆU 'ˆU / T  m  1k 
ˆU ' ˆU   ˆi ' ˆi , i indicates a subsample between breaks i and i  1
i  0 to m

• Intuition:
• If the sum of squared errors for the U-model is substantially smaller, than that for
the
h R-modeld l ˆR ' ˆR  ˆU ' ˆU
•  which makes the F-stat.
F stat is “larger”
larger than a critical value
value,
•  there are breaks indeed and R R-model
model accounts for them!
•  the H0 of no breaks is rejected
j

EViews example: pagefile “M7

M7_S7_Exmpl_Russia_infl
S7 Exmpl Russia infl” 6
 Quandt-Andrews Breakpoint Test
• Testing Sample is trimmed in the beginning and in the end:
• default value is 15% (7.5% on each end)

1 t1 t2 T

7.5% ................... 7.5%

Ft1  Ft1 1 Ft2 
• Sequential application of the Chow breakpoint test to each date
• F-statistics are calculated as in Chow (see slide 5 above)
• and 3 test statistics are calculated based on FF’ss
1 t 1  1 t
MaxF  max F t  ExpF  ln  exp( F t )  AveF   F t 
2 2

t1  t  t 2
 k t t 2 1  k t t 1

• EViews example: pagefile “M7

M7_S7_Exmpl_Russia_infl
S7 Exmpl Russia infl”
7
Optional: Chow Forecast Test
 Chow Forecast Test: Ingredients
• Chow Forecast Test
• Full-Sample
Full Sample Model:
yt  β0  β1 x1t  ...  βk xkt   FS ,t , t  1,..., T
• Model estimated over a smaller sample T1  T :
~ ~ ~
y t   0   1 x1 t  ...   k x kt   SS ,t , t  1,..., T1

• The test compares in-sample residuals of the 2 models

• H0: there is no break
9
 Chow Forecast Test: Intuition
• F-statistic:
F
ˆ 'ˆ FS FS
 ˆSS ' ˆSS  /(T  T1  1)
 F T  T1  1, T1  k 
ˆSS ' ˆSS / T1  k 
where ˆ FS ,t  y t  ˆ1 x1t  ...  ˆ k x kt for t  1,..., T

and
~
ˆ SS ,t  y t  ˆ1 x1t  ...  ~ˆ k x kt for t  1,..., T1

• Intuition:
• if no break  the fit of FS-model and SS-model are of the same degree of accuracy
•  F-stat. is small
• If there is a break at T1  the fit of the FS-model should deteriorate after T1
•  F-stat.
F t t becomes
b “l
“large”,
” andd if it exceeds
d th
the critical
iti l value
l
•  the H0 of no breaks is rejected

EViews example: pagefile “M7_S7_Exmpl_Russia_infl” 10

 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models

Coefficient Testing
 Coefficient Testing
• Main questions:
• Should some coefficients in the regression be restricted to particular (e
(e.g.,
g economic-
theory-consistent) values?
• Should some regressors be excluded (i.e., respective coefficients restricted to 0’s) from
our model?
• In this session:
• t-test of single coefficient
• Wald test
• Redundant variables test

Please have “M7_Data.wf1”, pagefile “M7_S8_exmpl_brazil_pstar” opened

 t test of Restriction on a Single Coefficient
t-test
• Regression model yt  β0  β1 x1t  ...  βk xkt   t

• Test if βi can be restricted to a value β

• H0: βi  β

• t-statistic:
i i
ˆ i  
t ˆ 
i
s . e .(( ˆ i )

4
 tt-test:
test: Intuition
• t-statistic:
i i
ˆ i  
t ˆ 
i
s . e .(( ˆ i )
• Intuition:
• if restriction is not correct  β
β̂i and β are different values
•  βi  β is “large”
g  t̂ is “large”
g
i

• if t̂ˆ iss larger

a ge than
t a a critical
c t ca value
a ue
i

•  H0 is rejected

• Particular case: testing if xit can be excluded from the model

H 0 : βi  β  0
EViews Example: pagefile “M7_S8_exmpl_brazil_pstar” 5
 Wald test of Multiple Linear Restrictions
• Regression model: yt  β0  β1 x1t  ...  βk xkt   t
• Linear restrictions on coefficients: R  r
• R : ( m  k ) - matrix
restrictions k – number of coefficients in vector β  [ β0 , β1 ,..., βk ]]'
• m – number of restrictions,

• r : ( m  1) - vector
• Example: test β1  0, β2  β3 , β4  2
• Note: β2  β3  β2  β3  0
 0 1 0 0 0 ... 0 0 0 
R   0 0 1  1 ... ... 0 0, r  0 
   
 0 0 0 1 ... ... 0 0    2 
  [  0 ,  1 ,  2 ,  3 ,  4 ,...,  k ]]' 6
 Wald Test: Intuition
• H0: R  r
• Wald Test Statistic

   
A
W  R ˆ  r ˆ R  X ' X  R ' R   r   m ,
' 1
2 1
ˆ 2

• Intuition:
• if restrictions R  r are correct
•  ( R ˆ  r ) ~ 0  then W ~ 0
• i.e., if W  WCV (not statistically significant)
•  the H0 is not rejected (can impose the restrictions)

EViews Example: pagefile “M7_S8_exmpl_brazil_pstar”

7
 Redundant Variables Test: Ingredients
• Regression: Residuals:
yt  β0  β1 x1t  ...  βk xkt   t ˆt  yt  βˆ0  βˆ1 x1t  ...  βˆk xkt
 
• For ˆt ~ N 0, the density of distribution of yt is:
2

1 ˆt2
f  y t | X ,   exp{  }
2 ˆ 2
2ˆ 2

where  is the set of estimated parameter values

• And the log-likelihood function is:

 1 t 
ˆ 2

ln L     ln  f ( y t | X , )    ln 
T T
exp{  }
t 1 t 1  2ˆ
2
2ˆ 2

• Note: smaller ̂t (i.e.,

(i e “better”
better fit) lead to a larger value of the log-likelihood
log likelihood

• this
h property is key
k for
f the
h test… 8
 Redundant Variables Tests: Intuition
• Unrestricted model:
yt  β0  β1 x1t  ...  βn xnt  ...  β j x jt  ...  βk xkt   t
• Restricted one: xnt , x jt are dropped
yt  β0  β1 x1t  ...  βk xkt   t
• H0: xnt , xjt are redundant
• Test
T t statistic
t ti ti - the
th Lik
Likelihood
lih d RRatio:
ti

LR  2ln LR  ln LU  
 r,A 2

where r is the number of redundant variables

• Intuition:
• if dropping variables (going from U to R)
•  substantial decrease of LU to LR  LR is “large”
•  if LR LRCV  the R-model fit deteriorates significantly
•  reject H0  variables should not be dropped from the model 9
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models

Dealing with Error Irregularities

Part I
 Dealing with Error Irregularities

• A few practical approaches to deal with:

• Residual autocorrelation

• Residual heteroskedasticity

• Please,
Please have “M7
M7_data.wf1
data.wf1” opened
 Residual Autocorrelation
• What if tests suggest that residuals are autocorrelated?
• Recall 2 cases:
cases
• No lagged dependent variables in the model and t are autocorrelated : t  1t1  ... mtm  t
• In our example: AR(1)
yt   zt   t  t   t 1   t

• A model includes lagged

gg dependent
p yt and t are autocorrelated:
• e.g., yt 1 and AR(1) t

y t   1 y t 1   2 z t   t  t   t 1   t

Reminder:
d check
h k sessions 2 and
d 3 ffor the
h ddiscussion off consequences off autocorrelation
l

4
 Dealing with Autocorrelation: Case 1
• No lagged variables in the model and t are autocorrelated:
• In general add lags zt 1 , yt 1 , zt 2 , yt 2 ,...

• In our case of AR(1)-errors: add lags zt 1 , yt 1

yt   zt   t y t   1 z t   2 y t 1   3 z t 1   t

This is the rho

coefficient in
 t  ˆ t 1  t
i.e.,
ˆ  ˆ2

5
 Dealing with Autocorrelation: Case 1 (cont
(cont…))
• Alternatively, add AR-terms in the regression and estimate using MLE:
• In general: AR(1), AR(2),…, AR(m) This is the rho
• In our case: AR(1) coefficient in
 t  ˆ t 11   t

In EViews: Open “m7_s9 - autocorr_case01-dealing_irreg.prg” and re-specify the “incorrect” model 6

 Dealing with Autocorrelation: Case 1 (cont…)

• If our aim is to check significance

g of a regression:
g HAC estimate of the error variance

• Input for the estimate of the coefficient standard error

• Please, have pagefile “M7_S9_russia_infl”, equation “infl_rus” opened

7
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models

Dealing with Error Irregularities

Part II
Autocorrelation, Lagged Dependent Variable in the
RHS
• 2 A model includes lagged dependent variable yt and t are autocorrelated:
Case 2:
• In our example, yt 1 and AR(1)  t

y t   1 y t 1   2 z t   t  t   t 1   t

Reminder: revisit sessions 2 and 3 for the discussion of consequences of autocorrelation

3
 Dealing with Autocorrelation: Case 2
• Errors are autocorrelated, lagged variables of the same order are in the RHS of the equation:
• Add lags: in general – yt2 , zt 1, yt 3, zt2 ,...
• Add llags: iin our case – yt2 , zt1
y t   1 y t 1   2 z t   t yt   1 yt 1   2 zt   3 yt  2   4 zt 1   t
Consistent
estimates of 
and β
β’s can be
recovered from:

  1    c (1)

  1     c ( 3)
   c (2)
 2

In EViews: open “M7_S9-autocorr_case02-dealing_irreg.prg” and re-specify the “incorrect” model 4

 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models

Dealing with Error Irregularities

Part III
 Residual Heteroskedasticity

• Heteroskedasticity:
Heteroskedasticity

yt   0  1 x1t   2 x1t  ...   k xkt   t , Var { t }   t

2

• OLS estimator still produces unbiased/consistent estimates of 

OLS-estimator
• … but
b t the
th constant
t t variance
i estimate
ti t isi nott correctt

3
 Dealing with Heteroskedasticity
• Model the variance explicitly,
p y, estimate the model usingg Maximum Likelihood
Estimator (MLE):

• e g as an ARCH/GARCH process – is only one of many possible specifications!

e.g.,

• Use Akaike information criterion to select p and q orders in GARCH(p,q)

• Open pagefile
O fil “M7_S9_garch_no_bias”,
“M7 S9 h bi ” check
h k objects
bj t named
d
p [ ] q [ ]
p_[…]_q_[…]

4
 Dealing with heteroskedasticity (cont
(cont…))
q 1 2
p
1 3.424123 3.428592
2 3.429648 3.434166

• GARCH(1,1) model… (… it was also used as a DGP in this example):

yt  1 yt 1   2 yt  2   t   0   
t
2 2
1 t 1
  2 2
t 1

In EViews: Open “m7_s9 - garch-dealing_irreg.prg” and specify the corrected

“incorrect” model
5
 Dealing with heteroskedasticity (cont
(cont…))

• If the
h aim
i is
i to check
h k significance
i ifi off our regression:
i

• use the HAC estimate of the error variance to correct the estimates of
standard errors…

• …White estimate (if errors are heteroskedastic) or Newey-West (if both,

heteroskedastic and autocorrelated)

In EViews: have pagefile “M7_S9_garch_no_bias”, equation “rb_ols” opened

6
 Summary
• Testing residuals for autocorrelation and heteroskedasticity is an important part of
model
d l di
diagnostics
ti
• Autocorrelation may lead to:
• biased estimates of model coefficients or incorrectly estimated error variance

• Negative effect on the model’s forecasting performance and incorrectly estimated forecast uncertainty
(confidence bounds)

• Heteroskedasticity,
Heteroskedasticity if not accounted for:
• Does not affect
ff model’s
d l’ fforecasting performance
f iff terms off RMSE

• … but
b may llead
d to incorrectly
l estimated
d uncertainty (confidence
( fd b
bounds)
d)

• A number of practical approaches are available to deal with residual irregularities:

• Generally, some sort of model reformulation is required
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models

Dealing with Structural Breaks

Part I
 Dealing with Structural Breaks
• Practicalities of dealing with:
• Structural breaks in the intercept and slope parameters

• Single and multiple breaks

• Please,
Please have “M7
M7_Data.wf1
Data.wf1” opened
 Single Break in the Intercept
• The intercept may break in the estimation sample [1, ... ,T ] in N points (one or more)
yt   0  1 x1t  ...   k xkt   t , t  1,..., T1
yt  (  0   1 )  1 x1t  ...   k xkt   t , t  T1  1,,...,, T2
...
yt  (  0   N )  1 x1t  ...   k xkt   t , t  TN  1,..., T

• i.e., shift by  i at Ti and new intercept value is  0   i

• We do not know the date of a break – we test for it
• We first consider the case of only one break in the intercept
In EViews: select p
pagefile
g “M7_S10_str_break_interc”,
_ _ _ _ equation
q “y_mod”
y_ and Quandt-
Andrews test to estimate a date of a single break
4
Dealing with a Single Break in the Intercept (cont
(cont…))
• We correct the “incorrect” model for a break in the intercept by adding a shift dummy:

yt   0    dum _ 802  1 yt 1   t

• In EViews: Use “m7_s10 – generate_shift_dummy.prg” to create a shift dummy for a break in “802”

• In EViews: Open “m7_s10 - single_break_intercept-dealing_irreg.prg”:

• Correct the “incorrect” model
• Compare 3 model “Correct (DGP)”, “Incorrect” (does not account for a break) and “Corrected”
(includes a shift dummy corresponding to the estimated moment of a break)

5
 Single Break in the Slope
• It is quite similar to how we treated breaks in the slope…
• Break in the slope, e.g. 1:
yt   0  1 x1t  ...   k xkt   t , t  1,..., T1
yt   0  ( 1   1 ) x1t  ...   k xkt   t , t  T1  1,,...,, T2
...
yt   0  ( 1   N ) x1t  ...   k xkt   t , t  TN  1,,...,, T

• i.e.,
i shift by  i att Ti and
hift b d new slope
l l iis 1   i
value
• We do not know the date of a break and test for it
• We consider the case of a single break in the slope
In EViews: select pagefile “M7_S10_str_break_slope” equation “y_mod” and Quandt-
Andrews
d test to estimate a d
date off a single
l bbreakk 6
Dealing with a Single Break in the Slope (cont
(cont…))

• We correct the “incorrect” model for a break in the slope by adding a cross-
product of a shift dummy and zt :

yt   0  1  zt    dum _ 794  zt   t

• In EViews: Open
p “m7_s10
_ - single_break_slope-dealing_irreg.prg”:
g _ _ p g_ g p g
• Correct the “incorrect” model
• Compare 3 model “Correct (DGP)”, “Incorrect” (does not account for a break) and “Corrected”
(include a product of z and shift dummy corresponding to the estimated moment of a break)

7
 MFx – Macroeconomic
Forecasting

IMFx

This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF
I tit t ffor C
Institute Capacity
it D
Development
l t (ICD) courses. A
Any reuse requires
i ththe permission
i i off th
the ICD
ICD.
EViews® is a trademark of IHS Global Inc.
Evaluating Regression Models

Dealingg with Structural Breaks

Part II
 Dealing with Multiple Breaks
• Previously:
i l
• Single break
• Either
Eith slope
l OR iintercept
t t parameters
t may break
b k
• More
M generall case:
• Multiple
M lti l bbreaks
k
• Both,
Both slope and intercept parameters
parameters, may break
• Again – we do not know dates of multiple breaks

I EViews:
In EVi S l pagefile
Select fil “M7_S10_mult_str_breaks”,
“M7 S10 l b k ” equation
i “y_mod”
“ d”
• Bai-Perron
i tests off LL+1
1 vs. L sequentially
i ll ddetermined
i db breaks
k to estimate
i d
dates off multiple
li l b breaks
k

3
 Dealing with Multiple Breaks (cont
(cont…))
• We correct the “incorrect” model for by accounting for multiple breaks
• We use shift dummies:

yt  0  1 yt 1  ( 01  11 yt 1 )  dum _ 417  ...

..  ( 02  12 yt 1 )  dum
d _ 627  ( 02  13 yt 1 )  dum
d _ 771   t

In EViews:
• Use “m7_s10 - generate_shift_dummy.prg” to create dummies for breaks at “417”, “627” and
“771”
• Open “m7
m7_s10
s10 - multiple_str_breaks-dealing_irreg.prg
multiple str breaks dealing irreg prg”::
• correct the “incorrect” model and run the script
p

4
 Summary
• Testing for beaks is an important part of model diagnostics

• Single or multiple beaks in the estimation sample are possible

• If not accounted for,

for breaks may lead to:
to

• biased estimates of model coefficients

• Negative effect on the model

model’ss forecasting performance (see session 4)

• We can use shift dummies to account for breaks in the intercept or slope
coefficients of a liner regression model