Review

ECE 531: Detection and Estimation Theory
Natasha Devroye
devroye@ece.uic.edu
http://www.ece.uic.edu/~devroye
Spring 2010
Course outline
ECE 531: Detection and Estimation
University of Illinois at Chicago, ECE
Spring 2010
Instructor: Natasha Devroye, devroye@ece.uic.edu
Course coordinates: Tuesday, Thursday from 2-3:15pm in TH 208 (Taft Hall).
Ofce hours: Monday from 4-5:30pm in SEO 1039, or by appointment
Welcome to ECE 531! This course is a graduate-level introduction to detection and estimation theory,
whose goal is to extract information from signals in noise. A solid background in probability and
some knowledge of signal processing is needed.
Course Textbook: Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by
Steven M. Kay, Prentice Hall, 1993 and (possibly) Fundamentals of Statistical Signal Processing,
Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
Other useful references:
Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, II, III, IV
H. Vincent Poor, Introduction to Signal Detection and Estimation
Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, and Time
Series Analysis
Carl Helstrom, Elements of Signal Detection and Estimation. It's out of print, so here's my pdf copy.
Notes: I will follow the course textbooks fairly closely, using a mixture of slides (highlighting the
main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I
will post a pdf version of the slides as they become ready here, but the derivations will be given in
class only.
Topics: Estimation Theory:
General Minimum Variance Unbiased Estimation, Ch.2, 5
Cramer-Rao Lower Bound, Ch.3
Linear Models+Unbiased Estimators, Ch.4, 6
Maximum Likelihood Estimation, Ch.7
Least squares estimation, Ch.8
Bayesian Estimation, Ch.10-12
Detection Theory:
Statistical Detection Theory, Ch.3
Deterministic Signals, Ch.4
Random Signals, Ch.5
Statistical Detection Theory 2, Ch.6
Non-parametric and robust detection
Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).
http://www.ece.uic.edu/~devroye/courses/ECE531/
1 of 3 1/11/10 8:50 PM
Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by Steven M. Kay, Prentice Hall, 1993
Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
Spring 2010
Series Analysis
class only.
Detection Theory:
1 of 3 1/11/10 8:50 PM
Problems - attack!
Problems - attack!
Problems - attack!
Problem 2.5
In the binary communication system shown in Fig. 5-1, messages m = 0 and m = 1
occur with a priori probabilities 1/4 and 3/4 respectively. Suppose that we observe
r ,
r = n + m,
where n is a continuous valued random variable with the pdf shown in Fig. 5-2. The
random variable n is statistically independent of whether message m = 0 or m = 1
occurs.
source
n
r
+
m
receiver m
^
m=0 or 1
message
received
signal
m=0 or 1
^
Figure 5-1
p (n)
n
n
-3/4 3/4
2/3
Figure 5-2
(a) Find the minimum probability of error detector, and compute the associated
probability of error, Pr [ m = m].
(b) (practice) Suppose that the receiver does not know the a priori probabilities,
so it decides to use a maximum likelihood (ML) detector. Find the ML detector
and the associated probability of error. Is the ML detector unique? Justify your
answer. If your answer is no, nd a dierent ML receiver and the associated
probability of error.
Problem 2.6
Consider the binary discrete-time communication system shown in Fig. 6-1. Assume
that m = 0 and m = 1 are equally likely to occur. Under hypothesis Hm (m = 0, 1),
the received signal r [n] is given by
r [n] = ym[n] + w[n] = sm[n] h[n] + w[n], for all n,
where denotes convolution, and where the w[n]s are zero-mean statistically in-
dependent Gaussian random variables with variance
2
. The two signals, s0[n] and
4
Problems - attack!
Massachusetts Institute of Technology
Department of Electrical Engineering and Computer Science
6.432 Stochastic Processes, Detection and Estimation
Problem Set 6
Spring 2004
Issued: Tuesday, March 16, 2004 Due: Thursday, April 1, 2004
Reading: This problem set: Sections 4.0-4.5 and Section 4.7 except 4.7.5
Next: Sections 4.3-4.6, 4.A, 4.B, Chapter 5
Final Exam is on May 19, 2004, from 9:00am to 12:00 noon in Walker
Memorial (50-340). You are allowed to bring three 8
1
2
11
sheets of notes (both

sides). If you have a conict with this time and need to schedule an alternate time,
you must see Prof. Willsky by April 6, 2004 at the absolute latest.
Problem 6.1
Consider the estimation of a nonrandom but unknown parameter x from an observa-
tion of the form y = x+w where w is a random variable. Two dierent scenarios are
considered in parts (a) and (b) below.
(a) Suppose w is a zero-mean Laplacian random variable. i.e.,
p
w
(w) =

2
e
|w|
for some > 0. Does an unbiased estimate of x exist? Explain. Does an ecent
estimate of x exist? If so, determine x
e
(y). If not, explain.
Hint:
w
2
2
e
|w|
dw =
2
2
.
(b) Suppose w is a zero-mean random variable with probability density p
w
(w) > 0
depicted in Figure 6-1. Does an ecient estimate of x exist? If so, determine
x
e
(y). If not, explain.
p
w
(w )
0 1 2 -1 -2 -3 3
w
Figure 6-1
1
Problems-attack!
3. Suppose that h, x, v are mutually independent random variables with
x N(3, 4), v N(0, 4)
and where h takes on only two possible values, 1 and 3 with
Pr(h = 1) = Pr(h = 3) = 1/2.
Let
z = hx + v
(a) Determine the MMSE estimator of x given z and h (i.e. x
MMSE
).
(b) Determine the MMSE achieved by the above estimator.
4
Course outline
Spring 2010
Series Analysis
class only.
Detection Theory:
1 of 3 1/11/10 8:50 PM
Spring 2010
Series Analysis
class only.
Detection Theory:
1 of 3 1/11/10 8:50 PM
Estimation: General Minimum Variance Unbiased Estimation
Bias: (expected value of estimator - true value of data)
MVUE:
Estimation: Cramer-Rao lower bound
Lower bound on variance of ANY unbiased estimator!
Usage:
assert whether an estimator is MVUE
benchmark against which to measure the performance of
an unbiased estimator
feasibility studies
Depends on?
Nature
Transmission /
measurement
Processing
Noise
The Cramer-Rao Lower Bound (CRLB)
CRLB examples
CRLB T or F
all the way to....
CRLB for General Gaussian Case
When observations are Gaussian and one knows the dependence of the
mean and covariance matrix on the unknown parameters, we know the
closed form of the CRLB (or Fisher information matrix):
Finding MVUE so far
Finding the MVUE
Estimation: linear models
Whats a linear model and why is it useful?
What can be said?
Best Linear Unbiased Estimators (BLUE)
MVUE for the linear model
Examples
Examples
General linear model
Re-do the derivation
Key idea: whiten the observations,
then apply previous results
Gauss-Markov theorem for BLUEs:
BLUE - vector
Estimation: Maximum Likelihood Estimation
Alternative to MVUE which is hard to nd in general
Easy to compute - very widely used and practical
What is the MLE?
Properties?
Properties of the MLE
Invariance property
Examples
MLE for linear model
Numerical methods
Estimation: Least Squares
Alternative estimator with no general optimality properties, but nice and
intuitive and no probabilistic assumptions on data are made - only need a
signal model
Advantages?
Disadvantages?
Examples
Linear LSE: theory
Linear model without any noise assumptions!
Linear LSE vs other estimators
7
Comparing the Linear LSE to Other Estimates
Model Estimate
e H! x ! "
# $ x H H H !
! !
"#
1
%
"
No Probability Model Needed
# $ x H H H !
! !
$"%&
1
%
"
w H! x ! "
PDF Unknown, White
# $ x H H H
! !
'"
1
%
" &
w H! x ! "
PDF Gaussian, White
# $ x H H H !
! !
'(%
1

%
"
w H! x ! "
PDF Gaussian, White
If you
assume
Gaussian &
apply
these.
BUT you
are
WRONG.
you at least
get the
LSE!
http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm
Estimation: Bayesian Estimation
Parameter to be estimated is assumed to be random, according to some prior
distribution which models our knowledge of it
Bayesian Minimum Mean Squared Error (MMSE):
Applications to Gaussian noise / linear model
Bayesian estimation on Gaussian priors/noise
This is crucial to know!
Bayesian estimation on Gaussian priors/noise
This is crucial to know!
Bayesian linear model
Examples: MMSE in Gaussian noise
Properties of MMSE
Estimation: Bayesian Estimation
General risk functions - arbitrary cost functions
Maximum a posteriori (MAP) estimation
Linear MMSE: constrain estimator to be linear - very practical
MAP properties
Sequential LMMSE
Algebraic approach Geometric approach
LMMSE properties
Course outline
Spring 2010
Series Analysis
class only.
Detection Theory:
1 of 3 1/11/10 8:50 PM
Spring 2010
Series Analysis
class only.
Detection Theory:
1 of 3 1/11/10 8:50 PM
Problems - attack!
ECE 531 Detection and Estimation - Final Exam Name:
5. The random variables X and Y are uniformly distributed over the trapezoidal region shown in the gure
below, described by the equations
p(x, y) =
1 |y| < 1/2, y 1/2 < x < y + 1/2

0 else
Next:
X|Y
(y) = E
_
[x x
BLS
(y)]
2
| y
_
=
_

_
x
2
_
y +
1
a
_
_
2
p
X|Y
(x | y) dx
=
_

0
_
x
2
_
y +
1
a
_
_
2
x
_
y +
1
a
_
2
e
x(y+
1
a
)
dx
=
_

0
_
x
3
4x
2
1
y +
1
a
+ 4x
1
_
y +
1
a
_
2
_
_
y +
1
a
_
2
e
x(y+
1
a
)
dx
= 3!
_
y +
1
a
_
24
4 2!
_
y +
1
a
_
213
+ 4 1!
_
y +
1
a
_
222
=
2
(y +
1
a
)
2
Finally:
BLS
= E
_
X|Y
(y)
=
_

X|Y
(y) p
Y
(y) dy =
_

0
2
(y +
1
a
)
2
1
a
1
(y +
1
a
)
2
dy =
2
a
_
1
3
__
y +
1
a
_
3
0
=
2a
2
3
(b) Now we want to compute x
MAP
(y). Weve already done most of the work in part (a).
x
MAP
(y) = argmax
x
p
X|Y
(x | y) = argmax
x
_
y +
1
a
_
2
xe
x(y+
1
a
)
u(x) = argmax
x
xe
x(y+
1
a
)
, x 0
Taking derivatives we nd:
d
dx
_
xe
x(y+
1
a
)
_
= e
x(y+
1
a
)
x(y +
1
a
)e
x(y+
1
a
)
= 0
= x =
1
y +
1
a
Thus x
MAP
(y) =
1
y+
1
a
. We really need to also verify that the second derivative is negative, so that we
have actually found a maximum. You can verify that:
d
2
dx
2
_
x e
x(y+
1
a
)
_
x=
1
y+
1
a
= 2(y +
1
a
)e
1
+ (y +
1
a
)e
1
< 0
Problem 9.4 (Old Exam Question)
The random variables X and Y are uniformly distributed over the region shown in the gure.

;
:
F
: ;
N O

(a) Find x
BLS
the Bayes least square estimate of X given Y ,
X|Y
the corresponding conditional covari-
ance, and
BLS
the corresponding mean square error.
5
(a) (5 points) Find the MMSE of X given Y and the corresponding minimum mean squared error.
(b) (3 points) Find the Linear MMSE of X.
(c) (2 points) Dene the Quadratic MMSE as the estimator

X = aY
2
+ bY + c where the constants
a, b, c, are chosen to minimize the mean squared error. Find the quadratic MMSE estimator of X
based on Y and the corresponding MMSE. HINT: think rst!
(d) (5 points) Find the MAP estimate of X based on Y .
Points earned: out of a possible 15 points
Problems - attack
T/F: The performance of a detector in colored noise is always worse than that
of a detector in white noise.
T/F: The Wiener lter is a Bayesian lter than optimizes the MMSE criterion.
T/F: The Cramer-Rao bound seen in class only applies to unbiased
estimators.
Explain the difference between MAP and ML estimators.
Explain the difference between MAP and ML detectors.
T/F: The sample mean is the MVUE for the DV level of any signal in noise.
Course outline
Spring 2010
Series Analysis
class only.
Detection Theory:
1 of 3 1/11/10 8:50 PM
Spring 2010
Series Analysis
class only.
Detection Theory:
1 of 3 1/11/10 8:50 PM
Detection: Statistical Detection Theory
Binary hypothesis testing
Detection: Statistical Detection Theory
Neyman-Pearson hypothesis testing
Useful problem 2.1
Bayesian risk
Multiple hypothesis testing
Detection: Deterministic Signals
How to detect known signals in noise?
The famous matched lter!
X x[n]
T(x)
s[n]
Generalized matched lter
> 2 hypotheses
Performance of matched lter
Performance of generalized matched lters
Designing signals
Attack?
Be able to show this!
Multiple Deterministic Signals in Gaussian Noise
Lets derive this decision rule!
Geometric meaning?
Detection: Random Signals
What if s[n] is random?
Key idea behind estimator-correlator:
Linear model simplies things again...
The problem
Example: Linear Model
Detection: Statistical Decision Theory II
Uniformly most powerful test
Generalized likelihood ratio test
Bayesian approach
Wald test
Rao test
Further resources

Review

Caricato da

Informazioni sul documento

Descrizione originale:

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Review

Caricato da

Copyright:

Formati disponibili

ECE 531: Detection and Estimation Theory

sheets of notes (both

1 |y| < 1/2, y 1/2 < x < y + 1/2

Potrebbero piacerti anche