Spectrum estimation — nonparametric methods Material comes from Hayes a Summarit 2008 Tr Introduction ‘As we discussed previously, the power spectrum of a wss process is the Fourier transform of its autocorrelation sequence. Therefore, estimating the spectrum is equivalent to estimating the autocorrelation. For an autocorrelation ergodic process: @21) Therefore, if x, is known for all n, estimating the power spectrum is (in theory) straightforward. However, there are two primer limitations making spectral estimation an extremely challenging problem. 1. The amount of data available for the analysis is never unlimited and, in many situations, might be very small (short observation time or quasi-stationarity limitations). 2. The data is often corrupted by noise or contaminated with an interfering signal. Therefore, spectrum estimation is a problem involving estimating P,(e*) from a finite number of noisy data sample. a Summa 2008Introduction Spectral estimation is a problem that is of great importance in many applications including data analysis, Wiener filtering (to build an optimum filter, power spectra of desired signal and noise must be computed), signal detection, classification, and tracking, etc. There are two major classes of spectral estimators: 1. Nonparametric (classical) methods — begin by estimating the autocorrelation sequence from a given data. The power spectrum then is estimated via Fourier transform of an estimated autocorrelation sequence; 2. Parametric (non-classical) methods — the analyzed process is replaced by an appropriate model with known (for the specific parameters) spectrum sume TE) The Periodogram The periodogram method was introduced by Schuster in 1898. The power spectrum of a wss process is the Fourier transform of its autocorrelation sequence’ P(e@)= Vin (ker 44) i Therefore, spectrum estimation is, in some sense, an autocorrelation estimation problem. For an autocorrelation ergodic process and an unlimited amount of data, the autocorrelation may (theoretically) be determined with the time-average, ¥ “(8)=tin{ 1 Esa} (842) Noe LING, J However, if x, is only available for a finite interval (say, [0, NV-1]), the autocorrelation can be estimated, for example, with a finite sum sumer 2008The Periodogram we A= TS nat ws Niwa To ensure that the values of x, outside the interval [0, N-1] are excluded from the sum, (8.5.1) must be modified as pNcke A(k) Snakes Oy =1 082) Now Using the conjugate symmetry, the values of 7, () for k< 0 can be defined as ACK) =F (k) asa) Outside the interval: R(k)=0 — for|kl=N sa) The Periodogram Taking next the DTFT (actually, DFT) of the autocorrelation estimate leads to an estimate of the power spectrum called the periodogram: A(er)= SE (Re™ een |twould be more convenient to express the periodogram in terms of the process. itself rather then its autocorrelation. Let xy, be the finite length NV signal such that x, O Xy yer” xe" 73) Therefore, the periodogram is proportional to the squared magnitude of DFT of x, (20k ber 1 5 fat Xue > Xya P—[Xv al = P| eo ® ) (74) aN ) summers TE) The Periodogram: Ex Periodogram of white noise. If x, is white noise with a variance of o,2, then r,(k) = 6,2, and the power spectrum is a constant: P(e@)=02 881) Assample realization of unit variance white noise of length N = 32: 2 bad | 4 sumer 2008The Periodogram: Ex An autocorrelation estimate using (8.5.1). Although the 98 autocorrelation is zero for |k| > 32, itis nonzero for all other k. ° 0 © % «© The periodogram 1 (solid) and the true power spectrum (dashed). Although the g- periodogram is approximately equal 27" the true spectrumon 4 average, there are + *) 0) 02 0304 0s 08 07 08 os 1 visible variations. Freavency (unite of i) a summer 2008 TT 10 The Periodogram There is an interesting property. Let h,, be an FIR filter of length N defined as follows: me Oa sumer 200818 Performance of the Periodogram is a Bartlett (triangular) window, Therefore, the autocorrelation estimate is biased. The expected value of the periodogram is Bffo (0) =e{ ¥ nue} ¥ Eff. (k)pe = Yr, (kW Ze" 8.154) Since it is a Fourier transform of a product, using the frequency convolution, we Fe(e*)} where W®(e/*) is the Fourier transform of the Bartlett window 1 [seen iernay (c)#w* (e”") (e182) Qn sin(@/2) “ oe oe TT 16 Performance of the Periodogram Thus, the expected value of the petiodogram is the convolution of the power spectrum P,(e!*) with the Fourier transform of a Bartlett window. Therefore, the periodogram is a biased estimate. However, since W(e*) converges to an impulse as NV goes to infinity, the periodogram is asymptotically unbiased ime (A(e™)=R(e*) on For illustration, consider a random-phase sinusoid in white noise: = Asin(n@+9)+v, (0162) where gis a uniform over [-z,] random variable and v,, is white noise with a variance 0,2, The power spectrum of x, is Ple*)=o2 +h [u,(0-a,)+u4(0+0,)] (a8) « aw Performance of the Periodogram Therefore, the expected value of periodogram is 3 P. (ei*)ew" (e*)= a2 Lar [ we (eles!) art (ere I 7.) The power spectrum“) (0%) for N= 64. 18 Performance of the Periodogram Fe) The expected value 7 of the periodogram | mie power spectrum estimate for N= 64 There are two major effects: 1.The spectral smoothing produced by W®(o!) that leads to a spreading of the power in the sinusoid over a frequency band with bandwidth of approximately 4n/N. 2. The power leakage through the sidelobes of the window creating secondary spectral peaks at frequencies 2a @, = 0+ — a sine (@.18.1)19 Performance of the Periodogram With A= 5, a = 0.4x, and N = 64, 50 different realizations of (8.16.2) were generated and periodograms were estimated. Realizations and the average. Increased the number of data values: N= 256, realizations and the average The power is spread out over a narrower frequency band 20 Performance of the Periodogram ‘Smoothing introduced by the Bartlett window also limits the ability of the periodogram to resolve closely-spaced narrowband components in x,, Let a random process consists of two sinusoids in white noise: A sin(na, + ¢,)+ A, sin(n@, +, )+v, (20.1) where ¢, and g, are uncorrelated uniform over [-2,] random variables and v,, is white noise with a variance o,2. The power spectrum of x, is Ple")=ot+4 7A [u(o-@)+u,(ore) +224 [u,(@-@) +44(+e) (8202) 2 The expected value of periodogram is Ff, (e*)}=LR(e)om*(e*) wo ebaLit(oe)jaart (elon )]at al wt(etn) art (eto)] zo « aa Performance of the Periodogram Pee) The power spectrum (0!) for A, = Az, N= 64, Abie The expected value of the periodogram al power spectrum for N=64. 2 Performance of the Periodogram Since the width of the main lobe of W&(e!*) increases as the data length decreases, there is a limit on how closely two sinusoids (or two narrowband processes) may be located, for the given data length NV, in order to be resolved. Usually, the resolution limit is defined to be equal to the width of the main lobe of the spectral window at its “half-power" (-6 dB) point. For the Bartlett window: Res| &.,, (e””) | 08922 (9221) However, itis important to note that (8.22.1) is just a rule of thumb since it works “on the average” and really depends on the phase between two sinusoids. With A=5, @, = 0.4n, @ = 0.45x, 50 different realizations of (8.20.1) were generated and periodograms were estimated. According to (8.22.1), data length of N= 36 is required to resolve the frequency components, sumer 200823 Performance of the Periodogram 50 periodogram estimates and their average for N= 40. Barely resolved. 50 periodogram estimates and their average for N= 64. i Clearly resolved. | sae a summer 2008 TT 2 Performance of the Periodogram 2. Variance of the Periodogram. The periodogram estimate is asymptotically unbiased, In order for it to be a consistent estimate, it is necessary that the variance goes to zero as N >=. Itis difficult to evaluate the variance of the periodogram for an arbitrary input process x,. The variance may be estimated for the special case of white noise x, with variance 2. In this case, the periodogram may be found as > 17a i Not 10 | =—| VY xe% xe aldo" | Eee | SY yave e281) mm Therefore, the 2" moment of periodogram is, 1 Xavava ya a i a TOT ono 4 sumer 2008 E{P.., (e) perPerformance of the Periodogram which actually depends on the 4" order moments of x,. However, since the process is Gaussian, we may use the moment factoring theorem to simplify these moments. For complex Gaussian variables, the moment factoring theorem is, EXx,xix,x,}=E {xx} E{x,x,} +2 {x00} E{x,%7} (625.1) ‘Substitution (8.24.2) into (8.25.1), we observe that the first term simplifies to 1 xy UES ot =! ons N° imo The second term becomes 1 tvs Hite, = (kD) arte Hina Hts Na (oon i(o,-02) (oa) Sol 10 of [1-e on ']__.[sinv(o,-0,)/2) = |= | SNOT OH) gas) NL ere) | ae) || Nsin(@ —@,)/2 “ eve et TT) Performance of the Periodogram Therefore, the 2" moment is £{B., (e*)B., (e)}=e! 28) 20 (e)he(P,..(e")} (6282) Since Cov, (e) and js then, the covariance of the periodogram is . ~ = /2 cov( (eA, (e)} = (a-a) (axear Performance of the Periodogram Finally, setting @, = @», we have for the variance Var {?.,, (e”)} =o! e274) Therefore, the variance does not go to zero as N+ and the periodogram is not a consistent estimate of the power spectrum. Since for the white noise P(e”)=07 (e272) the variance of the periodogram of white Gaussian process is var{B,, (e)} = P(e”) e273) sume TE) 28 Performance of the Periodogram Let x, be white Gaussian noise with P. (e”) 1 (6.28.1) It follows that the expected value is (e)}=1 (e282) e| veel (e*) Therefore, although the periodogram is unbiased in this case, the variance equals a constant that is independent of the data length N. and the variance is (028.3) 50 realizations were generated and the periodograms were evaluated for them. sumer 2008Performance of Periodogram N=64 N= 128 N= 256 Variance does not decrease when the amount of data increases. Performance of the Periodogram The analysis of non-Gaussian processes is more difficult. However, the variance of the periodogram for non-Gaussian processes can be approximately described by (8.27.3). 2 5 WS ja ine Prer(el®) = Se xe Summary of the n= periodogram: Bias - 1 E {Prer(e!*)} = 5= Pale!) « Wale’) Resolution . Aw = 0.897% N Variance Var { Bper(e!)} * P2(e!) sumer 2008The Modified Periodogram The periodogram is proportional to the squared magnitude of the DTFT of the windowed signal x,,,, = x,W,°. Would it be beneficial to use instead of a rectangular some other window? We examine next the effect of data window on the bias of the periodogram, which may be expresses as (e3t.1) The expected value of the periodogram is abetopel Sonim [Beat] DL Lr (nem) we ear.ay ~ xe {= > axcwtersooni| _ The Modified Periodogram Changing variables k = n-m #{2.(e") rb Scie fe (e321) where We =WE ewe = > Wwiwe, (6.32.2) is a Bartlett window, Using the frequency convolution theorem, the expected value: E{Pa(e")}=s rR (e)*|0" (e"] (632.3) Sin(N@/2) __v-por2 sin(@/2) 4 sumer 2008 where w*(er) 22033 The Modified Periodogram is the Fourier transform of the rectangular window. Therefore, the amount of ‘smoothing in the periodogram is determined by the window that is applied to the data. Although a rectangular window has a narrow main lobe compared to other windows and, therefore, produces the last amount of spectral smoothing, it has relatively large sidelobes that may lead to masking of weak narrowband components. For example, 2 sinusoids in white noise: x, =0.1sin(n@, + ¢,)+sin(na, +¢,)+v, (633.1) With @,= 0.2% = 0.3n,and | j 28, the | expected value, ofperiodogram | (rectangular, left) and the Hamming window (right). ‘summa 2008 4 The Modified Periodogram Using a Hamming window instead of the rectangular one allows to clearly observe the second frequency component due to the much lower sidelobes of Hamming windows. On the other hand, this reduction in the sidelobe amplitude is accompanied with an increase in the width of the mainlobe, which affects the resolution. The periodogram of a process that is windowed with a general window W, is called the modified periodogram method: (034.1) (8.34.2) Nim is a constant. sumer 200835 The Modified Periodogram The expected value of the modified periodogram is (835.1) We note that with (8.352) it follows: 1 fF 2 fotos jv(e*)f domi wx 2nNU ?, and, with an appropriate window, 1 jo)? pr(e") NU will converge to an impulse of unit area as N->.0 and the modified periodogram will be asymptotically unbiased. “ oe oe TT 36 The Modified Periodogram Since the modified periodogram is simply the periodogram of a windowed data sequence, its variance is approximately the same as that for the periodogram: Var{ A, (c)} = P(e) (636.1) Therefore, the modified periodogram is not a consistent estimate of the power spectrum and the data windows offer no benefit in terms of reducing the variance. The windows provide a Sidelobe 34B BW trade-off between Window Level (B) (A@)38 spectral resolution (main lobe with) and spectral Rectangular -13 0.89(21/N) masking (sidelobe Bartlett 27 1.28(2/N) amplitude). Hanning -32 1.44(Q2n/N) Properties of afew |” Hamming -4B3 1,30(2n/N) ‘common time windows: Blackman —58 1.68(21/N) « aar The Modified Periodogram w(n)x(nje"? ‘Summary of the modified periodogram: Bias 1 py (el*)) = je joyp E {Pulel)) = =a Pele) # 1WleI Resolution Window dependent Variance Var { P(e!) = P2(e!*) sume TE) 38 Bartlett’s method: averaging Bartlett's method of periodogram averaging produces a consistent estimate of the power spectrum, Averaging a set of uncorrelated measurements of a random variable x yields a consistent estimate of the mean E(x). This suggests that we consider estimating the power spectrum of a random process by periodogram averaging. Let x,,, for 1,2,...K be K uncorrelated realizations of a random process x, over the interval 0 < n =, the Bartlett's estimate is consistent. Additionally, for a given value of N, Bartlett's method allows to trade a reduction in spectral resolution in variance by changing the values K and L. LE a(el®) = ies yn +ibye no Bias Summary of the Paley) = + peo je ae £ {Pale!™)} = 5 Pale’) » Wale’) method: Resolution 2n dw = 089K 5 Variance Var { Pa(e’)} © en (ei) sumer 2008Bartlett’s 1 ] method ' Estimate the power spectrum of 50. =!) )))/111" different unit variance white noise 7 >= sa ; sequences of length N= 512 K = 1 (periodogram) 24.2100 ———> | a K=16,L=32 ——————> Considerable decrease in variance. 4“ Bartlett’s J ] method ] Consider a process consisting of 2. sinusoids with @, = 0.21, = 0.257 * and unit variance noise. With N= 512 and for 50 realizations, } 4 the Bartlett's estimates for K = 1, K = 4, and K= 16 were formed. "gl | Hal | Although the variance of the estimate decreases with K, the aoe ee resolution decreases correspondingly as indicated by the > , 7 broadening of spectral peaks. - Qi Q ut Mata |45 Welch’s method Welch proposed two modifications to Bartlett's method: 1, Allow the sequences x, to overlap; 2. Apply a data window w, to each sequence, thereby producing a set of modified periodograms to be averaged. ‘Assuming that successive sequences are offset by D points and that each sequence is L points long, then the # sequence is given by Xin = Xnsip i=0,1,..,L-1 (645.1) Thus, the amount of overlap between x,,, and X;.;, is L-D points, and if K sequences cover the entire NV data points, then N=L+D(K-l) (e452) For example, with no overlap (D = L) we have K = N/L sections of length Las in Barllett’s method. If the sequences are allowed to overlap by 50% (D = L/2), we may form N K=2—-1 (8.45.3) L summer 2008 TT “6 , Welch’s method sections of length L while maintaining the same resolution (section length) as Bartlett's method and reducing the variance (doubling number of modified periodograms). On the other hand, with 50% overlap we could also form N K=—-1 (8.46.1) L sequences of length 2L, thus, increasing the resolution while maintaining the same variance as Bartlett's method. Therefore, it is possible to trade a reduction in variance for a reduction in resolution by allowing the sequences to overlap. B ,, (2.46.2) or in terms of modified periodograms 5 (0) 1S su (jo B, (e" Jee (e”) (6.46.3) = sumer 2008ar Welch’s method The expected value of Welch's estimate is 5 5 ls 2 ESP, (e”)\\=E{B, (e”)\= P(e” + ei” (Fele"=8 Bele) = thle”) (e") where W(e/*) is the Fourier transform of the L-point data window W, used to form the modified periodograms. Therefore, Welch's method is asymptotically unbiased estimate of the power spectrum, The resolution depends on the data window. (e471) The variance is hard to evaluate, however, assuming the 50% overlap and the Bartlett window: var A (e”*)} = 2p? (e”) wary Itis possible to average more sequences for a given amount of data by increasing the amount of overlap. However, this would increase computational load and the correlation between sequences x,,, diminishing benefits of increasing overlap. The typical amount of overlap is either 50% or 75%, sume TE) Welch’s method kot fust a 1 Pule”) = ery DL |, woven + ide tao | n=0 ‘Summary of the Welch's method: Bias Resolution Window dependent Variancet Var { Pw (e!*)} z Eero) sumer 200849 Welch’s method Considering the same process with two sinusoids in white noise and using Welch's method with N = 512, L = 128, 50% overlap (7 sections), a Hamming window, a plot of Welch's estimates for 50 realizations and the average are shown. tj | Tj | The variance and the resolution (rectangular vs. longer Hamming) are approximately the same as for Bartlett's with K = 8 and L = 64, The gain is in the reduction of spectral leakage through the sidelobes. a sume TE) 50 Blackman-Tukey method: smoothing Bartlett's and Welch's methods reduce the variance of the spectral estimates by averaging periodograms and modified periodograms. Another method to decrease the statistical variability of periodogram is the periodogram smoothing referred to as the Blackman-Tukey method. The periodogram is a DTFT of a consistent estimate of the autocorrelation sequence. For any finite data record of length N, the variance of autocorrelation estimate will be large for values of lag k that are close to N. For example, the estimate of r,(k) at lag k = N-1 is (@50.1) Since there is little averaging involved into formation of (8.50.1) for [kl = N, no matter how large N becomes, these estimates will always be unreliable. Consequently, the only way to reduce the variance of the periodogram is to reduce the variance of these estimates or to reduce the contribution that they make to the periodogram, In B-T method, the variance of the periodogram is reduced by applying a window to the autocorrelation estimate to decrease the contribution of unreliable estimates to the periodogram. a See ¢51 Blackman-Tukey method: smoothing The Blackman-Tukey estimate is: B,(e*)= > AGM" (e511) F Me where W,, is a lag window applied to the autocorrelation estimate. For example, if W, is a rectangular window in [-M, Mj with M> M >> 1 (@522) sumer 200853 Blackman-Tukey method: smoothing Therefore, the trade-off is again between bias and variance. For a smalll bias, M should be large to minimize the width of the main lobe of W(e), while M should be ‘small to minimize the variance... Generally, it is recommended that M has a maximum value of NA. " Parle!) =) Fe(Ryw(be** ‘Summary of the ™ Blackman-Tukey Bias as 1 method: E [Parte] = 5 Pale) « Weel) Resolution Window dependent Variance 5 i< Var {Par(e™)} © PRe!*)— D> wk) mh a summer 2008 TT Performance comparison The selection of a non-parametric spectral estimator depends on the particular needs and requirements. ‘To compare the performance of different estimator, two additional criteria are introduced. The variability of the estimate: - aval (e") - FLA) v which is the normalized variance, The figure of merit of the estimate: M4 Vao (0.542) which should be as small as possible. sumer 200855 Performance comparison Variability Resolution _ Figure of Merit Periodogram Bartlett Welcht Blackman-Tukey The figure of merit is inversely proportional to the data length. The overall performance of classical estimators is fundamentally limited by the amount of data available. a summer 2008 TT More nonparametric methods Another nonparametric approach to estimating the coherence function has been proposed by Thomson. This method is based on Thomson's spectral estimation procedure, also referred to as the multiple — taper method, which is known to produce less biased spectral estimates in comparison with the periodogram method. This method was proposed for signals with complicated spectral densities. The appropriate procedure is briefly described next. ‘Assuming an NV-sample long time record x, with zero mean, the raw eigen- coefficients are specified as a discrete Fourier transform of the input signal: vt ,(e”) = x,y, (NW) eo" (8561) w= where v,"’(V,W)is the discrete prolate spheroidal sequence, which is a Fourier transform of the discrete prolate spheroidal wave function U, (VW; f) also called the Slepian function, These sequences are orthonormal providing spectral windows with well-concentrated energy, over which the data is observed. W denotes the bandwidth: 0< W< 0.5. 4 sumer 2008‘7 More nonparametric methods The raw eigen-coefficients are usually weighted by \/4,(V.37) to form estimates of idealized eigen-coefficients $,(e!”) = JA, (NW) -%,(e”) (esr) Here 4,(.V,IV) are the eigenvalues of the N x N matrix sin2xW(m-n) NP) PNW), a(m-n) m,n =O,lyonN=1 (057.2) ‘Thus, the spectral estimates are formed as follows (e873) However, the Thomson's estimate still suffers from the nonparametric limitations: high resolution is possible for long data records only. sume TE)