Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Maya Hristakeva
1 Linear Regression
Setup
X = [x1 ... xT ] ∈ RN x T
Reference values:
y = [y1 ... yT ]T ∈ RT x 1
Outline
1 Linear Regression
Linear Regression
Least Squares optimization problem:
T
1X T 1
L(w) = min (xi w − yi )2 ≡ min ||XT w − y||2
w 2 w 2
i=1
Differentiate w.r.t. w:
∇w L(w) = X(XT w − y) = 0
XXT w = Xy
Exact solution:
w? = (XXT )−1 Xy
Note: XXT is not always invertible
Ridge Regression
1 λ
L(w) = min ||XT w − y||2 + ||w||2
w 2 2
Differentiate w.r.t. w:
∇w L(w) = X(XT w − y) + λw = 0
(XXT + λI)w = Xy
Exact solution:
w? = (XXT + λI)−1 Xy
Note: XXT + λI is always invertible for λ > 0
Outline
1 Linear Regression
Properties of P:
P2 = P ∈ RNxN
P = ki=1 pi pTi = P̃P̃T for P̃ = [p1 ...pk ] ∈ RNxK
P
Variance Maximization
Find k projection directions P̃ = [p1 ...pk ] for which the variance of
the compressed data (P̃T X) is maximized:
T T
X 1 X T
max tr var (P̃T xi ) ≡ max tr (P̃ xi )(P̃T xi )T
P̃
i=1
P̃ T i=1
N
1 X T
= max tr (xi P̃P̃T} xi )
P̃ T i=1 | {z
P
N
1 X
= max tr (P (xi xTi ))
P T i=1
1
= max tr (P XXT )
P
|T {z }
C
PCA Solution
Let C = XXT : covariance matrix of X
X
max tr (PC) = max tr (P( γi ci cTi )
P P
i
X
= max γi tr (cTi Pci )
P | {zP }
i
cT
i Pci ≤1, i cT
i Pci =k
X
≤ max
P γi δi
0≤δi ≤1, i δi =k
i
k
X
= max γij = k largest eigenvalues of C
1≤i1 <i2 <ik ≤n
j=1
Summary of PCA
Outline
1 Linear Regression
Finding p1
PLS Regression
PLS Regression ≡ PLS Decomposition + Linear Regression
Use PLS to find a projection directions pi
Summary