Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
equation
f =
f
f
f
x1 +
x2 +
x3
x1
x2
x3
The six kinds of derivatives that can be most neatly organized in matrix form are collected in the following
table.[1]
f =
Scope
f
x
[
=
f
x1
f
x2
f
x3
]
.
Here, we have used the term matrix in its most general sense, recognizing that vectors and scalars are simply
matrices with one column and then one row respectively.
Moreover, we have used bold letters to indicate vectors
and bold capital letters for matrices. This notation is used
throughout.
1.1
The matrix derivative is a convenient notation for keeping track of partial derivatives for doing calculations. The
Frchet derivative is the standard way in the setting of
functional analysis to take derivatives with respect to vectors. In the case that a matrix function of a matrix is
Frchet dierentiable, the two derivatives will agree up
to translation of notations. As is the case in general
for partial derivatives, some formulae may extend under weaker analytic conditions than the existence of the
derivative as approximating linear mapping.
1.2
Usages
Matrix calculus is used for deriving optimal stochastic estimators, often involving the use of Lagrange multipliers.
This includes the derivation of:
two possible notational choices involved. The reason is that the choice of numerator vs. denominator (or in some situations, numerator vs. mixed)
can be made independently for scalar-by-vector,
vector-by-scalar, vector-by-vector, and scalar-bymatrix derivatives, and a number of authors mix and
match their layout choices in various ways.
2. The choice of numerator layout in the introductory
sections below does not imply that this is the correct or superior choice. There are advantages
and disadvantages to the various layout types. Serious mistakes can result from carelessly combining formulas written in dierent layouts, and converting from one layout to another requires care to
avoid errors. As a result, when working with existing formulas the best policy is probably to identify
whichever layout is used and maintain consistency
with it, rather than attempting to use the same layout in all situations.
Kalman lter
Wiener lter
2.1 Alternatives
Expectation-maximization algorithm for Gaussian The tensor index notation with its Einstein summation
mixture
convention is very similar to the matrix calculus, except
one writes only a single component at a time. It has the
advantage that one can easily manipulate arbitrarily high
2 Notation
rank tensors, whereas tensors of rank higher than two are
quite unwieldy with matrix notation. All of the work here
The vector and matrix derivatives presented in the sec- can be done in this notation without use of the singletions to follow take full advantage of matrix notation, us- variable matrix notation. However, many problems in esing a single variable to represent a large number of vari- timation theory and other areas of applied mathematics
ables. In what follows we will distinguish scalars, vectors would result in too many indices to properly keep track
and matrices by their typeface. We will let M(n,m) de- of, pointing in favor of matrix calculus in those areas.
note the space of real nm matrices with n rows and m Also, Einstein notation can be very useful in proving the
columns. Such matrices will be denoted using bold cap- identities presented here, as an alternative to typical eleital letters: A, X, Y, etc. An element of M(n,1), that is, ment notation, which can become cumbersome when the
a column vector, is denoted with a boldface lowercase explicit sums are carried around. Note that a matrix can
letter: a, x, y, etc. An element of M(1,1) is a scalar, be considered a tensor of rank two.
denoted with lowercase italic typeface: a, t, x, etc. XT
denotes matrix transpose, tr(X) is the trace, and det(X)
is the determinant. All functions are assumed to be of 3 Derivatives with vectors
dierentiability class C 1 unless otherwise noted. Generally letters from rst half of the alphabet (a, b, c, ) will
Main article: Vector calculus
be used to denote constants, and from the second half (t,
x, y, ) to denote variables.
Because vectors are matrices with only one column, the
NOTE: As mentioned above, there are competing notasimplest matrix derivatives are vector derivatives.
tions for laying out systems of partial derivatives in vectors and matrices, and no standard appears to be emerg- The notations developed here can accommodate the usual
ing yet. The next two introductory sections use the operations of vector calculus by identifying the space
n
numerator layout convention simply for the purposes of M(n,1) of n-vectors with the Euclidean space R , and
convenience, to avoid overly complicating the discussion. the scalar M(1,1) is identied with R. The correspondThe section after them discusses layout conventions in ing concept from vector calculus is indicated at the end
of each subsection.
more detail. It is important to realize the following:
NOTE: The discussion in this section assumes the
1. Despite the use of the terms numerator layout and numerator layout convention for pedagogical purposes.
denominator layout, there are actually more than Some authors use dierent conventions. The section on
3.3
Vector-by-vector
x1
y2
x1
y
=
.
x
..
y1
x2
y2
x2
..
.
y1
xn
y2
xn
..
.
.
..
Example Simple examples of this include the velocity
.
vector in Euclidean space, which is the tangent vector
ym
ym
ym
x1
x2
xn
of the position vector (considered as a function of time).
Also, the acceleration is the tangent vector of the velocity. In vector calculus, the derivative of a vector function y
with respect to a vector x whose components represent a
space is known as the pushforward or dierential, or
3.2 Scalar-by-vector
the Jacobian matrix.
The pushforward along a vector function f with respect to
x1
f
vector v in Rm is given by d f(v) = x
v.
x2
The derivative of a scalar y by a vector x = . , is
..
xn
4 Derivatives with matrices
written (in numerator layout notation) as
[
]
y
y y
y
=
.
x
x1 x2
xn
In vector calculus, the gradient of a scalar eld y in the
space Rn (whose independent coordinates are the components of x) is the derivative of a scalar by a vector.
In physics, the electric eld is the vector gradient of the
electric potential.
The directional derivative of a scalar function f(x) of the NOTE: The discussion in this section assumes the
space vector x in the direction of the unit vector u is de- numerator layout convention for pedagogical purposes.
Some authors use dierent conventions. The section on
ned using the gradient as follows.
layout conventions discusses this issue in greater detail.
The identities given further down are presented in forms
that can be used in conjunction with all common layout
u f (x) = f (x) u
conventions.
Using the notation just dened for the derivative of a
scalar with respect to a vector we can re-write the directional derivative as u f = f
Matrix-by-scalar
x u. This type of notation 4.1
will be nice when proving product rules and chain rules
that come out looking similar to what we are familiar with The derivative of a matrix function Y by a scalar x is
for the scalar derivative.
known as the tangent matrix and is given (in numerator
5 LAYOUT CONVENTIONS
layout notation) by
y11
x
y21
x
y12
x
y22
x
..
.
..
.
y1n
x
y2n
x
ym1
x
ym2
x
ymn
x
= .
x
..
4.2
.. .
.
The dierential or the matrix derivative of a matrix function F(X) that maps from nm matrices to pq matrices,
F : M(n,m) M(p,q), is an element of M(p,q) ? M(m,n),
a fourth-rank tensor (the reversal of m and n here indicates the dual space of M(n,m)). In short it is an mn
matrix each of whose entries is a pq matrix.
F
=
X
Scalar-by-matrix
F
X1,1
..
.
..
.
F
Xn,1
..
.
X1,m
Xn,m
The derivative of a scalar y function of a matrix X of
independent variables, with respect to the matrix X, is
F
and note that each X
is a pq matrix dened as above.
ij
given (in numerator layout notation) by
Note also that this matrix has its indexing transposed; m
rows and n columns. The pushforward along F of an nm
y
y
y
matrix Y in M(n,m) is then
xp1
x11
x21
y
y
y
x12 x22 xp2
y
=
(
)
.
..
.. .
..
X
F
.
..
.
.
Y ,
dF(Y) = tr
y
y
y
X
x1q
x2q
xpq
y(X)
X
)
f
Y .
X
4.3
x1,q
x1,1
(X)
.
..
..
=
.
.
..
X
x
xn,1
n,q
Given F = (fs,t ) , a dierentiable m n function of an n m matrix X ,
f1,1
f1,p
X
X
F(X) .
..
..
= ..
.
.
X
fm,1
fm,p
X
X
The Jacobian matrix, according to Magnus and
Neudecker,[2] is
The three types of derivatives that have not been considered are those involving vectors-by-matrices, matricesby-vectors, and matrices-by-matrices. These are not as D F (X) = vec F (X)
.
(vec X)
widely considered and a notation is not widely agreed
upon. As for vectors, the other two types of higher matrix
derivatives can be seen as applications of the derivative of
a matrix by a matrix by using a matrix with one column 5 Layout conventions
in the correct place. For this reason, in this subsection we
consider only how one can write the derivative of a matrix This section discusses the similarities and dierences between notational conventions that are used in the various
by another matrix.
5
numerator layout lays out according to Y and XT , while
consistent denominator layout lays out according to YT
and X. In practice, however, following a denominator layT
out for Y
x , and laying the result out according to Y , is
rarely seen because it makes for ugly formulas that do not
The fundamental issue is that the derivative of a vector correspond to the scalar formulas. As a result, the followy
with respect to a vector, i.e. x
, is often written in two ing layouts can often be found:
competing ways. If the numerator y is of size m and the
1. Consistent numerator layout, which lays out Y
x acdenominator x of size n, then the result can be laid out as
y
T
cording
to
Y
and
according
to
X
.
either an mn matrix or nm matrix, i.e. the elements
X
of y laid out in columns and the elements of x laid out in
2. Mixed layout, which lays out Y
x according to Y and
rows, or vice versa. This leads to the following possibiliy
X according to X.
ties:
elds that take advantage of matrix calculus. Although
there are largely two consistent conventions, some authors
nd it convenient to mix the two conventions in forms that
are discussed below. After this section equations will be
listed in both competing forms separately.
y
3. Use the notation X
, with results the same as consistent numerator layout.
y
x
and
y
x ,
Similarly, when it comes to scalar-by-matrix derivatives The results of operations will be transposed when switchy
Y
X and matrix-by-scalar derivatives x , then consistent ing between numerator-layout and denominator-layout
6
y
notation.
5.1
x
y11
x2
y
=
.
x
..
Numerator-layout notation
y1
xn
..
.
y2
xn
y
x11
y
x21
y
x12
y
x22
y
xp1
y
xp2
y
= .
X
..
[
]
y
y y
y
=
.
x
x1 x2
xn
y2
x1
y2
x2
..
.
..
.
..
.
IDENTITIES
ym
x1
ym
x2
.. .
.
ym
xn
y
x1q
y
x2q
.. .
.
y
xpq
y1
x
y2
y
x
= .
x ..
6 Identities
As noted above, in general, the results of operations will
be transposed when switching between numerator-layout
and denominator-layout notation.
ym
x
y1
x2
y2
x2
x1
y2
x1
y
= .
x
..
..
.
y1
xn
y2
xn
..
.
.
y
y
x
x11
x21
tained, since matrix products are not commutative. The
p1
y
y
y
x12 x22 xp2
chain rule applies in some of the cases, but unfortunately
y
.
=
.
.
.
.
does not apply in matrix-by-scalar derivatives or scalar.
.
.
.
X .
.
.
.
by-matrix derivatives (in the latter case, mostly involving
y
y
y
x
x1q
x2q
the trace operator applied to matrices). In the latter case,
pq
The following denitions are only provided in numerator- the product rule can't quite be applied directly, either, but
the equivalent can be done with a bit more work using the
layout notation:
dierential identities.
..
.
y11
x
y21
x
y12
x
y22
x
..
.
..
.
y1n
x
y2n
x
ym1
x
ym2
x
ymn
x
dx11
dx21
dX = .
..
dx12
dx22
..
.
..
.
dx1n
dx2n
.. .
.
dxm1
dxm2
dxmn
= .
x
..
.. .
.
Denominator-layout notation
[3]
y
x
y1
x2
y
=
.
x
..
y
xn
[
]
y
y1 y2
ym
=
.
x
x x
x
6.7
assume the matrices are laid out consistent with the vec- 6.6.2 With matrices involved
tor layout, i.e. numerator-layout matrix when numeratorlayout vector and vice versa; otherwise, transpose the
vector-by-vector derivatives.
6.4
Scalar-by-matrix identities
tr(A) = tr(A )
tr(ABCD) = tr(BCDA) = tr(CDAB) = tr(DABC)
For example, to compute
tr(AXBX C)
X
7 See also
Derivative (generalizations)
Product integral
8 Notes
= tr(CAXB(dX) ) + tr(CA(dX)BX )
(
)
= tr (CAXB(dX) ) + tr(CA(dX)BX
) Minka, Thomas P. Old and New Matrix Algebra Useful
[1]
= tr((dX)B X A C ) + tr(CA(dX)BX ) for Statistics. December 28, 2000.
= tr(B X A C (dX)) + tr(BX CA(dX))
[2] Magnus, Jan R.; Neudecker, Heinz (1999). Matrix Dier(
)
ential Calculus with Applications in Statistics and Econo
= tr (B X A C + BX CA)dX
metrics. Wiley Series in Probability and Statistics (2nd
ed.). Wiley. pp. 171173.
Therefore,
[3]
tr(AXBX C)
= B X A C + BX CA.
X
(For the last step, see the `Conversion from dierential to
derivative form' section.)
6.5
Matrix-by-scalar identities
6.6
6.6.1
Scalar-by-scalar identities
With vectors involved
External links
Linear Algebra: Determinants, Inverses, Rank appendix D from Introduction to Finite Element Methods book on University of Colorado at Boulder.
Uses the Hessian (transpose to Jacobian) denition
of vector and matrix derivatives.
Matrix Reference Manual, Mike Brookes, Imperial
College London.
The Matrix Cookbook (2006), with a derivatives
chapter. Uses the Hessian denition.
The Matrix Cookbook (2012), an updated version
of the Matrix Cookbook.
Linear Algebra and its Applications (author information page; see Chapter 9 of book), Peter Lax,
Courant Institute.
Matrix Dierentiation (and some other stu), Randal J. Barnes, Department of Civil Engineering,
University of Minnesota.
Notes on Matrix Calculus, Paul L. Fackler, North
Carolina State University.
Matrix Dierential Calculus (slide presentation),
Zhang Le, University of Edinburgh.
Introduction to Vector and Matrix Dierentiation
(notes on matrix dierentiation, in the context of
Econometrics), Heino Bohn Nielsen.
A note on dierentiating matrices (notes on matrix
dierentiation), Pawel Koval, from Munich Personal
RePEc Archive.
Vector/Matrix Calculus More notes on matrix differentiation.
Matrix Identities (notes on matrix dierentiation),
Sam Roweis. (needs user login)
EXTERNAL LINKS
10
10.1
10.2
Images
File:Gnome-searchtool.svg Source: https://upload.wikimedia.org/wikipedia/commons/1/1e/Gnome-searchtool.svg License: LGPL Contributors: http://ftp.gnome.org/pub/GNOME/sources/gnome-themes-extras/0.9/gnome-themes-extras-0.9.0.tar.gz Original artist: David
Vignoni
File:Nuvola_apps_edu_mathematics_blue-p.svg Source: https://upload.wikimedia.org/wikipedia/commons/3/3e/Nuvola_apps_edu_
mathematics_blue-p.svg License: GPL Contributors: Derivative work from Image:Nuvola apps edu mathematics.png and Image:Nuvola
apps edu mathematics-p.svg Original artist: David Vignoni (original icon); Flamurai (SVG convertion); bayo (color)
10.3
Content license