Variational Calculus and Discrete Alternatives

jplewis
CGIT/USC
The basic idea: instead of considering the change in a function wrt its arg, consider the variation in a functional wrt a variation in the function, this should be zero at the minimum of the functional.
The trick: if

f (x)q(x)dx = 0 regardless of what q is, then f must be zero.

y = f (x)

definitions:

F (y, y0 , x)

function of the function, eval at a particular point, scalar

q(x)

test function added to f, zero at ends

f = (f + q) f = q

variation of function at a particular point, scalar

E(F (f (x), f 0 (x), x)) e.g. E =

E[f ] =

F (y, y0 , x)dx

f 0 (x)2 dx

functional maps function to scalar

general form of functional, maps function to scalar
note E is function of f, not F

E = 0 = dE
f
d

f (x)dx =

condition for minimum

functional does not change with small variation

f dx

properties: variation commutes with integration, differentiation

F (y, y0 , x) = F (y + q, y 0 + q0 , x) F (y, y 0 , x)

dy0
dy
= F
 + F0
 + F dx 
y d
x d
y d
=

F q + F q0
y
y0

R
R

E =

F dx =

recognize 1. dx/d is zero, 2.  is small, so say  instead of d

now look at variation of functional

F dx = 

dE =
F q + F q0 dx
R F 0 y F y0 R d
0 q dx = y0 q
dx
y
a

F q
=
0
y0 
bR
R d F

F
dE
=

d
y
F d F = 0
0
y
dx y

represent variation of function (not functional) by taylor exp

dx y0

F q + F q0
y
y0

F
y0

dx

divide by  to get rate of change

now do integration by parts on second term

qdx

ab0 = ab

a0 b

because q is zero at both ends

qdx

and because q is arbitrary

euler-lagrange eqn

Worked Example: Laplace eqn in 1D

minimize

(f 0 (x))2 dx

should come out like

F (y, y0 , x) = y02
dy0

F = F0
dy d
= F0 q0 
dy
F q0 dx
dE =
d
dy0
F q0 dx = F q
d F qdx
dx dy0
dy0
dy0
b
F q
=0
dy0
a
d F qdx = 0
dE =
d
dx dy0
df
d2 f
d
= 2
= 2
= 22 f = 0
dx dx
dx2

d2 f
= 2 = 0
dx2

F = 2y0 = 2 df
dx
dy0
now change q to q
integration by parts
because q is zero at both ends
variation of functional is zero at minimum

Dan Piponis Physics Style approach to Variational Calculus

Need these:

u(z)

is variation on the left, dirac on the right

= (z w)

u(w)
d u(z)
dx

u(w)

true if a,b are zero at boundaries of integration

0
0 0
(this is like finite difference aDb = (Da) b = a D b

(z w)

dx

b =

dx

da

dx

which is true because if D = -1 1 ... then D = 1 -1 ...)

chain rule

u(x)
0
f (u(x)) = f (u(x))
u(z)

du(z)

( = derivative )

goal
min

0 2
(u ) dx

0 =

set first variational derivative to zero

u(z)

0
2
u (x) dx

0
2
u (x) dx

u(z)

u0 (x)
0
dx
2u (x)
u(z)

apply second relation above

d
0
2u (x)
(x z)dx
dx

apply third relation above

apply chain rule

= 2

integral and delta cancel each other, true at all z:

0
u (x)(x z)dx

dx
d2 u
dx2

= 0

Ken Anjyos approach

This approach starts directly with a general expression for the first variational derivative. (Setting this to zero is called first order optimality condition in the Chan&Wong paper).
Given a functional E that is an integral of a function F (~
x, u, u0 , u00 , . . .) of

the coordinates x1 , x2 , . . .,

the unknown solution u

and its nth order derivatives u0 = u1 , u00 = u2 ,

the general expression for the first variational derivative is:

E

= Fu +

(1)

n
n d
F un = 0
dxn

(where the sum is over the order of derivative of u).

0
If d
is read as identity this reduces to
dx0
E

(1)

n
n d
F un = 0
dxn

Apply this to the Laplace eqn: here n=1, F (x, u, u0 , . . .) = (u0 )2 , F 0 = 2u0 ,
u

E
u

= (1)n

d
dx

F 0 = 2
u

See 2d application below

d
dx

u0 = 2

d2
dx2

u = 0

Discrete alternative 1
By considering the function as being a point in hilbert space, the functional is simply a function of an (infinite dimensional) point, so standard function minimization can be used.

Discrete alternative 2
From Lanczos book,
y , y, y+

notation: yk1 , yk , yk+1

z = y0 = (y+ y)/(x+ x)

discrete derivative

E =

F (y, z, x)(x+ x)
dE =
d [F (y, z, x)x] +
d
F (y+ , z+ , x+ )x
dy+
dy+
dy+
d A+ d B
=
dy+
dy+
dA = F dy x + F dz x + F dx x
dy+
dy dy+
dz dy+
dx dy+

two terms in the sum involve y+

dx = 0, dy = 0,
dy+
dy+
dz = 1
dy+
x

= F 1 x
dz x
dB = F dy+ x + F dz+ x + F dx+ x
dy+
dy+ dy+
dz+ dy+
dx+ dy+

dx+
dy+
= 0,
= 1,
dy+
dy+
dz+
= 1
x
dy+

1 x
F x F
dy+
dz+ x
F x + F F
dy+
dz
dz+
F F
F 1
dy+
x
dz+
dz
F =
1 F
dy+
x dy0

total =

divide by x

as x 0

d
x
dx

Modified continuous
Take the Laplace eqn in 1D again, different derivation setup:

E[f ] =

df
dx

2

dx

(a) continuous

df 2
df
) +2
q0 + (q)2
dx
dx
df 0
q dx
E = E[f + q] E[f ] = 2
dx
df 0
dE = 2
q dx
d
dx

E[f + q] =

df
df 0
q dx =
q
dx
dx

dx = E[f ] + 2

df 0
q + Terms(q2 )
dx

get rid of q

d df qdx
dx dx

integration by parts

dE = 2
d df qdx
d
dx dx
2
d df = d f = 0
dx dx
dx2

for arbitrary q

Discrete alternatives 3,4

Is there a discrete equivalent to the variational calculus approach (which is: the functional is minimized then a variation of the function gives zero variation to the functional)?

Discrete 3
Now do something similar (to the modified continuous) discretely:

E[f ] = f 0 D0 Df

Df?

E = (f + q)0 D0 D(f + q) f 0 D0 Df

discrete variation

E[f + q] = f 0 D0 Df + 2f 0 D0 Dq + terms(q2 )

(b) similar to continuous (b)

E = 2f 0 D0 Dq

note this is a scalar

E = 2D0 Df
q

note this is a vector

want change in this wrt q

D =

D0 =

1
1

1
...

1
1

1
1
...

DD is 2 on the diagonal, -1 on the neighbors to diagonal, zero elsewhere, so -1 2 1 this is 2 . The minus sign does not matter since the rhs is zero, but it comes out the same way as the continuous, both are negative. (Bishop
appendix on variational calc talks about adjoint of derivative being the negative of the derivative).

Discrete 4
But easier to just directly minimize wrt f:
E[f ] = f 0 D0 Df

discrete

dE/df = 2D0 Df

TV deconvolution problem
Next apply these methods to the total variation blind deconvolution functional.

(Part of) that functional: E =

|u| =

Rp

2
u2
x + uy .

Apply the general solution expression (Ken Anjyos approach):

(1)n d Fun = 0
dx

E = F +
u
u

rewrite this for 2 dimensions, p==du/dx, q==du/dy, pn = dn u/dxn .

F (x, y, u, p, q) =

E = F +
u
u

p2 + q 2 .

n
(1)n d n Fpn +
dx

for TV problem n = 1, Fu = 0, Fp =

E = 1 d
u
dx

u
|u|

p2 +q2

1 d
dy

n
(1)n d n Fqn
dy

P
p

, Fq =

p2 +q2
q

p2 +q2

p2 +q2

p2 +q2

( d p + d q)
dx
dy

Variational version of Lagrange multiplier

With a constraint such as

pk = c, the variational formulation of this constraint (to be added to some equation with a Lagrange multiplier) is:

X
X
and then subtract the

(pk + pk ) = c

pk +

pk and the c from this.

pk = c

pk = 0. This is derived like so: