differentiation in linear spaces - idsim/cs724/smdc1.pdfsimilarly, the g^ateaux derivative is...

Differentiation in Linear Spaces - I

Prof. Dan A. Simovici

UMB

1 / 28

Outline

1 The Frechet and Gateaux Differentiation

2 Gateaux Differential

2 / 28

The Frechet and Gateaux Differentiation

Definition

Let (S , ‖ · ‖) and (T , ‖ · ‖) be two normed spaces and let X be an openset in (S , ‖ · ‖).A function f : X −→ T is Frechet differentiable at x0, where x0 ∈ X , ifthere exists a linear operator (Dx f )(x0) : X −→ T such that

limh→0

‖ f (x0 + h)− f (x0)− (Dx f )(x0)(h) ‖‖ h ‖

= 0.

The operator (Dx f )(x0) : X −→ T is referred to the Frechet derivative atx0.

3 / 28


The Case of Functions f : R −→ R

If f : R −→ R is a function differentiable at x0 and having the derivativef ′(x0) at x0, then

limh→0

f (x0 + h)− f (x0)

h= f ′(x0).

Therefore,

limh→0

f (x0 + h)− f (x0)− f ′(x0)h

h= 0,

hence

limh→0

|f (x0 + h)− f (x0)− f ′(x0)h||h|

= 0,

4 / 28


This shows that for functions of the form f : R −→ R theoperator (Dx f )(x0) from the definition of the Frechet differentialapplied to h, (Dx f )(x0)(h) is simply the multiplication of h byf ′(x0);

5 / 28


Frechet Differential

The function δf : X × S −→ T defined by δf (x0; h) = (Dx f )(x0)(h) is thedifferential of f at x0. Note that the differential is linear in its secondargument h. To emphasize the distinct roles played by x0 and h thearguments of the differential are separated by a semicolon.We stress that (Dx f )(x0) is a linear operator in L(S ,T ).If (Dx f )(x0) is continuous on X , where X ⊆ S , we say that f iscontinuously differentiable on X .

6 / 28


Example

A constant function k : S −→ T is Frechet differentiable at every point x0

of S and (Dx f )(x0) = 0.

7 / 28


Example

If f : S −→ T is a continuous linear mapping thenf (x0 + h)− f (x0) = f (h) for x0 ∈ S , so (Dx f )(x0) = f .For instance, consider the linear operator f : C [a, b] −→ C [a, b] defined as

(fu)(x) =

∫ b

aK (x , s)u(s) ds.

Its Frechet derivative is (Dx f )(u) = f (u).

8 / 28


We denote by o (with or without subscripts) a function o : S −→ R thathas the property:

limh→0

o(h)

‖ h ‖= 0.

The introduction of the o notation enables us to say that f is Frechetdifferentiable at x0 if and only if there exists a linear transformation(Dx f )(x0) in Hom(S ,T ) such that

‖ f (x0 + h)− f (x0)− δf (x0; h) ‖= o(h),

or‖ f (x0 + h)− f (x0)− (Dx f )(x0)(h) ‖= o(h).

The Frechet derivative (Dx f )(x0) will be simply denoted by (Dx f )(x0)when x is clear from the context.

9 / 28


Theorem

Let (S , ‖ · ‖) and (T , ‖ · ‖) be two normed spaces and let X be an opensubset of S. If a function f : X −→ T has a Frechet differential, whereX ⊆ S, then this differential is unique.

10 / 28


Proof

Suppose that both δf (x0; h) and δ1f (x0, h) are differentials of f at x0. Wehave

‖ δf (x0; h)− δ1f (x0; h) ‖ 6 ‖ f (x0 + h)− f (x0)− δf (x0; h) ‖+ ‖ f (x0 + h)− f (x0)− δ1f (x0; h) ‖= o(h).

11 / 28

Gateaux Differential

Definition

Let (S , ‖ · ‖) and (T , ‖ · ‖) be normed F-linear spaces, X be an open setin (S , ‖ · ‖) and let f : X −→ T be a function.The function f is Gateaux differentiable in x0, where x0 ∈ X if there existsa linear operator (Dx f )(x0) : S −→ T such that

(Dx f )(x0)(u) = limt→0

f (x0 + tu)− f (x0)

t

for every u ∈ S . The linear operator (Dx f )(x0) is the Gateaux derivative off in x0.The Gateaux differential of f at x0 is the the linear operator δf (x0; h)given by

δf (x0; u) = limt→0

f (x0 + tu)− f (x0)

t.

12 / 28


Notations

The Gateaux differential is denoted by the same δf (x0; h) as theFrechet differential.

Similarly, the Gateaux derivative is denoted by (Dx f )(x0), as wedenoted the Frechet derivative and the subscript x will be omittedwhen possible.

The specific differential (or derivative) we are referring to will resultfrom the context.

13 / 28


Remarks

The function f : S −→ T is Gateaux differentiable in x0 if for everyε > 0 there exists δ(x0, u) > 0, which depends on x0 and u such thatt < δ(x0, u) implies∣∣∣∣∣∣ f (x0 + tu)− f (x0)

t− (Dx f )(x0)(u)

∣∣∣∣∣∣ < ε.

The function f : S −→ T is Frechet differentiable in x0 if for everyε > 0 there exists δ(x0) > 0, which does not depend on u such that

t < δ(x0) implies∣∣∣∣∣∣ f (x0+tu)−f (x0)

t − (Dx f )(x0)(u)∣∣∣∣∣∣ < ε.

In this sense, Frechet differentiability implies that the convergence off (x0+tu)−f (x0)

t to (Dx f )(x0)(u) is uniform relative to u.

14 / 28


Indeed, if h ∈ S we can write h =‖ h ‖ uh, where ‖ uh ‖= 1. Thus, ifh→ 0, we have ‖ h ‖→ 0. Since f is Gateaux uniformly differentiable atx0, for every ε > 0 there is δ(x0) such that if ‖ h ‖< δ(x0) then

‖ f (x0 + h)− f (x0)− (Dx f )(x0)(h) ‖‖ h ‖

=‖ f (x0 + h)− f (x0)− ‖ h ‖ (Dx f )(x0)(uh) ‖

‖ h ‖

=∣∣∣∣∣∣ f (x0 + h)− f (x0)

‖ h ‖− (Dx f )(x0)(uh)

∣∣∣∣∣∣ < ε

which shows that f is Frechet differentiable at x0.

15 / 28


Example

Let a be a vector in Rn. Define the function f : Rn −→ R as f (x) = x′a.We have:

(Dx f )(x0)(u) = limt→0

f (x0 + tu)− f (x0)

t

= limt→0

(x0 + tu)′a− x′0a

t

= limt→0

tu′a

t= u′a.

16 / 28


Example

Let A ∈ Rn×n be a matrix and let f : Rn −→ R be the functionalf (x) = x′Ax. We have (Df )(x0) = x′0(A + A′).By applying the definition of Gateaux differential we have

(Df )(x0)(u) = limt→0

f (x0 + tu)− f (x0)

t

= limt→0

(x′0 + tu′)A(x0 + tu)− x′0Ax0

t

= limt→0

tu′Ax0 + tx′0Au + t2u′Au

t= u′Ax0 + x′0Au = x′0A′u + x′0Au

= x′0(A + A′)u,

which yields(Df )(x0) = x′0(A + A′).

If A ∈ Rn×n is a symmetric matrix and f : Rn −→ R is the functionalf (x) = x′Ax, then (Df )(x0) = 2x′0A. 17 / 28


Example

Let (S , ‖ · ‖) be a normed space. The norm ‖ · ‖: S −→ R>0 is notGateaux differentiable in 0S . Indeed, suppose that ‖ · ‖ were differentiablein 0S , which would mean that the limit:

limt→0

‖ tu ‖t

= limt→0

|t|t‖ u ‖

would exist for every u ∈ S , which is contradictory.However, the square of the norm, ‖ · ‖2 is differentiable in 0S because

limt→0

‖ tu ‖2

t= lim

t→0t ‖ u ‖= 0.

18 / 28


Example

Consider the norm ‖ · ‖1 on Rn given by

‖ x ‖1= |x1|+ . . .+ |xn|

for x ∈ Rn. This norm is not Gateaux differentiable in any point x0 locatedon an axis. Indeed, let x0 = aei be a point on the i th axis. The limit

limt→0

‖ x0 + tu ‖1 − ‖ x0 ‖1

t

= limt→0

‖ aei + tu ‖1 − ‖ aei ‖1

t

= limt→0

|t||u1|+ · · ·+ |t||ui−1|+ (|t||ui | − |a|) + |t||ui+1|+ · · ·+ |t||un|t

does not exists, so the norm ‖ · ‖1 is not differentiable in any of thesepoints.

19 / 28


Theorem

Let (S , ‖ · ‖) and (T , ‖ · ‖) be two normed F-linear spaces, X be an openset in (S , ‖ · ‖) and let f : X −→ T be a function.If f is Frechet differentiable in x0 ∈ X , then it is also Gateauxdifferentiable in x0 and the two differentials are the same.

20 / 28


Proof

Let f be Frechet differentiable in x0, that is,‖ f (x0 + h)− f (x0)− δf (x0; h) ‖= o(h), where δf (x0, h) is the Frechetdifferential.For the Gateaux differential δ′(x0; h) of f in x0 we have

‖ δ′f (x0; h)− δf (x0; h) ‖

=∣∣∣∣∣∣ limt→0

f (x0 + tu)− f (x0)

t− δf (x0; h)

∣∣∣∣∣∣= lim

t→0

∣∣∣∣∣∣ f (x0 + tu)− f (x0)− δf (x0; th)

th

∣∣∣∣∣∣ ‖ h ‖,

and limt→0

∣∣∣∣∣∣ f (x0+tu)−f (x0)−δf (x0;th)th

∣∣∣∣∣∣ = 0 because limt→0 ‖ th ‖= 0. Thus,

δ′f (x0; h) = δf (x0; h).

21 / 28


Theorem

Let (S , ‖ · ‖) and (T , ‖ · ‖) be two normed F-linear spaces, X be an openset in (S , ‖ · ‖) and let f : X −→ T be a function.If f is Gateaux differentiable on X , then

‖ f (u)− f (v) ‖6‖ u − v ‖ sup{f ′(au + (1− a)v) | a ∈ [0, 1]}.

22 / 28


Proof

Let w ∈ X such that ‖ w ‖= 1 and ‖ f (u)− f (v) ‖= (w , f (u)− f (v)).Define the the real-valued function g as g(t) = (w, f (u + t(v − u))) fort ∈ [0, 1]. We have the inequality

‖ f (u)−f (v) ‖= (w, f (v)−f (u)) = |g(1)−g(0)| 6 sup{|g ′(t)| | t ∈ [0, 1]}.Since

g ′(t) =

(w ,

d f (u + t(v − u))

d t

)=

(w , lim

r→0

f (u + (t + r)(v − u))− f (u + t(v − u))

r

)=

(w , f ′u+t(v−u)(v − u)

),

we have |g ′(t)| 6‖ f ′u+t(v−u)(v − u) ‖, hence

|g ′(t)| 6 ‖ f ′u+t(v−u)(v − u) ‖6 ‖ f ′u+t(v−u) ‖‖ v − u ‖ .

23 / 28


Theorem

Let (S , ‖ · ‖), (T , ‖ · ‖) be normed F-linear spaces, X be an open set in(S , ‖ · ‖) and let f : X −→ T be a function.If f is Gateaux differentiable at x0 ∈ X and the Gateaux derivative iscontinuous in x0, then f is Frechet differentiable in x0.

24 / 28


For v ∈ S define the function gv : [0, 1] −→ T be the function defined asgv (t) = f (x0 + tv)− f (x0)− t(Df )(x0)v ; we have g(0) = 0. From thecontinuity of the Gateaux derivative it follows that

‖ gv (1)− gv (0) ‖ = f (x0 + v)− f (x0)− f ′(x0)v

6 ‖ v ‖ sup{(Df )(x0 + tv)− (Df )(x0)} = o(‖ v ‖).

Thus, gv = 0.

25 / 28


Example

A homogeneous polynomial u of degree k has the property

u(tx1, . . . , txn) = tku(x1, . . . , xn)

for t, x1, . . . , xn ∈ R.Let p(x1, x2) and q(x1, x2) be two homogeneous polynomials of degrees rand s, respectively, where r > s + 1, and let f : R2 −→ R be the functiondefined by

f (x1, x2) =

{p(x1,x2)q(x1,x2) if x 6= 02,

0 otherwise,

where x 6= 02 implies q(x) 6= 0.

26 / 28


Example (cont’d)

e claim that f is Gateaux differentiable but not Frechet differentiable.Indeed,

limt→0

f (tu1, tu2)

t= lim

t→0tr−s−1f (u1, u2) = 0,

and the constant function 0 is linear in u.Frechet differentiability in 02 requires the existence of a linear operatorg : Rn −→ R such that

limh→02

‖ f (h)− f (x0)− g(h) ‖‖ h ‖

= limh→02

‖ f (h)− g(h) ‖‖ h ‖

= 0,

which is impossible because f (h) grows faster than a linear function in hassuming that r > s + 1. Thus, f is not differentiable Frechet.

27 / 28


Definition

Let X be a open subset in a real normed linear space S and let f : X −→ Rbe a functional that has a Gateaux derivative on X . A stationary point off is a point x0 ∈ X such that δf (x0; h) = 0 for every h ∈ X .

The term critical point is also used as an alternative to stationary point.

28 / 28

differentiation in linear spaces - idsim/cs724/smdc1.pdfsimilarly, the g^ateaux derivative is...

Documents