optical flow estimation using variational techniques darya frolova

Optical Flow Estimation Optical Flow Estimation using using

Variational Techniques Variational Techniques

Darya FrolovaDarya Frolova

Part IPart I

Mathematical Preliminaries

AgendaAgenda

Minimization problem: ? )( min

xFAx

Investigate the existence of a solution with respect to reasonable assumptions on functional F and space A

- conditions for A and F,- main theorem: existence and uniqueness of minimum

Optical Flow - what is this? - different approaches - formulation of minimization problem for Optical Flow - existence of solution (using main theorem)

minimization problem: ? )( min

xFAx

TheoremA is a compact set, function F is continuous on A.

Ax 0 Then such that )( )( 0 min xFxFAx

What happens when A is not a compact?

What happens when functional F is not continuous?

Does the minimum exist?

a bx0

F (x)

New TheoremNew Theorem

Theorem

A is a compact set

Ax 0 Then such that )( )( 0 min xFxFx

function F is continuous on A

A = BV (Ω) – space of functions of bounded variation

functional F is coercive and lower semicontinuous on A

Bounded Variation -1D caseBounded Variation -1D case

Rbaxf ],[:)(

f has bounded variation over ],[ ba if exists const M such that

|)()(| ... |)()(||)()(| 1121 nxfbfxfxfafxf

for all bxxxa n 121 ...

Consider

a x1 x2 xn -1 b……

)(af )( 1nxf )(bf

)( 2xf)( 1xf

M | )( )(| 1

1

n

iii xfxf

Definition

Bounded VariationBounded Variation

Definition

The space of functions of bounded variation on Ω is denoted BV (Ω)

Total variation of function f can be represented as follows:

dx

dffD

sup|)(|φ

where - function φ is compactly supported (is zero outside of a compact set),

] 1 ,1[ -

If f is differentiable, then dx

df

|f

dx

df

Supremum for

and

1

-1

f(x)

sign(df/dx)

dx

dfsign

dx

df

dx

d

Then variation of function f =

max loc min loc

)( )(x x

xfxf

Function f does not need to be differentiable

Bounded Variation – ND caseBounded Variation – ND case

sup |)(| dxdivffD

)( x

N

1i i

i xdiv

1 || ,)( ), ..... ,( )(

101

L

NN C

bounded open subset, function NR )( 1 Lf

Variation of over f

where

φ

Bounded Variation – exampleBounded Variation – example

xxxf 1sin)( 21(

xxxxf 1cos1sin2)(

1

0

1

0

|1cos1sin2||)(| dxxxxdxxf

functions under integral are bounded on [0,1]

xxxf 1sin)( 2(

xxxxf 11cos1sin)(

1

0

1

0

|11cos1sin||)(| dxxxxdxxf - problem at x = 0, 2nd term is not bounded

DefinitionsDefinitions

Definition Banach space – complete, normed linear space

Definition

Space X is said to be complete, when any Cauchy sequence xn from X converges

DefinitionConsider sequence xn from X.

If N 0 natural, such that for any m, n > N

|| || mn xx , then xn - Cauchy sequence

Space of functions of bounded variation BV(Ω) is a Banach space

Definition

Functional f is coercive if

)( lim||

xfx

DefinitionConsider sequence xn. Denote . inf n

knk xX

Then lower limit of sequence xn is a limit of sequence Xk:

nknk

kk

nn

xXx inflimlimlim

+ +

x

f (x)

Dual SpaceDual Space

Let X denote a real Banach space. Dual space of X : X * – space of linear bounded operators

X * = f : X → R

Definition

Linear operator:

f (αx + βy) = α f (x) + β f (y)

Bounded operator:

bounded set converts to bounded set

|||| ||)(||

such that 0

xcxfx

c

Dual spaceDual space

Definition

X is called reflexive if ( X *) *

Reflexive Not reflexive

e.g. finite-dimensional (normed) spaces,

Hilbert spaces

e.g. the space of sequences ℓ∞

||max nxℓ∞ = xn :

( if bidual space of X is equal to X )

Topologies on XTopologies on X

Definition Sequence xn from X strongly converges to point x if

)(n 0 | | Xn xx

If sequence strongly converges, then it weakly converges

Definition Sequence xn from X weakly converges to point x if

)(n )( )( xfxf n

for every functional f from X*

( if converges sequence of real numbers f (xn) for any linear bounded f )

Topologies on XTopologies on X**Definition

Sequence fn from X* strongly converges to f if

)(n 0 | | * Xn ff

Definition Sequence fn from X weakly converges to f if

)(n )( )( fgfg n

for every g from (X*)* (bidual space of X)

Definition Sequence fn from X weakly* converges to f if

)(n ffn

for every x from X

Direct MethodDirect Method

Problemf : X → R , where X is a Banach space

inf f (x)xX

Does the solution exist?

The proof consists of three steps , which is called

Direct Method of Variational Calculus

TheoremTheorem

If

functional f is coercive and lower semicontinuous on X,

X is Banach and reflexive space

Thenfunctional f has its minimum on X :

)( )( such that min00 xfxfXxXx

Direct MethodDirect Method

Step A Construct minimizing sequence xn for functional f :

)( )( inflim xfxfXx

nn

Step Ba) If f is coercive, then minimizing sequence is bounded

Step C If f is lower semicontinuous at point x0 then x0 is a point of minimum: )( )( min0 xfxf

Xx

b) If X is reflexive, then there exists weakly convergent subsequence of minimizing sequence:

Xxxjn 0

weakly exists

Step A

There exists an infimum of the set f (x), x X R

and exists a sequence R , which converges to this infimum

f (x), x X R

Step B (a)If f is coercive, then minimizing sequence is bounded :

CxC n || such that 0const

Proof )( )( inflim xfxfXx

nn

Consider minimizing sequence xn:

Assume that it is not bounded: CxnC n || such that 0 (equivalently, for any ball with radius C = 1,2… there

exists an element outside this ball)cnx

cnxIf we form a subsequence of these , it will satisfy: ) ( || cn nxc

f is coercive

)( lim||

xfx

)( lim||

c

cn

nx

xf

But is subsequence of f (xn), that is why )(cnxf

)( lim||

nx

xfn

So, unbounded xn cannot be minimizing sequence.

To prove

Step B (b)

we can conclude that exists x0 X and exists subsequence

such that:

jnx

0weakly xx

jn

We proved that minimizing sequence is bounded.

If X is reflexive ( ( X* )* = X ) then using theorem about weak sequential compactness, which states that:

for any bounded sequence in reflexive Banach space there exists weakly convergent subsequence

If X is reflexive, then there exists weakly convergent subsequence of minimizing sequence

To prove

Proof

Step C

If f is lower semicontinuous at x0 x0 is a minimum point of f

0weakly xx

jn We proved the existence of such that : jnx

)()( )( inflimlim xfxfxfXx

nn

nn

j

j

If there exists limit of f (xn) then there exists a limit of its subsequence

and these limits are equal:

)( jnxf

On the other hand, from the lower semicontinuity it follows that )( )( 0lim xfxfjn

Hence )( )( inf0 xfxfXx

)( )( min0 xfxfXx

This means that

To prove

Proof

Part IIPart II

Optical Flow

Image SequenceImage Sequence

Sequence of images contains information about the scene,We want to estimate motion (using variational formulation)

2D motion field2D motion field

Optical center

2D motion field

Projection on the image plane of the 3D velocity of the scene

3D motion field

Image intensity

I1

I2

Motion vector - ?

Optical FlowOptical FlowWhat we are able to perceive is just an apparent motion, called

Optical flow

(motion, observable only through intensity variations)

Intensity remains constant – no motion is perceived

No object motion, moving light source produces intensity variations

Optical flow-methodsOptical flow-methods

Correlation-based techniques - compare parts of the first image with parts of the second in terms of the similarity in brightness patterns in order to determine the motion vectors

Feature-based methods - compute and analyze Optical Flow at small number of well-defined image features

Gradient-based methods - use spatiotemporal partial derivatives to estimate flow at each point

Brightness constancyBrightness constancy

Intensity of a point keeps constant along its trajectory

(reasonable for small displacements)

),( xtI intensity of the pixel ),( 21 xxx at time t

Start from point x0 at time t0. Trajectory → ( t, x (t) )

txtItxtI allfor ),( ))( ,( 00) ,( ) )( ,( 0000 xttxt

0at 0 ttt

II

dt

dx

Differentiating →

)( )( 00 tdt

dxtv We will search the Optical Flow as the velocity field:

Brightness constancyBrightness constancy

Given sequence I (t, x) and time t0

Find the velocity v (x) such that:

We need to find the velocity field - 2 componentsWe have 1 scalar equation

Can find only “normal flow”

Component in the direction of gradient I

0 ),( ),( )( 000 xtIxtIxv t

Optical Flow constraint (OFC)

Aperture problemAperture problem

Solving the aperture problemSolving the aperture problem

Second order derivative constraint : conservation of along trajectories

I

0 ) ,(

xtdt

Id

Rigid deformations are not considered (object moves locally in one direction)

0

0

2

1

2221

1211

2

1

tx

tx

xxxx

xxxx

I

I

v

v

II

II

Sensitive to noise

Weighted least squaresWeighted least squares

Velocities are constant in small window (spatial neighborhood)

dxIIvxwrx

tv

)( 2

),(Ball

2

0

inf

w (x) is a window function ( gives more influence to the constraint at the center of the neighborhood than at the periphery)

Too local, no global regularity

Regularizing the velocity fieldRegularizing the velocity field

)( )( inf vSvAv

smoothing term

dxvdxIIvj

tv

j ||

22

1

2

inf

A (v) S (v)

Velocity should change slowly in spatial domain (in image plain)

Optical Flow constraint

Horn and Schunck

DiscontinuitiesDiscontinuities

But smoothing term does not allow to save discontinuities

Discontinuities near edges are

lost

Synthetic example(method of Horn and Schunck)

Discontinuity-preserving approachDiscontinuity-preserving approach

2

inf

dxIIv tv

dxvj

j ||

22

1

dxvj

j) || (

2

1

Where function η permits noise removal and edge conservation

2

inf

dxIIv tv

dxvj

j ||

22

1

dxv div ) )( (

[Black et.al, Cohen, Kumar, Balas, Tannenbaum, Blanc-Feraud]

[Suter, Gupta and Prince, Guichard and Rudin]

Discontinuity-preserving approach Discontinuity-preserving approach summarysummary

Given: sequence I (t, x)

Find: velocity field v that minimizes the energy functional E

22

1

||)( )( | | )( vDIcDvIDIvvE h

j

st j

S (v)

Smoothing term, we need to find

conditions on η for saving discontinuities

H (v)

Is related to homogeneous

regions

A (v)

L1 norm of the Optical Flow

constraint

Smoothing termSmoothing term

)( )(2

1

j

jDvvS

Function η : R+ → R+ is strictly convex, nondecreasing

η (0) = 0 (without loss of generality)

)( lim ss

bassbas )( η (s)

bas

bas

Homogeneous termHomogeneous term

2||)( )( vDIcvH

Idea: if there is no texture (there is no gradient), then there is no possibility to correctly estimate the flow field

So, we may force it to be zero

0 )( and 1 )( limlim0

scscss

)( Cc

]1 ; [ )( such that const exists cc mxcm

Without loss of generality:

Existence of solutionExistence of solution

Theorem Under the following hypotheses:

]1 ; [ )( such that const exists cc mxcm

)( Cc

bassbas )(

)( lim ss

η : R+ → R+ is strictly convex, nondecreasing, η (0) = 0

22

1

||)( )( | | )( min vDIcDvIDIvvE h

j

st j

The minimization problem

has a unique solution in BV(Ω)

Existence of a solutionExistence of a solution

ProofUsing direct method of Variational Calculus:

Step A Construct minimizing sequence vn for functional E :

)( )( inflim)(

vEvEBVv

nn

Step Ba) If E is coercive, then minimizing sequence is bounded

b) If BV (Ω) is reflexive, then there exists weakly convergent subsequence of minimizing sequence:

Step C If E is lower semicontinuous at point x0 then x0 is a point of minimum: )( )( min

)(0 vEvE

BVv


22

1

||)( )( | | )( vDIcDvIDIvvE h

j

st j

)( lim ss

v v v

+ +

c is bounded

+

So, functional E is coercive




)( )( inflim)(

vEvEBVv

nn




)(0 vEvE

BVv


Space of functions of bounded variation is not reflexive: ( BV *)* ≠ BV

But it has such a property that

every bounded sequence Ij from BV (Ω) has a subsequence

that weakly* converges to some element I from BV (Ω)




)( )( inflim)(

vEvEBVv

nn




)(0 vEvE

BVv

The End

optical flow estimation using variational techniques darya frolova

Documents