optical flow. distribution of apparent velocities of movement of brightness pattern in an image
TRANSCRIPT
![Page 1: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/1.jpg)
Optical Flow
![Page 2: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/2.jpg)
Optical Flow
• Distribution of apparent velocities of movement of brightness pattern in an image
![Page 3: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/3.jpg)
Optical Flow
• Perspective projection (pinhole camera)
• Instantaneous 2D velocity
![Page 4: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/4.jpg)
4
Definitions
The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ]
The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck ]
+ =
frame #1 frame #2flow field
![Page 5: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/5.jpg)
5
Ambiguity of optical flow
Frame 1
flow (1): true motionflow (2)
![Page 6: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/6.jpg)
Determining Optical Flow
Berthold K.P. Horn and Brian G. RhunckArtificial Intelligence Laboratory, MIT.
![Page 7: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/7.jpg)
Optical Flow• Optical flow field: Estimate the 2D motion field, which are
the 2D velocities for all visible points
• Two key problems– Determine what image property to track– Determine how to track it
![Page 8: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/8.jpg)
8
Brightness Constancy
0),,()1,,( tyxItvyuxI
frame t
frame t+1
)1,,( tvyux
),,( tyx
v
u
![Page 9: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/9.jpg)
Brightness Constancy
• Track points of constant brightness, assuming that surface radiance is constant over time
• Brightness constancy is often violated by Mother Nature, so the resulting optical flow field is sometimes a very poor approximation to the 2D motion field
1 2( , , 1) ( , , )I x y t I x u y u t
![Page 10: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/10.jpg)
Optical Flow
• The optical flow cannot be computed at a point independently of neighboring points without introducing additional constraints
![Page 11: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/11.jpg)
Brightness Constancy Constraint
• The brightness of a particular point in the pattern is constant
• Cannot determine the flow velocity locally without additional constraints
d0
d
E
t
d d0
d d
E x E y E
x t y t t
d
d
xu
t
d
d
yv
t 0x y tE u E v E
u
v
( , )x yE EtE
( , )u vConstraint line
![Page 12: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/12.jpg)
Brightness Constancy Constraint
• Normal Velocity• No constraint when the gradient magnitude is
zero
![Page 13: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/13.jpg)
13
Optical Flow Assumptions:
* Slide from Michael Black, CS143 2003
![Page 14: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/14.jpg)
14
Optical Flow Assumptions:
* Slide from Michael Black, CS143 2003
![Page 15: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/15.jpg)
Smoothness Constraint
• Constraint Assumption– Neighboring points on the objects have similar
velocities and – The velocity field of the brightness patterns in the
image varies smoothly almost everywhere– Discontinuities in flow can be expected where one
object occludes another
• Algorithms based on smoothness constraint are likely to have difficulties with occluding edges
![Page 16: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/16.jpg)
Smoothness Constraint
• Minimize the square of the gradient magnitude of the optical flow velocity
• Sum of the squares of Laplacians of x- and y- components of the flow
22u u
x y
22v v
x y
2 22
2 2
u uu
x y
2 22
2 2
v vv
x y
![Page 17: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/17.jpg)
Estimate the Partial Derivatives
![Page 18: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/18.jpg)
Estimate the Laplacian of Flow Velocities
![Page 19: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/19.jpg)
Minimization
• Minimize the sum of errors in rate of change of image brightness
• And measure of departure from smoothness
• Total error to be minimized
![Page 20: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/20.jpg)
Minimization
• Good (u,v) occurs at
![Page 21: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/21.jpg)
Difference of Flow from Local Average
• (u,v) lies in the direction towards the constraint line along a line that perpendicular to it
• a^2 plays significant role when Ex, Ey is small: roughly equal to the expected noise in the estimate of
0x y tE u E v E
![Page 22: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/22.jpg)
Iterative Solution
![Page 23: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/23.jpg)
24
Optical Flow: 1D Case Brightness Constancy Assumption:
)),(()),(()( dttdttxIttxItf
0)(
txt t
I
t
x
x
I
Ix v It
x
t
I
Iv
{0
)(
t
xfBecause no change in brightness with time
![Page 24: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/24.jpg)
25
v
?
Tracking in the 1D case:
x
),( txI )1,( txI
p
![Page 25: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/25.jpg)
26
v
xI
Spatial derivative
Temporal derivativetI
Tracking in the 1D case:
x
),( txI )1,( txI
p
tx x
II
px
t t
II
x
t
I
Iv
Assumptions:• Brightness constancy• Small motion
![Page 26: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/26.jpg)
27
Tracking in the 1D case:
x
),( txI )1,( txI
p
xI
tI
Temporal derivative at 2nd iteration
Iterating helps refining the velocity vector
Can keep the same estimate for spatial derivative
x
tprevious I
Ivv
Converges in about 5 iterations
![Page 27: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/27.jpg)
28
Algorithm for 1D tracking:
x
tI
Ivv
Compute local image derivative at p: Initialize velocity vector: Repeat until convergence:
Compensate for current velocity vector: Compute temporal derivative:Update velocity vector:
For all pixel of interest p:
xI
0v
)1,()1,(' tvxItxI
),()1,(' tpItpII t
Need access to neighborhood pixels round p to compute Need access to the second image patch, for velocity compensation:
The pixel data to be accessed in next image depends on current velocity estimate (bad?)
Compensation stage requires a bilinear interpolation (because v is not integer) The image derivative needs to be kept in memory throughout the iteration
process
Requirements:
xI
xI
![Page 28: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/28.jpg)
Iterative Solution
• Very costly to solve these equations simultaneously • Iterative algorithms, such as Gauss-Seidel method
It is interesting to note that the new estimates at a particular point do not depend directly on the previous estimate at the same point
![Page 29: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/29.jpg)
Fill in Uniform Region
• Uniform region: brightness gradient is zero
• No local information to constrain the vecocity• Fill in uniform region– Given the values on region boudnary– Solve the Laplace equation under Dirichlet
boundary condition
0x y tE u E v E
2 *min | |f
f with f f
![Page 30: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/30.jpg)
Tightness of Constraint
• In general, the solution is most accurately determined in the region where the brightness is not too small and varies in direction from point to point– Constraint information is available in a relative
small neighborhood
![Page 31: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/31.jpg)
Choice of Iterative Scheme
• How to interlace the iterations with the time step? – Get stable values before going to next frame– Only one iteration per time-step gives good initial
guess • Ability to deal with more images per unit time• Better estimation in uniform regions because of the
noise in measurements of images will be independent and tend to cancel out
![Page 32: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/32.jpg)
Experiment
• Very low resolution video, say, 32*32
• A rotating ball with smoothly varying patterns, but no shading
![Page 33: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/33.jpg)
Results I
• Simple linear translation of the entire pattern• Estimate between two images• Showing the velocity • Few changes occur after 32 iterations when
errors ~ 10%
![Page 34: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/34.jpg)
Results I
1 time step 4 time steps
![Page 35: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/35.jpg)
Results I
16 time steps 64 time steps
![Page 36: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/36.jpg)
Results II
• Only one iteration per time-step• More rapid • Few changes after 16 iteration when errors ~
7%
![Page 37: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/37.jpg)
Results II
1 time steps 4 time steps
![Page 38: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/38.jpg)
Results II
16 time steps 64 time steps
![Page 39: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/39.jpg)
Results III
• Simple rotation and contraction of the brightness pattern
![Page 40: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/40.jpg)
Result III
initial 32 time steps
![Page 41: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/41.jpg)
Result IV
• Rigid body motion– Laplacian for one of the velocity components
becomes infinite on the occluding bound– Reasonable results are still obtained with finite
velocities
![Page 42: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/42.jpg)
Results IV
Rotating cylinder Rotating sphere
![Page 43: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/43.jpg)
Results IV
Exact flow pattern for a rotating cylinder and rotating sphere
![Page 44: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/44.jpg)
Summary
• A method for computing optical flow from a sequence of images
• Two components of flow velocity• Additional constraint: smoothness • Iterative method
![Page 45: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/45.jpg)
46
Horn&Schunck
Optical Flow Research: Timeline
Lucas&Kanade
1981 1992 1998 now
Benchmark:Barron et.al.
Benchmark:Galvin et.al.
many methods
more methods
A slow and not very consistent improvement in results,
but a lot of useful ingredients were developed
Seminal papers
![Page 46: Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image](https://reader035.vdocuments.mx/reader035/viewer/2022062518/5697bf891a28abf838c8a354/html5/thumbnails/46.jpg)
http://people.csail.mit.edu/celiu/ECCV2008/z`z`