moving object detection
DESCRIPTION
moving object detection using optical flowTRANSCRIPT
Moving Object Detection
04/12/2023DSG, CEERI Pilani2
Tracking Tracking is the problem of estimating the
trajectory of an object in the image plane as it moves around a scene.
04/12/2023DSG, CEERI Pilani3
What is Motion Tracking…?• Technologies that collect data on human movement
(input) used to control sounds, music, recorded or projected text, video art, stage lighting (output) via performer actions / gestures / movements / bio-data.
04/12/2023DSG, CEERI Pilani4
What is Motion Tracking…?• other uses:
• Animation modeling (motion capture)
• Scientific research (musicology, semantics, ergonomics, medicine, sports medicine, architecture)
• Therapy for physically and mentally handicapped
04/12/2023DSG, CEERI Pilani5
Motion tracking vs. Motion captureMotion capture
• Tracks location of fixed positions on body
• Highly accurate, expensive ($200k-2m)
• Generally not realtime
• Used for data collection (research) and for making human or animal motion in animations (films, games, etc.) more realistic
Motion tracking
• less equipment, less data,
• less cost ($1k-2k)
• concerned with motion qualities like dynamic, direction of motion
• real time
• used for live applications: installation art, dance, theater and music
04/12/2023DSG, CEERI Pilani6
Motion capture Motion capture is defined as "The creation of a 3D
representation of a live performance." Tracks location of fixed positions on body with reflective
markers 12-24 cameras, each lens is ringed with infrared lights
04/12/2023DSG, CEERI Pilani7
Motion capture Motion capture is used to be considered a tool
for creating animation.
04/12/2023DSG, CEERI Pilani8
Motion capture Typical uses
Human movement research (sports, musicology, ergonomics, medicine,...)
Film and Animation -- often used with 3-D animation (modeling) programs to make animations
maya (http://www.animationarena.com)
houdini (http://www.sidefx.com)
04/12/2023DSG, CEERI Pilani9
Motion capture
Vicon is a leading company in motion capture
04/12/2023DSG, CEERI Pilani10
Motion tracking
input
physical
human action
media output
sounds,
music, text,
projections,
lighting
04/12/2023DSG, CEERI Pilani11
Motion tracking
input
physical
human action
computersensor
(e.g. video camera)
output device (e.g. loud speakers)
media output
sounds,
music, text,
projections,
lighting
analogue to digital
conversion
digital to analogue
conversion
04/12/2023DSG, CEERI Pilani12
Digital vs. Analogue Analogue data
• hard to reproduce• "rich data" (infinite values)• very high resolution• more details• contaminated data (becomes noisy, but rarely fails completely)
Digital data
• easy to reproduce• lower resolution, less human-feel.• easy to store• easy to process• contaminated data remains clean (errors can be filtered) or signal fails altogether
04/12/2023DSG, CEERI Pilani13
What you need to track motion Computer needs video input
Digital video (Firewire, USB2)
+ digital cameras (camcorder, webcams)
+ low noise
+ works with laptops
- latency issues
- image resolution issues (smaller chip sizes)
- limited cable length
Analog video
+ "unlimited" cable length
+ lower latency
+ even digital cams usually have analog output
- cost more (although many older cameras work quite well)
- works less well with laptops i.e. need an external or internal framegrabber
04/12/2023DSG, CEERI Pilani14
Palindrome Intermedia Performance Group Krisztina de Chatel Igloo Ventura Dance (Pablo Ventura) Robert Lepage André Werner Marlon Barrios Solano La la la Human Steps Georg Hobmeier Leine Roebana Dans Kompanie Troika Ranch Blue Man Group you
Who is using motion tracking?
The Problem in Motion Tracking Given a set of images in time which are similar
but not identical, derive a method for identifying the motion that has occurred (in 2d) between different images.
Motion Detection and Estimation in Literature
Image differencing based on the threshold difference of successive images difficult to reconstruct moving areas
Background subtraction foreground objects result by calculating the difference
between an image in the sequence and the background image (previously obtained)
remaining task: determine the movement of these foreground objects between successive frames
Block motion estimation Calculates the motion vector between frames for sub-
blocks of the image mainly used in image compression too coarse
What Is Optical Flow? Optical flow is the displacement field for
each of the pixels in an image sequence.
For every pixel, a velocity vector is found which says:
how quickly a pixel is moving across the image
the direction of its movement.
dt
dy
dt
dx,
04/12/2023DSG, CEERI Pilani18
Optical Flow Examples
Image #1 Optical Flow Field Image #2
Optical Flow ExamplesTranslation Rotation Scaling
04/12/2023DSG, CEERI Pilani20
Estimation of the optical flow Sequences of ordered images allow the
estimation of motion as either instantaneous image velocities or discrete image displacements.
The optical flow methods try to calculate the motion between two image frames which are taken at times
t and t + δt at every voxel position.
04/12/2023DSG, CEERI Pilani21
Voxel Position A voxel (volumetric pixel or Volumetric Picture
Element) is a volume element, representing a value on a regular grid in three dimensional space.
A series of voxels in a stack with a single voxel highlighted
04/12/2023DSG, CEERI Pilani22
Estimation of the optical flow Optical Flow methods are called differential since
they are based on local Taylor series approximations of the image signal; that is, they use partial derivatives with respect to the spatial and temporal coordinates.
In mathematics, a Taylor series is a representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point.
04/12/2023DSG, CEERI Pilani23
Taylor series The Taylor series of a real or complex function ƒ(x) that is
infinitely differentiable in a neighborhood of a real or complex number a is the power series
which can be written in the more compact sigma notation as
where n! denotes the factorial of n and ƒ (n)(a) denotes the nth derivative of ƒ evaluated at the point a. The zeroth derivative of ƒ is defined to be ƒ itself and (x − a)0 and 0! are both defined to be 1. In the case that a = 0, the series is also called a Maclaurin series.
04/12/2023DSG, CEERI Pilani24
Estimation of the optical flow For a 2D+t dimensional case (3D or n-D cases are similar) a
voxel at location (x,y,t) with intensity I(x,y,t) will have moved by δx, δy and δt between the two image frames, and the following image constraint equation can be given:
I(x,y,t) = I(x + δx,y + δy,t + δt)
Assuming the movement to be small, the image constraint at I(x,y,t) with Taylor series can be developed to get:
H.O.T
(higher-order terms)
04/12/2023DSG, CEERI Pilani25
Estimation of the optical flow
04/12/2023DSG, CEERI Pilani26
Estimation of the optical flowwhere Vx,Vy are the x and y components of the velocity or optical flow of I(x,y,t) and are the derivatives of the image at (x,y,t) in the
corresponding directions. Ix,Iy and It can be written for the derivatives in the following.
Thus: IxVx + IyVy = − It
or
This is an equation in two unknowns and cannot be solved as such. This is known as the aperture problem of the optical flow algorithms. To find the optical flow another set of equations is needed, given by some additional constraint. All optical flow methods introduce additional conditions for estimating the actual flow.
Our Solution Optical flow: maximum one pixel large
movements
Optical flow: larger movements
Morphological filter
Contour detection (demo purposes)
Optical Flow: maximum one pixel large movements
The optical flow for a pixel given 2 successive images and :
),(),( yxjimk
ji,k 1k
so that
),(),( 1 yjxiIjiI kk
is minimum for 11,11 yx
(1)
k k+1
Optical Flow: maximum one pixel large movements
More precision: consider a 3×3 window around the pixel:
Optical flow for pixel becomes: ji,
),(),( yxjimk so that
1
1
1
11
1
1
1
1
),(),(u v
ku v
k yvjxuiIvjuiI
is minimum for 11,11 yx
(2)
Optical Flow: larger movements Reduce the size of the image
=> reduced size of the movement
Solution: multi-resolution analysis of the images
Advantages: computing efficiency, stability
Multi-resolution Analysis Coarse to fine optical flow estimation:
Original image k Original image k+1
256×256
128×128
64×64
32×32
Optical Flow: Top-down StrategyAlgorithm (1/4 scale of resolution
reduction): Step 1: compute optical flow vectors for the
highest level of the pyramid l (smallest resolution)
Step 2: double the values of the vectors Step 3: first approximation: optical flow vectors
for the (2i, 2j), (2i+1, 2j), (2i, 2j+1), (2i+1, 2j+1) pixels in the l-1 level are assigned the value of the optical flow vector for the (i,j) pixel from the l level
Level l Level l-1
Optical Flow: Top-down Strategy Step 4:
adjustment of the vectors of the l-1(one) level in the pyramid
method: detection of maximum one pixel displacements around the initially approximated position
Step 5:
smoothing of the optical flow field (Gaussian filter)
Filtering the Size of the Detected Regions
Small isolated regions of motion detected by the optical flow method are classified as noise and are eliminated with the help of morphological operations:
Step 1: Apply the opening:
Step 2: Apply the closing: BBXBX
BBXBX
Contour Detection For demonstration purposes, the contours of the
moving regions detected are outlined Method: the Sobel edge detector:
Compute the intensity gradient:
using the Sobel masks:
Compute the magnitude of the gradient:
if then edge pixel
else non-edge pixel
yx ffy
f
x
fyxf ,,,
(5)
121
000
121
4
1,
101
202
101
4
1yx GG (6)
22,, yx ffyxfyxM (7)
thresholdyxM ,
A Block Diagram of the System
Conclusions What we did:
We managed to estimate the motion with a certain level of accuracy
The results might be good enough for some applications, while other applications might require better accuracy
What remains to be done: Reduce computational complexity
use the computed background image to separate foreground objects
Parallelism of the algorithms Experiment with specific problems, calibrate
the parameters of the algorithms
04/12/2023DSG, CEERI Pilani38
Thanks