determining optical flow horn schunck

8/4/2019 Determining Optical Flow Horn Schunck

1/19

A R T I F I C I A L I N T E L L I G E N C E 1 8 5

Determining Optical FlowB e r t h o l d K . P . H o r n a n d B r i a n G . S c h u n c kArtificial Intelligence Laboratory, Massachusetts Institute ofTechnology, Cambridge, MA 02139, U.S.A.

A B S T R A C TOptical f low cannot be computed local ly , s ince only one independent measurement is avai lable fromthe image sequence at a point, wh ile the flow velocity has two components. A second constraint isneeded. A meth od for findin g the optical flow pattern is presented which assu mes that the apparentvelocity of the brightness pattern varies smoo thly almo st everywhere in the image. An iterativeimplementat ion is shown which success ful ly computes the opt ical f low for a number of synthe t ic imagesequences. The algorithm is robust in that it can handle image sequences that are quantized rathercoarsely in space and time. It is also insensitive to quantizatio n of brightness levels and additive noise.Examples are included where the assumption of smoothness is violated at singular points or alonglines in the image.

1. IntroductionOptical flow is the distribution of apparent velocities of movement of bright-ness patterns in an image. Optical flow can arise from relative motion ofobjects and the viewer [6, 7]. Consequently, optical flow can give importantinformation about the spatial arrangement of the objects viewed and the rateof change of this arrangement [8]. Discontinuities in the optical flow can help insegmenting images into regions that correspond to different objects [27].Attempts have been made to perform such segmentation using differencesbetween successive image frames [15, 16,. 17, 20, 25]. Several papers address theproblem of recovering the motions of objects relative to the viewer from theoptical flow [10, 18, 19, 21, 29]. Some recent papers provide a clear expositionof this enterprise [30, 31]. The mathemat ics can be made rather difficult, by theway, by choosing an inconvenient coordinate system. In some cases in-formation about the shape of an object may also be recovered [3, 18, 19].These papers begin by assuming that the optical flow has already beendetermined. Although some reference has been made to schemes for comput-

Artific ial Intelligence 17 (1981) 185-2030004-3702/81/0000-0000/$02.50 North-Holland


2/19

1 8 6 B .K .P . H O R N A N D B .G . S C H U N C K

ing the f low f rom success ive v iews of a scene [5 , 10], the specif ics of a schemef o r d e t e rm i n i n g t h e f l ow f ro m t h e i m ag e h av e n o t b een d e s c r i b ed . R e l a t edwo rk h a s b een d o n e i n an a t t em p t t o f o rm u l a t e a m o d e l f o r t h e s h o r t r an g emot ion de tec t ion p rocesses in human v is ion [2 ,22 ] . The p ixe l recu rs iveequa t ions o f Net rava l i and Robb ins [28 ] , des igned fo r cod ing mot ion inte lev is ion s igna ls , bear some s imi la r i ty to the i t e ra t ive equa t ions deve loped inth i s paper . A recen t rev iew [26] o f computa t iona l t echn iques fo r the ana lys i s o fi m ag e s eq u en ces co n t a i n s o v e r 1 5 0 r e f e r en ces .

T h e o p t i c a l f l o w can n o t b e co m p u t ed a t a p o i n t i n t h e i m ag e i n d ep en d en t l yo f n e i g h b o r i n g p o i n t s w i t h o u t i n t ro d u c i n g ad d i t i o n a l co n s t r a i n t s , b ecau s e t h eve loc i ty f ie ld a t each im age po in t has two co mp on en ts wh i le the change inimage b r igh tness a t a po in t in the image p lane due to mot ion y ie lds on ly onecons t ra in t . Cons id er , fo r exa mple , a pa tch o f a pa t te rn whe re b r igh tness I var iesa s a f u n c t i o n o f o n e i m ag e co o rd i n a t e b u t n o t t h e o t h e r . Mo v em en t o f t h epa t te rn in one d i rec t ion a l t e rs the b r igh tness a t a par t i cu la r po in t , bu t mot ioni n t h e o t h e r d i r ec t i o n y i e l d s n o ch an g e . T h u s co m p o n en t s o f m o v em en t i n t h el a t t e r d i r ec t i o n can n o t b e d e t e rm i n ed l o ca l l y .

2. Relat ionship to Object MotionThe re la t ionsh ip be tween the op t ica l f low in the image p lane and the ve loc i t i eso f ob jec t s in the th ree d imens iona l wor ld i s no t necessar i ly obv ious . Wep e rce i v e m o t i o n wh en a ch an g i n g p i c t u re i s p ro j ec t ed o n t o a s t a t i o n a ry s c reen ,fo r example . Converse ly , a mov ing ob jec t may g ive r i se to a cons tan t b r igh t -ness pa t te r n . C ons ider , fo r exam ple , a un i fo rm sph ere wh ich exh ib i t s shad ingbecause i t s su r face e lemen ts a re o r ien ted in many d i f f e ren t d i rec t ions . Yet ,when i t i s ro ta ted , the op t ica l f low i s ze ro a t a l l po in t s in the image , s ince theshad ing does no t move wi th the su r face . Also , specu la r re f lec t ions move wi th ave loc i ty charac te r i s t i c o f the v i r tua l image , no t the su r face in wh ich l ight isre f lec ted .

F o r co n v en i en ce , we t a ck l e a p a r t i cu l a r l y s i m p l e wo r l d wh e re t h e ap p a ren tv e l o c i ty o f b r i g h tn e s s p a t t e rn s can b e d i r ec tl y i d en t if i ed w i t h t h e m o v em en t o fsu r faces in the scene .

3. The Restricted Problem DomainTo avo id var ia t ions in b r igh tness du e to shad ing e f f ec t s we in i t ia l ly assume tha tthe su r face be ing imaged i s f l a t . We fu r ther assume tha t the inc iden t i l lumina-t ion i s un i fo rm across the su r face . The b r igh tness a t a po in t in the image i s thenp ro p o r t i o n a l t o t h e r e f l e c t an ce o f t h e s u r face a t t h e co r r e s p o n d i n g p o i n t o n t h eob jec t . Also , we assum e a t f ir s t tha t re f lec tance var ies smo o th ly and has no

l l n t h is pa pe r , t he t e r m b r i gh tn e ss m e a ns i m a g e i rr a d i a nc e . Th e b r i gh tne ss pa t t e r n i s t he d i s t r i bu t i onof i r r a d i a nc e i n the i m a ge .


3/19

DETERMINING OPTICAL FLOW 187spatial discontinuities. This latter condition assures us that the image brightnessis differentiable. We exclude situations where objects occlude one another, inpart, because discontinuities in reflectance are found at object boundaries. Intwo of the experiments discussed later, some of the problems occasioned byoccluding edges are exposed.In the simple situation described, the motion of the brightness patterns in theimage is determined directly by the motions of corresponding points on thesurface of the object. Computing the velocities of points on the object is amatter of simple geometry once the optical flow is known.

4. ConstraintsWe will derive an equation that relates the change in image brightness at apoint to the motion of the brightness pattern. Let the image brightness at thepoint (x, y) in the image plane at time t be denoted by E ( x , y, t). Now considerwhat happens when the pattern moves. The brightness of a particular point inthe pattern is constant, so that

dE~ 0 ,dtUsing the chain rule for differentiation we see that,

c~E d x a E d_y_+ OE = O.Ox dt + -~y dt St(See Appendix A for a more detailed derivation.) If we let

dx dyu=~ - 7 and V= dt ,then it is easy to see that we have a single linear equation in the two unknownsu and v,

Exu + Eyv + E, = O,where we have also introduced the additional abbreviations Ex, Ey, and E, forthe partial derivatives of image brightness with respect to x, y and t, respec-tively. The constraint on the local flow velocity expressed by this equation isillustrated in Fig. 1. Writing the equation in still another way,

(Ex, Ey). (u, v) = - Et .Thus the component of the movement in the direction of the brightnessgradient (Ex, Ey) equals

E,


4/19

188 B.K.P. HORN AND B.G. SCHUNCK

\ comrtaint lineFIG. I. The basic rate of change of image brightness equation constrains the optical flow velocity.The velocity (u, u) has to lie along a line perpendicular to the brightness gradient vector (& E,).The distance of this line from the origin equals E, divided by the magnitude of (Ex. I?,,).

We cannot, however, determine the component of the movement in thedirection of the iso-brightness contours, at right angles to the brightnessgradient. As a consequence, the flow velocity (u, U) cannot be computed locallywithout introducing additional constraints.

5. The Smoothness ConstraintIf every point of the brightness pattern can move independently, there is littlehope of recovering the velocities. More commonly we view opaque objects offinite size undergoing rigid motion or deformation. In this case neighboringpoints on the objects have similar velocities and the velocity field of thebrightness patterns in the image varies smoothly almost everywhere. Dis-continuities in flow can be expected where one object occludes another. Analgorithm based on a smoothness constraint is likely to have difficulties withoccluding edges as a result.

One way to express the additional constraint is to minimize the square of themagnitude of the gradient of the optical flow velocity:

Another measure of the smoothness of the optical flow field is the sum of thesquares of the Laplacians of the X- and y-components of the flow. The


5/19

DETERM INING OPTICAL FLOW 189Lap lac ians o f u and v a re def ined as

02u + 02---E and V2D 02D -{ - 92/.)W u = ~ 0 y 2 = 0 x 2 0 y 2.In s imple s i tua t ions , bo th Lap lac ians a re zero . I f the v iewer t rans la tes para l le lto a f l a t ob jec t , ro ta tes abou t a l ine perpend icu la r to the su r face o r t rave lso r t h o g o n a l l y t o t h e s u r f ace , t h en t h e s eco n d p a r t i a l d e r i v a ti v e s o f b o t h u a n d vv an i s h ( a s s u m i n g p e r s p ec t i v e p ro j ec t i o n i n t h e i m ag e f o rm a t i o n ) .

W e wi l l u s e h e re t h e s q u a re o f t h e m ag n i t u d e o f t h e g rad i en t a s s m o o t h n es sm eas u re . No t e t h a t o u r ap p ro ach is i n co n t r a s t w i th t h a t o f F e n n e m a an dT h o m p s o n [5 ] , wh o p ro p o s e an a l g o r i t h m t h a t i n co rp o ra t e s ad d i t i o n a l a s s u m p -t ions such as cons tan t f low ve loc i t i es wi th in d i sc re te reg ions o f the image . The i rmethod , based on c lus te r ana lys i s , canno t dea l wi th ro ta t ing ob jec t s , s ince theseg ive r i se to a con t inuum of f low ve loc i t i es .

6. Quantization and NoiseImages may be sampled a t in te rva l s on a f ixed g r id o f po in t s . Whi le t es se la t ionso t h e r t h an t h e o b v i o u s o n e h av e ce r t a i n ad v an t ag es [9, 23 ], fo r co n v en i en c e wewi l l as sume tha t the image i s sampled on a square g r id a t regu la r in te rva l s . Le tt h e m eas u red b r i g h t n e s s b e Eij.k a t the in te rsec t ion o f the i th row and j thco l u m n i n t h e k t h i m ag e f r am e . Id ea l l y , e ach m eas u rem en t s h o u l d b e anav e rag e o v e r t h e a r ea o f a p i c t u re ce l l an d o v e r t h e l en g t h o f t h e t i m e i n t e rv a l .In t h e ex p e r i m e n t s c i t ed h e re we h av e t ak en s am p l e s a t d i s c r e t e p o i n t s in s p acean d t i m e i n s t ead .

In ad d i t i o n t o b e i n g q u an t i zed i n s p ace an d t i m e , t h e m eas u rem en t s w i l l i np rac t ice be quan t ized in b r igh tness as wel l . Fu r ther , no ise wi l l be apparen t inm eas u rem en t s o b t a i n ed i n an y r ea l s y s t em .

7. Estimating the Partial DerivativesW e m u s t e s t i m a t e t h e d e r i v a t i v e s o f b r i g h t n e s s f ro m t h e d i s c re t e s e t o f i m ag ebr igh tness m ea sur em en t s ava i lab le . I t i s imp or ta n t th a t the es t ima tes o f Ex , Ey ,an d 13, be cons i s ten t . Tha t i s , they shou ld a l l re f e r to the same po in t in thei m ag e a t t h e s am e t i m e . W h i l e t h e re a r e m an y f o rm u l a s f o r ap p ro x i m a t edif ferent iat ion [4 , 11] we wil l use a set which g ives us an es t imate of Ex, Ey, E,a t a p o i n t i n t h e cen t e r o f a cu b e f o rm ed b y e i g h t m eas u rem en t s . T h ere l a t i o n s h i p i n s p ace an d t i m e b e t we en t h e s e m ea s u re m en t s i s s h o wn i n Fi g. 2.Each o f the es t imates i s the average o f fou r f i r s t d i f f e rences t aken overad j acen t m eas u rem en t s i n t h e cu b e .

Ex "~ l {Ei j+l , k - - E i j , k + Ei+l , j+l ,k - - E i+l j , k+Eid+l , k+l - - E id , k+l + Ei+l j+ l , k+l - - E i+l j , k+l},

1Ey ~ ~{Ei+l j . k - - E i j . k + Ei+l j+l . k - - E i j+ l . k+ Ei+l j . k+l - E i j , k+ 1 I+ Ei+l j +l . k+l - - Eid+l./+l},E, -~ ~{Eij.k+l - Ei j. i + Ei+I.j.k+l - E i+l j , k

+ Ei j+l , k+ I - - E i j+ l , k + Ei+l j+ l , k+l - - E i+l j+l . k } .


6/19

190 B . K . P . H O R N A N D B . G . S C H U N C K

i 1 - - - - - - . - - - ~

1

J j!l kFIG. 2. The three partia l derivatives of images brightness at the center of the cube are eachestimated from the average of f irst differences a long four para l le l edges of the c ube . H e r e theco lumn index j corresponds to the x di rec t ion in the image , the row index i to the y di rec t ion,whi le k l ies in the t ime di rec t ion.

Here the unit of length is the grid spacing interval in each image frame and theunit of t ime is the image frame sampling period. We avoid est imation formulaewith larger support, s ince these typical ly are equivalent to formulae of smallsupport applied to smoothed images [14] .

8 . Estimating the Laplaeian of the Flow Velocit iesW e a l so n e e d to approximate the Laplacians of u and v. O n e c o n v e n i e n tapproximation takes the fol lowing form

V2 U "~" I((fi i,j, k -- lgij,k ) a n d VE u ~- K(Oi,], k - 1)i.j,k) ,whe re the local av erages t~ and i] are defined as follow s

~lij.k -~- 16{Ui_lj, k + Ui,j+l, k + Ui+l,j,k + Ui, j-l ,k }+ ~2{ ~i- l,j 1,k + Ui-lj+l, k + Ui+I .j+I,k + /'/i+LJ 1,k},

~i,j,k = i6{Ui-l,j,k + Ui,j+l,k + Ui+1,j,k + Ui,j-l,k}1+ ~{1 -) i- l, j 1,k "~- l.) i-l ,j+l,k + 1.)i+l,j+l,k "~- l-)i+l,j 1,k}.

T h e proportionality factor r equals 3 i f the average is computed as shown andwe again assume that the unit of length equals the grid spacing interval. Fig. 3i l lustrates the ass ignment of weights to neighboring points .


7/19

DETERMINING OPTICAL FLOW

1112 1~ 1/12

191

1/6 - , 1/6

iFIG. 3. The Laplacian is estimated by subtracting the value at a point from a weighted average ofthe values at neighboring points. Shown here are suitable weights by which values can bemultiplied.

9. MinimizationT h e p r o b l e m t h e n i s t o m i n i m i z e t h e s u m o f t h e e r r o r s i n t h e e q u a t i o n f o r t h er a t e o f c h a n g e o f im a g e b r i g h t n e s s,

~b = Exu + E, ,v + E, ,a n d t h e m e a s u r e o f t h e d e p a r t u r e f r o m s m o o t h n e s s i n t h e v e lo c i t y f lo w ,

~2 = \-b--X-X + +W h a t s h o u l d b e t h e r e l a t i v e w e i g h t o f t h e s e t w o f a c t o r s ? I n p r a c t i c e t h e i m a g eb r i g h t n e s s m e a s u r e m e n t s w il l b e c o r r u p t e d b y q u a n t i z a t i o n e r r o r a n d n o i se sot h a t w e c a n n o t e x p e c t ~gb t o b e i d e n t i c a l l y z e r o . T h i s q u a n t i t y w i ll t e n d t o h a v ea n e r r o r m a g n i t u d e t h a t is p r o p o r t i o n a l t o t h e n o is e in t h e m e a s u r e m e n t . T h i sf a c t g u i d e s u s i n c h o o s i n g a s u i t a b l e w e i g h t i n g f a c t o r , d e n o t e d b y a 2, a s wi ll b eseen l a t e r .

L e t t h e t o t a l e r r o r t o b e m i n i m i z e d b e~2 = f f (a2~g2+ ~2) dx dy.

T h e m i n i m i z a t i o n i s t o b e a c c o m p l i s h e d b y f i n d i n g s u i t a b l e v a l u e s f o r t h eo p t i c a l f l o w v e l o c i t y ( u , v ) . U s i n g t h e c a l c u l u s o f v a r i a t i o n w e o b t a i n

E2u + E,,Eyv = ot2V2u - E,,E,,E,,Eyu + E2v = a2V2v - EyE , .

U s i n g t h e a p p r o x i m a t i o n t o th e L a p l a c i a n i n t r o d u c e d i n t h e p r e v i o u s s e ct io n ,(Of 2 "q- E2)u ~- ExEyv = ( a 2 a - ExEt),E,,Eyu + ( c t 2+ E2)v = (a2~ - EyEt).


8/19

19 2 B.K.P. HORN AND B.G. SCHUNCKTh e de t e rm ina n t o f the coef f ic ien t mat r ix equa ls a2 ( a 2 + Ex + E2) . So lv ing fo r uand v we f ind that

(a2 + E~ + 2 E2y)a - E~E y~ - E~Et,y ) u = + ( a 2 +(a 2 + E~ + E ~)v = - E ~ E y a + (a 2 + E~ )O - EYE,.

10. Dif ference of F low at a Point from Local AverageT h es e eq u a t i o n s can b e wr i t t en i n t h e a l t e rn a t e f o rm

(a 2 + E~ + E2y)(u - ~) = -E x[E I(~ + Eyb + El l ,( a 2+ E 2 + E ~ ) ( v - ~ ) = -E y[ E x~ + E y~ + E , I .

This shows tha t the va lue o f the f low ve loc i ty (u , v ) wh ich min imizes the e r ro r4 2 l ies in the d i rec t io n tow ards the cons t ra in t l ine a long a l ine tha t in te rsec t sthe cons t ra in t l ine a t r igh t ang les . Th is re la t ionsh ip i s i l lu s t ra ted geomet r ica l lyin F ig . 4 . The d i s tance f rom the loca l average i s p ropor t iona l to the e r ro r in thebas ic fo rmula fo r ra te o f change o f b r igh tness when ~ , ~ a re subs t i tu ted fo r uand v . F ina l ly we can see tha t a2 p lays a s ign i f ican t ro le on ly fo r a reas wheret h e b r i g h t n e s s g rad i en t i s s m a l l , p r ev en t i n g h ap h aza rd ad j u s t m en t s t o t h ees t imated f low ve loc i ty occas ioned by no ise in the es t imated der iva t ives . Th isp a ram e t e r s h o u l d b e ro u g h l y eq u a l t o t h e ex p ec t ed n o i s e i n t h e e s t i m a t e o fE ~ + E ~ .

V

- U

o n s t r a l n t l i n e

FIG. 4. The value of the flow velocity which minimizes the er ror lies on a line drawn fro m the localaverage of the flow velocity perpendicular to the constraint line.


9/19

DETERMINING OPTICAL FLOW 19311. Constrained Minimization

When we a l low a2 to t end to zero we ob ta in the so lu t ion to a cons t ra inedmin imiza t ion p rob lem. App ly ing the me tho d o f Lagran ge mul t ip l ie rs [33, 34] tothe p rob lem of minimizin g g ,2 whi le m ainta ining ~b = 0 leads to

E~V2u = Ex V% , Exu + E~v + E, = 0Approx imat ing the Lap lac ian by the d i f fe rence o f the ve loc i ty a t a po in t andthe average o f i t s ne ighbors then g ives us

(E~ + E~)( u - r,) = -E~[ EX~ + E~ + E,] ,(E~ + E~) (v - ~) = -E y[ E ,a + Ey~ + E,].

Refer r ing aga in to F ig . 4 , we no te tha t the po in t computed here l i es a t thein te rsec t ion o f the cons t ra in t l ine and the l ine a t r igh t ang les th rough the po in t(~, ~) . W e wil l not use these equ at io ns s ince we do expec t errors in thees t imat ion o f the pa r t i a l der iva tives .

12. Iterative SolutionWe now have a pa i r o f equa t ions fo r each po in t in the image . I t wou ld be verycos t ly to so lve these equa t ions s imul taneous ly by one o f the s t andard methods ,such as Gauss -Jo rdan e l imina t ion [11 , 13] . The cor respond ing mat r ix i s sparseand very l a rge s ince the number o f rows and co lumns equa l s twice the numberof p ic tu re ce l l s in the image . I t e ra t ive methods , such as the Gauss -Se ide lm et ho d [11, 13] , suggest themselv es . W e can c om pu te a ne w set of veloci tyes t ima tes (u "+~, v "~) fro m the es t im ated derivat ives and the avera ge of thepreviou s veloci ty es t ima tes (u", v n) by

u "+' = ~" - E,[Ex~" + Eyf;" + E,]I(o~ 2 + E 2 + E~),v "+' = ~" - Ey[Exa" + EyF" + E, ]/ (a 2 + E 2 + E2).

( I t i s in te res t ing to no te tha t the new es t imates a t a par t i cu la r po in t do no tdepend d i rec t ly on the p rev ious es t imates a t the same po in t . )

The na tu ra l boundary cond i t ions fo r the var ia t iona l p rob lem tu rns ou t to bea zero normal der iva t ive . At the edge o f the image , some o f the po in t s neededto compute the loca l average o f ve loc i ty l i e ou t s ide the image . Here we s implycopy ve loc i t i es f rom ad jacen t po in t s fu r ther in .

13. Filling In Uniform RegionsIn par t s o f the image where the b r igh tness g rad ien t i s zero , the ve loc i tyes t imates wi l l s imply be averages o f the ne ighbor ing ve loc i ty es t imates . Therei s no loca l in fo rmat ion to cons t ra in the apparen t ve loc i ty o f mot ion o f thebr igh tness pa t t e rn in these a reas . Even tua l ly the va lues a round such a reg ionwi l l p ropaga te inwards . I f the ve loc i t i es on the border o f the reg ion a re a l l


10/19

194 B.K.P. HORN AND B.G. SCHUNCKequa l to the same va lue , then po in t s in the reg ion wi l l be ass igned tha t va luetoo , a f te r a su f f ic ien t number o f i t e ra t ions . Veloc i ty in fo rmat ion i s thus f i l l ed inf ro m t h e b o u n d a ry o f a r eg i o n o f co n s t an t b r i g h t n e s s .

If the value s on the b or de r ar e not a l l the sam e, i t is a l i tt le mo re d if ficul t topredict what wil l happen. In al l cases , the values f i l led in wil l correspond to theso lu t ion o f the Lap lace equa t ion fo r the g iven boundary cond i t ion [1 , 24 , 32 ] .

The p rogress o f th i s f i l l ing - in phenomena i s s imi la r to the p ropaga t ion e f f ec t sin the so lu t ion o f the hea t equa t ion fo r a un i fo rm f ia t p la te , where the t ime ra teo f ch an g e o f t em p e ra t u re is p ro p o r t i o n a l t o t h e Lap l ac i an . T h i s g iv e s u s a m ean so f u n d e r s t an d i n g t h e i t e r a t i v e m e t h o d i n p h y s i ca l t e rm s an d o f e s t i m a t i n g t h en u m b er o f s t eps r eq u i r ed . T h e n u m b er o f i t e r a ti o n s s h o u ld b e l a rg e r t han t h enu mb er o f p ic tu re ce l l s ac ross the la rges t reg ion tha t m us t b e f i ll ed in . I f the s ize o fsuch reg ions i s no t known in advance one may use the c ross - sec t ion o f the who leimage as a conserva t ive es t imate .

14. Tightness of ConstraintWhen b r igh tness in a reg ion i s a l inear func t ion o f the image coord ina tes wecan o n l y o b t a i n t h e co m p o n en t o f o p t i c a l f l o w i n t h e d i r ec t i o n o f t h e g rad i en t .The componen t a t r igh t ang les i s f i l l ed in f rom the boundary o f the reg ion asd es c r i b ed b e f o re . I n g en e ra l t h e s o l u t i o n i s m o s t a ccu ra t e l y d e t e rm i n ed i nreg ions where the b r igh tness g rad ien t i s no t too smal l and var ies in d i rec t ionf ro m p o i n t t o p o i n t . I n f o rm a t i o n wh i ch co n s t r a i n s b o t h co m p o n en t s o f t h eop t ica l f low ve loc i ty i s then ava i lab le in a re la t ive ly smal l ne ighborhood . Toov io len t f luc tua t ions in b r igh tness on the o ther hand a re no t des i rab le s ince thees t imates o f the der iva t ives wi l l be co r rup ted as the resu l t o f undersampl ing andaliasing.

15. Choice of Iterative SchemeAs a p rac t i c a l m a t t e r o n e h a s a ch o i ce o f h o w t o i n t e r l a ce t h e i t e r a t i o n s w i t hthe t ime s teps . On the one hand , one cou ld i t e ra te un t i l the so lu t ion hass t ab i l i z ed b e f o re ad v an c i n g t o t h e n ex t i m ag e f r am e . On t h e o t h e r h an d , g i v ena good in i t i a l guess one may need on ly one i t e ra t ion per t ime-s tep . A goodin it i al guess fo r the op t ica l f low ve loc i t i es is u sua lly ava i lab le f r om the p rev io ust ime-s tep .

The advan tages o f the la t t e r approach inc lude an ab i l i ty to dea l wi th moreimages per un i t t ime and be t te r es t imates o f op t ica l f low ve loc i t i es in cer ta inreg ions . Areas in wh ich the b r igh tness g rad ien t i s smal l l ead to uncer ta in , no isyes t imates ob ta ined par t ly by f i l l ing in f rom the su r round . These es t imates a rei m p ro v ed b y co n s i d e r i n g f u r t h e r i m ag es . T h e n o i s e i n m eas u rem en t s o f t h ei m ag es w i l l b e i n d ep en d en t an d t en d t o can ce l o u t . P e rh ap s m o re i m p o r t an t l y ,d i f f e ren t par t s o f the pa t te rn wi l l d r i f t by a g iven po in t in the image . Thed i rec t ion o f the b r igh tness g rad ien t wi l l vary wi th t ime , p rov id ing in fo rmat ionab o u t b o t h co m p o n en t s o f t h e o p t i c a l f l o w v e l o c i t y .


11/19

DETERMINING OPTICAL FLOW 195A practical implementat ion would most l ikely employ one i terat ion per t ime

step for these reasons. We i l lustrate both approaches in the experiments.16. Experiments

The i te ra t ive scheme has been implemented and appl ied to image sequencescorrespo nding to a nu mb er of simple flow patterns. T he results shown here arefor a relat ively low resolut ion image of 32 by 32 picture cel ls . The br ightnessmeasurements were in tent iona l ly cor rupted by approximate ly 1% noise and

A la

DFIG. 5. Four frames out of a sequence of images of a sphere rotating about an axis inclined towardsthe viewer. The sphere is covered with a pattern which varies smoothly from place to place. Thesphere is portrayed against a fixed, lightly textured background. Image sequences like these areprocessed by the optical flow algorithm.


12/19

196 B.K.P. HORN AND B.G. SCHUNCKthen quan t ized in to 256 leve ls to s imula te a rea l imag ing s i tua t ion . Theunder ly ing su r face re f lec tance pa t te rn was a l inear combina t ion o f spa t ia l lyo r t h o g o n a l s i n u s o i d s . T h e i r wav e l en g t h was ch o s en t o g i v e r ea s o n ab l y s t ro n gb r i g h t n e s s g rad i en t s w i t h o u t l e ad i n g t o u n d e r s am p l i n g p ro b l em s . D i s co n -t i n u i t i e s we re av o i d ed t o en s u re t h a t t h e r eq u i r ed d e r i v a t i v e s ex i s t ev e ry wh e re .

Shown in F ig . 5 , fo r example , a re fou r f rames o f a sequence o f imagesdep ic t ing a sphere ro ta t ing abou t an ax is inc l ined towards the v iewer . As m o o t h l y v a ry i n g r e f l e c t an ce p a t t e rn i s p a i n t ed o n t h e s u r f ace o f t h e s p h e re .The sphere i s i l lumina ted un i fo rmly f rom a l l d i rec t ions so tha t there i s noshad ing . We chose to work wi th syn the t ic image sequences so tha t we canco m p are t h e r e s u l ts o f t h e o p t i c a l fl ow co m p u t a t i o n w i th t h e ex ac t v a lu e sca l cu l a t ed u s i n g t h e t r an s f o rm a t i o n eq u a t i o n s r e l a t i n g i m ag e co o rd i n a t e s t oco o rd i n a t e s o n t h e u n d e r l y i n g s u r f ace r e f l e c t an ce p a t t e rn .

17. Results"l]ae f i rs t f low to be invest igated was a s imple l inear t ranslat ion of the ent i reb r igh tness pa t te rn . The resu l t ing computed f low i s shown as a need le d iag ramin Fig . 6 for 1 , 4 , 16 , and 64 i terat ions . The es t imated f low veloci t ies ared ep i c t ed a s s h o r t l i n e s , s h o wi n g t h e ap p a ren t d i s p l acem en t d u r i n g o n e t i m es tep . In th i s example a s ing le t ime s tep was t aken so tha t the computa t ions a rebased on jus t two images . In i t ia l ly the es t imates o f flow ve loc i ty a re zero .Consequen t ly the f i r s t i t e ra t ion shows vec to rs in the d i rec t ion o f the b r igh tnessg rad ien t . La te r , the es t ima tes a ppro ach the co r re c t va lues in a ll par t s o f thei m ag e . F ew ch an g es o cc u r a f t e r 3 2 it e r a t i o n s wh en t h e v e l o c i t y v ec t o r s h av ee r ro r s o f ab o u t 1 0 %. T h e e s t i m a t e s t en d t o b e t wo s m a l l , r a t h e r t h an t o ol a rg e , p e rh ap s b ecau s e o f a t en d en cy t o u n d e re s t i m a t e t h e d e r i v a t i v e s . T h ewors t e r ro rs occur , as one migh t expec t , where the b r igh tness g rad ien t i s smal l .

I n t h e s eco n d ex p e r i m en t o n e i t e r a t i o n was u s ed p e r t i m e s t ep o n t h e s am el inear t rans la t ion image sequence . The resu l t ing computed f low i s shown in F ig .7 fo r 1 , 4 , 16 , and 64 t ime s teps . The es t imates approach the co r rec t va luesmore rap id ly and do no t have a t endency to be too smal l , as in the p rev iousex p e r i m en t . F ew ch an g es o ccu r a f t e r 16 i t e r a t io n s wh en t h e v e l o c i ty v ec t o r sh av e e r ro r s o f ab o u t 7 %. T h e wo rs t e r ro r s o ccu r , a s o n e m i g h t ex p ec t , wh e rethe no ise in recen t measuremen ts o f b r igh tness was wors t . Whi le ind iv idua le s t i m a t e s o f v e l o c it y m ay n o t b e v e ry accu ra t e , t h e av e rag e o v e r t h e wh o l eimage was wi th in 1% of the co r r ec t va lue .

Nex t , t h e m e t h o d was ap p l i ed t o s i m p l e ro t a t i o n an d s i m p l e co n t r ac t i o n o fthe b r igh tness pa t te rn . The resu l t s a f te r 32 t ime s teps a re shown in F ig . 8 . No tet h a t t h e m ag n i t u d e o f t h e v e l o c it y is p ro p o r t i o n a l t o t h e d i s tan ce f ro m t h eor ig in o f the f low in bo th o f these cases . (By o rig in we m ean the po in t in theimage w her e the ve loc i ty i s ze ro . )

In the exam ples so fa r the Lap lac ian o f bo th f low ve loc i ty com pon en ts i s ze roeve ryw here . We a l so s tud ied mo re d i ff icu l t cases where th is was no t the case.


13/19

~.

~

~

~.~

~.

~~-~.~

~

~-o~

~~.~'~

~~~.~

-~.~'~

-~~

~~o

,~.

i~,~,~~s

,~:

,~~~"

~~...~~~

~~"~

,~'.

-*~~'o

~

~

:~~~

~

o~~

W.~

F

~

,

.\~

.~

.

.

~-

r

.

~

_

~

.

.

-

~r

~r~

.

~.

,

+,!~i~i~i

"7

~~~~~.L;:;~~:~:'~-";

!Ti:!.!!!

!!

Z 0 0 I


14/19

~.~=~.~

=oE~n

~~a~o

~a

oZ~

~=

.o

~.~'o

~-

,0 o

=~-~~

O~0~

I~

.

==

~e~-~~

~

~

~~~.~

,

N,~"

E~.J

=.a

~~

eE

~g. N~

=a

a~ ~

!~i:?~i~i~!::~~:~~:2::~i

?!

i!:!

0

:?:

.

:

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

t......

~

.

.

.

.

ml

t

:

~

.

90

W"

'~t~'~

rro~-

"'t

"

"

"~

.

.

.

.

.

t"

-

.

""

.

.

""

Z

:

::::

:22:

:

:::

:[2:

:::

>


15/19

D E T E R M I N I N G O P T I C A L F L O W 199~ - ~ ~ .~.

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . ' ' ' I ~ I ~ % t 1 1 1 1 1 1 1 1 ~. . . . . . . . . . . . . ~ ' I r i l l T I i l ~ l l l l l I" . . . . . . . . , , , , , r r ? r l ? ? ~. . . . . . . . . . . . . . . .

.......... , , , , , ~ r r r r y r r z z

Z 7 ? / 7 / / 7 / / 7'/ ~,' ;


16/19

s-m~OAUjmB

g

s"OVD

.

.

.

.

"1

..

"omLDsOLa

:.. .

.

~


17/19

DETERMINING OPTICAL FLOW 201provides only one constraint. Smoothness of the flow was introduced as asecond constraint. An iterative method for solving the resulting equation wasthen developed. A simple implementation provided visual confirmation ofconvergence of the solution in the form of needle diagrams. Examples ofseveral ditterent types of optical flow patterns were studied. These includedcases where the Laplacian of the flow was zero as well as cases where it becameinfinite at singular points or along bounding curves.The computed optical flow is somewhat inaccurate since it is based on noisy,quantized measurements. Proposed methods for obtaining information aboutthe shapes of objects using derivatives (divergence and curl) of the optical flowfield may turn out to be impractical since the inaccuracies will be amplified.

ACKNOWLEDGMENTThis research was conducted at the Artificial Intelligence Laboratory of theMassachusetts Institute of Technology. Support for the laboratory's research isprovided in part by the Advanced Research Projects Agency of the Depart-ment of Defense under Office of Naval Research contract number N00014-75-C0643. One of the authors (Horn) would like to thank Professor H.-H. Nagelfor his hospitality. The basic equations were conceived during a visit to theUniversity of Hamburg, stimulated by Professor Nagel's long-standing interestin motion vision. The other author (Schunck) would like to thank W.E.L.Grimson and E. Hildreth for many interesting discussions and much knowledg-able criticism. W.E.L. Grimson and Katsushi Ikeuchi helped to illuminate aconceptual bug in an earlier version of this paper. We should also like to thankJ. Jones for preparing the drawings.

Appendix A. Rate of Change of Image BrightnessConsider a patch of the brightness pattern that is displaced a distance 8x in thex-direction and 8y in the y-direction in time St. The brightness of the patch isassumed to remain constant so that

E( x, y, t) = E( x + 8x, y + By, t + 80.Expanding the right-hand side about the point (x, y, t) we get,c~E + c~E + 8EE ( x , y , t ) = E ( x , y , t ) + S X ~ x 8 y-~ y 8 I--~ -+ ~.

Where ~ contains second and higher order terms in 8x, By, and St. Aftersubtracting E(x, y, t) from both sides and dividing through by 8t we have8x OE ~ OE OE + cT(St)= O,8t ax ~ 8t -~y + -d?

where ~(St) is a term of order 8t (we assume that 8x and 8y vary as 80. In the


18/19

202 B.K.P . HORN AND B.G. SCHUNCKlimit as 6t--> 0 this becomes

dE dx tgEd~. ,gEO x a-7 + d t + = .

Appendix B. Sm oothness of Flow for Rigid Body MotionsLet a rigid bod y rotate abo ut an a xis (wx, toy, o)z), wh ere the ma gnitud e of thevector equals the angular velocity of the motion. If this axis passes through theorigin, then the velocity of a point (x, y, z) equals the cross product o f(cox, toy, toz), and (x, y, z). Th ere is a direct relation ship bet we en the ima gecoord inates and the x and y coord inates here if we assume that the image isgenerated by orthographic projection. The x and y components of the velocitycan be written,U = toyZ -- tozY, I) = tOzX -- toxZ.Consequently,

V2u = +toyV2z, V2v = -toxV2Z.This illustrates that the smoothness of the optical flow is related directly to thesmoothness of the rotating body and that the Laplacian of the flow velocity willbecome infinite on the occluding bound, since the partial derivatives of z withrespect to x and y become infinite there.

R E F E R E N C E S1. Ames, W.F., Numerical Methods for Partial Differential Equations (Academic Press , New York,1977).2. Batal i , J . and Ullman, S. , Motion detect ion and analysis, Proc. of the ARPA Image Under-

standing Workshop, 7-8 November 1979 (Sc ience Appl ica t ions Inc . , Arl ington, VA 1979) pp.69-75.3. Clocksin, W., Determining the orientat ion of surfaces from optical flow, Proc. of the ThirdAISB Conference, Hamburg (1978) pp. 93-102.4. Conte , S .D. and de Boor , C. , Elementary Numerical Analysis (McGraw-Hil l , New York, 1965,

1972).5 . Fennema, C.L. and Thompson, W.B. , Veloc i ty de te rmina t ion in scenes conta ining severa lmoving objec t s , Computer Graphics and Image Processing 9 (4) (1979) 301-31 5.6. Gibson, J .J . , The Perception of the Visual World (Riverside Press, Cambridge, 1950).7. Gibson, J .J . , The Senses Considered as Perceptual Systems (Houghton-Miff l in , Boston, MA.1966).8. Gibson, J .J . , On the analysis of change in the optic array, Scandinavian J. PsyehoL 18 (1977)161-163.9. Gray, S .B. , Loca l proper t ies of binary images in two dimensions , IEEE Trans. on Computers 20(5) (1971) 551-561.

10. Hadani , I . , Isha i , G. and Cur , M. , Visua l s tabi l i ty and space percept ion in monocular vi s ion:Mathemat ica l model , J. Optical Soc. Am. 70 (1) (1980) 60-6 5.11. Hamming, R.W., Numerical Methods for Scientists and Engineers (McGraw-Hi l l , New York,

1962).12. Hildebrand, F.B. , Methods of Applied Mathematics (Pre ntice- Hall, E ng lew oo d Cliffs, N J, 1952,1965).


19/19

D E T E R M I N I N G O P T I C A L F L OW 20313. Hildebrand, F.B., Introduction to Num erical Ana lysis (McGraw-Hill , New York, 1956, 1974).14. Horn, B.K.P., (1979) Hill shading and the reflectance map, Proc. I E E E 69 (1) (1981) 14--47.15. Jain, R., Martin, W.N. and Aggarwal, J .K., Segmentation through the detection of changes

due to motion, Computer Graphics and Image Processing I1 (1) (1979) 13-34.16. Jain, R. Militzer, D. and Nagel, H.-H., S eparating no n-stationary from stationary scenecom pone nts in a sequence o f real world TV-images, Proc. of the 5th Int. Joint Conf. on ArtificialIntelligence, Augu st 1977, Cam bridge, MA , 612--618.

17. Jain, R. and Nagel, H.-H., On the analysis of accumulative difference pictures from imagesequences of real world scenes, IEEE Trans. on Pattern Analysis and Machine Intelligence 1 (2)(1979) 206-214.

18. Koen derink, J .J . and van D oo m, A.J., Invariant properties of the motio n parallax field due tothe m ovem ent of r igid bodies relat ive to an observer , Optica Acta 22 (9) 773-791.19. Koenderink, J .J . and van Doorn, A.J., Visual perception of rigidity of solid shape, J. Math.Biol. 3 (79) (1976) 79-85.

20. Lim b, J .O. and Murphy, J .A., E stimating the velocity of moving images in television signals,Computer Graphics and Image Processing 4 (4) (1975) 311-327.21. Longue t-Higgins, H.C. and Prazdny, K., The interpreta tion of moving retinal image, Proc. ofthe Royal Soc. B 208 (1980) 385--387.

22. Man ', D. an d U llman, S., Direction al selectivity and its use in early visual processing, ArtificialIntelligence Laboratory Memo No. 524, Massachusetts Institute of Technology (June 1979), toappear in Proc. Roy. Soc. B.

23. Merserea u, R.M., T he processing of hexagonally sampled tw o-dimensional signals, Proc. of theI E E E 67 (6) (1979) 930-949.

24. Milne, W.E., Nu me rica l Solution o f Differential Equ ation s (Dover, N ew Yo rk, 1953, 1979).25. Nagel, H.-H., Analyzing sequences of TV-frames, Proc. of the 5th Int. Joint Conf. on Artificial

Intelligence, Augu st 1977, Camb ridge, M A, 626.26. Nagel, H.-H., Analysis techniques for image sequences, Proc. of the 4th Int. Joint Conf. onPattern Recognition, 4-10 November 1978, Kyoto, Japan.27. Nakaya ma, K. and Loomis, J .M., O ptical velocity patterns, v elocity-sensitive neurons andspace perception, Perception 3 (1974) 63-80.28. Netravali, A.N. and R obbins, J .D., Motio n-com pensa ted television coding: Part I , The Bell

System Tech. J. 58 (3) (1979) 631--670.29. Prazdny, K., C ompu ting eg omo tion and surface slant from optical flow. Ph.D. Thesis,

Computer Science Department, University of Essex, Colchester (1979).30. Prazdny, K., Egomotion and relative depth map from optical flow, Biol. Cybernet. 36 (1980)

87-102.31. Prazdny, K., The in formation in optical flows. Com pute r Science Departm ent, University ofEssex, Colchester (1980) mimeographed.32. Richtmyer, R.D. and Mortin, K.W., Difference Methods for Initial-Value Problems (Inter-science, John Wiley & Sons, New York, 1957, 1967).33. Russell, D.L., Calculus of Variations and Control Theory (Academic Press, New York, 1976).34. Yourgau, W. and Mandelstam, S., Variational Principles in Dy na mic s and Q uan tum Theory

(Dover, New York, 1968, 1979).

R e c e i v e d M a r c h 1 98 0

determining optical flow horn schunck

Documents