Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting

M. Bertalmio A. L. Bertozzi G. SapiroComputerEng.Dept. MathematicsDept. Elec.& Comp.Eng.Dept.

UniversityPompeuFabra Duke University Univ. of Minnesota08003Barcelona,SPAIN Durham,NC 27708 Minneapolis,MN 55455

Abstract

Image inpainting involvesfilling in part of an image orvideo using information from the surroundingarea. Ap-plicationsincludetherestoration of damagedphotographsandmoviesandtheremoval of selectedobjects.In this pa-per, we introducea classof automatedmethodsfor digitalinpainting. Theapproachusesideasfromclassicalfluid dy-namicsto propagateisophotelines continuouslyfrom theexterior into the region to be inpainted. Themain idea isto think of the image intensityasa ‘streamfunction’ for atwo-dimensionalincompressibleflow. TheLaplacianof theimage intensityplays the role of the vorticity of the fluid;it is transportedinto the region to be inpaintedby a vec-tor field definedby the streamfunction. Theresultingal-gorithm is designedto continueisophoteswhile matchinggradientvectors at the boundaryof the inpainting region.ThemethodisdirectlybasedontheNavier-Stokesequationsfor fluid dynamics,which hasthe immediateadvantage ofwell-developedtheoretical and numericalresults. This isa new approach for introducingideasfrom computationalfluid dynamicsinto problemsin computervisionandimageanalysis.

1. IntroductionImageinpainting [2, 10, 20, 38] is theprocessof filling inmissingdatain adesignatedregionof astill or videoimage.Applicationsrangefrom removing objectsfrom a scenetore-touchingdamagedpaintingsandphotographs.Thegoalis to producea revised imagein which the inpaintedre-gion is seamlesslymergedinto the imagein a way that isnot detectableby a typical viewer. Traditionally, inpaintinghasbeendoneby professionalartists.For photographyandfilm, inpaintingis usedto revert deterioration(e.g.,cracksin photographsorscratchesanddustspotsin film), or to addor removeelements(e.g.,removal of stampeddateandred-eye from photographs,theinfamous“airbrushing”of polit-ical enemies[20]). A currentactive areaof researchis toautomatedigital techniquesfor inpainting[2, 3, 16, 21, 22].

In this paper, we introducea novel algorithmfor digi-tal inpaintingof still imagesthat attemptsto replicatethe

basictechniquesusedby professionalrestorators.Our al-gorithm,motivatedby amethodproposedin [2], involvesadirectsolutionof theNavier-Stokesequationsfor anincom-pressiblefluid. Theimageintensityfunctionplaystheroleof thestreamfunctionwhoseisophotelinesdefinestream-lines of the flow. After the userselectsthe regions to berestored,the algorithm automaticallytransportsinforma-tion into the inpaintingregion. The fill-in is donein sucha way that isophotelines arriving at the region’s bound-ariesarecompletedinside. The techniqueintroducedheredoesnot requirethe userto specifywherethe novel infor-mation comesfrom. This is doneautomatically(and ina fastway), therebyallowing for simultaneouslyfill-in ofmultiple regionscontainingcompletelydifferentstructuresandsurroundingbackgrounds.In addition,no limitationsareimposedon the topologyof the region to be inpainted.The only user interactionrequiredby the algorithm is tomark the regionsto be inpainted. Sinceour inpaintingal-gorithm is designedfor both restorationof damagedpho-tographsand for removal of undesiredobjectson the im-age,theregionsto beinpaintedmustbemarkedby theuser.Our methodinheritsa mathematicaltheoryalreadydevel-opedfor the fluid equations,includingwell-posednessandthedesignof efficientconvergentnumericalmethods.

1.1. Prior Work

First note that image denoisingis different to filling-in,sincethe regions of missingdataare usually large. Thatis, regionsoccupiedby top to bottomscratchesalongsev-eralfilm frames,long cracksin photographs,superimposedlarge fonts, andso on, areof significantlylarger sizethanthe type of noisetreatedby commonimageenhancementalgorithms.In addition,in commonimageenhancementap-plications,thepixelscontainbothinformationabouttherealdataandthenoise(e.g.,imageplusnoisefor additivenoise),while in our applicationthereis no significantinformationin theregion to beinpainted.

A very active arearelatedto our work is the restora-tion of damagedfilms. The basicidea is to useinforma-tion from pastandfutureframesto restorethecurrentone,e.g.,[16, 22]. Of course,this generalapproachcannotbe

1

usedwhen dealingwith still images. In addition, it cannot deal with movies wherethe region to be inpaintedisstaticwith respectto its background(e.g.,a logoonashirt),sinceconsecutive framesdo not provide new information.This is also the casewhen the region to be inpaintedoc-cupiesa large numberof frames. Anotherarearelatedtoour work is texturesynthesis,in which a textureis selectedandsynthesizedinsidethe region to be filled-in (the hole)[9, 13, 15, 34]. Thesealgorithmsoften requirethe usertoselectthe textureandarenot often well-designedto fill instructurefrom boundarydata.

Theclosestmethodstoourapproacharethefundamentalworkson disocclusionandline continuation.A pioneeringcontribution in this areais describedin [28]. The authorspresentedatechniquefor removing occlusionswith thegoalof imagesegmentation.Sincetheregion to befilled-in canbeconsideredasoccludingobjects,removing occlusionsisanalogousto imageinpainting. The basicidea suggestedby the authorsis to connectT-junctionsat the occludingboundariesof objectswith elasticaminimizingcurves.Thetechniquewasprimarily developedfor simple imagesob-tainedfrom a segmentation,with only a few objectswithconstantgray-levels.Thus,they endedupby connecting

�-

junctionsat thesamegraylevel. (Otherresearchers,e.g.,D.Jacobs,R.Basri,andS.Zucker, havefollowedthis interest-ing researcharea,mainlydevelopingtechniquesfor smoothcurvecontinuation.)

Masnou and Morel [25, 26] recently extendedtheseideas,presentinga formal variationalformulationfor dis-occlusionand a particularpractical implementation. Thealgorithmusesgeodesiccurvesto join the isophotesarriv-ing at the boundaryof the region to be inpainted. The in-paintingregionsrequirea simpletopology. In addition,theangle,with which the level lines arrive at the boundaryofthe holes,is not (well) preserved, and the algorithmusesstraightlinesto join equalgrayvaluepixels.

1.2. Image inpainting along isophotesRecently, the conceptof smoothcontinuationof informa-tion in the level-linesdirectionhasbeenaddressedin [2].This is the work, aswe will seebelow, that we follow tomake the connectionwith fluid dynamicsand the Navier-Stokesequations.The proposedalgorithmpropagatestheimage Laplacian in the level-lines (isophotes)direction.The algorithm attemptsto imitate basicapproachesusedby professionalrestorators.The algorithmalsointroducesthe importanceof propagatingboth the gradientdirection(geometry)andgray-values(photometry)of theimagein abandsurroundingtheholeto befilled-in. Someof theideasof [2] whereadoptedin [1], while deviatingfrom thepartic-ularmodelin orderto beableto definea formalvariationalapproachto thefilling-in/inpaintingproblem.

Thework in [2] inspireda very elegantapproachto the

filling-in problemrecentlyreportedin [7] (this work wasperformedindependentlyof the onereportedin [1]). Theauthorspresenta clearandintuitive axiomaticapproachtothe problem. The main algorithmthey proposeis to min-imize the Total Variation (TV) [33], of the image insidethe hole (they alsouse,asproposedin [1, 2] andhere,abandsurroundingthe region). As in the work of MasnouandMorel, their interpolationis limited to creatingstraightisophotes,notnecessarilysmoothlycontinuedfrom theholeboundary, andmainly is developed(asthe authorsclearlystate)for small holes. Although straightconnectionsgivevisually pleasantresultsfor small holes,it is importanttodevelop a theory that permits interpolationof level linesacrosslargegaps,whereconnectingwith straightlineswillbe unpleasanteven for simple images. In order to obtainsucha smoothinterpolationandcontinuationof isophotes,it is necessaryto go into high-orderPartial DifferentialEquations(PDE’s) or systemsof PDE’s, asdonein [1, 2]andhere(the authorsof [7] have alsorecentlyintroducedhigher order models). Researchin perception,from theGestaltto more recentwork (e.g. [31]) supportsthe ideaof performinga smoothcontinuationof theangleof arrivalof thelevel linesat thegap.

Thepaper[2] proposesanalgorithmdesignedto projectthegradientof thesmoothnessof theimageintensityin thedirectionof the isophotes.The resultingschemeis a dis-creteapproximationof thePDE

����������� �����(1)

for theimageintensity�, where

� �denotestheperpendic-

ulargradient��������������� and�

denotestheLaplaceoperator����� ���� . Additionalanisotropicdiffusionof theimagecanproduceaPDEof theform

� � �!��"�#�$���%� '& �(� �*)+�-, ��� , � ��� �/. (2)

Thegoalis to evolve (1) or perhapsmoreappropriately(2)to a steadystatesolution,satisfyingtheconditionin thein-paintingregion that the isophotelines, in the directionof� � �

, mustbeparallelto thelevel curvesof thesmoothness���of theimageintensity, which for & �!0 becomes

��"�#�$���%�%�!0 . (3)

We note that while the authorsof [2] presenttheirmethodas moving image intensity along isophotelines,thereis anothersensein which to think of the dynamics.Equation(1) is a transportequationthatconvectstheimageintensity

�alonglevel curvesof thesmoothness,

���. This

canbeseenby notingthat(1) is equivalentto 1 ��2 143 �50where1 2 143 is thematerialderivative � 2 ��3 76 �8� for thevelocity field 6 �9� � �%�

. In particular�

is convectedbythevelocity field 6 which is in thedirectionof level curvesof thesmoothness

���. In thenext sectionwe discusshow

2

this is relatedto a classicalproblemin incompressiblefluiddynamics.Our goal is to make useof expertisedevelopedin thatfield to designbetterinpaintingalgorithms.

2. Analogy to Transport of Vorticity inIncompressible Fluids

IncompressibleNewtonian fluids are governed by theNavier-Stokesequations,which couplethe velocity vectorfield :6 to ascalarpressure; :

6 � <6 �$� 6 � � � ; =& � 6 � �(� 6 �>0 . (4)

In two spacedimensions,thedivergencefreevelocity field6 possessesa streamfunction ? satisfying

� � ? � 6 . Inaddition,in 2D thevorticity, @ �A�'B 6 , satisfiesaverysim-ple advectiondiffusion equation,which canbe computedby takingthecurl of thefirst equationin (4) andusingsomebasicfactsaboutthegeometryin 2D:

@ � <6 � � @ � & � @C. (5)

Note herethat in 2D the vorticity is a scalarquantitythatis relatedto the streamfunction throughthe Laplaceop-erator,

� ? � @ . In the absenceof viscosity & �D0,

we obtain the Euler equationsfor inviscid flow. Both theinviscid andviscousproblems,with appropriateboundaryconditions,are globally well-posedin two spacedimen-sions. Solutionsexist for any smoothinitial conditionandthey dependcontinuouslyon the initial andboundarydata[17, 18,23,24,36, 40].

In termsof thestreamfunction,equation(5) impliesthatsteadystateinviscidflowsmustsatisfy

� � ? �$��� ? �>0 (6)

which saysthat the Laplacianof the streamfunction, andhencethe vorticity, musthave thesamelevel curvesasthestreamfunction.Theanalogyto imageinpaintingin thepre-vioussectionis now clear: thestreamfunctionfor inviscidfluids in 2D satisfiesthe sameequationasthe steadystateimageintensityequation(3).

Thepoint is thatin orderto solvetheinpaintingproblemproposedin the previoussection,we have to find a steadystatestreamfunctionfor theinviscidfluid equations,whichis aproblempossessingarich andwell developedhistory.

2.1. The stream function-image intensity anal-ogy

Themainanalogythatwebuild onin thispaperis theparal-lel betweenthestreamfunctionin a2D incompressiblefluidandtherole of imageintensityfunction

�in theinpainting

methoddescribedin Section1.2. This allows us to design

a new inpaintingmethodthatwill achieve thesamesteadyequation(3).

Let E bearegionin theplanein whichwewantto inpaintfrom surroundingdata.Assumethattheimageintensity

�GFis a smoothfunction(with possiblylargegradients)outsideof E andwe know both

��Fand

���GFon the boundary��E .

We now designa ‘Navier-Stokes’ basedmethodfor imageinpainting.In thismethodthefluid dynamicquantitieshavethefollowing parallelto quantitiesin theinpaintingmethod.

Navier-Stokes Imageinpainting

streamfunction ? Imageintensity�

fluid velocity 6 ��� � ? isophotedirection� � �

vorticity @ �!� ? smoothnessH �����fluid viscosity & anisotropicdiffusion &

Our goal is to solve a form of the Navier-Stokesequa-tions in the region to be inpainted. The methoddescribednext is basedonthevorticity-streamform (5) of theNavier-Stokes equations,however it is also possibleto considerothermethodsbasedon theprimitive variablesform (4).

For easeof notationwedenoteby H thesmoothness���

of the imageintensity. Insteadof solvinga transportequa-tion for

�asin (2), we solve a vorticity transportequation

for H :

�IH 2 ��3 <6 �8� H � & �(� �J)K�L, � H�, � � H#�/� (7)

wherethefunction ) allows for anisotropicdiffusionof thesmoothnessH . Theimageintensity

�whichdefinestheve-

locity field 6 �M� � � in (7) is recoveredby solvingsimul-taneouslythePoissonproblem

���N� HO� � , P8Q �>�GF . (8)

For ) �SR, the direct numericalsolution of of (7-8) is a

classicalway to solveboththedynamicfluid equationsandto evolvethedynamicstowardsasteadystatesolution[30].

For fluid problemswith smallviscosity & , theabovedy-namicscan take a long time to converge to steadystate,makingthemethodlesspractical.Insteadtherearepseudo-steadymethodsthatinvolve replacingthePoissonequation(8) with a dynamicrelaxationequation

�G� �'TVU ��� H�W ��0 �XTZY 0 � � P8Q �!�GF . (9)

wherethe parameterT determinesa rateof relaxation. Inour situation,diffusioncanresultin a blurring of sharpin-terfaces,gradientsof

�, in the inpaintingregion. Henceit

is oftendesirableto includeanisotropicdiffusionin theso-lution of

�. This canbe addeddirectly to the dynamical

problem(9) or asanadditionalstepin conjunctionwith thePoissonstep(8). In our test caseswith anisotropicdiffu-sion,wedid notfind asignificantdifferencebetweendirectsolutionof thePoissonequationin (8) versustherelaxationmethod(9).

3

Oncesteadystateis achieved,we haveeffectively founda solution of (3) for the intensity

�(perhapsmodified

slightly by theanisotropicdiffusion).Hencewe expectandindeedfind that the Navier-Stokes inpaintingmethodper-formsin many wayslike themethodproposedin [2]. Thereareseveraladvantagesto usingthe Navier-Stokesmethod.First, thereis a well-developedtheoreticaland numericalliteraturefor thisproblemonwhichwecanbuild inpaintingalgorithms.Secondthereis thepossibilityto testtheperfor-manceof themethodagainstanumberof classicalexamplesfrom fluid dynamics.Finally, weareableto implementthismethodefficiently andwe have a theoreticalframework inwhich to understandthe transportof informationfrom theexterior into theinpaintingregion.

2.2. Isophote continuity and boundary condi-tions for Navier-Stokes

Whenusingany PDE-basedmethodto do inpainting, theissueof boundaryconditionsbecomesvery important. Inorderto producea resultwhich, to theeye,doesnot distin-guishwherethe inpaintinghastakenplace,we mustat thevery leastcontinueboththeimageintensityanddirectionofthe isophotelines continuouslyinto the inpaintingregion.This meansthat any PDE-basedmethodinvolving the im-ageintensity

�mustenforceDirichlet (fixed

�) boundary

conditionsaswell asa conditionon thedirectionof�%�

ontheboundary. Immediatelyweseethatthisposesaproblemfor lower-order PDE-basedmethods. Indeed,any first orsecondorderPDE(includinganisotropicdiffusion) for thescalar

�couldtypically only enforceoneof theseboundary

conditions,the resultbeingan inpaintingwith discontinu-ities in the slopeof the isophotelines, or a methodwitha jump in

�itself on the boundary. From a mathematical

pointof view, to fix this,onecaneitherchoosea higheror-der equationfor

�, as in [2], that requiresmoreboundary

conditions,or considera vectorevolution for���

, which istheideaof theNavier-Stokesmethod.

The Navier-Stokesanalogyguarantees,in a very natu-ral way, continuity of the imageintensity function

�and

its isophotedirectionsacrosstheboundaryof theinpaintingregion. First considera solutionof theNavier-Stokesequa-tion (4) in primitive variablesform satisfyingthe classicalno-slipcondition 6 �[0

on the boundary�\E . This condi-tion guaranteestwo features:(a) that the streamfunction? mustbeconstanton theboundary, sincetheboundaryistrivially a streamlineof the flow; (b) that the directionofthe fluid velocity :6 is always tangentto the boundary. Amoregeneralform of the no-slip boundarycondition, forwhichwell-posednessis known, is to prescribethevelocityvector 6 � :6 F on the boundary. This would be the natu-ral choicefor a moving boundary. Specifyingthe velocityon theboundaryis equivalentto specifyingboththenormalandtangentialderivativesof the streamfunction ? on the

boundary, since 6 �]� � ? . However, specifyingthe tan-gentialderivativeof ? determines? on theboundaryup toa constantof integration,by simply integratingaroundtheboundarywith respectto its arc length. Similarly this in-formationdeterminesthedirectionof flow ontheboundary.The result is that if we solve the Navier-Stokesequationswith 6 fixed on the boundary, we obtaina solutionwith astreamfunction ? and velocity field 6 both of which arecontinuousup to theboundary.

For theNavier-Stokesinpaintingmethod,we inherit thecontinuityacrossthe boundary. For example,supposewefix� � �

on theboundary. Thensolving theNavier-Stokesinpaintingequationwith theseboundaryconditionswill notonly resultin continuousisophotes,but alsowill produceanimageintensityfunctionthatis continuousacross�\E .

In this paper, we are interestedin solving the vortic-ity streamform (7) which, practically speaking,requiresboundaryconditionsfor the streamfunction and the vor-ticity. For viscousNavier-Stokes, thereare several well-studiedmethodsfor doing this, all of which involve somesort of Dirichlet condition for @ on the boundary. In thecaseof afluid, informationabout@ outsideof theboundaryis typically not known, hencefirst andsecondorderaccu-ratemethodsuseinformationaboutthe streamfunctionattheprevioustimestepin orderto numericallycompute

� ?ontheboundary. In thecaseof images,wehavemoreinfor-mationabout

���outsidethe boundaryandcanusethis to

constructaccurateboundaryconditionsfor H �M���at the

boundary. For a discussionof how this is treatedin fluids,see[30] Chapter6.

2.3. Existence and uniqueness of solutions

TheNavier-Stokesbasedinpaintingmethodinheritsa the-oretical framework from the mathematicaltheory of theNavier-Stokesequations.Althougha full treatmentof suchissuesis beyond the scopeof this paper, we discusstheproblemof uniquenessandits relevanceto inpainting.

Firstwenotethatwithoutthepresenceof viscosityin themethodwedonothaveauniquesteady-statesolution.Thiscanbeseenin thefollowing simpleexample,motivatedbyproblemsinvolving themathematicalconcentrationof vor-ticity in solutionsequencesof the Euler equations[8, 12].Considerthe problem(6) with 6 �^0

on ��E . Consideradisk 1 �L_ of radius `ba , centeredat thepoint c F , containedinside E . Let Hed$�gf�� and H � �gf�� be two differentcompactlysupportedfunctions on U 0 � R W so that h dF Hji��Jf��kfmlmf �X0

.FromeachH i wecanconstructanexact“radial eddy”solu-tion of theinviscid ( & �n0 ) fluid equations:thevorticity @oiis givenas @ i �pcK� � H i ��q �Gr8sK� q_ � , , cV�Oc F ,�tua , andzerooth-erwise. This resultsin a streamfunctionsatisfying(where

4

wedenotef � , c��'c F , )?�vi �gf�� �

Rfw'xFzy @ i � y ��l y

for0 tufOt!`{a . Notethat ? vi �gf�� �>0 for aztuf%t>`ba . We

canintegratethe equationfor ? vi to recover ? i wheretheconstantof integration is chosenso that ?�i is identicallyzeroin a(tuf�tA`{a . Outsideof thedisk 1 �L_ we continue?|i and @}i aszero.Note thatthis is a ~�� continuation,byconstruction. Sincethe initial functions H d and H � werechosento be different, but both satisfyingthe meanzeroconstraint,the result is two different smoothsolutionsofthe inviscid steadystateproblem. In termsof the actualimage,theresultwould betwo differentbull’s eye patternson a disk in the interior of E . Note that this constructionworksfor any diskor for multipledisks.Soit is possibletoconstructa wide rangeof differentsolutionssatisfyingthesamezerovelocityboundarydata.

At theotherextremewhenviscosityis sufficiently large,for smoothboundariesthere is a unique solution of thesteadyNavier-Stokesequations[11]. Forviscousflowswithmoderateandsmallviscositytheproblemis morecomplex;for example,thereareclassicalexperimentalexamplesinwhich known steadystatesolutionshave varying stabilitypropertiesdependingon the viscosity. For example,in thecaseof Couetteflow, the fluid is constrainedto move be-tweentwo concentriccylindersrotatingatdifferentspeeds.Thereis anexact,radiallysymmetric,solutionthatsatisfiestheNavier-Stokesequationfor all viscosities.Howeverthissolutionis unstablefor sufficiently high viscosity, resultingin thecreationof multiple eddiesin suchexperiments(seee.g.[37] p. 80-81).

We expectthat Navier-Stokesbasedinpaintingmay in-herit someof thestability anduniquenessissuesknown forincompressiblefluids,althoughtheeffectof anisotropicdif-fusionis not clear. In fact,somedegreeof non-uniquenessof the steadystateproblemcouldbe favorablefor inpaint-ing,sincelargeinpaintingregionsmighthavemultiple‘pos-sible’ solutions,the bestchoiceof which might be deter-minedby patternmatchingor informationfrom a previousframe,asin thecaseof video. Ultimately we hopethatthechoiceof magnitudeandanisotropy of viscosity, aswell astheinitial conditionin theinpaintingregion,maydeterminethebestpossibleoutcomefor theinpaintingmethod.

Existenceanduniquenessof time-dependentsolutionsoftheNavier-Stokesequationswith isotropicviscosity() �(Rin (7)) is well established[17, 18, 23, 24, 36, 40]. In ourproblemweconsiderthedynamicswith anisotropicviscos-ity, both at the level of the smoothnessandat the level oftheintensity. This methodof selectivesmoothingis knownto beill-posedfor a wide classof functions) , however theill-posednesscanbe easily removed with a small amountof smoothnessappliedto thegradientinsidethefunction ) .

Catteet. al. [6] establishedanexistenceanduniquenessthe-ory which couldbeextendedto includenonlineartransporttermsfrom theNavier-Stokesequations.In fact,nonlineardiffusion hasa naturalphysicalanalogy;Non-Newtonianfluidshave a viscositythatdependslocally andnonlinearlyon the shear( , � 6 , ) of thefluid [32]. Anotherform of vis-cosityfor Navier-Stokesincludefourthorderhyperviscositywhichis commonlyusedin numericalsimulationsof turbu-lence[4].

3. Computational ExamplesWe now show via someexampleshow the Navier-Stokesinpaintingmethodperforms.In eachexampledescribedbe-low, theNavier-Stokesequationsaresolvedin the inpaint-ing region. We start by computingthe vorticity H fromthe image

�, usinginformationfrom the exterior to deter-

minetheboundaryvorticity. Weevolvethevorticity streamform (7), usinga simpleforwardEulertime stepping,withcentereddifferencesin spacefor the diffusion anda min-modmethod[29] for theconvectionterm. Thediffusionisanisotropic.

After onetime stepof (7) we computethe imageinten-sity

�by solvingthePoissonequation(8) usingtheJacobi

iterationmethod.Fromthisupdated�

werecomputeH andstartagain. Every few stepswe performanisotropicdiffu-sion on

�, which helpsto sharpenedges. Steadystateis

achievedafter N iterationsof this cycle, typically N=300.Wecansetthealgorithmto stopautomaticallywhen

�does

not changeappreciably. Parametersfor thealgorithmhavebeenchosenin sucha way asto work for a wide rangeofexamples: l�3 �[0 . 0�R , l�c � lm� ��R

, & � ` , � 0 stepsforJacobi,� stepsof anisotropicdiffusionof

�every

R�0cycles.

Theevolution(7) mightbemademoreefficientby usinganADI (alternatedirectionimplicit) method[39].

Whenworking with a color image,we performinpaint-ing on its threecomponentsseparately(oneluminanceim-ageandtwo chromaimages),andjoin theresultsat theend(in thesamewasasin [2]). Thealgorithmis programmedin tensof linesof C++ code. Theresultsshown herewereobtainedin a few secondsof CPU time of a standardPCunderLinux.

3.1. Inpainting of stillsIn this examplewe considera still image,shown in Fig-ure1 top,with thick linesobscuringpartsof thephotograph.Thebottomimagein thefigureshows theresultof Navier-Stokesinpaintingappliedto the photowith the inpaintingregioncorrespondingto theobscuringlines.

Notice the result is sharp,edgesarenot blurred insideE nor do color artifactsappear. This is a clearexampleforwhich texturesynthesisor manualselectionalgorithmsarenot a goodchoice,sincethe inpaintingregion is complex.

5

Figure1: (top)Colorphotographwith linesobscuringpartsof theimage,(bottom)Navier-Stokesinpaintingrestorationof thephotograph.

Thetopologyof theinpaintingregiondoesnotposeaprob-lem,unlike theapproachin [25, 26].

3.2. Video inpaintingFigure2 showsfour framesof the‘Foreman’videoin whichletteringoverlayhasbeenremovedusingNavier-Stokesin-painting. The inpaintingis doneframe-by-frameusingthesamemethodoutlined in the previous subsection.Noticethatthisapproach,thoughstraightforward,showsverygoodresults:sharp,nocolorartifacts,nomotionartifacts.

3.3. Super-resolutionThis exampleusesNavier-Stokesinpaintingto increasetheresolutionof an image. We have taken a detail

�of size

(n,m)of theoriginalimage,theright eyeof thegirl in Figure3. Thenwe increasedits size9-fold with replication(zero-th orderinterpolation)to obtain

� v of size(9n,9m).FinallyweapplyNavier-Stokesinpaintingto

� v , while fixing�

and� � �, from theoriginalsmallimage,onthelatticeof points

with coordinates(9i,9j). Noticehow inpaintingreproducesaroundiris.

In this example,we useNavier-Stokes inpaintingonlyfor the brightnesscomponentof the color image. A sim-

Figure2: Four framesfrom the ‘Foreman’video in whichtheletteringhasbeenremovedusingNavier-Stokesinpaint-ing (availableelectronically).

plerfilter is usedfor thechromacomponents.Betterresultsmay be achieved via harmonicmapsmoothingof vectors(see[35]). For an axiomatic/PDEbasedtechniquefor im-ageiterpolation,see[5].

4. Summary and Conclusions

In this paper, we show the importanceof computationalfluid dynamics(CFD) in general,andNavier-Stokesequa-tionsin particular, to visionproblems.AlthoughCFDideashave beenusedin computergraphicsfor such problemsasmodelingnaturalphenomena,in shapeanalysisfollow-ing [29], andasan interpretationfor anisotropicdiffusion[27], the connectionand applicationhere is to the bestof our knowledgenovel. Having sucha direct interdisci-plinaryconnectionhasseveralbenefitsfor thecomputervi-sioncommunity. First it bringswell-establishednumericalmethodsandtheoryto a problemof centralimportancein

6

Figure3: Top left: Original imageof girl, 120X160pixels.Top right: oneof her eyes (19x27pixels) magnifiedwithzerothorderzoomto 169x241pixels,Bottomleft: bicubicinterpolation,Bottomright: Navier-Stokesinpainting.Notehow theedgesaresmootherwith theNSapproach,e.g.,theeyeball.

vision research.Second,thevery closeconnectionto CFDhasthepotentialto draw morepeoplefrom thatcommunityto exploreproblemsin imageprocessingandcomputervi-sion. This canonly serve to acceleratethedevelopmentofthefield.

Although we presenta new methodthat seemsto per-form well, thereare several interestingproblemsand is-suesto beworkedout thatshouldgive improvements.Firstthereis the designof the numericalalgorithm. We useavorticity-streambasedmethoddueto its direct connectionto themethodintroducedin [2]. Howeveronecanconsidermany othermethods,includingdirectsolutionof theprim-itive variablesform of theNavier-Stokesequation(4) withlinearviscosityreplacedby anisotropicdiffusion.Thereis avastliteratureof CFD methodsthatcanbemodifiedto suitthe imageproblems.Moreover, thecloseanalogybetweenNavier-Stokesandthemethoddescribedin [2] suggeststhepossibilityto try hybridmethodsthatcrossbetweenthetwoequations,therebygiving a rangeof parametersthatcanbe

tunedto optimize the end result. Moreover, the theoreti-cal knowledgebasedevelopedin connectionwith Navier-Stokesmightprovidemoreinsightintoatheoryfor thePDEproposedin [2]. Finally thereis a very interestingconnec-tion thatwe have not yet exploredrelatedto thedynamicalevolution of the fluid equationsandstability of solutions.For example it is possiblethat classicaltwo-dimensionalfluid dynamicsinstabilities,like the Kelvin-Helmholtzin-stability [19, 14] for shearlayers,might allow us to gen-erateinterestingpatternformation in large inpainting re-gions. Thecontrolof suchinstabilitiesshouldbelinked tothemagnitudeof diffusionpresentin themodelandto thechoiceof initial condition.

At the very practicallevel, we have not dealtwith tex-tures,andthe parametersaresetmanually. In this respectwebelievethattheusermayberequiredto tuneboththepa-rametersandinitial datain theinpaintingregion to suit theparticularproblemat hand. However, our experiencesug-gestthatlargeclassesof problemscanbeefficiently solvedwith the sameor similar rangesof parametersand initialconditions.

We alsoexpect that this connection,whenextendedtohigherdimensions,will allow us to generalizethe inpaint-ing problemto otherfeatures(e.g.,optical flow) andothermodalities. We arecurrentlyworking on a video inpaint-ing technique,basedon the framework presentedhere,toautomaticallyswitchbetweentextureandstructureinpaint-ing, aswell asto permit theuseof informationfrom otherframeswhenit is available.

Acknowledgments

We thankS. Betelu, P. Constantin,R. Kohn,andA. Pardofor helpful comments. This work was supportedby IIEUruguay, ONR grantsN00014-01-1-0290andN00014-97-1-0509, the ONR Young InvestigatorAward, the Presi-dential Early CareerAwardsfor Scientistsand Engineers(PECASE),aNSFCAREERAward,andby theNSFLearn-ing andIntelligentSystemsProgram(LIS).

References

[1] C. Ballester, M. Bertalmio, V. Caselles,G. Sapiro, and J.Verdera,“Filling-In by Joint Interpolationof Vector FieldsandGrayLevels,” IEEE Trans.Img. Proc., to appear.

[2] M. Bertalmio,G. Sapiro,C. BallesterandV. Caselles,“Imageinpainting,” ComputerGraphics,SIGGRAPH2000, pp.417-424,July 2000.

[3] C. Braverman.Photoshopretouching handbook. IDG BooksWorldwide,1998.

7

[4] G. L. BrowningandH. O. Kreiss.“Comparisonof numericalmethodsfor thecalculationof two-dimensionalturbulence”,Math.Comp., 52(186), 369-388,1989.

[5] V. Caselles,J. M. Morel, and C. Sbert, “An axiomaticap-proach to image interpolation”, IEEE Trans. Img. Proc.,March1998.

[6] F. Catte,P. Lions, J. Morel, and T. Coll, “Image selectivesmoothingandedgedetectionby nonlineardiffusion” SIAMJ. Num.Anal., 29, 1: 182-193,1992.

[7] T. ChanandJ. Shen,“Mathematicalmodelsfor local deter-ministic inpaintings,” UCLACAMReport00-11, March2000(availableatwww.math.ucla.edu).

[8] R. J. DiPernaandA. J. Majda, “Concentrationsin Regular-izationsfor 2-D IncompressibleFlow”, Commun.Pur. Appl.Math., 40, 301-345,1987.

[9] A. EfrosandT. Leung,“Texturesynthesisby non-parametricsampling,” Proc. IEEE International ConferenceComputerVision, pp.1033-1038,Corfu,Greece,September1999.

[10] G. Emile-Male,TheRestorer’s Handbookof EaselPainting.VanNostrandReinhold,New York, 1976.

[11] G. P. Galdi, An Introduction to the MathematicalTheoryof the Navier-StokesEquations,Vol II., Nonlinear SteadyProblems, SpringerTracts in Natural Philosophy, vol. 39,Springer-Verlag,New York,1994.

[12] C. GreengardandE.Thomann,“On DiPerna-Majdaconcen-trationsetsfor two-dimensionalincompressibleflow”, Com-mun.Pur. Appl.Math.,41, 295-303,1988.

[13] D. Heeger and J. Bergen. Pyramid basedtexture analy-sis/synthesis. ComputerGraphics,229-238,SIGGRAPH95,1995.

[14] H. von Helmholtz. “Uber discontinuierlicheflussigkeitsbe-wegungen”,Monatsber. Berlin. Akad., 215–228,1868.

[15] A. Hirani andT. Totsuka.CombiningFrequencyandspatialdomaininformationfor fast interactiveimagenoiseremoval.ComputerGraphics,pp.269-276,SIGGRAPH96,1996.

[16] L. Joyeux,O. Buisson,B. Besserer, S. Boukir. “Detectionandremoval of line scratchesin motionpicturefilms,” Pro-ceedingsof CVPR’99,IEEE Int. Conf. on ComputerVisionandPattern Recognition,Fort Collins, Colorado,USA, June1999.

[17] V. Judovich, Math.Sb. N. S., 64, 562-588,1964.

[18] T. Kato,“On classicalsolutionsof thetwo dimensionalnon-stationaryEuler equation”,Arch. Rat.Mech. Anal., 25, 188-200,1967.

[19] Lord Kelvin. Nature, 50:524,549,573,1894.

[20] D. King. TheCommissarVanishes. Henry Holt andCom-pany, 1997.

[21] A.C. Kokaram,R.D. Morris, W.J.Fitzgerald,P.J.W. Rayner,“Detectionof missingdatain imagesequences,” IEEETrans.Img. Proc.11(4), 1496-1508,1995.

[22] A.C. Kokaram,R.D. Morris, W.J.Fitzgerald,P.J.W. Rayner.“Interpolation of missing data in image sequences,” IEEETrans.on Img. Proc.11(4), pp.1509-1519,1995.

[23] O. A. Ladyzhenskaya,TheMathematicalTheoryof ViscousIncompressibleFlow, GordonandBreachSciencePublishers,New York-London,1963.

[24] A. J.MajdaandA. L. Bertozzi,Vorticity andincompressibleflow, CambridgeUniv. Press,2001.

[25] S.Masnou,Filtr ageetDesocclusiond’Imagespar Methodesd’EnsemblesdeNiveau, These,Univ. Paris-Dauphine,1998.

[26] S. MasnouandJ.M. Morel, Level-linesbaseddisocclusion.5th IEEE Int’l Conf. on ImageProcessing,Chicago,IL. Oct4-7,1998.

[27] J. Monteil andA. Beghdadi,“A new interpretationandim-provementof theNonlinearAnisotropicDif fusionfor ImageEnhancement”,IEEE Trans. Patt. Anal. Mach. Int., 21(9),940-946,1999.

[28] M. Nitzberg, D. Mumford,andT. Shiota,Filtering, Segmen-tation,andDepth, Springer-Verlag,Berlin, 1993.

[29] S.OsherandJ.Sethian,“Frontspropagatingwith curvaturedependentspeed:algorithmsbasedon Hamilton-Jacobifor-mulations,” J. Comp.Phys., 79, 12-49,1988.

[30] R. Peyret andT. D. Taylor, Computationalmethodsfor fluidflow, SpringerVerlag,1983.

[31] D. L. RingachandR. Shapley, “Spatialandtemporalproper-tiesof illusory contoursandamodalcompletion”,Vision Re-search 36, 3037-3050,1996.

[32] M. Renardy, MathematicalAnalysisof ViscoelasticFlows,CBMS-NSFConferenceSeriesin Applied Mathematics73,Soc.Ind. Appl. Math.,Philadelphia,2000.

[33] L. Rudin,S.OsherandE. Fatemi.“Nonlineartotal variationbasednoiseremoval algorithms,” Phys.D 60, 259-268,1992.

[34] E. Simoncelli andJ. Portilla, Texture characterizationviajoint statisticsof waveletcoefficient magnitudes. 5th IEEEInt’l Conf.on ImageProcessing,Chicago,IL. Oct4-7,1998.

[35] B. Tang,G. Sapiro,andV. Caselles,“Dif fusion of generaldataon non-flat manifoldsvia harmonicmapstheory: Thedirectiondiffusioncase,” Int. Journal ComputerVision 36:2,pp.149-161,February2000.

[36] R. Temam,Navier-StokesEquations,North–HollandPub-lishingCompany, Amsterdam,secondedition,1986.

[37] M. VanDyke, An Album of Fluid Motion, The ParabolicPress,Stanford,CA, 1983.

8

[38] S. Walden, TheRavishedImage. St. Martin’s Press,NewYork, 1985.

[39] J. Weickert,B. M. ter HaarRomeny, andM. A. Viergever,“Efficient andreliableschemesfor nonlineardiffusionfilter-ing”, IEEE Trans.Img. Proc., 7(3):398–410,March1998.

[40] W. Wolibner, Math.Zeit., 37, 698-726,1933.

9