shadowdraw - university of texas at...
TRANSCRIPT
ShadowDrawReal-TimeUserGuidanceforFreehandDrawing
HarshalPriyadarshi
Demo
ComponentsofShadow-Draw
• InvertedFileStructureforindexing• Databaseofimages• CorrespondingEdgemaps
• Querymethod• Dynamicallyretrievesmatchingimages• Alignsthemtoevolvingdrawing• Weighsthembasedonmatchingscore,toformshadow
• UI• Displaysashadowofweightededgemapsbeneaththeuser’sdrawing
DatabaseCreation
• ImageAcquisition• EdgeExtraction• PatchDescription• Min-hashEncoding
ImageAcquisition
• 30kimagesspanning40categoriesobtainedfrominternet• Scaledtoobtain300x300images
ProblemsHandled:1. DirectSketchimagesarenotabundant2. DiverseBackground– Stillhasgood
edgesyoumightwanttodraw3. Photographerbiastorescue
DatabaseCreation
• ImageAcquisition• EdgeExtraction• PatchDescription• Min-hashEncoding
EdgeExtraction(Step1)
• Computethelocaledgemagnitude(pm)andorientation(po)ateachpixelusingsteerablefilters
Input Output
EdgeExtraction(Step2)• Normalizetheedgemagnitude• Needtodetectlong,coherentedgesevenwhenfaint(i.e.,notsimplyedgeswithstrongmagnitude)
Input Output
EdgeExtraction(Step3)• MessagePassingforlengthestimation
Input Output
Seethedifference
CannyEdgeDetection(NonMaxSuppression)
Credits- KristenGrauman (bothimages)
DatabaseCreation
• ImageAcquisition• EdgeExtraction• PatchDescription• Min-hashEncoding
PatchDescriptors
• LowDimensionalBiCE descriptor• Encodesahistogramofedgepositionsandorientations
• Doneoveroverlapping60x60patcheswith50%overlap• AsmappinganedgeimageEtoincompleteandevolvingdrawing.
• SIFT/DaisyvsBiCE• Formerreliesonrelativestrengthofedgemagnitudestoprovidediscriminability• ThusreducedperformancecomparedtoBiCE onourtask,whereedgestrengthsarenotimportant.
BiCE descriptor(Steps)1. LocalNormalizationofimagepatchgradients• Removevariationinrelativegradientmagnitudes
2. Binningofnormalizedgradients• usingposition,orientation,andlocallinearlength ofimage
3. Binarization ofnormalizedgradienthistogram• Encodesthepresenceofedges
Presence/absenceofEdgeispreservedacrossmatchingpatches,Theremagnitudemightnot.
Step1 NormalizedGradient(g_cap)
OriginalGradient(g)
Step2
OrientationalignedbinningRobustnesstoorientationchanges
InitialBinningw.r.t.justpositionandorientation.
IncreasingDiscriminabilityLongCoherentEdgesvsShorterTexturedEdges
CalculateEdgeLength
Discretizationinto2binsbyweightbasednormalizedgradientsplitting
Alpha,betaà Tunablethresholdingparams
Withgaussian Bluralongx,y,thetadimension
WHY??
Step3(SubsamplingandBinarization)
• Subsampletodiscretesetofvaluesforx,y,thetaandlength
• Value=1(topTpercentofbinswithhighestfrequency)• Value=0(rest)
• FlattentogettheBiCE desciptor
DatabaseCreation
• ImageAcquisition• EdgeExtraction• PatchDescription• Min-hashEncoding
RetrievalandClusteringEfficiency
• Reducedimensionà ImproveClusteringà ImproveRetrieval
PreservesMaximumJaccard Similarity
Minhashing (Iapologizeformyterribleanimation)
Index VectorA VectorB
1 1 0
2 1 0
3 0 1
4 0 0
5 1 0
6 0 0
H(Index)4
6
1
5
3
2
Index VectorA VectorB
1 1 0
2 1 0
3 0 1
4 0 0
5 1 0
6 0 0
Index VectorA VectorB
1 5 3
Whatarethesevectors?
Sketches(n,k)andInvertedIndexing• Toincreaseprecision
Index VectorA VectorB
1 5 3
2 2 1
3 1 1
4 4 6
5 3 2
6 1 6
Khashfunctions
• ToincreaserecallIndex VectorA VectorB
1 5 3
2 2 1
3 1 1
4 4 6
5 3 2
6 1 6
Index VectorA VectorB
1 5 3
2 2 1
3 1 1
4 4 6
5 3 2
6 1 6
Index VectorA VectorB
1 5 3
2 2 1
3 1 1
4 4 6
5 3 2
6 1 6
Index VectorA VectorB
1 5 3
2 2 1
3 1 1
4 4 6
5 3 2
6 1 6
Index VectorA VectorB
1 5 3
2 2 1
3 1 1
4 4 6
5 3 2
6 1 6
NsetsofKhashfunctions
ComponentsofShadow-Draw
• InvertedFileStructureforindexing• Databaseofimages• CorrespondingEdgemaps
• Querymethod• Dynamicallyretrievesmatchingimages• Alignsthemtoevolvingdrawing• Weighsthembasedonmatchingscore,toformshadow
• UI• Displaysashadowofweightededgemapsbeneaththeuser’sdrawing
QuerySteps
• DynamicallyRetrievingMatchingImages
• Aligning MatchingImagestoDrawnSketch
• ShadowCreationbyWeighting
ImageMatching
• Obtain CandidateMatches• Align CandidateMatcheswiththepartiallydrawnsketch• Assignweighttoeachcandidate’sedgeimage• ConstructShadowImage
Votecountforpatchforeachcandidateimage
CandidateMatchFinding
EdgeImageofSketch
BiCE sketchdescriptorforeachpatch
Top100imagesandcorrespondingoffsetforthehighlyvotedoffsetNote:Not100bestpatches
Xoffsetofthecandidatepatchfromtheusersketchpatch
ResultantCandidateFormat(ImageId,patchoffset-xdirection,patchoffset-ydirection)
QuerySteps
• DynamicallyRetrievingMatchingImages
• Aligning MatchingImagestoDrawnSketch
• ShadowCreationbyWeighting
AligningCandidateMatches
QuerySteps
• DynamicallyRetrievingMatchingImages
• Aligning MatchingImagestoDrawnSketch
• ShadowCreationbyWeighting
ImageWeightingWeightImage
ShadowImageConstruction
ShadowImage
EdgeCandidateImage
BlendingWeightImage
GlobalmatchingtermSpatiallyvaryingmatchterm
NormalizationTerm
3variables:• Globalmatchingterm(v)• Spatiallyvaryingmatchingterm(V)• VisibilityEnhancer(alpha) VisibilityEnhancer
ObtainingGlobalMatchingTerm(V)
UserSketch(8orientations)
CandidateImage(8orientations)
PositiveCorrelation
NegativeCorrelation
Imagenotoriented,theedgesitcapturesareoriented
GlobalMatchingSpatialMatching
Averageof5highesthi fromthecandidateset
GaussianBluronthepositivecorrelationimage
VisibilityEnhancer(alpha)
Whyisitavisibilityenhancer?
ExperimentalFindings
Robustnessto________ ????
PoorvsAveragevsGood
WHY?? WHY??
Complexity
• Strengths• Canhelpdrawingstructurallycomplexobjects• Helpspreservetheuniquestyleoftheusers• Isareal-timealgorithm
•Weaknesses• Leadstogoodshadowsonlyiftheinitialusersketchisnotallovertheplace.Otherwisemightconfusetheuser.• Atusslebetween guidanceand freedom.• Sketchingflowbias– Thewaywestartdrawingthesketchmightaffecttheshadowretrieved,andthusleadtoconfusioninitially,iftheuserisnotverycertainofeachdetail.
References
• OriginalPaper• http://vision.cs.utexas.edu/projects/shadowdraw/ShadowDrawSiggraph11.pdf
• SupportingPapers• LongEdgeDetector
• http://grail.cs.washington.edu/projects/gradientshop/demos/gs_paper_TOG_2009.pdf• BiCE descriptor
• http://larryzitnick.org/publication/BiCE_ECCV10.pdf
ThankYou