Компьютерное зрение и библиотека opencv, весна 2011:...
Post on 13-Jan-2017
445 Views
Preview:
TRANSCRIPT
OpenCVTutorial
VictorEruhimovItseez,CTO
http://opencv.willowgarage.comwww.willowgarage.com
www.itseez.com
Itseez
Outline
• OpenCVOverview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications
2
OpenCV Czar
OpenCVOverview:OpenCVOverview:
General Image Processing Functions
Machine Learning:• Detection,• Recognition
Segmentation
Tracking
Matrix Math
Utilities and Data Structures
Fitting
Image Pyramids
Camera calibration,Stereo, 3D
TransformsFeatures
Geometric descriptors
Robotsupport
opencv.willowgarage.com >500algorithms
4GaryBradski
CLASSIFICATION/REGRESSION
CLUSTERING
TUNING/VALIDATION
CLASSIFICATION/REGRESSION(new)FastApproximateNN(FLANN)(new)ExtremelyRandomTreesCARTNaïveBayesMLP(Backpropagation) StatisticalBoosting,4flavorsRandomForestsSVMFaceDetector(Histogrammatching) (Correlation)
CLUSTERINGK-MeansEM(Mahalanobisdistance)
TUNING/VALIDATIONCrossvalidationBootstrappingVariableimportanceSamplingmethods
Machine Learning Library (MLL) AACBAABBCBCC
AAA
AACACB CBABBC
CCB
BCC
ABBCCB
BC A BBC
CBB
55http://opencv.willowgarage.com
OpenCV History
6GaryBradski
Willow
10
5
0
• Originalgoal:• Acceleratethefieldbyloweringthebartocomputervision• FindcompellingusesfortheincreasingMIPSoutinthemarket
• Timeline:
• Staffing:• Climbedin1999toaverage7firstcoupleofyears• Starting2003supportdeclinedbetweenzeroandonewithexception
oftransferringthemachinelearningfrommanufacturingworkIled(equivalentof3people).
• SupporttozerothecoupleofyearsbeforeWillow.• 5peopleoverthelastyear 6
NewDirectoryStructure• Re-Organizedintermsofprocessingpipelines• Codesite:https://code.ros.org/gf/project/opencv/– Core– Calibration,features,I/O,imgprocessing– MachineLearning,Obj.Rec– Python
• ~2.5Mdownloads
OtherLanguages
OpenCVConceptualStructure
Python
Java(TBD)
Machinelearning
HighGUI
SSETBBGPUMPU
Modules
CORECORE
imgprocimgproc
ObjectDetection Features2d Calib3d
Stereo
VOSLAM(TBD)
Stitching(TBD)
UserContrib
OperatingsystemOperatingsystem
C
C++
OpenCV Tends Towards Real Time
http://opencv.willowgarage.com
SoftwareEngineering• Workson:
– Linux,Windows,MacOS
• Languages:– C++,Python,C
• Onlinedocumentation:– Onlinereferencemanuals:C++,C andPython.
• We’vebeenexpandingUnittestcode• WillsoonstandardizeoncxxorGoogle’stestsystem.• TESTCOVERAGE:
License• BasedonBSDlicense• Freeforcommercialorresearchuse
– Inwholeorinpart– Doesnotforceyourcodetobeopen– Youneednotcontributeback
• Wehopeyouwillcontributeback,recentcontribution,C++wrapperclassusedforGoogleStreetMaps*
GaryBradski(c)2008 11
*ThankstoDanielFilip11GaryBradski,2009
What’saddedinDecember2010OpenCV 2.2?
• Detector/Descriptorpipeline(Features2D)– Manysupportingdetectorsanddescriptorfeatures
• EasyinterfacetoPascalVOC• BOWandLatentSVMclassificationengines• ExperimentalUserContrib• Focusdetector?• Visualization(“HighGUI”)willbebasedonQt• OfficialsupportofAndroidOS• UpdatedFLANNlibrary• LimitedCudasupport(stereo)
What’sinProgress?
• GPUsupportthroughoutthelibrary• Morefunctionalityinfeatures2d• Betterposeestimationalgorithms
– ePnP– Stereoposeestimation
• Circlespatterndetector• BettersupportofAndroid• Supportforgoogle test
WhereisOpenCVUsed?
2Mdownloads
•Wellover2Mdownloads
ScreenshotsbyGaryBradski,2005
• GoogleMaps,Googlestreetview,GoogleEarth,Books• AcademicandIndustryResearch• Safetymonitoring(Damsites,mines,swimmingpools)
• Securitysystems• Imageretrieval• Videosearch• Structurefrommotioninmovies• Machinevisionfactoryproductioninspectionsystems• Robotics
Useful OpenCV Links
1515GaryBradski,2009
OpenCVWiki:http://opencv.willowgarage.com/wiki
OpenCVCodeRepository:svn co https://code.ros.org/svn/opencv/trunk/opencv
NewBookonOpenCV:http://oreilly.com/catalog/9780596516130/
Or,directfromAmazon:http://www.amazon.com/Learning-OpenCV-Computer-Vision-Library/dp/0596516134
Codeexamplesfromthebook:http://examples.oreilly.com/9780596516130/
Documentationhttp://opencv.willowgarage.com/documentation/index.html
UserGroup(39717members):http://tech.groups.yahoo.com/group/OpenCV/join
Outline
• OpenCVOverview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications
GaryBradski,2009 16
MainStructures
New“Image”:cv::Mat
Matdoesreferencecounting, soitdoestherightthingwhenitgoesoutofscopeyoucanalsoeasilymakestlvectortsormapsoutofMat.
MatareSimple
MatM(480,640,CV_8UC3);//Makea640x480imgRectroi(100,200,20,40);//MakearegionofintMatsubM=M(roi);//Takeasubregion,
//nocopyisdone
Mat_<Vec3b>::iteratorit=subM.begin<Vec3b>(),itEnd=subM.end<Vec3b>();
//0outplacesinsubMwhereblue>redfor(;it!=itEnd;++it)if((*it)[0]>(*it)[2])(*it)[0]=0;
MatrixManipulation
SimpleMatrixOperations
SimpleImageProcessing
ImageConversions
Histogram
I/O
SerializationI/O
SerializationI/O
GUI(“HighGUI”)
CameraCalibration,Pose,Stereo
ObjectRecognition
samples/c
31
In...\opencv\samples\c
bgfg_codebook.cpp - Useofaimagevaluecodebookforbackgrounddetectionforcollectingobjects
bgfg_segm.cpp - Useofabackgroundlearningengineblobtrack.cpp - Engineforblobtrackinginimagescalibration.cpp - CameraCalibrationcamshiftdemo.c - Useofmeanshiftinsimplecolortrackingcontours.c - Demonstrateshowtocomputeanduse
objectcontoursconvert_cascade.c - Changethewindowsizeinarecognition
cascadeconvexhull.c - Findtheconvexhullofanobjectdelaunay.c - Triangulatea2Dpointclouddemhist.c - Showhowtousehistogramsforrecognitiondft.c - Discretefouriertransformdistrans.c - distancemapfromedges inanimagedrawing.c - Variousdrawingfunctionsedge.c - Edgedetectionfacedetect.c - Facedetectionbyclassifiercascadeffilldemo.c - Floodfillingdemofind_obj.cpp - DemouseofSURFfeaturesfitellipse.c - Robustelipse fittinghoughlines.c - Linedetectionimage.cpp - Showsuseofnewimageclass,CvImage();inpaint.cpp - Textureinfilltorepairimagerykalman.c - Kalman filterfortrackignkmeans.c - K-Meanslaplace.c - Convolveimagewithlaplacian.
letter_recog.cpp - ExampleofusingmachinelearningBoosting,Backpropagation(MLP)andRandomforests
lkdemo.c - Lukas-Canadaopticalflowminarea.c - Foracloudofpointsin2D,findmin
boundingboxandcircle.ShowsuseofCv_SEQ
morphology.c - DemonstratesErode,Dilate,Open,Closemotempl.c - Demonstratesmotiontemplates
(orthogonalopticalflowgivensilhouettes)mushroom.cpp - Demonstratesuseofdecisiontrees(CART)
forrecognitionpyramid_segmentation.c - Colorsegmentationinpyramidsquares.c - Usescontourprocessingtofindsquares
inan imagestereo_calib.cpp - Stereocalibration,recognitionanddisparity
mapcomputationwatershed.cpp - Watershedtransformdemo.
samples/C++
32
Samples/python
BookExamples
GaryBradski,2009 34
ch2_ex2_1.cpp Loadimagefromdiskch2_ex2_2.cpp Playvideofromdiskch2_ex2_3.cpp Addaslidercontrolch2_ex2_4.cpp Load,smoothanddsiplayimagech2_ex2_5.cpp Pyramiddownsamplingch2_ex2_6.cpp CvCannyedgedetectionch2_ex2_7.cpp PyramiddownandCannyedgech2_ex2_8.cpp Aboveprogramsimplifiedch2_ex2_9.cpp Playvideofromcameraorfilech2_ex2_10.cpp Readandwritevideo,doLogpolar
ch3_ex3_1.txt Matrixstructurech3_ex3_2.txt Matrixcreationandreleasech3_ex3_3.cpp Creatematrixfromdatalistch3_ex3_4.cpp AccessingmatrixdataCV_MAT_ELEM()ch3_ex3_5.cpp SettingmatrixCV_MAT_ELEM_PTR()ch3_ex3_6.txt Pointeraccesstomatrixdata
ch3_ex3_7.txt ImageandMatrixElementaccessfunctionsch3_ex3_8.txt Settingmatrixorimageelementsch3_ex3_9.cpp Summingallelementsin3channelmatrixch3_ex3_10.txt IplImageHeaderch3_ex3_11.cpp Useofwidthstepch3_ex3_12.cpp UseofimageROIch3_ex3_13.cpp ImplementinganROIusingwidthstepch3_ex3_14.cpp Alphablendingexamplech3_ex3_15.cpp SavingandloadingaCvMatch3_ex3_16.txt Filestoragedemoch3_ex3_17.cpp WritingconfigurationfilesasXMLch3_ex3_19.cpp ReadinganXMLfilech3_ex3_20.cpp HowtocheckifIPPaccelerationison
BookExamples
GaryBradski,2009 35
ch4_ex4_1.cpp Useamousetodrawboxesch4_ex4_2.cpp Useatrackbarasabuttonch4_ex4_3.cpp Findingthevideocodec
ch5_ex5_1.cpp UsingCvSeqch5_ex5_2.cpp cvThresholdexamplech5_ex5_3.cpp Combiningimageplanesch5_ex5_4.cpp Adaptivethreshiolding
ch6_ex6_1.cpp cvHoughCirclesexamplech6_ex6_2.cpp Affinetransformch6_ex6_3.cpp Perspectivetransformch6_ex6_4.cpp Log-Polarconversionch6_ex6_5.cpp 2DFourierTransform
ch7_ex7_1.cpp Usinghistogramsch7_ex7_2.txt EarthMover’sDistanceinterfacech7_ex7_3_expanded.cpp EarthMover’sDistancesetupch7_ex7_4.txt UsingEarthMover’sDistancech7_ex7_5.cpp Templatematching/CrossCorr.ch7_ex7_5_HistBackProj.cpp Backprojectionofhistograms
ch8_ex8_1.txt CvSeqstructurech8_ex2.cpp Contourstructurech8_ex8_2.cpp Findingcontoursch8_ex8_3.cpp Drawingcontours
BookExamples
GaryBradski,2009 36
ch9_ex9_1.cpp Samplingfromalineinanimagech9_watershed.cpp ImagesegmentationusingWatershedtransformch9_AvgBackground.cpp Backgroundmodelusinganaverage imagech9_backgroundAVG.cpp Backgroundaveragingusingacodebookcomparedtojustanaveragech9_backgroundDiff.cpp Usethecodebookmethodfordoingbackgrounddifferencingch9_ClearStaleCB_Entries.cpp Refinecodebooktoeliminatestaleentriescv_yuv_codebook.cpp CorecodeusedtodesignOpenCVcodebook
ch10_ex10_1.cpp OpticalflowusingLucas-Kanadeinanimagepyramidch10_ex10_1b_Horn_Schunck.cpp OpticalflowbasedonHorn-Schunckblockmatchingch10_ex10_2.cpp Kalmanfilterexamplecodech10_motempl.cpp Usingmotiontemplatesforsegmentingmotion.
ch11_ex11_1.cpp Cameracalibrationusingautomaticchessboardfindingusingacamerach11_ex11_1_fromdisk.cpp Doingthesame,butreadfromdiskch11_chessboards.txt Listofincludedchessboardsforcalibrationfromdiskexample
ch12_ex12_1.cpp Creatingabird’seyeviewofascene usinghomographych12_ex12_2.cpp ComputingtheFundamentalmatrixusingRANSACch12_ex12_3.cpp Stereocalibration,rectificationandcorrespondencech12_ex12_4.cpp 2Drobustlinefittingch12_list.txt ListofincludedstereoL+Rimagepairdata
ch13_dtree.cpp Exampleofusingadecisiontreech13_ex13_1.cpp Usingk-meansch13_ex13_2.cpp Creatingandtrainingadecisiontreech13_ex13_3.cpp Trainingusingstatisticalboostingch13_ex13_4.cpp FacedetectionusingViola-Jonescvx_defs.cpp Somedefinesforusewithcodebooksegmentatio
PythonFaceDetectorNode:1
37
#!/usr/bin/python"""Thisprogramisdemonstrationpython ROSNodeforfaceandobjectdetectionusinghaar-likefeatures.Theprogramfindsfacesinacameraimageorvideostreamanddisplaysaredboxaround them.Python implementationby:RomanStanchak,JamesBowman"""importroslibroslib.load_manifest('opencv_tests')importsysimportosfromoptparseimportOptionParserimportrospyimportsensor_msgs.msgfromcv_bridgeimportCvBridgeimportcv
#Parametersforhaardetection#FromtheAPI:#Thedefaultparameters(scale_factor=2,min_neighbors=3,flags=0)aretuned#foraccurateyetslowobjectdetection.Forafasteroperationonrealvideo#imagesthesettingsare:#scale_factor=1.2,min_neighbors=2,flags=CV_HAAR_DO_CANNY_PRUNING,#min_size=<minimumpossiblefacesize
min_size=(20,20)image_scale=2haar_scale=1.2min_neighbors=2haar_flags=0
TheSetup
PythonFaceDetectorNode:2
38
if__name__ == '__main__':
pkgdir=roslib.packages.get_pkg_dir("opencv2")haarfile=os.path.join(pkgdir,"opencv/share/opencv/haarcascades/haarcascade_frontalface_alt.xml")
parser=OptionParser(usage="usage:%prog[options][filename|camera_index]")parser.add_option("-c","--cascade", action="store",dest="cascade", type="str",help="Haarcascadefile,default%default", default=haarfile)(options,args)=parser.parse_args()
cascade=cv.Load(options.cascade)br=CvBridge()
defdetect_and_draw(imgmsg):img=br.imgmsg_to_cv(imgmsg,"bgr8")#allocatetemporaryimagesgray=cv.CreateImage((img.width,img.height),8,1)small_img=cv.CreateImage((cv.Round(img.width/image_scale),
cv.Round(img.height/image_scale)),8,1)
#convertcolorinputimagetograyscalecv.CvtColor(img,gray,cv.CV_BGR2GRAY)
#scaleinputimageforfasterprocessingcv.Resize(gray,small_img,cv.CV_INTER_LINEAR)
cv.EqualizeHist(small_img,small_img)
if(cascade):faces=cv.HaarDetectObjects(small_img,cascade,cv.CreateMemStorage(0),
haar_scale,min_neighbors,haar_flags,min_size)iffaces:for((x,y,w,h),n)infaces:#theinputtocv.HaarDetectObjectswasresized,soscalethe#boundingboxofeachfaceandconvertittotwoCvPointspt1=(int(x*image_scale),int(y*image_scale))pt2=(int((x+w)*image_scale),int((y+h)*image_scale))cv.Rectangle(img,pt1,pt2,cv.RGB(255,0,0),3,8,0)
cv.ShowImage("result",img)cv.WaitKey(6)
rospy.init_node('rosfacedetect')image_topic=rospy.resolve_name("image")rospy.Subscriber(image_topic,sensor_msgs.msg.Image,detect_and_draw)rospy.spin()
TheCore
Outline
• OpenCV Overview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications
GaryBradski,2009 39
NewC++API:UsageExample
doublecalcGradients(constIplImage*src,intaperture_size=7){CvSizesz=cvGetSize(src);IplImage*img16_x=cvCreateImage(sz,IPL_DEPTH_16S,1);IplImage*img16_y=cvCreateImage(sz,IPL_DEPTH_16S,1);
cvSobel(src,img16_x,1,0,aperture_size);cvSobel(src,img16_y,0,1,aperture_size);
IplImage*imgF_x=cvCreateImage(sz,IPL_DEPTH_32F,1);IplImage*imgF_y=cvCreateImage(sz,IPL_DEPTH_32F,1);
cvScale(img16_x,imgF_x);cvScale(img16_y,imgF_y);
IplImage*magnitude=cvCreateImage(sz,IPL_DEPTH_32F,1);cvCartToPolar(imgF_x,imgF_y,magnitude);double res=cvSum(magnitude).val[0];
cvReleaseImage(&magnitude);cvReleaseImage(&imgF_x);cvReleaseImage(&imgF_y);cvReleaseImage(&img16_x);cvReleaseImage(&img16_y);
returnres;}
doublecontrast_measure(constMat&img){Matdx,dy;
Sobel(img,dx,1,0,3,CV_32F);Sobel(img,dy,0,1,3,CV_32F);magnitude(dx,dy,dx);
returnsum(dx)[0];}
FocusDetector
C: C++:
40
Pyramid/**Makeanimagepyramidwithlevelsofarbitraryscalereduction (0,1)*M Input image*reduction Scalingfactor1>reduction>0*levels Howmanylevelsofpyramid*pyr stdvectorcontainingthepyramid*sz Thewidthandheightofblurringkernel,DEFAULT3*sigma ThestandarddeviationoftheblurringGaussianDEFAULT0.5*RETURNS Numberoflevelsachieved*/intbuildGaussianPyramid(constMat&M,double reduction,intlevels,
vector<Mat>&pyr, intsz=3,floatsigma=0.5){if(M.empty())return0;pyr.clear();//Clearitupif((reduction<=0.0)||(reduction>=1.0))return0;MatMblur,Mdown=M;pyr.push_back(Mdown);Sizeksize=Size(sz,sz);intL=1;for(;L<=levels;++L){if((reduction*Mdown.rows)<=1.0||(reduction*Mdown.cols)<=1.0)break;GaussianBlur(Mdown,Mblur,ksize,sigma,sigma);resize(Mblur,Mdown,Size(),reduction,reduction);pyr.push_back(Mdown);
}returnL;
}
Outline
• OpenCV Overview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications
GaryBradski,2009 42
Canny Edge DetectorCanny Edge Detector
43
DistanceTransformDistanceTransform• Distancefieldfromedgesofobjects
FloodFilling
44
Hough Transform
45GaryBradski,AdrianKahler2008
SpaceVariantvision:Log-PolarTransform
46
ScreenshotsbyGaryBradski,2005
Scale Space
voidcvPyrDown(IplImage* src,IplImage* dst,IplFilter filter=IPL_GAUSSIAN_5x5);
voidcvPyrUp(IplImage* src,IplImage* dst,IplFilter filter=IPL_GAUSSIAN_5x5);
47
ChartbyGaryBradski,2005
Thresholds
48
ScreenshotsbyGaryBradski,2005
Histogram Equalization
49ScreenshotsbyGaryBradski,2005
Contours
50
MorphologicalOperationsExamplesMorphologicalOperationsExamples
• Morphology- applyingMin-Max. Filtersanditscombinations
Opening IoB= (IΘB)⊕BDilatation I⊕BErosion IΘBImage I
Closing I•B= (I⊕B)ΘB TopHat(I)= I - (IΘB) BlackHat(I)= (I⊕B) - IGrad(I)= (I⊕B)-(IΘB)
Imagetextures
• Inpainting:• Removesdamagetoimages,inthiscase,itremovesthetext.
Segmentation
• Pyramid,mean-shift,graph-cut• Here:Watershed
5353
ScreenshotsbyGaryBradski,2005
• GraphCutbasedsegmentation
54
RecentAlgorithms:RecentAlgorithms: GrabCut
ImagesbyGaryBradski,©2010
Motion Templates Motion Templates (work with James Davies)
• Objectsilhouette• Motionhistoryimages• Motionhistorygradients• Motionsegmentationalgorithm
silhouette MHI MHG
55
ChartsbyGaryBradski,2005
Segmentation,MotionTrackingand
GestureRecognition
PoseRecognition
MotionSegmentation
GestureRecognition
MotionSegmentation
ScreenshotsbyGaryBradski,2005
NewOpticalFlowAlgorithms
…)
// opencv/samples/c/lkdemo.cint main(…){…CvCapture* capture = <…> ?
cvCaptureFromCAM(camera_id) : cvCaptureFromFile(path);
if( !capture ) return -1;for(;;) {
IplImage* frame=cvQueryFrame(capture);if(!frame) break;// … copy and process image
cvCalcOpticalFlowPyrLK( …)cvShowImage( “LkDemo”, result );c=cvWaitKey(30); // run at ~20-30fps speedif(c >= 0) {
// process key}}cvReleaseCapture(&capture);}
lkdemo.c,190lines(needs cameratorun)
∑∑ ⎥⎦
⎤⎢⎣
⎡=
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡=∂∂=∂
=∂⋅
⋅∂∂+⋅∂∂=∂∂−
=+++
y
xt
yyx
yxx
II
IbIII
IIIGyxX
bXG
dtdyyIdtdxxItItyxIdttdyydxxI
,,
,),,(
,
);/(/)/(//);,,(),,(
2
2
TrackingwithCAMSHIFT
• Controlgamewithhead
ScreenshotsbyGaryBradski,2005
Projections
ScreenshotsbyGaryBradski,2005
Stereo…DepthfromTriangulation
• Involvedtopic,herewewilljustskimthebasicgeometry.
• Imaginetwoperfectlyalignedimageplanes:
60
Depth“Z”anddisparity“d”areinverslyrelated:
Stereo• Inalignedstereo,depthisfromsimilartriangles:
• Problem:Camerasarealmostimpossibletoalign• Solution:Mathematicallyalignthem:
61
rl
rl
xxfTZ
ZT
fZxxT
−=⇒=
−
−− )(
All:GaryBradskiandAdrianKaehler:LearningOpenCV
StereoRectification• Algorithmstepsareshownatright:• Goal:
– Eachrowoftheimagecontainsthesameworldpoints– “Epipolar constraint”
62
Result:Epipolaralignmentof features:
All:GaryBradskiandAdrianKaehler:LearningOpenCV
Outline
• OpenCV Overview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications
GaryBradski,2009 63
Features2dcontents
Detection
Detectorsavailable• SIFT• SURF• FAST• STAR• MSER• HARRIS• GFTT (Good Features
ToTrack)
Description
Descriptorsavailable• SIFT• SURF• Calonder• Ferns• Oneway
Matching
Matchersavailable• BruteForce• FlannBased• BOW
Matchesfilters(underconstruction)• Crosscheck• Ratiocheck
Detectorinterfaces
classFeatureDetector{public:virtual~FeatureDetector() {}
//Detectkeypoints inanimage.virtualvoiddetect(constMat&image,vector<KeyPoint>&keypoints,
constMat&mask=Mat())const=0;//Detectkeypoints inanimageset.voiddetect(constvector<Mat>&imageCollection,
vector<vector<KeyPoint>>&pointCollection,constvector<Mat>&masks=vector<Mat>())const;
virtualvoid read(constFileNode&fn){}virtualvoidwrite(FileStorage& fs )const{}
protected:…};
Creatingadetector
• StaticallySurfFeatureDetector detector;
• Usingclassfactorycv::Ptr<FeatureDetector>detector=createFeatureDetector(“SURF”);
Runningdetector
Matimg =imread("test.png");vector<KeyPoint>keypoints;
SurfFeatureDetectordetector;detector.detect(img,keypoints );
Descriptorinterfaces
• Fordescriptorsthatcanberepresentedasvectorsinmultidimensionalspace:DescriptorExtractor andDescriptorMatcher
• Moregeneralinterface(oneway,decision-tree-baseddescriptors):GenericDescriptorMatcher
DescriptorExtractor interfaces
classCV_EXPORTSDescriptorExtractor{public:virtual~DescriptorExtractor(){}//Computethedescriptorsforasetofkeypoints inanimage.virtualvoidcompute(constMat&image,vector<KeyPoint>&keypoints,
Mat&descriptors)const=0;//Computethedescriptorsforakeypoints collectiondetectedinimagecollection.voidcompute(constvector<Mat>&imageCollection,
vector<vector<KeyPoint>>&pointCollection,vector<Mat>&descCollection )const;
virtualvoidread(constFileNode&){}virtualvoidwrite(FileStorage&)const{}virtualint descriptorSize()const=0;virtualint descriptorType()const=0;
protected:…};
DescriptorExtractor creating
• StaticallySurfDescriptorExtractor descriptorExtractor;
• Usingclassfactorycv::Ptr<DescriptorExtractor>descriptorExtractor =createDescriptorExtractor("SURF");
DescriptorExtractor running
Ptr<FeatureDetector>detector=createFeatureDetector("FAST");
Ptr<DescriptorExtractor>descriptorExtractor =createDescriptorExtractor("SURF");
vector<KeyPoint>keypoints;detector->detect(img,keypoints );Matdescriptors;descriptorExtractor->compute(img,keypoints,descriptors);
DescriptorMatcher interfaces
• Twogroupsofmatchmethods– tomatchdescriptorsofimagepair– tomatchdescriptorsofoneimagetoimageset
• Eachgroupconsistsfromtreetypemethods– match()– knnMatch()– radiusMatch()
Matchingofimagepair//detectingkeypointsSurfFeatureDetector detector;vector<KeyPoint>keypoints1, keypoints2;detector.detect(img1,keypoints1 );detector.detect(img2,keypoints2 );
//computing descriptorsSurfDescriptorExtractor extractor;Matdescriptors1,descriptors2;extractor.compute(img1,keypoints1, descriptors1 );extractor.compute(img2,keypoints2, descriptors2 );
//matchingdescriptorsBruteForceMatcher<L2<float>>matcher;vector<DMatch>matches;matcher.match(descriptors1,descriptors2,matches);
Visualizekeypoints
Visualizematches
Matimg_points;drawKeypoints(img,keypoints,img_points );namedWindow(“keypoints”,1);imshow(“keypoints”,img_points );waitKey();
Matimg_matches;drawMatches(img1,keypoints1,
img2,keypoints2,img_matches);namedWindow(“matches”,1);imshow(“matches”,img_matches );waitKey();
Runningthesample
• DownloadOpenCV• Compile• Runmatcher_simple:
bin/matcher_simple ../../opencv/samples/c/box.png../../opencv/samples/c/box_in_scene.png
• Selectadetectorthatgivesthemaximumnumberofkeypoints
• SwitchSIFTandSURFdescriptors
Cross-checkoutliermatchfiltering
BruteForceMatcher<L2<float>>descriptorMatcher;vector<DMatch>filteredMatches12,matches12,matches21;descriptorMatcher.match(descriptors1,descriptors2,matches12);descriptorMatcher.match(descriptors2,descriptors1,matches21);
for(size_t i =0;i <matches12.size(); i++){DMatch forward=matches12[i];DMatch backward=matches21[forward.trainIdx];if(backward.trainIdx ==forward.queryIdx )filteredMatches12.push_back(forward);
}
Ratiotesttofiltermatches
]1,0(21∈=
MinDistMinDistRatio (lessisbetter)
outlierelseinlierthresholdRatioif ,)3.0( ⇒<
Calculatinginliers(planarobjectscase)
• Detectkeypoints• Findmatchesusingdescriptors• Filtermatchesusingcross-check• Calculatebesthomography• Filteroutliers• Run
bin/descriptor_extractor_matcher SURFSURF../../opencv/samples/c/box.png../../opencv/samples/c/box_in_scene.png3
Thelastparameteristhereprojection thresholdforRANSAC
Detectortestbench
• Measuresofdetectorrepeatabilityaretakenfrom– K.Mikolajczyk,Cordelia Schmid,“Scale&AffineInvariantInterestPoint
Detectors”,IJCV60(1),63–86,2004.– K.Mikolajczyk etal,AComparisonofAffineRegionDetectors,IJCV
65(1/2):43-72,2005.
• Testimagesaretakenfromhttp://www.robots.ox.ac.uk/~vgg/data/data-aff.html
• Testbench islocatedinopencv_extra/testdata/cv/detectors_descriptors_evaluation/detectors
Descriptortestbench
• Measuresofdescriptormatchingaccuracyaretakenfromhttp://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolaj
czyk_pami2004.pdf
• Testimagesaretakenfromhttp://www.robots.ox.ac.uk/~vgg/data/data-aff.html
• Testbench islocatedinopencv_extra/testdata/cv/detectors_descriptors_evaluation/descriptors
OpenCV andROS
• Opencv2packagetofetchandcompileopencv• Messages:
– sensor_msgs::Image– sensor_msgs::CameraInfo
• cv_bridge toconvertbetweenmessagesandimages
• image_geometry::PinholeCameraModel andimage_geometry::StereoCameraModel tomanage2d<->3dconversions
Q&A
• Foilswillbeavailableathttp://itseez.com
top related