mesa lab multi-view image stitching guimei zhang mesa lab mesa (mechatronics, embedded systems and...
TRANSCRIPT
![Page 1: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/1.jpg)
MESA LABMESA LABMESA LABMESA LAB
Multi-view image stitching Multi-view image stitching
Guimei ZhangMESA MESA (Mechatronics, Embedded Systems and Automation)LABLAB
School of Engineering,University of California, Merced
E: [email protected] Phone:209-658-4838Lab: CAS Eng 820 (T: 228-4398)
June 16, 2014. Monday 4:00-6:00 PMApplied Fractional Calculus Workshop Series @ MESA Lab @ UCMerced
![Page 2: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/2.jpg)
MESA LABMESA LABMESA LABMESA LAB
Introduction
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-2/1024
Why to do some work on stitching?
1.Generate one panoramic image from a series of smaller, overlapping images.
2.The stitched image can also be of higher
resolutions than a panoramic image acquired by
a panoramic camera. The other is a panoramic
camera is more expensive.
![Page 3: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/3.jpg)
MESA LABMESA LABMESA LABMESA LABIntroduction
![Page 4: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/4.jpg)
MESA LABMESA LABMESA LABMESA LAB
Applications:
interactive panoramic viewing of
images , architectural walk-through , multi-
node movies and other applications associated
with modelling the 3D environment using images
acquired from the real world (digital surface
models/digital terrain models/ true orthophoto
and full 3D models)06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-4/1024
Introduction
![Page 5: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/5.jpg)
MESA LABMESA LABMESA LABMESA LAB
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-5/1024
Introduction
true orthophoto and full 3D models
![Page 6: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/6.jpg)
MESA LABMESA LABMESA LABMESA LAB
• What is multi-view?
capture images at different time, at different view points or using different sensor, such as camera, laser scanner, radar, multispectral camera…..
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-6/1024
Introduction
![Page 7: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/7.jpg)
MESA LABMESA LABMESA LABMESA LAB
2. MethodThe flowchart of producing a panoramic image
![Page 8: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/8.jpg)
MESA LABMESA LABMESA LABMESA LAB
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-8/1024
The main work is as following:
1. image acquired
2.How to perform image effective registration
3. How to fulfill image merging
Introduction
![Page 9: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/9.jpg)
MESA LABMESA LABMESA LABMESA LAB
• Image acquisition1. Use one camera, at different time or different viewpoints, to
capture images, so there is rotation or translation transformation or both of them. (R,T)
2. Use several cameras located in different viewpoint to capture
images. (R,T)
3. Use different sensors, such as camera, laser scanner, radar, or
multispectral scanner ….(fuse multi sensor information)
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-9/1024
Introduction
![Page 10: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/10.jpg)
MESA LABMESA LABMESA LABMESA LAB
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-10/1024
Introduction
Camera and tripod for acquisition by camera rotations Geometry of overlapping images
Need perform coordination transformation
![Page 11: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/11.jpg)
MESA LABMESA LABMESA LABMESA LAB
• Since the orientation of each imaging plane is different in acquisition method. Therefore, images acquired need to be projected onto the same surface, such as the surface of a cylinder or a sphere, before image registration can be performed.
• that means we have to perform coordinate transformation.
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-11/1024
Introduction
![Page 12: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/12.jpg)
MESA LABMESA LABMESA LABMESA LAB
• Image registration:
To form a larger image with a set of overlapping images, it
is necessary to find the transformations matrix(usually
rigid transformation, only parameters R and T) to align the
images. The process of image registration aims to find
the transformations matrix to align two or more
overlapping images. Because the projection( from the view
point through any position in the aligned images into the
3D world) is unique.
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-12/1024
Introduction
![Page 13: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/13.jpg)
MESA LABMESA LABMESA LABMESA LAB
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-13/1024
5th frame image 6th frame image
Registrated image
Introduction
![Page 14: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/14.jpg)
MESA LABMESA LABMESA LABMESA LAB
Multi-view point clouds registration and stitching based on SIFT feature
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-14/1024
1. Motivation
2. Method
3. Experiments
4. Conclusion
5. Discussion
SIFT: scale invariant feature transform
![Page 15: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/15.jpg)
MESA LABMESA LABMESA LABMESA LAB
1. Motivation
Problems: (Existed method for multi-view
point clouds registration in large scene)
1. Be restricted by the view angle of camera,
single or two viewpoint image can only obtain
local information of local scene;
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
![Page 16: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/16.jpg)
MESA LABMESA LABMESA LABMESA LAB
2. Existed methods need to add special markers
in large reconstruction scenes.
3. Existed method also need ICP(iterative
closest point ) iterating calculation, and can’t
eliminate interference of holes and invalid 3D
point clouds.
06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-16/1024
1. Motivation
![Page 17: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/17.jpg)
MESA LABMESA LABMESA LABMESA LAB
2. Method
• Our work based on Bendels[8] work, put forward a new algorithm of multi-view registration and stitching.
1. Generation 2D texture image of effective point
clouds;
2. we extract SIFT features and match them in
2D effective texture image;
![Page 18: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/18.jpg)
MESA LABMESA LABMESA LABMESA LAB
2. Method
3. Then we map the extracted SIFT features and
matching relationship into the 3D point
clouds data and obtain features and matching
relationship of multi-view 3D point clouds;
4. Finally we achieve multi-view point clouds
stitching.
![Page 19: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/19.jpg)
MESA LABMESA LABMESA LABMESA LAB
2.1 Generation texture image of effective point
clouds data
Why: 3D point clouds can’t avoid holes and noise, in order
to decrease the effect of registration and stitching precision
about multi-view point clouds, we use mutual mapping
method between 3D point clouds and 2D texture image to
obtain texture image of effective point clouds data.
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-19/1024
2. Method
![Page 20: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/20.jpg)
MESA LABMESA LABMESA LABMESA LAB
2.1 Generation texture image of effective point
clouds data
•How: Firstly, we projected the 3D point clouds to 2D
plane, secondly, used projection binary graph to make 8
neighborhood seed filling and area filling, so that we can
obtain projection graph of effective point clouds data.
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-20/1024
2. Method
![Page 21: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/21.jpg)
MESA LABMESA LABMESA LABMESA LAB
2. Mehtod2.1 Generation texture image of effective point clouds data
Texture image of effective point clouds data of a scene
![Page 22: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/22.jpg)
MESA LABMESA LABMESA LABMESA LAB
2.2 Extraction and matching with SIFT feature
1.Extract SIFT feature
SIFT is a local feature which is proposed by David[7], we can
extract the SIFT feature which is invariant under translation,
scale and rotation. This paper used SIFT algorithm to extract 2D
features, then used RANSAC method[9]to eliminate error
matching.
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-22/1024
2. Mehtod
![Page 23: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/23.jpg)
MESA LABMESA LABMESA LABMESA LAB
• 2. 3 3D feature point extractionPixel point of effective texture image,
which is obtained by point clouds
texture mapping has one-to-one
correspondence relationship
to 3D point clouds[10], as shown in
Fig.
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-23/1024
2. Mehtod
correspondence relationship
![Page 24: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/24.jpg)
MESA LABMESA LABMESA LABMESA LAB
the method is as follows:
(1) We extract SIFT feature points in 2D texture image,
then calculated the coordinate of the point.
(2) Because 2D feature points and 3D point clouds
have one-to-one correspondence relationship, we can
calculate coordinate of corresponding feature points of 3D
point clouds.
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-24/1024
2. Method
![Page 25: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/25.jpg)
MESA LABMESA LABMESA LABMESA LAB
• 2.4 3D point clouds stitching
The Multi-view 3D point clouds stitching is to make
coordinate transformation of point clouds in different
coordinate systems, the main problem is to estimate the
coordinate transformation R (rotation matrix) and
T(translation matrix).
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-25/1024
2. Mehtod
![Page 26: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/26.jpg)
MESA LABMESA LABMESA LABMESA LAB
• According to the matching point pairs which are obtained through the above step, we can estimate coordinate transformation relationship, that is to
• estimate parameters R and T, which make the objective function get minimum:
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-26/1024
Where pi and qi are matching points pairs of 3D point clouds in two consecutive viewpoint.
2. Method
![Page 27: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/27.jpg)
MESA LABMESA LABMESA LABMESA LAB3. Experiments
(a) SIFT feature effective texture image 1(b) effective texture image 2
(c) SIFT feature of 3D cloud points 1
(d) SIFT feature of 3D cloud points 2
![Page 28: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/28.jpg)
MESA LABMESA LABMESA LABMESA LAB
3. Experiments
Our method
Ref [8] Ref [1]
Experimental results
![Page 29: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/29.jpg)
MESA LABMESA LABMESA LABMESA LAB
3. Experiments
Performances evaluation criteria
Accuracy: registration rate and
stitching error rate
Efficiency: time consume
![Page 30: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/30.jpg)
MESA LABMESA LABMESA LABMESA LAB
3. Experiments
![Page 31: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/31.jpg)
MESA LABMESA LABMESA LABMESA LAB
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-31/1024
![Page 32: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/32.jpg)
MESA LABMESA LABMESA LABMESA LAB
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-32/1024
![Page 33: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/33.jpg)
MESA LABMESA LABMESA LABMESA LAB
4. Conclusion
1. Use SIFT feature of effective texture image to
achieve registration and stitching of dense multi-view
point clouds, we obtain texture image of effective
point clouds through mutual mapping between 3D
point clouds and 2D texture image, this algorithm
eliminate interference of holes and invalid point
clouds;
![Page 34: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/34.jpg)
MESA LABMESA LABMESA LABMESA LAB
4. Conclusion
2. Ensure that effective 3D feature which
corresponding to every 2D feature can be found
in the 3D point clouds, it can eliminate the
unnecessary error matching, so matching
efficiency and matching precision have been
improved;
![Page 35: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/35.jpg)
MESA LABMESA LABMESA LABMESA LAB
3. Our algorithm use the correct matching
point pairs to stitch, so it can avoid stepwise
iterative of ICP algorithm, and decrease
computational complexity of matching, it can
also reduce stitching error which is brought
by error matching.
![Page 36: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/36.jpg)
MESA LABMESA LABMESA LABMESA LAB
05/05/2014 AFC Workshop Series @ MESALAB @ UCMerced
Slide-36/1024 MESA LAB
Problems:• how to extract feature which is robust to
illumination and geometry transformation.
• how to eliminate registration error
• how to perform image merging to decrease the
difference between two adjacent stitched images
5. Discussion
![Page 37: MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56649e8f5503460f94b93506/html5/thumbnails/37.jpg)
MESA LABMESA LABMESA LABMESA LAB
Thanks