iarpa multi-view stereo (mvs) 3d mapping challenge multi-vi… · intelligence advanced research...

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

IARPA Multi-View Stereo (MVS) 3D Mapping Challenge

1 July 2016

HakJae Kim, IARPA Myron Brown, JHU/APL


Why Are We Doing This?

• Multi-View Stereo (MVS) • Reconstruction of a 3D object or scene model from a collection of images

taken from multiple camera viewpoints • MVS for 3D Mapping

• A broad range of MVS research can apply • No prior public benchmark exists for 3D mapping with satellite imagery

Image Acquisition Camera Pose 3D Reconstruction

Illustrations from http://carlos-hernandez.org


Recent MVS Evaluation Benchmarks

• Seitz, Curless, Diebel, Scharstein, and Szeliski, CVPR 2006 • Laser scans of small indoor 3D objects • http://vision.middlebury.edu/mview/

• Stretcha, Hansen, Van Gool, Fua, and Thoennessen, CVPR 2008 • Lidar scans of large outdoor 3D objects • http://cvlabwww.epfl.ch/data/multiview/

• Geiger, Lenz, and Urtasun, CVPR 2012 • Velodyne lidar scans for autonomous driving • http://www.cvlibs.net/datasets/kitti/

• Jensen, Dahl, Vogiatzis, Tola, and Aanaes, CVPR 2014 • Structured light scans of small indoor 3D objects • http://roboimagedata.imm.dtu.dk/

http://vision.middlebury.edu/mview/

http://cvlabwww.epfl.ch/data/multiview/

http://www.cvlibs.net/datasets/kitti/

http://roboimagedata.imm.dtu.dk/


MVS for Mapping with Satellite Imagery

31 images of EPFL campus from helicopter Tola, Strecha, and Fua (2011)

131 Flickr images of Notre Dame Fuhrmann, Langguth, and Goesele (2014)

73 satellite images of Bangalore Ozcanli, Dong, Mundy, Webb, Hammoud, and Tom (2015)

• MVS research with satellite imagery has been limited to few with access to data • Sources and commercial uses of satellite imagery are rapidly expanding • 50 images for this challenge are provided as a courtesy of Digital Globe

Proximal / Local Remote / Global


IARPA MVS 3D Mapping Challenge

https://www.iarpa.gov/challenges/3dchallenge.html

Explorer contest launches 11 July


IARPA’s Objectives

• Promote and benchmark research in multi-view stereo algorithms applied to commercial satellite imagery

• Encourage participation from a broad range of disciplines in both academia and industry, with or without satellite imagery experience

• Cultivate and sustain an ongoing collaborative community dedicated to this technology and research


Challenge Contests and Events • Explorer Contest (3 weeks)

• Small financial prizes for top 5 ranked solutions • Small data sets with limited scene diversity • Chance for both solvers and evaluation team to learn

• Master Contest (3 weeks) • Substantial financial prizes for top 5 ranked solutions • Larger data sets with significant scene diversity

• White Paper Contest (3 weeks) • Encourage good documentation of methods and results • Reward interesting results not fully characterized by the metrics • Evaluated by government stakeholder community

• Open Source Incentive • Encourage open source solutions to maximize community benefit

• Workshops • CVPR Workshop, Las Vegas, Nevada, 1 July • DC Area Workshop, November


Challenge Execution Team • Booz Allen Hamilton

• Developed challenge framework and strategy • Managing challenge execution and communications • Conducting social media and marketing campaign and community outreach

• TopCoder

• Provided best practice guidance for challenge formulation • Providing and managing challenge platform, which serves over 1M members • Conducting outreach to TopCoder data scientist community

• JHU/APL

• Developed challenge data sets, ground truth, metric evaluation software, and example code to help solvers get started quickly

• Developed example MVS algorithms and providing example point cloud results to demonstrate compliant submissions

• Organizing workshops at CVPR 1 July and in the DC area in November • Providing independent test and evaluation of solutions


Challenge Data Package for Solvers

• Inputs • 50+ Digital Globe WorldView images in NITF file format

(approx. 40GB) • KML geographic bounding boxes for challenge areas

• Sample Output • Example of compliant LAZ compressed point cloud file

• Sample Source Code for Linux and Windows • Read KML and LAZ and test LAZ file compliance with

requirements • Read KML and NITF and project KML polygon XYZ corner

coordinates into image coordinates using GDAL • Read KML and generate LAZ file with correct geospatial tags

that satisfy submission requirements; this LAZ file is for example only, and all output points have the same Z value


Public Resources to Support Solvers

• LizardTech GeoViewer • Free image viewer that supports NITF file format • https://www.lizardtech.com/products/geoviewer/

• Applied Imagery QT Reader • Free tool for viewing 3D point clouds and other geospatial data with

support for LAS, LAZ, KML, and NITF files • http://appliedimagery.com/download/

• LibLAS and LASzip • Open source for reading and writing LAS files and LAZ files with LASzip • http://www.liblas.org/ • http://rapidlasso.com/laszip/

• Geospatial Data Abstraction Library (GDAL) • Open source for reading and writing images with support for NITF file

format and RPC00B image metadata used for commercial imagery • http://www.gdal.org/

https://www.lizardtech.com/products/geoviewer/

http://appliedimagery.com/download/

http://www.liblas.org/

http://rapidlasso.com/laszip

http://www.gdal.org/


Judging Criteria for Explorer Contest*

• Registration • Point cloud will be registered to ground truth using X, Y, and Z

translational search to optimize accuracy metric • Accuracy Metric

• Median Z error measured in meters compared to ground truth • Accuracy must be less than 1 meter to be considered for ranking

• Completeness Metric • Fraction of points with Z error less than 1 meter • Completeness must be greater than 0.5 to be considered for ranking

• Rank on Leaderboard • Normalized sum of accuracy and completeness metric scores • Ties resolved in favor of best accuracy score

* Judging criteria may be adjusted for the Master contest based on lessons learned from the

Explorer contest and feedback provided by the solvers.


Example Benchmark Evaluation 1m Airborne Lidar (2005)

Courtesy of OpenTopography

* Area shown and data used is for example only. Locations for challenge data sets will be announced when each contest begins. Ground truth will be independently acquired.

Example MVS Solution Developed using Open Sources

• Foliage and cars are automatically labeled in lidar and not compared • Vertical edges the width of lidar footprint are also automatically labeled • Other scene change is manually labeled (not shown above) • Example MVS algorithm Accuracy = 0.67m and Completeness = 0.6

1m Airborne Lidar After Labeling Uncertain Points


Longer Term Objectives

• Promote active research in MVS for global 3D mapping • By making data and ground truth more broadly available after

the challenge is complete • By funding additional challenges • Suggestions and collaboration are welcome

• Leverage results of this challenge to support upcoming

IARPA CORE3D program • CORE3D seeks to develop solutions for global 3D scene

modeling from the broad range of current and planned active and passive remote sensing capabilities

• https://www.iarpa.gov/index.php/research-programs/core3d • BAA released Thursday 30 June

https://www.iarpa.gov/index.php/research-programs/core3d

iarpa multi-view stereo (mvs) 3d mapping challenge multi-vi… · intelligence advanced research...

Documents