2015 microsoft indoor localization competition dimitrios lymberopoulos (microsoft research) jie liu...
TRANSCRIPT
2015 Microsoft Indoor Localization Competition
Dimitrios Lymberopoulos (Microsoft Research)Jie Liu (Microsoft Research)Xue Yang (Intel)Niki Trigoni (University of Oxford)Anthony Rowe (CMU)Ayman Naguib (Qualcomm)
Background
• 48 systems from 44 teams submitted abstracts• 28 systems showed up• 23 systems actually evaluated• Academia, Industry
• Two categories• Infrastructure-free (7 teams)
• No hardware deployment• Conference venue WiFi and phone sensors
• Infrastructure-based( 16 teams)• Custom hardware deployment
Evaluation Process
• 2-day event• Day 1: Monday
• Teams were given 7 hours to setup and calibrate their systems
• Day 2: Tuesday• Each team was asked to provide the
coordinates of 20 test points • Test points’ coordinates were manually
measured using laser range finders
• Evaluation Metric• Average localization error across the 20
test points
2000m2 Evaluation Area
Origin Point(0,0)
Y Ax
is
X Axis
Fun Times…
Towards a standardized evaluation of indoor location technologies
Multi-institutional effort NIST, TU Berlin, and iMinds at University of Ghent
Goal: create a large repository of labeled indoor location data traces 5 buildings on the NIST campus in Gaithersburg, MD 1800 test points More than 30000m2
Data Collection September – December 2015 Available through a web service
Ideas/Feedback/Suggestions Nader Moayeri ([email protected])
Official Results
Sanchez e
t al.
Lazik
et al.
Time Domain
Campbell et a
l.
Quantitec G
mbH Intra
nav
Simon et a
l.
Klipp et a
l.
Chen et al.
Zengen et al.
SPIRIT Navigation
Elias e
t al.
Symington et a
l.
Zou et al.
Pirkl e
t al.
Wu et a
l.
Bata et al.
Qiu et al.
Navix
Nikodem et al.
Deora et al.
Kuo et al.
Mirs
hekari et a
l.Tata
0
5
10
15
20
25
0.2 0.31 0.39 0.6 0.73 0.739 0.78 1.37 1.63 1.9 2.63.5 3.65 3.93 4.22 4.31 4.67 4.92 5.6
7.398.24
11.24
22.99
Team
Loca
lizati
on E
rror
(m)
infrastructure-free industry
1st
2nd
3rd
Infrastructure Free Infrastructure Based
$1000
$600
$400
STeAM: Sensor Tracking and MappingSanchez et al.(Institute for Transuranium Elements) 0.2m
error
ALPS: An Ultrasonic Localization SystemLazik et al.(CMU) 0.31m
error
SPIRIT NavigationBerkovich et al.(SPIRIT Navigation) 1.9m
error
Precise Indoor LocationD. Elias et al. (Fraunhofer Portugal Research Center)
2.6m error
Time DomainDewberry et al.(Time Domain) 0.39m
error
Exploring State of the Art Commercial LocalizationCampbell et al.(Univ. of Michigan)
0.6m error
Point-Level View
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
1
2
3
4
5
6
Sanchez et al. SPIRIT Navigation Elias et al.
Evaluation Point
Loca
lizati
on E
rror
(m)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200123456789
10
Lazik et al. Time Domain Campbell et al.
Evaluation Point
Loca
lizati
on E
rror
(m)