dddams-based border surveillance and crowd control via uvs ... · 3-levels sensors measurement data...
TRANSCRIPT
1
DDDAMS-based Border Surveillanceand Crowd Control via UVs and
AerostatsSponsor:
Air Force Office of Scientific Research
FA9550-17-1-0075
Program Manager: Dr. Erik Blasch
PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2
Students: S. Lee1, Y. Yuan1, H. Na1, H. Yang1
1Systems and Industrial Engineering, University of Arizona2Computer Science, George Mason University
PI Contact: [email protected]; 1-520-626-9530
AFOSR DDDAS PI Meeting Sep. 7th, 2017
2
Agenda
• Problem Definition and Modeling Framework
• Model Description and Implementation via
Sensors
• Systems Software
3
Border Surveillance and 3-Level Framework
High altitude
level
(HAL)
Low altitude
level (LAL)
Surface level (SL)
4
Major Challenges in Border Surveillance
• 3-D Surveillance System for aerial and ground targets
• Effective detection, recognition and identification of
targets
• Heterogeneous data from complex targets by 3 levels
of sensors
• Multi-level information aggregation
• Active or pro-active surveillance strategies
• Realistic scenarios and model validation based on
data collection from our research partners (AFRL,
Raytheon, University Partners …)
5
High altitude level (HAL)
Low altitude level (LAL)
Surface level (SL)
3-Level Measurement System in Border Surveillance
3-Levels Sensors Measurement Data
SAR
Image
EO/IR
Image
Lidar
Image
Thermal
Images
Magnetic
Data
Spectral
Image
6
LiDAR Data Processing
A Velodyne HDL32 LiDAR integrated with
a powered wheelchair for indoor scanningA Velodyne HDL64 LiDAR integrated
with a golf cart for outdoor scanning
LiDAR+Photos = Better Localization [Arsalan, Kosecka, Lien, ICRA 2015]
Registered LiDAR data
captured on George
Mason Campus
7
DDDAMS Framework
Group
Prediction
Group
Prediction
8
Agenda
• Problem Definition and Modeling Framework
• Model Description and Implementation via
Sensors
• Systems Software
9
DDDAMS Framework : Target DRI(Detection, Recognition, Identification)
Group
Prediction
Group
Prediction
10
Target DRI (1) : Detection
• Goal: Discover the presence of a person, object, or phenomenon
* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense.
• Sensing technologies: Cameras (UAV, UGV), Seismic Sensors
• Algorithms: Motion detection, Control Charts
Group
Prediction
11* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense.
• Algorithms: Wavelet decomposition,
Classification
Target DRI (2) : Recognition
• Goal: Determination of the nature of a detected person, object or
phenomenon, and its class or type *
Group
Prediction
12* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense.
Target DRI (3) : Identification
• Goal: Discrimination between recognizable objects as being friendly
or enemy *
• Algorithms: Information-aggregation method,
Extended BDI (Belief-Desire-Intention) framework
Group
Prediction
13
Crowd Detection Module UAV
Moving
Target
Detection
via
Sliding
Window
𝕊1. Feature
extraction:
Good features to
track
𝕊2. Feature
tracking:
Optical flow
.
𝕊3. Image
registration:
RANSAC,
Homography
𝕊4. Background
elimination:
Absolute
differences
𝕊5. Targets
segmentation:
Motion history,
Dilation-erosion
𝑰𝒕𝑰𝒕+∆𝒕𝑰𝒕+𝟐∆𝒕
𝑲𝒕
𝑰𝒕+∆𝒕(𝑻)
𝑰𝒕+𝟐∆𝒕(𝑻)
𝑲𝒕+∆𝒕
𝑲𝒕+𝟐∆𝒕
𝑰𝒕+𝟐∆𝒕(𝑻)
− 𝑰𝒕+∆𝒕(𝑻)
Goal:
(a) (b) (c)
Minaeian, S., Liu, J., & Son, Y. J. (2016). Vision-Based Target Detection and Localization via a Team of Cooperative UAV and UGVs. Systems, Man,
and Cybernetics: Systems, IEEE Transactions on, 46(7), 1005-1016.
Minaeian, S., Liu, J., & Son, Y.-J. (2017). Effective and Efficient Detection of Moving Targets from a UAV’s Camera. Submitted to Intelligent
Transportation Systems, IEEE Transaction on, (Under Review).
14
Individual Detection/Recognition Module UGV
: OpenCV classifier
: 3x3 derivative mask
[-1, 0 , 1]
: Weighted voting over cells
6x6 pixel cells
: Grouping cells into blocks
3x3 cell blocks
: L2-norm
Gradient
computation
Orientation
binning
HOG over
des. blocks
Block
normalization
Classify the
target
Blocks
Cells
• Goal: HOG (Histogram Oriented Gradient) based Target D/R
Minaeian, S., Liu, J., & Son, Y.-J. (2015, January). Crowd Detection and Localization Using a Team of Cooperative UAV/UGVs. In IISE Annual
Conference. Proceedings: 595-604. Institute of Industrial Engineers-Publisher.
15
Detection Ground Seismic Sensor
• Goal: Target Detection based on Control Charts
• Detect presence of an object
Detected
Geophone Amplifier Gateway Software
System
Components:
(sec) (sec)
Normal Target Appearance(mV) (mV)
16
𝑋 𝑡 = 𝚽𝑟𝑢𝑛 𝑡 𝑪𝑟𝑢𝑛 +𝚽𝑤𝑎𝑙𝑘 𝑡 𝑪𝑤𝑎𝑙𝑘
Training Data Knowledge
New Observation
𝑪 =𝑪𝑟𝑢𝑛𝑪𝑤𝑎𝑙𝑘
Wavelet
Coefficients
Model: Wavelet decomposition Regression
Fe
atu
re E
xtr
ac
tio
n M
eth
od
s:
Fis
he
r’s
Dis
cri
min
an
t A
na
lys
is
SVM Classifier
• Goal: Target’s Characteristics Recognition (e.g., walking/running)
• Challenge: Raw data are noisy and mathematically inseparable
Recognition Ground Seismic Sensor
17
Time
Location
Route Choice En-Route
Planning
Fastest Way
Rugged Road
Selection at
Departure
Hiding/Avoiding
Sudden Direction
Change
Decision 1 Decision 2 Decision 3
Behavioral Models of Drug Traffickers
• Goal: Target identification via behavior models of drug traffickers and ground
patrol agents under varying environmental conditions
Identification Behavior Models
Hiding Direction
Change
18
Bratman, 1987
Rao and Georgeff, 1998
Zhao and Son, 2008
Extended Belief-Desire-Intention (EBDI)
Framework
Lee, S., Y.-J. Son, and J. Jin (2010), Integrated human decision making and planning model under extended belief-desire-intention framework,
ACM Transactions on Modeling and Computer Simulation, 20(4), 23(1)~23(24).
19
Identification Fusion of Sensor Data
• Goal: Target’s Identification (e.g., Friend/Foe)
• Challenge: Fusion of different sensor data (UAV vision and seismic)
20
DDDAS in border patrol’s computing unit
• Task: Augment patroller agent’s vision system with real-time imagery data collected by UAVs and the patrol agent
• Inputs: • (1) Low-resolution but less-occluded
image/3D data from UAV • (2) high-resolution but much-occluded
captured by the border patrol
• Output: Fused and registered data captured by the UAV to images captured by the border patrol agents
• Method: The key in enhancing the border patrol’s vision is in finding the correspondences between the features extracted from the images captured by the UAV and the patrol agents and identifying occluded objects in patrol agents’ view.
Motion control
Wide rangevisibility occluded
subjects
Aerostat
Border patrol
UAV
Superhuman Vision via Information Fusion
21
Agenda
• Problem Definition and Modeling Framework
• Model Description and Implementation via
Sensors
• Systems Software
22
Agent-based Hardware-in-the-loop Simulation
Agent-based Simulation
Repast Simphony
with 3D GIS
Wi-Fi / XBee PRO 900HP; APM one
Assembled UAV
(APM:Copter / Arducopter)
Assembled UGV
(APM:Rover / Ardurover)
Sensory Data
(e.g. GPS)
Control Commands
(MAVLink Messages)
Hardware Interface:
MAVproxy
Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV
surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA.
23
Physics Based Simulation for Data Generation
Real Pictures from UAV Generated Pictures from Simulation
24
Physics Based Simulation for DataGeneration
• Goal: Visionary data generation using various simulation objects.
Ground Patrol
UAV (fly high)UAV (fly low)
UGV
25
Detection module (Simulated) UAV
• Goals: Applying detection algorithm using data from simulated UAV.
26
Detection module (Simulated) UGV
• Goal: Applying HOG detection algorithm to simulated data from UGV.
27
DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats
Son and Liu, Arizona; Lien, George Mason
• Summary of Effort
– AF relevance to autonomous systems;
collaborative/cooperative control; sensor-based processing;
multi-scale simulation technologies; cognitive modeling
• Key Focus of Scientific Research
– Active or pro-active border surveillance strategies
– Multi-level information aggregation involving heterogeneous
data from 3 levels of sensors
– Handling latency in detection, recognition, & identification of
targets
• Other performers on project
– Y. Son (UA), J. Liu (UA), J. Lien (George Mason)
– Students: S. Lee, Y. Yuan, H. Na, H. Yang
28
• Accomplishments
– Developed/refined a DDDAMS-framework for a 3 level border
surveillance system
– Developed preliminary models and algorithms for target DRI,
group prediction, and mission control
– Hardware in the loop simulation (agent; physics simulation)
• Awards
– (PhD graduate at U of Arizona) Dr. Sara Minaeian, August 2017
(Joined Siemens)
– (Best paper award in “Service and Work Systems” track) S. Lee
(PhD student) and Y. Son, “Extending Decision Field Theory to
a Multi-agent Decision-making with Forgetting,” Proceedings
of 2016 IISE Annual Meeting, Anaheim
DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats
Son and Liu, Arizona; Lien, George Mason
29
• Reporting
– Minaeian, S., Liu, J., & Son, Y.-J. (2016). Vision-Based Target
Detection and Localization via a Team of Cooperative UAV and UGVs.
Systems, Man, and Cybernetics: Systems, IEEE Transactions on,
46(7), 1005-1016
– Minaeian, S., Liu, J., & Son, Y.-J. Effective and Efficient Detection of
Moving Targets from a UAV’s Camera. Submitted to Intelligent
Transportation Systems, IEEE Transaction on, Special Issue on
Robust & Efficient Vision Techniques for Intelligent Vehicles, (Under
Review)
– Minaeian, S., Liu, J., & Son, Y.-J. Effective and Efficient Multi-target
Data Association via Dynamically Adjusted Affinity Scores, (to be
submitted to a journal in September, 2017)
– Minaeian, S., Liu, J., & Son, Y.-J. (2016). Analysis of Network
Communications between Cooperative Unmanned Vehicles for
Autonomous Surveillance. In IIE Annual Conference, 2016
Proceedings. Institute of Industrial Engineers-Publisher.
DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats
Son and Liu, Arizona; Lien, George Mason
30
• Coordination/Synergy
– Exploring collaboration and access to aerostat data with
Raytheon (no access yet)
– Discussed collaboration opportunities with Tathagata
Mukherjee (Intelligent Robotics, Inc) at the 2017 PI meeting in
Dayton, and will schedule follow-up meetings together with
Eduardo Pasiliao at AFRL
• Exposure/Use by other groups
– Hardware-in-the-loop demos to visitors (STEM students,
summer students, visitors from industry and other
universities)
DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats
Son and Liu, Arizona; Lien, George Mason
31
Acknowledgements
PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2
Students: S. Lee1, Y. Yuan1, H. Na1, and H. Yang1
1Systems and Industrial Engineering, University of Arizona2Computer Science, George Mason University
PI Contacts:
[email protected]; 1-520-626-9530
[email protected]; 1-520-621-6548
[email protected]; 1-703-993-9546
Air Force Office of Scientific Research
FA9550-17-1-0075
Program Manager: Dr. Erik Blasch