a simplex architecture for intelligent and safe unmanned ... · a simplex architecture for...
TRANSCRIPT
![Page 1: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/1.jpg)
A Simplex Architecture for Intelligent and Safe
Unmanned Aerial Vehicles
Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul Yun+, Shawn Keshmiri*
Electrical Engineering and Computer Science +
Aerospace Engineering *
University of Kansas
1
![Page 2: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/2.jpg)
Intelligent UAVs
• Many applications
– Commercial, military, police,…
– $10B in 3 years*
2
http://abarry.org/
(*) http://gizmodo.com/some-good-things-drones-can-actually-do-1475717696
Amazon prime airFollow me
Search & rescuesurveillance
![Page 3: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/3.jpg)
Intelligent UAVs
• Powerful computer hardware
– Multicore SoC, GPU
• High performance, Low cost, size,weight, and power
• Powerful software framework
– Linux, middleware, libraries
• Productivity, ease of development
– Like a PC
3
![Page 4: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/4.jpg)
Safety Challenges
4
http://rochester.nydatabases.com/map/domestic-drone-accidents
http://petapixel.com/2015/12/23/crashing-camera-drone-narrowly-misses-top-skiier/
http://www.nytimes.com/2015/01/28/us/white-house-drone.html
![Page 5: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/5.jpg)
Safety Challenges
UAVs are safety critical systems
4
http://rochester.nydatabases.com/map/domestic-drone-accidents
http://petapixel.com/2015/12/23/crashing-camera-drone-narrowly-misses-top-skiier/
http://www.nytimes.com/2015/01/28/us/white-house-drone.html
![Page 6: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/6.jpg)
Sources of Failures
• Sensors
• Airframe
• Actuators
• Onboard computing platform– Software
– Hardware
5
![Page 7: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/7.jpg)
Safety Challenges: Software
• Increasing complexity– E.g., Linux: > 15M SLOC
• Concurrency– Multithreading is hard
• Race condition. Order violation
• Timing unpredictability– Shared resource contention affects timing
• >21X slowdown on a cache partitioned multicore (*)
6
https://www.quora.com/How-many-lines-of-code-are-in-the-Linux-kernel
(*) Prathap Kumar Valsan, Heechul Yun, Farzad Farshchi. Taming Non-blocking Caches to Improve Isolation in Multicore Real-Time Systems. IEEE Intl. Conference on Real-Time and Embedded Technology and Applications Symposium (RTAS), IEEE, 2016.. Best Paper Award
![Page 8: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/8.jpg)
Safety Challenges: Software
• Increasing complexity– E.g., Linux: > 15M SLOC
• Concurrency– Multithreading is hard
• Race condition. Order violation
• Timing unpredictability– Shared resource contention affects timing
• >21X slowdown on a cache partitioned multicore (*)
Software bugs are hard to weed out
6
https://www.quora.com/How-many-lines-of-code-are-in-the-Linux-kernel
(*) Prathap Kumar Valsan, Heechul Yun, Farzad Farshchi. Taming Non-blocking Caches to Improve Isolation in Multicore Real-Time Systems. IEEE Intl. Conference on Real-Time and Embedded Technology and Applications Symposium (RTAS), IEEE, 2016.. Best Paper Award
![Page 9: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/9.jpg)
Safety Challenges: Hardware
• Hardware bugs
– Pentium floating point bug (FDIV bug)
– Intel CPU bugs in 2015: http://danluu.com/cpu-bugs/• “Certain Combinations of AVX Instructions May Cause Unpredictable System Behavior”
• “Processor May Experience a Spurious LLC-Related Machine Check During Periods of High Activity”
• …
• Transient hardware faults (soft errors)
– Single event upset (SEU) in SRAM, logic• Due to alpha particle, cosmic radiation
– Manifested as software failures• Crashes, wrong output: silent data corruption
– Bigger problem in advanced CPU• Increased density, freq higher soft error
7
http://www.cotsjournalonline.com/articles/view/102279
![Page 10: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/10.jpg)
Safety Challenges: Hardware
• SRAM SER vs. technology scaling– Per-bit SER decreases– Per-chip SER increases (due to higher density)
8
Ibe et al., “Scaling Effects on Neutron-Induced Soft Error in SRAMs Down to 22nm Process” (Hitachi)
![Page 11: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/11.jpg)
Safety Challenges: Hardware
• SRAM SER vs. technology scaling– Per-bit SER decreases– Per-chip SER increases (due to higher density)
Complex hardware is buggy and less reliable
8
Ibe et al., “Scaling Effects on Neutron-Induced Soft Error in SRAMs Down to 22nm Process” (Hitachi)
![Page 12: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/12.jpg)
How to Improve Safety of a System?
• Correct by design
– Formal method based software development
• Difficult for a complex system
– Radiation hardened processors
• Expensive and low performance
• Deal with failures
– Run-time monitoring and redundancy
9
![Page 13: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/13.jpg)
Outline
• Motivation
• UAV Simplex Architecture
• Prototype and Case Study
10
![Page 14: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/14.jpg)
Simplex Architecture (*)
• Protect an untrusted complex controller with a trusted backup controller
• General architectural principal
11(*) L. Sha, Using Simplicity to Control Complexity, IEEE Software, 2001
Safety Controller
PerformanceController
UAVPlant
Decision Logic Plant
![Page 15: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/15.jpg)
UAV Simplex Architecture
• Idea: use two hardware/software platforms with distinct performance and reliability characteristics to realize Simplex
12
High Performance (HP) Platform
High Assurance (HA) Platform
Safety controller
Performance controller
UAVPlant
Decision Logic
GPS,IMU
Radar, Camera
HA Platform(Arduino)
HP Platform:(Tegra TK1)
Rich OS (Linux), Middleware (ROS)
![Page 16: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/16.jpg)
Two Platforms
• High Assurance (HA) Platform– Simple hardware and software for verification and reliability– Hardware: low frequency and density to reduce SEUs– Software: certifiable, simple, low SLOC
• High Performance (HP) Platform– Complex hardware and software for performance– Hardware: performance oriented multicore, multi-gigahz, gpu– Software: productivity oriented software framework, millions SLOC
13
![Page 17: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/17.jpg)
Outline
• Motivation
• UAV Simplex Architecture
• Prototypes and Case Study
14
![Page 18: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/18.jpg)
Prototype Avionics
• AFS: our custom built avionics
– Arduino based custom DAQ
• Basic sensors: IMU, GPS
– Nvidia Tegra TK1
• 4 x ARM cores + 192 GPU cores
• Advanced sensors: camera, radar
• UAVs with the AFS
– Applied to four UAVs in Dr. Keshmiri’slab in KU Aerospace Engineering
– Fixed wing (DG 808, G1XD, G1XB) and a Quadcopter
15
![Page 19: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/19.jpg)
UAVs with AFS
16
DG 808G1XB
Quadcopter
G1XB
G1XD
![Page 20: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/20.jpg)
Performance Controller
• Hardware– Nvidia Tegra TK1, 4 x ARM Cortex-A15 @ 2.3GHz, 192 core GPU– 28nm process, > a billion transistors complex, high potential SEUs
• Performance controller – Intelligent adaptive non-linear control using advanced sensor packages (goal)– Use Linux (Ubuntu), Robot Operating System (ROS) difficult to verify
17
ROS nodesRadar Vision
Performance controller
![Page 21: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/21.jpg)
Safety Controller
• Hardware– Arduino Due, a single ARM Cortex-M3 @ 80MHz– Low density, low operating freq. less susceptible for SEUs
• Safety controller– Matlab Simulink coder + Arduino sketch, no OS small and
easier to verify
18
Safety controller (Simulink model)
![Page 22: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/22.jpg)
Decision Logic
• Fault models– HA (safety controller, decision logic) is trusted– HP is not trusted
• Decision logic– Detect crash, connect failure, timing violation, invalid outputs
(e.g., NaN)– Recovery: reboot the HP platform– Limitation: Currently don’t know “unsafe” states
19
Detectable faults
![Page 23: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/23.jpg)
Execution Flow
20
![Page 24: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/24.jpg)
Case Study: Fault (Crash) Injection
• Experiment
– Kill the performance controller in the middle flight
• Hardware-in-the-loop (HIL) setup
21
![Page 25: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/25.jpg)
Case Study: Fault (Crash) Injection
22
![Page 26: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/26.jpg)
Case Study: Fault (Crash) Injection
• Monitored from the ground station software
23
![Page 27: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/27.jpg)
Conclusion and Future Work
• Safety challenges of intelligent UAVs– Software: increasing complexity, concurrency and
timing non-determinism
– Hardware: increasing reliability issues. E.g., transient hardware faults (SEUs)
• UAV Simplex architecture– Two platform based realization of Simplex
• High assurance (HA) platform: simple, verifiable
• High performance (HP) platform: performant, unverifiable
24
![Page 28: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/28.jpg)
Conclusion and Future Work
• Prototype development and case study– Nvidia Tegra TK1 + Arduino based
– Can survive from performance controller crash
• Ongoing and Future work– Radar and vision based sense & avoid
– Define and detect unsafe state (not just crash)
– Detect and recover intrusion (security)
– Handling of sensor faults
25
![Page 29: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/29.jpg)
Thank You
Disclaimer: This work is supported by the National Aeronautics and
Space Administration's (NASA's) Leading Edge AeronauticsResearch for NASA (LEARN) fund under grant number
NNX15AN94A and Paul G. Allen Family Foundation(PGAFF) grant number KUAE#40956.
More details can be found in the following publication.Prasanth Vivekanandan, Gonzalo Garcia, Heechul Yun, Shawn Keshmiri. “A Simplex Architecture for Intelligent and Safe
Unmanned Aerial Vehicles.” IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), IEEE, 2016
26
![Page 30: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/30.jpg)
UAV Guidance System
27
G
1W
3Wx
1Wx
Gx
l h
C
2W
3WGxGu
x
Gu
Gtrimu Gtrimx
Gx
Gtrimx
w
hC
w1z
2z
3z
G
( ) ( ) G G G G G Gx f x g x u
( ) ( ) G G G G G Gx f x g x u
Nonlinear Aircraft and Guidance Model G:
Shifted to the origin Model:
1 1 1 11
1 1 1 1
2 2 2
3 3 3 13
3 3 3 3
( ( ))
( ( ))
( )
( )
W W W W G
W W W W G
W W
W W W W G
W W W W G
G
G
G
G
G
x A x B w h xW
z C x D w h x
W z D u
x A x B h xW
z C x D h x
Weighting Matrices
![Page 31: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/31.jpg)
Robustness approach improves: Mixed Sensitivity Concept
Low frequency: reference tracking and disturbance rejection
High frequency: noise attenuation andUncertainty neutralization
10-2
10-1
100
101
-25
-20
-15
-10
-5
0
5
10
15
20
Singular Values
Frequency (Hz)
Sin
gu
lar
Va
lue
s (
dB
)
W1
W3
ux
Optimization
Algorithm
NMPC
1W
2W2W
x
1Wx
3W3W
x
System Modelku
ke
ky
[ ]cmd ky
[ ]ku ˆ[ ]kx
1
kz2
kz3
kz
kz
Robust and Adaptive Control
• Advanced adaptive non-linear model predictive controller (NMPC)
![Page 32: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/32.jpg)
Common Issues in UAVs
• Divergence or low quality state estimations by the EKF due to external factors such as vibration with nonzero mean, low GPS update rate, noisy sensors (e.g. MEMS based rate gyro or accelerometer).
• Divided by zero and/or divergence of EKF when UAS is on the tarmac with no body velocity.
• Oscillatory and overestimated airflow angles due to accelerometer/rate gyro noise
• Nonlinear aerodynamic forces and moments when aircraft is in loss of control conditions (e.g. impairment of servos) or when it encounters environmental hazards (e.g. wind, gust, icing, etc) where most controllers are designed under linear time invariant assumption. Such situations degrade the performance of UAS controllers dramatically.
• EMI issues
29
![Page 33: A Simplex Architecture for Intelligent and Safe Unmanned ... · A Simplex Architecture for Intelligent and Safe Unmanned Aerial Vehicles Prasanth Vivekanandan+, Gonzalo Garcia*, Heechul](https://reader034.vdocuments.mx/reader034/viewer/2022042218/5ec4671a4e7aca03a87f2d68/html5/thumbnails/33.jpg)
Example Issues
30
1740 1745 1750 1755 1760 1765 1770-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Time [sec]
X-a
xis
Accele
ration [
g]
Flight Test Data
ANN without Adaptation
ANN with Adaptation
1740 1745 1750 1755 1760 1765 1770-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Time [sec]
Y-a
xis
Accele
ration [
g]
Flight Test Data
ANN without Adaptation
ANN with Adaptation
1740 1745 1750 1755 1760 1765 1770-3.5
-3
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
Flight Test Data
ANN without Adaptation
ANN with Adaptation
Aircraft’s Nonlinear Aerodynamic
Divergence of EKF due to high noise in the accelerometer