deep representation and reinforcement learning for anomaly ...€¦ · rome, italy joined utc in...

Soumalya Sarkar, PhD Senior Research scientist, UTRC

Deep Representation and Reinforcement Learning for Anomaly Detection and Control

in Multi-modal Aerospace Applications

This document contains no technical data subject to the EAR or the ITAR.

May 9 @ GTC 2017

2

“UTRC is where

you bring your

toughest problems.”

Our

business units


IIoT

52% 36%

12%

Military Aerospace

& Space

Commercial

Aerospace

Commercial

& Industrial

END MARKETS

56%

44%

Original

Equipment

Manufacturing

Aftermarket

SALES BY TYPE SALES BY GEOGRAPHY

38%

27%

20%

15%

Asia

Pacific

Europe

United

States

Other

3

A

global leader

$56B 2015 UTC Sales

$3.9B invested in R&D


4

Cork, Ireland Established in 2010,

focuses on energy, security

and aerospace systems

Shanghai, China Established in 1997, focuses on

integrated buildings, fluid and

mechanical systems

Rome, Italy Joined UTC in 2012,

focuses on model-based

design and embedded

systems engineering

East Hartford, CT Founded in 1929, focuses on a

broad range of system engineering,

thermal, fluid, material, and

informational sciences

Berkeley, CA Established in 2009, focuses

on cyber physical systems

and embedded intelligence

A

global presence


5

Focused on

performance

Physical Sciences Systems

Thermal & Fluid

Sciences

– Advanced Materials

– Applied Physics

– Environmental Science

– Materials Chemistry

– Measurement Science

– Solid Mechanics

– Surface Mechanics

– Advanced Laboratory for Embedded Systems

– Control Systems

– Cyber-Physical Systems

– Decision Support & Machine Intelligence

– Electromagnetics & Networks

– Power Electronics

– Software Systems

– System Dynamics & Optimization

– Acoustics

– Aerodynamics

– Aero-Thermal Testing

– Combustion Science

– Propulsion Technology

– Thermo-Fluid Dynamics

– Thermal Management


Topics


Deep Representation Learning - DAE Big PHM (Prognostics & Health Monitoring) SHM (Structural Health Monitoring)

Deep Reinforcement Learning (DRL) for additive manufacturing

Deep Auto-Encoder (DAE)

• Parameter learning by Stochastic gradient descent (Hinton&Salakhutdinov, 2006; Bengio et al., 2007)

• Variants: De-noising (RBM), variational etc

• Static DAE instead of LSTM AE due to ease / speed of training

Multi-layer neural network based learner of non-linear representation of the data

W

Input:

Hidden representation:

Sigmoid connecting two layers:

Parameters:

Sigmoid function at reverse mapping of reconstruction layers:

Where

Cost function for back-propagation:


,𝑊′= 𝑊𝑇

Topics




Motivation

Big, multi-modal & heterogeneous data; unsupervised visualization


Data: ~100 sensors, ~200 dimensional condition data Size ~ TB Zero/ few labels Problem: Understanding / separating different missions or faults Challenges: Low-dimensional visualization, robust separation of faults (FDI) / mission, real-time application and generalizability

FDI approaches and challenges

• Lack of high-fidelity non-linear models,

• Tedious hand-crafting (domain knowledge) of fault features,

• Lack of scalability to large data,

• Insufficient robustness to noise and

• The presence of various operating modes,

• Presence of multi-modal sensors for fault disambiguation

Previous Work and Challenges

Methods of fault detection and identification

Model based 1. Residual methods 2. Parity based

3. Kalman filter based

Data-driven 1. Time, frequency, symbolic domain

features 2. SVM, k-NN, artificial neural net based

learning systems

Hybrid 1. Parity Equation Approach and wavelet based signal features 2. PCA based system models

Deep Learning


Database* for Validation

Apparatus: A set of electromechanical actuators (EMA), constructed by Moog Corporation, were used by Balaban et. al. (Balaban et al., 2009, 2015). To increase the horizon of available operating conditions, flyable electromechanical actuator (FLEA) testbed was also constructed.

13 multi-modal sensors @100Hz: Actuator Z Position, Measured Load, Motor Current X-Y-Z, Motor Voltage X-Y, Motor Temperature X-Y-Z, Nut X-Y Temperature, Ambient Temperature.

Baseline an 2 fault classes:

1. A jam fault injected via a mechanism mounted on the return channel of the ball screw that can stop circulation of the bearing balls through the circuit.

2. A spall fault injected by introducing cuts of various geometries via a precise electrostatic discharge process. The initial size and subsequent growth of these cuts were confirmed by using an optical inspection and measurement system.

*open database available at NASA Dashlink, collected by Balaban et. al. (Balaban et al., 2009, 2015)

Fault Detection and Identification


DAE architecture

650 (50x13) -> 256 -> 196 -> 136 -> 76 -> 14 -> 76 -> 136 -> 196 -> 256 -> 650

Window size = 0.5 seconds, shifted by each time point

11-layer DAE


DAE Reconstruction Error

Actual signals and reconstructed signals for Motor X voltage, Motor Y temperature, and load sensors (from top to bottom) with bottleneck layer of 14 dimension

Multi-modal Reconstruction Error Fault Detection and Identification


Training and Parameter Learning

Variation of normalized RMS error at the reconstructed output layer with increasing dimension of the bottleneck

Individual sensor-wise reconstruction errors at the output layer for 3 different bottleneck layer dimensions


Tuning bottleneck Layer

Fault Diagnostics

ROC curves via varying detection threshold on testing data for different bottleneck dimensions of 11-layer DAE and few single layer AE models

Precision-Recall curves for the same conditions as data is unbalanced


ROC and Precision-Recall curves

Unsupervised Fault Disambiguation

Spider charts showing the NRMS error across different sensors during testing phase for nominal and fault scenarios

Disambiguation by Multi-dimensional Reconstruction Error


Why Deep Architecture?

Spider charts of the average (over nominal and fault scenarios) NRMS error across different sensors during testing phase for

(a) single hidden-layer AE with 512-dimensional bottleneck (b) proposed 11-layer DAE with 14-dimensional bottleneck

DAE Reconstruction Error increases fault separability with low over-fitting


Why Deep Architecture?

Clusters of two largest principal components obtained from PCA on 13-dimensional NRMS error distributions

Multi-dimensional NRMS from DAE increases inter-fault distance at low dimension

single hidden-layer AE model with 512-dimensional bottleneck layer proposed 11-layer DAE with 14-dimensional bottleneck layer


Discussions

• trained directly on raw time series from heterogeneous sensors without feature hand-crafting and extensive data preprocessing.

• A high fault detection rate ( ~97.8%) along with zero false alarm on a large set of realistic data (available on NASA DASHlink)

• Disambiguation among different types of faults with high confidence in an unsupervised way

• Proposed DAE more robust than single-hidden layer AE


Fault separation even at a low dimension in an unsupervised way

Topics




Autonomous SHM from Imaging

Background


Vision-based SHM is required to get details of the damage such as size, configuration, shape, topological networking, geometrical statistics for material characterization, damage prognostics , RUL analysis etc

• Image processing as important tool in material/structural characterization for over three decades (Krakow, 1982; Duval et al., 2014; Robertson et al., 2011; Leach, 2013).

• Texture analysis (Comer & Delp, 2000) and segmentation (Ruggiero, Ross, & Porter, 2015; Park, Huang, Ji, & Ding, 2013) are applied image processing techniques to SHM.

• Pre-processing steps like filtering and enhancement techniques (Tomasi & Manduchi, 1998; Angulo & Velasco-Forero, 2013; Buades, Coll, & Morel, 2005) used to denoise image and perform alignment and artifact correction.

• Recent breakthroughs of deep learning are mostly in image processing because it models multiple levels of abstractions (low-level features to higher-order representations, Erhan, Courville, & Bengio, 2010).

• Broad applications to medical imaging (similar to vision-based SHM from computer vision perspective), recent application to video-based combustion PHM (Sarkar et al., 2015)

Why DAE for SHM?

Not explored enough yet!


Methodological challenges: Computational Challenges:

Extensive heuristics for parameter tuning in existing tools

Limited availability of annotation causing small number of training labels (no CNN)

Robustness issue in computer vision (segmentation) techniques

Seamless incorporation of domain expert in the loop

• Lengthy and tedious process of manual annotation by domain experts on large number of samples

• Human error (expert bias) in labelling the ground truth

• Process changes significantly with new experimental setup and material

Use case: Damage characterization

Experimental setup: Variable load-induced cracks on a composite

(a) Scheme of coupon testing and (b) Representative damage pattern


Thick multi-layer composite sub-elements used in numerous rotorcraft and aircraft applications.

Usually under conditions of multi-axial loading with dominant influence of bending, generating complex patterns of internal damages

Carbon fiber polymer-matric 55 layered composite (IM7/977-3 materials with lay-up [+454 / -454 / 03]2S[03/- 454 /+ 454] representing thickness of 0.290 in) coupons were considered under conditions of five-point bending.

Video starts with a straight coupon and slowly it is bent under monotonically increasing displacement-controlled load till full fracture.

Framework for Damage Characterization

Image frame to crack-length distribution

Distribution of crack lengths

Video frames with dynamic crack and non-linear bending


Patching DAE and Guided Segmentation

Modeling nominal surface and segmenting cracks from reconstruction error map

Input layer Output layer

Multiple Hidden layers

Frame with NO crack

Learning Nominal Surface via Reconstructing DAE (patch size 3 by 7 pixels)

𝑆 𝑥, 𝑦 = 𝐼 𝑥, 𝑦 − 𝑐 1 − 𝑝 𝑥, 𝑦

Similarity measure Raw image intensity of pixel 𝑥, 𝑦

Probability (intensity of reconstruction error map) of a pixel 𝑥, 𝑦 being a crack


Patching DAE Guided region-growing segmentation

Mean intensity of current region

A = true crack zone B = detected crack zone A ∩ B = correctly detected crack area

1. 𝐷𝑖𝑐𝑒 𝑠𝑐𝑜𝑟𝑒 = 2(𝐴 ∩ 𝐵)/(𝐴 ∪ 𝐵)

A

B A ∩ B

Performance metrics

3. Average minimum distance between true crack and detected crack areas

2. Distance between histograms, d

4. Number of cracks


For characterization number of cracks and d are the most significant metrics

Tuning thresholds (on intensity of reconstruction error map) on medium level load

Refining parameter

Chosen threshold


Tuning thresholds based on medium load = 0.55

Crack detection at medium load from Tuned parameter



Actual

Medium level load

High level load

Low level load

Crack detection across various load levels Predicted

Shows better robustness (to different load level) even crack thickness of only 2-3 pixels than sophisticated contour detector, edge detectors, morphological segmentations and single step region growing segmentation.

Discussions

• High characterization accuracy and satisfactory performance over a wide range of loading conditions with limited number labeled training image data.

• Meets required robustness (to different load level) via DAE error map in comparison to other benchmarks

• This approach can applied to field inspection and borescope inspection on complex surface structures.


Less heuristics, Validation on real data, Robustness to varying condition

Topics



Deep Reinforcement Learning (DRL) for additive manufacturing (AM)

Noisy robot state Surface imaging

Nozzle Trajectory

𝑜𝑡

Global feedback controller, No need to solve online MPC

AM in aerospace domain: high precision standards & high variability of tasks

No need for state estimation

• 3D printing • Cold Spray • Arc welding • Powder deposition

Reduce reliance on expert (rather expert guided) Self-learning/adaptation to optimal behavior Closing the loop with real time perception

Cost reduction, easier commissioning, improved performance & scalability, high precision geometric and material property

for printing and repair

DRL for AM


Cold Spray

Robot

Scanner

Sensors Perception/

State Estimation

Policy /Control

Low Level

Control

Actuator

Actions/ Control

Sensing

Explore recent advances in deep learning to address • End-to-end sensinglearning/control • Incorporate expert/prior knowledge

Automate

Expert engineered on a case by case basis

What are we trying to do?


DRL for AM

Different Flavors

End-to-end perception to learning/control

End-to-end guided policy search (Levine et. al. 2015)

Deep Q Learning (Mnih et. al., 2013, Lillicrap et. al., 2016)

Learning features/dynamics to use in control/RL Deep Q Fitted Network (Lang et. al, 2012) Embed to control-iLQR (Watter et. al., 2015) Deep Dynamical Models-NMPC (Assael et. al, 2015)

Learning value function/policy in RL

Guided Policy Search Levine e.t al., 2014)


DRL for AM

How it works?

NN to represent value function Purely exploratory No need for generative model Gaming applications / simulated environments

NN to represent policy Exploit trajectory optimization to guide search Need (approx.) generative model Robotics application

Agent

Simulated/Real Environment

Action, 𝑢𝑡 ∼ 𝜋(. |𝑠𝑡) State, 𝑠𝑡+1

Policy: 𝜋: 𝑆 → 𝑃(𝑈)

Value/Q function: 𝑄(𝑠, 𝑢) = E𝜋[J(𝜏)|s1 = s, u1 = u]

𝑠𝑡+1 = 𝑓(𝑠𝑡, 𝑢𝑡)

min𝜋𝐸𝜋[𝐽 𝜏 ]

𝐽 𝜏 = 𝑐(𝑠𝑡, 𝑢𝑡)

𝑇

𝑡=1

𝜏 = {𝑠1, 𝑢1, 𝑠2, 𝑢2, … . , 𝑠𝑇 , 𝑢𝑇}

Cost, 𝑐(𝑠𝑡, 𝑢𝑡)

Key advance: Stable training

NN for function approximation

Deep Q Learning Guided Policy Search


DRL for AM

*(Levine et. al. 2015)

Guided Policy Search Problem Formulation

Approach I: One Shot “Demonstrator” Trajectory optimizer to generate optimal guiding

sample NN trained to match samples

Approach II: Incremental “Good teacher” Provides training to NN in small steps Modified cost for trajectory optimizer, so that it solves the

control problem similar to how student (fails to) solve it

min𝜃𝐸𝜋𝜃[𝐽(𝜏)]

Policy Search

Guided Policy Search = Supervised Learning

Traj

ect

ory

O

pti

miz

atio

n

Guiding samples

𝑞(𝑢𝑡|𝑠𝑡)

Simulated/Real Environment

𝜃

𝜋𝜃(𝑢𝑡|𝑠𝑡)


DRL for AM

Deep Reinforcement Learning (DRL)

Guided Policy Search: Incremental Approach

Dynamic Model

Approx.

Cost Function Approx.

Iterative LQR

Sample

Trajectory

NN

Training

𝑞𝑖(𝑢𝑡|𝑠𝑡) 𝜏𝑖𝑗

𝑖 = 1, . . , 𝑀 𝑗 = 1, . . , 𝑁

Lagrange Multiplier Update

𝜏𝑖𝐼 , 𝑖 = 1, . , 𝑀

Simulator/ Real System

Quadratic Approx.

𝜋𝜃,Σ 𝑢𝑡 𝑠𝑡

𝜇(. ; 𝜃)


DRL for AM: Cold Spray for high precision repair

Guided DRL for cold spray control

Challenges Complex physics:

Coupled geometric/structural properties

High dimensional nonlinear system

Path planning:

Laborious manual process ~multiple hours per part

Difficult to assess frustration points

Part machined to simplify path planning

Dynamics:

Objective: Compute 𝑢1, 𝑢2, … , 𝑢𝑇 to minimize

𝑠𝑡+1 = 𝑓 𝑠𝑡, 𝑢𝑡 𝑠𝑡 = (𝐷 𝑟1, 𝑡 , …𝐷 𝑟𝑁, 𝑡 , 𝑥𝑠 𝑡 , 𝛼(𝑡))

𝑢𝑡 = (𝑣𝑠, 𝜔𝑡)

𝐽(𝜏) =1

2 𝑠𝑡 1:𝑁 − 𝐷𝑟𝑒𝑓

∗𝑅 𝑠𝑡(1:𝑁) − 𝐷𝑟𝑒𝑓 + 𝑢𝑡

∗𝑄𝑢𝑡

𝑇

𝑡=1

─ distribution of particles in spray cone ─ deposit efficiency function ─ total deposit in the point r during one run ─ position of the nozzle at time t

𝜑 tan𝛼

𝑓 cot 𝛽

𝐷 𝑟, 𝑡

xs(t), hs

𝜕𝐷 𝑟, 𝑡

𝜕𝑡= 𝜑

𝑟 − 𝑥𝑠 𝑡

ℎ𝑠 − 𝐷 𝑟, 𝑡

𝑇

0

𝑓𝑟 − 𝑥𝑠 𝑡 − ℎ𝑠 − 𝐷 𝑟, 𝑡

𝜕𝜕𝑟 𝐷 𝑟, 𝑡

ℎ𝑠 − 𝐷 𝑟, 𝑡 + 𝑟 − 𝑥𝑠 𝑡𝜕𝜕𝑟 𝐷 𝑟, 𝑡

d𝑡

α

β

γ

D(r,t)

α

β

γ

xs(t) r

hs

𝐷𝑟𝑒𝑓

Goal: Automate cold spray control


Results: GPS leads to optimal behavior as MPC


500 randomly generated surfaces

Much Faster!


Co

st

Better Training!

machining

Constant Speed DRL Based Control

Generalizability to various surface topology with limited training and benchmarking against constant speed nozzle control (state-of-the-art in industry)


Generalizability to various surface imperfection


CstSpeed_5types_12.avi

DNNDirect_5types_12.avi

DRL for AM: Conclusions

Cost reduction from time and material saving,

easier commissioning due to generalizability,

improved performance & scalability,

high precision geometric and material property

Questions?

Explainability & Certification of these approaches for safety-critical systems…

Publications

• Amit Surana, Soumalya Sarkar, and Kishore K. Reddy, “Guided Deep Reinforcement Learning for Additive Manufacturing Control Application”, NIPS 2016 Deep Reinforcement Learning Workshop, December 2016.

• Soumalya Sarkar, Kishore K. Reddy, Michael Giering, and Mark Gurvich, “Deep Learning for Structural Health Monitoring: A Damage Characterization Application”, Conference of the Prognostics and Health Management Society, August 2016.

• Kishore K. Reddy, Soumalya Sarkar, Vivek Venugopalan, and Michael Giering, “Anomaly Detection and Fault Disambiguation in Large Flight Data: A Multi-modal Deep Autoencoder Approach”, Conference of the Prognostics and Health Management Society, August 2016.


deep representation and reinforcement learning for anomaly ...€¦ · rome, italy joined utc in...

Documents