learning fault patterns from data in streaming applications

Noname manuscript No.(will be inserted by the editor)

Learning fault patterns from data in streaming applications

Sida Chen ¨ Shigeru Imai ¨ Wennan Zhu ¨ Carlos A. Varela

Received: date / Accepted: date

Abstract In sensor-based systems, spatio-temporal data

streams are often related in non-trivial ways. For ex-

ample in avionics, while the airspeed that an aircraft

attains in cruise phase depends on the weight it carries,

it also depends on many other factors such as engine

inputs, angle of attack and air density. It is therefore a

challenge to develop failure models that can help recog-

nize errors in the data, such as an incorrect fuel quantity

or an incorrect airspeed. In this paper, we learn failure

models from data streams using two approaches: (1)

create static parameterized functional models using re-

gression and use them to detect error modes based on

error signatures, and (2) create statistical models based

on the naıve Bayes classifier that dynamically update

themselves to detect new error modes online. We ex-

tend our own PILOTS system to evaluate the accu-racy, response time, and adaptability of these learning

techniques. While error signatures can be more accu-

rate and responsive than Bayesian learning, the latter

method adapts better due to its data-driven nature.

Keywords data streaming, spatio-temporal data,

declarative programming, linear regression, Bayesian

statistics.

This research is partially supported by the DDDAS programof the Air Force Office of Scientific Research, Grant No.FA9550-15-1-0214, NSF Grant No. 1462342, and a YamadaCorporation Fellowship.

Sida Chen, Shigeru Imai, Wennan Zhu, and Carlos A. Varela110 Eighth Street, Troy NY USA 12180Tel.: (518)276-6000E-mail: {chens15, imais, zhuw5}@rpi.edu, [email protected]

1 Introduction

Detecting and recognizing patterns from stream data

generated by multiple aircraft sensors has become an

important research area for flight safety. In the Air

France flight 447 (AF447) accident in 2009, iced pitot

tubes caused an error in air speed data, and the pilots

failed to react correctly, leading to the crash [7]. While

there have been advances in information fusion [6] and

virtual modeling [27] for avionics control and user warn-

ings, there is still a need for further research in methods

that allow for fault detection and recovery techniques to

be easily realized and implemented with minimal risk

of software errors. Using redundant information pro-

vided by different sensors, the AF447 tragedy could

have been avoided by Dynamic Data-Driven Avionics

Systems (DDDAS) based on the concept of Dynamic

Data-Driven Application Systems [11]. DDDAS can ex-

pand the flight safety envelope of automation and sup-

port pilots with real-time decision making.

To realize the concept of Dynamic Data-Driven Avion-

ics Systems, we are developing the ProgrammIng Lan-

guage for spatiO-Temporal data Streaming applications

(PILOTS) and its runtime system for fault detection

and correction in data streams. PILOTS has evolved

gradually to date. We first designed the PILOTS pro-

gramming language and proposed the concept of error

signatures [19], which identify the shape of error func-

tions to detect error modes. Next, we implemented a

compiler and a runtime system for PILOTS [20] and

then added error detection and correction support to

the compiler and the runtime [23]. Finally, we applied

PILOTS to data streams obtained from actual acci-

dents, Air France 447 [7] and Tuninter 1153 [28], and

confirmed the effectiveness of its error detection and

correction capabilities [18,17].

2 Sida Chen et al.

In some situations, detecting and recovering from

sensor data errors is non-trivial, even for human experts

and flight assistant systems. In the Tuninter 1153 flight

accident in 2005, the fuel quantity indicator of a dif-

ferent aircraft model was erroneously installed, causing

the instrument to display an incorrect amount of fuel,

which led to fuel exhaustion of the aircraft [28]. This

accident could have been avoided if the weight error

could be detected by checking the relationship between

lift and weight during the cruise phase of flight. Lift de-

pends on airspeed, air density, wing surface area, and

coefficient of lift. The coefficient of lift itself depends

on the angle of attack and this relationship will change

with different aircraft types. Understanding such com-

plex relationships from multiple sensor data streams is

critical to accurately detecting sensor faults.

Using linear regression and known relationships among

redundant stream data, in prior research, we were able

to detect and correct for sensor data errors using actual

flight accident data [18,17]. However, aircraft models

might change due to significant aircraft emergencies,

e.g., loss of a wing or loss of engines. To get more accu-

rate results, an online system should be able to incre-

mentally update pre-calculated models, and detect new

modes that may not have been anticipated. The naıve

Bayes classifier is a suitable method for offline training

and incremental learning, but needs to be extended to

detect previously unknown modes.

In this paper, we enhance machine learning tech-

niques originally proposed in [9] to estimate parame-

terized functional models of aircraft sensor data rela-

tionships, and statistically determine aircraft operating

modes. Namely, we extend our PILOTS programming

language and the runtime system to learn failure mod-

els using the following two approaches:

1. Create static parameterized functional models using

regression and use them to detect error modes based

on error signatures

2. Create statistical models based on the naıve Bayes

classifier that dynamically update themselves to de-

tect new error modes online

Characteristics of the model creation approaches are

summarized in Table 1. Functional models are expressed

as a linear combination of fixed-number parameters β,

which are learned using linear regression. We compute

the error using the created model and use it to de-

termine error modes based on error signatures. Unlike

the functional models, statistical models are described

by probability distributions. We implement a dynamic

Bayes classifier for offline training and incremental on-

line learning of different modes, which also detects new

modes as the stream data switches to an unknown pat-

tern. Notice that the former approach is well-suited

to the existing PILOTS framework due to the static

and functional nature of the error signature approach,

whereas the latter approach is radically different from

the existing framework since the models are dynami-

cally updated and are expressed as probability distri-

butions. To evaluate the performance of the proposed

approaches, we have performed two case studies: air-

craft weight estimation and aircraft speed estimation.

Table 1 Characteristics of the model creation approaches.

Static Dynamic

functional model statistical model

Model form y “ Xβ ` ε P py|xq

Learning Linear regression Bayes classifier

method (offline) (offline & online)

ErrorError signature Bayes classifier

detection

ErrorFunctional model Functional model

correction

The rest of the paper is organized as follows. Sec-

tion 2 describes error signature-based error detection

and correction methods as well as the PILOTS pro-

gramming language and the architecture of its runtime

system. Section 3 discusses the design and integration

of the newly developed machine learning components

in PILOTS. Section 4 describes a linear model and its

parameter estimation as an instantiation of the param-

eterized functional models. Section 5 introduces the dy-

namic Bayes classifier and online incremental learning.

Section 6 and 7 discuss the methods and results of the

proposed learning approaches using case studies of air-

plane weight estimation and speed estimation respec-

tively. Section 8 describes related work. Finally, we de-

scribe potential future research directions and conclude

the paper in Section 9.

2 Background

PILOTS1 is a highly-declarative programming language

that has been applied to both the Air France 447 [7]

and the Tuninter 1153 [28] accidents data, showing that

PILOTS was able to successfully detect the data errors

in both cases, and correct the error in the case of Air

France 447 [17].

2.1 Error detection and correction methods

Error functions Error functions are used to detect pos-

sible faults among redundant input stream data. An

1The latest version of PILOTS is available athttp://wcl.cs.rpi.edu/pilots

Learning fault patterns from data in streaming applications 3

error function should have the value zero if there is no

error in the input data, when the whole system is work-

ing in the normal mode.

For example, in the cruise phases of a flight, the lift

equals the weight of the airplane. The lift can also be

calculated using other input data, including airframe’s

angle of attack, air density, temperature, pressure and

airspeed. In this case, an error function could simply be

defined as:

eplift, weightq “ lift´ weight (1)

The lift in Equation (??) is calculated using other

input data. In the normal cruise phase mode, the value

of this equation should be zero. If there is an error in

the weight indicator, and the input weight data is lower

than the real weight, Equation (??) should be greater

than zero. Similarly, if the input weight data is higher

than the real weight, Equation (??) should be smaller

than zero. Thus, the validity of the input data could be

determined from the value of the error function.

The values of input data are assumed to be sampled

periodically from corresponding spatio-temporal data

streams. Thus, an error function e changes its value as

time proceeds and can be represented as eptq.

Error signatures An error signature is a constrained

mathematical function pattern that is used to capture

the characteristics of an error function eptq. Using a vec-

tor of constants K “ xk1, . . . , kmy, a function fpt, Kq,

and a set of constraint predicates P “ tp1pKq, . . . , plpKqu,

the error signature SpK, fpt, Kq, P pKqq is defined as

follows:

Spfpt, Kq, P pKqq fi t f | p1pKq ^ ¨ ¨ ¨ ^ plpKqu. (2)

Mode likelihood vectors Given a vector of error signa-

tures xS0, . . . , Sny, we calculate δipSi, tq, the distance

between the measured error function eptq and each er-

ror signature Si by:

δipSi, tq “ mingptqPSi

ż t

t´ω

|eptq ´ gptq|dt. (3)

where ω is the window size. Note that our convention

is to capture “normal” conditions as signature S0. The

smaller the distance δi, the closer the raw data is to the

theoretical signature Si. We define the mode likelihood

vector as Lptq “ xl0ptq, l1ptq, . . . , lnptqy where each liptq

is:

liptq “

#

1, if δiptq “ 0mintδ0ptq,...,δnptqu

δiptq, otherwise.

(4)

Mode estimation Using the mode likelihood vector, the

final mode output is estimated as follows. Observe that

for each li P L, 0 ă li ď 1 where li represents the ratio

of the likelihood of signature Si being matched with

respect to the likelihood of the best signature.

Because of the way Lptq is created, the largest el-

ement lj will always be equal to 1. Given a threshold

τ P p0, 1q, we check for one likely candidate lj that is

sufficiently more likely than its successor lk by ensur-

ing that lk ď τ . Thus, we determine j to be the correct

mode by choosing the most likely error signature Sj . If

j “ 0 then the system is in normal mode. If lk ą τ ,

then regardless of the value of k, unknown error mode

(´1) is assumed.

Error correction Whether or not a known error mode i

is recoverable is problem dependent. If there is a math-

ematical relationship between an erroneous value and

other independently measured values, the erroneous value

can be replaced by a new value estimated from the other

independently measured values.

2.2 PILOTS: spatio-temporal data stream processing

system

Figure 1 shows the architecture of the PILOTS runtime

system, which implements the error detection and cor-

rection methods described in Section 2.1. It consists of

three parts: the Data Selection, the Error Analyzer, and

the Application Model modules. The Application Model

obtains homogeneous data streams pd11, d12, . . . , d

1N q from

the Data Selection module, and then it generates out-

puts (o1, o2, . . . , oM ) and data errors (e1, e2, . . . , eL).

The Data Selection module takes heterogeneous incom-

ing data streams (d1, d2, . . . , dN ) as inputs. Since this

runtime is assumed to be working on moving objects,

the Data Selection module is aware of the current lo-

cation and time. Thus, it returns appropriate values

to the Application Model by selecting or interpolating

data in time and location, depending on the data se-

lection method specified in the PILOTS program. The

Error Analyzer collects the latest ω error values from

the Application Model and keeps analyzing errors based

on the error signatures. If it detects a recoverable error,

then it replaces an erroneous input with the estimated

one by applying a corresponding estimation equation.

The Application Model computes the outputs based on

the estimated inputs produced by the Error Analyzer.

4 Sida Chen et al.

�

��

��

�

��

��

�

��

��

�

�

�

��

��

��

��

��

�

�

�

�

�

�

�

�

�

�

�

�

�

�

�

��

�

��

��

�

�

�

�

�

�

�

�

�

� � �

��

� � ��

�

�

�

��

� �

��

��

��

�

��

�

��

�

�

��

��

��

��

��

�

��

�

��

�

�

��

� � �

��

��

��

��

��

��

��

� � ��

��

��

��

��

��

Fig. 1 Architecture of the PILOTS runtime system.

3 Machine learning in PILOTS

We integrate machine learning component into the PI-

LOTS runtime system to support online/offline model

training and online prediction. To support the newly

added functionalities, we also update the PILOTS pro-

gramming language grammar.

3.1 Modifications to PILOTS runtime system

As shown in Figure 2, we design the machine learn-

ing component as an independent server system for a

better modularity. The server is designed to work with

arbitrary learning algorithms, and clients (i.e., other

components in the PILOTS runtime system) can re-

quest the following to the server: model training, value

prediction using the trained model, and error mode es-

timation when using Bayes classifiers.

For the offline training, the Learning Engine trains

learning models using three major parts: (1) Training

Definition including learning hypothesis, learning algo-

rithm configuration, preprocessing methods, and data

file configuration (see for example Figure 6 for linear

regression and Figure 11 for dynamic Bayes classifier);

(2) learning algorithms such as least squares, Bayes

classifier or others; (3) training data, which refers to

data stored in files. For the online training, the Learn-

ing Engine updates the dynamic online learning model

for every prediction made and gives prediction result

p back to the Data Selection component, whcompo-

nentich sends the prediction result to the Application

Model as requested.

Prediction requires runtime interaction between the

PILOTS runtime and the machine learning component.

Data Selection is altered to support communication to

outside component through a socket. When Applica-

tion Model requests p from Data Selection module, it

first computes the input vector x “ rd1i1d1i2¨ ¨ ¨ d1ins

T us-

ing data selection method defined for each di, and then

sends model along with input vector x to the Learn-

ing Engine where the prediction is made by requested

model using input vector x. Error Analyzer now allows

detection of error modes using methods other than error

signatures by sending request of error mode to learning

engine and retrieving mi as the error mode through the

same interface used by Data Selection module.

3.2 Enhancements to PILOTS programming language

To support prediction in the language, a new data selec-

tion method predict is defined in addition to closest,

euclidean, and interpolate. Method predict(model, d1i1 ,d1i2 ,¨ ¨ ¨ ,d1in) takes the identifier of the model

as used for prediction as the first argument, model,

and the data streams, d1i1 , d1i2, ¨ ¨ ¨ , d1in , used for input

as the following arguments. predict method is im-

plemented as an interface, accepting different machine

learning models including online models, offline mod-

els, regressors, and classifiers. Figure 3 shows a simple

example PILOTS program PredictionTest, where

aptq and bptq are data streams retrieved by closestmethod meaning that the values of aptq and bptq with

closest timestamp t to current time are chosen, and the

predicted data stream cptq uses the prediction method

with linear regression as predictive model and

aptq, retrieved by closest, as input stream to the

linear regression. Assuming data streams aptq and

bptq have a linear relationship, which is captured by

the linear regression, cptq is the prediction result

of the linear regression from aptq. The outputssection compares cptq with bptq to produce pairwise dif-

ference between actual data (stream b) and output of

the linear regression (stream c).

�

�

�

�

program PredictionTest;inputsa,b (t) using closest (t);c (t) using predict(linear_regression, a);

outputsdifference: b - c at every 1 sec;

end

Fig. 3 A simple PILOTS program example outputting error.

4 Learning static linear models

Linear regression is a well-studied and powerful tool for

estimating inter-variable relations in linear models. The


Fig. 2 Updated PILOTS runtime architecture with machine learning component.

equation for a linear regression model is

y “ Xβ ` ε

where

y “

¨

˚

˚

˚

˝

y1

y2

...

yn

˛

‹

‹

‹

‚

, X “

¨

˚

˚

˚

˝

x11 ¨ ¨ ¨ x1m

x21 ¨ ¨ ¨ x2m

.... . .

...

xn1 ¨ ¨ ¨ xnm

˛

‹

‹

‹

‚

, β “

¨

˚

˚

˚

˝

β1

β2

...

βm

˛

‹

‹

‹

‚

,

ε “

¨

˚

˚

˚

˝

ε1ε2...

εn

˛

‹

‹

‹

‚

yi is dependent variable; xij is independent variable; βiis regression coefficient; εi is error term.

4.1 Learning algorithm

There are multiple methods to solve linear models. One

of the learning algorithms implemented in our system is

ordinary least squares, of which the target is minimizing

square of the Euclidean norm ||y´Xβ||2 by finding the

best coefficient vector β

β “ arg minβ

||y ´Xβ||2

Assuming the columns in X are linearly independent,

we could retrieve β in closed form

β “ pXTXq´1XT y

4.2 Linear model accuracy metrics

– Coefficient of determination: This metric is used

to evaluate goodness of regression model fitting on

training set R2 “ 1 ´ř

ipyi´yiq2

ř

ipyi´yq2 where yi is mea-

sured/dependent variable, yi is estimated variable,

y is the average of all yi.

– Root mean squared error : This metric is used to

evaluate the amount of error produced by prediction

on average RMSE “b

1n

řni“1pyi ´ fpxiqq

2 where

f is an estimator, xi is an independent variable vec-

tor rxi1 xi2 ¨ ¨ ¨ ximsT .

5 Learning dynamic statistical models

Naıve Bayes classifiers [29,21] are commonly used in

supervised training and classification. For continuous

data, if the values of samples in each class are assumed

to be normally distributed, the classifiers are called

Gaussian naıve Bayes classifiers [22]. In the training

phase, tagged samples of different classes are processed

to train the parameters of the classifier. The param-

eters include the mean value, standard variance, and

prior probability of each class. In the testing phase, the

trained Bayes classifier decides the class of untagged

input samples.

One limitation of the traditional naıve Bayes clas-

sifier is that the input samples in the testing phase

will only be classified into classes that appeared in the

training phase. If a sample of a previously unknown

6 Sida Chen et al.

class appears, it will be classified into one of the known

classes, even if the probability that it belongs to that

class is very low. However, with dynamic stream data,

new modes not in the training set could occur in some

complex situations. For example, if a Bayes classifier is

trained to recognize the “normal weight” and “under-

weight” modes of the weight indicator on an airplane

during flights, and a previously unknown mode “over-

weight” appears in testing phase, the classifier will not

be able to detect this new mode, but will classify the

samples to “normal weight” or “underweight” based on

the value and prior probability of the modes.

To tackle this limitation of the naıve Bayes classi-

fier, we extend it into a dynamic Bayes classifier that

has two phases: (1) Offline: Supervised learning, which

is the same as Gaussian naıve Bayes classifiers. (2) On-

line: Unsupervised dynamic incremental learning, that

classifies samples in known modes, updates parameters

of the model, and create new modes for samples in pre-

viously unknown modes. Because we focus on process-

ing stream data during flights and deciding the normal

or error operational modes of an airplane, the words

“mode” and “class” are used interchangeably and have

the same meaning in this paper.

5.1 Offline supervised learning

5.1.1 Gaussian naıve Bayes classifiers

In a Gaussian naıve Bayes classifier [22], each input

sample X is described by a feature vector px1, . . . , xnq,

and each sample is classified into a target class y P

ty1, . . . , ymu. In this paper, we consider samples of only

one feature x, but the results can be generalized to n

features. By Bayes’ theorem, the conditional probabil-

ity P py|xq is:

P py|xq “P pyqP px|yq

P pxq(5)

As the samples in each feature are assumed to be

normally distributed, P px|yq is calculated by:

P px|yq “1

b

2πσ2y

e´px´µyq

2

2σ2y (6)

Where µy is the mean of the values in x associated

with class y, and σy is the standard deviation of the

values in x associated with class y.

The corresponding classifier y is:

y “ arg maxP py|xq (7)

Because P pxq is the same for each class, y is:

y “ arg maxP py|xq “ arg max P pyqP px|yq (8)

5.1.2 Offline learning phase

In the offline supervised learning phase, input data tagged

with mode labels are processed by a Gaussian naıve

Bayes classifier. The mean value µy, standard devia-

tion σy, and the prior probability P pyq of each mode y,

are calculated by the classifier, as in Figure 4.

5.2 Dynamic online unsupervised learning by dynamic

Bayes classifier

5.2.1 Major and minor modes

To support dynamically changing modes during the on-

line learning phase, the concepts of major and minor

modes are introduced. The modes in the offline super-

vised learning phase are major modes. In the prediction

and online learning phase, before a sample is processed

by the Bayes classifier, the value is checked by a pre-

processor to decide if it is in the range of each major

mode. As 95% of the values in a normal distribution lie

within µ˘ 2σ, if the sample is not within that range of

any major mode, a new minor mode is created. As more

data are processed, when the number of samples in a

minor mode exceeds a threshold, it will be turned into

a major mode. Minor modes are used to keep samples

differentiated from known major modes. A threshold is

used to diminish the impact of noise. Mode ID is auto-matically assigned when a new mode is detected.

5.2.2 Online learning phase

The process of dynamic online unsupervised learning

is shown in Figure 4. The parameters of initial major

modes are from the training results of the offline train-

ing phase. As untagged samples are processed, if the

value is within µ ˘ 2σ of any major mode, the sample

will be classified by naıve Bayes classifier, and the pa-

rameters are incrementally updated. If the value is not

within µ˘ 2σ of any major mode, but is within µ˘ 2σ

of a minor mode, it will be classified into the closest

minor mode, and the parameters of minor modes are

updated accordingly. Finally, if the value of the sample

is not within µ ˘ 2σ of any major or minor mode, a

new minor mode will be created for this sample. σ of

the new minor mode is initially set as the average σ of

the existing major modes. When the size of the minor

mode is greater than a threshold, we start to calculate


and use the real σ of the minor mode. The reason is

that the σ might be biased if the number of samples is

too small. Each time when the parameters of a minor

mode are updated, if the number of samples exceeds

a certain threshold, it will be upgraded into a major

mode.

Fig. 4 Online classification and incremental learning usingdynamic Bayes classifier.

6 Case study: airplane weight estimation

To help prevent accidents caused by fuel quantity in-

dicator errors such as the Tuninter 1153 flight acci-

dent, we use the X-Plane flight simulator to generate

flight sensors data and simulate airplane weight error

scenarios. With the synthetic data, machine learning

techniques are applied for inferring airplane model pa-

rameters from data, airplane weight error detection and

actual weight estimation.

6.1 Estimation by static linear model

6.1.1 Training weight model using linear regression

Weight model design In cruise phase, when yaw, roll

angles are close to zero and pitch is small, we assume

L “ W , in which L is total lift and W is gross weight.

Based on the assumption, we can estimate W by the

lift equation:

W “ L “1

2v2SρCl, (9)

where v is true air speed, S is wing surface area, ρ is air

density and Cl is coefficient of lift. From ideal gas law,

we know ρ “ pR1T , where p is ambient air pressure, R1

is special gas constant and T is ambient temperature,

and replace ρ with pR1T in Equation (9) to get:

W “pv2SCl2R1T

(10)

and by transforming Equation (10), Cl, coefficient of

lift could be represented by:

Cl “2WR1T

pv2S. (11)

Generally Cl depends on the shape of airfoil and the

shape of an aircraft. To roughly estimate Cl, the com-

plex physical model is simplified using Thin-Airfoil the-

ory, which predicts a linear relationship [5] between co-

efficient of lift, Cl, and the angle of attack, α, for low

values of α, as shown in Figure 5 between dashed ver-

tical lines. This relationship can be expressed as:

Cl “ β1α` β2 ` ε (12)

where ε is noise and α is known while β1 and β2 are

distinct values for different aircrafts. A linear model

could be formulated as the following:

y “ Xβ ` ε (13)

y “

¨

˚

˚

˚

˝

Cl1Cl2

...

Cln

˛

‹

‹

‹

‚

, Cli “2WiR

1Tipiv2

i S,X “

¨

˚

˚

˚

˝

α1 1

α2 1...

αn 1

˛

‹

‹

‹

‚

, β “

ˆ

β1

β2

˙

Because each column in X is independent, we could

use the least squares method defined in Section 4.1 to

retrieve β, and predict W using the following equation:

W “pv2Spβ1α` β2q

2R1T(14)

where we have substituted the linear estimation of Cl,

in Equation (10).

Fig. 5 Typical graph of section coefficient of lift ver-sus angle of attack for a cambered airfoil, adapted fromhttps://en.wikipedia.org/wiki/Lift coefficient.

8 Sida Chen et al.

Experimental settings Synthetic data with simple rela-

tionships are used to verify the integration of machine

learning approaches into PILOTS. In this example, sim-

ulated ATR-72 500 airplane data is used for PILOTS to

detect and correct for weight error in the data streams,

and the relation between coefficient of lift and angle

of attack is investigated. To simulate and test linear

regression implemented in PILOTS machine learning

component, we assume certain known variables. It is

required that the following variables are correctly mea-

sured and known: gross weight W , ambient pressure p,

true airspeed v, wing surface area S, special gas con-

stant for dry air R1, and ambient temperature T .

The linear regression model is trained with synthetic

data using offline training definition file as shown in

Figure 6. The estimator file will generate an estimator

with ID linear regression, where the model is de-

fined in model section with definition of feature, label

and algorithm for the solving model. In this case, a lin-

ear regression model solved by pseudo inverse is used.

The model will be trained by data defined in data sec-

tion, in which also the variables are initialized. schemadefines the additional attributes to the variables and

preprocess defines the actions to be taken on input

dataset before training. The generated estimator will

be used for estimation of coefficient of lift from angle of

attack.

'

&

$

%

estimator linear_regression;/* v: true air speed,

a: angle of attack,p: pressure, t: temperature,w: gross weight, S: wing area,R: Gas constant

*/constantsR = 286.9;S = 61.0;

datav, p, t, w, a using file( training.csv );schema

unit(v:knot, p:in_Hg, t:celsius,w:force_pound, a:degree);

modelpreprocess

changeunit usingunit(v:m/s, p:pascal, t:kelvin,

w:newton, a:radian);feature: a;label: 2*w/(vˆ2 * (p/R/t)*S);algorithm: LinearRegression using PseudoInverse();

end

Fig. 6 Offline training parameters for the linear regressionmodel.

Training result Figure 7 shows the training result of

linear relationship between angle of attack and coef-

ficient of lift, where the learned parameters are β1 “

6.3888 and β2 “ 0.3589. The evaluation of the trained

model gives R2 “ 0.9949, RMSE “ 0.00794, showing

a strong linear relationship with low in-sample error.

Using Equation (9), we compute the training error be-

tween measured weight and estimated weight, resulting

in RMSE “ 2687N .

Fig. 7 The linear relation between angle of attack and coef-ficient of lift in cruise phase of training set.

6.1.2 Performance metrics for error mode detection

We evaluate the performance of error detection based

on accuracy and response time, which are defined as

follows:

– Accuracy: This metric is used to evaluate how ac-

curately the algorithm determines the true mode.

Assuming the true mode transition mptq is known

for t “ 0, 1, 2, . . . , T , let m1ptq for t “ 0, 1, 2, . . . , T

be the mode determined by the error detection al-

gorithm. We define accuracypm,m1q “ 1T

řTt“0 pptq,

where pptq “ 1 if mptq “ m1ptq and pptq “ 0 other-

wise.

– Maximum/Minimum/Average Response Time: This

metric is used to evaluate how quickly the algorithm

reacts to mode changes. Let a tuple pti,miq repre-

sent a mode change point, where the mode changes

to mi at time ti. Let

M “ tpt1,m1q, pt2,m2q, . . . , ptN ,mN qu,

and

M 1 “ tpt11,m11q, pt

12,m

12q, . . . , pt

1N 1 ,m

1N 1qu,

where M and M 1 are the sets of true mode changes

and detected mode changes respectively. To remove

noise in the detected changes, we further apply the

following operation to M 1: if t1i`1 ´ t1i ď δ, remove

a tuple pt1i`1,m1i`1q from M 1, where δ is a small

threshold value to eliminate frequent mode changes.


For each i “ 1 . . . N , we can find the smallest t1j such

that pti ď t1jq ^ pmi “ m1jq; if not found, let t1j be

ti`1. The response time ri for the true mode mi

is given by t1j ´ ti. We define the maximum, min-

imum, and average response time by max1ďiďN ri,

min1ďiďN ri, and 1N

řNi“1 ri respectively.

6.1.3 Error detection and correction using error

signatures

Error signatures The error function e is given by Equa-

tion (15) as the percentage of discrepancy between pre-

dicted weight W and measured weight W . The error

signatures use a threshold of 3.5% because this number

is more rigorous than the percentage of discrepancy be-

tween error weight and actual weight in Tuninter 1153

accident, which is about 10%.

e “W ´ W

W(15)

Mode Error SignatureFunction Constraints

Normal e “ k ´0.035 ă k ă 0.035Overweight e “ k k ą 0.035

Underweight e “ k k ă ´0.035

Table 2 Error signatures set for weight correction.

PILOTS program The WeightCorrection PILOTS

program implementing the error signatures in Table 2

is shown in Figure 8. If the error signature s1 or s2 is

detected, the program estimates weight using Equation

(14). The data selection module computes v1, a1, p1, te1,

w1 using data points with the closest time stamp, and

uses a1 as an input matrix to predict cl1 using model

the with id linear regression.

Experimental settings X-Plane 9.7 is used to generate

flying data of ATR72-500 in different altitudes, gross

weights and power settings. The data is split by se-

lecting three flights’ 25 cruise phases, 1251 minutes in

total, as training set, and one 20-minutes flight with

four cruise phases as testing set. The model is trained

by 25 cruise phases in the training set and tested by

four cruise phases in the testing set. To evaluate the

PILOTS error detection accuracy, the whole testing set

is modified to introduce artificial measurement errors

as follows: weight data in the range from 1 to 100 and

750 to 800 seconds is multiplied by 0.9, from 1025 to

1099 seconds is multiplied by 1.1, from 390 to 490 sec-

onds is multiplied by 1.05, from 570 to 648 seconds is

'

&

$

%

program WeightCorrection;/* v: true air speed [m/s],

a: angle of attack [rad],p: pressure [Pa], te: temperature [K],w: gross weight [N], S: wing area [mˆ2],R: Gas constant [J kgˆ-1 Kˆ-1],cl: coefficient of lift

*/constantsR = 286.9;S = 61.0;

inputsv, a, p, te, w (t) using closest(t);cl (t) using predict(linear_regression, a);

outputscorrected_weight: w at every 1 sec;

errorse: (w - p*(v*v)*S*cl/(2*R*te)) / w;

signaturess0: e = K, -0.035 < K < 0.035 "Normal";s1: e = K, K > 0.035 "Underweight"estimate w = p*(v*v)*S*cl/(2*R*te);

s2: e = K, K < - 0.035 "Overweight"estimate w = p*(v*v)*S*cl/(2*R*te);

end

Fig. 8 Declarative specification of WeightCorrection PI-LOTS program using error signature.

multiplied by normal distribution of error with mean

at 1 and standard deviation at 0.1, from 291 to 377

seconds are multiplied by uniform distribution of error

ranging from 0.9 to 1.1. The cruise phases of testing set

lie between 5 to 164, 230 to 395, 470 to 688 and 780 to

1108 seconds. We can visualize this data as “measured”

in Figure 9.

PILOTS program WeightCorrection in Figure 8

is executed with different combinations of window sizes

ω P t1, 2, 4, 8, 16u and thresholds τ P t0.2, 0.4, 0.6, 0.8, 0.99u

to investigate the accuracy and average response time,

which is computed using δ = 0, because of the instabil-ity of error modes in the dataset.

Results Figure 9 shows the estimated weight and mea-

sured weight during the 18 minutes flight where ω “ 1

and τ “ 0.99, the best combination among all combina-

tions in accuracy and response time. The PILOTS pro-

gram can successfully detect and correct underweight

and overweight conditions in cruise phases, with root

mean squared error close to 1617N. The program per-

forms the best in system failure simulation regions of

cruise phase 1 and 4, where the weight drifts by 10%

or 5%. The accuracy is 100% and max/min/average

response time is 0/0/0 seconds. In random error simu-

lation regions of cruise phase 2 and 3, performance is

decreased to have accuracy 89.1% and max/min/aver-

age response time 84/0/1.76 seconds. The overall accu-

racy is 95.2% and max/min/average response time is

84/0/1.65 seconds. Outside cruise phases, the program

does not estimate weight properly since the assumption

L “W does not hold.

10 Sida Chen et al.

Fig. 9 Error detection and correction using ω “ 1, τ “ 0.99for X-Plane simulated data.

6.2 Estimation by dynamic statistical model

6.2.1 Offline training of dynamic Bayes classifier

We use W ´ W as the feature for the dynamic Bayes

classifier. Estimated weight is calculated by Equation

(9) using the method described in Section 6.1.1. The

dynamic Bayes classifier is trained with both “normal”

and “underweight” tagged data in the offline learning

phase. Figure 10 shows the parameters setting for the

offline training phase. Similar to Figure 6, PILOTS uses

the definition of the program to generate an estima-

tor with ID bayes weightcorrection, in which the

model is defined in model with dynamic bayesian clas-

sifier using sigma scale 2, which represents µ ˘ 2σ for

range of a mode for classification and threshold 100 for

turning a minor mode into a major mode. Estimator

linear regression is used for estimating Cl, which

is then used for estimation of W .

'

&

$

%

estimator bayes_weightcorrection;/* v: true air speed,

a: angle of attack,p: pressure, t: temperature,w: gross weight, S: wing area,R: Gas constant, mode: labeled mode for training,cl: coefficient of lift

*/constantsS = 61.0;R = 286.9;

datav, p, t, w, a, mode using file(training.csv);cl using predict(linear_regression, a);schema

unit(v:knot, p:in_Hg, t:celsius,w:force_pound, a:degree);

modelpreprocess

changeunit usingunit(v:m/s, p:pascal, t:kelvin,

w:newton, a:radian);feature: w - 0.5*vˆ2*(p/R/t)*2*cl;label: mode;algorithm: BayesianClassifier using

DynamicBayesClassifier(sigma_scale:2, threshold:100);

end

Fig. 10 Offline training parameters for the dynamic Bayesclassifier.

6.2.2 Online error detection and correction using

dynamic Bayes classifier

PILOTS program The WeightCorrectionBayes PI-

LOTS program is shown in Figure 11. This program

is used for the online learning and classification to de-

tect and give weight estimation to different weight error

modes.

'

&

$

%

program WeightCorrectionBayes;/* v: true air speed [m/s],

a: angle of attack [rad],p: pressure [Pa], te: temperature [K],w: gross weight [N], S: wing area [mˆ2],R: Gas constant [J kgˆ-1 Kˆ-1],cl: coefficient of lift

*/constantsS = 61.0;R = 286.9;

inputsv, a, p, te, w (t) using closest(t);cl (t) using predict(linear_regression, a);mode (t) using predict(bayes_weightcorrection,

v, p, te, w, cl);modesm0: mode = 0 "Normal";m1: mode = 1 "Underweight"

estimate w = p*(v*v)*S*cl/(2*R*te);end

Fig. 11 A declarative specification of theWeightCorrectionBayes PILOTS program using thedynamic Bayes classifier.

Experimental settings We use the same performance

metrics for major mode prediction: accuracy and re-

sponse time as in Section 6.1.2.

See Section 6.1.3 for data generation. We use the

same testing data, and 8000 seconds training data in

cruise phase modified as follows: weight data in the

range from 1526 to 3129 second are multiplied by 1.1

to simulate underweight mode. There are two major

modes in the tagged training data: mode 0 for normal

status and mode 1 for underweight status. For online

learning, we set the threshold of the sample number to

turn a minor mode into a major mode to 100. Instead of

using average σ, the sample number threshold for cal-

culating σ of a new mode is also set as 100. Figure 12

(a) shows the feature and Figure 12 (b) shows tagged

mode of training data. The major modes of the classi-

fier after training phase is shown in Table 3 where n is

the number of datapoint classified to a mode.

Results Figure 13 and Table 4 show the results of weight

error mode detection by dynamic Bayes classifier. Us-

ing the same testing data as in Figure 9, the dynamic

Bayes classifier successfully detects three major modes


in the cruise phases: mode 0 for normal, mode 1 for

underweight, and mode 3 for overweight. Mode 0 and

mode 1 are major modes that appeared in the tagged

training data, mode 3 is a new major mode detected

by the classifier during the online incremental learning

and prediction phase. There are also 22 minor modes

generated by the noise and non-cruise phase data in the

testing set. Using the metric described in Section 6.1.2

on only major modes, in systematic failure simulation

regions of cruise phase 1 and 4, the accuracy is 95.3%

and max/min/average response time is 19/0/3.83 sec-

onds, while in random error simulation regions of cruise

phase 2 and 3, the accuracy is largely decreased to

66.7% and max/min/average response time is increased

to 98/0/5.38 seconds. The overall accuracy of major

mode detection on all cruise phases is 82.7% and max/min/av-

erage response time is 98/0/5.28 seconds. Figure 14 il-

lustrates the result of weight estimation in detected un-

derweight mode. When in underweight mode and cruise

phase, the estimation gives root mean squared error of

approximately 1420N.

Table 3 Major modes of dynamic Bayes classifier after run-ning training dataset.

Mode description Mode µ σ nNormal 0 49 2534 5937

Underweight 1 16068 3249 1604

Fig. 12 Weight error mode training data for the dynamicBayes classifier.

Table 4 Major and minor modes of dynamic Bayes classifierafter error mode detection on testing dataset.

Modestatus

Modedescription

Mode µ σ n

MajorNormal 0 -50 2481 6522

Underweight 1 15961 3201 1746Overweight 3 -14472 1151 121

Minor Noise 2, 4-24 N/A N/A N/A

Fig. 13 Weight error mode detection using dynamic Bayesclassifier.

Fig. 14 Weight error correction using dynamic Bayes classi-fier.

6.3 Summary of results

In systematic failure simulation regions, the average

response time of the error signatures approach with

0.035 as threshold, ω “ 1, and τ “ 0.99, is 3.83 sec-

onds shorter than that of the dynamic Bayes classifier,

and the error signatures approach is 4.7% more accu-

rate than the dynamic Bayes classifier. And in random

error simulation regions, the error signature approach

produces 3.62 seconds shorter average response time

and 22.4% more accurate than those of the dynamic

Bayes classifier. One of the reason is that in random

12 Sida Chen et al.

error simulation regions, many minor modes are cre-

ated instead of major modes, resulting in poor perfor-

mance of major modes on the regions. However, the

dynamic Bayes classifier discovers discrete error modes

dynamically and automatically while the error signa-

tures approach is static, that is, every signature must

be predefined manually.

7 Case study: airplane speed estimation

To help prevent accidents caused by pitot tube failures

such as the Air France 447 flight accident, we use ac-

tual speed data recovered from the Air France 447’s

flight data recorder for evaluation. For training, we de-

rive a functional model based on the laws of physics for

the static linear model, whereas we use synthetic data

streams which consist of airspeed, ground speed, and

wind speed for the statistical model.

7.1 Estimation by static linear model

7.1.1 Derivation of speed check model from physics

Unlike the weight model in Section 6, the relationship

between airplane speeds can be modeled in a simpler

way. The relationship between ground speed, airspeed,

and wind speed follows the following formula:

ÝÑvg “ ÝÑva `ÝÑvw. (16)

where ÝÑvg ,ÝÑva, and ÝÑvw represent the ground speed, the

airspeed, and the wind speed vectors. A vector ÝÑv can

be defined by a tuple pv, αq, where v is the magnitudeof ÝÑv and α is the angle between ÝÑv and a base vector.

Following this expression, ÝÑvg ,ÝÑva, and ÝÑvw are defined as

pvg, αgq, pva, αaq, and pvw, αwq respectively as shown in

Figure 15.

To examine the relationship in Equation (16), we

can compute ÝÑvg by applying trigonometry to 4ABCas shown in Figure 15. We can define an error function

as the difference between measured vg and computed

vg as follows:

epÝÑvg ,ÝÑva,ÝÑvwq “ |ÝÑvg ´ pÝÑva `ÝÑvwq|

“ vg á

v2a ` 2vavw cospαa ´ αwq ` v2

w.

(17)

7.1.2 Error detection and correction using error

signatures

Error signatures We consider the following four error

modes: 1) normal (no error), 2) pitot tube failure due

to icing, 3) GPS failure, 4) both pitot tube and GPS

vw

va

αa-α

w

vg

αa

C

A

B

αw

Fig. 15 Trigonometry applied to the ground speed, airspeed,and wind speed.

failures. Suppose the airplane is flying at airspeed va.

For computing error signatures for different error condi-

tions, we will assume that other speeds as well as failed

airspeed and ground speed can be expressed as follows.

– ground speed: vg « va.

– wind speed: vw ď ava, where a is the wind to air-

speed ratio.

– pitot tube failed airspeed: blva ď va ď bhva, where

bl and bh are the lower and higher values of pitot

tube clearance ratio and 0 ď bl ď bh ď 1. 0 repre-

sents a fully clogged pitot tube, while 1 represents

a fully clear pitot tube.

– GPS failed ground speed: vg “ 0.

We assume that when a pitot tube icing occurs, it is

gradually clogged and thus the airspeed data reported

from the pitot tube also gradually drops and eventually

remains at a constant speed while iced. This resulting

constant speed is characterized by ratio bl and bh. On

the other hand, when a GPS failure occurs, the ground

speed suddenly drops to zero. This is why we model the

failed ground speed as vg “ 0.

In the case of pitot tube failure, let the ground

speed, wind speed, and airspeed be vg “ va, vw “ ava,

and va “ bva. The error function (17) can be expressed

as follows:

e “ va á

v2apb

2 ` 2ab cospαa ´ αwq ` a2q.

Since ´1 ď cospαa ´ αwq ď 1, the error is bounded by

the following:

va á

v2apa` bq

2 ď e ď va á

v2apa´ bq

2

p1´ a´ bqva ď e ď p1´ |a´ b|qva. (18)

In the case of GPS failure, let the ground speed, wind

speed, and airspeed be vg “ 0, vw “ ava, and va “ va.

The error function (17) can be expressed as follows:

e “ 0á

v2ap1` 2a cospαa ´ αwq ` a2q.


Similarly to the pitot tube failure, we can derive the

following error bounds:

´pa` 1qva ď e ď ´|a´ 1|va. (19)

We can derive error bounds for the normal and both

failure cases similarly. Applying the wind to airspeed

ratio a and the pitot tube clearance ratio bl ď b ď bh to

the constraints obtained in Inequalities (18) and (19),

we get the error signatures for each error mode as shown

in Table 5.

Table 5 Error signatures for speed data.

ModeError Signature

Function Constraints

Normal e “ k k P r´ava, avas

Pitot tube failure e “ kk P rp1 ´ a ´ bhqva,p1 ´ |a ´ bl|qvas

GPS failure e “ k k P r´pa ` 1qva,´|a ´ 1|vasBoth failures e “ k k P r´pa ` bhqva,´|a ´ bl|vas

When a “ 0.1, bl “ 0.2, and bh “ 0.33, the error

signatures shown in Table 5 are visually depicted in

Figure 16.

0.1va

e

-0.1va

0.57va

0.9va

-0.43va

-0.9va

-1.1va

tNormal

Both failures

Pitot tube failure

GPS failure

Fig. 16 Error Signatures for speed data (a “ 0.1, bl “ 0.2,and bh “ 0.33).

PILOTS program The SpeedCheck PILOTS program

implementing the error signatures shown in Table 5 is

presented in Figure 17. This program checks if the wind

speed, airspeed, and ground speed are correct or not,

and computes a crab angle, which is used to adjust the

direction of the aircraft to keep a desired ground track.

For this program to be applicable to the aircraft used

for Air France 447, we use a cruise speed of 470 knots

as va.

'

&

$

%

program SpeedCheck;/* va: airspeed, aa: airspeed angle,

vw1: wind speed,aw1: wind speed angle,vw2: estimated wind speed,aw2: estimated wind speed angle,vg: ground speed, ag: ground angle */

inputsva, vg, vw1, vw2 (t) using closest(t);aa, ag, aw1, aw2 (t) using closest(t);

outputsva_out: va at every 1sec;

errorse: vg - sqrt(va*va + vw1*vw1 +

2*va*vw1*cos((PI/180)*(aw-aa)));signatures/* v_a=470, a=0.1, b=0.2...0.33 */s0: e = K, -47 < K, K < 47 "No error";s1: e = K, 220.9 < K, K < 517 "Pitot tube failure"estimate va =

sqrt(vg*vg + vw2*vw2 -2*vg*vw2*cos((PI/180)*(ag-aw2)));

s2: e = K, -517 < K, K < -423 "GPS failure"estimate vg =sqrt(va*va + vw2*vw2 +

2*va*vw2*cos((PI/180)*(aw2-aa)));s3: e = K, -203.66 < K, K < -47"Pitot tube + GPS failure";

end

Fig. 17 Declarative specification of the SpeedCheck PI-LOTS program.

Experimental settings The ground speed and airspeed

are collected based on Appendix 3 in the final report of

Air France flight 447 [7]. Note that the (true) airspeed

was not recorded in the flight data recorder so that we

computed it from recorded Mach (M) and static air

temperature (SAT ) data. The airspeed was obtained

by using the relationship: va “ a0Ma

SAT {T0, where

a0 is the speed of sound at standard sea level (661.47

knots) and T0 is the temperature at standard sea level(288.15 Kelvin). Independent wind speed information

was not recorded either. According to the description

from page 47 of the final report: “(From the weather

forecast) the wind and temperature charts show that

the average effective wind along the route can be esti-

mated at approximately ten knots tail-wind.” We fol-

lowed this description and created the wind speed data

stream as ten knots tail wind.

According to the final report, speed data was pro-

vided from 2:09:00 UTC on June 1st 2009 and it be-

came invalid after 2:11:42 UTC on the same day. Thus,

we examine the valid 162 seconds of speed data in-

cluding a period of pitot tube failure which occurred

from 2:10:03 to 2:10:36 UTC. so the set of true mode

changes is defined as M “ tp1, 0q, p64, 1q, p98, 0qu. The

accuracy and average response time are investigated

for window sizes ω P t1, 2, 4, 8, 16u and threshold τ P

t0.2, 0.4, 0.6, 0.8u with the true mode changes M “

tp1, 0q, p64, 1q, p98, 0qu.

14 Sida Chen et al.

Results From the metric described in Section 6.1.2 with

δ = 5 for response time, the best results, accuracy =

94.4%, max/min/average response times = 5/0/1.67

seconds, are observed when ω “ 1 and τ “ 0.8. The

transitions of the corrected speed and detected modes

that show the best accuracy with ω “ 1 and τ “ 0.8

are shown in Figure 18. Looking at Figure 18(b), the

pitot tube failure is successfully detected from 70 to 96

seconds except for the interval 63 to 69 seconds due to

the slowly decreasing airspeed. The response time for

the normal to pitot tube failure mode is 5 seconds and

for the pitot tube failure to normal mode is 0 seconds

(thus the average response time is 1.67 seconds). From

Figure 18(a), the airspeed successfully starts to get cor-

rected at 70 seconds and seamlessly transitions to the

normal airspeed when it recovers at 99 seconds.

��

�

��

��

��

��

��

��

� ��

�

�

�

�

�

�

�

�

�

�

�

�

�

�

�

�

�

��

��

��

��

��

��

��

��

�

�

�

�

� ��

�

�

�

��

��

��

��

��

��

� ��

Fig. 18 Corrected airspeed and detected modes for AF447flight.

7.2 Estimation by dynamic statistical model

7.2.1 Offline training of dynamic Bayes classifier

We create a training dataset for three error modes (i.e.,

normal, pitot tube failure, GPS failure) by simulation

using the parameters as shown in Table 6. Note that

all the speeds are relative to airspeed va and all the

angles are in degrees. N pµ, σq is a normal distribu-

tion with mean µ and standard deviation σ. Upa, bq is

a uniform distribution that randomly choose a num-

ber between ra, bs. ε is a Gaussian measurement noise.

fpva, αa, vw, αwq gives ground speed vg such that the

error defined in Equation (17) becomes zero. Similarly,

gpva, αa, vw, αwq gives ground speed angle αg such that

an error defined for angles becomes zero.

Table 6 Simulation parameters to generate speed estimationmodel training data.

ParametersError modes

NormalPitot tube

GPS failurefailure

Number of samples 10,000 10,000 10,000va: airspeed 1.0` ε „ N p0.5, 0.067q 1.0` εαa: airspeed angle 45` εvw: wind speed 0.1` εαw: wind speed angle „ Up0, 360qvg : ground speed fpva, αa, vw, αwq ` ε 0.0 + εαg: ground speed angle gpva, αa, vw, αwq ` ε

Similar to what we define the error signatures set for

speed data in Section 7.1.2, we make some assumptions

on how pitot tube and GPS failures occur. In the case

of pitot tube failure, we assume the pitot tube clearance

ratio b has some randomness and follows the normal dis-

tribution N p0.5, 0.067q. We define failed airspeed va “

b ¨ va, but since va “ 1.0, va follows N p0.5, 0.067q. In

the case of GPS failure, the ground speed drops to zero

with a small measurement error ε.

Offline training of the dynamic Bayes classifier is

performed with training parameters in Figure 19, where

the feature is chosen to be the difference between mea-

sured ground speed and estimated ground speed calcu-

lated using other variables.

'

&

$

%

estimator bayes_speedcheck;/* vw: wind speed, aw: wind speed angle

va: airspeed, aa: airspeed anglevg: ground speed, ag: ground angle,mode: the error mode

*/datavw, va, vg, aa,ag, aw, mode using file( training.csv );

modelfeature: vg - sqrt(vaˆ2 + vwˆ2+ 2*va*vw*cos((PI/180)*(aw-aa)));

label: mode;algorithm: BayesianClassifier usingDynamicBayesClassifier(sigma_scale:2, threshold:200);

end

Fig. 19 Offline training parameters for the dynamic Bayesclassifier.

7.2.2 Online error detection and correction using

dynamic Bayes classifier

PILOTS program Figure 20 shows the PILOTS pro-

gram using dynamic Bayes classifier to classify Normal,

Pitot tube failure and GPS failure and gives estima-

tion of airspeed in Pitot tube failure mode and ground


speed in GPS failure mode. The program outputs the

airspeed or estimated airspeed depending on mode.

'

&

$

%

program SpeedCheckBayes;/* vw: wind speed, aw: wind speed angle

va: airspeed, aa: airspeed anglevg: ground speed, ag: ground angle */

inputsva, vg, vw (t) using closest(t);aa, ag, aw (t) using closest(t);mode (t) using predict(bayes_speedcheck,

va, vg, vw, aw, aa);outputsva_out: va at every 1 sec;

modesm0: mode = 0 "Normal";m1: mode = 1 "Pitot tube failure"

estimate va =sqrt(vg*vg + vw*vw- 2*vg*vw*cos((PI/180)*(ag-aw)));

m2: mode = 2 "GPS failure"estimate vg =

sqrt(va*va + vw*vw+ 2*va*vw*cos((PI/180)*(aw-aa)));

end

Fig. 20 Declarative specification of the SpeedCheckBayesPILOTS program.

Experimental settings The same dataset used in error

signature experiment in Section 7.1.2 is utilized as test-

ing set for PILOTS program described in Figure 20

with trained dynamic Bayes classifier in Section 7.2.1.

For dynamic Bayes classifier, we set the threshold of

creating new mode as 2σ and threshold of converting

minor mode to major mode as 200. Table 7 shows the

major modes contained in the dynamic Bayes classifier.

Table 7 Major modes of dynamic Bayes classifier aftertrained by training data.

Mode description Mode µ σ nNormal 0 0.11 14.19 10000

Pitot tube failure 1 233.61 39.50 10000GPS failure 2 -471.14 39.78 10000

Results Using Section 6.1.2 as metric with delta = 5 for

response time, the accuracy is 91.0%, max/min/average

response times is 11{0{3.67 seconds. Since the dynamic

Bayes classifier changes its states as new data comes in,

the updated states are described in Table 8. Because the

training set is large and only the data points residing

inside 2σ of a mode are capable of changing attributes of

the mode, for major modes with id 0 to 2, the standard

deviation and mean stay nearly the same. The classifier

discovers new modes with id 3 to 5, which may suggest

some unconsidered cases are not covered in the training

dataset or the threshold of creating new mode is low

and results in noises and outliers to become new modes.

Since the threshold of converting minor mode to major

mode is 200, no new major modes are created.

The measured and estimated airspeed is illustrated

in Figure 21(a) and the detected modes are shown in

Figure 21(b). Pitot tube failure is captured and the

airspeed is estimated during 70, 75 to 97 seconds but

not 64 to 69, 71 to 74 seconds since the error func-

tion gives numbers greater than 2σ of every states in

trained dynamic Bayes classifier, and being classified

to new modes. The classifier successfully detects end of

pitot tube failure immediately under the resolution of 1

second. Interestingly, at 124 second, the classifier gives

mode 3, which is created at 62 second.

Table 8 Major and minor modes of dynamic Bayes classifierafter running testing data.

Modestatus

Modedescription

Mode µ σ n

MajorNormal 0 -0.07 14.0 10130

Pitot tube failure 1 233.62 39.50 10024GPS failure 2 -471.14 39.78 10000

MinorNew mode 3 -38.26 9.59 5New mode 4 104.01 8.80 2New mode 5 332.25 14.30 5

Fig. 21 Speed error mode detection and correction usingdynamic Bayes classifier.

16 Sida Chen et al.

7.3 Summary of results

The max/min/average response times for both dynamic

Bayes classifier and error signatures method for PI-

LOTS program are identical. However, since the dy-

namic Bayes classifier contains newly created modes

that have no corresponding estimation function pro-

vided, the failure mode accuracy is less than the one

produced by error signature, and thus, the estimation

of airspeed is less complete than that computed by error

signature. The dynamic Bayes classifier discovers new

modes, which could be attached with new estimation

function, and requires offline training, while the error

signatures are not able to change dynamically but have

no requirement of training.

8 Related work

Stream data processing has been an important tech-

nique in flight safety systems. Fault detection, isolation,

and reconfiguration (FDIR) has also been actively stud-

ied in the control community [16]. The FDIR systems

evaluate a set of residuals (what we call error func-

tions) to detect if a fault has occurred, then isolate the

type of the fault, and reconfigure the system to recover

from the fault. To alleviate the effect of noise on resid-

uals, robust residual generation techniques, such as a

Kalman Filter based approach [25], have been used.

Error residuals from different sources including ground

speed, estimated wind speed, and propeller speed, are

used to successfully detect and isolate airspeed sensor

faults [15]. In some study, the residuals are not con-

sidered as binary value, but are assumed to have dif-

ferent distributions according to different modes [30].

The false positive and false negative detection rate of

a FDIR method can be evaluated by several statistical

models [34,24]. The PILOTS language was designed for

spatio-temporal data stream filtering, error detection

and correction. PILOTS has been shown to detect and

recover from sensor errors using actual flight data from

commercial accidents [17]. The PILOTS framework en-

ables users to implement fault detection and correction

with tens of lines of code to describe error conditions.

There have been many systems that combine data

stream processing and data base management, i.e., Data

Stream Management Systems (DSMS). PLACE [26] and

Microsoft StreamInsight [2] are DSMS-based systems

supporting spatio-temporal streams. Also, the concept

of the moving object data base (MODB) which adds

support for spatio-temporal data streaming to DSMS

is discussed in [4]. Also, a DSMS-based traffic conges-

tion estimation system has been proposed [14]. These

DSMS-based spatio-temporal stream management sys-

tems support general continuous queries for multiple

moving objects such as “Find all the cars running within

a diameter of X from a point Y in the past Z time”. Un-

like these DSMS-based systems which handle multiple

spatio-temporal objects, a PILOTS program is assumed

to be moving and tries to extrapolate data that is rel-

evant to the current location and time. This approach

narrows down the applicability of PILOTS; however,

users can more easily design error signatures to estimate

data on the fly thanks to the declarative programming

approach.

In the context of big data processing, distributed,

scalable, and fault-tolerant data streaming systems have

been widely used. Such systems include MillWheel [1],

Storm [32], and Spark Streaming [33]. Since these sys-

tems are expected to run over many computer nodes,

they are designed to continue producing correct results

with reasonably degraded performance even in the case

of node failures. Unlike PILOTS, they are not aware

of application-level data failures. On the other hand,

PILOTS itself does not have any fault-tolerance mech-

anism to node failures.

Machine learning techniques have been widely used

in stream data processing. There is a multi-dimensional

regression method for time-series data streams [10], and

a regression-based temporal pattern mining scheme for

data streams [31]. Neural networks have been applied

for supervised real-time learning and classification [8],

and unsupervised active mining methods could be used

to estimate the error of the model on new data streams

[13]. In this paper, we extend PILOTS to support lin-

ear regression of stream data, and also combined of-

fline supervised learning and dynamic online incremen-

tal learning as implemented by the dynamic Bayes clas-

sifier.

9 Discussion and conclusion

In this paper, we investigated both static regression-

based model and dynamic Bayesian learning approach

to data-driven fault detection in avionics. We also de-

signed changes to the PILOTS programming language

and runtime system to support these machine learn-

ing techniques. We applied a linear regression approach

to learn the relationship between coefficient of lift and

the angle of attack during flights. With the training

results, and Equation (9) to calculate airplane weight

during cruise phase, the PILOTS program successfully

detects and corrects underweight and overweight condi-

tions in simulated flight data by using error signatures.

In practice, error signatures need to be complemented


with error residuals methods for weight data error iso-

lation [16]. Using dynamic Bayes classifier, when the

system is trained by normal and underweight data, the

PILOTS program is able to detect a new mode when an

overweight situation occurs in the online learning phase.

Error signatures require more specific patterns for de-

tecting faults, while the dynamic Bayes classifier uses

data to learn different operating modes. Dynamic Bayes

classifier detects statistically significant new modes dur-

ing the online learning phase, while error signatures can

only detect pre-defined modes.

When using the dynamic Bayes classifier to detect

weight error, we noticed that the system could not only

detect “normal” and “underweight” modes, but also

classifies “5% overweight” and “15% overweight” as two

different modes as shown in Figure 22. This informa-

tion is useful if different strategies need to be taken

for different extent of weight errors, otherwise it would

be unnecessarily misleading to classify them into differ-

ent modes. Thus, the dynamic Bayes classifier should

be adjusted to the requirements of various use cases.

This would result in a semi-supervised online learning

approach.

Fig. 22 Another weight error mode detection using dynamicBayes classifier.

Future work for the dynamic Bayes classifier in-

cludes to involve human feedback in the online learn-

ing phase, especially when a new mode is detected, to

get more accurate classification parameters and deci-

sion making. Our dynamic Bayes classifier produces re-

sults that depend on the order it sees new data. We

Table 9 Major and minor modes of dynamic Bayes classifierafter running testing data.

Modestatus

Modedescription

Mode µ σ n

MajorNormal 0 -33 2462 6544

Underweight 1 15898 3312 1708

Minor5% Overweight 2 -6922 239 5415% Overweight 3 -17450 277 91

need to devise new techniques to make data ingestion

associative and commutative, so that classification (es-

pecially new modes) is data-order independent. Sup-

port for multiple features and technics for determin-

ing thresholds for converting minor modes into major

modes will also be explored. Machine learning tech-

niques could also be used to learn parameters in er-

ror signatures from data. Another possible direction is

to combine logic programming and probabilistic pro-

gramming, as in ProbLog [12], to help analyze spatio-

temporal stream data. Finally, uncertainty quantifica-

tion [3] is an important future direction to associate

confidence to data and error estimations in support of

decision making.

References

1. Tyler Akidau, Alex Balikov, Kaya Bekiroglu, SlavaChernyak, Josh Haberman, Reuven Lax, Sam McVeety,Daniel Mills, Paul Nordstrom, and Sam Whittle. Mill-wheel: fault-tolerant stream processing at internet scale.Proceedings of the VLDB Endowment, 6(11):1033–1044,2013.

2. Mohamed H. Ali, Badrish Chandramouli, Balan SethuRaman, and Ed Katibah. Spatio-temporal stream pro-cessing in Microsoft StreamInsight. IEEE Data Eng.Bull., pages 69–74, 2010.

3. Douglas Allaire, David Kordonowy, Marc Lecerf, LauraMainini, and Karen Willcox. Multifidelity DDDAS meth-ods with application to a self-aware aerospace vehicle. InDDDAS 2014 Workshop at ICCS’14, pages 1182–1192,June 2014.

4. Kyounghwan An and Juwan Kim. Moving objects man-agement system supporting location data stream. In Pro-ceedings of the 4th WSEAS International Conference onComputational Intelligence, Man-Machine Systems andCybernetics, CIMMACS’05, pages 99–104, Stevens Point,Wisconsin, USA, 2005. World Scientific and EngineeringAcademy and Society (WSEAS).

5. John David Anderson Jr. Fundamentals of aerodynamics.Tata McGraw-Hill Education, 2010.

6. Erik P Blasch, Dale A Lambert, Pierre Valin, Mieczys-law M Kokar, James Llinas, Subrata Das, Chee Chong,and Elisa Shahbazian. High level information fu-sion (hlif): survey of models, issues, and grand chal-lenges. IEEE Aerospace and Electronic Systems Mag-azine, 27(9):4–20, 2012.

7. Bureau d’Enquetes et d’Analyses pour la Securite del’Aviation Civile. Final Report: On the accident on 1stJune 2009 to the Airbus A330-203 registered F-GZCPoperated by Air France flight AF 447 Rio de Janeiro -Paris. Accessed 09-15-2016.

18 Sida Chen et al.

8. Gail A Carpenter, Stephen Grossberg, and John HReynolds. Artmap: Supervised real-time learning andclassification of nonstationary data by a self-organizingneural network. Neural networks, 4(5):565–588, 1991.

9. Sida Chen, Shigeru Imai, Wennan Zhu, and Carlos A.Varela. Towards learning spatio-temporal data streamrelationships for failure detection in avionics. In DynamicData-Driven Application Systems (DDDAS 2016), Hart-ford, CT, Aug 2016. To appear.

10. Yixin Chen, Guozhu Dong, Jiawei Han, Benjamin WWah, and Jianyong Wang. Multi-dimensional regressionanalysis of time-series data streams. In Proceedings ofthe 28th International Conference on Very Large DataBases, pages 323–334. VLDB Endowment, 2002.

11. Frederica Darema. Dynamic data driven applications sys-tems: A new paradigm for application simulations andmeasurements. In Computational Science-ICCS 2004,pages 662–669. Springer, 2004.

12. Luc De Raedt, Angelika Kimmig, and Hannu Toivonen.Problog: A probabilistic prolog and its application in linkdiscovery. In International Joint Conference on ArtificialIntelligence, volume 7, pages 2462–2467, 2007.

13. Wei Fan, Yi-an Huang, Haixun Wang, and S Yu Philip.Active mining of data streams. In Proceedings of the 2004SIAM International Conference on Data Mining, pages457–461. SIAM, 2004.

14. Sandra Geisler, Christoph Quix, Stefan Schiffer, andMatthias Jarke. An evaluation framework for traffic in-formation systems based on data streams. Transporta-tion Research Part C: Emerging Technologies, 23:29–55,2012.

15. Soren Hansen and Mogens Blanke. Diagnosis of air-speed measurement faults for unmanned aerial vehicles.IEEE Transactions on Aerospace and Electronic Sys-tems, 50(1):224–239, January 2014.

16. Inseok Hwang, Sungwan Kim, Youdan Kim, andChze Eng Seah. A survey of fault detection, isolation,and reconfiguration methods. Control Systems Technol-ogy, IEEE Transactions on, 18(3):636–653, 2010.

17. Shigeru Imai, Alessandro Galli, and Carlos A. Varela.Dynamic data-driven avionics systems: Inferring failuremodes from data streams. In Dynamic Data-Driven Ap-plication Systems (DDDAS 2015), Reykjavik, Iceland,June 2015.

18. Shigeru Imai, Richard Klockowski, and Carlos A. Varela.Self-healing spatio-temporal data streams using error sig-natures. In 2nd International Conference on Big DataScience and Engineering (BDSE 2013), Sydney, Aus-tralia, December 2013.

19. Shigeru Imai and Carlos A. Varela. A programmingmodel for spatio-temporal data streaming applications.In Dynamic Data-Driven Application Systems (DDDAS2012), pages 1139–1148, Omaha, Nebraska, June 2012.

20. Shigeru Imai and Carlos A. Varela. Programming spatio-temporal data streaming applications with high-levelspecifications. In 3rd ACM SIGSPATIAL InternationalWorkshop on Querying and Mining Uncertain Spatio-Temporal Data (QUeST) 2012, Redondo Beach, Califor-nia, USA, November 2012.

21. Edwin T Jaynes. Probability theory: The logic of science.Cambridge university press, 2003.

22. George H John and Pat Langley. Estimating continuousdistributions in Bayesian classifiers. In Proceedings ofthe Eleventh conference on Uncertainty in artificial in-telligence, pages 338–345. Morgan Kaufmann PublishersInc., 1995.

23. Richard S. Klockowski, Shigeru Imai, Colin Rice, andCarlos A. Varela. Autonomous data error detection andrecovery in streaming applications. In Proceedings ofthe International Conference on Computational Science(ICCS 2013). Dynamic Data-Driven Application Sys-tems (DDDAS 2013) Workshop, pages 2036–2045, May2013.

24. J. Marzat, H. Piet-Lahanier, F. Damongeot, and E. Wal-ter. Model-based fault diagnosis for aerospace systems:a survey. Proceedings of the Institution of MechanicalEngineers, Part G: Journal of Aerospace Engineering,226(10):1329–1360, January 2012.

25. T.E. Menke and P.S. Maybeck. Sensor/actuator failuredetection in the Vista F-16 by multiple model adaptiveestimation. IEEE Transactions on Aerospace and Elec-tronic Systems, 31(4):1218–1229, oct 1995.

26. Mohamed F. Mokbel, Xiaopeng Xiong, Walid G. Aref,and Moustafa a. Hammad. Continuous query processingof spatio-temporal data streams in PLACE. GeoInfor-matica, 9:343–365, 2005.

27. J Tinsley Oden, Ernesto E Prudencio, and Paul T Bau-man. Virtual model validation of complex multiscalesystems: Applications to nonlinear elastostatics. Com-puter Methods in Applied Mechanics and Engineering,266:162–184, 2013.

28. Agenzia Nazionale per la Sicurezza del Volo. Final Re-port: Accident involving ATR 72 aircraft registrationmarks TS-LBB ditching off the coast of Capo Gallo(Palermo - Sicily), August 6th, 2005. Accessed 03-31-2015.

29. Irina Rish. An empirical study of the naive Bayes clas-sifier. In IJCAI 2001 workshop on empirical methods inartificial intelligence, volume 3, pages 41–46. IBM NewYork, 2001.

30. C. Svard, M. Nyberg, E. Frisk, and M. Krysander. Data-driven and adaptive statistical residual evaluation forfault detection with an automotive application. Mechan-ical Systems and Signal Processing, 45(1):170–192, 2014.

31. Wei-Guang Teng, Ming-Syan Chen, and Philip S Yu.A regression-based temporal pattern mining scheme fordata streams. In Proceedings of the 29th InternationalConference on Very large data bases-Volume 29, pages93–104. VLDB Endowment, 2003.

32. The Apache Software Foundation. Apache Storm, Jan-uary 2015. Accessed 09-15-2016.

33. Matei Zaharia, Tathagata Das, Haoyuan Li, ScottShenker, and Ion Stoica. Discretized streams: an effi-cient and fault-tolerant model for stream processing onlarge clusters. In Proceedings of the 4th USENIX confer-ence on Hot Topics in Cloud Computing, pages 10–10.USENIX Association, 2012.

34. Ali Zolghadri. Advanced model-based FDIR techniquesfor aerospace systems: Today challenges and opportuni-ties. Progress in Aerospace Sciences, 53:18–29, August2012.

learning fault patterns from data in streaming applications

Documents