aerodynamic uncertainty quanti cation and estimation of

Aerodynamic Uncertainty Quanti�cation and Estimation of Uncertainty

Quanti�ed Performance of Unmanned Aircraft Using Non-Deterministic

Simulations

Lawrence E. Hale

Dissertation submitted to the Faculty of the

Virginia Polytechnic Institute and State University

in partial ful�llment of the requirements for the degree of

Doctor of Philosophy

in

Aerospace Engineering

Mayuresh Patil, Co-Chair

Christopher J Roy, Co-Chair

Chraig A. Woolsey

Mazen Farhood

Dec. 1, 2016

Blacksburg, Virginia

Keywords: Uncertainty Quanti�cation, Non-Deterministic Simulations, Unmanned Aircraft

Copyright 2016, Lawrence E. Hale

Aerodynamic Uncertainty Quanti�cation and Estimation of Uncertainty Quanti�ed

Performance of Unmanned Aircraft Using Non-Deterministic Simulations

Lawrence E. Hale

Abstract

This dissertation addresses model form uncertainty quanti�cation, non-deterministic simulations, and sen-

sitivity analysis of the results of these simulations, with a focus on application to analysis of unmanned

aircraft systems. The model form uncertainty quanti�cation utilizes equation error to estimate the error

between an identi�ed model and �ight test results. The errors are then related to aircraft states, and predic-

tion intervals are calculated. This method for model form uncertainty quanti�cation results in uncertainty

bounds that vary with the aircraft state, narrower where consistent information has been collected and wider

where data are not available. Non-deterministic simulations can then be performed to provide uncertainty

quanti�ed estimates of the system performance. The model form uncertainties could be time varying, so

multiple sampling methods were considered. The two methods utilized were a �xed uncertainty level and a

rate bounded variation in the uncertainty level. For analysis using �xed uncertainty level, the corner points

of the model form uncertainty were sampled, providing reduced computational time. The second model

better represents the uncertainty but requires signi�cantly more simulations to sample the uncertainty. The

uncertainty quanti�ed performance estimates are compared to estimates based on �ight tests to check the

accuracy of the results.

Sensitivity analysis is performed on the uncertainty quanti�ed performance estimates to provide information

on which of the model form uncertainties contribute most to the uncertainty in the performance estimates.

The proposed method uses the results from the �xed uncertainty level analysis that utilizes the corner points

of the model form uncertainties. The sensitivity of each parameter is estimated based on corner values of all

the other uncertain parameters. This results in a range of possible sensitivities for each parameter dependent

on the true value of the other parameters.

Aerodynamic Uncertainty Quanti�cation and Estimation of Uncertainty Quanti�ed

Performance of Unmanned Aircraft Using Non-Deterministic Simulations

Lawrence E. Hale

General Audience Abstract

This dissertation examines a process that can be utilized to quantify the uncertainty associated with an

identi�ed model, the performance of the system accounting for the uncertainty, and the sensitivity of the

performance estimates to the various uncertainties. This uncertainty is present in the identi�ed model

because of modeling errors and will tend to increase as the states move away from locations where data has

been collected. The method used in this paper to quantify the uncertainty attempts to represent this in a

qualitatively correct sense. The uncertainties provide information that is used to predict the performance

of the aircraft. A number of simulations are performed, with di�erent values for the uncertain terms chosen

for each simulation. This provides a family of possible results to be produced. The uncertainties can be

sampled in various manners, and in this study were sampled at �xed levels and at time varying levels. The

sampling of �xed uncertainty level required fewer samples, improving computational requirements. Sampling

with time varying uncertainty better captures the nature of the uncertainty but requires signi�cantly more

simulations. The results provide a range of the expected performance based on the uncertainty.

Sensitivity analysis is performed to determine which of the input uncertainties produce the greatest un-

certainty in the performance estimates. To account for the uncertainty in the true parameter values, the

sensitivity is predicted for a number of possible values of the uncertain parameters. This results in a range

of possible sensitivities for each parameter dependent on the true value of the other parameters. The range

of sensitivities can be utilized to determine the future testing to be performed.

Acknowledgment

This work received support from the ICTAS at Virginia Tech. Additional funding was provided by Mr.

Robert J. Hanley, Director, USN & USMC Airworthiness and CYBERSAFE O�ce, AIR-4.0P. This research

is funded by prime contract #HC1047-05-D-4005, purchase order APSC01797 (Clearance statement TBD).

iv

Contents

1 Introduction 1

2 Aerodynamic Parameter Identi�cation and Uncertainty Quanti�cation for Small Un-

manned Aircraft 4

2.1 Title . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.3 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.4 Parameter Estimation and Uncertainty Quanti�cation . . . . . . . . . . . . . . . . . . . . . . 7

2.4.1 Aerodynamic Parameter Identi�cation . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.4.2 Aerodynamic Parameter Uncertainty Quanti�cation . . . . . . . . . . . . . . . . . . . 8

2.5 Model Form Uncertainty Quanti�cation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.6 Example Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.6.1 Synthetic Aircraft Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.6.2 Parameter Estimation and Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . 14

v

2.6.3 Estimation of Model Form Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.7 E-SPAARO Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.7.1 Parameter Estimation and Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.7.2 Estimation of Model Form Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.7.3 Validation of Model Form Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3 Estimation of Uncertainty Quanti�ed Performance of Unmanned Aircraft Using Non-

Deterministic Simulations 37

3.1 Title . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.2 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.3 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.4 Parameter Estimation and Uncertainty Quanti�cation . . . . . . . . . . . . . . . . . . . . . . 40

3.4.1 Aerodynamic Parameter Identi�cation . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.4.2 Aerodynamic Parameter Uncertainty Quanti�cation . . . . . . . . . . . . . . . . . . . 41

3.5 E-SPAARO Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.6 Model Form Uncertainty Quanti�cation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.7 Uncertainty Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.8 Nondeterministic Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.9 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

vi

4 Sensitivity of Input Epistemic Uncertainty on Uncertainty Quanti�ed Performance Es-

timates Using Non-Deterministic Simulations 64

4.1 Title . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.2 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4.3 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4.4 Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

4.4.1 Estimation of Output Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.5 Existing Sensitivity Analysis Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.5.1 Local Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.5.2 Scatter Plot Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.5.3 Variance Based Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4.5.4 p-box Based Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4.5.5 Comparison of Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.6 Proposed Sensitivity Analysis Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.7 Example Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.7.1 Sensitivity of Aircraft Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.7.2 Sensitivity of Nondeterministic Performance Estimates . . . . . . . . . . . . . . . . . . 82

4.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5 Conclusions 92

vii

Bibliography 94

viii

List of Figures

2.1 Comparison of Simulated Results Using the True Synthetic Model (Blue Lines) and Identi�ed

Model (Red Lines) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2 Validation of Parameter Estimates from Synthetic Data . . . . . . . . . . . . . . . . . . . . . 18

2.3 Synthetic Data, Model Form Uncertainty Bounds CL (2nd order) . . . . . . . . . . . . . . . . 19

2.4 Synthetic Data, Model Form Uncertainty Bounds CL (2nd order) . . . . . . . . . . . . . . . . 19

2.5 Tuned Simulation vs. Experimental Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.6 Experimental Parameter Identi�cation Validation . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.7 Experimental Data, Model Form Uncertainty Bounds CZ . . . . . . . . . . . . . . . . . . . . 23

2.8 Experimental Data, Model Form Uncertainty Bounds CX . . . . . . . . . . . . . . . . . . . . 24

2.9 Experimental Data, Model Form Uncertainty Bounds Cm . . . . . . . . . . . . . . . . . . . . 24

2.10 Experimental Data, Model Form Uncertainty Bounds Cm (Blue Baseline, Red With Roll Rate) 25

2.11 Experimental Data, Model Form Uncertainty Bounds Cy . . . . . . . . . . . . . . . . . . . . . 25

2.12 Experimental Data, Model Form Uncertainty Bounds Cl . . . . . . . . . . . . . . . . . . . . . 26

2.13 Experimental Data, Model Form Uncertainty Bounds Cn . . . . . . . . . . . . . . . . . . . . 26

ix

3.1 E-SPAARO UAV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.2 Example Prediction Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.3 Probability Density Function of δCZ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

3.4 p-box representing aleatory and epistemic uncertainties . . . . . . . . . . . . . . . . . . . . . 49

3.5 Simulated performance CDF convergence check with 500 hours. . . . . . . . . . . . . . . . . . 50

3.6 Experimental loiter performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.7 Simulated performance (3 m/s steady wind, medium turbulence) . . . . . . . . . . . . . . . . 52

3.8 Simulated performance (3 m/s steady wind, medium turbulence) with sensor noise added to

recorded simulation measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3.9 Simulated performance (3 m/s steady wind, severe turbulence) with sensor noise added to


3.10 Time varying model form uncertainty (3 m/s steady wind, severe turbulence) with sensor

noise added to recorded simulation measurements . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.1 p-box Representing Aleatory and Epistemic Uncertainties . . . . . . . . . . . . . . . . . . . . 67

4.2 Diagram Illustrating P-box Based Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4.3 Simulated performance (3 m/s steady wind, sever turbulence) with sensor noise added to


4.4 E-SPAARO UAV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

x

List of Tables

2.1 Measurement Noise Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2 Mass and Size Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.3 Validation of Model Form Uncertainty Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.4 Synthetic Model Parameter Estimation Results (SL model) . . . . . . . . . . . . . . . . . . . 29

2.5 Synthetic Model Parameter Estimation Results (SLM model) . . . . . . . . . . . . . . . . . . 30

2.6 E-SPAARO Parameter Estimation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.1 Number of path excursions per hour from nondeterministic simulations (25 m radius limit) . 55

3.2 Number of path excursions per hour from nondeterministic simulations (15 m radius limit) . 56

4.1 Comparison of Number of Required Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.2 Aleatory Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.3 Epistemic Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.4 Phugiod Damping Sensitivity results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

4.5 Short Period Damping Sensitivity results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

xi

4.6 Short Period Natural Frequency Sensitivity Results . . . . . . . . . . . . . . . . . . . . . . . . 81

4.7 Normalized Output Uncertainty from Input Uncertainty (Maximum and Minimum) . . . . . 84

xii

Chapter 1

Introduction

Performance estimation of unmanned vehicles involves the quanti�cation of the response of the vehicle to

disturbances. For unmanned vehicles there tends to be uncertainty in the identi�ed models. The performance

estimates should re�ect this uncertainty in the estimates as well. The work presented in this paper uses

parameter identi�cation methods and estimates the model form uncertainties associate with the estimates.

These uncertainties are then sampled producing a family of simulations that provide information on the

uncertainty of the performance estimates.

The estimation and analysis of parametric uncertainties has been examined using a variety of approaches

in the literature. [1, 2]. Some information on uncertainty quanti�cation in general computational models is

presented by Liang and Mahadevan[3]. The methods for estimation of model form uncertainty focus on the

comparison of experimental values to the predictions of a candidate model. The method described in this

paper follows this general form, and proposes a method utilizing information form the veri�cation process

for use on aircraft systems. Previous model form uncertainty predictions for aircraft have primarily focused

on aeroelastic analyses, often assuming a probability distribution for the uncertainty [4, 5]. These methods

are su�cient if the �ight conditions obey a probability distribution. However, as the aircraft deviates from

1

Lawrence E. Hale Chapter 4.3. Introduction 2

the �ight conditions used to estimate the uncertainty, there is no guarantee that the �ight condition will

obey a probability distribution. Polynomial chaos could be utilized to propagate uncertainties through the

system, however the formulation of polynomial chaos results in a degradation of the estimates over time [6].

The degradation of the estimates is due to the �nite dimensional approximation of the probability space.

Some current work is also underway by a team from Boeing and NASA Langley [7]. This work is part of the

NASA Vehicle Systems Safety Technologies project Technical Challenge #3. The paper discusses the goals

of the project, and describes the current static and dynamic identi�cation testing that has been performed.

UAS performance measures are still in development. Performance measures based on the Cooper-Harper

Piloted Rating scale have been examined in Ref. [8]. These methods relate the UAV performance measures

to manned aircraft performance measures, however as UAS are mostly �own autonomously performance

limits may be better associated with the payload requirements. The method proposed in Ref. [8] allows for

the performance measure of interest to be adjusted to the requirements of a variety of payloads.

The performance limits can also be used to predict if the system will enter a loss of control state. These limits

could be used to de�ne operational limits on the UAV. The acceptable number of loss of control events per

�ight hour is dependent both on the size of the aircraft and the operational area of the vehicle. Estimates of

the required mean time between failures for di�erent classes of vehicles is presented in [9]. Based on required

safety levels, nondeterministic simulations could be utilized to provide estimates the limits on the conditions

that are safe to �y in for tactical UAVs over larger portions of the country.

The overview of the dissertation is as follows. In Chapter 2, the methods for parameter identi�cation and

uncertainty quanti�cation are presented. The parameter identi�cation method presented is the output error

method. Model form uncertainty is estimated through a modi�ed equation error technique combined with

least squares estimation and prediction intervals. The parameter identi�cation and model form uncertainty

estimations are performed for a simulated �ight using synthetic �ight test data and for �ight test data.

The method provides information on portions of the �ight that match the identi�ed model, and insight into

possible terms to be added to the model if less uncertainty is desired.

Lawrence E. Hale Chapter 4.3. Introduction 3

In Chapter 3, nondeterministic simulations are utilized to provide estimates of uncertainty quanti�ed per-

formance. Nondeterministic simulations are performed based on the identi�ed model and model form un-

certainty bounds estimated in the previous chapter. The model form uncertainty bounds are time varying

and the distance between the bounds is associated with the system states. Two possible methods to sample

the model form uncertainty are presented. Sampling with a �xed uncertainty level, and with a time varying

level based on the rate of change of the uncertainty utilized. Performance estimates are made from a family

of simulations for each of theses sampling methods for an example �xed wing aircraft. The simulated per-

formance is compared to experimental results. The results of the nondeterministic simulations are also used

to provide information about which model form uncertainties have the largest impact on the uncertainty in

the estimated performance.

In Chapter 4, proposed a method for sensitivity analysis of uncertainty quanti�ed performance estimates from

nondeterministic simulations. A number of current sensitivity analysis methods are examined, however none

are ideally suited to this application. A new proposed method is presented based on the p-box method, but

better aligned with the format of the uncertainty quanti�ed performance estimates. The proposed method

does not require additional simulations to be performed beyond those required to obtain the uncertainty

quanti�ed performance estimates. The proposed method provides a range of possible sensitivities for each

parameter depending on the value of the other parameters.

Chapter 2

Aerodynamic Parameter Identi�cation

and Uncertainty Quanti�cation for Small

Unmanned Aircraft

2.1 Title

Aerodynamic Parameter Identi�cation and Uncertainty Quanti�cation for Small Unmanned Aircraft

By: Lawrence E. Hale, Mayuresh Patil, and Christopher J Roy

2.2 Abstract

The paper presents a method to identify aerodynamic parameters from �ight test data, determine the

uncertainty in the identi�ed parameters, and estimate the model form uncertainties in the aerodynamic

4

Lawrence E. Hale Chapter 2. Uncertainty Quanti�cation 5

model. This information is required for simulations designed to estimate the reliability of the system.

Output error method is used in parameter estimation, and uncertainty in these estimates is based on the

Cramér-Rao bounds. Validation data are then used to compare the values of the aerodynamic coe�cients as

predicted by the model to experimental data. These data are used to estimate the model form uncertainty,

which is represented as prediction intervals. These methods are utilized to analyze both synthetic and true

�ight test data. The results validate the proposed method to estimate the aerodynamic parameters and

the associated model form uncertainties. The model form uncertainties can be directly used to predict the

uncertainty in the aircraft performance and probability of loss of control.

2.3 Introduction

There has been a major push recently to integrate unmanned aircraft systems (UAS) into the national

airspace[10, 9]. The certi�cation process for UAS is still being developed and di�erent classes of UAS will

likely face di�erent certi�cation requirements [11, 9]. To help provide uncertainty quanti�ed information on

the performance of proposed designs, there is a desire to perform nondeterministic simulations of aircraft

�ights. Nondeterministic simulations are comprised of multiple simulations where uncertain parameters

are sampled with di�erent values in each simulations. By performing these nondeterministic simulations

in diverse atmospheric conditions, information can be provided that could aid in the certi�cation process.

Regardless of the �nal requirements, some assessment of the performance/risk will be required. Families of

simulations sampling model form uncertainties can provide nondeterministic information on how the aircraft

will react in adverse conditions allowing for the performance/risk to be quanti�ed. In order to understand the

uncertainty in these predictions, the aerodynamic parameters must be properly identi�ed, their uncertainty

quanti�ed, and the uncertainty in the resulting aerodynamic model estimated.

A framework for the inclusion of model-form uncertainty is provided by Oberkampf et al.[12]. The estima-

tion and analysis of parametric uncertainties has been examined by Paw and Balas [1] and Nannapaneni and


Mahadevan [2]. Paw and Balas utilized a LFT model and simpli�ed uncertainty models to develop robust

controllers. Nannapaneni and Mahadevan proposed a framework including aleatory and epistemic uncer-

tainty in reliability estimation applied to a aeroelastic application. Uncertainty quanti�cation in general

computational models is presented by Liang and Mahadevan[3]. The methods for estimation of model form

uncertainty focus on the comparison of experimental values to the predictions of a candidate model. The

method described in this paper follows this general form, and proposes a speci�c method for use on aircraft

systems. Previous model form uncertainty predictions for aircraft have primarily focused on aeroelastic anal-

yses, often assuming a probability distribution for the uncertainty [4, 5]. These methods are su�cient if the

uncertainties obey a probability distribution relative to the aircraft states. However, as the aircraft deviates

from the aircraft states used to estimate the uncertainty there is no guarantee that the uncertainties will fall

within the probability distribution. Polynomial chaos could be utilized to propagate uncertainties through

the system, however the formulation of polynomial chaos results in a degradation of the estimates over

time [6]. The degradation of the estimates is due to the �nite dimensional approximation of the probability

space. A team from Boeing Company and NASA Langley Research Center is working to perform uncertainty

quanti�cation and propagation as part of the uncertainty quanti�cation challenge of the NASA Vehicle Sys-

tems Safety Technologies project [7]. The work is expanding NASA's GTM dynamic derivative database to

provide additional information in the characterization of aircraft responses in o�-nominal conditions.

Section 3.4 presents a method for estimation of the aerodynamic parameters for an aircraft model along

with their associated uncertainty. Methods for estimating model form uncertainty are discussed in Section

2.5. The proposed methods are formulated to account for our lack of knowledge in cases where experimental

data is not available. An example analysis is performed using synthetic data to demonstrate the process in

section 2.6. The analysis is then applied to the Virginia Tech E-SPAARO unmanned aircraft[13] in section

2.7.


2.4 Parameter Estimation and Uncertainty Quanti�cation

2.4.1 Aerodynamic Parameter Identi�cation

Aerodynamic parameter identi�cation is performed through the use of system identi�cation. A number of

methods could be used when performing identi�cation of aircraft parameters as shown in previous detailed

studies [14, 15, 16, 17]. For aircraft, output error method is often utilized for parameter identi�cation. This

method provides parameter estimates that approach a best estimate of the value (the best estimate is the

true value if the assumed model structure is correct) as the amount of data increases.

For the general formulation of the output error method, the aircraft dynamic model is of the form

x(t) = f [x(t), u(t),Θ], x(t0) = x0 (2.1)

y(t) = g[x(t), u(t),Θ] (2.2)

z(tk) = y(tk) + v(tk), k = 1, . . . N (2.3)

where Θ is the vector of unknown parameters, x is state vector and u is the vector of control inputs. f and

g represent the model dynamics and observer transformation, respectively. The measured data z is provided

at N discrete times and in a perfect model represents the observed output y plus some measurement noise v.

For the output-error formulation, the noise is assumed to be a sequence of independent zero mean Gaussian

random numbers.

The output error method does not account for process noise which can include wind, steady as well as

turbulent. Extensions to the output error method that can better account for process noise are detailed in

work by Klein and Morelli [18, 19]. Output error method is acceptable in the present work as �ight test data

is generated on calm, low-turbulence days. Furthermore the primary focus of the present work is on model

form uncertainty estimation which works with any system identi�cation method.


The estimates of Θ are found by minimizing the cost function [15]:

J =1

2

N∑k=1

[z(tk)− y(tk)]TR−1[z(tk)− y(tk)] (2.4)

where R is the prediction error covariance matrix. The minimum of the cost function is found by using

Gauss-Newton method which updates the parameters by:

Θ(i+1) = Θi + ∆Θ (2.5)

∆Θ = −M−1G (2.6)

The information matrix M and the gradient vector G are [14]:

M =

N∑k=1

[∂y(tk)

∂Θ

]TR−1

[∂y(tk)

∂Θ

](2.7)

G =

N∑k=1

[∂y(tk)

∂Θ

]TR−1[z(tk)− y(tk)] (2.8)

These values only require �rst order sensitivities, that here are approximated by �nite di�erence [20].

Once the parameter updating is completed we have parameter estimates that minimize the error between

the response of our proposed model and the measured response, based on our assumed model form and the

�ight test data.

2.4.2 Aerodynamic Parameter Uncertainty Quanti�cation

Uncertainty quanti�cation for the aerodynamic parameters is based on the calculation of con�dence intervals

about the parameters. When using output error the con�dence intervals are estimated based on the Cramér-

Rao lower bound [21]. The Cramér-Rao lower bound represents the minimum variance of the estimated

value. The size of the intervals can be reduced with additional or better information about the parameters.

These bounds indicate whether or not su�cient data were used in the parameter identi�cation process. Large

bounds relative to the estimated parameter would indicate that the estimation should be performed with

additional data. If additional data does not improve the bounds, the assumed model may be inadequate and

require reformulation.


The Cramér-Rao bound is calculated as

σj ≥√djj (2.9)

where σj is the minimum con�dence interval. djj is the trace of the inverse of the information matrix:

D = M−1 (2.10)

2.5 Model Form Uncertainty Quanti�cation

The parametric uncertainties calculated in the previous section provide an estimate of how accurately the

aerodynamic parameters have been identi�ed, based on the assumed model form. The approach does not

provide information on how accurately the model represents the true physics, i.e., the uncertainty due to

the assumed model form. This section proposes a method to provide estimates of theses uncertainties. The

estimates are made in a conservative manner as sensor noise is included in the estimates. The inclusion of

sensor noise results in wider, more conservative, uncertainty bounds

The residuals used in the estimation of model form uncertainty are calculated utilizing di�erent data than

that used in the estimation of the parameters. The use of distinct data additionally provides validation

of the estimated model. The bounds on the uncertainty are based on prediction intervals [22] calculated

from the residuals of the force and moment coe�cients. The residuals are the di�erence in the coe�cient

calculated from the measured linear accelerations and rotational rates and those predicted by model based

on the measured aircraft states. The linear accelerations and rates are corrected to the aircraft CG and then

multiplied by the mass to calculate the forces and the inertia to calculate the moments. The aerodynamic

coe�cients based on experimental data are then calculated from these forces and moments. The equations

used to �nd these residuals are given as:

m

.5ρV 2SaX = CX(y,Θ) +

T (y,Θ)

.5ρV 2S+ ∆CX (2.11)

m

.5ρV 2SaY = CY (y,Θ) + ∆CY (2.12)


m

.5ρV 2SaZ = CZ(y,Θ) + ∆CZ (2.13)

Ix.5ρV 2Sb

[p+

Iz − IyIx

qr − IxzIx

(pq + r)

]= Cl(y,Θ) + ∆Cl (2.14)

Iy.5ρV 2Sc

[q +

Ix − IzIy

pr +Ixz

Iy(p2 − r2)

]= Cm(y,Θ) + ∆Cm (2.15)

Iz.5ρV 2Sb

[r +

Iy − IxIz

pq +Ixy

Ix(qr − p)

]= Cn(y,Θ) + ∆Cn (2.16)

Θ are the identi�ed parameter values and y are the model states on which the aerodynamic model is

dependent. The term m is the mass of the system, and Ix, Iy, Iz, and Ixz are the inertia terms. The

identi�ed parameters and model states provide the model predicted force and moment coe�cients. The

terms p, q, r are the angular rates, and aX , aY , aZ are the linear acceleration measurements (corrected to

the CG location). The angular rates and linear accelerations, along with the airspeed and vehicle inertial

properties are used to calculate the estimates of the aerodynamic coe�cients based on �ight test data. The

terms ∆CX , ∆CY , ∆CX , ∆CY , ∆CX , and ∆CY represent the residual between the experimental data and

identi�ed model and are used to estimate the model form uncertainty.

The residual is calculated for each experimental time step and is then used to estimate the model form

uncertainty bounds. These bounds are calculated based on the prediction interval. To calculate these

intervals we �rst perform a linear regression with respect to multiple variables, to provide a curve �t over

the data. The curve �t is formulated as:

Y = β0 + β1x1 + β2x2 + · · ·+ βkxk + ε (2.17)

In this equation Y is a vector of the residuals for one of the aerodynamic coe�cients, the β terms are the

�t coe�cients and the x terms are the �ight states the �t is made over. The �ight states in this equation

are based on the implementation, and can include the vehicle states, control surface de�ections, or other

measured information. The x terms can include terms that have been calculated from the �ight states (e.g.

α2).

The selection of the model structure of the multiple linear regression can be accomplished with a variety of

methods [23]. These methods compare the importance of the proposed model states to be utilized in the


regression. The �nal states chosen will depend on the distribution of the ∆C's relative to the aircraft states.

Prediction intervals are utilized to estimate the uncertainty bounds and are calculated by:

yi ± tα2 ,n−(k+1)

√σ2(1 + xi(X ′X)−1x′i) (2.18)

where yi is the curve �t estimate at the point where the interval is cacluated (xi), tα2 ,n−(k+1) is from the

student's t-distribution, σ2 is the mean squared error, X is the design matrix formed from the �ight states

used to perform the curve �t, and xi is the point at which the intervals are calculated [24]. The value of α

in the student's t-distribution determines the percent of the data that will be within the prediction bounds.

The degrees of freedom of the student's t-distribution is the number of samples used in the estimation minus

1. Using this approach to estimate the model form uncertainty bounds allows for bounds that vary in size

as a function of the aircraft state with larger bounds as the amount of available data decreases or as the

error between model predictions and �ight data increases. The higher the order of the �ight states used in

the �t (e.g. α vs. α2 vs. α3), the more rapid the expansion of the prediction intervals as the system moves

away from the region where data is present. To ensure that the identi�ed model is included in the bounds,

the upper bound is set ≥ 0 and the lower bound is set ≤ 0.

The method presented above provides a good estimate for the uncertainty in regions where data is available,

however it is not guaranteed to capture the true model in regions with sparse data. By using the prediction

intervals and allowing the size of the intervals to increase in regions with sparse data, the lack of knowledge

about the true model is represented in a heuristic sense. The increased size of the intervals will lead to

greater uncertainty in the performance estimates in cases where the states enter regions where �ight test

data are not available.

2.6 Example Analysis

In this section the dynamics of an aircraft is studied to illustrate the model form uncertainty estimation

method presented in Section 2.5. Synthetic �ight test data based on a simulated aircraft model is generated


to perform the analysis. This synthetic data was then used to identify the parameters of a simpli�ed aircraft

model. After identifying the model parameters and associated uncertainty, the model form uncertainty was

calculated based on the method described in section 2.5.

2.6.1 Synthetic Aircraft Model

To provide synthetic �ight test data, a longitudinal aircraft model representing a wide range of �ight physics

was developed. The vehicle dynamics are described by

u =Fx + T

m− g sin θ + rv − qw (2.19)

v =Fym

+ g cos θ sinφ+ pw − ru (2.20)

w =Fzm

+ g cos θ cosφ+ qu− pv (2.21)

p =Mx + Ixz(r + qp) + qr(Iyy − Izz)

Ixx(2.22)

q =My + (Izz − Ixx)pr − Ixz(p2 − r2)

Iyy(2.23)

r =Mz + Ixz(p+ qr) + qp(Ixx − Iyy)

Izz(2.24)

In these equations the forces and velocities are in the body frame, however the aerodynamic model gives the

forces in the wind frame. The relation between these forces is:

Fx = L sinα−D cosα (2.25)

Fz = −L cosα−D sinα (2.26)

The thrust force in these equations is given by the relation

T = CTδT δT (2.27)

where CTδT is the thrust coe�cient and δT is the throttle position. For the synthetic model, the thrust

model has be simpli�ed to omit in�ow e�ects. The aerodynamic forces and moments are given by:

L =1

2ρV 2SCL (2.28)


D =1

2ρV 2SCD (2.29)

Fy =1

2ρV 2SCy (2.30)

Mx =1

2ρV 2SbCl (2.31)

My =1

2ρV 2ScCm (2.32)

Mz =1

2ρV 2SbCn (2.33)

The V in these equations is the airspeed, S is the wing area, c is the mean chord, b is the wing span, and ρ

is the air density. The aerodynamic coe�cients are assumed to be:[25]

CL = CL0+ CLαα

{1 +√X

2

}2

+ CLδe δe (2.34)

CD = Cd0 +C2L

πeAR(2.35)

Cy = Cy0 + Cyββ + Cyppb/(2V ) + Cyrrb/(2V ) + Cyδr δr (2.36)

Cl = Cl0 + Clββ + Clppb/(2V ) + Clrrb/(2V ) + Clδr δr + Clδa δa (2.37)

Cm = Cm0 + Cmαα+ Cmqqc/(2V ) + Cmδe δe (2.38)

Cn = Cn0 + Cnββ + Cnppb/(2V ) + Cnrrb/(2V ) + Cnδr δr + Cnδa δa (2.39)

The X term in the CL equation provides a quasi-steady model of dynamic stall described by [25]:

X =1

4{(1− tanh [a1(α− τ2α− α∗1])(1− tanh [a1(α∗2 − τ2α− α])} (2.40)

where a1 is the airfoil static stall characteristic, τ2 is the time constant associated with the unsteady aero-

dynamics, and α∗1/α∗2 indicate the angle of attack for 50% separation in the positive/negative α directions.

This model is used to create synthetic �ight test data. The synthetic model parameters are given in the

appendix. Gaussian white noise was added to the synthetic data to represent measurement noise in actual

�ight test data. The variances of the various noise components were selected to be similar to those of

Ardupilot autopilot system as utilized on the E-SPAARO UAS, and are given in Table 2.1.


Table 2.1: Measurement Noise Characteristics

Sensor Output V α, β θ, φ q, p, r ax, ay, az h

Standard Deviation (σ) 0.5 m/s 0.1 deg 0.5 deg 0.2 deg/s 0.16 m/s2 1.33 m

The low altitude Dryden model was used to simulate a gust �eld [26]. Light simulated wind turbulence

intensity was selected as system identi�cation �ights ideally occur on days with minimal wind gusts. The

wind model is incorporated into the �ight dynamic model through changes in the values of V , α, and β.

The mass and geometry parameters of the system are given in Table 2.2. These properties were chosen to

match those of the E-SPAARO unmanned aircraft.

Table 2.2: Mass and Size Properties

Property M Ixx Iyy Izz Ixz S c b

Value 21.34 kg 6.055 kg-m2 8.26 kg-m2 11.51 kg-m2 0.25 kg-m2 1.97 m2 0.53 m 3.66 m

The control inputs used by the synthetic model are based on actual control inputs recorded during an E-

SPAARO test �ight. Use of control inputs from �ight test data ensures that the inputs will result in a

realistic data set.

2.6.2 Parameter Estimation and Uncertainty

The parameters were estimated utilizing two simpli�ed versions of the synthetic model described previously.

One model only has a modi�ed CL term (SL), and the other has a modi�ed CL and CM term (SLM). Both

simpli�ed models have the same modi�cations in the form of the CL term. The simpli�cation involved


removing the quasi-steady dynamic stall terms. The modi�ed equation is shown below:

CL = CL0+ CLαα+ CLδe δe (2.41)

The second simpli�ed model additionally has a modi�cation of the Cm term with the damping term removed.

The modi�ed equation is:

Cm = Cm0+ Cmαα+ Cmδe δe (2.42)

The parameters of both models are tuned and the corresponding Cramér-Rao bounds are estimated as

described in Section 3.4, using the synthetic �ight data. The results of this estimation are shown in the

appendix (Tables 2.4 and 2.5). The initial parameter values used in tuning the simpli�ed model were

set equal to the values used in the original model. The Cramér-Rao bounds indicate that the parameter

estimates have converged to accurate values. Comparing the system identi�cation results, the estimated

parameters for the SL model are near the actual values, however the estimated parameters for the SLM

model di�er from the actual values in the longitudinal terms. The Cramér-Rao bounds are larger for the

SLM model than the SL model.

The simulation results and the true motion are compared in Figure 2.1. From this �gure we can see how

the converged model, the red line, does not overlap the true motion, the blue line, in some regions. This

di�erence in the lines does not decrease with additional parameter tuning as it is not due to errors in the

parameter estimates but in the selected model form (i.e., due to the missing terms in the CL or CL and Cm

expressions of the simpli�ed model). As expected the identi�ed model based on the SLM model has more

error than that based on the SL model.

2.6.3 Estimation of Model Form Uncertainty

To estimate the model form uncertainty a new synthetic data set was created based on control inputs from

a di�erent �ight test. This data was not used for parameter estimation to ensure that the uncertainty is not

arti�cially lowered via calibration. The model output based on the identi�ed parameter values is compared


(a) SL Model

(b) SLM Model

Figure 2.1: Comparison of Simulated Results Using the True Synthetic Model (Blue Lines) and Identi�ed

Model (Red Lines)


to the synthetic output in Figure 2.2. The blue line in these plots are the sensor data (including measurement

noise), and the red line is the motion predicted by the identi�ed model. The sensor data and modeled motion

are similar but do not match exactly. This indicates that the model is valid with some associated model

form uncertainty. Again, the identi�ed model based on SLM has more error than the model based on SL,

indicating that the SLM model is less accurate.

Using the method described in section 2.5, the model form uncertainty is calculated. The error in the CL

term was calculated at each time step and Figure 2.3 shows the model form uncertainty bounds for a second

order �t in velocity, angle of attack, pitch rate, and pitch angle. The �gures are selected two-dimensional

slices of a �ve-dimensional space. The values of the three dimensions for the prediction intervals not shown

on any slice are set as α = 2.4 degrees, V = 20.6 m/s, q = 0 rad/s, and θ = 0.8 degrees. The blue points in

the �gure are the values calculated for ∆CL that are near the slice and bounded by 0.4 ≤ α ≤ 4.4 degrees,

18.6 ≤ V ≤ 22.6 m/s, −.1 ≤ q ≤ .1 rad/s, and −1.2 ≤ θ ≤ 2.8 degrees. The red lines are the calculated

bounds on the model form uncertainty with 95% prediction intervals. From these plots it can be seen that

the uncertainty bounds are small in the region where a large amount of �ight test data is available and

increases in regions with less data. The model form uncertainty bounds are particularly large for high alpha

values. The bound is large in this region due to the lack of data to inform the prediction. As expected,

the bounds are nearly constant for the pitch values indicating that they have little in�uence on the model

form uncertainty. Comparing the two simpli�ed models, the the uncertainty bounds follow similar trends.

The identi�ed parameters based on SLM model has an o�set in the uncertainty leading to arti�cially larger

uncertainty bounds near the nominal state values.

The uncertainty bounds for the Cm terms is given in Figure 2.4. Comparing the two simpli�ed models, the

identi�ed model based on SLM has larger uncertainty bounds than the identi�ed model based on SL. The

di�erence between these models represents how the uncertainty bounds can vary as the identi�ed model

form is varied. The removal of the Cmq term lead to additional uncertainty in the CL terms and the Cm

uncertainty relative to α.


(a) SL Model

(b) SLM Model

Figure 2.2: Validation of Parameter Estimates from Synthetic Data


(a) SL Model (b) SLM Model

Figure 2.3: Synthetic Data, Model Form Uncertainty Bounds CL (2nd order)

(a) SL Model (b) SLM Model

Figure 2.4: Synthetic Data, Model Form Uncertainty Bounds CL (2nd order)

These bounds can be utilized in non-deterministic simulations by applying an uncertainty on the value of

aerodynamic coe�cient from within the bounds [27]. This uncertainty represents the lack of knowledge in

the model. When the bounds are larger the simulations will correctly estimate larger uncertainty in the

performance. Methods to sample the uncertainties will be presented in future work.


2.7 E-SPAARO Analysis

In this section the methods described in this paper are used to identify the aerodynamic parameters and

associated uncertainty of the E-SPAARO UAV. The E-SPAARO is a small UAV developed by Virginia Tech

for research purposes [13]. The mass properties of the aircraft are the same as those described in Table 2.2.

For this model the thrust T is set as a function of throttle position δT by the equations

RPM = −0.00522886δ3T + 1.04281575δ2T + 11.215278δT + 2329.50786049 (2.43)

T =0.07806ρπ((0.00508RPM)2 − (0.00508RPM)Vin)0.415

12ρV

2S; (2.44)

The RPM map is based on the measured RPM data in �ight and the thrust calculations are based on the

propeller pitch and diameter from experimental studies [28]. The Vin is the in�ow velocity to the propeller,

which is currently assumed to be the airspeed. For the E-SPAARO analysis the aerodynamic coe�cients

were based on the body frame, so it was not necessary to rotate into the wind frame. Additionally, the

aerodynamic coe�cients are then given by:

CX = CX0 + CXαα+ CXα2α2 (2.45)

CY = CY0 + CYββ + CYppb/(2V ) + CYrrb/(2V ) + CYδr δr (2.46)

CZ = CZ0 + CZαα+ CZδe δe (2.47)

Cl = Cl0 + Clββ + Clppb/(2V ) + Clrrb/(2V ) + Clδr δr + Clδa δa (2.48)

Cm = Cm0 + Cmαα+ Cmqqc/(2V ) + Cmδe δe (2.49)

Cn = Cn0 + Cnββ + Cnppb/(2V ) + Cnrrb/(2V ) + Cnδr δr + Cnδa δa (2.50)

The δe and δr terms show up in all of the moment coe�cients as the E-SPAARO has a v-tail con�guration

and the de�ections of the control surfaces on the two tail surface are slightly di�erent, resulting in coupling

between elevator and rudder de�ections.


2.7.1 Parameter Estimation and Uncertainty

In this case the initial parameter guesses were made based on the known aircraft geometry. The results of

the parameter tuning are shown in the appendix (Table 2.6). Again, since Cramér-Rao bounds are small

compared to the parameter values for most cases, the parametric uncertainty is negligible. In some cases the

uncertainty in the parameter values is signi�cant and would result in larger model form uncertainty bounds.

From Table 2.6, CXα2 and CZq have signi�cant Cramér-Rao bounds.

The simulation results and the sensor data for the �ight data used in parameter estimation are compared in

Figure 2.5. The red line in these plots show the modeled motion and the blue lines are the sensor data. The

Figure 2.5: Tuned Simulation vs. Experimental Data

vertical lines separate the plot into the section showing di�erent doublet tests: the �rst section shows the

pitch double, the second section shows the aileron doublet, and the third section shows the rudder doublet.

As this data was used to tune the parameters, the strong match is expected. Comparing Figure 2.5 to Figure

2.1, the proposed model does a poorer job matching the experimental data than the case with synthetic data.

This poorer match is due to the increased uncertainty from all sources (sensor noise, process noise, model

form) during actual �ight tests.


2.7.2 Estimation of Model Form Uncertainty

Validation of the estimated parameters was performed using sensor data from a second �ight test. Figure

2.6 shows the modeled motion and the sensor data from the second �ight. Red lines in these plots show the

Figure 2.6: Experimental Parameter Identi�cation Validation

modeled motion and the blue lines are the sensor data. The vertical lines again separate the plot into the

sections showing the di�erent doublet tests. Comparing the sensor data and the modeled motion there is

again a good agreement between the two. This agreement indicates that the identi�ed model parameters are

valid with some amount of model form uncertainty.

Following the method described in section 2.5, the errors in the aerodynamic coe�cients are calculated and

plotted against the aircraft states. Figure 2.7 shows the model form uncertainty bounds for the CZ term

using a second order �t in velocity, angle of attack, pitch rate, and pitch angle. The �t was selected as the

second order terms provided a good match over the residual data. The �gures are selected two-dimensional

slices of a �ve-dimensional space. The values of the dimensions for the prediction intervals not shown by

the slices were set as α = 1.58 degrees, V = 22.2 m/s, q = 0 rad/s, and θ = 6.44 degrees. The blue points

in the �gure are the values calculated for ∆CZ that are near the slice and bounded by 0.58 ≤ α ≤ 2.58


degrees, 20.2 ≤ V ≤ 24.2 m/s, −0.02 ≤ q ≤ 0.02 rad/s , and 1.44 ≤ θ ≤ 11.44 degrees. The red lines are

the estimated 95% bounds on the model form uncertainty. Examining these plots, we can again see how

the bounds increase as the model moves away from the region where we have data. The large change in the

width of the uncertainty bounds in V is partially due to the lower bound being constrained to be at most

zero.

Figure 2.7: Experimental Data, Model Form Uncertainty Bounds CZ

With the same bounds on the slices, Fig. 2.8 shows the model form uncertainty bounds of the CX term. The

uncertainty in this case seems to be mostly associated with the pitch rate (q). This correlation is indicated

by the slope associated with the uncertainty bounds. This slope leads to a change in the relative value of

the bounds even without a change in the amplitude of the uncertainty.

Similarly, Fig. 2.9 shows the model form uncertainty bounds for the Cm term. The points that are outside of

the bounds on the slice in pitch rate (q) occur during the aileron doublet. These points are not located near

the nominal pitch rate and because of this do not appear in slices used for the other plots. This indicates

that some factor present in this maneuver produces additional error in the model. If high roll rates are

desired, the model should be modi�ed to account for the error at high roll rates.

The model form uncertainty could be recalculated with additional terms to help account for these points.

New bounds were calculated with second order roll rate terms included. Figure 2.10 shows a comparison


Figure 2.8: Experimental Data, Model Form Uncertainty Bounds CX

Figure 2.9: Experimental Data, Model Form Uncertainty Bounds Cm

between the original and the updated bounds. The updated bounds have a lower minimum width of the

uncertainty compared to the prior bounds. The updated bounds are now over a six-dimensional space,

resulting in an additional state that can in�uence the uncertainty bounds.

To examine the lateral directional terms (Cy Cl Cn), a second order �t of the error in aerodynamic coe�cients

to velocity, sideslip, roll rate, and yaw rate is constructed. Figures 2.11-2.13 show selected two-dimensional

slices of a �ve-dimensional space. The values for the prediction intervals of the dimensions not shown by the

slice were set as β = −0.34 degrees, V = 22.2 m/s, p = 0 rad/s, and r = 0 rad/s. The blue points in the

�gures for lateral directional terms are data points near the slice. They are bounded by −1.34 ≤ β ≤ 0.76


Figure 2.10: Experimental Data, Model Form Uncertainty Bounds Cm (Blue Baseline, Red With Roll Rate)

degrees, 20.2 ≤ V ≤ 24.2 m/s, −0.02 ≤ p ≤ 0.02 rad/s, and −0.02 ≤ r ≤ 0.02 rad/s. Figure 2.11 shows the

model form uncertainty bounds for the Cy term. The uncertainty in Cy seems to be correlated to sideslip

(β) but the bounds do not change signi�cantly over the region plotted.

Figure 2.11: Experimental Data, Model Form Uncertainty Bounds Cy

Fig. 2.12 shows the model form uncertainty bounds for the Cl term. The uncertainty bounds do not

signi�cantly change over the selected region for any of the states.

Fig. 2.13 shows the model form uncertainty for the Cn term. The uncertainty is mostly related to sideslip

angle (β) as this term has the greatest slope resulting in the most signi�cant change in the boundaries over


Figure 2.12: Experimental Data, Model Form Uncertainty Bounds Cl

the region plotted.

Figure 2.13: Experimental Data, Model Form Uncertainty Bounds Cn

2.7.3 Validation of Model Form Uncertainty

Validation of these bounds was accomplished by using a third set of �ight test data that had not been used

in either of the previous steps. This validation was performed for bounds shown above (95%) and for bounds

calculated with a con�dence level of 68% (1-sigma) bounds. The model form uncertainty was calculated at

each data point. Table 2.3 shows the percentage of these new points that are within the previously calculated


bounds. Between 92.8% and 99.4% of the data from the third �ight test fell within the 95% bounds predicted

Table 2.3: Validation of Model Form Uncertainty Bounds

Model Form Uncertainty Type Estimated 95% Bounds Estimated 68% Bounds

CZ 93.9% 63.5%

CX 99.4% 64%

Cm 98.4% 89%

CY 94.7% 63.7%

Cl 92.8% 72.7%

Cn 99.4% 94.6%

using th model form uncertainty calculated in the previous section. Three of the coe�cients have less than

95% of the points within the predicted bounds and three coe�cients have more than 95% of the points within

the bounds. This is very close to the expected 95% and indicates the the calculated bounds are accurate for

the region where �ight data has been measured. Between 63.5% and 94.6% of the data from the third �ight

test fell within the 68% bounds predicted using th model form uncertainty calculated in the previous section.

The worst case values are again close to the expected value and indicate that these lower bounds are also

accurate. Again three coe�cients have less than the expected percentage in the bounds and three have more

than the expected percentage. If it were decided that these uncertainty bounds indicate more uncertainty in

the model than was acceptable, a new more complex model could be formulated and the analysis repeated.

2.8 Conclusion

This paper has presented a general method for combined aerodynamic parameter identi�cation and model

form uncertainty quanti�cation. Parameter identi�cation was accomplished through the use of output error

method and the associated con�dence in the parameters was estimated from the Cramér-Rao lower bound.


Validation data was then used to compare the values of the aerodynamic coe�cients as predicted by the

model and from experimental data. This data was used to estimate the model form uncertainty, which

was represented as prediction intervals. The prediction intervals provide tight bounds on the distribution

of errors in regions where data is available. The percentage of points that fall between the bounds depends

on the chosen value of the student's t-distribution. The method also leads to an increase in the uncertainty

bounds in regions with little or no experimental data. This increase in the uncertainty indicates that there

is less con�dence in the model in these regions as expected.

These methods were utilized to analyze both synthetic and true �ight test data. For the synthetic data, two

di�erent simpli�ed models were used. The results indicate that the model form uncertainty increases as the

assumed model structure moved away from the true model. Three �ight test data sets were used. The �rst

one for parameter identi�cation, the second one for uncertainty quanti�cation, and a third for validation of

uncertainty quanti�cation. From these example applications the ability to both estimate the aerodynamic

parameters and the associated model form uncertainties has been demonstrated and the method has been

validated. These estimates provides information that can be utilized in the prediction of nondeterministic

simulations of aircraft �ights.

Appendix


Table 2.4: Synthetic Model Parameter Estimation Results (SL model)

Parameter Actual Value Estimated Value Cramér-Rao bound

CL00.38 0.3805 0.00018

CLα 3.15 3.03 0.00008

CLδe-0.0004 -0.0023 0.00007

Cd0 0.1 0.0986 0.00005

Cy0 0 -0.00094 0.000053

Cyβ -0.248 -0.317 0.0017

Cyp -0.033 0.0058 0.0042

Cyr 0.17 0.27 0.004

Cyδr0.0024 0.0031 0.0001

Cl0 0 0.000004 0.0000003

Clβ -0.0092 -0.0094 0.00008

Clp -0.32 -0.34 0.0006

Clr 0.044 0.045 0.00013

Clδr0.0002 0.00022 0.0000024

Clδa0.0027 0.0029 0.0000049

Cm0 0.023 0.0246 0.000052

Cmα -0.55 -0.588 0.0012

Cmq -8.5 -9.4 0.019

Cmδe-0.012 -0.013 0.000025

Cn0 0 0.000006 0.000001

Cnβ 0.046 0.046 0.000034

Cnp -0.023 -0.027 0.00036

Cnr -0.077 -0.076 0.000093

Cnδr-0.0011 -0.0011 0.000002

Clδa-0.00006 -0.00004 0.000003

CTδT85 85.6 0.0339

a1 0.5 N/A N/A

α∗1 11 N/A N/A

α∗2 -20 N/A N/A

τ2 1 N/A N/A


Table 2.5: Synthetic Model Parameter Estimation Results (SLM model)

Parameter Actual Value Estimated Value Cramér-Rao bound

CL00.38 0.28 0.0013

CLα 3.15 1.95 0.0008

CLδe-0.0004 -0.019 0.0004

Cd0 0.1 0.088 0.00023

Cy0 0 0.00065 0.00007

Cyβ -0.248 -0.191 0.0038

Cyp -0.033 0.0373 0.0086

Cyr 0.17 0.186 0.01

Cyδr0.0024 0.0042 0.00013

Cl0 0 -0.00004 0.000002

Clβ -0.0092 -0.0109 0.00022

Clp -0.32 -0.195 0.0014

Clr 0.044 0.0352 0.00048

Clδr0.0002 -0.00013 0.000006

Clδa0.0027 0.0016 0.000009

Cm0 0.023 0.0233 0.00039

Cmα -0.55 -0.852 0.0103

Cmq -8.5 N/A N/A

Cmδe-0.012 -0.004 0.0001

Cn0 0 -0.00004 0.000004

Cnβ 0.046 0.0328 0.00012

Cnp -0.023 -0.0115 0.0011

Cnr -0.077 -0.0701 0.0004

Cnδr-0.0011 -0.00071 0.000006

Clδa-0.00006 -0.00013 0.000008

CTδT85 118.1 0.2227

a1 0.5 N/A N/A

α∗1 11 N/A N/A

α∗2 -20 N/A N/A

τ2 1 N/A N/A


Table 2.6: E-SPAARO Parameter Estimation Results

Parameter Initial Guess Estimated Value Cramér-Rao bound

CZ0-0.327 -0.312 0.0017

CZα -2.77 -3.34 0.046

CZδe0.038 0.036 0.0006

CX0 -0.028 -0.038 0.00054

CXα 0.013 0.264 0.022

CXα2

2.4 1.86 0.16

Cy0 -0.0005 -0.00015 0.00017

Cyβ -0.292 -0.298 0.0033

Cyp -0.066 -0.075 0.0069

Cyr 0.033 0.135 0.0117

Cyδr0.0017 0.0023 0.00013

Cl0 -0.00003 -0.00009 0.000008

Clβ -0.01 -0.01 0.00064

Clp -0.19 -0.296 0.0053

Clr 0.0217 0.0094 0.0013

Clδr-0.00007 -0.0003 0.000026

Clδa0.0018 0.0024 0.000042

Cm0 0.0085 0.0104 0.00029

Cmα -0.364 -0.487 0.01

Cmq -9.7 -13.8 0.28

Cmδe-0.0092 -0.014 0.00028

Cn0 0.00004 0.000007 0.00002

Cnβ 0.0557 0.0473 0.00024

Cnp -0.0203 -0.0225 0.0019

Cnr -0.447 -0.075 0.00098

Cnδr-0.0011 -0.0011 0.000013

Cnδa-0.00009 -0.000007 0.000014

Bibliography

[1] Paw, Y. C. and Balas, G. J., �Uncertainty Modeling, Analysis and Robust Flight Control Design for

a Small UAV System,� Proceedings of the AIAA Guidance, Navigation, and Control Conference and

Exhibit , 2008.

[2] Nannapaneni, S. and Mahadevan, S., �Model and data uncertainty e�ects on reliability estimation,� 17th

AIAA Non-Deterministic Approaches Conference, American Institute of Aeronautics and Astronautics

Inc., 2015.

[3] Mahadevan, S. and Liang, B., �Error and Uncertainty Quanti�cation and Sensitivity Analysis in Me-

chanics Computational Models,� International Journal for Uncertainty Quanti�cation, Vol. 1, No. 2,

2011, pp. 147�161.

[4] Schmidt, D. and Lee, S., �New Methods for Structured Uncertainty Modeling for Flexible Aircraft,� 17th

AIAA Non-Deterministic Approaches Conference, American Institute of Aeronautics and Astronautics,

1999.

[5] Riley, M. E. and Grandhi, R. V., �Quanti�cation of model-form and predictive uncertainty for multi-

physics simulation,� Computers and Structures, Vol. 89, 2011, pp. 1206�1213.

[6] Prabhakar, A., Fisher, J., and Bhattacharya, R., �Polynomial chaos-based analysis of probabilistic

uncertainty in hypersonic �ight dynamics,� Journal of guidance, control, and dynamics, Vol. 33, No. 1,

2010, pp. 222�234.

32


[7] Hyde, D. C., Brown, F., Shweyk, K. M., and Shah, G., �Advanced modeling and uncertainty quan-

ti�cation for �ight dynamics; interim results and challenges,� AIAA Atmospheric Flight Mechanics

Conference - SciTech Forum and Exposition 2014 , 2014.

[8] Cotting, M., �UAV Performance Rating Scale Based on the Cooper-Harper Piloted Rating Scale,� 49th

AIAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposition, 2011.

[9] Weibel, R. E. and Hansman, R. J., �Safety Considerations for Operation of Unmanned Aerial Vehicles

in the National Airspace System,� Tech. Rep. March, Massachusetts Inst. of Technology International

Center for Air Transportation, Cambridge, MA, 2005.

[10] U.S. Department of Transportation, Federal Aviation Administration, �Integration of civil Unmanned

Aircraft Systems (UAS) in the National Airspace System (NAS) roadmap,� Tech. Rep. 2012-AJG-502,

Washington, D.C., 2013.

[11] Cotting, M., �Proposed Longitudinal Flying Qualities Criterion for Unpiloted Autonomous Aircraft,

Starting the Conversation,� AIAA Atmospheric Flight Mechanics Conference, No. August, 2010.

[12] Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., and Alvin, K. F., �Error and

uncertainty in modeling and simulation,� Reliability Engineering & System Safety , Vol. 75, No. 3, 2002,

pp. 333 � 357.

[13] Cotting, M. C., Wolek, A., Murtha, J., and Woolsey, C., �Developmental Flight Testing of the SPAARO

UAV,� 48th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Ex-

position, Aerospace Sciences Meetings, American Institute of Aeronautics and Astronautics, Jan 2010.

[14] Maine, R. and Ili�, K., �Identi�cation of dynamic systems- applications to aircraft. Part 1: The output

error approach,� AGARD , Vol. 3, No. 300, 1986.

[15] Klein, V., �Estimation of aircraft aerodynamic parameters from �ight data,� Progress in Aerospace

Sciences, Vol. 26, No. 1, Jan. 1989, pp. 1�77.


[16] Morelli, E. and Klein, V., �Accuracy of aerodynamic model parameters estimated from �ight test data,�

Journal of Guidance, Control, and Dynamics, Vol. 20, No. 1, 1997.

[17] Raol, J., G. G. and Singh, J., Modeling and Parameter Estimation of Dynamical Systems, chap. 3, The

Institution of Electrical Engineers, London, UK, 2004, p. 37 â��64.

[18] Morelli, E.A. and Klein, V., �Determining the Accuracy of Maximum Likelihood Parameter Estimates

With Colored Residuals,� NASA CR 194893, March 1994.

[19] Klein, V. and Morelli, E., Aircraft System Identi�cation - Theory and Practice, AIAA Education Series,

AIAA, Reston, VA.

[20] Hamel, P. G. and Jategaonkar, R. V., �Evolution of �ight vehicle system identi�cation,� Journal of

Aircraft , Vol. 33, No. 1, Jan. 1996, pp. 9�28.

[21] Maine, R. and Ili�, K., The theory and practice of estimating the accuracy of dynamic �ight-determined

coe�cients, NASA RP 1077, 1981.

[22] Roy, C. J. and Oberkampf, W. L., �A comprehensive framework for veri�cation, validation, and uncer-

tainty quanti�cation in scienti�c computing,� Computer Methods in Applied Mechanics and Engineering ,

Vol. 200, No. 25-28, June 2011, pp. 2131�2144.

[23] Nathans, L., Oswald, F., and Nimon, K., �Interpreting Multiple Linear Regression: A Guidebook of

Variable Importance,� Practical Assesment Research & Evaluation, Vol. 17, No. 9, 2012, pp. 19.

[24] Seber, G. A. and Lee, A. J., Linear regression analysis, Vol. 936, John Wiley & Sons, 2012.

[25] Jategaonkar, R. V., Fischenberg, D., and Gruenhagen, W., �Aerodynamic Modeling and System Identi-

�cation from Flight Data-Recent Applications at DLR,� Journal of Aircraft , Vol. 41, No. 4, July 2004,

pp. 681�691.

[26] Hoblit, F. M., "Continuous Turbulence Gust Loads Criteria", Gust Loads on Aircraft: Concepts and

Applications, AIAA Education Series, pp. 69-96, 1988.


[27] Hale, L. E., Patil, M., and Roy, C. J., �Nondeterministic Simulation for Probability of Loss of Control

Prediction for Unmanned Aircraft Systems,� AIAA Modeling and Simulation Technologies Conference,

AIAA Aviation, American Institute of Aeronautics and Astronautics, jun 2015.

[28] Staples, G., �Propeller Static & Dynamic Thrust Calculation - Part 2 of 2 - How Did I Come Up

With This Equation,� http://www.electricrcaircraftguy.com/2014/04/propeller-static-dynamic-thrust-

equation-background.html, April 2014.

[29] Hale, L. E., Patil, M., and Roy, C. J., �Aerodynamic Parameter Identi�cation and Uncertainty Quan-

ti�cation for Small Unmanned Aircraft,� AIAA Guidance, Navigation, and Control Conference, AIAA

SciTech, American Institute of Aeronautics and Astronautics, Jan. 2015.

[30] Moorhouse, D. J. and Woodcock, R. J., �Background Information and User Guide for MIL-F-8785B,

Military Speci�cation - Flying Qualities of Piloted Airplanes,� Tech. rep., WRIGHT-PATrERSON AIR

FORCE BASE, 1982.

[31] Ferson, S. and Ginzburg, L. R., �Di�erent methods are needed to propagate ignorance and variability,�

Reliability Engineering & System Safety , Vol. 54, No. 2-3, Nov. 1996, pp. 133�144.

[32] Crespo, L. G., Kenny, S. P., and Giesy, D. P., �The NASA Langley Multidisciplinary Uncertainty

Quanti�cation Challenge,� 16th AIAA Non-Deterministic Approaches Conference, Jan. 2014, pp. 1�9.

[33] Broggi, M., Alvarez, D. A., and Angelis, M. D., �An integrated and e�cient numerical framework for un-

certainty quanti�cation : application to the NASA Langley multidisciplinary Uncertainty Quanti�cation

Challenge,� 16th AIAA Non-Deterministic Approaches Conference, Jan. 2014.

[34] Chaudhuri, A., Waycaster, G., Matsumura, T., Price, N., and Haftka, R. T., �Framework for Quan-

ti�cation and Risk Analysis for Layered Uncertainty using Optimization : NASA UQ Challenge,� 16th

AIAA Non-Deterministic Approaches Conference, Jan. 2014, pp. 1�30.

[35] Liang, C. and Mahadevan, S., �Bayesian Framework for Multidisciplinary Uncertainty Quanti�cation

and Optimization,� 16th AIAA Non-Deterministic Approaches Conference, 2014.


[36] McFarland, J., Bichon, B. J., and Riha, D. S., �A Probabilistic Treatment of Multiple Uncertainty Types:

NASA UQ Challenge,� 16th AIAA Non-Deterministic Approaches Conference, Jan. 2014, pp. 1�25.

[37] Safta, C. and Eldred, M. S., �Uncertainty Quanti�cation Methods for Model Calibration , Validation ,

and Risk Analysis,� 16th AIAA Non-Deterministic Approaches Conference, Vol. 8, Jan. 2014, pp. 1�17.

[38] Srivastava, A., Subramaniyan, A. K., and Wang, L., �A Hybrid Bayesian Solution to NASA Langley

Multidisciplinary Uncertainty Quanti�cation Challenge,� 16th AIAA Non-Deterministic Approaches

Conference, Jan. 2014, pp. 1�34.

[39] Oberkampf, W. L. and Roy, C. J., Veri�cation and Validation in Scienti�c Computing , Cambridge

University Press, 2010.

[40] Sobol, I., �Global sensitivity indices for nonlinear mathematical models and their Monte Carlo esti-

mates,� Mathematics and Computers in Simulation, Vol. 55, No. 1-3, Feb. 2001, pp. 271�280.

[41] Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., and Tarantola, S., �Variance based

sensitivity analysis of model output. Design and estimator for the total sensitivity index,� Computer

Physics Communications, Vol. 181, No. 2, Feb. 2010, pp. 259�270.

[42] Guo, J. and Du, X., �Sensitivity Analysis with Mixture of Epistemic and Aleatory Uncertainties,� AIAA

Journal , Vol. 45, No. 9, Sept. 2007, pp. 2337�2349.

[43] �Military Speci�cation Flying Qualities of Piloted Airplanes,� Tech. Rep. MIL-F-8785C, 1980.

Chapter 3

Estimation of Uncertainty Quanti�ed

Performance of Unmanned Aircraft

Using Non-Deterministic Simulations

3.1 Title

Estimation of Uncertainty Quanti�ed Performance of Unmanned Aircraft Using Non-Deterministic Simula-

tions

By: Lawrence E. Hale, Mayuresh Patil, and Christopher J Roy

37

Lawrence E. Hale Chapter 3. Uncertainty Quanti�ed Performance 38

3.2 Abstract

This paper examines a method based on nondeterministic simulations to estimate the uncertainty quanti�ed

performance of unmanned aircraft. The aerodynamic parameters in the �ight dynamic model are identi�ed

using the output error method applied to �ight test data. The model form uncertainty is estimated using

a new approach based on the equation error method. A method to sample the time varying model form

uncertainty is formulated. Nondeterministic simulations are performed based on model form uncertainty

modeled as time varying epistemic uncertainty and atmospheric turbulence modeled as aleatory uncertainty.

The response of the closed-loop system to various wind speeds and atmospheric turbulence levels is analyzed.

3.3 Introduction

There has been a major push recently to integrate unmanned aircraft systems (UAS) into the national

airspace [10, 9]. The use of these systems in civil applications has been limited due to a lack of appropriate

regulations. One of the primary obstacles to the formulation of these regulation is the large number of

potential designs. To help provide additional information on the performance of proposed designs, there is

a desire to perform nondeterministic simulations of aircraft �ights. By performing these nondeterministic

simulations in diverse atmospheric conditions, information can be provided that could help inform appropri-

ate regulations and aid in the certi�cation process. These simulations provide nondeterministic information

on how the aircraft will react in adverse conditions allowing for the risk to be quanti�ed. A framework for

the inclusion of error and uncertainty in modeling is provided by Oberkampf et al.[12]. Additionally, sensi-

tivity analysis of the simulation results can be utilized to derive information on how changes in atmospheric

conditions can change the predicted aircraft performance.

The estimation and analysis of parametric uncertainties has been examined using a variety of approaches

[1, 2]. Some information on uncertainty quanti�cation in general computational models is presented by

Liang and Mahadevan [3]. Previous model form uncertainty predictions for aircraft have primarily focused


on aeroelastic analyses, often assuming a probability distribution for the uncertainty [4, 5]. These methods

are su�cient if the �ight conditions obey a probability distribution. However, as the aircraft deviates from

the �ight conditions used to estimate the uncertainty there is no guarantee that the �ight condition will

obey a probability distribution. Polynomial chaos could be utilized to propagate uncertainties through the

system, however the formulation of polynomial chaos results in a degradation of the estimates over time [6].

The degradation of the estimates is due to the �nite dimensional approximation of the probability space.

Some current work is also underway by a team from Boeing and NASA Langley [7]. This work is part of the

NASA Vehicle Systems Safety Technologies project Technical Challenge #3. The paper discusses the goals

of the project, and describes the current static and dynamic identi�cation testing that has been performed.

UAS performance measures are still in development. Performance measures based on the Cooper-Harper

Piloted Rating scale have been examined in Ref. [8]. These methods relate the UAV performance measures

to manned aircraft performance measures, however as UAS are mostly �own autonomously performance

limits may be better associated with the payload requirements. The method proposed in this paper allows

for the performance measure of interest to be adjusted to the requirements of a variety of payloads.

This paper utilizes nondeterministic simulations to predict the performance of an unmanned aerial vehicle

(UAV). This work can be used to generate information supporting certi�cation and informing operational

bounds on the use of UAVs. The design of small UAVs is of particular interest as they are used in the

civil sector due to their low cost and variety of uses. To keep the cost of these small UAVs reasonable,

they are designed with tight budgets that limit how precisely the aircraft can be modeled. The resulting

simpli�ed models may not accurately represent the true aircraft motion, resulting in greater uncertainty in

the performance.


3.4 Parameter Estimation and Uncertainty Quanti�cation

3.4.1 Aerodynamic Parameter Identi�cation

Aerodynamic parameter identi�cation is performed through the use of system identi�cation. A number of

methods could be used when performing identi�cation of aircraft parameters as shown in previous detailed

studies [14, 15, 16, 17]. For aircraft, output error method is often utilized for parameter identi�cation. This

method provides parameter estimates that approach a best estimate of the value (the best estimate is the

true value if the assumed model structure is correct) as the amount of data increases.

For the general formulation of the output error method, the aircraft dynamic model is of the form

x(t) = f [x(t), u(t),Θ], x(t0) = x0 (3.1)

y(t) = g[x(t), u(t),Θ] (3.2)

z(tk) = y(tk) + v(tk), k = 1, . . . N (3.3)

where Θ is the vector of unknown parameters, x is state vector and u is the vector of control inputs. f and

g represent the model dynamics and observer transformation, respectively. The measured data z is provided

at N discrete times and in a perfect model represents the observed output y plus some measurement noise v.

For the output-error formulation, the noise is assumed to be a sequence of independent zero mean Gaussian

random numbers.

The output error method does not account for process noise which can include wind, steady as well as

turbulent. Extensions to the output error method that can better account for process noise are detailed in

work by Klein and Morelli [18, 19]. Output error method is acceptable in the present work as �ight test data

is generated on calm, low-turbulence days. Furthermore the primary focus of the present work is on model

form uncertainty estimation which works with any system identi�cation method.


The estimates of Θ are found by minimizing the cost function [15]:

J =1

2

N∑k=1

[z(tk)− y(tk)]TR−1[z(tk)− y(tk)] (3.4)

where R is the prediction error covariance matrix. The minimum of the cost function is found by using

Gauss-Newton method which updates the parameters by:

Θ(i+1) = Θi + ∆Θ (3.5)

∆Θ = −M−1G (3.6)

The information matrix M and the gradient vector G are [14]:

M =

N∑k=1

[∂y(tk)

∂Θ

]TR−1

[∂y(tk)

∂Θ

](3.7)

G =

N∑k=1

[∂y(tk)

∂Θ

]TR−1[z(tk)− y(tk)] (3.8)

These values only require �rst order sensitivities, that here are approximated by �nite di�erence [20].

Once the parameter updating is completed we have parameter estimates that minimize the error between

the response of our proposed model and the measured response, based on our assumed model form and the

�ight test data.

3.4.2 Aerodynamic Parameter Uncertainty Quanti�cation

Uncertainty quanti�cation for the aerodynamic parameters is based on the calculation of con�dence intervals

about the parameters. When using output error the con�dence intervals are estimated based on the Cramér-

Rao lower bound [21]. The Cramér-Rao lower bound represents the minimum variance of the estimated

value. The size of the intervals can be reduced with additional or better information about the parameters.

These bounds indicate whether or not su�cient data were used in the parameter identi�cation process. Large

bounds relative to the estimated parameter would indicate that the estimation should be performed with

additional data. If additional data does not improve the bounds, the assumed model may be inadequate and

require reformulation.


The Cramér-Rao bound is calculated as

σj ≥√djj (3.9)

where σj is the minimum con�dence interval. djj is the trace of the inverse of the information matrix:

D = M−1 (3.10)

3.5 E-SPAARO Model

The parameter identi�cation was performed for the E-SPAARO UAV shown in Figure 3.1. The E-SPAARO

is a small UAV developed by Virginia Tech for research purposes [13]. The aircraft uses the 3DR Pixhawk

autopilot during autonomous �ights. The Pixhawk software is open source allowing access to controller fore

modeling in the simulation. The simulated controller includes the computation time required by the Pixhawk

with the commanded control inputs held constant until the next command is received. The controller gains

were set by performing a �ight test and following the procedure outlined in the Pixhawk documentation.

Figure 3.1: E-SPAARO UAV

The vehicle dynamics are described by

u =Fx + T

M− g sin θ + rv − qw (3.11)


v =FyM


w =FzM


p =Izzl + Ixzn− (Ixz(Iyy − Ixx − Izz)p+ (I2xz + Izz(Izz − Iyy))r)q

Ixx ∗ Izz − I2xz(3.14)

q =m+ (Izz − Ixx)pr − Ixz(p2 − r2)

Iyy(3.15)

r =Ixzl + Ixxn− (Ixz(Iyy − Ixx − Izz)r + (I2xz + Ixx(Ixx − Iyy))p)q


The terms p, q, r are the angular rates, u, v, w are the body velocities, and θ and φ are the pitch and roll

angles. The thrust T is set as a function of throttle position δT by the equations

RPM = −0.00522886δ3T + 1.04281575δ2T + 11.215278δT + 2329.50786049 (3.17)

T =0.07806ρπ((0.00508RPM)2 − (0.00508RPM)Vin)0.415

12ρV

2S; (3.18)


propeller pitch and diameter from experimental studies [28]. The Vin is the in�ow velocity to the propeller,

which is currently assumed to be the airspeed. The aerodynamic forces and moments are given by:

Fz =1

2ρV 2SCz (3.19)

Fx =1

2ρV 2SCx (3.20)

Fy =1

2ρV 2SCy (3.21)

l =1

2ρV 2SbCL (3.22)

m =1

2ρV 2ScCM (3.23)

n =1

2ρV 2SbCN (3.24)

The V in these equations is the total velocity, S is the wing area, c is the chord, b is the wing span, and ρ

is the air density. The aerodynamic coe�cients are given as:

CZ = CZ0+ CZαα+ CZqq + CZδe δe (3.25)


CX = CX0+ CXαα+ CXα2α

2 (3.26)

Cy = Cy0 + Cyββ + Cypp+ Cyrr + Cyδr δr (3.27)

Cl = Cl0 + Clββ + Clpp+ Clrr + Clδr δr + Clδa δa + Cmδe δe (3.28)

Cm = Cm0+ Cmαα+ Cmqq + Cmδe δe + Cmδr δr (3.29)

Cn = Cn0+ Cnββ + Cnpp+ Cnrr + Cnδr δr + Cnδa δa + Cmδe δe (3.30)

The δe and δr terms show up in all of the moment coe�cients as the E-SPAARO has a v-tail con�guration

and the de�ections of the control surfaces on the two tail surface are slightly di�erent, resulting in coupling

between elevator and rudder de�ections. These equations consist of the aircraft states and the aerodynamic

parameters. These parameters are the terms in Θ that are tuned in the output error method.

3.6 Model Form Uncertainty Quanti�cation

The parametric uncertainties calculated in the previous section provide an estimate of how accurately the

aerodynamic parameters have been identi�ed, based on the assumed model form. The approach does not

provide information on how accurately the model represents the true physics, i.e., the uncertainty due to

the assumed model form. This section proposes a method to provide estimates of theses uncertainties. The

estimates are made in a conservative manner as sensor noise is included in the estimates. The inclusion of

sensor noise results in wider, more conservative, uncertainty bounds

The residuals used in the estimation of model form uncertainty are calculated utilizing di�erent data than

that used in the estimation of the parameters. The use of distinct data additionally provides validation

of the estimated model. The bounds on the uncertainty are based on prediction intervals [22] calculated

from the residuals of the force and moment coe�cients. The residuals are the di�erence in the coe�cient

calculated from the measured linear accelerations and rotational rates and those predicted by model based

on the measured aircraft states. The linear accelerations and rates are corrected to the aircraft CG and then


multiplied by the mass to calculate the forces and the inertia to calculate the moments. The aerodynamic

coe�cients based on experimental data are then calculated from these forces and moments. The equations

used to �nd these residuals are given as:

m

.5ρV 2SaX = CX(y,Θ) +

T (y,Θ)

.5ρV 2S+ ∆CX (3.31)

m

.5ρV 2SaY = CY (y,Θ) + ∆CY (3.32)

m

.5ρV 2SaZ = CZ(y,Θ) + ∆CZ (3.33)

Ix.5ρV 2Sb

[p+

Iz − IyIx

qr − IxzIx

(pq + r)

]= Cl(y,Θ) + ∆Cl (3.34)

Iy.5ρV 2Sc

[q +

Ix − IzIy

pr +Ixz

Iy(p2 − r2)

]= Cm(y,Θ) + ∆Cm (3.35)

Iz.5ρV 2Sb

[r +

Iy − IxIz

pq +Ixy

Ix(qr − p)

]= Cn(y,Θ) + ∆Cn (3.36)

Θ are the identi�ed parameter values and y are the model states on which the aerodynamic model is

dependent. The term m is the mass of the system, and Ix, Iy, Iz, and Ixz are the inertia terms. The

identi�ed parameters and model states provide the model predicted force and moment coe�cients. The

terms p, q, r are the angular rates, and aX , aY , aZ are the linear acceleration measurements (corrected to

the CG location). The angular rates and linear accelerations, along with the airspeed and vehicle inertial

properties are used to calculate the estimates of the aerodynamic coe�cients based on �ight test data. The

terms ∆CX , ∆CY , ∆CX , ∆CY , ∆CX , and ∆CY represent the residual between the experimental data and

identi�ed model and are used to estimate the model form uncertainty.

The residual is calculated for each experimental time step and is then used to estimate the model form

uncertainty bounds. The correlated states are then used to calculate prediction intervals on the coe�cients.

The use of prediction intervals to represent the model form uncertainty has the advantage of increasing

the uncertainty in regions with less information. Figure 3.2 shows an example of the prediction intervals

based on the E-SPAARO. The blue dots on this plot represent the calculated residualss in the aerodynamic

parameters and the red lines are the 95% prediction intervals (i.e., the estimated range where at least 95%

of additional data is expected to fall). Note that the prediction intervals grow outside the range where


Figure 3.2: Example Prediction Intervals

data is available. This high uncertainty could likely be reduced by taking additional data at more extreme

values of angle of attack, velocity, etc. Further information on this method and an example of the system

identi�cation and model form uncertainty process can be found in Ref. [29].

3.7 Uncertainty Sampling

The uncertainties to be sampled are categorized as either aleatory uncertainty, de�ned as �uncertainty due

to inherent randomness", or epistemic uncertainty, de�ned as �uncertainty due to lack of knowledge."[22] For

this study, the epistemic uncertainties have been treated as intervals with precise bounds but no associated

probability distribution. Each type of uncertainty is additionally subdivided into static and time varying

uncertainties. When sampled, static uncertainties have a single value for the entire simulation and time

varying uncertainties vary during the simulation. The aleatory uncertainties in the study are the wind

turbulence and sensor noise. Wind turbulence is a time varying uncertainty with a known probability

distribution based on the Dryden turbulence model [30]. Sensor noise is based on a Gaussian zero mean

distribution


Time varying epistemic uncertainties could be sampled with multiple methods. The time varying epistemic

uncertainties in this study are associated with the model form uncertainties. The calculated prediction

intervals (shown in Figure 3.2) provide bounds on the model form uncertainty that are associated with the

aircraft states. Thus as the aircraft states change the uncertainty range also changes. Several methods

could be utilized to sample this uncertainty for the simulation. The �rst proposed method to sample the

uncertainty is to set the uncertainty to a constant percentage between the upper and lower bounds for any

given state. This method represents the simplest possible scenario where the uncertainty only varies with

the states used to calculate the bounds. A second method allows the uncertainty to vary randomly between

the upper and lower bounds instantaneously at every time step. This method would be overly conservative,

as the true values of the aerodynamic parameters do not vary in this manner, resulting in an overestimation

of the number of loss of control events. A �nal method to sample the uncertainty allows for the uncertainty

to vary between the prediction intervals randomly at a bounded rate. The rate of variation can be estimated

based on the rate of change of the ∆C∗ data used to calculate the model form uncertainty bounds. This

data is calculated for each bound separately so each term can vary at a di�erent rate. The time history of

the ∆C∗ values are �ltered to minimize the e�ect of the sensor noise on the estimation of the rate of change

of uncertainty bounds. The rate of change is then estimated by calculating the discrete change in the value

of the �ltered ∆C∗ data at each time step.

δC∗ = ∆C∗k −∆C∗k−1(3.37)

Figure 3.3 shows the probability density function of the δCZ values from data used to calculate the prediction

intervals. These values have been normalized by the width of the prediction interval. Fitting the data with

a normal �t provides an estimate of how fast the model form uncertainty could change. This �t is shown in

the �gure in red and has a standard deviation of 0.17 percent. The rate of change of the uncertainty is based

on data rate of the �ight recorder tpre. If the nondeterministic simulations are performed at di�erent data

rate tSim then the rate of change in the uncertainty must be scaled. To account for the di�erence between


Figure 3.3: Probability Density Function of δCZ

the sample time and the simulation time the standard deviation for simulation is scaled by

σsim = σPre

√tPretSim

(3.38)

where σPre is the standard deviation from the probability density function.

The uncertainty level is allowed to vary using a one dimensional Gaussian random walk with re�ecting

boundaries. A Gaussian random walk is a random walk with a variable step size de�ned by a Gaussian

distribution, in this case σsim. To ensure that the uncertainty value remains within the uncertainty bounds,

re�ecting boundaries are utilized. If the uncertainty value would cross this boundary, it is re�ected by back

into the acceptable range. The re�ection results in the uncertainty value being within the bounds by an

amount equal to the amount that the bound would have been violated.


3.8 Nondeterministic Simulation

Nondeterministic simulations allow for the e�ects of the epistemic uncertainties to be propagated through

the model. Epistemic and aleatory uncertainties are propagated separately as described by Refs. [31, 22].

To propagate the uncertainties separately a nested formulation is used with the outer loop sampling the

epistemic uncertainties and the inner loop sampling from the aleatory uncertainties. For each iteration of

the outer loop a cumulative distribution function (CDF) is formed based on the aleatory samples. This

is repeated for each sample of epistemic uncertainties resulting in a family of CDFs. The maximum and

minimum values from the family of CDFs are selected producing a probability box (p-box). The p-box

Figure 3.4: p-box representing aleatory and epistemic uncertainties

represents the upper and lower bounds on the cumulative distribution function (CDF) of the parameter.

The width between the curves indicates the epistemic uncertainty and the shape of the curves represents

the aleatory uncertainty. If the epistemic uncertainty were removed only one curve would remain and if the

aleatory uncertainty were removed the plot would show two vertical lines some distance apart. The aleatory

uncertainties in this study are the atmospheric turbulence and sensor noise. The epistemic uncertainties are

the model form uncertainties.

Atmospheric turbulence in the simulations is modeled using the Dryden turbulence model [30]. The Dryden

model provides turbulence velocities acting on the aircraft based on the aircraft altitude, velocity, and


reference turbulence intensity. Turbulence intensities for low, moderate, and severe turbulence are given in

[30]. The measurements used for the simulation controller have zero-mean Gaussian noise included in each

sample used to calculate the controller commands..

To ensure that the turbulence and sensor noise are su�ciently sampled, the convergence of the simulation

results are examined. Convergence is checked by simulating the aircraft using the nominal model with low

turbulence and sensor noise. The simulation is run multiple times using di�erent combinations of turbulence

and sensor noise and the results are compared. The lowest acceptable simulation length was found to be

500 hours of simulated �ight. Figure 3.5 shows the results of the convergence check for 500 simulated �ight

hours. Each point on the plotted curves represents a hour of �ight, and each curve is composed of 500 points.

The curves in the �gure represent the average and maximum values of the cross track, altitude and velocity

errors. For these simulations the nominal model was used in each of the 10 simulation with di�erent sensor

noise and wind speed time histories. As the curves for each set of 500 simulated �ight hours are almost

identical we can conclude that 500 hours of �ight is su�cient to estimate the uncertain response due to

combinations of sensor noise and atmospheric turbulence.

Figure 3.5: Simulated performance CDF convergence check with 500 hours.

For the epistemic uncertainties at a �xed level between uncertainty bounds, method one, samples were se-


lected at each combination of maximum and minimum bound. Sampling these points examines the corner

points of the uncertainty range and allow for the maximum �xed perturbations to be examined. To ensure

that a su�cient number of epistemic uncertainty time histories were sampled by simulations where the uncer-

tainties were allowed to vary at a bounded rate, the third method, a similar convergence check to that used

for the aleatory uncertainties was performed. Based on this check it was determined that 2000 simulations

would su�ciently sample the possible combinations of time histories for the model form uncertainty.

3.9 Simulation Results

This section reports the results from simulated aircraft �ights. The simulations represent the �ight char-

acteristics of the empty aircraft and do not consider possible mass and inertia variations due to possible

payloads that may be used in the aircraft. Wind and sensor noise are included in the simulation along with

the model form uncertainties. The initial simulations have been performed using the corner points of the

hypercube of uncertainty, all combinations of upper and lower prediction interval bounds, of the model-form

uncertainties were used in the simulations. These samples matches the �rst method of sampling the model

form uncertainty proposed in Section 3.7.

To provide validation of the simulations the aircraft was �own using the autopilot in a loiter mission. Figure

Figure 3.6: Experimental loiter performance


3.6 shows the vehicle path in blue, and the desired path in red. The o�set in the two paths is due to the

method utilized by the autopilot in the loiter algorithm. The weather conditions on this day were 3 m/s

steady wind with medium to severe turbulence. Using the data from �ight test the performance of the system

was estimated giving a median absolute error in velocity as 0.61 m/s, in altitude as 2.16 m, and in cross track

as 9.41 m. The performance calculations form the �ight test data are based on the sensor measurements.

These estimations include sensor noise in the measurements and this noise may be resulting in experimental

performance being slightly worse than the actual performance.

To validate that the nondeterministic simulations provided results that matched the �ight tests, nondeter-

ministic simulations of the �ight were conducted using the same controller gains as in the experimental �ight

and using a 3 m/s steady wind with medium turbulence. Figure 3.7 shows the median cross track, velocity,

Figure 3.7: Simulated performance (3 m/s steady wind, medium turbulence)

and altitude errors. The blue and red curves represent the corner points of the model form uncertainty. The

curves are separated into 6 sections with blue color representing the lower bound and red representing the

upper bound. The Corresponding uncertainties are labeled on the right side of the plot. The black vertical

line in each plot is the experimentally determined value. The �ight test performance is a single vertical line

and not a CDF because the data all comes from a single �ight test. If additional �ight tests were performed


in similar weather conditions, a CDF could be formed using the additional data points. The experimental

results indicate a larger error in both velocity and altitude. The performance calculations from the �ight

test data are based on the sensor measurements that have sensor noise included. The performance from the

simulations were based on the exact values. Using data that includes the senor noise the simulated perfor-

mance was recalculated. Figure 3.8 shows the performance measurements with the sensor noise included.

A comparison of Figures 3.7 and 3.8 shows how the inclusion of sensor noise in the performance measures

Figure 3.8: Simulated performance (3 m/s steady wind, medium turbulence) with sensor noise added to

recorded simulation measurements

results in degraded performance estimates for the simulation. For validation Figure 3.8 is apropriate as the

experimental data includes the sensor noise. For prediction of performance sensor noise should not be in-

cluded in the performance measure as the true performance of the system is of interest. Using Figure 3.8 the

performance of the aircraft in the �ight tests and simulations is similar. The match between the simulations

and experimental results indicates that these simulations provide a good estimate of the performance of the

vehicle.

The simulations were also performed using the same controller and 3 m/s steady wind and severe turbulence.

Figure 3.9 shows the median cross track, velocity, and altitude errors.


Figure 3.9: Simulated performance (3 m/s steady wind, severe turbulence) with sensor noise added to

recorded simulation measurements

Again in this case the experimental results indicate a larger error in both velocity and altitude. Comparing

the results from the medium and severe turbulence simulations, the cross track error does not change much

with changes in turbulence. Both velocity and altitude error increase with the increased turbulence intensity.

The comparison of results indicates that the velocity and altitude errors are sensitive to changes in turbulence,

and that cross track error is insensitive to changes in turbulence intensity.

Examining the distribution of blue and red in the curves allows for an initial estimate of the sensitivity of

these performance measures to model form uncertainty to be made. Examining the cross track error curves

the most extreme values occur when both Cn and Cy uncertainties are at maximum or minimum values,

with results near the center of the performance range when they are at opposite bounds. The clustering

of the uncertainties indicates that both of these terms have an e�ect on the cross-track error. Examining

the altitude error curves the only section where there is a clear distribution between upper and lower error

bounds is the Cx uncertainty. The distribution of the results indicates that altitude error is most sensitivity

to the Cx uncertainty. The velocity error also is primarily sensitive to changes in the Cx uncertainty. The

curves further indicate that that there is a secondary sensitivity to the Cz uncertainty.


Table 3.1: Number of path excursions per hour from nondeterministic simulations (25 m radius limit)

Steady Wind Speed Low Turbulence Medium Turbulence Severe Turbulence

0 m/s [0,0] [0,0] [0,0]

1 m/s [0,0] [0,0] [0,0]

2 m/s [0,0] [0,0] [0,0]

3 m/s [0,0] [0,0] [0,0]

4 m/s [0,0] [0,0] [0,0]

5 m/s [0,0] [0,0] [0,0]

6 m/s [0,0] [0,0] [0,0]

7 m/s [0,0] [0,0] [0,0.002]

To examine how di�erent steady winds a�ect the �ight of the aircraft, simulations were run for the system

with the same controller used in �ight for low, medium and high atmospheric turbulence with steady winds

ranging from 0 to 7 m/s. For these simulations the performance measure of interest was the ability of the

aircraft to remain within a 25 m radius of the commanded path. The performance requirement comes from

the need to avoid potential obstacles the path. Table 3.1 shows the e�ect of steady wind and turbulence on

number of path excursions. The results in the table show the average number of path excursions per hour

for both the best and the worst case combinations of model form uncertainty. These results indicate that

the aircraft can remain within a 25 m radius around the path.

The simulations were repeated with the performance requirement rede�ned as an excursion from a 15 m

radius around the path. The tighter bounds on the path could allow for the aircraft to �y closer to potential

obstacles. Table 3.2 shows the e�ect of steady wind and turbulence on probability of excursions with tighter

bounds on the �ight path. Assuming 0.1 excursions per hour as acceptable, the aircraft can meet the

performance goal for all the examined wind speeds under low turbulence, for up to 5 m/s steady wind and


Table 3.2: Number of path excursions per hour from nondeterministic simulations (15 m radius limit)

Steady Wind Speed Low Turbulence Medium Turbulence Severe Turbulence

0 m/s [0,0] [0,0] [0,0.068]

1 m/s [0,0] [0,0] [0,0.092]

2 m/s [0,0] [0,0] [0,0.136]

3 m/s [0,0] [0,0] [0,0.31]

4 m/s [0,0] [0,0.002] [0,0.868]

5 m/s [0,0] [0,0.036] [0,2.44]

6 m/s [0,0] [0,0.324] [0,7.762]

7 m/s [0,0.028] [0,3.886] [0.002,25.342]

medium turbulence, and for up to 1 m/s steady wind with severe turbulence. In the other cases, it is not

clear if the performance will be acceptable. In these cases the best case model-form uncertainty leads to zero

excursions, and the worst case has a large number of loss of control events per �ight hour. If the number of

excursions per hour has to be less than 0.1, the model-form uncertainty would need to be reduced. In order

to reduced the model form uncertainty, a more complex aircraft model that includes additional dynamics

will have to be used. Sensitivity analysis can be utilized to help determine which portions of the model

should be changed to reduce the bounds by the greatest amount.

The very large number of path excursions predicted with the high atmospheric turbulence is likely due to

the atmospheric disturbances causing the aircraft states to enter regions where the model form uncertainty

bounds are large.

A �nal set of simulations were conducted where the uncertainty was allowed to vary between the upper and

lower bounds at a bounded rate, the third method of sampling the model form uncertainty from Section 3.7.

These simulations again used the same controller gains as in the experimental �ight, and using a 3 m/s steady


wind with severe turbulence. Figure 3.10 shows the average cross track, velocity, and altitude errors with

Figure 3.10: Time varying model form uncertainty (3 m/s steady wind, severe turbulence) with sensor noise

added to recorded simulation measurements

time varying model form uncertainty. The simulations were run for 2000 epistemic uncertainty time histories,

with each time history simulated for 500 one hour aleatory time histories. These simulations result in 2000

CDFs, one for each set of epistemic time histories. Figure 3.10 only shows the maximum and minimum

values from set of CDFs found in the simulation. There were no loss of control events per hour for these

simulations when a 25 m radius was used. Comparing these results with those of the constant model form

uncertainty, the results from the time varying uncertainty are within the upper and lower bounds of results

obtained with constant model form uncertainty. The time varying model form uncertainty does not degrade

the performance beyond that from the worst case bounds. The results being bounded by the worst case result

indicates that the uncertainty variation is su�ciently slow that the changes in coe�cients does not lead to

a degradation in performance. The results also indicate that the outputs are not highly nonlinear functions

of the model form uncertainty, with the maximum/minimum response occurring at maximum/minimum

uncertainties.


3.10 Conclusion

This paper presented a general method for nondeterministic simulation of small unmanned systems. This

method estimates the aerodynamic parameters and quanti�es the model form uncertainty. A method to

sample the time varying model form uncertainty was developed allowing for the uncertainty value to change

during the simulation. The uncertainties were then used in nondeterministic simulations producing a p-box

representing the aleatory and epistemic uncertainty in aircraft performance which can be used for predicting

the probability of loss of control.

The nondeterministic simulation results in this paper are based on a model of the E-SPAARO UAV. The

results could be used to estimate atmospheric conditions for safe �ight operations.

Results that allowed the model form uncertainty level to vary between the prediction intervals at a bounded

rate during the simulations were also presented. The results from the time varying uncertainty are within the

upper and lower bounds of results obtained by sampling the corner point of the uncertainty. This indicates

that the time varying model form uncertainty does not degrade the performance beyond that from the worst

case bounds.

Acknowledgments

The authors greatfully acknowledge Mr. Robert J. Hanley, Director, USN & USMC Airworthiness and

CYBERSAFE O�ce, AIR-4.0P. This research is funded by prime contract #HC1047-05-D-4005, purchase

order APSC01797 (Clearance statement TBD).

Bibliography



Exhibit , 2008.



Inc., 2015.



2011, pp. 147�161.



1999.





2010, pp. 222�234.

59

















pp. 333 � 357.
















AIAA, Reston, VA.







Vol. 200, No. 25-28, June 2011, pp. 2131�2144.






pp. 681�691.















FORCE BASE, 1982.













Chapter 4

Sensitivity of Input Epistemic

Uncertainty on Uncertainty Quanti�ed

Performance Estimates Using

Non-Deterministic Simulations

4.1 Title

Sensitivity of Input Epistemic Uncertainty on Nondeterministic Performance Estimates Using Non-Deterministic

Simulations

By: Lawrence Hale, Mayuresh Patil, Christopher J. Roy

64


4.2 Abstract

This paper examines various sensitivity analysis methods which can be used to determine the relative im-

portance of input epistemic uncertainties on the uncertainty quanti�ed performance estimate. These results

would then indicate which input uncertainties would most merit additional study. The following existing

sensitivity analysis methods are examined and described: local sensitivity analysis by �nite di�erence, scatter

plot analysis, variance based analysis, and p-box based analysis. As none of these methods are ideally suited

for analysis of dynamic systems with epistemic uncertainty, an alternate method is proposed. This method

uses aspects of both local sensitivity analysis and p-box based analysis to provide improved computational

speed while removing dependence on the assumed nominal model parameters.

4.3 Introduction

The use of modeling and simulation in engineering applications is becoming increasingly common as com-

putational power has increased. The results of these models are used to inform design decisions and provide

uncertainty quanti�ed performance estimates. In order to correctly report the simulation output the un-

certainties in the prediction must be taken into account. These uncertainties can be caused by a variety of

factors including modeling errors, unknown modeling parameters, and variations from manufacturing di�er-

ences. These errors can be roughly categorized as aleatory (random), epistemic (lack of knowledge), or mixed

uncertainties. In this application epistemic uncertainty is characterized as an interval. This characterization

is utilized to represent the lack of knowledge associated with epistemic uncertainties. An common example

of a mixed uncertainty is a random variable where one does not know the mean precisely and thus treats it

as an interval. The approaches discussed herein are also applicable to such mixed uncertainties by treating

the input as an aleatory uncertainty and treating the mean value as an interval (i.e., the single uncertain

input is treated as two separate inputs: one aleatory and one epistemic). In some cases the uncertainties

in simulation results are high enough that they do not provide acceptable estimate of performance. In such


cases there is a need to reduce the level of input epistemic uncertainties by adding information (including

doing more experiments) in order to reduce the output uncertainty. Adding information has an associated

cost and it is important to ensure that any expenditures are made in an informed manner.

A variety of sensitivity analysis methods may be used to determine the sensitivity of output to input epis-

temic uncertainty. The formulation of these methods should be consistent with the treatment of epistemic

uncertainties to ensure consistent results. This paper examines a range of existing methods for performing

sensitivity analyses of epistemic uncertainties and compares the number of calculations needed to perform

the analysis (computational cost) while indicating any limitations that may be associated with the method.

The paper then proposes and examines a new method for performing sensitivity analysis on dynamic systems

with epistemic uncertainty. This method is consistent with the format used in the estimation of uncertainty

quanti�ed performance.

Section 4.4 of this paper further describes the types of uncertainty that may be present on the input param-

eters and how these are represented in the system output. Section 4.5 discusses various sensitivity analysis

methods and where they are applicable. Section 4.6 describes the proposed sensitivity analysis method for

uncertainty quanti�ed performance estimates. Section 4.7 provides an example application of the proposed

method applied to uncertainty quanti�ed performance estimates.

4.4 Uncertainty

This paper examines methods to calculate the sensitivity of the system to epistemic uncertainties. A general

system containing both aleatory and epistemic uncertainties is considered. Aleatory uncertainty is de�ned as

"uncertainty due to inherent randomness" and epistemic uncertainty is de�ned as "uncertainty due to lack

of knowledge."[22] These de�nitions inform the method used to represent the uncertainties in simulations.

Aleatory uncertainty is modeled with a known probability distribution, (Gaussian, uniform, etc.) based on

the known variations of the parameter. Epistemic uncertainties can be represented in various forms. We


are choosing to represent pure epistemic uncertainty as an interval with precise bounds but no associated

probability distribution. In cases where a parameter has both aleatory and epistemic uncertainty, the

uncertainty is represented by a probability box (p-box) as shown in Figure 4.1.

Figure 4.1: p-box Representing Aleatory and Epistemic Uncertainties

The p-box represents the upper and lower bounds on the cumulative distribution function (CDF) of the

parameter. The width between the curves indicates the epistemic uncertainty and the shape of the curves

represents the aleatory uncertainty. If the epistemic uncertainty were removed only one curve would remain

(a single CDF) and if the aleatory uncertainty were removed the plot would show two vertical lines some

distance apart (i.e., an interval).

4.4.1 Estimation of Output Uncertainty

To estimate the uncertainty in the outputs of the model the parameter uncertainties are propagated through

the system. In systems where there are both epistemic and aleatory uncertainties the uncertainties must be

propagated separately as describe by Ferson (1996)[31].


To propagate the uncertainties separately a nested formulation is used with the outer loop sampling the

epistemic uncertainties and the inner loop sampling from the aleatory uncertainties. In each iteration of the

outer loop a CDF is formed based on the aleatory uncertainty samples. This is repeated for each sample of

epistemic uncertainties resulting in a family of CDFs. The maximum and minimum values from this family

are then used to produce a p-box as discussed previously[22]. The goal of this work is to examine methods

which can be used to determine what uncertainty is contributing most to the width of the output p-box (the

epistemic uncertainty).

4.5 Existing Sensitivity Analysis Methods

There are a variety of existing methods for performing sensitivity analysis on systems in general that can be

applied to systems with epistemic and aleatory uncertainties. In this study we are primarily interested in the

sensitivity of the system output to epistemic uncertainty in the inputs. This section examines how local sen-

sitivity analysis by �nite di�erence, scatter plot analysis, variance based analysis, and p-box based analysis

could be applied to the analysis of epistemic uncertainties. As part of a recent NASA uncertainty quanti�-

cation challenge problem [32] a number of papers applied a subset of these methods to obtain sensitivities

[33, 34, 35, 36, 37, 38].

4.5.1 Local Sensitivity Analysis

Local sensitivities provide a basic method to quickly calculate sensitivities. This method utilizes �nite

di�erences to calculate the sensitivity of the outputs to the epistemic uncertainties. The formulation does

not require the calculation of a p-box for the sensitivity to be estimated. Instead, the method propagates

the nominal value through the simulation to obtain the nominal output. The values of each input epistemic

uncertainties is then varied by a small amount and the output is recalculated. This is done for each of

the epistemic uncertainty providing an estimate of the sensitivity of output uncertainty to input epistemic


uncertainty:

Si =

∣∣∣∣ ∂y∂Θi

∣∣∣∣ (4.1)

where y is the simulation output and Θ is epistemic parameter. The index i indicates the epistemic input

that is being studied in this case while all the other epistemic inputs have nominal values.

To provide a metric that measures the epistemic uncertainty associated with each variable, the sensitivity is

multiplied by the associated epistemic uncertainty range. This corrects for cases where the uncertain slope

is high but the uncertain range of the input is very small, resulting in a reduced e�ect on the output. The

drawback to this approach is that it assumes that the local sensitivity is indicative of the variation over

the full range of uncertain inputs. To normalize this result in a manner similar to that used in subsequent

methods the value is normalized as:

Pri =RaiSi

Rp(4.2)

In this case Pr is the percent reduction in area of the system response probability box, Rai are the ranges of

the input epistemic uncertainty, and Rp is the average P-box width including all of the uncertainties. This

additional step is used so that the results from this method can be compared to the other methods. If this

were the only method utilized this step would not be required. The reduction in area of the probability box

provides an indication of the reduction of epistemic uncertainty.

4.5.2 Scatter Plot Analysis

Scatter plot analysis examines a number of scatter plots showing one of the outputs as a function of one

of the epistemic inputs [39]. For the scatter plots, sampling is performed across the epistemic and aleatory

uncertainties in the same manner as would be to create a p-box. For each sample, the system response

quantity is calculated providing a data point that is used to create the scatter plot.

A linear regression is used to provide a measure of how sensitivity the outputs are to a variety of inputs .


From this regression a slope Si is formed as

Si =

∣∣∣∣ ∂y∂Θi

∣∣∣∣ (4.3)

The value here is di�erent than that obtained from local sensitivities as it incorporates points from the global

exploration of the input uncertainty space. The relative costs can then be normalized in the same manner

as that used in equation 4.2.

From the regression we can also obtain an R2 value to indicate how much of the observed variation in the

output can be represented by the �t. By summing the R2 values found from each of the �ts, the total amount

of observable variation is calculated. If this value is near 1 then there is evidence that this �t will provide a

good estimate. If the sum of R2 values is not near 1, the �t is not good and there is no guarantee that the

estimate will be correct.

4.5.3 Variance Based Analysis

Analysis of Variance (ANOVA) is a method commonly used to examine sensitivities of probabilistic uncer-

tainties. To perform analysis of epistemic variables a uniform probability distribution is assumed for the

uncertainty. The variance of the simulation outputs due to variations in the input samples are used to

estimate the sensitivity of the system. The general ANOVA formulation for the calculation of the Sobol

indices [40] is provided in Saltelli et al. [41]. Sobol indices provide both a �rst-order and total e�ect metric.

The total e�ect metric measures the contribution to the output variance from an input including any caused

by interaction with other inputs. This is the primary metric of interest in these studies as it indicates what

input will provide the greatest reduction in output variance. The general total e�ect equation is given by:

PrTi =EX∼i(V arXi(Y |X∼i))

V ar(Y )(4.4)

which is the expected value of the variance of Y (the simulation output), given all of the input uncertainties

except Xi divided by the total variance. To examine the change in variance only from the input epistemic

uncertainties, the mean value from a set of aleatory samples is used as the simulation output. The expected


value in Equation 4.4 can be approximated by the equation:

EX∼i(V arXi(Y |X∼i)) ≈1

2N

N∑j=1

(f(A)j − f(AiB)j)2 (4.5)

where f(.) is function used to calculate the system response quantity. The matrices A and B are k (the

number of input variables) by N (the number of samples), where the columns of these matrices are the

samples taken for each of the epistemic uncertainties. The AiB matrix is formed by replacing the i-th column

of the A matrix with the i-th column of the B matrix.

The results of these calculations indicate the source of output variance of the epistemic uncertainty due to

each of the input uncertainties. This measure is based on assumed uniform probabilities for the epistemic

uncertainty.

4.5.4 p-box Based Analysis

p-box based sensitivity analysis is similar to the ANOVA based analysis but directly examines how the width

of the p-box is a�ected by the various input epistemic uncertainties. This method was describe by Guo and

Du [42] and can be demonstrated by Figure 4.2.

Figure 4.2: Diagram Illustrating P-box Based Analysis

In this method the �rst step is calculating the p-boxes of the outputs of interest utilizing the full uncertainty

range (the black lines in the rightmost graph in �gure 4.2). The area between these curves is then calculated.

The simulation is then repeated k times (where k is the number of input parameters with epistemic uncer-


tainty). In each of these repeated simulations one of the uncertain input epistemic parameters is assumed

to be a nominal value (usually taken as the center of the bound) and the outputs are recalculated (the red

lines in �gure 4.2). This reduced area is then calculated for each output p-box. The percent change in the

area in these probability boxes indicates the total sensitivity of the output uncertainty to the given input

uncertainty with a greater change indicating a greater sensitivity to that input.

4.5.5 Comparison of Methods

The �rst step in comparing these methods is a comparison of the number of simulations required for each

method (an indication of computational cost). Table 4.1 lists the number of simulations needed for the

various methods. In this table d is the number of inputs represented by epistemic variables variables, E is

the number of samples used to propagate epistemic variables and A is the number of samples used for aleatory

variables. In general the value of d is much smaller than that of either A or E in order to accurately sample

the uncertainty. This table indicates that the local sensitivity analysis will require the fewest simulations,

followed by scatter plot analysis, with the number of simulations for the variance based and p-box based

analysis being very high.

Table 4.1: Comparison of Number of Required Simulations

Method Computational Cost

Local Sensitivity d+1

Scatter Plot Analysis E*A

Variance Based Analysis E*A*(d+2)

p-box Based Analysis E*A*(d+1)

An additional consideration is the expected accuracy of the results from these methods. The local sensitivity

analysis can be very accurate but only in very speci�c cases. The method is accurate when the input to output

relation is linear or that the uncertainty range is small enough that the nonlinearities are not important.


Additionally the method does not consider any coupling between the various inputs. Unfortunately in most

cases it can not be assumed that these conditions will be met thus leading to incorrect sensitivity results.

Scatter plot analysis is in general the next most computationally e�cient method. It can also provide correct

result but again in speci�c conditions. In this method the accuracy of the result is dependent on the ability

to produce a good �t for the data from the scatter plot. This ability can be degraded if there is a large

amount of cross correlation, or if the aleatory uncertainties contribute a signi�cant amount to the total

uncertainty. Aleatory uncertainties are not considered in this study as they are not reducible, however they

are included in the simulations since as their interaction with the epistemic uncertainties could a�ect the

simulation results. The R2 value check should be performed with this method to determine if it will provide

a reasonable estimate of the relative sensitivity. If the sum of the R2 values is not near 1, then a more

advanced technique should be used.

Variance based analysis is the most common method used for sensitivity analysis of aleatory uncertainties.

In studies that focus on epistemic uncertainties, however, this method examines changes in variance, a value

that is not associated with interval-characterized epistemic uncertainties. The method uses an assumed

probability distribution for the epistemic variables, often chosen as uniform. The measurement of variance

is based on the scatter of the system response quantities and produces acceptable results in most cases.

p-box based analysis is an alternative to variance based analysis with a similar computational cost but tailored

for used with epistemic uncertainties. In this method the assumed nominal value used in the recalculation

can have a signi�cant e�ect on the results. If the nominal value is not near the true value and there is

dependence between variables, the results may have signi�cant error. In a case where these values are chosen

equal to or near the true value this method, the method will give a direct indication of how much reduction

in output epistemic uncertainty would be removed if the input was known.


4.6 Proposed Sensitivity Analysis Method

Amodi�ed sensitivity analysis method was developed to provide sensitivity analysis results that are consistent

with the uncertainty analysis. The uncertainty propagation method for calculating the nondeterministic

performance estimates is detailed in [27]. In this paper it is determined that the model form uncertainties

(epistemic input uncertainty) are dependent on the aircraft states and vary with time. The uncertainty

could be sampled in several ways. We utilized a �xed uncertainty level sampling. For the �xed uncertainty

sampling the corner points of a hypercube representing the combinations of upper and lower uncertainty

bounds were sampled.

An example of the output from the analysis with �xed bounds is shown in Figure 4.3. The blue and red line

Figure 4.3: Simulated performance (3 m/s steady wind, sever turbulence) with sensor noise added to recorded

simulation measurements

segments represent the corner points of the model form uncertainty. The curves are separated into 6 sections

with blue color representing the lower bound and red representing the upper bound. The corresponding

uncertainties are labeled on the right side of the plot.

The proposed method uses these results to determine which uncertainties have the greatest impact on the


nondeterministic performance estimates. To perform this analysis, the average width of the bounding p-box

(largest and smallest values) is calculated in the same manner as in p-box based analysis. The average width

of each combination curves where only one of the bounds have been changed is calculated. This analysis

provided a number of calculated widths for each of the uncertainties (one associated with each combination

of the other uncertainty bounds). By comparing the widths associated with each uncertainty to the full

uncertainty the total e�ect of each uncertainty (normalized by the total width) can be calculated. For each

uncertainty, multiple possible widths are calculated providing a range of possible reductions in uncertainty.

The actual reduction in uncertainty is not known exactly due to possible interaction between the various

uncertainties.

The method described above provides several advantages over the current sensitivity analysis methods. The

proposed method is most similar to the p-box method which means that the method is aligned with how

epistemic and aleatory uncertainties are sampled. The proposed method also does not assume that the

nominal model is correct and instead estimates the sensitivity for each uncertainty at various combinations

of the other uncertainties. The proposed method also requires signi�cantly fewer simulations than p-box

based analysis. The proposed method requires A*2d simulations compared to E ∗ A ∗ (d + 1). The value

of E depend on the number of samples used for the epistemic uncertainties. If only 5 samples were utilized

fore each of the epistemic uncertainties, then the value of E would be 5d. In the limit when the minimum of

two samples are used E = 2d. Additionally if the �xed sampling method is used for uncertainty quanti�ed

analysis, the proposed method does not require any additional information or further simulations.

A requirement of this method is that the uncertainty be somewhat smooth such that intermediate uncertainty

value will not result in estimated performance signi�cantly outside the performance obtained at endpoints.

For aircraft systems, changes in performance estimates are associated with the change in uncertainty such

that it is unlikely that an intermediate value would have signi�cantly changed performance. This suggests

that the proposed method can be utilized to perform sensitivity analysis of nondeterministic performance

estimates for aircraft system.


4.7 Example Analysis

4.7.1 Sensitivity of Aircraft Modes

A comparison of these methods was performed based on an a linear longitudinal aircraft motion model and

associated coe�cient uncertainty. The model is based on the Virginia Tech E-SPAARO Unmanned Aerial

Vehicle (UAV) shown in Figure 4.4. The E-SPAARO UAV is a small remotely piloted vehicle used by the

Figure 4.4: E-SPAARO UAV

Nonlinear Systems Lab at Virginia Tech for a variety of research. For this study the mass and geometric

properties are considered aleatory with a normal distribution, shown in Table 4.2, and the aerodynamic

parameters are considered epistemic with bounds, shown in Table 4.3 The velocity (V ) is set as 20 m/s at

sea level for the analysis. These parameters were selected based on measured mass properties and estimates

of the aerodynamic parameters based on �ight tests. These aerodynamic parameter bounds have been

simpli�ed for this analysis.

The example linearized longitudinal system dynamics are given by:

x = Ax+Bη (4.6)

x = [u w q θ]T (4.7)


Table 4.2: Aleatory Parameters

Parameter Mean Variance

Mass M (kg) 21.34 0.2846

Wing Area S (m2) 1.97 0.0024

Chord c (m) 0.53 0.0002

Inertia Iyy (kg-m2) 8.26 0.0426

Table 4.3: Epistemic Parameters

Parameter Lower Bound Upper Bound

CZ0-0.376 -0.2964

CZα -3.507 -3.33165

CZδe 0.03591 0.0378

CX0-0.0399 -0.037905

CXα 0.26334 0.2772

CXα2 1.85535 1.953

Cm00.010374 0.01092

Cmα -0.51135 -0.4857825

Cmq -14.49 -13.7655

Cmδe -0.0147 -0.013965


η = δe (4.8)

A =

qSMV 2CX |eq qs

MV 2CXα |eq 0 −9.81

qSMV 2CZ |eq qs

MV CZα V 0

0 qScIyyV

CmαqSc2

2IyyVCmq 0

0 0 1 0

(4.9)

B = [0 CZδe Cmδe 0]T (4.10)

where:

CX |eq = CX0+ CXαα|eq + CXα2α|2eq (4.11)

CXα |eq = CXα + 2CXα2α|eq (4.12)

CZ |eq = CZ0 + CZαα|eq + CZδe δe|2eq (4.13)

where α is the angle of attack, δe is the elevator de�ection, u and w are body velocities, q is the pitch rate,

θ is the pitch angel, and q is the dynamic pressure.

The analysis is based on the trimmed angle of attack and elevator de�ection. These are found by �nding

the elevator de�ection and angle of attack required for the vehicle to maintain straight and level �ight. The

required elevator de�ection and angle of attack are found by solving the equations:

M ∗ 9.81 = qS(CZ0+ CZαα+ CZδe δe) (4.14)

0 = Cm0+ Cmαα+ Cmδe δe (4.15)

The linearized system model is used to estimate the short period and phugoid damping and short period

natural frequency. The phugoid and short period are the longitudinal modes of an aircraft. The phugoid

mode involves large amplitude variations in speed, pitch angle and altitude, and tends to have low damping.

The short period mode involves an angle of attack variation and tends to be highly damped. The damping

of these two modes along with the short period natural frequency are used in military speci�cation (MIL-

F-8785C) to assess the �ying-quality of aircraft.[43] It is likely that modi�ed versions of these measures will

be used in certi�cation of unmanned systems.


Table 4.4: Phugiod Damping Sensitivity results

Parameter Local Sensitivity Scatter Analysis ANOVA p-box Based Proposed Method

CZ0 0.453 0.458 0.519 0.290 [0.316, 0.361]

CZα 0.158 0.162 0.063 0.025 [0.103, 0.135]

CZδe 0.009 0.007 0.0002 0.004 [0.003, 0.011]

CX00.359 0.369 0.320 0.162 [0.245, 0.290]

CXα 0.011 0.031 0.0003 0.005 [0.004, 0.013]

CXα2 0.015 0.037 0.001 0.002 [0.008, 0.015]

Cm0 0.017 0.016 0.001 0.009 [0.010, 0.016]

Cmα 0.124 0.146 0.040 0.017 [0.076, 0.112]

Cmq 0.185 0.187 0.084 0.038 [0.124, 0.153]

Cmδe 0.009 0.032 0.0002 0.002 [0.003, 0.011]

The sensitivity analysis results for the phugoid damping are shown in Table 4.4. The results indicate the

phugoid damping is most sensitive to changes in CZ0followed by CX0

. Comparing the results from the

di�erent methods, the parameters with the most impact on the output are the same. The proposed method

additionally indicates that there is uncertainty in the order of importance of the parameters (CZα , Cmq )

because it depends on the values of the other parameters.

The sensitivity analysis results for the short period damping are shown in Table 4.5. The results indicate the

short period damping is most sensitive to changes in Cmq followed by Cmα . Comparing the results from the

di�erent methods, the parameters with the most impact on the output are the same. The proposed method

additionally indicates that the sensitivity to Cmq is dependent on other parameters.

The sensitivity analysis results for the short period natural frequency are shown in Table 4.6. The sensitivity

of the short period natural frequency is very similar to the short period damping. The results for the short


Table 4.5: Short Period Damping Sensitivity results


CZ0 0.005 0.011 0.0001 0.001 [0.004, 0.006]

CZα 0.186 0.179 0.080 0.141 [0.137, 0.206]

CZδe 0.0001 0.023 0 0 [0, 0.0002]

CX00.0004 0.017 0 0.0002 [0.0003, 0.001]

CXα 0.002 0.004 0 0.001 [0.001, 0.002]

CXα2 0.001 0.005 0 0.001 [0.0005, 0.001]

Cm00.0002 0.022 0 0.0001 [0.0001, 0.0002]

Cmα 0.356 0.355 0.290 0.302 [0.317, 0.341]

Cmq 0.532 0.532 0.640 0.473 [0.460, 0.522]

Cmδe 0.0001 0.032 0 0 [0, 0.0002]


Table 4.6: Short Period Natural Frequency Sensitivity Results


CZ0 0.005 0.012 0.0001 0.001 [0.004, 0.006]

CZα 0.186 0.177 0.078 0.140 [0.154, 0.190]

CZδe 0.0001 0.023 0 0 [0, 0.0001]

CX00.0004 0.017 0 0.0002 [0.0003, 0.0005]

CXα 0.002 0.004 0 0.001 [0.0015, 0.002]

CXα2 0.001 0.005 0 0.001 [0.0006, 0.001]

Cm00.0002 0.022 0 0.0001 [0.0001, 0.0002]

Cmα 0.353 0.355 0.290 0.304 [0.309, 0.352]

Cmq 0.533 0.534 0.640 0.474 [0.460, 0.528]

Cmδe 0.0001 0.032 0 0 [0, 0.0001]


period natural frequency again indicate that the output is most sensitive to changes in Cmq followed by Cmα .

Comparing the results from the di�erent methods, the parameters with the most impact on the output are

the same. The sensitivity of the output is to Cmq is again dependent on other parameters

These results indicate that the proposed method is able to produce results similar to those produced by the

existing methods. It additionally provided information on how variations in other parameters could a�ect

the sensitivity of a given parameter. The proposed method additionally required fewer calculations than any

of the other methods except local sensitivity analysis.

4.7.2 Sensitivity of Nondeterministic Performance Estimates

The proposed sensitivity analysis method was applied to nondeterministic performance estimates of the

Virginia Tech E-SPAARO Unmanned Aerial Vehicle (UAV). In this example a full nonlinear simulation is

used instead of the linearized longitudinal dynamics. Due to the increased complexity of this example, the

existing sensitivity analysis methods were not applied to this example. System identi�cation and model

form uncertainty quanti�cation of this aircraft was performed in [29], and nondeterministic simulations were

performed in [27]. An example of the results from this analysis was shown in Figure 4.3.

The vehicle dynamics are described by

u =Fx + T

M− g sin θ + rv − qw (4.16)

v =FyM


w =FzM


p =Izzl + Ixzn− (Ixz(Iyy − Ixx − Izz)p+ (I2xz + Izz(Izz − Iyy))r)q


q =m+ (Izz − Ixx)pr − Ixz(p2 − r2)

Iyy(4.20)

r =Ixzl + Ixxn− (Ixz(Iyy − Ixx − Izz)r + (I2xz + Ixx(Ixx − Iyy))p)q



The terms p, q, r are the angular rates, u, v, w are the body velocities, and θ and φ are the pitch and roll

angles. The thrust T is set as a function of throttle position δT by the equations

RPM = −0.00522886δ3T + 1.04281575δ2T + 11.215278δT + 2329.50786049 (4.22)

T =0.07806ρπ((0.00508RPM)2 − (0.00508RPM)Vin)0.415

12ρV

2S; (4.23)


propeller pitch and diameter from experimental studies [28]. Vin is the in�ow velocity to the propeller, which

is currently assumed to be the airspeed. The aerodynamic forces and moments are given by:

Fz =1

2ρV 2SCz (4.24)

Fx =1

2ρV 2SCx (4.25)

Fy =1

2ρV 2SCy (4.26)

l =1

2ρV 2SbCL (4.27)

m =1

2ρV 2ScCM (4.28)

n =1

2ρV 2SbCN (4.29)

The V in these equations is the total velocity, S is the wing area, c is the chord, b is the wing span, and ρ

is the air density. The aerodynamic coe�cients are given as:

CZ = CZ0+ CZαα+ CZqq + CZδe δe (4.30)

CX = CX0+ CXαα+ CXα2α

2 (4.31)

Cy = Cy0 + Cyββ + Cypp+ Cyrr + Cyδr δr (4.32)

Cl = Cl0 + Clββ + Clpp+ Clrr + Clδr δr + Clδa δa + Cmδe δe (4.33)

Cm = Cm0+ Cmαα+ Cmqq + Cmδe δe + Cmδr δr (4.34)

Cn = Cn0+ Cnββ + Cnpp+ Cnrr + Cnδr δr + Cnδa δa + Cmδe δe (4.35)


Table 4.7: Normalized Output Uncertainty from Input Uncertainty (Maximum and Minimum)

Parameter Cross Track Error Altitude Error Velocity Error

CZ [0.0283, 0.035] [0.0145, 0.1227] [0.0853, 0.3324]

CX [0.0017, 0.0034] [0.5843, 0.7427] [0.5108, 0.8083]

Cm [0.0379, 0.0432] [0.011, 0.0864] [0.02, 0.1412]

CY [0.3884, 0.4019] [0.0711, 0.1525] [0.0098, 0.0628]

Cl [0.0314, 0.0347] [0.0121, 0.0401] [0.0067, 0.0314]

Cn [0.4891, 0.5065] [0.0794, 0.1509] [0.0111, 0.0581]

These equations consist of the aircraft states and the aerodynamic parameters. These parameters are the

terms in Θ that are tuned in the output error method.

The input uncertainties of interest are the total aerodynamic coe�cients, CX , CY , CZ , Cl, Cm, and Cn.

Additional information on the coe�cient values, uncertainty bounds and nondeterministic simulation results

are available in [27]. The sensitivity analysis performed here is base on the results from the 3 m/s stead

wind sever turbulence case. As there are 6 uncertain epistemic terms the plot has 26 = 64 curves. Thirty

two possible reductions in uncertainty can be calculated for each uncertain input by comparing curves where

only one uncertainty has changed.

Table 4.7 lists the minimum and maximum average widths of the p-boxes formed when only that uncertainty

is changed, normalized by the overall p-box width, for various performance measures. Examining the table a

number of things can be deduced. The Cn term is the most important epistemic uncertainty for the predicted

cross track error. The range between the upper and lower values is small, indicating that the results are not

sensitive to the value of the other parameters. CX is the most important term for the predicted altitude error

while most of the other terms have lower ranges. The range between the upper and lower values is moderate,

indicating that the other parameter values have a minor impact on how much uncertainty reduction a given


parameter could provide. CX is the most important input for the sensitivity of the predicted velocity error.

The range of sensitivity values in this case is quite large, especially for CX and CZ indicating that the values

of other parameters has a strong in�uence on how much reduction in uncertainty could be expected. Even

with this uncertainty, the order of which parameters are most important does not signi�cantly change.

4.8 Conclusion

This paper discussed a number of existing sensitivity analysis methods that could be applied to perform

sensitivity analysis on the nondeterministic performance estimates. As none of these methods were ide-

ally suited to the analysis of nondeterministic performance estimates from non-deterministic simulations, a

modi�ed method, consistent with the formulation used in the estimation of nondeterministic performance,

is proposed. This method does not rely on a nominal set of results and better captures the nature of the

epistemic uncertainties by giving a range of possible sensitivities depending on the values of the other param-

eters. Additionally the proposed method requires signi�cantly fewer simulations than required for standard

p-box based analysis.

The existing methods along with the proposed method were applied to an example modal analysis problem

for the linearized longitudinal dynamics of the E-SPAARO UAV. The results indicated that each method

provided qualitatively similar results. The proposed method provided additional information on how varia-

tions in the value of other parameters could a�ect the estimated sensitivity. Finally, the proposed sensitivity

analysis method provided results based on less simulations than any of the other methods except local

sensitivity analysis.

The proposed sensitivity analysis method was applied to uncertainty quanti�ed performance estimates of

the full nonlinear model of the E-SPAARO UAV. The results were obtained without the need for additional

simulations and provide information on which model form uncertainties have the greatest impact on the

overall performance. The sensitivity analysis results also provided insight into how signi�cant a change in


value of the other parameters is to the sensitivity analysis results.

Bibliography



Exhibit , 2008.



Inc., 2015.



2011, pp. 147�161.



1999.





2010, pp. 222�234.

87

















pp. 333 � 357.
















AIAA, Reston, VA.







Vol. 200, No. 25-28, June 2011, pp. 2131�2144.






pp. 681�691.















FORCE BASE, 1982.













Chapter 5

Conclusions

In Chapter 2, a general method for the identi�cation of aerodynamic parameters and quanti�cation of model

form uncertainty was presented. Parameter Identi�cation is accomplished utilizing the output error method,

a method that has been used in may prior studies. To calculate the model form uncertainty, the data

used in model validation is utilized. The validation data is compared to the identi�ed model through a

modi�ed version of the Equation Error method. The error between the coe�cients predicted by the model

and those from the experimental data is calculated for each data sample. These errors are related to aircraft

state through a multiple linear regression. The coe�cients from the regression should have small values in

the regions where data is available for states that were used in the identi�cation. If additional terms are

used, the values of these coe�cients can provide some insight into additional states that would improve the

system identi�cation results. Uncertainty bounds on the curve �t are calculated using prediction intervals.

Prediction intervals account for the spread of the data where it is available and the uncertainty increases

in regions with less data. The method provides good estimates for uncertainty in regions where data is

available but is not guaranteed to capture the true model in regions with sparse data. Allowing the size

of the intervals to increase in regions with sparse data, the true model is represented in a heuristic sense.

To prevent the uncertainty bounds from applying a calibration to the model, the bounds are set such that

92

Lawrence E. Hale Chapter 5. Conclusions 93

the upper bound will be greater than 0 and the lower bound is lower than 0. Without this condition the

simulation including the uncertainty might deviate from the identi�ed model in the full �ight envelope.

The method described above was examined using both synthetic and true �ight data. The example appli-

cation demonstrated the ability of the method to both estimate aerodynamic parameters and provide an

associated model form uncertainty.

In Chapter 3, a method for performance analysis utilizing nondeterministic simulation of dynamic system

with model from uncertainty was presented. This method was based on the uncertainty quanti�cation

previously detailed. To sample from these uncertainties, a number of potential sampling techniques were

presented. The primary method examined �xed the uncertainty at a single level between the bounds.

By �xing the uncertainty level, fewer samples are required to sample the uncertainties. The hypercube

corner points with the various combinations of maximum/minimum uncertainty level can be sampled to

give am approximate worst case analysis. The other method for sampling from the uncertainty that was

considered was allowing the uncertainties to vary randomly at a bounded rate. To estimate the rate at

which the uncertainty would vary, the time history of the errors between the model and �ight test data

were �rst �ltered. This �ltering was performed to minimize the impact of sensor noise on the system. The

change in the error between each time step was then calculated and used to estimate the rate of change

of the uncertainty. To correct for the di�erence in time step between the �ight test and nondeterministic

simulations, the rate of change was scaled based on the square root of the di�erence in the sample times.

The uncertainty bounds were set as re�ecting boundaries, to keep the uncertainty within the bound, and to

prevent the uncertainty from sticking to the bounds. To sample the time varying uncertainty, many more

simulations are required.

The results produced from the sampled uncertainties are presented in a p-box. p-boxes allow for both

aleatory and epistemic uncertainties to be represented on a single plot. The shape of the curves forming the

box represent the aleatory uncertainties and the width of the box represents the epistemic uncertainty. When

using the �xed uncertainty level sampling method and sampling from the corner points of the hyper cube, all


of the curves can be plotted along with the uncertainty sampling. This allows for visual sensitivity analysis

to be performed on the results. Sensitivity analysis can inform what additional test should be performed to

provide the maximum reduction in uncertainty in the performance of the system.

The performance analysis method was demonstrated based on �ight data. In this analysis the performance

estimates are provided and compared to performance from �ight testing.

In Chapter 4, a method for sensitivity analysis of uncertainty quanti�ed performance estimates was presented.

The method is similar to the p-box method, but only requires the data calculated in the uncertainty quanti�ed

performance estimates results. By utilizing these results, the analysis can be performed without additional

simulations. Additionally the proposed method does not make an assumption of what the true value of the

uncertain parameters. Instead the sensitivity is presented as a range, with the true value dependent on the

values of the other parameters. The results are normalized by the maximum width of the uncertainty to

provide a better idea of how much uncertainty is present. The sensitivity analysis provides information on

which of the uncertainties create the greatest uncertainty in the output. This information can be utilized

to inform what addition tests should be performed to re�ne the model and reduce the uncertainty in the

performance estimates.

Bibliography



Exhibit , 2008.



Inc., 2015.



2011, pp. 147�161.



1999.





2010, pp. 222�234.

95

















pp. 333 � 357.
















AIAA, Reston, VA.







Vol. 200, No. 25-28, June 2011, pp. 2131�2144.






pp. 681�691.















FORCE BASE, 1982.













aerodynamic uncertainty quanti cation and estimation of

Documents