model-based rigorous uncertainty quantification in complex ... · – determination of mean...

Model-Based Rigorous UncertaintyQuantification in Complex Systems

Michael OrtizMichael OrtizCalifornia Institute of Technology

Applied Mathematics SeminarWarwick Mathematics Institute, June 23, 2010

Work done in collaboration with: Marc Adams, Addis Kadani, Bo Li, Mike McKerns, Ali Lashgari, Jon Mihaly, Houman

PSAAP: Predictive Science Academic Alliance Program

o , e c e s, as ga , Jo a y, ou aOwhadi, G. Ravichandran, Ares J. Rosakis, Tim Sullivan

Michael OrtizWarwick 06/23/10- 1

ASC/PSAAP Centers


Michael OrtizPSAAP Peer Review, 10/23/2008- 2

Quantification of margins and uncertainties (QMU)( )

• Aim: Predict mean performance and uncertainty in the behavior of complex physical/engineered systems

• Example: Short-term weather prediction,– Old: Prediction that tomorrow will rain in Warwick…

New: Guarantee same with 99% confidence– New: Guarantee same with 99% confidence…

• QMU is important for achieving confidence in high-consequence decisions, designs

• Paradigm shift in experimental science, modeling and simulation, scientific computing (predictive science):

Deterministic Non deterministic systems– Deterministic → Non-deterministic systems– Mean performance → Mean performance + uncertainties– Tight integration of experiments, theory and simulation



– Robust design: Design systems to minimize uncertainty– Resource allocation: Eliminate main uncertainty sources

Certification view of QMU

system inputs

performance measures

responsefunctioninputs measures

• Random variables

• Known or

• Observables• Subject to

performance• Known or unknown pdfs

• Controllable,

performance specs

• Random due

S t bl k b

uncontrollable, unknown-unknowns

to randomnessof inputs or of system



System as black boxunknowns system

Hypervelocity impact as an example of a complex systemp y

Challenge: Predict hypervelocity impact phenomena (10Km/s ) with quantified margins and uncertainties

NASA Ames Research Center E fl h f h l it t tEnergy flash from hypervelocity test

at 7.9 Km/s


Hypervelocity impact test bumper shield(Ernst-Mach Institut, Freiburg Germany)


Hypervelocity impact as an example of a complex systemp y

• Hypervelocity impact is of interest to a broad scientific community: Micrometeorite shields, geological impact cratering…

Hypervelocity impact test of The International Space Station uses


multi-layer micrometeorite shield 200 different types of shield to protect it from impacts


Hypervelocity impact at Caltech



Caltech’s Small Particle Hypervelocity Impact Range facility(A.J. Rosakis, Director)

Hypervelocity impact at Caltech

BREECH PUMP TUBE

LAUNCH TUBE

FLIGHT TUBE

TARGET TANK

Front Spall Cloud

Debris Cloud

Witness Plate

STAGE 1 STAGE 2α

Impactor

TargetWitness Plate

aluminum witness plates replaced by capture media

Target Materials•SteelAluminum Ø 71 mil (1x10-3 in)

by capture media

• Impact Speeds: 2 to 10 km/sI t Obli iti 0 t 80 d

•Aluminum•Tantalum

Impactor Materials•Steel

Ø 71 mil (1x10 3 in) launch tube bore

PSAAP: Predictive Science Academic Alliance Program 8

• Impact Obliquities: 0 to 80 degrees• Impactor Mass: 1 to 50 mg

Steel•Nylon


Hypervelocity impact as system

System Outputs (Y)System inputs (X)

Diagnostics

Conoscope

MetricsProfilometry

Projectile velocity

Projectile mass Conoscope

CGS

yPerforation area

Real-time, full-field back-surface deformation

Projectile mass

Number of target plates

VISAR

S t

deformation

Real-time back-surface velocimetry

I t fl h d b i d

plates

Plate thicknesses

Spectro-photometer

Capture media

Impact flash, debris and spall clouds, spectra over IR to UV range

Debris & spall clouds, Particle consistency

Plate obliquities

Projectile/plate


Michael Ortiz

Capture media Particle consistency, size & velocity vector

j pmaterials

Warwick 06/23/10- 9

Certification view of QMU

• Certification = Rigorous guarantee that complex system will perform safely and according to specifications

• Certification criterion: Probability of failure must be below tolerance,

• Alternative (conservative)Alternative (conservative) certification criterion: Rigorous upper bound of probability of failure must be below tolerancemust be below tolerance,



• Challenge: Rigorous, measurable/computable upper bounds on the probability of failure of systems

Concentration of measure (CoM)

• CoM phenomenon (Levy, 1951): Functions over1951): Functions over high-dimensional spaces with small local oscillations in each variable are almost constant

• CoM gives rise to a classCoM gives rise to a class of probability-of-failure inequalities that can be

d f iused for rigorous certification of complex systems



Paul Pierre Levy (1886-1971)

The diameter of a function

• Oscillation of a function of one variable:

• Function subdiameters:

• Function diameter:evaluation requires



qglobal optimization!

McDiarmid’s inequality

McDiarmid C (1989) “On the method of bounded differences” In J Simmons (ed ) Surveys in



McDiarmid, C. (1989) On the method of bounded differences . In J. Simmons (ed.), Surveys inCombinatorics: London Math. Soc. Lecture Note Series 141. Cambridge University Press.

McDiarmid’s inequality

• Bound does not require distribution of inputs• Bound depends on two numbers: Function



pmean and function diameter!

McDiarmid’s inequality and QMU

Probability of failure Upper bound Failure tolerance

• Equivalent statement (confidence factor CF):

• Rigorous definition of margin (M)



• Rigorous definition of uncertainty (U)

Extension to empirical mean

• Equivalent statement (confidence factor CF):

• Rigorous definition of margin (margin hit!)



Rigorous definition of margin (margin hit!)• Rigorous definition of uncertainty (U = DG)

Extension to multiple performance measures

• Equivalent statement (confidence factor CF):Equivalent statement (confidence factor CF):



Multiple performance measures and unknown mean performancep

E i l t t t t ( fid f t CF)• Equivalent statement (confidence factor CF):



McDiarmid’s inequality and QMU

• Direct evaluation of McDiarmid’s upper bound requires:– Determination of mean performance (e.g., by sampling)– Determination of system diameter by solving a sequence of

global optimization problems

• Viable approach for systems that can be tested cheaplyViable approach for systems that can be tested cheaply• Prohibitively expensive or unfeasible in many cases!

– Tests too costly, time-consuming– Operating conditions are not observable– Political/environmental constraints…

• Alternative: Model-based certification!Alternative: Model based certification!• Challenge: How can we use physics-based models to

achieve rigorous certification with a minimum of testing?



Model-based QMU – The model

System inputs

performance measures

Modelinputs measures



Model-based QMU – McDiarmid

• Two functions that describe the system:– Experiment: G(X) F(X) G(X) M d li f tiExperiment: G(X)– Model: F(X)

• Linearity:

F(X)-G(X) ≡ Modeling-error function

• Corollary: A conservative certification criterion is:

y• Triangular inequality:

Corollary: A conservative certification criterion is:

• E[F]: Model mean; E[F-G]: Model mean error• DF: Model diameter (variability of model)



DF: Model diameter (variability of model)• DF-G: Modeling error (badness of model)


• Working assumptions:– F-G far more regular than F

or G aloneor G alone– Global optimization for DF-G

converges fast– Evaluation of D requires



– Evaluation of DF-G requires few experiments


• Calculation of DF requires exercising model only

• Evaluation of DF-G requires (few) experiments

• Uncertainty Quantification burden mostly shifted to modeling and simulation!

• Rigorous certification not achievable by modeling and simulation alone!



g

Model-based QMU – Implementation



Model-based QMU – Implementation

UQ

Modeling d

ExperimentalS i



andSimulation

Science

Sample UQ Analysis – Ballistic range

• Target/projectile materials:– Target: Al 6061-T6 plates (6”x 6”)– Projectile: S2 Tool steel balls (5/16”)

• Performance measure (output): Perforation area

Target and projectile

e o at o a ea• Admissible operation range:

Perforation area > 0!M d l t (i t ) • Model parameters (inputs): – Plate thickness (0.032’’-0.063’’)– Impact velocity (100-400 m/s)

• Optimal Transportation Meshfree (OTM) solver (sequential)

• Modifier adaption BFGS; in-


Michael Ortiz

• Modifier adaption, BFGS; inhouse UQ pipeline (Mystic)

04/23/10- 32

OTM simulation


BarrelPressure

Barrel Gun

Light detector(velocity )

D

front view

ZY X

Data recorder LeCroy

OptimetMiniConoscan

3000


Michael Ortiz04/23/10 - 33

rear view


400

300

350

400

y (m

/s)

40

50

60

mm

2)

200

250not perforated

perforated

ctile

Velocity

20

30

40

rated area

(m

32 milli‐inch

40 milli‐inch

50 milli‐inch

100

150

30 40 50 60 70

Projec

0

10

0 100 200 300 400

Perfor 63 milli‐inch

30 40 50 60 70Plate thickness (milli‐inch)Impact Velocity (m/s)

Perforation area vs. impact velocity(note small data scatter!)

Perforation/non-perforationbo ndar



(note small data scatter!) boundary




Computed vs. measured perforation area


OTM simulations

/s)

experimental

OTM simulations

a (m

m2 )

velo

city

(m perforation

ratio

n ar

ea

vno perforation

Per

for

OTM simulationsexperimental

thickness (thousands of an inch)

p

velocity (m/s)



Measured vs. computed perforation area


Modelthickness 4.33 mm2

velocity 4.49 mm2operating range(180 400 m/s)

50

60

m2)

diameter DF

ytotal 6.24 mm2

Modelingthickness 4.96 mm2

velocity 2 16 mm2

(180-400 m/s)

20

30

40

ed area (m

m

32 milli‐inch40 milli‐i h

Modeling error DF-G

velocity 2.16 mm2

total 5.41 mm2

Uncertainty DF + DF-G11.65 mm2

0

10

20

Perforate inch

50 milli‐inch

y F F G

Empirical mean <G> 47.77 mm2

Margin hit α (ε’=0.1%) 4.17 mm2

0 100 200 300 400

Impact Velocity (m/s)Confidence factor M/U 3.74

• Perforation can be certified with ~ 1-10-12 confidence!



Perforation can be certified with 1 10 confidence!• Total number of tests ~ 50 → Approach feasible!

Beyond McDiarmid - Extensions

• A number of extensions of McDiarmid may be required in practice:– Some input parameters cannot be controlled– There are unknown input parameters (unknown unknowns)– There is experimental scatter (G defined in probability)There is experimental scatter (G defined in probability)– McDiarmid may not be tight enough (convergence?)– Model itself may be uncertain (epistemic uncertainty)

D t t b il bl ‘ d d’ (l d t )– Data may not be available ‘on demand’ (legacy data)

• Extensions of McDiarmid that address these challenges include:– Martingale inequalities (unknown unknowns, scatter…)– Partitioned McDiarmid inequality (convergent upper bounds)

Optimal Uncertainty Quantification (OUQ)


– Optimal Uncertainty Quantification (OUQ)– Optimal models (least epistemic uncertainty


Beyond McDiarmid – Scatter

• Added challenges:– Experimental

!10

11

12

Plot assumes average of obliquities (a) for all targets tested of a given thickness (h)

Vbl = Ho*( h/Dp)^n / (cos a)^sA = if V<Vbl, 0, K*Dp^2 * {(h/Dp)^p} * {[cos(a)]^u} * {tanh[(V/Vbl)-1]}^m

A = perforation area (mm^2)) scatter!– Impact velocity

uncontrollable!7

8

9

ea (m

m)

A = perforation area (mm 2)h = target thickness (mm)a = imapct obliquity (rad)V = impact speed (km/s)Dp = imactor diameter (mm)Ho, K, n, s, p, u, m are curve fit parametersre

a (m

m2 )

Distribution of Speeds Obtained

0 8

0.9

1

peed

Fit Errors: Uniform = 0.32 Gaussian = 0.08

GaussianMean = 2.49StDev = 0.25

Measured speed distribution

bilit

y

4

5

6

Perfo

ratio

n A

re

1.9mm Data

2.3mm Data

2 6mm Datarfora

tion

a

0.4

0.5

0.6

0.7

0.8

roba

bilit

y of

Low

er S

pat

ive

prob

ab

1

2

32.6mm Data

1.9mm Model

2.3mm Model

2.6mm Model

Impactor area

Per

0

0.1

0.2

0.3

0.4

Cum

ulat

ive

PrC

umul

a

01.7 1.8 1.9 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3

Impact Speed (km/s)Impact speed (km/s)Experimental ballistic curves (SPHIR)



1.6 1.8 2 2.2 2.4 2.6 2.8 3

Impact Speed (km/sImpact speed (km/s)Experimental ballistic curves (SPHIR)

440 C Steel spherical projectiles304 Stainless Steel plate targets


known controllable performance Response

inputs performance

measuresfunction

uncontrollableinputs

& unknown unknown unknowns




• Let denote averaging with respect to uncontrollable variables and unknown unknowns

• Let be the fluctuation

• Theorem [Lashgari Owhadi MO] A conservative• Theorem [Lashgari, Owhadi, MO] A conservative certification criterion is:

f i t l tt !• Simulations and experiments must be averaged

wrt uncontrolled variables and unknown unknowns

measure of experimental scatter!



wrt uncontrolled variables and unknown unknowns• Data scatter contributes to uncertainty!


Model D‹F›

thickness 1.82 mm2

obliquity 2.41 mm2

Di t

‹F› q ytotal 3.02 mm2

Modeling errorthickness 1.80 mm2

obliquity 4 50 mm2Diameters Modeling error D‹F-G›

obliquity 4.50 mm2

total 4.85 mm2

Experimentaltt D

total 7.78 mm2

scatter DG’

Mean values

Model E[F] total 3.30 mm2

Modeling error total 0.32 mm2

Steel-on-steel, 2.6 km/sPerforation and impactor

E[F-G]

• Perforation cannot be certified with any reasonablefid !



confidence!

Beyond McDiarmid - Partitioning

1 1

cliff!operating

range



Beyond McDiarmid - Partitioning



Beyond McDiarmid – Optimal UQ

• What is the least probability of failure upper bound given what is known about the system?

• Best probability of failure upper bound given that probability μ of inputs and response function G are in a set A:set A:

• Can be reduced, to finite-dimensional optimization (Choquet theory, representation of linear functionals by measures on extreme points moment problems )measures on extreme points, moment problems…)

• Example: Mean performance and diameter known• Explicit solutions for finite-dimensional inputs (Owhadi et



Explicit solutions for finite dimensional inputs (Owhadi et al.), optimal McDiarmid-type inequalities!

Concluding remarks…

• QMU represents a paradigm shift in predictive science:– Emphasis on predictions with quantified uncertaintiesp p q– Unprecedented integration between simulation and experiment

• QMU supplies a powerful organizational principle in predicti e science Theorems r n entire centers!predictive science: Theorems run entire centers!

• QMU raises theoretical and practical challenges:– Tight and measureable/computable probability-of-failure upper g / p p y pp

bounds (need theorems!)– Efficient global optimization methods for highly non-convex,

high-dimensionality, noisy functionsg y, y– Effective use of massively parallel computational platforms,

heterogeneous and exascale computing– High-fidelity models (multiscale effective behavior )


High fidelity models (multiscale, effective behavior…)– Experimental science for UQ (diagnostics, rapid-fire testing…)…


model-based rigorous uncertainty quantification in complex ... · – determination of mean...

Documents