verification and validation of cfd asme

Verification and Validation of CFD Simulations

F. Stern1, R. Wilson1, H. Coleman2, and E. Paterson1

1. Iowa Institute of Hydraulic Research, University of Iowa; 2. Propulsion Research Center, University of Alabama in Huntsville

Tutorial2001 Fluids Engineering Division Summer Meeting

May 29 - June 1, 2001, New Orleans, Louisiana

Sheraton New Orleans Hotel

OutlineBackgroundApproachOverall Verification and Validation MethodologyVerification ProceduresValidation ProceduresExample for RANS CFD CodeConclusions

2

BackgroundDiscussion and methodology for estimating errors and uncertainties in CFD simulations has reached a certain level of maturity– Editorial policies and early recognition of importance and need

for “quality control” (ASME JFE, 1993, Vol. 115, pp. 339-340)

– Increased attention and recent progress on common terminology through published guidelines (AIAA, 1998)

– Advocacy and detailed methodology in recent textbook (Roache, 1998)

– Numerous case studies, including a special AIAA J. issue (e.g.,Mehta, 1998)

3

BackgroundProgress accelerated in response to the urgent need for achieving consensus on concepts and definitions and useful methodology and procedures– CFD is applied to increasingly complex geometry and physics

and integrated into the engineering design process

– Realization simulation-based design

– Other uses of CFD such as simulating flows for which experiments are difficult

In spite of progress and urgency, various viewpoints have not converged and current approaches fall short of providing practical methodology and procedures

4

ApproachComprehensive, pragmatic approach to verification and validation (V&V) methodology and procedures for estimating errors and uncertainties for industrial CFDAlready developed CFD code without requiring source code Specified objectives, geometry, conditions, and available benchmark experimental dataDeveloped for RANS CFD codes, but should also be applicable to boundary-element methods and some aspects LES and DNS

5

Approach

Definitions of errors and uncertainties consistent with experimental uncertainty analysisConcepts, definitions, and equations derived for simulation errors and uncertainties provide overall mathematical frameworkNumerical error treated both as deterministic or stochasticGeneralized Richardson extrapolation for J input parameters and correction factorsUse of quantitative estimates for errors and uncertainties constitute new V&V approach

6

Approach

Previous work on verification (Stern et al., 1996) extended and and combined with subsequent work on validation (Coleman and Stern, 1997)– Stern, F., Wilson, R.V., Coleman, H., and Paterson, E.,

“Verification and Validation of CFD Simulations,” Iowa Institute of Hydraulic Research, The University of Iowa, IIHR Report No. 407, September 1999 (in review ASME JFE)

Nearly two years experience through International Towing Tank Conference (ITTC) community and Gothenburg 2000 Workshop on CFD in Ship Hydrodynamics (Larsson et al., 2000)

7

Overall V&V Methodology Concepts and Definitions

Error, δ - difference between a simulation value (or an experimental value) and the truth

Error estimate, δ* - estimate of both sign and magnitude of error δUncertainty estimate, U - estimate of magnitude(but not sign) of error such that interval ±U contains true value 95 times out of 100

8


Simulation error, δS - difference between simulation S and truth T postulated as comprised of addition of modeling and numerical errors

The uncertainty equation corresponding to the error equation is

SNSMS TS δδδ +=−=

222SNSMS UUU +=

9


For certain conditions, the numerical error δSN can be considered as

– where δ*SN is an estimate of sign and magnitude of δSN and εSN is the error in that estimate

The corrected simulation value SC (numerical benchmark) is

With error and uncertainty equations

SNSNSN εδδ += *

SNSMSNC TSS εδδ ++=−= *

SNSMCS TSC

εδδ +=−= 222NSSMS CC

UUU +=

10


Verification - assessment of numerical uncertainty USN and when conditions permit, estimating the sign and magnitude of the numerical error δ*SNitself and the uncertainty in that estimate

Validation - assessment of modeling uncertainty USM by using benchmark data and, when conditions permit, estimating the sign and magnitude of the modeling error δSM itself

NSCU

11

Overall V&V Methodology Verification

Numerical errors decomposed into contributions from iteration number grid size, time step, and other parameters

Similarly,∑+==

∗∗∗ J

jjISN 1

δδδ

SNSM

J

jjIC TSS εδδδ ++=∑+−=

=

∗∗ )(1

∑+==

J

jjINS CCC

UUU1

222

∑=

+=+++=J

jjIPTGISN

1δδδδδδδ

∑=

+=+++=J

jjIPTGISN UUUUUUU

1

2222222

12

Overall V&V Methodology Validation

The comparison error E is defined by

– with δSM decomposed into errors from modeling assumptions δSMA and use of previous data δSPD

The uncertainty UE in the comparison error is

Standard methodology and procedures available for estimating UD (Coleman and Steele, 1999)

( )SNSPDSMADSDSDE δδδδδδ ++−=−=−=

22222SNSPDSMADE UUUUU +++=

13

2 2 2STE SPD SNU U U= +


Ideally, like to postulate if |E|<UE, validation achieved; however, no known method for estimating USMA

More stringent alternative is, |E|<UV, where UVis validation uncertainty

– USTE is the total estimated simulation uncertainty

If |E|<UV then validation achieved at UV levelIf |E|>>UV then

2 2 2 2 2 2 2 2V E SMA D SPD SN D STEU U U U U U U U= − = + + = +

SMAE δ≈

14


Definition of comparison error

S + U

E

UD

Ux

r

X

D

S

STE

15


Corrected comparison error

Corrected validation uncertainty

– SC and EC can either be larger or smaller than S and E

– and should be smaller than UE and UV

( )SNSPDSMADCC SDE εδδδ ++−=−=

222222NSSPDDSMAEV CCC

UUUUUU ++=−=

CEUCVU

16

Verification ProceduresConvergence Studies

Verification procedures based on

Parameter convergence studies– use multiple (m) solutions by varying the kth input

parameter ∆xk while holding all other parameters constant– input parameters (step sizes) defined such that for

finest resolution– uniform parameter refinement

• Not required

Solutions corrected for iterative errors∑

≠=

∗∗∗ ++=−=J

kjjjkCIkk mmmkmm

SSS,1

ˆ δδδ

)(1

∑=

∗∗ ++=J

jjICSS δδ

2312// kkkkk xxxxr ∆∆=∆∆=

0→∆ kx

17


Convergence studies require a minimum of m=3solutions corresponding to fine, medium, and coarse values for the kth input parameter

Solution changes ε for medium-fine and coarse-medium solutions and their ratio Rk are defined by

( )321

ˆ,ˆ,ˆkkk SSS

21 2 1ˆ ˆ

k k kS Sε = −

32 3 2ˆ ˆ

k k kS Sε = −

21 32k k kR ε ε=

18


Converging condition: 0 < Rk < 1, monotonic convergence and generalized RE is used to estimate Uk or and

Oscillatory condition: Rk < 0, solutions exhibit oscillations, which may be erroneously identified as convergent or divergent

– uncertainties estimated using upper (SU) and lower (SL) bounds of solution oscillation and require more than 3 solutions

Diverging condition: Rk > 1, solutions exhibit divergence and errors and uncertainties can not be estimated

CkU

)(21

LUk SSU −=

kδ ∗

19

Verification ProceduresMonotonic Convergence: Generalized RE

Generalized Richardson Extrapolation (RE) for convergent condition– Modified and numerical error equations show error terms

are in the form of a power series expansion in input parameter

– Subtraction of multiple solutions eliminates terms and provides equations for SC, p(i)

k, and g(i)k (assuming p(i)

k, and g(i)k

independent ∆xk)

– Since each term contains 2 unknowns, m=2n+1 solutions required (i.e., for n=1, m=3 and for n=2, m=5, etc.)

∑∑≠=

∗

=

+∆+=J

kjjj

n

i

ik

pkCk m

ik

mmgxSS

,11

)()(

)(ˆ δ

∗mkδ

∗mj

δ

20

Verification ProceduresMonotonic Convergence: Generalized RE

For m=3 solutions, only leading term estimated

– for sufficiently small ∆xk solutions in the asymptotic range such that higher-order terms negligible

– achieving asymptotic range for practical geometries and conditions not possible and m>3 undesirable from resource point of view

21

1 1 1k k

kk RE p

krε

δ δ∗ ∗= =−

( )( )

32 21ln /

lnk k

kk

pr

ε ε=

21

Verification ProceduresEstimating Errors and Uncertainties using Generalized RE

with Correction FactorsResults for analytical benchmarks (1D wave and 2D Laplace equations) show that when solutions not in asymptotic range correct form but pkpoorly estimatedAnalysis results suggests concept of correction factors Ck– provide quantitative metric to determine proximity of

solutions to asymptotic range– account for effects of higher-order terms– use for defining and estimating errors and uncertainties

REδ ∗

22


with Correction FactorsMultiplication by Ck provides estimate accounting for effects higher order terms

Ck based on leading and first two terms

– where pkest, qkest are improved estimates of orders of accuracy (e.g., modified equation or simplified geometry with similar grid expansion)

– Ck<1 or Ck>1 indicates that the leading-order term over predicts or under predicts the error, respectively

11)1(

−−

=estk

k

pk

pk

k rrC

( ) )1)(()1)(/(

)1()1)(/( 12231223)2(

−−

−−+

−−

−−=

estkestkestk

kestk

kk

estkestkestk

kestk

kk

qk

qk

pk

pk

pk

pk

qk

pk

pk

qk

k rrrrr

rrrrr

Cεεεε

21

1 1( )

1)k k

kk k RE k p

k

C Cr

εδ δ∗ ∗= =

−

∗1kδREδ ∗

23


with Correction FactorsFor Ck<1 or Ck>1 and lacking confidence, Uk is estimated, but not and Ukc

For Ck≈1 and having confidence, and Ukc are estimated

– In the limit of the asymptotic range, Ck=1, = , and Ukc=0

∗∗ −+=11

)1(kk REkREkk CCU δδ

21 21

1 1 1 1k k th

k kk k RE k p p

k k

C Cr r

ε εδ δ∗ ∗

= = = − −

1(1 )

C kk k REU C δ ∗= −

kδ ∗

kδ ∗

1kδ ∗1kREδ ∗

24


with Factors of SafetyRoache (1998) proposes factor of safety approach

Can be extended for estimate of corrected simulation numerical uncertainty

In this approach, fixed percentage of three-gird error estimate used to define uncertainty: FS=1.25 for careful grid study otherwise =3– Results for analytical benchmark show FS overly

conservative compared to Ck approach

1kk S REU F δ ∗=

( )1

1C kk S REU F δ ∗= −

25

Verification ProceduresFundamental and Practical Issues

Fundamental Issues– Convergence power series expansion– Assumption p(i)

k, and g(i)k independent ∆xk

– Estimating based on theoretical values or solutions for simplified geometry and conditions with stretched grids

– Ck vs. FS or other approachesPractical Issues– For complex flows with relatively coarse girds, solutions far from

asymptotic range such that while some variables convergent other variables may be oscillatory or even divergent

– pk shows variability between different variables same grid study and same variables different grid studies

– More than 3 grids required– Selection parameter refinement ratio– Nature asymptotic range for practical applications unknown– Interpretation results important since limited experience for

guidance

estkp

26

Validation ProceduresSix combinations of |E|, UV, and Ureqd (program validation requirement)1. |E| < UV < Ureqd2. |E| < Ureqd < UV3. Ureqd < |E| < UV4. UV < |E| < Ureqd5. UV < Ureqd < |E| 6. Ureqd < UV < |E| In cases 1, 2, and 3, validation achieved at UV levelIn cases 4, 5, and 6, validation not achieved at UV. If E>>UVthen

In cases 1 and 4, validation successful programmatically Similar conclusions for corrected simulation results

SMAE δ≈

27

Example for RANS CFD Code

CFDSHIP-IOWASeries 60 cargo/container ship– ITTC benchmark data (Toda et al., 1992)– Conditions

• Froude number Fr = 0.316• Reynolds number Re = 4.3x106

– V&V for resistance CT (integral variable) and wave profile ζ (point variable)

28

Example for RANS CFD CodeGrid Studies

Grid refinement ratio m=4 grids with systematic grid refinement in each coordinate direction enables two separate grid studies: grids 1-3 (GS1) and grids 2-4 (GS2)

2Gr =

Table 1. Grid dimensions and y+ values for grid refinement studies.

Grid Grid

Dimensions

Total Number

of points y+

1 287x78x43 876,211 0.7

2 201x51x31 317,781 1

3 144x36x22 114,048 1.4

4 101x26x16 42,016 2

29

Example for RANS CFD CodeGrid Studies

Grids and wave contours

30

Example for RANS CFD CodeVerification for Resistance

Verification performed for iterative and grid convergence

Limiting order of accuracy estimated as formal order of accuracyof the CFD code

Iterative convergence negligible (i.e., at least one order of magnitude smaller grid convergence)

Grid 1 UI=0.07%S1

Grid 2 UI=0.02%S1

Grid 3 UI=0.03%S1

Grid 4 UI=0.01%S1

S1=solution on finest grid

2 2 2SN I GU U U= +

2.0est thk kp p= =

31


Iteration history: solution change; forces; and magnified CT last two periods: fine grid; 4 order of magnitude drop in residuals; UI=.07%S1

Iteration

Res

idua

l

0 5000 10000 15000 2000010-7

10-6

10-5

10-4

UVWP

(a)

Iteration12000 14000 16000 18000

0

0.002

0.004

0.006

0.008

(b)

CFCPCT

Iteration12000 14000 16000 18000

0.00504

0.00505

0.00506

SL=5.046x10-3

SU=5.053x10-3

(c)

32


Monotonic convergence; variability pG and counter expectation

Table 2. Grid convergence study for total CT, pressure CP, and frictional CF resistance (x10-3) for Series 60.

Grid S4 (grid 4) S3 (grid 3) S2 (grid 2) S1 (grid 1) Data

CT

ε 6.02 5.39

-10% 5.11

-5.2% 5.05

-1.2% 5.42

CP

ε 1.88 1.61

-14%

1.60 -0.6%

1.60 0.0%

CR = 2.00

CF

ε 4.14 3.69

-11% 3.51

-4.9% 3.45

-1.7% 3.42 ITTC

% SG.

Table 3. Verification of CT (x10-3) for Series 60. Study RG pG CG

1 (grids 1-3)

0.21 4.4 3.7

2 (grids 2-4)

0.44 2.3 1.3

% SG

33


CF=70%CT and within 1%CF(ITTC)For GS1, CP grid independent and CF convergent with pG>pth

For GS2, both convergent with pG>pth

Fact that CP and CF converge with different pG and depend different physics helps explain variability

Table 4 Verification of CP and CF (x10-3) for Series 60. CP CF Study

RG pG CG RG pG CG

1 (grids 1-3)

0.00 - - 0.33 3.2 2.0

2 (grids 2-4)

0.04 9.5 26 0.40 2.6 1.5

% S1

34


Error and uncertainty values reasonable in consideration number grid pointsFS approach less conservative, which is opposite results for analytical benchmark

Table 5. Errors and uncertainties for CT (x10-3) for Series 60. CT uncorrected CT corrected Grid

GU (CG) GU (FS) *Gδ

CGU (CG) CGU (FS) SC

1 2.1% 0.5% 1.2% 0.9% 0.1% 4.99 2 6.7% 5.6% 5.5% 1.1% 1.1% 4.83

% S1

35


Next finer grid requires 2.4M grid pointsExpect but with UI similar order magnitudeFrom resource point of view, accept present Scand for finest gridBased on overall verification: four solutions display monotonic convergence with notwithstanding variability pG which precludes complete confidenceAdditional solutions desirable for knowledge asymptotic range for practical applications

2kr =

CG GU U→

cGU

0Gδ ∗ >

36

Example for RANS CFD CodeValidation for Resistance

Uncorrected solution– For GS1, and UD>USN

– For GS2, so CT validated at 7%D and USN >UD

Corrected Solution– For GS1 and GS2, and

Table 6. Validation of uncorrected total resistance for Series 60. Grid E% UV% UD% USN%

1 6.8 3.1 2.5 1.9

2 5.7 6.7 2.5 6.3

%D.

Table 7. Validation of corrected total resistance for Series 60. Grid CE %

CVU % DU % CS NU %

1 7.9 2.6 2.5 0.8 2 11 2.7 2.5 1.0

%D.

7%V SMAE U E Dδ> ⇒ ≈ =

VE U<

8%cc V c SMAE U E Dδ>> ⇒ = =

cS N DU U<<

37

Example for RANS CFD CodeV&V for Wave Profile

Wave height at free-surface hull intersectionConvergence ratio and order of accuracy defined using L2 norms of and

As with CT, UI<<UG such that USN=UG

21kε32kε

21 322 2/G G GR ε ε=

( )32 21 22ln /

ln( )G G

GG

pr

ε ε=

38

Example for RANS CFD CodeVerification for Profile Average Wave Profile

For both GS1 and GS2, monotonic convergenceUncertainties GS1=1/2GS2Trends pG consistent expectationUncertainty values reasonable in consideration number of grid points

Table 8. Profile-averaged verification results for wave profile for Series 60.

Study RG pG CG GU CGU

1 (grids 1-3)

0.64 1.3 0.56 2.0% 0.9%

2 (grids 2-4)

0.68 1.1 0.47 4.1% 2.2%

%ζmax .

39

Example for RANS CFD CodeValidation for Profile Average Wave Profile

Uncorrected solution– For GS1, not validated at E=5.2%ζmax but margin small– For GS2, nearly validated at 5.6%ζmax

Corrected Solution– Not validated but margins relatively small and cS N DU U<<

Table 9. Profile-averaged validation results for uncorrected wave profile for Series 60. Grid E% UV% UD% USN%

1 5.2 4.2 3.7 2.0

2 5.6 5.5 3.7 4.1

%ζmax .

Table 10. Profile-averaged validation results for corrected wave profile for Series 60. Grid CE %

CVU % DU % CS NU %

1 5.6 3.8 3.7 0.9 2 6.6 4.3 3.7 2.2

%ζmax .

40

Example for RANS CFD CodeV&V for Wave Profile

Regions not validated indicate under prediction crests and troughs

x/L

ζ/L

0 0.25 0.5 0.75 1

-0.01

0

0.01 Grid 1 (287x71x43)Grid 2 (201x51x31)Grid 3 (144x36x22)Grid 4 (101x26x16)Toda et al. (1992)

x/L

ζ/L

0 0.25 0.5 0.75 1

-0.01

0

0.01

(a)x/L

E

0 0.25 0.5 0.75 1

-0.2

-0.1

0

0.1

0.2 E=D-S+UV

-UV

(b)

x/L

E C

0 0.25 0.5 0.75 1

-0.2

-0.1

0

0.1

0.2 EC=D-SC

+UV

-UV

(c)

x/L

E

0 0.25 0.5 0.75 1

-0.2

-0.1

0

0.1

0.2 E=D-S+UV

-UV

(d)

x/LE C

0 0.25 0.5 0.75 1

-0.2

-0.1

0

0.1

0.2 EC=D-SC

+UV

-UV

(e)

41

Example for RANS CFD CodeOverall V&V Conclusions for CT and Wave Profile

CT and ζ not validated due to 8%D and 6%ζmaxmodeling errorsImprove modeling assumptions for dynamic sinkage and trim, free surface boundary conditions, and turbulence for validation at 3%D and 4%ζmaxlevelsReduction level validation UV requires reduction UD

42

Conclusions

V&V methodology and procedures successful in assessing levels of verification and validation or modeling errors. For practical applications many issues– Solutions far from the asymptotic range

– Analysis and interpretation results important in assessing variability for order of accuracy, levels of verification, and strategies for reducing numerical and modeling errors and uncertainties

43

Conclusions

Future work on verification should focus on both fundamental and practical issues, as previously discussedV&V methodology and procedures should facilitate– Documented V&V studies for transition CFD codes to

design– Sufficient number of documented solutions should enable

accreditation of CFD code for a certain range of applications

44

Analytical Benchmarks

1D Wave equation

Exact analytical solution

( ) ( )

0),(:

exp0,:

0)()()(

2

0

=−∞

−=

=∂∂

+∂∂

===

tABC

BxAxAIC

xAc

tAALMLTL AMT

( )2

0( , ) expx ct

A x t AB

−= −

x

S(x,

t)

-1 0 1 20

0.5

1

1.5

c=1

t=0t=1

45


Simulation error and uncertainty

Corrected Simulation error and uncertainty

Verification

S SNS Aδ δ= − = 2 2S SNU U=

2 2C CS S NU U=

CS C SNS Aδ ε= − =

CC C S NE A S U= − <SNE A S U= − <

46


Verification results 1st order solution 1D wave equation

∆x=∆t/2 10-510-410-310-20.4

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2(b)

(A-S)/δ*RE

C(1)SN

C(2)SN

∆x=∆t/2

Erro

r

Ord

er,p

10-510-410-310-2

0

0.05

0.1

0.15

0.2

1

2(a) A-S

δ*RE

C(1)SNδ*

REC(2)

SNδ*RE

p

∆x=∆t/2

Erro

r

10-510-410-310-2

10-3

10-2

10-1

100

|A-S|

U(1)SN=|CKδRE|+|(1-CK)δRE|

U(2)SN=|CKδRE|+|(1-CK)δRE|

USN=FSδRE, FS=1.25

(d)

∆x=∆t/2

Erro

r

10-510-410-310-210-5

10-4

10-3

10-2

10-1 |A-SC|

U(1)SCN=|(1-CK)δRE|

U(2)SCN=|(1-CK)δRE|

USCN=(FS-1)|δRE|, FS=1.25

(c)

verification and validation of cfd asme

Documents

procedures cfd

validation of cfd simulationsf

industrial cfd

validation vv methodology

verification stern

rans cfd codes

practical methodology

detailed methodology