1

Institu

tt for m

arin tek

nik

k

Error and Uncertainty Analysis

Experimental Methods in Marine Hydrodynamics

Week 37

Objectives:

• Understand the importance of getting a measure of the uncertainty of an

experimental result

• Learn how to find the uncertainty of the outcome of an experiment

Covers chapter 12 in the Lecture Notes

2

Institu

tt for m

arin tek

nik

k

Contents

• What is the difference between error and uncertainty?

• Introduction of some basic concepts

• Why uncertainty analysis?

• How to determine the uncertainty of measurements?

• Error propagation and sensitivity to errors on final results

• Calculation of total uncertainty of experiments

• Impact on final conclusions?

• Scale effects and other error sources

3

Institu

tt for m

arin tek

nik

k

Model resistance – all groups (2014)Based on the spreadsheets with averages made by the groups

4

Institu

tt for m

arin tek

nik

k

Residual Resistance – all groups (2014)

5

Institu

tt for m

arin tek

nik

k

Residual Resistance – all groups (2014)

6

Institu

tt for m

arin tek

nik

k

Repeated tests (2014)

Standard deviations Averages

Group # no of tests Speed RTm Speed RTm

1 5 0.00210 0.064 1.298 5.685

2 6 0.00188 0.457 0.800 1.943

3 6 0.00104 0.064 0.904 2.246

4 5 0.00072 0.284 1.200 4.294

5 5 0.00061 0.064 1.099 3.833

6 5 0.00073 0.077 0.800 1.988

8

Institu

tt for m

arin tek

nik

k

Model resistance – all groups (2014)Based on the spreadsheets with averages made by the groups

Standard deviations Averages

Group # no of tests Speed RTm Speed RTm

1 5 0.00210 0.064 1.298 5.685

2 6 0.00188 0.457 0.800 1.943

3 6 0.00104 0.064 0.904 2.246

4 5 0.00072 0.284 1.200 4.294

5 5 0.00061 0.064 1.099 3.833

6 5 0.00073 0.077 0.800 1.988

9

Institu

tt for m

arin tek

nik

k

Error

Suppose we know the true value

Then, this is the error of a single

measurement

The problem is that we usually don’t know the true value

(That is why we do the experiment)

10

Institu

tt for m

arin tek

nik

k

Definition of error and uncertainty

• Error is the difference between the measured value and the true value:

|Error| = | measured value – true value |

– Problem: The true value is very seldom known

• Uncertainty is the statistical representation of error

→ the expected error of a measurement

• Confidence interval: The range of probable values of an experiment

– Example: A 95% confidence interval of 2 N means that 95% of all readings of a particular measurement will be within 2 N from the ”true” value

11

Institu

tt for m

arin tek

nik

k

Bias and Precision Error

• Precision error: ”scatter” in the experimental results

– Found from repeated measurements

• Bias error: Systematic errors, not found from repeated measurements

• Replication level: How much of an experimental set-up that is repeated when finding the precision error

– Example of different repetition levels of a resistance test:

1. Only the test itself (running the same speed twice)

2. Repeat also the connection to the carriage

3. Repeat also the ballasting of the model

4. Making a new model, testing in the same tank

5. Making a new model at another towing tank, testing in a different tank (facility bias)

12

Institu

tt for m

arin tek

nik

k

Results of repeated resistance tests

1

1.2

1.4

1.6

1.8

2

2.2

0.1 0.15 0.2 0.25 0.3 0.35

Froude number Fn [-]

Resid

ua

l re

sis

tan

ce C

R*1

000 [

-]

Group 1 2004

Group 2 2004

Group 3 2004

Group 4 2004

Group 3&4 2005

Group 1&2 2005

All groups 2006

14

Institu

tt for m

arin tek

nik

k

Why do we want to know the uncertainty

of an experimental result?

15

Institu

tt for m

arin tek

nik

k

5000

7000

9000

11000

13000

15000

17000

19000

21000

23000

17 18 19 20 21 22 23 24 25

Ship speed [knots]

To

tal

Bra

ke

Po

we

r [k

W]

Prediction from model test

Contractual condition

Example: Speed prediction of a car carrier

Cancellation limit

Contract: The ship shall do 23 knots at

a brake power of 20 000 kW

16

Institu

tt for m

arin tek

nik

k

Uncertainty of PB calculated to 2%

5000

7000

9000

11000

13000

15000

17000

19000

21000

23000

25000

17 18 19 20 21 22 23 24 25

Ship speed [knots]

To

tal

Bra

ke

Po

we

r [k

W]

Prediction from model test

Contractual conditionUncertainty: 2% on PB

17

Institu

tt for m

arin tek

nik

k

Uncertainty of PB calculated to 10%

5000

7000

9000

11000

13000

15000

17000

19000

21000

23000

25000

17 18 19 20 21 22 23 24 25

Ship speed [knots]

To

tal

Bra

ke

Po

we

r [k

W]

Prediction from model test

Contractual condition

Uncertainty: 10% on PB

In this case there might be a problem!

18

Institu

tt for m

arin tek

nik

k

Example 2a: Comparison of experimentally

and numerically calculated RAO in heave acc.

Response amplitude operators

accelerations at pos AP

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

0 5 10 15 20 25 30

Wave period [s]

RAO Heave acc.

h 3 / za [m/s²]

VERES

Model test

Model test, mean

Conclusion: Calculations not in agreement with experiments

Uncertainty: 2.08%

19

Institu

tt for m

arin tek

nik

k

Example 2b: Comparison of experimentally

and numerically calculated RAO in heave acc.

Response amplitude operators

accelerations at pos AP

0.00

0.50

1.00

1.50

2.00

2.50

0 5 10 15 20 25 30

Wave period [s]

RAO Heave acc.

h 3 / za [m/s²]

VERES

Model test

Model test, mean

Conclusion: Calculations in agreement with experiments

Uncertainty: 30%

20

Institu

tt for m

arin tek

nik

k

mean true

1.0-0.95

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

-0.5 0 0.5 1 1.5 2 2.5 3 3.5

Result X

Gau

ssia

n d

istr

ibu

tio

nf(

X)

Bias error

Confidence interval

Calculation of precision error

2

221

2

X

f X e

21

Institu

tt for m

arin tek

nik

k

Calculation of precision error

• Assumption: if infinitely many, the measured

values follows a Gaussian distribution (called

the parent distribution) around a mean:

– is mean value and standard deviation

• Mean of N samples is:

• Standard deviation of N samples is:

• Standard deviation of the mean value of N

samples:

accuracy increases with repeated

measurements

2

221

2

X

f X e

1

1 N

jj

X XN

2

1

1

1

N

x jj

X XN

S

X

X

S

NS

22

Institu

tt for m

arin tek

nik

k

Calculation of precision error

• For the parent distribution the confidence interval of a sample is given

by:

– t 1.96 for a normal distribution

– is the confidence interval, = 0.95 when t 1.96

• For a finite number of samples N, σ and t of the parent distribution is

unknown

• Re-write eq. 1 as:

• Then is random and follow a Student’s t distribution with N-1

degrees of freedom

• The precision limit of a sample is now easily found from

Prob j jX t X t

Prob j

x

X

St t

j

X

X

S

x xP t S

Means: 95% chance that the

value is within ±1.96st.dev from

the true value

23

Institu

tt for m

arin tek

nik

k

The weight t for estimating confidence

intervals using Student’s t distribution

0

1

2

3

4

5

6

7

8

0 5 10 15 20 25 30

Degrees of freedom (N-1)

Weig

ht

t

95% confidence

99% confidence

Means: 95% chance that the

value is within ±tst.dev from the

true value

24

Institu

tt for m

arin tek

nik

k

Finding the precision limit

• The precision limit of a sample:

• Note that more than one sample is needed to calculate Sx:

• The precision limit of the average of N samples:

Repeated measurements are required to calculate the precision limit

of a single sample

Repeated measurements is a good (but time-consuming) way of

decreasing the precision error of the results

x xP t S

XX

SP t

N

2

1

1

1

N

x jj

X XN

S

In practical terms, the precision limit with 95%

confidence is two times the standard deviation.

When the standard deviation is calculated based

on a few samples, the precision limit is more

than two times …

25

Institu

tt for m

arin tek

nik

k

Repeated tests in Labtest 1 (2014)

Standard deviations Averages Precision limit Px

Group

#

no of

tests Speed RTm Speed RTm

Students t Speed RTm

1 5 0.00210 0.064 1.298 5.685 2.132 0.0045 0.136

2 6 0.00188 0.457 0.800 1.943 2.015 0.0038 0.921

3 6 0.00104 0.064 0.904 2.246 2.015 0.0021 0.130

4 5 0.00072 0.284 1.200 4.294 2.132 0.0015 0.605

5 5 0.00061 0.064 1.099 3.833 2.132 0.0013 0.136

6 5 0.00073 0.077 0.800 1.988 2.132 0.0016 0.165

There is 95% chance that the true value is within

4.2920.605 N for a speed of 1.2 m/s

This is only the precision limit. What about bias errors? What about the replication level

26

Institu

tt for m

arin tek

nik

k

0

1

2

3

4

5

6

7

8

0 5 10 15 20 25 30

Degrees of freedom (N-1)

Weig

ht

t

95% confidence

99% confidence

Typical number of repeated tests:

27

Institu

tt for m

arin tek

nik

k

Example – repeated resistance tests

Repeated resistance test with a conventional ship

model

41.3

41.4

41.5

41.6

41.7

41.8

41.9

42

42.1

1.7015 1.702 1.7025 1.703 1.7035 1.704 1.7045 1.705 1.7055

Model speed Vm [N]

Mo

del re

sis

tan

ce R

Tm

[N

]

41.45

41.5

41.55

41.6

41.65

41.7

0 5 10 15

Number of repetitions

Me

an

va

lue

[N

]

0

0.5

1

1.5

2

2.5

3

3.5

4

Sta

tis

tic

al v

alu

es

(s

t.d

ev

, t,

P)

Mean

st.dev

t

P

28

Institu

tt for m

arin tek

nik

k

Repeated tests in practice

1. Repeat all test conditions to reduce the uncertainty of the results

– Too time consuming (and expensive) to be done in practice (in marine

hydrodynamics), except on some special research projects

2. Repeat one (or a few carefully selected) test conditions to calculate the

precision error

– Recommended practice in all research projects

– Typically not done in routine commercial verification tests

3. Do a thorough uncertainty analysis, including repeated tests, of a

typical standard tests (for instance resistance), and use the results as

being representative for all standard tests of the same type

• Recommended practice in standardized commercial testing

29

Institu

tt for m

arin tek

nik

k

Chauvenet’s criterion for rejecting outliers

1 12(1 )

chauvenett F p

j xchauvenetX X St

Reject samples with larger

deviation from the mean than

given by:

F is the cumulative density

function of the normal

distribution

p=1-1/(2N)

30

Institu

tt for m

arin tek

nik

k

Estimating Bias Errors

• Can not be found from repeated tests

• No standard way of calculating bias errors

– That’s why we say ”estimating”

• Examples of bias errors:

– Calibration factors of the sensors

• Parts of the bias error can be estimated from the precision error of the calibration factor

– Geometrical accuracy of a ship model

• Find the geometrical accuracy (for instance by control measurements)

• Estimate the sensitivity of the measurement results from the geometrical deviations (that is the hard part!)

– Inaccurate calibration of (wave) environment

– Tank wall effects

• Blockage (it is a pure bias error)

• Wave reflections (can give both bias and precision)

«The

rest»

Can be

calculated

from

repeated

tests

Bias

Precision

Total

31

Institu

tt for m

arin tek

nik

k

How to reduce bias errors?

• Careful calibration of sensors and environment

• Well-designed test set-up

• Accurate manufacture of model

• Careful installation of the model

• Increase the replication level

• Correlation

– Empirically based correction factors

32

Institu

tt for m

arin tek

nik

k

What is the uncertainty of the final end result?

Example:

• We want to find the uncertainty of the full-scale speed-power

prediction of a ship based on model tests

• Assume we have found the uncertainty of the model resistance

measurement and of the measurement of model propeller thrust and

torque from repeated measurements (and by estimating bias errors)

• How do we find the uncertainty of the final prediction in full scale?

Two key concepts:

– Data reduction equations

– Error Propagation

34

Institu

tt for m

arin tek

nik

k

Data reduction equations

The mathematical equations describing the conversion from test

result to final result

• The data reduction can in general be written as:

• Taylor-expansion of the data reduction equation gives:

• The influence coefficient is defined as:

• The elemental error is then:

1 2, ,..., ,NX f Y Y Y

1

1

22 3

2

1( )

2

N

i i

N

i i

i i ii

i

X XX X Y Y O Y

Y Y

XX Y

Y

ii

X

Y

i i iii

Y YX

eY

35

Institu

tt for m

arin tek

nik

k

Error propagation

• Assuming elemental error sources are independent, total error is found

by summation:

• Total bias and precision errors are determined independently.

Total error is found as:

– For 95% coverage

– For 99% coverage

2

1

N

ii

e e

2 2BSe e e

S Be ee

36

Institu

tt for m

arin tek

nik

k

Propagation of errors in a resistance test.

37

Institu

tt for m

arin tek

nik

k

Monte-Carlo simulation

Data reduction

equations

Mea

sure

d v

aria

ble

s

Final result

Identify range of variation for each “measured” variable

Create many (typically 10 000) randomly different sets of “measured” variables

Gives you 10 000 slightly different final results

Analysis of the variation of final result, compared to the variation of inputs, shows the

sensitivity of uncertainty in measured values of the final result

Useful technique when the data reduction process is complicated

38

Institu

tt for m

arin tek

nik

k

Steps in an uncertainty analysis

1. Identify all error sources

2. Determine the individual bias error for each error source

3. Determine the precision error of each measured variable

4. Determine the sensitivity of the end result to error sources

5. Create the total precision interval

6. Create the total bias uncertainty

7. Combine the total bias and precision

8. Declare results from steps 4-6 separately in the report

40

Institu

tt for m

arin tek

nik

k

Summary of the uncertainty analysis

• We always need knowledge about the uncertainty of an experimental result– Often, this “knowledge” is based on “gut feeling”, experience, or a crude order-of-

magnitude estimate

– Increasingly, there is a demand for a formal analysis. Examples:• When verifying numerical calculation methods by comparison with experiments

• When it is very important to know that the experimental result is true within a known uncertainty band

• Error vs. Uncertainty:– Error is the actual difference between a result and the true value

– Uncertainty is the expected error – calculated based on statistics

• Problem: We don’t know the true value (that’s why we do the experiment!)

• Error estimates are commonly divided in two categories:– Precision errors – found from the scatter in repeated measurements

– Bias errors – systematic errors, that can not be found from repeated measurements

• To find the uncertainty of the final result (for instance ship resistance) one needs to consider how the experimental error propagates through the data analysis (error propagation)

– We have seen how to use the data reduction equations to set up an analysis of the error propagation

41

Institu

tt for m

arin tek

nik

k

Further reading

• ITTC has a number of Recommended Procedures for assessing the

uncertainty in different types of tests – see ittc.info

• ITTC now recommends following ISO/IEC Guide 98-3:2008 “Guide

to the expression of uncertainty in measurements”

• The treatment in this lecture mainly follows Coleman and Steele

“Experimentation and Uncertainty Analysis for Engineers” 2nd ed.

Wiley 1999

• A very comprehensive (and quite theoretical) treatment is given by

Bendat and Piersol “Random data – Analysis and Measurement

Procedures” 4th ed, Wiley, 2010

• And there is a lot more out there …

42

Institu

tt for m

arin tek

nik

k

Scale effects and other error sources

43

Institu

tt for m

arin tek

nik

k

How can the accuracy be increased?

• Improve calibration procedures– More careful

– Include more of the test set-up in the calibration

– Non-linear calibration relation

• Change to more accurate transducers– More expensive

– More sensitive (but then usually less robust)

• Re-design test set-up– For instance using larger model

• More careful test execution– For instance longer waiting times between runs

• Repeated measurements– Accuracy increases with √N

44

Institu

tt for m

arin tek

nik

k

Examples of Bias Error Sources

• Scale effects

• Model inaccuracies

• Errors in test set-up

• Calibration errors

• Errors due to environmental modeling

• Wave parameters and spectral shape

• Tank wall effects

45

Institu

tt for m

arin tek

nik

k

Model inaccuracies

• Inaccurate draught/ballasting

– Error is minimized by ballasting to correct weight, not to specified draught

• Model surface too rough

– Poor paintwork

– The surface finish deteriorates with timeIt is common for the resistance to increase about 2% after one year storage

• Inaccurate shape (production errors)

• Model deformations

– Could be the reason for the fact that model resistance tends to increase with time

46

Institu

tt for m

arin tek

nik

k

Errors in test set-up

• Influence of Model Connections (Example: resistance test)

– Towed at a position too high in the model

– Model not correctly aligned

– External forces acting, for instance strain in cables

– The force transducer might not measure the force exactly in the horizontal direction

• Uncertainty of calibrations

– The uncertainty in calibrations can be calculated

• Systematic errors in measurement systems

– Use end-to-end calibrations to find and correct such error(End-to-end calibrations means to calibrate the entire test set-up)

47

Institu

tt for m

arin tek

nik

k

Errors due to environmental modeling

• Wave parameters and spectral shape

– Always calibrate the waves used in an experiment

– Use the results of the calibrations rather than the specified wave

– Quality of wave makers and wave generation system is important!

– Wave reflections from tank walls is an important problem

• Effective wave damping is important

• Reflection will often limit the length of measurement

• Seiching and temperature layers in the water

– Might turn up also as precision errors

• For offshore testing, quality of modeled current is often critical

Seiching:

Standing wave in the tank or

basin

Termperature layers:

Might cause internal waves,

since the different layers have

different density

48

Institu

tt for m

arin tek

nik

k

Seiching – standing waves in the tank

Length Ltank

Depth h

Amplitude za

Horisontal velocity Vx

)sin()cos( kxta zz

)sin()sin( kxtgk

V ax

z

hg

LT Tank

2

•Wave elevation:

•Horizontal velocity:

•Wave period:

49

Institu

tt for m

arin tek

nik

k

Error from seishing on total resistance

- Example from the large towing tank

•Wave amplitude za = 1 cm

•Horizontal max velocity Vx = 0.03 m/s

•Carriage speed Vm = 1.5 m/s

•Total resistance: ½V2

•Induced max. Error: 4%

50

Institu

tt for m

arin tek

nik

k

Tank wall effects

• Usually, we want our test to represent our model in the open sea

• When we do the experiments in a model basin there will inevitably be

boundaries

• We must make sure that these boundaries doesn’t influence the results

• Types of influence:

– Blockage – influencing steady velocity and pressure around a forward-

moving model

– Wave reflections

• Generated waves are reflected from imperfect wave damping devices

• Reflected and radiated waves from the model are reflected from tank walls and

wave maker back to the model

51

Institu

tt for m

arin tek

nik

k

Tank wall effects - Blockage

• Effective speed is increased due to the presence of the model

Blockage correction

0

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0.008

0.009

0.01

0 0.5 1 1.5 2 2.5 3 3.5

Vm [m/s]

Ve

loc

ity

co

rre

cti

on

fa

cto

r

V/V

[-]

Scott

Schuster

53

Institu

tt for m

arin tek

nik

k

Tank wall effects - Wave reflections

Comparison between numerical and experimental results for first order

vertical wave forces on a hemisphere – Effect of tank wall interference

54

Institu

tt for m

arin tek

nik

k

Wave reflections from moving models

•Radiated waves=created by model motions

•Diffracted waves=incoming waves reflected

• by the model

2g

e

gc

•Wave group speed:

2

M gM M

w e

L cL L gUcrit

t B B

•This is why you should not run zero speed tests in a towing tank!

21 1

2

Mcrit

LgU

B

Ch. 7.5

55

Institu

tt for m

arin tek

nik

k

Ways to minimize the wave reflection

problem for zero and low speed

• Keep the model small relative to the size of the basin (or width of tank)

• Install effective wave damping devices

– Hardly possible along the sides of a towing tank

– Even the best wave beaches aren’t 100% effective at all wave lengths

• Limit the duration of the test

– Not much of an alternative for irregular wave conditions

– To be considered for regular wave and impulse wave tests

57

Institu

tt for m

arin tek

nik

k

Scale Effects

• Reynolds effects

– Frictional resistance

– Boundary layer thickness

– Different flow separation pattern

• Cavitation

– Important only for performance of foils and propulsors

• Surface tension, spray, Weber number

– Gives a limit for minimum model size

• Air pressure ratio

– The atmospheric pressure is not scaled

– Relevant for seakeeping testing of ships with air cushion

58

Institu

tt for m

arin tek

nik

k

Reynolds scale effects - Forces

• Froude scaling requires:

• When C is dependent on scale we are most likely having a

Reynolds scale effect

212

equal in model and full scaleF

CV A

(When V is scaled by square root of scale ratio)

59

Institu

tt for m

arin tek

nik

k

Reynolds Scale Effects – Skin Friction

60

Institu

tt for m

arin tek

nik

k

Correcting for Reynolds scale effect on full

scale resistance

61

Institu

tt for m

arin tek

nik

k

Reynolds Scale Effect on Lift of Foils

3

1

2 2

3

1

62

Institu

tt for m

arin tek

nik

k

Wake scale effect

Model Ship

Reynolds number

Boundary layer thickness

0.04 ( 0.04 ) Fs F

s m

Fm

C Cw t w t

C

63

Institu

tt for m

arin tek

nik

k

Reynolds Scale Effects – Drag forces

Squared

cylinder

221 DUCdF DD

Circular

cylinder

64

Institu

tt for m

arin tek

nik

k

Reynolds scale effects on propellers

• Scale effect on blade drag → propeller torque → KQ

– Due to Rn-effect on skin friction

– Possibly due to flow separation

• Scale effect on blade lift→ propeller thrust → KT

– Rn-effect on foil lift

• Minimize effect by requiring model Rn>2·105

• Established correction method (ITTC’78 method ):

D

Zc

D

PCCKK DSDMTPMTPS

.3.0

D

ZcCCKK DSDMQMQS

.25.0

65

Institu

tt for m

arin tek

nik

k

Reynolds scale effects – short summary

• On skin friction

– Established correction methods exists (friction lines)

• On pressure drag

– Depends on flow separation and vortex generation

– Established correction methods exist for simple cases (geometries)

• On flow fields (f.i. boundary layers)

– Established correction methods exists only for simple cases

– CFD (RANS codes) might play an increasing role

• On lift from lifting surfaces (foils and propellers)

– Commonly neglected effect

– Little data exists

– Depends on details in the geometry -> boundary layer development

66

Institu

tt for m

arin tek

nik

k

Summary of error sources

• Discussed sources of precision error

• Discussed sources of bias error:

– Scale effects

– Model inaccuracies

– Errors in test set-up

– Calibration errors

– Errors due to environmental modeling

– Wave parameters and spectral shape

– Tank wall effects